This post is related to Oracle HTTP Server (OHS) performance issue from our Oracle Fusion Middleware Training (next batch starts on 30th Aug, 2015) where we cover OHS on Day2 (Installation, Configuration, High Availability, Troubleshooting, integrating OHS with WebLogic as proxy etc) .
One of the trainee from our previous batch asked about the common issues that Oracle Fusion Middleware Administrator encounters. We recently implemented Oracle Fusion Middleware including OHS for one of our customer where file system on OHS server presented from SAN as NFS mount.
Error message while accessing the OHS Instance on Enterprise Manager “Failed to invoke operation Load on MBean”
After checking the ohs log file ($ORCALE_INSTANCE/ diagnostics/ logs/ OHS/ ohs1), I got the exact error behind the issue that was ” apr_proc_mutex_lock failed. Attempting to shutdown process gracefully”
[2015-05-06T08:18:23.8610+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22680] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:23.8628+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22679] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:23.8863+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22681] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:23.8894+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22678] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:24.8024+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22867] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
[2015-05-06T08:18:24.8506+00:00] [OHS] [INCIDENT_ERROR:10] [OHS-9999] [core.c] [host_id: example.company.com] [host_addr: 192.168.1.141] [pid: 22872] [tid: 140414451361536] [user: oracle] [VirtualHost: main] (5)Input/output error: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.
Root Cause : http.lock file (located in default location at $ORACLE_INSTANCE/ diagnostics/ logs/ OHS/ ohs1 ) was under file system on NFS mount causing performance issue which in turn causing OHS to fail. It is recommended to keep OHS lock file under Local Disk and not on the shared storage. If the $ORACLE_INSTANCE is present on NFS mount then the above error will occur as any delay in response from OHS will force OPMN process to restart OHS.
Solution: Change the http.lock file location to Local_Disk (like /tmp or any other place on local disk) which is not NFS mounted and update that location under mpm_prefork_module and mpm_worker_module in httpd.conf (located in $ORACLE_INSTANCE/ config/ OHS/ ohs1) file as below
LockFile ” <LOCAL DISK_PATH>”
- My Oracle Support Note 1460851.1 (NFS Locking Issues)
Oracle Fusion Middleware
Test your knowledge on Oracle Fusion Middleware or Use them to find your dream Job as Oracle Administrator
If you want to learn more about Oracle Fusion Middleware products, register for our Oracle Fusion Middleware Training (Register before 24th August and Get Discount of 100 USD Apply code: F1OFF at time of checkout)
Note: We provide 100% Money back guarantee so in unlikely case of you being not happy after 2 sessions, just drop us a mail before third session and We’ll refund FULL money.
The post Deploying OHS on NFS ? Think again : Locking/Performance Issue appeared first on Oracle : Design, Implement & Maintain.
This post is from our Oracle Apps DBA Training where trainees install Oracle E-Business Suite (R12) on our servers remotely.
We use Oracle Linux 5.5 to install Oracle Apps and on installing version 12.1.1 installation completes but at post install check failed with above screen shot (Post validation checks failed for HTTP Server).
If you get error like above then
1. Check status OHS logs under OPMN (OHS is managed by OPMN)
In log $LOG_HOME/ ora/ 10.1.3/ opmn/ HTTP_Server~1.log (where LOG_HOME is $ORACLE_BASE/inst/apps/$SID_[hostname]/logs )
/u01/oracle/PRD1211/inst/apps/PRD1211_iamdemo07/ora/10.1.3/Apache/Apache/bin/apachectl startssl: execing httpd
/u01/oracle/PRD1211/apps/tech_st/10.1.3/Apache/Apache/bin/httpd: error while loading shared libraries: libdb.so.2: cannot open shared object file: No such file or directory
Looked at Bala’s blog and
yum install gdbm
2. Created softlink for libdb.so.2 pointing to libgdbm.so.2
ln -s /usr/lib/libgdbm.so.2 /usr/lib/libdb.so.2
3. Start OHS (Set environment & then use OPMNCTL to start OHS)
. $ORACLE_BASE/ apps/ apps_st/ appl/ APPS[SID]_[hostname.env]
4. Click on retry on post install validation steps
- MyOracle Support784162.1 OHS 10.1.3 Fails to Start on Linux – ” apachectl startssl .. error loading shared libraries : libdb.so.2 “
Register before 24th August for next Oracle Apps DBA (R12) training batch and get 300 USD off .
The post EBS R12 (12.1) installation failed at Post Install Checks on Linux : libdb.so.2 appeared first on Oracle : Design, Implement & Maintain.
This is third part in series How to become Oracle Apps DBA from our Oracle Apps DBA (R12.2) Training (next batch starts on 8th Aug and only 1 seat left, Registration closes in 72 hours). After Architecture and Installation of Oracle E-Business Suite (R12.2) next topic you should learn in Oracle Apps DBA Training is Patching.
There are major changes around Patching in Oracle EBS 12.2 so if you are an Apps DBA working on R12 or 11i then pay special attention to this topic.
Here are some of the key points we are going to cover in Apps DBA Training
- Type of Patches in Oracle EBS
- Two files system (fs1 & fs2) in EBS 12.2 to support Online Patching
- Edition Based Redefinitions – EBR
- Version editions during patching – Run, Patch, Old
- ADOP – New Patching Tool in EBS 12.2
- Patching Phases in 12.2 : Prepare, Apply, Finalize, Cutover, Cleanup
- Useful notes related to Online Patching
- Lab Exercises where you’ll apply patches on EBS 12.2 installed by you on previous day.
Register for Oracle E-Business Suite Apps (R12.2) DBA Training (Only 1 seat left before we close registration so Hurry Up to avoid disappointment).
This is second part in series How to become Oracle Apps DBA from our Oracle Apps DBA (R12) Training (next batch starts on 8th Aug and only 3 seats left). In first part, I mentioned that first and foremost thing you need to understand is Architecture of Oracle E-Business Suite (WebLogic is introduced in WebLogic 12.2 so that is another topic an Apps DBA should learn) .
After architecture, You should look at installation of Oracle Apps (E-Business Suite). Refer to Oracle E-Business Suite 12.2 installation guide here
Here are key points for Oracle Apps R12 installation
- You can use Express Install or Normal Install (Express Install is for quick install with default values like username/mount points etc )
- You start R12 installer as root user on Unix
- You can do single user or multi user install (in multi user Application Tier and Database Tier file system are owned by different user)
- From 12.2 onwards you can configure Oracle RAC for database during Oracle Apps Installation (In prior release RAC was not possible during R12 installation. You change database to RAC after installation)
- 12.2 uses two application tier file system (FS1 & FS2) – Run Time & Patch Time so space requirement on 12.2 is slightly higher than previous R12 versions.
Attend Oracle Apps DBA (R12 – 12.2) training to learn from experts and know more about Oracle E-Business 12.2 installation
Get 100 USD off by registering by 4th Aug for Oracle E-Business Suite Apps (R12.2) DBA Training and use code A1OFF at time of checkout (We limit seats per batch and only 4 seats are left so register early to avoid disappointment).
The post Oracle Apps DBA Training : How to become Apps (R12) DBA Part II : Installation appeared first on Oracle : Design, Implement & Maintain.
Database validation failed while applying EBS R12.2 application patch : Oracle Apps DBA Training : Patches
This post is from our Oracle Apps DBA – R12 Training ( next batch starts on 8th Aug, 2015) where we cover installation, patching, cloning and other maintenance tasks. We also cover upgrade of Oracle E-Business Suite 12.2.0 to 12.2.4 to show how to apply patches and new patch process in Oracle Apps.
One of the trainee from previous batch encountered issue while applying EBS R12.2 application patches 19330775 and 20677045 where command used is highlighted below. The new utility to apply patch is adop (prior to 12.2 you apply patches using adpatch)
adop phase=apply patches=19330775,20677045 hotpatch=yes merge=yes
But patch was failing with error message like:
“adop is not able to detect any valid application tier nodes in ADOP_VALID_NODES table. Ensure autoconfig is run on all nodes.”
So we run autoconfig (there is dedciated topic on Autoconfig in our Oracle Apps/R12 DBA Training) on both Apps Tier & DB Tier by using below commands
- Stop the application tier and Database tier
- Autoconfig on Database tier
a) Source the Oracle Database env
b) cd $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME
c) sh adautocfg.sh
(Check the autoconfig logfiles for any error at $ORACLE_HOME/appsutil/log/$CONTEXT_NAME)
- Autoconfig on Apps Tier
a) Please source the APPS environment
b) cd $ADMIN_SCRIPTS_HOME
c) sh adautocfg.sh
After running autoconfig on both the tiers, we again run the command to apply the patch. This time patch failed to execute some SQL statements and showing below error messages:
[Error] Failed to execute SQL statement :
[Error] Error Message :
[UNEXPECTED] Error occurred while performing database validations
To fix this issue, apply database patch 17693770 as per the instructions mentioned in Readme.
After applying 17693770 patch on database tier, issue got resolved and we were able to apply the patches successfully.
You can learn more from our expert team by registering to Oracle Apps DBA Training (next batch starts 8th Aug 2015- register early before 30th July to get 100 USD OFF)K21 Technologies provide Full Money back Guarantee (If you are not happy after 2 sessions then you can ask for Full Money Back )
For further details and registration check
After long gap I’m start writing blogs and I’m feeling for that.
Today I have faced login issue in WNA setup environment.
Requirement is user would need to login via WNA fallback authentication and access to the OAM WNA protected resources but it login request landed into error page “Account locked or disabled”.
From oam-server1.out logs
Note: If you are not able to see below then you should enable Kerberos trace level.<Jul 21, 2015 6:27:52 PM AEST> <Error> <oracle.oam.plugin> <BEA-000000> <Defective token detected (Mechanism level: GSSHeader did not find the right tag) GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag) at sun.security.jgss.GSSHeader.<init>(GSSHeader.java:80) at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:287) at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:267) at oracle.security.am.plugin.authn.SPNEGOLoginModule$1.run(SPNEGOLoginModule.java:139) at javax.security.auth.Subject.doAs(Subject.java:394) at oracle.security.am.plugin.authn.SPNEGOLoginModule.login(SPNEGOLoginModule.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) Normally this issue appears to be that something different from a Kerberos or NTLM token is being sent by the Microsoft IE browser client machine.
OAM only accepts Kerberos or NTLM tokens for now.
We noticed browser was sending the following token when accessing in company network domain.
And it keeps sending this similar like “Authorization: Negotiate” string over and over.
This is not a standard NTLM value, as normally when we review the headers we would expect to see either:
Authorization: Negotiate TlRMTVNTUAABAAA…. (NTLM)
Authorization: Negotiate YIIGeAYGK…(Kerberos)
then this will still not work for OAM WAN Fallback, since the token received by OAM Server is NOT an NTLM token like, but appears to be more related to a NEGOEXTS token, which the Windows 7 clients sometimes send.
So, the token was not sent correctly by the browser to OAM server.
On the UNIX host, use kinit on your user account and use klist to verify that you have a ticket to the HTTP/DOMAIN.NAME@REALM.NAME principal or not.
In our cause we have encountered below exceptionkinit(v5): Client not found in Kerberos database while getting initial credentials
We have found a DNS issue for application OAM hostname. OAM VIP host name was resolving to different hostname and Keytab was created based on VIP hostname not actual hostname different and frontend host which is critical specially for creating a keytab
Re-generated the keytab for DNS resolve hostname as follow
ktpass -princ HTTP/DOMAIN.NAME@REALM.NAME
-mapuser aurdev\srv-oam-iap1 -pass <Password> -out master.keytab -kvno 0
Copy the new keytab into <Oracle Home>/server/config/ and restart OAM server.
Hope above information helped you to get out of the issues.
I started this blog 9 years back with first post as How to become Oracle Apps DBA (back then it was 11i) and with 225 comments, this is still the most common question I get in mail or on this blog.
We are starting our new batch for Oracle Apps DBA training (R12.2) from August 8, 2015 and first thing we cover is Architecture of Oracle E-Business Suite. If you are learning (getting trained) on Oracle E-Business Suite on your own then first thing you should learn is Architecture of Oracle Apps.
As shown below Oracle E-Business suite is Three Tier Architecture
a) Database Tier : With Oracle Database where data resides
b) Application Tier : With Application & Web Server where business logic resides
c) Client Tier : browser based client from where end user access application
Note: Till Oracle E-Business Suite R12.1 (prior versions include 12.0 & 11i), Application Tier uses 10g Application Server (or 9 for some versions of 11i). From Oracle E-Business Suite 12.2 onwards Application Tier is deployed on Oracle WebLogic Server as application Server.
You can get more information on Architecture of Oracle E-Business Suite in Concepts Guide or learn it from our expert team by registering to Oracle Apps DBA Training (starting on 8th August) where Day1 coversArchitecture and File System
- Architecture of R12.2
- Changes in Oracle Apps from previous version
- Requirement/Hardware Sizing Guidelines
- File System Overview
- Benefit of New Architecture
- File System including Changes from previous version
- Provide one working instance of R12.2 to the Trainee with Front end and backend access
- Get comfortable with the Terminology/File system/Environment Variables
- Understand the Architecture via Navigation
Get 200 USD off by registering before 20th July and use code A2OFF at time of checkout (We limit seats per batch to register early to avoid disappointment).
The post How to become/learn Oracle Apps DBA R12.2 : Part I appeared first on Oracle : Design, Implement & Maintain.
- All the Freshers, Newbies or may be who want to enter Oracle Applications Area.
- Who is into Core DBA from years and want new technology to learn.
For further details check
The post Learn Oracle Apps DBA (R12) with us:Training Starts on 8th of August appeared first on Oracle : Design, Implement & Maintain.
This post covers procedure installs the Java Development Kit (JDK) for 64-bit RPM-based Linux platforms, such as Red Hat and SuSE, using an RPM. This post is from our Oracle Fusion Middleware (FMW) or Oracle Access Manager (OAM) training where we provide dedicated machine to trainees to practice but if you need to install similar setup on your local machine (We use Oracle Virtual Box with Oracle Linux 5.5 ).
You must be login as root user to install this installation (Assumption is that you are installing JDK on 64 bit Linux)
1. Download the JDK software from here (jdk-7u60-linux-64.rpm)
2. The installation process should be carried out with the “root” user.
su – root
when prompted for password, enter the root password.
3. Navigate to the directory where your JDK software is downloaded
4. Install the package using the command : rpm -ivh <package_name>
rpm -ivh jdk-7u60-linux-x64.rpm
Note: This step will install JDK 1.7 under /usr/java/jdk1.7.0_60
5. To verify the version of java, navigate to the directory /usr/java/jdk1.7.0_60/bin and check javac and java versions. The version should be the latest installed JDK version.
Note: JDK is default installed under the directory /usr/java/jdk1.7.0_60
6. Delete the .rpm file if you want to save disk space.
7. Exit the root shell. No need to reboot.
If you are part of our training program and have not yet registered for closed Facebook Group then send request and post any technical queries.
The post Installation steps of JDK 7 for Linux for Oracle Fusion Middleware appeared first on Oracle : Design, Implement & Maintain.
This post is coming from our Oracle Fusion Middleware Training where we cover Oracle WebLogic Server on Day1 . One of the performance issue that commonly encountered in poorly written application (or on not so performant Fusion Middleware infrastructure) is Stuck Threads.
Stuck Threads in WebLogic Server means a thread performing the same request for a very long time and more than the configurable Stuck Thread Max Time in WebLogic .
Thread dumps are diagnosis information that is used to analyse and troubleshoot performance related issues such as server hangs, deadlocks, slow running, idle or stuck applications etc.
How to generate Thread dumps?
In this post, I will walk you through the steps to generate Thread dumps of a server using operating system (O.S.) commands.
1. Start the server from command line script (using nohup). Let us take managed server as an example for which we need to generate the thread dumps so start the server using script as shown below.
nohup ./startManagedWeblogic.sh <Server_name> &
2. Now identify the PID (java Process ID) for the managed server using the below command:
ps auxwww | grep –i java | grep –i <server_name> (This command is for Solaris)
3. Now run the below command to create the thread dump.
kill -3 <PID>
(This will send a signal to the process whose dump we require. This signal causes the Java Virtual Machine to generate a stack trace of the process.)
This command will create thread dump in the nohup.out file (where we started the managed server)
4. Open the nohup.out file to see generated thread dumps:
- How To Take Thread Dumps With WLST (Doc ID 1274713.1)
- How to get java thread dump (stack trace) of an application deployed in Weblogic? (Doc ID 1468660.1)
The post WebLogic Server (FMW) : Generating Thread Dumps using OS commands appeared first on Oracle : Design, Implement & Maintain.
I had convinced myself the Apple Watch was an overpriced fitness band and that it wasn’t for me and was set to get a Garmin to track my running instead. Then out of the blue I was given an Apple Watch. So you can certainly put me down as a cynic, but I certainly like to think I am open minded, so here are my thoughts after a week with the watch.
The experience of getting it set up was surprisingly frustrating, I had to upgrade my phone to iOS 8 before I could activate the watch and that meant deleting things to free a few Gb of memory (to upgrade my Operating System, really?). So everything had to wait until after I got home and backed up my phone.
First I got this rather cool visual on my watch to scan with the phone and then it was paired and I got this screen telling me the model that I had bought. OK so I still could not get the time from this watch and I have had the thing all day, I’m getting a little impatient at this point.
After waiting about 5 minutes for it to synch, suddenly a load of my apps, including my email, texts, calendar, twitter fitness apps and more are available on my watch. This is about to get interesting.
The first thing I noticed is that it is actually really easy to ready and see at a glance the notifications that are sent to your watch, such as Calendar reminders, text messages and Oracle Social Network updates (glad to see we are quick to the new platform with our own mobile apps). This is good for me, I get a lot of these alerts and I found a glance at my wrist was much nicer than pulling out my phone and unlocking it and starting at it. This sounds like a very small thing, but it is these small improvements in frequent interactions that make for a great user experience. I also agree with Jeremy Ashley about the huge value in being able to retain eye contact, notifications on my watch are far less obtrusive and the glance at my wrist it is a great experience.
So I wanted to try using it for some different things so I decided to test out text messages first, a quick SMS to respond to my wife’s text ‘ETA?’ to let her know what time I am planning to get home.
So I tap once on that nice Reply button
I can now either pick from a set of pre-defined responses and they would be sent without any other interaction from me. However I like the personal touch, this is my wife after all, so I decide I will click on the microphone icon to dictate a response. I speak in my answer and see the sound wave at the bottom and the text comes up perfectly first time.
So now I click done and get a really option to either send the audio or to just tap on the text and send that. This is a great feature if maybe the voice to text didn’t work properly and I don’t want to waste time correcting it or speaking it again.
After tapping on the text I am now done. The whole interaction was very fast and felt very natural. At this point I am really starting to like the Apple Watch. In the next few days I try driving directions, twitter, my calendar, a variety of fitness apps and more and pretty much across the board I find the interactions are natural and quick and the fact I have to pull out my phone less is a much bigger deal than I expected. I find I can glance down at my watch see a text or meeting reminder and carry on a conversation in a way that was not really possible if I had to pull my phone out. The one app I haven’t yet mentioned is the time, I haven’t worn a watch for over 10 years and I have realized in the last week it’s much easier to glance at my wrist than to pull out my phone – who knew?