OEM DB console on 10gR2 broken.

From: <gbahut_at_ravenpack.com>
Date: Thu, 29 Nov 2007 04:29:50 -0800 (PST)
Message-ID: <05b6a452-3e69-4cf8-87f1-5d28de754d1a@g30g2000hsb.googlegroups.com>

Hello,

I have an issue with an OEM database control installed with Oracle Database
10.2.0.1.0 on Linux(Flavour of linux) X86_64 platform. This is a production environment.

Here is a summary of my problem:
About a week ago, I noticed that an RMAN backup session scheduled from the Database Control page had hung for a long period of time. I killed the Perl OS Process for the backup session and tried to restart it.
After restarting, the new Perl process also hung, this time consuming 100% CPU.
I also killed this.

At this point, I decided to drop and recreate the OEM repository using emca script:
> emca -dbconfig dbcontrol db -repos recreate

The script hung while dropping the repository. Once again, I saw a perl process at 100% CPU.
It exactly hang at:
  $ORACLE_HOME/sysman/admin/emdrep/bin/emrepmgr.pl: getTempTablespace() method
<code>
my $lda = DBI->connect('dbi:Oracle:', "SYS@".$EM_REPOS_CONNSTRING, "$EM_REPOS_DBSYSPWD",
    {ora_session_mode => 2, PrintError => 0, RaiseError => 0, AutoCommit =>
0})

    or (info "Could not connect to SYS/$EM_REPOS_CONNSTRING: $DBI::errstr\n"

        && exit 50);
</code>

I made a simple Perl file to test connections to the database using DBI
Perl interface with different users and all attempts to connect hung. No error is
returned. I am starting to believe that perhaps the Perl environment could
be corrupted.

I manage to recreate the OEM environment by following metalink Note: 278100.1 instructions:
1) Deleting DB Control Configuration Files using emca script:
> emca -deconfig dbcontrol db

2) Manually dropping OEM related objects: Logon SQLPLUS as user SYS or SYSTEM, and drop the sysman account and management objects:

 SQL> drop user sysman cascade;
 SQL> drop role MGMT_USER;
 SQL> drop user MGMT_VIEW cascade;
 SQL> drop public synonym MGMT_TARGET_BLACKOUTS;
 SQL> drop public synonym SETEMVIEWUSERCONTEXT;

3) Create the DB Control Repository Objects and Configuration Files.
> emca -config dbcontrol db -repos create

However the database control page shows the general status as "?Status Pending?"
and no information available. The home page shows the error: java.lang.Exception: Exception in sending Request :: null

I forced the agent to do an ?upload? by issuing:
> emctl upload

And finally got access to the OEM database control page.

My current situation is:
1) The OS process emagent is consuming 100% of 1 CPU while the db console
is started.
2) The database control page shows incomplete information. For instance, I
have access to performance and database objects information, but I cannot
get system statistics, ADDM recommendations, most of the metrics, and lots of
other valuable information.

The emca.trc file shows, among others, the following errors: 2007-11-29 11:44:56 Thread-4136450272 ERROR TargetManager: target {[SID], oracle_database} is broken: cannot compute dynamic properties in time.
2007-11-29 11:44:56 Thread-4136450272 ERROR util.files: ERROR: nmeufis_new: failed in lfiopn on file: $ORACLE_HOME/[host]_[sid]/ sysman/emd/agntstmp.txt. error = 2 (No such file or directory) 2007-11-29 11:44:56 Thread-4136450272 ERROR collector: nmecs_validateStateFile(): state file invalid : $ORACLE_HOME/ [host]_[sid]/sysman/emd/state/snapshot

The DB is completely operational and I can perform Backups using the RMAN client.

My questions are:
1. Does anyone have an idea of how to resolve this problems? 2. Is there a sure way to reinstall, purge or completely clean the OEM environment and start from scratch?

Best Regards,

G. Bahut Received on Thu Nov 29 2007 - 06:29:50 CST