Re: hung db
Date: Sat, 9 Jan 2016 17:46:36 -0500
Message-ID: <56918DCC.2010206_at_gmail.com>
On 01/08/2016 11:21 PM, MJ Mody wrote:
> Suspected resource contention of sorts. Follow-up with backup vendor, CommVault, yielded a file lock with cv/oracle library (i.e. oci.dll) from an earlier backup. Will need to review the Oracle shadow copy interface.
>
> Still a mystery as to why nothing was written to alert log.
Was there anything in the ClOraAgent.log? Did you open a case with
CommVault support? Do you have any other backup system on the machine?
Simpana client library is called ORASBT.DLL and usually resides in
C:\Program Files\Commvault\Simpana\Base. If there was another backup
software on the machine, it can put ORASBT.DLL either in
%ORACLE_HOME%\lib or in C:\Windows\system32. Please, search the entire
machine for the ORASBT.DLL libraries, since the library clash can be
quite serious problem. The name of the library is mandated by Oracle
Corp. and all backup suites (Commvault, TSM, NetBackup, Avamar) must use
the same name.
Regards
>
> Cheers
> MJ
>
>> On Jan 8, 2016, at 9:26 PM, De DBA <dedba_at_tpg.com.au> wrote:
>>
>> Usually I'd suspect an out-of-memory situation. The scenario reminds me of a case I had where the backup software (Tivoli, I think), would use Windows shadow copy to execute an RMAN backup. The shadow copy process never exited and after a (large) number of backups the Oracle process memory was exhausted and the database hang. From memory that was also W2K8R2/11.2.0.3. I believe there was a bug in the Oracle shadow copy interface.
>>
>> You did check the Windows Event Logs (Application and System) I presume?
>>
>> Hth,
>> Tony
>>
>> On 09/01/16 11:03, MJ Mody wrote:
>>> Nothing like ringing in a new year than with a hung database.
>>> In this case, running 11.2.0.3 EE with ASO/TDE on W2K8-R2 with last CPU.
>>> From what is known, external backups (tape writes) were taking place in backup software, both rman and data pump (the .dmp files were created early in the evening). Additionally, nightly EOD processing was also taking place.
>>>
>>> Performed a server reboot to bring the database online. Point of concern is from 9pm onwards, until the database came back online (~1:30am), there is no entries in the alert log.
>>>
>>> Feel free to share if anyone has come across this or have some words of wisdom for troubleshooting and remediation.
>>>
>>> Thank you in advance.
>>>
>>> Best
>>> MJ--
>>> http://www.freelists.org/webpage/oracle-l
>>>
>>>
>>>
> --
> http://www.freelists.org/webpage/oracle-l
>
>
-- Mladen Gogala Oracle DBA http://mgogala.freehostia.com -- http://www.freelists.org/webpage/oracle-lReceived on Sat Jan 09 2016 - 23:46:36 CET