Re: Backups failing
Date: Tue, 30 Aug 2011 12:11:52 +0100
Message-ID: <CALQkqm-C3DVt=1qCG+XqxV8mmnpKZpU_zTxUOmOufORDGdXZOw_at_mail.gmail.com>
Thanks.
That command works well ... I saw well the backups in the EM, and via command too. But it fails from some days. I'm recreating the EM following this:
Is it ok? I'm doing recreate, it is deleting it now. I remember do this in the past, for a similar problem. When I restart agent, I always see in emagent.log the next:
2011-08-30 11:15:50 Thread-4136376000 target {xxx.xxx.xxx, orac le_database} is broken: cannot compute dynamic properties in time. (00155)
Many people fix it modifying emd.properties, changing a parameter, I've
changed it but it continues failing. Other people fix it recreating EM, I'm
with it now.
Other people fix patching Oracle ...
I don't know why it has failed without do nothing in the OS/Oracle.
Thanks beforehand.
Cheers...
2011/8/30 Niall Litchfield <niall.litchfield_at_gmail.com>
> Hi
>
> Oliver, John's note is for Grid Control and not for Database Control, that
> doesn't mean it won't work - but it may not do so. If the problem you are
> trying to address is RMAN backups failing then using the rman command line
> directly maybe of more immediate use. At the very least I would do the
> following in a terminal/command window on the server ( & having set the
> oracle environment correctly for your database. )
>
> rman target /
> list backup of database;
>
>
>
>
> On Tue, Aug 30, 2011 at 9:59 AM, Oliver <ofabelo_at_gmail.com> wrote:
>
>> I did that yesterday ... I will try again.
>> Thanks.
>>
>>
>> 2011/8/30 Pavel <ocp.pauler_at_gmail.com>
>>
>>> Hi
>>> Try this:
>>> emctl stop agent
>>> cd $AGENT_HOME/sysman/emd
>>> rm -rf collection/*
>>> rm -rf state/*
>>> rm -rf upload/*
>>> emctl start agent
>>>
>>> Regards,
>>> Pavel.
>>>
>>>
>>> 2011/8/30 Oliver <ofabelo_at_gmail.com>
>>>
>>>> I'm looking emagent.trc, just when it crashes ... The log output is:
>>>>
>>>> 2011-08-30 00:11:12 Thread-4039011216 ERROR scheduler: Collection
>>>> oracle_listener:LISTENER_xxx.xxx.xx:Response has not finished in 721 minutes
>>>> (allowed: 720).
>>>> Exiting agent.
>>>> 2011-08-30 00:11:12 Thread-4039011216 ERROR : Signalled to Exit with
>>>> status 3
>>>> 2011-08-30 00:11:12 Thread-4136433344 ERROR main: EMAgent abnormal
>>>> terminating
>>>> 2011-08-30 00:11:41 Thread-4136175296 WARN command: Job Subsystem
>>>> Timeout set at 600 seconds
>>>> 2011-08-30 00:11:41 Thread-4136175296 WARN upload: Upload manager has
>>>> no Failure script: disabled
>>>> 2011-08-30 00:11:41 Thread-4136175296 WARN upload: Recovering left
>>>> over xml files in upload directory
>>>> 2011-08-30 00:11:42 Thread-4136175296 WARN upload: Recovered 0 left
>>>> over xml files in upload directory
>>>> 2011-08-30 00:11:42 Thread-4136175296 WARN metadata: Metric Disk_Path
>>>> does not have any data columns
>>>> 2011-08-30 00:11:42 Thread-4136175296 WARN metadata: Metric
>>>> osm_diskGroupPolicies does not have any data columns
>>>> 2011-08-30 00:12:42 Thread-4136175296 ERROR TargetManager: TIMEOUT when
>>>> compute dynamic properties for target xxx.xxx.xxx
>>>> 2011-08-30 00:12:42 Thread-4136175296 ERROR TargetManager: target
>>>> {xxx.xxx.xxx, oracle_database} is broken: cannot compute dynamic properties
>>>> in time.
>>>> 2011-08-30 00:12:43 Thread-4136175296 WARN collector: the column name
>>>> DiskActivityavwait in this condition does not exist
>>>> 2011-08-30 00:12:43 Thread-4119530384 ERROR upload: Error in
>>>> uploadXMLFiles. Trying again in 60.00 seconds.
>>>>
>>>> I'm "googling" but without success for now ... I don't find the
>>>> solution.
>>>> Thanks beforehand.
>>>>
>>>>
>>>>
>>>> 2011/8/30 Oliver <ofabelo_at_gmail.com>
>>>>
>>>>> Same error today, initialization failure in status of the job and in
>>>>> the error log, "Step aborted because agent went down".
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2011/8/29 Oliver <ofabelo_at_gmail.com>
>>>>>
>>>>>> This is the output:
>>>>>>
>>>>>> Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
>>>>>> Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
>>>>>> ---------------------------------------------------------------
>>>>>> Agent Version : 10.1.0.4.1
>>>>>> OMS Version : 10.1.0.4.0
>>>>>> Protocol Version : 10.1.0.2.0
>>>>>> Agent Home : /u01/app/oracle/product/10.2.0/db_1/xxx_ora10g
>>>>>> Agent binaries : /u01/app/oracle/product/10.2.0/db_1
>>>>>> Agent Process ID : 19051
>>>>>> Parent Process ID : 18114
>>>>>> Agent URL : http://xxx:3938/emd/main
>>>>>> Started at : 2011-08-29 12:09:00
>>>>>> Started by user : oracle
>>>>>> Last Reload : 2011-08-29 12:14:14
>>>>>> Last successful upload : 2011-08-29 12:14:23
>>>>>> Total Megabytes of XML files uploaded so far : 4.40
>>>>>> Number of XML files pending upload : 0
>>>>>> Size of XML files pending upload(MB) : 0.00
>>>>>> Available disk space on upload filesystem : 87.14%
>>>>>> ---------------------------------------------------------------
>>>>>> Agent is Running and Ready
>>>>>>
>>>>>> I see it all ok, isn't it?
>>>>>> Jobs are many hours for seeing it has failed ... I will look tomorrow,
>>>>>> but I see that it doesn't writing in the backup dir ...
>>>>>>
>>>>>>
>>>>>> 2011/8/29 Niall Litchfield <niall.litchfield_at_gmail.com>
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> what is the full output on the target machine of emctl status agent?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 29, 2011 at 11:49 AM, Oliver <ofabelo_at_gmail.com> wrote:
>>>>>>>
>>>>>>>> When I do "emctl upload agent" step, it shows:
>>>>>>>>
>>>>>>>> EMD upload error: uploadXMLFiles skipped :: OMS version not checked
>>>>>>>> yet..
>>>>>>>>
>>>>>>>> Jus the error of that page ...
>>>>>>>>
>>>>>>>>
>>>>>>>> 2011/8/29 Oliver <ofabelo_at_gmail.com>
>>>>>>>>
>>>>>>>>> Yes, I did crosscheck in rman ...
>>>>>>>>> I'm going to do the steps of:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://jhdba.wordpress.com/2008/12/10/clearing-an-enterprise-manager-agent-that-fails-to-upload/
>>>>>>>>>
>>>>>>>>> Is it ok?
>>>>>>>>>
>>>>>>>>> Thanks beforehand.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2011/8/29 Pavel <ocp.pauler_at_gmail.com>
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>> Did you crosscheck after delete some backups? Clear state don't
>>>>>>>>>> remove created jobs.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Pavel.
>>>>>>>>>>
>>>>>>>>>> 2011/8/29 Oliver <ofabelo_at_gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Thanks, but I'm sorry, I'm a newbie, the dba is on vacation and I
>>>>>>>>>>> have to look it because I'm the person with more knowledges of Oracle.
>>>>>>>>>>> I've tried stop and start agent/dbconsole but it is failing
>>>>>>>>>>> again. I understand your commands, except remove all uploaded files :-?
>>>>>>>>>>> Clear state or remove uploads files remove created jobs?
>>>>>>>>>>> Ah, one thing, I think that it started to fail when it surpassed
>>>>>>>>>>> the flash recovery quota. I had 300GB, and it had 330GB written, in
>>>>>>>>>>> destination of backup dir. I deleted some backups and I will increase to
>>>>>>>>>>> 500GB the flash recovery quota (I've not restarted database, is it
>>>>>>>>>>> necessary?)
>>>>>>>>>>> Thanks beforehand.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2011/8/29 Pavel <ocp.pauler_at_gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>> Could you try these one:
>>>>>>>>>>>>
>>>>>>>>>>>> Issue emctl stop agent, emctl clear state agent, emctl start
>>>>>>>>>>>> agent.
>>>>>>>>>>>>
>>>>>>>>>>>> And I think remove all uploads files.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Pavel.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2011/8/29 Oliver <ofabelo_at_gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> backups of EM are failing suddenly. It shows "Initialization
>>>>>>>>>>>>> failure" in status and error log shows "Step aborted because agent went
>>>>>>>>>>>>> down". The error comes after many hours, I see that it doesn't write nothing
>>>>>>>>>>>>> in the oputut dir of the backup.
>>>>>>>>>>>>> If I do "$ tail -f emagent.trc", it shows some warns and the
>>>>>>>>>>>>> errors:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2011-08-29 09:43:25 Thread-4136277696 ERROR TargetManager:
>>>>>>>>>>>>> TIMEOUT when compute dynamic properties for target
>>>>>>>>>>>>> ora10g.localhost.localdomain
>>>>>>>>>>>>> 2011-08-29 09:43:25 Thread-4136277696 ERROR TargetManager:
>>>>>>>>>>>>> target {ora10g.localhost.localdomain, oracle_database} is broken: cannot
>>>>>>>>>>>>> compute dynamic properties in time.
>>>>>>>>>>>>> 2011-08-29 09:43:25 Thread-4136277696 WARN collector: the
>>>>>>>>>>>>> column name DiskActivityavwait in this condition does not exist
>>>>>>>>>>>>> 2011-08-29 09:43:25 Thread-4119632784 ERROR upload: Error in
>>>>>>>>>>>>> uploadXMLFiles. Trying again in 60.00 seconds.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is 10.2.0.1.0 on Linux. Someone can help me?
>>>>>>>>>>>>> Thanks beforehand.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers...
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Niall Litchfield
>>>>>>> Oracle DBA
>>>>>>> http://www.orawin.info
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> Niall Litchfield
> Oracle DBA
> http://www.orawin.info
>
-- http://www.freelists.org/webpage/oracle-lReceived on Tue Aug 30 2011 - 06:11:52 CDT