Incremental backup fails [message #324338] |
Mon, 02 June 2008 01:27 |
ora_2007
Messages: 430 Registered: July 2007 Location: Mumbai
|
Senior Member |
|
|
Hi,
On my production systems the backups are running well from last one year.
On monday we used to set the Incremental backup.
Today morning i surprised to see that the Incremental backup is failed becuase of some reason.
There is no change in the scripts of backup.
Please find the attachment log of Incremental backup which is failed.
Recovery Manager: Release 10.2.0.1.0 - Production on Mon Jun 2 01:00:02 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
RMAN>
RMAN> CONNECT TARGET *
connected to target database: ORA01 (DBID=639147973)
RMAN> CONNECT CATALOG *
connected to recovery catalog database
RMAN> **end-of-file**
RMAN>
RMAN> # INCREMENTAL LEVEL 1
2> RUN
3> {
4> ALLOCATE CHANNEL c1 DEVICE TYPE 'sbt_tape'PARMS
5> 'ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin/tdpo.opt)';
6>
7> BACKUP INCREMENTAL LEVEL 1 FILESPERSET 1
8> TAG = 'INC LVL 1 DB BKUP' DATABASE INCLUDE CURRENT CONTROLFILE
9> FORMAT = 'Inc_1_%U_%T.bkp';
10>
11>
12> BACKUP FILESPERSET 10
13> FORMAT = 'Arch_%U_%T.bkp'
14> ARCHIVELOG ALL
15> DELETE INPUT;
16> #DELETE ARCHIVELOG UNTIL TIME 'SYSDATE-1';
17> RELEASE CHANNEL c1;
18> }
allocated channel: c1
channel c1: sid=96 devtype=SBT_TAPE
channel c1: Data Protection for Oracle: version 5.3.3.0
Starting backup at 02-JUN-08
channel c1: starting incremental level 1 datafile backupset
channel c1: specifying datafile(s) in backupset
input datafile fno=00016 name=/u01/mannual/oracle/oradata/ora01/mannual_M_IND/mannual_M_IND_01.dbf
channel c1: starting piece 1 at 02-JUN-08
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on c1 channel at 06/02/2008 01:13:50
RMAN-10038: database session for channel c1 terminated unexpectedly
RMAN>
RMAN>
RMAN> RUN
2> {
3> ALLOCATE CHANNEL a1 TYPE DISK;
4> ALLOCATE CHANNEL c1 DEVICE TYPE 'sbt_tape'PARMS
5> 'ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin/tdpo.opt)';
6>
7>
8> CROSSCHECK BACKUP;
9> DELETE EXPIRED BACKUP;
10>
11> DELETE OBSOLETE;
12>
13> RELEASE CHANNEL c1;
14> RELEASE CHANNEL a1;
15> }
allocated channel: a1
channel a1: sid=96 devtype=DISK
allocated channel: c1
channel c1: sid=108 devtype=SBT_TAPE
channel c1: Data Protection for Oracle: version 5.3.3.0
RMAN retention policy will be applied to the command
RMAN retention policy is set to redundancy 2
Deleting the following obsolete backups and copies:
Type Key Completion Time Filename/Handle
-------------------- ------ ------------------ --------------------
Control File Copy 133418 31-MAY-08 /home/oracle/RMAN_BACKUP/cpcurrCTL_cf_D-ORA01_id-639147973_frjhngo8_20080531.ora
released channel: a1
released channel: c1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of delete command on c1 channel at 06/02/2008 01:13:53
RMAN-10015: error compiling PL/SQL program
RMAN-10014: PL/SQL error 0 on line 621 column 4: Statement ignored
RMAN-10014: PL/SQL error 201 on line 621 column 4: identifier 'EXITED' must be declared
RMAN-10014: PL/SQL error 0 on line 573 column 16: Statement ignored
RMAN-10014: PL/SQL error 201 on line 573 column 16: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 463 column 16: Statement ignored
RMAN-10014: PL/SQL error 201 on line 463 column 16: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 381 column 16: Statement ignored
RMAN-10014: PL/SQL error 201 on line 381 column 16: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 336 column 16: Statement ignored
RMAN-10014: PL/SQL error 201 on line 336 column 16: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 259 column 13: Statement ignored
RMAN-10014: PL/SQL error 201 on line 259 column 29: identifier 'STAMP2DATE' must be declared
RMAN-10014: PL/SQL error 0 on line 234 column 10: Statement ignored
RMAN-10014: PL/SQL error 201 on line 235 column 24: identifier 'BOOL2CHAR' must be declared
RMAN-10014: PL/SQL error 0 on line 231 column 16: Statement ignored
RMAN-10014: PL/SQL error 201 on line 231 column 16: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 226 column 16: Statement ignored
RMAN-10014: PL/SQL error 201 on line 226 column 16: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 208 column 16: Statement ignored
RMAN-10014: PL/SQL error 201 on line 208 column 16: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 201 column 16: Statement ignored
RMAN-10014: PL/SQL error 201 on line 201 column 16: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 199 column 13: Statement ignored
RMAN-10014: PL/SQL error 201 on line 199 column 13: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 196 column 7: Statement ignored
RMAN-10014: PL/SQL error 201 on line 196 column 50: identifier 'BOOL2CHAR' must be declared
RMAN-10014: PL/SQL error 0 on line 191 column 10: Statement ignored
RMAN-10014: PL/SQL error 201 on line 191 column 10: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 189 column 10: Statement ignored
RMAN-10014: PL/SQL error 201 on line 189 column 10: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 181 column 19: Statement ignored
RMAN-10014: PL/SQL error 201 on line 181 column 44: identifier 'GETERRM' must be declared
RMAN-10014: PL/SQL error 0 on line 109 column 7: Statement ignored
RMAN-10014: PL/SQL error 201 on line 109 column 7: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 105 column 7: Statement ignored
RMAN-10014: PL/SQL error 201 on line 105 column 7: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 68 column 4: SQL Statement ignored
RMAN-10014: PL/SQL error 201 on line 77 column 11: identifier 'X$DUAL' must be declared
RMAN-10014: PL/SQL error 0 on line 67 column 4: Statement ignored
RMAN-10014: PL/SQL error 201 on line 67 column 4: identifier 'DEB' must be declared
RMAN-10014: PL/SQL error 0 on line 66 column 4: Statement ignored
RMAN-10014: PL/SQL error 201 on line 66 column 4: identifier 'ENTERED' must be declared
RMAN>
RMAN> RUN
2> {
3> ALLOCATE CHANNEL t1 TYPE DISK;
4> BACKUP AS COPY CURRENT CONTROLFILE FORMAT '/home/oracle/RMAN_BACKUP/cpcurrCTL_%U_%T.ora';
5> RELEASE CHANNEL t1;
6> }
allocated channel: t1
channel t1: sid=96 devtype=DISK
Starting backup at 02-JUN-08
channel t1: starting datafile copy
copying current control file
output filename=/home/oracle/RMAN_BACKUP/cpcurrCTL_cf_D-ORA01_id-639147973_hajhskqh_20080602.ora tag=TAG20080602T011353 recid=512 stamp=656298834
channel t1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 02-JUN-08
released channel: t1
RMAN> **end-of-file**
RMAN>
Recovery Manager complete.
The error signifies that the problem is with allocating the channel.
But before some time i run the archive backup script and it is executed succesfully. In this the channels are allocated successfully.
Please find archive backup log file also.
Recovery Manager: Release 10.2.0.1.0 - Production on Mon Jun 2 07:54:03 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
RMAN>
RMAN> CONNECT TARGET *
connected to target database: ORA01 (DBID=639147973)
RMAN> CONNECT CATALOG *
connected to recovery catalog database
RMAN> **end-of-file**
RMAN>
RMAN>
RMAN>
RMAN>
RMAN> RUN
2> {
3> ALLOCATE CHANNEL c1 DEVICE TYPE 'sbt_tape'PARMS
4> 'ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin/tdpo.opt)';
5>
6> RESYNC CATALOG;
7> #BACKUP DATABASE FILESPERSET 1
8> #TAG = 'FULL DB BKUP' DATABASE INCLUDE CURRENT CONTROLFILE
9> #FORMAT = 'FULL%U_%T.bkp';
10> CROSSCHECK ARCHIVELOG ALL;
11>
12> BACKUP ARCHIVELOG ALL DELETE INPUT;
13> #CROSSCHECK ARCHIVELOG ALL;
14>
15> # BACKUP FILESPERSET 10
16> # FORMAT = 'Arch_%U_%T.bkp'
17> # ARCHIVELOG ALL
18> # DELETE ALL INPUT;
19> # DELETE ARCHIVELOG UNTIL TIME 'SYSDATE-1
20>
21> RELEASE CHANNEL c1;
22> }
allocated channel: c1
channel c1: sid=120 devtype=SBT_TAPE
channel c1: Data Protection for Oracle: version 5.3.3.0
starting full resync of recovery catalog
full resync complete
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20382_595759112.dbf recid=13949 stamp=656237509
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20383_595759112.dbf recid=13950 stamp=656244452
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20384_595759112.dbf recid=13951 stamp=656244502
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20385_595759112.dbf recid=13952 stamp=656244548
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20386_595759112.dbf recid=13953 stamp=656244596
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20387_595759112.dbf recid=13954 stamp=656244646
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20388_595759112.dbf recid=13955 stamp=656251605
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20389_595759112.dbf recid=13956 stamp=656266221
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20390_595759112.dbf recid=13957 stamp=656278664
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20391_595759112.dbf recid=13958 stamp=656278757
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20392_595759112.dbf recid=13959 stamp=656279076
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20393_595759112.dbf recid=13960 stamp=656287297
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20394_595759112.dbf recid=13961 stamp=656288234
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20395_595759112.dbf recid=13962 stamp=656300224
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20396_595759112.dbf recid=13963 stamp=656302018
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20397_595759112.dbf recid=13964 stamp=656303548
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20398_595759112.dbf recid=13965 stamp=656304847
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20399_595759112.dbf recid=13966 stamp=656306231
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20400_595759112.dbf recid=13967 stamp=656307632
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20401_595759112.dbf recid=13968 stamp=656308721
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20402_595759112.dbf recid=13969 stamp=656309753
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20403_595759112.dbf recid=13970 stamp=656316659
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20404_595759112.dbf recid=13971 stamp=656317156
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20405_595759112.dbf recid=13972 stamp=656317628
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20406_595759112.dbf recid=13973 stamp=656317973
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20407_595759112.dbf recid=13974 stamp=656318311
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20408_595759112.dbf recid=13975 stamp=656318685
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20409_595759112.dbf recid=13976 stamp=656319170
validation succeeded for archived log
archive log filename=/u06/ARCHIVE_LOG/1_20410_595759112.dbf recid=13977 stamp=656322259
Crosschecked 29 objects
Starting backup at 02-JUN-08
current log archived
channel c1: starting archive log backupset
channel c1: specifying archive log(s) in backup set
input archive log thread=1 sequence=20382 recid=13949 stamp=656237509
input archive log thread=1 sequence=20383 recid=13950 stamp=656244452
input archive log thread=1 sequence=20384 recid=13951 stamp=656244502
input archive log thread=1 sequence=20385 recid=13952 stamp=656244548
input archive log thread=1 sequence=20386 recid=13953 stamp=656244596
input archive log thread=1 sequence=20387 recid=13954 stamp=656244646
input archive log thread=1 sequence=20388 recid=13955 stamp=656251605
input archive log thread=1 sequence=20389 recid=13956 stamp=656266221
input archive log thread=1 sequence=20390 recid=13957 stamp=656278664
input archive log thread=1 sequence=20391 recid=13958 stamp=656278757
input archive log thread=1 sequence=20392 recid=13959 stamp=656279076
input archive log thread=1 sequence=20393 recid=13960 stamp=656287297
input archive log thread=1 sequence=20394 recid=13961 stamp=656288234
input archive log thread=1 sequence=20395 recid=13962 stamp=656300224
input archive log thread=1 sequence=20396 recid=13963 stamp=656302018
input archive log thread=1 sequence=20397 recid=13964 stamp=656303548
input archive log thread=1 sequence=20398 recid=13965 stamp=656304847
input archive log thread=1 sequence=20399 recid=13966 stamp=656306231
input archive log thread=1 sequence=20400 recid=13967 stamp=656307632
input archive log thread=1 sequence=20401 recid=13968 stamp=656308721
input archive log thread=1 sequence=20402 recid=13969 stamp=656309753
input archive log thread=1 sequence=20403 recid=13970 stamp=656316659
input archive log thread=1 sequence=20404 recid=13971 stamp=656317156
input archive log thread=1 sequence=20405 recid=13972 stamp=656317628
input archive log thread=1 sequence=20406 recid=13973 stamp=656317973
input archive log thread=1 sequence=20407 recid=13974 stamp=656318311
input archive log thread=1 sequence=20408 recid=13975 stamp=656318685
input archive log thread=1 sequence=20409 recid=13976 stamp=656319170
input archive log thread=1 sequence=20410 recid=13977 stamp=656322259
input archive log thread=1 sequence=20411 recid=13978 stamp=656322854
channel c1: starting piece 1 at 02-JUN-08
channel c1: finished piece 1 at 02-JUN-08
piece handle=hbjhtc98_1_1 tag=TAG20080602T075415 comment=API Version 2.0,MMS Version 5.3.3.0
channel c1: backup set complete, elapsed time: 00:10:06
channel c1: deleting archive log(s)
archive log filename=/u06/ARCHIVE_LOG/1_20382_595759112.dbf recid=13949 stamp=656237509
archive log filename=/u06/ARCHIVE_LOG/1_20383_595759112.dbf recid=13950 stamp=656244452
archive log filename=/u06/ARCHIVE_LOG/1_20384_595759112.dbf recid=13951 stamp=656244502
archive log filename=/u06/ARCHIVE_LOG/1_20385_595759112.dbf recid=13952 stamp=656244548
archive log filename=/u06/ARCHIVE_LOG/1_20386_595759112.dbf recid=13953 stamp=656244596
archive log filename=/u06/ARCHIVE_LOG/1_20387_595759112.dbf recid=13954 stamp=656244646
archive log filename=/u06/ARCHIVE_LOG/1_20388_595759112.dbf recid=13955 stamp=656251605
archive log filename=/u06/ARCHIVE_LOG/1_20389_595759112.dbf recid=13956 stamp=656266221
archive log filename=/u06/ARCHIVE_LOG/1_20390_595759112.dbf recid=13957 stamp=656278664
archive log filename=/u06/ARCHIVE_LOG/1_20391_595759112.dbf recid=13958 stamp=656278757
archive log filename=/u06/ARCHIVE_LOG/1_20392_595759112.dbf recid=13959 stamp=656279076
archive log filename=/u06/ARCHIVE_LOG/1_20393_595759112.dbf recid=13960 stamp=656287297
archive log filename=/u06/ARCHIVE_LOG/1_20394_595759112.dbf recid=13961 stamp=656288234
archive log filename=/u06/ARCHIVE_LOG/1_20395_595759112.dbf recid=13962 stamp=656300224
archive log filename=/u06/ARCHIVE_LOG/1_20396_595759112.dbf recid=13963 stamp=656302018
archive log filename=/u06/ARCHIVE_LOG/1_20397_595759112.dbf recid=13964 stamp=656303548
archive log filename=/u06/ARCHIVE_LOG/1_20398_595759112.dbf recid=13965 stamp=656304847
archive log filename=/u06/ARCHIVE_LOG/1_20399_595759112.dbf recid=13966 stamp=656306231
archive log filename=/u06/ARCHIVE_LOG/1_20400_595759112.dbf recid=13967 stamp=656307632
archive log filename=/u06/ARCHIVE_LOG/1_20401_595759112.dbf recid=13968 stamp=656308721
archive log filename=/u06/ARCHIVE_LOG/1_20402_595759112.dbf recid=13969 stamp=656309753
archive log filename=/u06/ARCHIVE_LOG/1_20403_595759112.dbf recid=13970 stamp=656316659
archive log filename=/u06/ARCHIVE_LOG/1_20404_595759112.dbf recid=13971 stamp=656317156
archive log filename=/u06/ARCHIVE_LOG/1_20405_595759112.dbf recid=13972 stamp=656317628
archive log filename=/u06/ARCHIVE_LOG/1_20406_595759112.dbf recid=13973 stamp=656317973
archive log filename=/u06/ARCHIVE_LOG/1_20407_595759112.dbf recid=13974 stamp=656318311
archive log filename=/u06/ARCHIVE_LOG/1_20408_595759112.dbf recid=13975 stamp=656318685
archive log filename=/u06/ARCHIVE_LOG/1_20409_595759112.dbf recid=13976 stamp=656319170
archive log filename=/u06/ARCHIVE_LOG/1_20410_595759112.dbf recid=13977 stamp=656322259
archive log filename=/u06/ARCHIVE_LOG/1_20411_595759112.dbf recid=13978 stamp=656322854
Finished backup at 02-JUN-08
released channel: c1
RMAN>
RMAN> **end-of-file**
RMAN> 2>
Recovery Manager complete.
Please suggest me the cause of the problem.
Thanks in advance.
[Updated on: Mon, 02 June 2008 01:32] Report message to a moderator
|
|
|
Re: Incremental backup fails [message #324364 is a reply to message #324338] |
Mon, 02 June 2008 03:22 |
ora_2007
Messages: 430 Registered: July 2007 Location: Mumbai
|
Senior Member |
|
|
The problem i think is because the network problem???????
Will network problem causes the previous error????
How can i check that during perticular time the network was having any problem??
I am using Suse Enterprise edition 9i unix.
Thanks,
|
|
|
|
|
Re: Incremental backup fails [message #324424 is a reply to message #324338] |
Mon, 02 June 2008 07:46 |
ora_2007
Messages: 430 Registered: July 2007 Location: Mumbai
|
Senior Member |
|
|
Thanks Mahesh for your reply,
In the alert log file it gives me following messae during this time.
Errors in file /home/oracle/admin/ora01/udump/ora01_ora_8924.trc:
ORA-07445: exception encountered: core dump [strlen()+11] [SIGSEGV] [Address not mapped to object] [0x126] [] []
Mon Jun 2 01:13:49 2008
Process 0x0xa22bb494 appears to be hung while dumping
Current time = 41772318, process death time = 41712193 interval = 60000
Attempting to kill process 0x0xa22bb494 with OS pid = 8924
OSD kill succeeded for process 0xa22bb494
Please find the attachment of ora01_ora_8924.trc file
Tomorrow there will be incremental backup on 1 AM. So tomorrow i can able to check that it is reproducable or not.
Thanks,
[Updated on: Mon, 02 June 2008 08:00] Report message to a moderator
|
|
|
|
|
|
|
Re: Incremental backup fails [message #330394 is a reply to message #324338] |
Mon, 30 June 2008 01:21 |
ora_2007
Messages: 430 Registered: July 2007 Location: Mumbai
|
Senior Member |
|
|
As i am observing my backups everyday and after 2 weeks observation i find that my backup fails on Monday only
1. Backup was running perfecty from last 1 year.
2. Backup start to fail on monday from last 1 month.
Please tell me me what the things which i need to look into it?
Or please sugget what might be the reason behind it?
Thanks,
[Updated on: Mon, 30 June 2008 01:21] Report message to a moderator
|
|
|
|