Large trace files generated in udump [message #528295] |
Mon, 24 October 2011 03:08 |
snowball
Messages: 229 Registered: April 2006 Location: China
|
Senior Member |
|
|
Hi, guys
We have Oracle 10g(10.2.0.4) RAC on AIX 5.3, there are 3 RAC instances on each node.
From Oct 9th, we found one of the instance in node 1 generated a large size trace files in udump. The largest size of the trace file take up 3G. But there's only about 15G $ORACLE_BASE direcotry. After some time, we should delete some trace files for releasing the space.
Here is a part of alert log when this issue happen:
Sun Oct 9 23:18:15 2011
Errors in file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3166258.trc:
ORA-00600: internal error code, arguments: [17087], [0x70000010DF9F580], [], [], [], [], [], []
Sun Oct 9 23:18:16 2011
Trace dumping is performing id=[cdmp_20111009231816]
Sun Oct 9 23:33:17 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Sun Oct 9 23:48:18 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3092524.trc
Sun Oct 9 23:49:30 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 00:00:18 2011
Thread 1 advanced to log sequence 4244 (LGWR switch)
Current log# 6 seq# 4244 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_6.262.738307031
Current log# 6 seq# 4244 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_6.262.738307031
Mon Oct 10 00:02:32 2011
Thread 1 advanced to log sequence 4245 (LGWR switch)
Current log# 7 seq# 4245 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_7.263.738307033
Current log# 7 seq# 4245 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_7.263.738307035
Mon Oct 10 00:03:19 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3445118.trc
Mon Oct 10 00:05:13 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 00:18:51 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 00:20:57 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
...
Mon Oct 10 00:49:05 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3445118.trc
Mon Oct 10 00:52:25 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 01:04:37 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 01:08:10 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 01:19:49 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3092524.trc
Mon Oct 10 01:23:56 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 01:34:52 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3445118.trc
Mon Oct 10 01:39:41 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 01:50:25 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 01:55:26 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 02:00:21 2011
Thread 1 advanced to log sequence 4248 (LGWR switch)
Current log# 5 seq# 4248 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_5.261.738307027
Current log# 5 seq# 4248 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_5.261.738307029
Mon Oct 10 02:05:37 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3092524.trc
Mon Oct 10 02:11:12 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 02:20:40 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3445118.trc
Mon Oct 10 02:26:58 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 02:36:12 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 02:42:44 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 02:51:25 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3092524.trc
Mon Oct 10 02:58:30 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 03:06:27 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3445118.trc
Mon Oct 10 03:14:15 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 03:22:01 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 03:30:02 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 03:37:13 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3092524.trc
Mon Oct 10 03:45:48 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 03:52:16 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3445118.trc
Mon Oct 10 04:01:33 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 04:07:49 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 04:17:19 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 04:23:01 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3092524.trc
Mon Oct 10 04:23:35 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 04:32:54 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 04:38:04 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3445118.trc
Mon Oct 10 04:39:08 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 04:48:26 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 04:53:35 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3092524.trc
Mon Oct 10 04:54:39 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 05:00:21 2011
Thread 1 advanced to log sequence 4249 (LGWR switch)
Current log# 6 seq# 4249 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_6.262.738307031
Current log# 6 seq# 4249 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_6.262.738307031
Mon Oct 10 05:03:43 2011
Thread 1 advanced to log sequence 4250 (LGWR switch)
Current log# 7 seq# 4250 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_7.263.738307033
Current log# 7 seq# 4250 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_7.263.738307035
Mon Oct 10 05:03:57 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 05:08:37 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3445118.trc
Mon Oct 10 05:10:11 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 05:19:32 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 05:24:08 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3092524.trc
Mon Oct 10 05:25:44 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 05:35:04 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 05:39:11 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3445118.trc
Mon Oct 10 05:41:16 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3158172.trc
Mon Oct 10 05:50:37 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3104982.trc
Mon Oct 10 05:54:41 2011
System State dumped to trace filTue Oct 11 09:20:16 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 09:33:25 2011
Thread 1 advanced to log sequence 4286 (LGWR switch)
Current log# 1 seq# 4286 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_1.257.738307021
Current log# 1 seq# 4286 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_1.257.738307023
Tue Oct 11 09:35:51 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 09:51:25 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 10:06:57 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 10:13:49 2011
Thread 1 advanced to log sequence 4287 (LGWR switch)
Current log# 2 seq# 4287 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_2.258.738307023
Current log# 2 seq# 4287 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_2.258.738307025
Tue Oct 11 10:22:30 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 10:29:29 2011
Thread 1 advanced to log sequence 4288 (LGWR switch)
Current log# 5 seq# 4288 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_5.261.738307027
Current log# 5 seq# 4288 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_5.261.738307029
Tue Oct 11 10:38:02 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 10:53:34 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 11:02:46 2011
Thread 1 advanced to log sequence 4289 (LGWR switch)
Current log# 6 seq# 4289 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_6.262.738307031
Current log# 6 seq# 4289 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_6.262.738307031
Tue Oct 11 11:07:08 2011
Thread 1 advanced to log sequence 4290 (LGWR switch)
Current log# 7 seq# 4290 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_7.263.738307033
Current log# 7 seq# 4290 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_7.263.738307035
Tue Oct 11 11:09:06 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 11:24:39 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 11:40:13 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 11:55:45 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 12:01:26 2011
Thread 1 advanced to log sequence 4291 (LGWR switch)
Current log# 1 seq# 4291 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_1.257.738307021
Current log# 1 seq# 4291 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_1.257.738307023
Tue Oct 11 12:04:55 2011
Thread 1 advanced to log sequence 4292 (LGWR switch)
Current log# 2 seq# 4292 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_2.258.738307023
Current log# 2 seq# 4292 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_2.258.738307025
Tue Oct 11 12:06:42 2011
Thread 1 advanced to log sequence 4293 (LGWR switch)
Current log# 5 seq# 4293 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_5.261.738307027
Current log# 5 seq# 4293 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_5.261.738307029
Tue Oct 11 12:09:36 2011
Thread 1 advanced to log sequence 4294 (LGWR switch)
Current log# 6 seq# 4294 mem# 0: +YW_DGSYSTEM/bzywk/onlinelog/group_6.262.738307031
Current log# 6 seq# 4294 mem# 1: +YW_DGRECOVER/bzywk/onlinelog/group_6.262.738307031
Tue Oct 11 12:11:17 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 12:26:50 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 12:42:24 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 12:57:58 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 13:13:32 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
Tue Oct 11 13:29:05 2011
System State dumped to trace file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_3424538.trc
...
Tue Oct 11 15:11:50 2011
Errors in file /oracle/app/oracle/admin/bzywk/udump/bzywk1_ora_2089118.trc:
ORA-00600: internal error code, arguments: [qctcfx : len], [0], [0], [], [], [], [], []
...
I checked with some trace files, I found all these files contains a huge info of processes.
I have no idea how to located the root cause.
Please kindly give me some advices.
Thanks very much,
Milo
|
|
|
|
|
|
|
|
|
|
Re: Large trace files generated in udump [message #528327 is a reply to message #528320] |
Mon, 24 October 2011 05:19 |
snowball
Messages: 229 Registered: April 2006 Location: China
|
Senior Member |
|
|
Hi, Michel
I have tried this on my test environment, but it seems that there is no differences before I swtich on and after I switch on the event.
alter session set events 'immediate trace name SYSTEMSTATE level 10';
Here is what I am trying in my enviroment:
SYS@myrac2>oradebug setmypid
Statement processed.
SYS@myrac2>
SYS@myrac2>oradebug dump events 4
Statement processed.
SYS@myrac2>alter session set events 'immediate trace name SYSTEMSTATE level 10';
Session altered.
SYS@myrac2>oradebug setmypid
Statement processed.
SYS@myrac2>oradebug dump events 4
Statement processed.
Here is what I can see from the trace file:
*** 2011-10-24 18:08:57.608
Dump event group for level SYSTEM
TC Addr Evt#(b10) Action TR Addr Arm Life
*** 2011-10-24 18:09:21.813
Dump event group for level SYSTEM
TC Addr Evt#(b10) Action TR Addr Arm Life
Is there something I am missing?
Thanks very much,
Milo
[Updated on: Mon, 24 October 2011 05:20] Report message to a moderator
|
|
|
|
|
|
|
|
|
|
|
|