RAC internals - GLOBAL ENQUEUE SERVICES DEADLOCK DETECTED

We're encountering many of "Global Enqueue Services Deadlock detected" messages in the alert log of one of our database instances (two node RAC database) together with quite big global cache cr request waits in STATSPACK report. The problem is that the only other available information is trace file that looks like:

• 2005-09-19 10:37:01.153 Global Wait-For-Graph(WFG) at ddTS[0.0] : BLOCKED 70000016f956fd8 5 [0x2b90011][0x5f20],[TX] [131321,1285] 1 BLOCKER 70000016b5ff288 5 [0x2b90011][0x5f20],[TX] [131317,1159] 1 BLOCKED 70000016b1c8440 5 [0x3fe001d][0x1a1d],[TX] [131317,1159] 1 BLOCKER 700000199574ea0 5 [0x3fe001d][0x1a1d],[TX] [131321,1285] 1

and sometimes also like:

----------enqueue------------------------
lock version     : 10665
Owner node       : 0
grant_level      : KJUSEREX
req_level        : KJUSEREX
bast_level       : KJUSERNL
notify_func      : 0
resp             : 7000001b13711d8
procp            : 700000160d71fa0
pid              : 5365832
proc version     : 39
oprocp           : 0
opid             : 0
gid              : 618
xid              : 67532 66652
dd_time          : 0.0 secs
dd_count         : 0
timeout          : 0.0 secs
On_timer_q?      : N
On_dd_q?         : N
lock_state       : GRANTED
Open Options     : KJUSERDEADLOCK
Convert options  : KJUSERNOQUEUE
History          : 0x9a514395
Msg_Seq          : 0x0
res_seq          : 0

The problem is that it looks like no one including local Oracle support knows what is the meaning of values in this tracefile. We are also pursuing this with Oracle Worldwide Support, but it can take a lot of time. Is there someone who investigated this kind of trace file before? Search on Metalink showed only a couple of similar, but never answered questions.

Thank you

Dusan Bolek Received on Wed Nov 02 2005 - 01:44:13 CST