Re: intermittent long "log file sync" waits

was just getting ready to sign off and noticed the archivelog backup scheduled to run every hour seems to be stuck and has been for at least 10 mins:

[oracle_at_lsst-oradb04 ~]$ ps -ef | grep oracle_backup.sh oracle 13369 31167 0 16:43 pts/0 00:00:00 grep --color=auto oracle_backup.sh
oracle 14649 14645 0 16:00 ? 00:00:00 /bin/bash /home/oracle/scripts/rman/oracle_backup.sh -d lsst2db -s archivelog.rman -c lsst2db_rman -e cs2018_at_ncsa.illinois.edu -r iddsdba_rman [oracle_at_lsst-oradb04 ~]$ pstack 15649 Process 15649 not found.
[oracle_at_lsst-oradb04 ~]$ pstack 14649
#0 0x00007f9a6537e41c in waitpid () from /lib64/libc.so.6
#1 0x0000000000440b84 in waitchld.isra.10 ()
#2 0x0000000000441e3c in wait_for ()
#3 0x0000000000433b0e in execute_command_internal ()
#4 0x0000000000433d2e in execute_command ()
#5 0x000000000041e365 in reader_loop ()
#6 0x000000000041c9ce in main ()

[oracle_at_lsst-oradb04 ~]$ pstack 14649
#0 0x00007f9a6537e41c in waitpid () from /lib64/libc.so.6
#1 0x0000000000440b84 in waitchld.isra.10 ()
#2 0x0000000000441e3c in wait_for ()
#3 0x0000000000433b0e in execute_command_internal ()
#4 0x0000000000433d2e in execute_command ()
#5 0x000000000041e365 in reader_loop ()
#6 0x000000000041c9ce in main ()

[oracle_at_lsst-oradb04 ~]$ pstack 14649
#0 0x00007f9a6537e41c in waitpid () from /lib64/libc.so.6
#1 0x0000000000440b84 in waitchld.isra.10 ()
#2 0x0000000000441e3c in wait_for ()
#3 0x0000000000433b0e in execute_command_internal ()
#4 0x0000000000433d2e in execute_command ()
#5 0x000000000041e365 in reader_loop ()
#6 0x000000000041c9ce in main ()

[oracle_at_lsst-oradb04 ~]$ pstack 14649
#0 0x00007f9a6537e41c in waitpid () from /lib64/libc.so.6
#1 0x0000000000440b84 in waitchld.isra.10 ()
#2 0x0000000000441e3c in wait_for ()
#3 0x0000000000433b0e in execute_command_internal ()
#4 0x0000000000433d2e in execute_command ()
#5 0x000000000041e365 in reader_loop ()
#6 0x000000000041c9ce in main ()

[oracle_at_lsst-oradb04 ~]$

On Tue, Jan 28, 2020 at 4:30 PM Chris Stephens <cstephens16_at_gmail.com> wrote:

> ok. i've had the following running since lunchtime and will let it run
> through the night in hopes that issue occurs again:
>
> _at_snapper ash,stats,trace 10 999999 lgwr
>
> I will run the following script as well:
>
> #! /bin/bash
>
> while [ 1 ]
> do
>     echo `date` >> /tmp/pstack.26552
>     pstack 26552 >> /tmp/pstack.26552
>     echo `date` >> /tmp/pstack.26556
>     pstack 26556 >> /tmp/pstack.26556
>     echo `date` >> /tmp/pstack.26560
>     pstack 26560 >> /tmp/pstack.26560
>     sleep 5
> done
>
> based off:
>
> [oracle_at_lsst-oradb05 bin]$ ps -ef | egrep "lg.*lsst2db2"
> oracle   26552     1  0 Jan17 ?        00:06:14 ora_lgwr_lsst2db2
> oracle   26556     1  0 Jan17 ?        00:00:13 ora_lg00_lsst2db2
> oracle   26560     1  0 Jan17 ?        00:00:12 ora_lg01_lsst2db2
>
> All sessions for this workload are connecting to service w/ tracing
> enabled so we'll have trace data as well.
>
> I will (hopefully) have updates in the morning.
>
> On Tue, Jan 28, 2020 at 4:01 PM Noveljic Nenad <
> nenad.noveljic_at_vontobel.com> wrote:
>
>> We need to find out what lg* are doing during long log file sync waits.
>> As ash samples are entirely missing I propose to sample all lg* processes
>> with pstack every second and store the information for the later analysis.
>> Another useful information would be snapper stats output, say, also once
>> per second and targeted to lg* from the same interval.
>>
>> This will generate lots of data but you can write a script to
>> periodically delete it if there weren’t any incidents. You can easily
>> detect them by counting log file sync samples.
>>
>> I’m really curious about this as usually long log file sync waits match
>> some instrumented bg activity which is, apparently, not being the case here.
>>
>>
>>
>>
>> *Von: *Noveljic Nenad <nenad.noveljic_at_vontobel.com>
>> *Datum *Dienstag, 28. Jan. 2020, 10:17 PM
>> *An: *cary.millsap_at_method-r.com <cary.millsap_at_method-r.com>, Stefan
>> Koehler <contact_at_soocs.de>
>> *Cc: *Chris Stephens <cstephens16_at_gmail.com>, oracle-l <
>> Oracle-L_at_freelists.org>
>> *Betreff: *RE: intermittent long "log file sync" waits
>>
>> In this case, though, there aren’t any ash entries for any of the lg*
>> processes while the app sessions were waiting in log file sync for a long
>> time.
>>
>>
>>
>>
>>
>> ____________________________________________________
>>
>> Please consider the environment before printing this e-mail.
>>
>> Bitte denken Sie an die Umwelt, bevor Sie dieses E-Mail drucken.
>>
>>
>> Important Notice
>> This message is intended only for the individual named. It may contain
>> confidential or privileged information. If you are not the named addressee
>> you should in particular not disseminate, distribute, modify or copy this
>> e-mail. Please notify the sender immediately by e-mail, if you have
>> received this message by mistake and delete it from your system.
>> Without prejudice to any contractual agreements between you and us which
>> shall prevail in any case, we take it as your authorization to correspond
>> with you by e-mail if you send us messages by e-mail. However, we reserve
>> the right not to execute orders and instructions transmitted by e-mail at
>> any time and without further explanation.
>> E-mail transmission may not be secure or error-free as information could
>> be intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also
>> processing of incoming e-mails cannot be guaranteed. All liability of
>> Vontobel Holding Ltd. and any of its affiliates (hereinafter collectively
>> referred to as "Vontobel Group") for any damages resulting from e-mail use
>> is excluded. You are advised that urgent and time sensitive messages should
>> not be sent by e-mail and if verification is required please request a
>> printed version. Please note that all e-mail communications to and from the
>> Vontobel Group are subject to electronic storage and review by Vontobel
>> Group. Unless stated to the contrary and without prejudice to any
>> contractual agreements between you and Vontobel Group which shall prevail
>> in any case, e-mail-communication is for informational purposes only and is
>> not intended as an offer or solicitation for the purchase or sale of any
>> financial instrument or as an official confirmation of any transaction.
>> The legal basis for the processing of your personal data is the
>> legitimate interest to develop a commercial relationship with you, as well
>> as your consent to forward you commercial communications. You can exercise,
>> at any time and under the terms established under current regulation, your
>> rights. If you prefer not to receive any further communications, please
>> contact your client relationship manager if you are a client of Vontobel
>> Group or notify the sender. Please note for an exact reference to the
>> affected group entity the corporate e-mail signature. For further
>> information about data privacy at Vontobel Group please consult
>> www.vontobel.com.
>>
>

--
http://www.freelists.org/webpage/oracle-l