RE: Centralized Alert Log Monitoring

I hadn't really considered centralizing my monitoring scripts, for fear of actually having to learn about TCP via perl. What I do have is a perl daemon for each instance that monitors the alert log using "tail -f flnm|". I search each line as it is written for troubling errors. If any major errors are detected, I write out an empty file to a specified directory. Then I use cron to look for that file every 5 minutes. If that file exists, a page is issued. The cron part is centralized I guess, as it only runs on a single server and does it's look ups via NFS.

After writing the above, I realize that we are actually not that far off in terms of concept. Except I bet your perl is much more ready for primetime than mine. As usual I employed brute force as often as possible.

Steve McClure

-----Original Message-----
Jared.Still_at_radisys.com
Sent: Monday, January 07, 2002 12:35 PM
To: Multiple recipients of list ORACLE-L

> Any ideas?

Yeah. It involves Perl of course.

This sounds like a natural evolution for the stuff I already have.

The distributed version will be in the Perl book. If I had time I would create the centralized version for it, but I can't do it right now.

'Why even mention it?' you ask.

Well, it never hurts to cast about for ideas on how to do it.

The cleanest way to me would be to have a daemon/Service ( one per database, unix or Win32 ) on the db server side that monitors the alert log. All it would do is read the alert log, and ship off new lines that appear ( via TCP socket ) to a centralized analyzer.

The analyzer would be a non-forking ( Win32, remember? ) TCP server accepting the data from the unix daemons and win32 services, writing it out to a master file, data files per database, or to Oracle tables . Lots of ways it could be stored. ( re the non-forking: Perl threads aren't ready for prime time )

The analyzer would check for patterns in the text that indicate that someone should be paged. These could be global or at the database level, probably a combination with db level overriding global where it exists.

Emails, pages, what have you would be sent, recorded, the errors recorded, etc.

Both client and server would require unix and win32 specific versions, as writing a win32 service is sufficiently different from a unix version to warrant a different code. base.

Might make a nice rev 2 tool.

Any thoughts on this?

Jared

                    dmeng_at_focal.co
                    m                    To:     Multiple recipients of list
ORACLE-L <ORACLE-L_at_fatcity.com>
                    Sent by:             cc:
                    root_at_fatcity.c       Subject:     Re: Centralized Alert
Log Monitoring
                    om


                    01/07/02 11:50
                    AM
                    Please respond
                    to ORACLE-L

Thanks all for the reply.
To make the problem more challenging, I would like to add that ( sorry for not mentioning about this in my original E-mail ) our shop has both NT and Unix.

Dennis Meng
Database Administrator
Focal Communications
847-954-8328

                    "Brian Haas"

                    <bhaas_at_musiciansf        To:     Multiple recipients of
list ORACLE-L <ORACLE-L_at_fatcity.com>
                    riend.com>               cc:

                    Sent by:                 Subject:     Re: Centralized
Alert Log Monitoring
                    root_at_fatcity.com



                    01/07/02 01:27 PM

                    Please respond to

                    ORACLE-L

Dennis,

I'm not sure if you have any NT boxes or not, but for the unix side have you considered setting up an NFS share? You could store one copy of the script there and mount that share on the db servers. Then just schedule it like you normally would (cron,at). Depending how your script works, you may need to abstract out hard coded values, and pass them as command line parameters for file locations, sid, etc. We do this for our web site code and all of our utility scripts. It makes maintenence much easier.

HTH, -Brian

On Mon, 7 Jan 2002 dmeng_at_focal.com wrote:
>Hi All,
>Currently I am the only DBA for the 15 productin databases in our company.
>I have written shell scripts on all database servers to parse the
alert.log
>file and look for Oracle errors and send page/E-mail if problems are
found.
>While this works it is sometimes painful to maintain - if I want to change
>something I have to go to different servers to change the scripts one by
>one. I figure it would be nice if we can have a centralized solution where
>I can monitor alert logs remotely and don't have to hop from server to
>server. Are there any tools/scripts that enable DBAs to do this? I know
>OEM can do that with the Advanced Events but now you have to buy the
>Diagnostics pack for that.
>TIA
>Dennis Meng
>Database Administrator
>Focal Communications
>847-954-8328
>

--
Please see the official ORACLE-L FAQ: http://www.orafaq.com
--
Author: Brian Haas
  INET: bhaas_at_musiciansfriend.com

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists
--------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).




--
Please see the official ORACLE-L FAQ: http://www.orafaq.com
--
Author:
  INET: dmeng_at_focal.com

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists
--------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).




--
Please see the official ORACLE-L FAQ: http://www.orafaq.com
--
Author:
  INET: Jared.Still_at_radisys.com

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists
--------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.com
-- 
Author: Steve McClure
  INET: smcclure_at_usscript.com

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists
--------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).