RE: Database management techniques and frameworks

Adam --

I've done this more times than I can count. The answer is "it depends on your environment, your desired results, and, more often than not, your corporate structure". Here's some examples:

1. Monitoring script pages DBA group if X happens, Unix group if Y happens, Network group if Z happens. Simultaneously, XTerm windows are popped up in both Operations and HelpDesk with the name and pager number of the person paged (via uucp)
2. Monitoring script sends messages to centralized Error Management System. Error Management System handles it
3. Monitoring script finds problem and corrects problem. If problem continues, email is generated
4. Error Management System has external handles (not APIs) which can be used to call Monitoring Scripts, which need to be modified to ustilize System's internal structures (sometimes written in French -- *that* was fun!)
5. Monitoring script simply sends emails
6. Monitoring script keeps track of the errors in log files which are compared to log files from X time ago and only the differences are reported
7. Monitoring script has redundancy built in such that the first X times a particular problem is encountered, the Monitoring System ignores it, then generates a page
8. Monitoring script has redundnacy built in such that after the first time the problem is encountered, a page is sent, and if there is still a problem 15 minutes later, someone else is paged and so on up the company ladder

It goes on and on. This is largely what I've been doing for the past 8 years. Note that the words "Monitoring script" as used above is generally an inherently complicated conglomeration of several different scripts, generally with a governor and/or one or more driver(s), infrequently on different operating systems, sometimes in multiple languages and/or utilizing, or integrating with, or extending the capabilities of, one or more COTS products, which use different mechanisms to trigger and synchronize them. Generally, there is some kind of "IGNORE" functionality which allows for specified downtime for maintenance, or "ALTERNATE" functionality for unusual yet definable situations, and hierarchy of tests (if the database is down, that implies that a subsequent error that a user cannot connect to it has already been dealt with) and, occasionally has sniffers on other boxes to determine whether remote scripts need to be run either dependent upon remote conditions or independent of them. Sometimes, there is a process which kicks off other jobs and manages the security. I particularly enjoy those where there is fault tolerance built in such that if Monitoring script X on Machine Y craps out, Machine Z takes over and runs the scripts until Y is back, then copies the logs back, kicks off Y, make sure it runs ok, then shuts itself down. (note to the Oracle-L historians who might be curious, this change in my utilization is largely why my posts from 10 years ago were a lot more DBMS/internals heavy and my posts nowadays are more OS/script heavy.)

Regardless, I hope this answers your question and shows some of the complexity of what you're asking for...

Bambi.

-----Original Message-----
Sent: Friday, December 05, 2003 1:44 PM
To: Multiple recipients of list ORACLE-L

So your approach is to write a series of custom scripts, add them to (I assume) oracle's crontab for periodic execution. Do you have one single machine (or pair of machines) that monitor remote databases? Or do you install these scripts on each database server? Do you leverage dbms_jobs?  And relying on email seems kind of iffy -- what happens if you're not around to check your email? Page system? Escalation matrix in place?

Not trying to ruffle any feathers here, and certainly, I appreciate the time requirements in fully answering a question as broad as the one I submitted, but I would like to probe further into various strategies. The whole "run scripts to check, install statspack, etc." approach seems both highly unscalable and leaves much to the whim of the individual DBA. So what, you've installed statspack? Do you use it regularly? Is this a manual review, or is some system in place to monitor changes? How easy is it to deploy this framework?

(Does anyone here use Oracle's SNMP agents for monitoring? I've leveraged these -- along with a home-grown SNMP NMS (in Perl) -- to some degree at a multiple database site to good effect.)

Are there any 'design patterns for databases' around? Should we come up with some?

(I'll post my own notes on the topic of management in a future post -- still compiling.)

Adam

<ryan_oracle_at_cox.net>
Sent by: ml-errors_at_fatcity.com
12/05/2003 11:09 AM
Please respond to
ORACLE-L_at_fatcity.com

To
Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com> cc

Subject
Re: Database management techniques and frameworks

We have about 20-25 instances here. Nearly all on SUN. I dont touch the ones on windows. I also have development responsibilities, so I dont have time for a checklist.

you need to automate tasks. You cant spend your time reading the alert log. you should poll it and get an email when something pops up. Same with chained rows, tablespace sizes, etc... Write scripts for this and send your self emails.

Have statspack snapshots run daily.

>
> From: AdamDonahue_at_maximus.com
> Date: 2003/12/05 Fri PM 01:49:30 EST
> To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>
> Subject: Database management techniques and frameworks
>
> Folks,
>
> I thought it'd be interesting to take a survey on what techniques and
> frameworks DBA's on this list use to manage their Oracle databases. I
> imagine that some of us manage only a single database and instance, but
in
> those configurations where there are many instances, multiple databases,

> different platforms/versions, etc., what are some of the strategies for
> management in place? What daily tasks do you perform, and how do you
> organize them? How do you manage user requests (individually or as part

> of a larger environment)? How do you handle jobs? Organization
> techniques? Naming standards? User/application deployment framework,
> etc., etc.?
>
> (Obviously we could write a book about this -- there's an idea! -- but
> summaries and pointers would be interesting. Perhaps we can come up
with
> a best practices document and associated framework for Oracle database
> management.)
>
> Thanks,
>
> Adam
> --
> Please see the official ORACLE-L FAQ: http://www.orafaq.net
> --
> Author:
> INET: AdamDonahue_at_maximus.com
>
> Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> San Diego, California -- Mailing list and web hosting services
> ---------------------------------------------------------------------
> To REMOVE yourself from this mailing list, send an E-Mail message
> to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> the message BODY, include a line containing: UNSUB ORACLE-L
> (or the name of mailing list you want to be removed from). You may
> also send the HELP command for other information (like subscribing).
>

-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: <ryan_oracle_at_cox.net
  INET: ryan_oracle_at_cox.net

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).


-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: 
  INET: AdamDonahue_at_maximus.com

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).
-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Bellow, Bambi
  INET: bbellow_at_chi.navtech.com

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).