RE: Recent reports on outages caused by DB2 and 9iRAC issues

WOW! Those are some good stories to keep in mind... And some folks wonder why techies like working with computers more than working with people. ;-)

Trying to be wise as a serpent and harmless as a dove...

-----Original Message-----
From: Steve Perry [mailto:sperry_at_sprynet.com] Sent: Friday, July 18, 2003 1:49 PM
To: Multiple recipients of list ORACLE-L Subject: Re: Recent reports on outages caused by DB2 and 9iRAC issues

my 2¢ is it all depends on who you are.
A few good ones I'm aware of.

Confessors
within the last 9 months, I know of one company where a DBA (unknown to mgt.) inadvertently dropped something and was able to fix it within a few minutes. I'm not sure if anyone other than operations noticed. Mgt. response: his mistake would be recognized on his yearly bonus.

A couple months later at the same company, a well known DBA (one of the boys) brought down the web database for a couple of hours. Nothing was said. Mgt. response "We all make mistakes. Be more careful".

Cowards
A few years earlier at the same company, I remember a DBA who was doing maintenance and messed up. He never confessed to creating the problem, but did take credit for fixing it. Mgt. response: he was awarded a certificate for "Going the Extra Mile"... :)

Steve

• Original Message ----- To: "Multiple recipients of list ORACLE-L" <ORACLE-L_at_fatcity.com> Sent: Friday, July 18, 2003 1:34 PM

> I've only made one mistake in my life and that was the time when I
> thought I was wrong. ;-)
>
> OK, getting serious... I've found it's best to QUICKLY be 100%
> truthful. If you get crucified for it then it may be that damagement
> needs to lighten up. Unfortunately the fear of damagement causes wheel
> spinning work-arounds to compromises of the facts. ;-)
>
> I once inadvertently brought a 24X7 production system down by shutting
> down a test database that was "managed" by Veritas Cluster Server
> junk. A hands on type director had added it to the cluster without
> telling me so when I shut down the test database the cluster stuff
> tried to fail over and everything came crashing down. I got the blame
> so it was a real "cluster-f***" because the damager wouldn't accept
> any responsibility for this episode. Later he decided to change some
> Unix stuff all on his own and the result was that the system was down
> quite some time. He as rather cavalier about the outage and danced
> around the truth. Ultimately his prevarications caught up with him and
> he was "dismissed" on the spot. "Justice" was served.
>
>
>
> -----Original Message-----
> Sent: Friday, July 18, 2003 11:59 AM
> To: Multiple recipients of list ORACLE-L
> Importance: High
>
>
> I always admit when I make a mistake. Gives me much more credibility
> when I say "this time it ISN'T me"
>
>
> --- "Goulet, Dick" <DGoulet_at_vicr.com> wrote:
> > I prefer my bosses attitude on the subject of mistakes:
> >
> > Admit that you did it & then go fix it. We'll crucify you
> later, if
> > we have the time.
> >
> > For some reason, known only to the Devil himself, they never find
> > the time.
> >
> > Dick Goulet
> > Senior Oracle DBA
> > Oracle Certified 8i DBA
> >
> > -----Original Message-----
> > Sent: Friday, July 18, 2003 1:14 PM
> > To: Multiple recipients of list ORACLE-L
> >
> >
> > This reminds of watching the Indy 500 a couple of years ago.
> >
> > One of the drivers managed to spin in turn 4 of the first lap of the
> > race, and subsequently was unable to finish the race.
> >
> > Before continuing, you need to know that when in a corner, one thing
> > you should *never* do is take your foot off of the gas pedal.
> >
> > The resulting weight transfer will greatly reduce the amount of
> > weight on the rear tires, and could easily cause you to spin.
> >
> > Back to the story: The driver was interviewed shortly afterward in
> > the pits, and put the blame squarely on the car. "It was something
> > with the car".
> >
> > Technology in the form of an onboard camera told another story. He
> > let off the gas twice in turn 4. The first time the rear of the car
> > squirmed a bit and he recovered. The second time it sounded like he
> > completely released the accelerator, and the spin came shortly
> > afterwards.
> >
> > He could have been a smart guy too, if he had remembered the camera.
> >
> > Too bad there wasn't a camera on the smart guys at Orbitz, maybe it
> > would tell a different story. ;)
> >
> > None of us are immune to making mistakes, and about the time you
> > start feeling invincible, watch out.
> >
> > Jared
> >
> >
> >
> >
> >
> >
> > "Goulet, Dick" <DGoulet_at_vicr.com>
> > Sent by: ml-errors_at_fatcity.com
> > 07/18/2003 09:44 AM
> > Please respond to ORACLE-L
> >
> >
> > To: Multiple recipients of list ORACLE-L
> > <ORACLE-L_at_fatcity.com>
> > cc:
> > Subject: RE: Recent reports on outages caused by DB2
> > and 9iRAC issues
> >
> >
> > Interesting quote in eWeek. We may all be "smart guys", but how
> > often
>
> > do we like to admit to causing a problem.
> >
> > Dick Goulet
> > Senior Oracle DBA
> > Oracle Certified 8i DBA
> > -----Original Message-----
> > Sent: Friday, July 18, 2003 11:55 AM
> > To: Multiple recipients of list ORACLE-L
> >
> >
> >
> > Reports on "Bugs" in IBM's DB2 which led to "a critical operational
> > situation" http://theregister.co.uk/content/archive/30095.html
> > and
> >
> http://www.danskebank.com/link/ITreport20030403uk/$file/ITreport200304
> 03
> uk.pdf
> >
> >
> > Report on Orbitz blaming an outage on Oracle's 9iRAC [and Orbitz
> > going
>
> > out of 9iRAC] http://www.eweek.com/article2/0,3959,1196879,00.asp
> > and
> >
> http://www.computerworld.com/databasetopics/data/software/story/0,1080
> 1,
> 83186,00.html
> >
> >
> >
> > Hemant K Chitale
> > Oracle 9i Database Administrator Certified Professional
> > My personal web site is : http://hkchital.tripod.com
> >
> > -- Please see the official ORACLE-L FAQ: http://www.orafaq.net --
> > Author: Hemant K Chitale INET: hkchital_at_singnet.com.sg Fat City
> > Network Services -- 858-538-5051 http://www.fatcity.com San Diego,
> > California -- Mailing list and web hosting services
> > --------------------------------------------------------------------
> > -
> > To
> > REMOVE yourself from this mailing list, send an E-Mail message to:
> > ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the
> > message BODY, include a line containing: UNSUB ORACLE-L (or the name
> > of
> > mailing list you want to be removed from). You may also send the HELP
> >
> > command for other information (like subscribing).
> >
> >
> > --
> > Please see the official ORACLE-L FAQ: http://www.orafaq.net
> > --
> > Author:
> > INET: Jared.Still_at_radisys.com
> >
> > Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> > San Diego, California -- Mailing list and web hosting services
> > --------------------------------------------------------------------
> > -
> > To REMOVE yourself from this mailing list, send an E-Mail message
> > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > the message BODY, include a line containing: UNSUB ORACLE-L (or the
> > name of mailing list you want to be removed from). You may also send
> > the HELP command for other information (like subscribing).
> > --
> > Please see the official ORACLE-L FAQ: http://www.orafaq.net
> > --
> > Author: Goulet, Dick
> > INET: DGoulet_at_vicr.com
> >
> > Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> > San Diego, California -- Mailing list and web hosting services
> > --------------------------------------------------------------------
> > -
> > To REMOVE yourself from this mailing list, send an E-Mail message
> > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > the message BODY, include a line containing: UNSUB ORACLE-L (or the
> > name of mailing list you want to be removed from). You may also send
> > the HELP command for other information (like subscribing).
> >
>
>
> __________________________________
> Do you Yahoo!?
> SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
> --
> Please see the official ORACLE-L FAQ: http://www.orafaq.net
> --
> Author: Rachel Carmichael
> INET: wisernet100_at_yahoo.com
>
> Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> San Diego, California -- Mailing list and web hosting services
> ---------------------------------------------------------------------
> To REMOVE yourself from this mailing list, send an E-Mail message
> to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> the message BODY, include a line containing: UNSUB ORACLE-L (or the
> name of mailing list you want to be removed from). You may also send
> the HELP command for other information (like subscribing).
>
> --
> Please see the official ORACLE-L FAQ: http://www.orafaq.net
> --
> Author: Orr, Steve
> INET: sorr_at_rightnow.com
>
> Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> San Diego, California -- Mailing list and web hosting services
> ---------------------------------------------------------------------
> To REMOVE yourself from this mailing list, send an E-Mail message
> to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> the message BODY, include a line containing: UNSUB ORACLE-L (or the
> name of mailing list you want to be removed from). You may also send
> the HELP command for other information (like subscribing).
>

-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Steve Perry
  INET: sperry_at_sprynet.com

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from).  You may also send the HELP command for other information (like subscribing).