Re: Controlfiles just got overwritten

I think what really scares people is the "mysterious incident", where somehow, *something* bad happened and nobody knows how or why. It's the IT corollary to a street murder with plenty of bystanders but no witnesses.

Another personal vignette...

A little over 10 years ago, I was working in downtown LA and arrived in the office early (5 am) to start a batch job. I had a card-key which got me into the building and into the office during the day, but at night there were locking doors in the elevator lobby of which I was previously unaware. I banged on the doors, tried calling people, to no avail. Finally, after a half hour, out of frustration, I grabbed one of the door handles and just YANKED HARD.

It popped open.

I looked at it in surprise, thought "sweet!", walked in to the cubicle farm, sat down, and started my batch job. All was good.

Around 7 am, the LAPD showed up. There were about a dozen people in the office now, so the two officers began questioning folks nearest the door. From the opposite side of the room, I stood up and called out, "Over here". I 'fessed up.

They told me that, if I hadn't called them over immediately, they would have arrested me by the time they got to me. Have a nice day, sir.

As Chris Rock said (after the Rodney King incident): "If the police have to chase you, they'll bring a whupping with them." That works for the every other part of the world, too...

On 7/18/14, 12:26, Maureen English wrote:
> Tim, rest of list,
>
> THANK YOU!
>
> I was ready to start cleaning out my desk after realizing what I did.
> My manager, though, was concerned about what
> to tell the users and letting our director know what had happened.
> Our director, someone I've worked with for almost
> 20 years, always remains calm and offered some suggestions for quick
> fixes since he was a DBA years ago. Our CITO,
> someone I expected to be very upset, responded with something like
> 'Mistakes happen, you're only human.'
>
> - Maureen
>
>
> On 7/17/2014 3:28 PM, Tim Gorman wrote:
>> Maureen,
>>
>> About 4 years ago, or 26 years into my IT career, I dropped an index
>> on a 60 Tb table with 24,000 hourly partitions;
>> the index was over 15 Tb in size. It was the main table in that
>> production application, of course.
>>
>> Over a quarter century of industry experience as a developer,
>> production support, systems administrator, and database
>> administrator; if that's not enough time to have important lessons
>> pounded into one's head, then how much time is needed?
>>
>> My supervisor at the time was amazing. After the shock of watching
>> it all happen and still not quite believing it had
>> happened, I called him at about 9pm local time, and told him what
>> occurred. I finished speaking, and waited for the axe
>> to drop, for the entirely-justified anger to crash down on my head.
>> He was silent for about 3 seconds, then just said
>> calmly, "Well, I guess we need to fix it." And that was it. No
>> anger, no recriminations, no humiliating
>> micro-management. We launched straight into planning what needed to
>> happen to fix it.
>>
>> He got to work notifying the organization what had happened, and I
>> got started on the rebuild, which eventually took
>> almost 2 weeks to complete.
>>
>> I hope you had the same experience. Because it truly happens to all
>> of us. And anyone who pretends otherwise simply
>> hasn't been doing important work.
>>
>> Hope this helps...
>>
>> -Tim
>>
>>
>>
>>
>> On 7/17/14, 17:01, Maureen English wrote:
>>> I ended up recovering the database from a backup done the day before
>>> and rolling forward.
>>>
>>> I also modified my instructions for recreating a standby database.
>>> Instead of 'cp -p' to copy
>>> my standby control file to the appropriate directory on the standby
>>> server, I will now use 'cp -pi'.
>>>
>>> I'm still feeling badly for making this mistake, but I'm amazed at
>>> how understanding my coworkers
>>> have been.
>>>
>>> - Maureen
>>>
>
>

--
http://www.freelists.org/webpage/oracle-l