Moans Nogood

I love to visit Iceland. Not because of the landscape (We Do Not Use landscapes), but because of the people there. Man, they're funny and good to talk to.

This time (last week, for two days only) I detected a deep, slow-burning anger in the folks up there, that I haven't felt before.

One of them - a very stylish, suit-dressed bartender in his 40's, who broke his principle about not talking politics with guests (hey, the bar was empty), said that he suspected there would be real riots in the fall of 2010 if the few guilty bastards were not punished for real in court.

The handful of real bastards have been driven out of Iceland, by the way: People simply spat on them when they met them on the street AND painted their houses and cars red at night. One of the bastards re-painted his house in its original color. Guess what happened the following night.

They all now live in the UK or Florida.

Fine and good. Now you know.

But this post is about the one thing you should never ask an Icelander to do: A list.

You see, back around year 900 several Norwegian vikings sailed to Iceland. They pretty much remembered everything: Clothes. Food. The ship. Oares. Live animals. Tools. Pen & Paper. You know - all the usual stuff for a 400 years voyage.

But whoever was in charge of The Norwegian Iceland Travel List forgot one thing, and whether they discovered it quickly (i.e. on the journey) or when they had settled in in their small mud huts, I do not know. But man, they must have told The List Guy a thing or two upon suddenly remembering what they had forgot:

Women.

Anyway, they apparently decided to do something about it, because DNA-tests of Icelandic women some years ago confirmed that they originate from the British isles.

The British scientist who found this out, and who was interviewed on Danish Radio, commented on the fact that Icelandic women look Pretty Damn Good by saying (rather drily): "They probably didn't take the ugly ones."

So now you know.

It's Friday morning and I'm on my way away from San Francisco after a splendid week of OOW, good guys, a few beers, and a lot of tech talk.

We ran OCW four times from Monday to Thursday, and it was really good presenters we had talked into showing up:

Monday: Jeff Needham on processors and how Oracle runs on them. Opteron good. Nehalem good. A reporter named Kate was present in order to write about OCW. Code: 41.

Tuesday: Jonathan Lewis showing why the crowd were not experts. Ouch. Code 43.

Wednesday: Jeremiah Wilton about the Cloud, and especially the Amazon Cloud. He seems to know a good deal about Amazon. Code 24.

Thursday: Uri Shaft on counting eg. NDV in the optimizer, and some compression theory - and then Dan Norris & Greg Rahn about the Database Machine. Code 42.

And Kate's funny article about OCW appeared in the daily conference newspaper on Thursday. She got all the technical and non-technical stuff right - very impressive! She also gave away the secret location (Thirsty Bear on 661 Howard, upstairs), but thankfully only on the very last day of OCW :).

I truly enjoyed it, and so did several others, so we'll probably do it again next year.

Apart from that, it also appears that the guys from Miracle who were here with me (Morten Tangaa, Jesper Haure, Kaj Christensen, Claus Sørensen) got good things out of the conference.

While I remember it: Thank you to Victoria Lira, Lillian Buziak, and Justin Kestelyn for allocating a reporter for OCW, for managing the whole ACE Director thing, and many other favors that make the conference work.

We had planned not to have any OCW presentations today in order not to steal Larry's audience from his planned keynote, but we're doing it anyway.

It's at 1200 hours, NOT 1300 hours as usual.

More details via text messages later, including todays codeword. If you want text messages from me for the OCW sessions, send me a text/SMS on +45 25277100.

Cloud computing is 'hot'. So is Larry when he talks about it on YouTube. Funny as Hell, actually.

There are at least these two videos. They are partly overlapping, but that doesn't matter- you'll want to see him do this standup routine a couple of times, trust me:

http://www.youtube.com/watch?v=0FacYAI6DY0
http://www.youtube.com/watch?v=8UYa6gQC14o

Which is why today, at the secret location, Oracle Closed World will present a couple of guys that know everything about 'the cloud'.

Mogens

I'm here in San Francisco for the Oracle Open World conference along with four other guys from Miracle, the two crazy Miracle Finland guys and some other crazy people - we've rented a couple of big apartments as usual, and are doing work, beer and other essential stuff together.

Last year at Oracle Open World (OOW) my friend Iggy Fernandez, who edits the NOCOUG (Northern California Oracle User Group) magazine/journal, suggested an Oracle Closed World conference, where REAL, TECHNICAL presentations would take place underground in secret locations, using secret passwords, and what have you.

Well, it's here. Monday, Tuesday and Thursdag at a secret location we'll do deep and very technical presentations about various topics. The secret location (which is indeed underground) has the capability to serve beer, by the way.

Let me know if you're interested in hearing more about OCW - email me on mno@MiracleAS.dk or text me on +45 2527 7100.

Mogens

Here's some information you will try to forget after reading. It explains why SAN's always cause trouble, why "a firmware upgrade" is really a complete change of an OS and therefor really dangerous (and impossible to plan or test for) and more.

From now on, think of the firmware in a SAN as a whole OS, just bigger. Scary, right?

My question to this very smart guy I know was this:

"Could you repeat what OS'es are used in what SAN's for me? And how many code lines the ExaData is using?"

=======================================================================

oh god...that is a huge question... First, Exadata software is small (less than about 100MB of bits)...but that is an unfair comparison to the glut of stuff in a full-featured array... Exadata has Linux underneath it, but then we execute about 98% User, 2% Kernel so really, the only thing we get from Linux is scheduling and I/O... Exadata is small because it doesn't do any of the fat stuff arrays like Clariion do (e.g., snapshots, remote mirror, etc).

Netapp's is called OnTap and it is a heavily developed BSD (Net/1 to be exact). It is huge and full featured as you can tell by how many add on packages it support, but just in protocol provisioning it is huge. Consider the fact that it can support front-end FC yet the LUNS are actually files in the WAFL filesystem! Wild.

Clariion OS is called FLARE and it sits on top of a full Windows distro (XP). EMC NAS (celerra) is called DART which is written from scratch.

HP StorageWorks Clustered Gateway is Linux +hundreds of thousands of very specialized PolyServe code.

EMC DMX OS is called Enginuity...DMX cpus are Power and I have no idea what the origin of this OS is. If I were a betting man I'd bet that it is scratch like DART.

IBM DS83XX is full blown AIX plus more (this is the old SHARK array) in fact, it is a cluster of AIX boxes in there...

I don't know what HP calls the stuff that runs inside EVA ...

as you can see...it is very confusing.
=======================================================================

There's been a discussion going on among some of my friends about all this horrible-looking (and often badly performing) auto-generated SQL coming out of Discoverer and other tools. Here are some of the comments made during the discussion, and some of my memories of how I got started with Oracle with the help of my good friend Mogens Egan...

=======================

Me:

"Oracle Discoverer - helping developers write ugly code for more than a decade."

=======================

NN:

"no no no!

the real beauty of discoverer (and similar tools) is not that it lets developers write ugly code, but it lets people who don't know what code is (business users), write code and share it with other users who also don't know what code is. It's entire purpose in life is to let people who don't know what they are doing, do it. developers do what they do with some understanding and can, sometimes, be educated. accountants and hr people can't."

=======================

Me:

"This brings me back. From 1987 to 1990 I was in a bank, sharing an office with Mogens Egan (the father of Morten Egan) and basically creating a datawarehouse (although we didn't know it) for internal users in the bank.

Our strategy was this:

1. Every night (or once a week or whatever) we would transfer data from the banks mainframe system via a SNA gateway to our VAX. The data came from IMS databases and was delivered as flat ASCII files (one physical record = one logical record) which often resultet in very very long records, of course, since IMS is hierachical. We would then load it into tables and let the users access it.

2. I would hold one- or two-day courses where I'd teach the attendees (who had probably only used a PC for a very short time) how to log onto the VAX using Smarterm, how to use VMS basic commands (including the editor), how to use SQL and SQL*Plus, how to create default forms in Forms 2.3 - and some other stuff.

3. Mogens Egan's idea was that it was better to turn users/experts (SME's in todays jargon) into "programmers" than vice versa. And then it should be our job to fix run-away jobs (read: SQL that performed bad or messed up things for others).

A rather anarchistic approach, you could say. But man, it worked. In three years we had 1000 users, some of who turned out to be natural super users, who started creating systems that helped their co-workers.

Since they were not officially named super users they couldn't demand to be given time to develop something they thought could be useful - they were by natural selection only allowed to spend time on something their co-workers thought useful.

Mogens and I are still in contact with many of those users. The machine is now an Alpha cluster, the data it manages runs a rather large banks' trading stuff, and all that - but its name is still Samson. And the super user we created back then is still called Supermule, which is the Danish name for Super Goof. With the introduction of English-speaking consultants in the last 10 years it has proved a minor mistake - they all ask "What's a super mule?"

So yes, we had many incidents of run-away jobs where the poor user had issued a SQL statement without the proper where-clause, etc. But then we would discover it, kill it, help the user - and all of the victims of this bad SQL knew it could be their turn one day, so they didn't get mad or upset.

That playground which we created back then generated a lot of Oracle-lovers who are still around in various higher positions, and perhaps it would have been even easier for them back then if we had had Discoverer.

So I think you're absolutely right: Discoverer will help computer-illeterates write really bad code even faster. But at least it gets them to use Oracle, and it creates wonderful problems that finances our fantastic lifestyles.

Mogens

PS: In the World as a whole, I think Discoverer had a presence (penetration) of about 2% of customers. In Denmark it was 20% due to my ex-wife Laila (Nathalie's mother), then product sales rep for Discoverer, who insisted that every single customer should have this product, like it or not. And notice how well Miracle is doing here. Perhaps there's a relationship."

===================================

SAP is a huge, mysterious, expensive animal.

In my very private opinion it is probably the worst ERP system you can buy today. Hence, most whiteshirts will choose it.

To compensate for the fact that it's old and silly technology, it's also exceedingly expensive. Introducing SAP to your company is the only reliable way to tell whether your company is so financially strong that it almost resembles a monopoly.

But what I really hate about SAP is that it removes people from the Oracle database field. I think most of us have experienced the following scenario:

A colleague or a bunch of colleagues are selected to help implement SAP. Until then they've been ordinary DBA's, fixing stuff, running databases and leading normal family lives.

Then they go away for EXTENSIVE training over a LONG period of time. In between the 42 week-long classes they have to take (per year, of course), they usually rest with their families and might show up for short, social functions among their (still) colleagues. But they have these myserious, far-away eyes... you can't quite reach them.

Then you get the famous message by mouth or email stating:

"We're now almost ready to , so for the next transition period of , I'll be working half of my time with my old stuff and half of my time with SAP before moving to full-time SAP obligations."

AND THAT'S THE LAST YOU EVER HEAR FROM THEM!

I think, but I could be wrong, that they're sucked into a place and time in space that the rest of us can't see or in other ways sense.

A Harry Potter-like parallel universe.

From which, mind you, they never return.

Numerous are the good Oracle DBA's who disappear from the real Oracle world this way.

Now it appears that in their parallel universe (the SAPPU, it could be called) they're slowly corrupted into thinking about Oracle databases the way the real (the few, the remaining) Oracle DBA's thought about databases in the 80's.

They're forced to unlearn all the right things they had learned. A kind of communist re-schooling or indoctrination. So sad.

Here follows a real mail thread from some friends of mine that know more about Oracle than most. They shall, of course, remain nameless - we still have no idea about the powers and general abilities of our disappeared friends in the SAPPU, and so a certain degree of fear for the unknown make us cautious....

======================================================================
Oracle Person One said:

I find myself once again embroiled in an SAP/Oracle Issue.

It seems I broke something in the production database by applying security patches, even though the 5 development and production servers have no such issues. (No, we can't afford a full system to test patches on - sigh...)

There are a number of ORA-7445 and ORA-600 errors being issued when DML is attempted, along with requisite trace files. (Yes, there is a an active SR already)

The SAP Basis team see's ORA-3113 and ORA-3114 errors.

I think I finally have them convinced to stop trying to 'fix' the ORA-3113...

Which brings me to the point: The usual method of troubleshooting SAP problems, as practiced by the SAP team, and nearly every SAP person I have worked with (not a terribly large sample - maybe I am just really lucky), goes something like this:

1. Search the SAP support site for every note containing ORA-600, ORA-7445. There are a few as you can well imagine.

2. Call a meeting to discuss which of the actions in these notes should be taken. Whether or not the contents of the note actually match the problem at hand seems to be irrelevant.

3. Ask the DBA to run a script (recommended by SAP support) to check for some problem or another in the data. This script will launch full table scans for every table in the database... Using a for/next loop. Fortunately for me, the script is broken as is. ... and I can't seem to find the problem with it.

4. Rinse and repeat - effectiveness is unimportant, only looking busy is important.

To memorialize this method, I have created a link to the following short, but accurately portrayed method of this troubleshooting methodology:

http://tinyurl.com/sap-troubleshooting-method
===============================================================
For that Person Two commented:

Make that, 'development and test'
===============================================================
Person Three need this off his chest:

So, regarding recent security patches that might cause havoc:

The one that prevents old client retries in clear text when the encrypted handshake is rejected affects some connection attempts.

“Repairing” the permissions of the archive log destination can ultimately get the archiver stuck far enough behind to toss a 7445 (I think) I didn’t look it up and since my databases never have problems I get rusty on the error messages. (tongue firmly in cheek). If memory serves (see previous sentence) one of the security patches suggests repairing the permissions on the archive log directory without telling you to make sure the ownership is correct.

What are your vintages? I’ve only done SAP stuff ONCE (and walked away quickly and quietly having proved that a certain physical reordering solved all their stated performance issues on the load testing system only to be informed that any manipulation of the data outside of SAP was not allowed.)
===============================================================
Person One felt he finally had someone to talk to who understood him:

Yup, that is SAP SOP.

So far I have refused to do that particular operation, except in a couple cases where it retrieved a lot of empty space due to archival of data.

The 'make work' analogy from "The Longest Yard" (the old one with Burt Reynolds and Eddie Albert) was used in reference to the "SAP Reorg" mentality when they last asked me to do a db reorg.

They didn't get it.

For those of you that don't already know it, the 'make work' for the prison workers in the facility where Reynold's character was incarcerated consisted of the ollowing:

Morning: shovel mud out of the swamp.
Afternoon: shovel mud back into the swamp.

By the classic definition of work, nothing was accomplished in the end.
But the inmates were still sweaty, tired, thirsty and hungry.
===============================================================
Person Five then finally could say this to a friendly crowd:

I wholeheartedly agree with both of you.

I spent almost 4 years as part of team that supported SAP. There were 3 dba's on the team, and both of the other two received their primary dba training from SAP. Their method was to manage EVERY database as if it were an SAP database.

I volunteered to do every non-SAP upgrade and rearranged everything back Oracle standards when they weren't looking. :) Took about 2 years to get to them all but it was well worth the trouble.

As for the SAP databases, patch application and upgrades always had different results on dev, test or prod. It was utterly baffling.
===============================================================

Ah well, based on all this, it is really no wonder that SAP is so wildly popular and has won the whole upper(ERP market) over less complicated, cheaper, more technologically advanced - and way more agile - competitors.

If you can sit in the local CEO club and claim: "We actually managed to pay for the WHOLE SAP implementation with our own money and we're still functioning in several departsments..." your fellow CEO's will know that you have more money than God or AIG's dealers.

You will have the uttermost respect from them, as they scramble to try and explain why their predecessors in their respective companies chose to implement something different from SAP. Cheaper, of course.

But one day, when they have gained enough financial strength....

Here's one final salute to all those lost colleagues from the Oracle database space. We'll always remember you. You'll never be forgotten. Long live your memory.

Wherever you are. In whatever lifeform.

It’s time that we admit it. We did horrible things at OOW in Berlin. We’ve not told anyone for all these years, but the pressure is building inside. So I’ve decided to come clean.

We had just started Miracle, so we were only about eight folks or so in total. So we decided to go to the conference in Berlin all of us. We rented two or three apartments and also invited our friends (customers) to stay with us.

We drove down there in a few cars and found out upon arrival that the apartments were empty except for the mattresses on the floor. Oh well, easier to find your way around.

I’m still not sure why Peter Gram or someone else decided to bring along our big office printer/scanner/copier, but the guys quickly set up the network, the printer and the laptops, and then we just sat around, worked on the laptops, drank beers and talked about all sorts of Oracle internals.

I went down to registration and got a badge, so that was good. Then someone (forget who) came up with the idea that we should simply copy my badge so the rest of the guys could get in for free.

It wasn’t because we didn’t have the money or anything. Oh no. It was just because it sounded stupid and a little risky. So that’s why you’ll find pictures here and there (including in my office) of the guys copying and modifying badges.

The biggest challenge was that the badges had an “Oracle-red” stripe at the bottom.

But Oracle Magazine had a special conference edition out which had a lot of “Oracle-red” on the front cover, so it was just a matter of using the scissors in the Swiss army knife.

It worked perfectly for the whole conference and we were very proud, of course.

It was also the conference where I was introduced to James Morle by Anjo Kolk, our old-time friend from Oracle. I had placed myself strategically in a café/bar between the two main halls in the conference center which meant that everybody came walking by sooner or later. So I met lots of old friends that way. And a new friend named James Morle, who was in need for an assignment – and we had a customer in Germany who badly need his skills, so he ended up working for Mobilcom for half a year or more.

So the next bad thing we did was to crash the Danish country dinner. Oracle Denmark might not have been too fond of us back then, because they thought we were too many who had left in one go. Nevertheless, we thought it was not exactly stylish of them not to invite us to the Danish country dinner – as the only Danish participants.

Our friend (and future customer) Ivan Bajon from Simcorp stayed with us in the apartments and he was invited to the country dinner. So we found out where it was, snooped around a little, and then simply climbed a rather high fence and gate-crashed the dinner.

That was fun. The Oracle folks there were visibly nervous when we suddenly stormed in, but what could they do in front of all the customers, who very well knew who we were? So we sat down at the tables and had a good evening with all the other Danes there.

We had lots of fun during those few days in Berlin, had many political debates and beers, and went home smiling but tired.

To my knowledge we’ve not faked badges or gate-crashed country dinners since.

There have been a few suggestions since then that the badges we copied were actually free to begin with, but that can't possible be. I strongly object to that idea.

My friend Ole told me this story many moons ago - and many times since:

A man decided to teach his horse not to eat anymore. Gradually he reduced the amount of food the horse got each day, and the programme worked really well.

Unfortunately, just as he had finally taught the horse not to eat at all something unforeseen and tragic happened to the horse and it died suddenly.

I was reminded of the story the other day when I was studying this article in my beloved The Economist (I've subscribed non-stop since 1983):

Stopping in a hurry

Dec 11th 2008
From The Economist print edition

Cars are getting better at avoiding collisions. Before long they may be communicating with each other to make roads safer.... and somewhere in the article this is stated:

"Jan Ivarsson, head of safety at Volvo, believes it should be possible to build a car in which people will not be killed or injured."

On the other hand I can read in various newspapers that Volvo is not doing too well, and may in fact soon be either sold or closed, just like Saab. Or maybe Sweden will try to put those two together and create a small (by international standards) entity that might survive with state funding and what have you.

So you have this carmaker - Volvo - who has been making cars safer and safer and safer over the last several decades, and JUST as they're sensing the possibility of making the perfectly safe car - in which people will not get killed - the carmaker Volvo unfortunately died. Like the horse.

In my own, little world I have also been witnessing how perfect the databases are getting, how much they can do, how much stuff you can put into them in order to save on the application coding and development side - and how coders, developers and programmers have stopped using them. Just as databases were getting damn near perfect ... people stopped using them.

I have for several years now claimed that any computer technology that reached a state of perfection, a plateau of predictability & stability and a high level of usefulness ... will be replaced with something more chaotic and hence much less productive. I have seen no exceptions.

I now realise it is connected: Technology reaching maturty, car safety reaching its logical conclusion - and feeding of horses.

Here's a conversation I've been having over the last several weeks. Hope you enjoy it... Debra Lilley told me to put it on this blog. I always do what she tells me to do. Mogens.

Dear CEO,

But how much for Miracle.China.Com??? Please email in written statement for good happyness here and everywhere for you!

Also, with respect to your irrevocable demands and undisputed popularity will we pay in Icelandic Kroner cheap cheap?
==============================================

I just transferred the picture from my digital camera. I have no idea how to turn it. But if you put your screen/laptop on its right-hand side you should be able to get a rough idea of the shirt layout.

Let me know if anyone wants one. I'll take care of it. I think they were about 15 dollars for a normal T-shirt. Viktors shirt was bought in haste by his mother in a near-by shop because the shirt guy didn't have very small sizes.

Mine is just a plain T-shirt in black with the YWC motif.

So when I first saw this speech and the hair on my arms did funny things (the relevant stuff starts 10.28 minutes into it):

http://www.youtube.com/watch?v=Fe751kMBwms

... I thought: He'll win. He's the first presidential candidate EVER to use the creed of Bob the Builder (Bygge-Bob in Danish, which is exactly what my son Viktor, three years of age, calls my friend and employee CarpenterTorben, by the way).

So he won. Of course.

Then I told Dan Morgan of PSOUG the story last Sunday. He went home and found out that the Obama campaign had used the Bob the Builder figure at a rally or two. It was no coincidence.

Which might explain why McCain invented a false plumber named Joe.

So last week in Seattle (at the SQL Server PASS conference) Anette stumbled on a shirt shop that had a T-shirt with a picture of Obama and the words 'Yes we can' on it.

We talked to the guy, told him the story, and then he promply went into some Adobe product on the laptop on his desk and replaced Obama with Bob the Builder carrying multiple tools, put Obama's campaign logo on his hardhat, and made his skin colour slightly brownished...

The result was perfect, and I think he is now displaying shirts with that motive in his window in a small mall in Seattle.

Viktor was overjoyed (size four years). Dan Morgan, too (3XL) - he might even wear it in Birmingham for the presentation he's doing for Jeremiah (who can't come) at the UKOUG conference next week.... I'll certainly bring mine (2XL).

And for those of you who don't know Bob the Builder: Whenever Bob and his friends (machines and a few humans alike) are going to do some heavy stuff together, like digging a hole in Mrs. Petersen's garden, Bob will ask them all:

"Can we do it?"

and they will answer with gusto, enthusiasm and unbound energy:

"Yes, we can!"

In Danish it's:

"Kan vi gøre det?"

and the answer is:

"Ja, dét kan vi!"

Just so you know.

My sincere thanks to Viktor and Bob the Builder for helping me re-learn a few fundamental things about human capacity, and the power of optimism.

So my wife Anette and I are on our way home from Tim's wedding.

We flew British Airways both ways. In Copenhagen I told a lady at the BA check-in counter that I might have discovered a way for terrorists to put bombs on planes without being on the plane themselves.

You see, the last couple of times where I have had to change terminals in Heathrow and there's been approximately two hours or less between the flights my bags haven't made it.

So if the bags are onboard a plane but the passenger doesn't show up, they'll pull the bags. But if the bags are delayed they'll let the passenger fly without his luggage.

Everyone knows there have been huge problems with luggage in Heathrow. At one point there was more than 42,000 pieces stacked up. IBM had stopped a DW/BI project without having created even indexes on the Oracle database tables, so every piece of luggage required a full table scan of a rather large table, so it took a while to get over that one.

So I told the lady at the checkin in Kastrup airport, Copenhagen, that there might be a security risk in Heathrow and she said she would relay the information.

Well, apparently she did, because the checkin guy in Denver yesterday suddenly started behaving very strange, went into the backoffice to "do a security check" and marked our boarding cars with the dreaded "ssss" code highlighted in yellow which means "pay very special attention", and which meant that both Anette and I had to go through the new machine that will blow air on you so that it can smell traces of explosives, etc etc.

We're currently in Heathrow, about to board for Copenhagen. I wonder if our bags will make it.

So much for trying to warn the folks about a security problem :-))).

Mogens

Anette and I are in Denver, Colorado these days, because Tim Gorman is getting married to Lori tonight (Saturday). It's a hot wedding: This is, I think, the 20th day in a row with over 90 degrees Fahrenheit, which makes it the hottest period since 1857 or something like that.

Tim is very well known in the Oracle community: He spent many years inside Oracle where I had the pleasure of communicating a lot with him on the wonderful HELPKERN list there.

He also wrote a couple of books and contributed to the Tales of The OakTable book. Here's his website: http://www.evdbt.com/

Good luck to Lori & Tim! (said the guy on his third marriage...)

I heard this story from a Dutch friend, and I'm sure a) he's completely wrong and b) I've misunderstood everything he said. We must have been drunk both of us. With that in mind, here's the story as I recall it....

It's legal to use hash in Holland. It's also legal for the "coffee shops" to sell it. We all know that.

But of course it's illegal for the coffee shops to purchase it. And as an utterly natural consequence it's illegal to grow it. Of course. Makes sense.

And yet - despite this state of things - the Dutch haven't quite managed to push the hash thing out of the gangsters' hands :-)).

So the gangsters are now growing hash in apartments, on boats, and other places.

They use artificial lights, of course. And carbon filters so it doesn't smell. They circumvent the normal electricity circuit in order to avoid detection. They force weak people to front this kind of activity, of course.

So the Dutch Police are of COURSE trying to find all these places that illegally grow the hash which can be sold legally in the coffee shops. They even use IR (infrared) devices to detect lofts and other places where the heat/lighting raise the temperature of the growth rooms. In response, the gangsters are of course insulating the rooms.

It's all very green and sustainable, I think. And lofts and bad apartments, maybe even boats? - get insulation, which saves energy. It rocks.

The Police has of course out-sourced the removal of illegal growth places that they detect. A private company is called upon, and they'll arrive shortly afterwards and remove the plants, the equipment, and what have you.

This, I think, is brilliant. The private removal company now has an interest in people creating these illegal places, of course. Just like the folks that bred rats in New York in the 1890's because they could get a bounty per rat tail they turned in - because New York City wanted to get rid of all the rats in the city, of course.

As I started out saying: I must have misheard most of this and misunderstood the rest. Even Frederic Forsyth couldn't dream this one up.

Mogens

I started working for a bank called Sparekassen SDS 1st of January 1987. They had just bought Oracle, and that's how I ended up in the database world.

In 1990 I joined Oracle Denmark's support organisation under the magnificient leadership of Jannik Ohl.

He was fired by Peter Perregaard in 1998 or so, because they didn't like each other. Until then things were fantastic. After that things were not.

Jannik was replaced by Allan Marker, who was not nearly his equal in any which way you choose to look. Especially when it comes to the art of thinking instead of wondering how you can survive in the corporate culture for the next few months.

But that's how things are. Peter made a mistake, and he regrets it to this day, I'm sure (as in: sure).

So Jannik went into geo-stationary orbit. In other words: He joined the Oracle EMEA organisation (Europe, Middle East, Africa).

When you "go into orbit", ie. join EMEA or some global stuff, you're never heard of again. In space, nobody can hear you scream, as they say.

Until it's time to lay off some bodies. So Jannik, uhm, resigned just now.

Today I served a bit of Miracle beer for my friend Jannik in Oracle Denmark's canteen.

To honour the best boss I ever had.

And to honour one of the most creative minds I've met. Really.

He was the one that came up with the idea of doing serious database stuff in Lalandia (which is why Miracle now do two conferences there a year).

He was the one that told me: "With all this internet stuff and not-being-able-to-call-a-person thing going on in Support, people will pay for extra services that allow them to talk to people and get their problems resolved without too much bullshit" - and we now have 130 Miracle Support customers.

He came up with the idea of having a credit-card thing for Good Oracle Customers (GOC).

Miracle Support shouldn't be allowed to live. It's feeding off the failings of the big vendor support organisations, because they're failing. That's wrong. But it's a fact.

I just hope Jannik doesn't do the boring thing of leaning back and waiting for the early-age pension to arrive. He's not old, he's not spent. We need him.

As for the headline (The beginning of Oracle Denmark) I'll just say this piece of information from an unknown source:

The beginning of Oracle Denmark: Jørgen Balle, Ole Bisgaard, Hanne Cederberg & Jannik started at the same time. Then came Pete Francis, og later Klaus Holse Andersen.

We need more details, folks :-))

Mogens

My ringtone on my mobile is currently Highway To Hell with AC/DC, but I thought Chris Rea's The Road to Hell was more appropriate as a title today. I hope you'll understand why after reading this.

I've just come home from 10 days in a Danish town called Horsens doing a reality TV show called "The Secret Millionaire", which has run for two seasons in England.

Now they've done 11 programs in Denmark. Mine will probably be shown in the fall of this year.

Basically, a TV crew of three followed me all day long while I (complete with a cover story) visited places where good souls help out people in need. At night I stayed in a borrowed, Turkish immigrant apartment.

At the end of the 10 days I put on one of my Armani suits and told the good people that in fact I was not that much down and out, and that I'd like to donate some of my own money to their cause (a total of 250.000 Danish kroner, to be exact).

In fact I'm not a millionaire in the sense that I can take out that amount from my bank account at all. Instead, we had to take a loan in our house, which my wife Anette was OK with (and thank you so much for that!).

The 10 other folks look a LOT more like millionaires than me, let me tell you that.

Folks like the guy behind JustEat, a guy with his own investment bank in London, a big IT-guy called Asger Jensby, and so on and so forth. Some of them with private chauffeurs, one live in a French castle, for crying out loud. You know the type.

The filming ended last Thursday - a week ago - and it was a good day. Lots of happiness, tears, and much more. And of course I threw a big party with more than 200participants at the end of it.

Fantastic. But perhaps the most emotinally draining thing I've tried.

Then last Friday (the day after) after spending 30 minutes in my house while re-packing and re-grouping, I found myself with my co-director Lasse racing snow scooters and drilling holes through 70 cm ice on frozen lakes in Northern Iceland with hard, Icelandic men around me.

Talk of a change of scene within 24 hours.

A couple of rough days here in Denmark, and it was time to relax on this beautiful Easter Thursday...

The plan was to eat brunch with my friend Søren (who buys breweries for Carlsberg) and his family. Well, I made it, but late of course, due to all sorts of things.

Then I left around 1400 hours in order to drive back to my town Maaloev and pick up three kids and then take them to a football match between Brondby and FC Midtjylland due to take place at 1500 hours. Running a bit late...

I had 10 free tickets from Brondby because I tried to help them with a social project called "Fra Bænken Til Banen" (from the bench to the field) where they try to get kids into jobs (they've been so successful that they're now starting to find jobs for the kids' fathers, too).

But nobody wanted my free tickets, so I ended up throwing six of them away. Bah.

I was running a bit late for the game. Perhaps that's why I was driving too fast on the street where the Police was checking speed.

I was charged with driving 97 where 60 was the limit. Ouch.

That means 2500 kroner in fine (that's OK) and I have to take a new driving test (which cost a lot more and takes a lot of time). Hmm.

But hey, I get to learn about all the new street signs and rules that have appeared since I learned to drive back in 1982. Might even get one of those new, fancy credit-card style driving licenses.

While I was standing there talking to the cops, another car was stopped for speeding in the opposite direction of me. Turned out he had just been at Brondby Stadium, but had discovered that the game had been moved from 1500 hours to 1800 hours due to demands from Viasat television, since they had another important sport thing to cover today, too.

So evidently, one should check game times and not rely on whatever is written on the tickets. Anyway, that's how I discovered that it wasn't neccessary for me to drive faster today :-)).

So we drove back, and then we went to Brondby and saw a fine match (2-1 to Brondby) in rain and snow, then back to Maaloev with the kids and then back into Copenhagen to pick up my girl Nathalie (9 years of age) who had stayed with Søren to play with his daughter Louise.

Shortly before arriving in the street where Søren lives, I hit something with my right front tire and all air went out. Then I spent half an hour in heavy rain and sleet trying to change the #%&/Q tire.

Due to very slippery cobble stones, the jack kept slipping and the car crashing down. That happened four times, the last time with the tire only half way off, and that's when I called for professionel help (and a professionel jack).

They came, tire was changed, we drove home.

All through this I was looking forward to a nice evening with my wife Anette (whom I hadn't seen too much of in the last couple of weeks) and some cheese and champagne that she had promised me.

Like in 'Driving home for Christmas'. Chris Rea. Can't wait to see those faces. Oh, I'm driving down that lane.

It was late when I finally made it home, and Anette had had to go to sleep, of course. She had been up early and had been taking care of little Viktor all day.

Bummer. But there's always email and blogging for you, then.

But I just got a text message from my oldest daughter Christine (18 years old), who's on some kind of survival training thing with the scouts.

She wrote: "By the way: I love you, dad. Have I ever told you that?".

No, you haven't. And you never needed to. But it was the finest of timings when you did :-)).

I've just poured myself a large Bowmore 12 year single malt (Enigma edition).

Here's to life.

You have to watch this:

http://www.youtube.com/watch?v=CHzV4LZnvHc

We'll follow it up with a few other initiatives in order to help the big companies bring down the time spent to install Oracle from, say, 50 hours to one or two.

While reading the book 'Deep Survival' (most kindly given to me at the UKOUG conference in Birmingham by Sir Graham Wood of Oracle after the fire in my house) I happened on a description on page 107 of a book called 'Normal Accidents' by a fellow named Perrow (get it? per row - a perfect name for database nerds).

Perrow's theses is that in any tightly coupled system - in which unexpected interactions can happen - accidents WILL happen, and they're NORMAL.

Also, he states that technological steps taken to remedy this will just make matters worse.

Perrow and IT systems
=====================
I have freely translated Perrow's thoughts into the following:

IT systems are tightly coupled. A change - a patch, a new application, or an upgrade - to a layer in the stack can cause accidents to happen, because they generate unexpected interactions between the components of the system.

This is normal and expected behaviour, and any technological gear added to the technology stack in order to minimize this risk will make the system more complex and therefor more prone to new accidents.

For instance, I find that two of the most complexing things you can do to an IT system are clusters and SAN's.

These impressive technologies are always added in order to make systems more available and guard against unexpected accidents.

Hence, they will, in and by themselves, guarantee other normal accidents to happen to the system.

Complexing and de-complexing IT systems
=======================================
So you could say that it's a question of complexing or de-complexing IT systems.

I have found four situations that can complex IT systems (I'm being a bit ironic here):

1. To cover yourself (politics).
2. Exploration.
3. SMS decisions.
4. Architects.

1. Reason One: To cover yourself (politics)
===========================================
You might want to complex systems in order to satisfy various parties that you depend on or who insist on buying certain things they've heard about at vendor gatherings:

"Yes, we've done everything humanely possible, including buying state-of-the-art technology from leading vendors and asking independant experts to verify our setup".

This is known as CYB (Cover Your Behind).

2. Reason Two: Exploration
==========================
Ah, the urge to explore unknown territories and boldly go where no man has ever gone before...

Because you can.

The hightened awareness thus enabled might be A Good Thing for your system and your customers.

It could also create situations that you and others find way too interesting.

Reason Two is often done by men, because we love to do stupid or dangerous things.

3. Reason Three: SMS decisions
==============================
A third reason for complexing IT systems could be pure ignorance in what is commonly referred to as Suit Meets Suit (SMS) decisions - where a person of power from the vendor side with no technical insight talks to a person of power from the customer side with no technical insight.

These SMS situations tend to cause considerable increases in the GNP (just like road accidents and fires) of any country involved because of all the - mostly unneccessary - work following.

The costs to humans, systems and users can be enormous. Economists tend to love it.

4. Reason Four: Architects
==========================
A fourth reason for complexing IT systems can be architects. Don't get me wrong: There are many good IT architects. The very best ones, though, tend not to call themselves architects.

One of my dear friends once stated that an architect is often a developer that can't be used as a developer any more. Very funny.

However, what I have witnessed myself is that the combination of getting further away from the technical reality and getting closer to the management levels (the C class, as it were) tend to make some architects less good at making architectural decisions after a while.

That's where the vendors get their chance of selling the latest and greatest and thus complexing new and upcoming systems.


Summary: The end of reasoning
=============================
Four reasons must be enough. There are probably more, but I cannot think of them right now.

Anyway, imagine what savings in costs and worries you can obtain by moving just a notch down that steep slope of complexity in your system.

You might be able to de-complex your system to a degree where it becomes
absolutely rock solid and enormously available.

That should be our goal in the years to come: To help our customers de-complex their systems, while of course trying everything we can to support those who chose to complex theirs.

Now and then some new angles and thoughts emerge in a field where a lot of people think there's not much new to be said.

Two examples:

1. James Morle told me a while ago, that he thinks all performance problems relate to skew, to latency, or to both. It's brilliant, I think. I hope James will one day write about it. He's a damn fine writer when he gets down to it.

2. This one from Dan Fink. Impressive piece, I think. Enjoy it.

http://optimaldba.blogspot.com/2007/12/how-useful-is-wait-interface.html

When I emailed Dan and told him I admired his angle on this, he responded:

"I think it is a matter of keeping an open mind and knowing that you have friends and colleagues who are open to new ideas. Support is absolutely critical, even when you don't necessarily agree with what is being said. That keeps the flow of information open.

I shall never forget walking into a conference room. In big letters on one of the whiteboards were the words "THINK OUTSIDE THE BOX". For emphasis...someone had drawn a nice large box around them! "

I like that one :-)).