How to cause a complete server meltdown

Andy Pedisich   March 25 2009 07:15:00 AM

Here's a sure-fire recipe to follow if you want to cause complete havoc and spend a bunch of hours trying to fix mail files and inboxes on your Domino servers.

First, allow unbridled growth of the size of mail files.  Have a bunch that are big, and not just 10 GB or 15 GB either.  Make them really big.  Take them all the way out to 58 GB if at all possible.

Next, let your users bully you.  That's right.  Let them take over and dictate their way into doing what ever they want to do with their mail files.  Especially, let them keep every stinking message since Notes was brought into the environment in their inbox.  Let them keep important messages from 1999 with subjects like, "There's a green Chevy truck with its lights on in the parking lot," and "Are you going fishing Saturday?"  These messages are truly priceless, if only for their classic down-home humor and reminders of how wonderfully simple life was before the Internet bubble burst.

Let those inboxes fills with those messages, and don't let them stop until they reach 80,000.  No, really, allow the users to go whole hog.  Let some of them take the inbox to a whopping 140,000 documents, because Administrators are only the jerks that make the system highly available every day for their now business-critical, can't-live-without-it-for-even-a-minute-email.  Screw your service level agreements, by the way.

As a little added bonus, don't put any common names of the servers into DNS.  Make the clients work for a living, forcing the use of connection documents so that a failover becomes a nightmare during which CEOs and VPs question whether Notes/Domino is worth the investment.  Unfortunately, this is an easy-to-fix DNS change, so the sabotage won't last longer than 10 or 15 minutes.

Now, here's the killer.

If you are running out of disk space because 1 TB just isn't enough to satisfy your Star Trek bit map graphic-saving Luddite users, add more disk.  But to make it more interesting, add it while the Domino server is up and running.  That's right.  Grow some cojones, and do it during prime time, preferably on a rainy day when everyone is inside.  Pick a really cool time to do it, like 10 AM when the peak of users is at its prime.  Make it very exciting!

At first, your users will begin getting message at the Notes client level telling them there were "insufficient system resources to complete operation."  They like that, because it's an error message that makes sense to them.  Then your disk queue length will climb to about 50, and your servers will begin to hemorrhage.  It won't be yellow.  It will be rage red.  The townspeople will be coming to your cube with pitchforks and torches, straight out of a 1930's Frankenstein movie.  Managers will be screaming, almost crying, asking you, "Why is the Domino server failing?"  The weeping will be pitiful. The sounds of their voices will be like fingernails on a blackboard reeking of the stench of colored chalk.

Afterwards, when the dust settles, and you've tried to save another corrupted inbox, seen your last "invalid or missing" design document, and received you final "database is corrupt" message, you'll be happily able correlate the start of the swan song with the addition of the disks, and you will emerge victorious.

With luck, you will only be hated and not be the recipient of more than a dozen death threats.  At best, management will come to understand that regardless of what Dell says, you should never add disks to an on-line Domino system. Dell apparently doesn't know squat about Domino, but you do.

The moral:  Don't be bullied into doing the wrong thing.  Follow your best practices.

Did this really happen?  

What do you think?  Could I possibly make this stuff up?

- Andy

07:15:00 AM March 25 2009