Stories from the front #3

In this edition of “Stories from the front” I’m going to tell you something about colours. Yeah. Colours, you got me right.

When I started working in the cluster-business I was a comissioning-engineer, doing all those pesky upgrade and updates of the cluster-software, our cluster-framework, the applications and such. It was a job which boiled down to “pkgadd“, “pkgrm” and “init 6“.

Our usual way of upgrading was upgrading the OS, upgrading the database, upgrading the cluster-software followed by the framework – and, at the very end – the application.

I deal mostly with high-available clusters optimized to run 24/7/365 with no downtime and only minor little impacts on the reliability; however, the software must keep running. The clusters are usually just two-node SMP-clusters, at that time typically a two-node 16x MIPS R10000 CPU cluster (RM600 E60 for the initiated of you).

When i started at The Company a tutor was assigned to me; someone who teached me how to do things, how to do them right, how to avoid errors and such.

Since the cluster i was about to upgrade was a two-node cluster, the upgrade procedure was quite simple:

  1. Split cluster
  2. Upgrade node #1
  3. Start up node #1, let it join the cluster, see if everything is running fine
  4. Upgrade node #2
  5. Let it join the cluster and check if everything is runnin as expected

If any of those steps failed, we were about to fall back to the old version of the software and everything.

All the installation-steps happened on /dev/console which we accessed trough cu inside two different xterms. My tutor had his own way of remembering which console is running in what xterm; the colours of the xterm showing console of node #1 was YELLOW on BLUE background, console #2 GREEN on PINK background. I found it very disgusting, distracting and it generally hurt my eyes.

I just said “well, console #1 is left, console #2 is on the right.

My tutor complained “NOOOO, you’ll never going to distinguish!!

Distinguish my arse, I thought and started to upgrade.

Eventually I was finished upgrading node #1, started the application, everything fine expect some processes which were new and refused to start; I expected this because node #2 didn’t yet know about those new processes so I started to upgrade the second node.

When I finished installing the second node I started the application and my tutor and me started to check the processes, if everything was running smooth.

It didn’t. Some processes still refused to start so I investigated what might have gone wrong.

After a couple of minutes I realized something really nasty… I installed the same cluster-node twice.

My tutor just said: “SEE? If you just have changed the colours as I’ve told you…

Now, 6 years after, I still refuse to change the colours as he proposed; but I use the “reverse video”-feature xterms offer you when pressing CTRL+right mousebutton.

A lesson was learned though:

  1. uname -n helps
  2. reverse video too
  3. follow rule number one: Think before pressing enter.

Got a story to share? Drop me a line!

(Markus: See it as a rebuttal to myself considering your _save-incident… Deep in the shit we’re all equal.)

Tech Tags:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: