Stories from the front #1

Now for something entirely different. In the current edition of “UpTimes“, the not-so-regular magazine of the German Unix User Group, we can read some stories from admins who experienced strange encounters in server-rooms. I’m an engineer myself and I got some stories to tell as well. I’m not going to mention on which site from which customer it happened, but that’s not necessary. Some of those stories didn’t happen to me, but to my colleagues, but never mind: They’re all true.

Some years ago one of my customers build a cage around some very sensitive systems inside their server-room. This cage was intended to protect the system of unauthorized access. Problem was that they had servers and equipment from lots of different vendors in the same room; also the system which should be protected was running a billing-system of sort with access to account-data (“money”).

The whole site was very secure anyway: Guards, dogs, high fences, barbed wire, CCTV everywhere and a very sophisticated door-control system. It was some kind of RFID (or inductive-loop system, i don’t know) – you literally needed the access-card to open the door to the toilet. Strange about this system was that you had to check in and check out at every door in order to move freely around the building; if you forgot to check out at one door because somebody hold the door open for you, you where stuck in that part of the building.

The only solution was to call up the building-security and ask them to release your card again.

As you might guess, in some parts of the building was no phone; on some phones the number of security wasn’t noted; and in some parts of the building mobile phones were banned as well.

Part of the security-concept was also that the card was encoded every day when you checked in at the guard-house. Someone from the customer had to give permission until a certain time – like 18:00 o’clock – until you must have left the site. If you were still in the building after that time the card became invalid and you couldn’t even leave the site.

On one occasion I, and a colleague of mine, were in the above-mentioned cage for the commissioning of a new cluster. We were already working since 08:00AM, it was already close before 18:00 but we knew that we would’ve to work for another two hours. We called up our contact from the customer and asked him if it’s possible to extend our stay and if he could call up the security to extend the validity of our security-badges. “Yes, no problem” he told us and gave us a callback like 5 minutes later – everything was sorted out, our cards were extended to work until 21:00.

So my colleague and me kept working on the system until like 20:50, when we decided it’s time to pack our gear and leave. We collected our tools, grabbed our stuff, hold our card to the cardreader of the cage – red. No way, my card didn’t work. Meep. OK, my colleague tried his card – red. No way.

Since i already knew the procedure I took my cellphone (which was actually forbidden in that place but there was no phone in the cage either) and called up the guard-house to ask them if they could enable our cards so that we can leave. The dialog we had was bizarre:

Me: “This is Janssen speaking from $company, we’re locked in the cage in server-room $n, would you be so kind to enable our cards again so that we can leave?”
Guard: “Oh, I’m afraid I’m not in the position to fiddle with the system, my colleague who’s entitled is just gone for the round.”
Me: “(…wtf…?) Well, I don’t care, we’re about to leave anyway, I just want you to enable my card for the next couple of minutes so that we can leave the cage and the site.”
Guard: “I’m sorry Sir, I’m afraid I can’t do that.”
Me: “(annoyed) Then would you please call up your colleague who’s gone for the round and ask him to enable our cards?”
Guard: “I’m afraid he wouldn’t do that either, I have no note in the computer that you’re allowed to stay in the cage until this time, you were meant to leave at 18:00, did you know that?”
Me: “But Mr. $x called you up to tell you that we’re allowed to stay longer!”
Guard: “Yes, I’ve a hand-written note here, but my colleague from the day-shift made a mistake; he can’t take that order on the phone, we need a written statement. And it’s not in the computer.”

I knew this was utter bullshit, the phoning-up thing worked for the past couple of months. However, I’m there as a vendor and can’t start fucking around with the guards, so i tried to keep calm.

Me: “OK, what do you want me to do? Shall I call up Mr. $x again?”
Guard: “Yes, and he should send form ABC as a fax to the guard-house.”
Me: “What, this guy already knocked off, what if he doesn’t have a fax-machine at home? Never mind, ok, ok, I’ll do it.”

I tried to call up my customer, with no effort. Then i was totally pissed and called up $customer’s project-leader explaining him the situation. I got a callback a minute later, telling that I should call up the Security NOC, they would take care of the problem. At that point it was already like 21:20.

I called up the Security NOC (also responsible for access) and explained them the situation. The dispatcher put me on hold and tried to connect me to the correct person.

Guard: “Guard-house?
Me: “Is this the Security NOC?”
Guard: “No, this is the guard-house; didn’t we talk like a couple of minutes ago?”
Me: “Yes, that’s me, would you please put me over to the Security NOC again?”
Guard: “Sorry, i don’t have their number.”

That was the point where I almost exploded:

Me: “If you not going to open that fucking door in the next couple of minutes I’m going to, I swear by god, I’m going to press the Fire-Alarm Button!”
Guard: “Er, well, err, what were your card’s serial-numbers again…”

And what’s the morale of this story?

  • Don’t be too patient.
  • Always carry a mobile-phone with you, even if it’s not allowed.
  • Do not hit the Fire-Alarm button if you’re not sure if that would open the door which prevents you from escaping, but a Argon/Halon-system is in place
  • Learn the phone number of the guard-house.

But you know what was really really pissing me off? It was 21:45 when my colleague and me left the site, went to the guard-house to check out: The other guard, who was doing his round earlier, was sitting in his bullet-proof box, smoking, and said:

Guard: “You look like you had a hard day at work, mate – see you tomorrow!”

Tech Tags:

3 Responses to Stories from the front #1

  1. […] Stories from the front #1 […]

  2. Marcus says:

    hey!

    Thanks for the great post, its good to read personal experiences.

    Marcus

  3. zywcu says:

    Hi!

    I have really experiance with that place. In Poland we have really great Serverrooms. Security Systems are too sophisticated ann there are a liitle bit bottleneck like computers, systems.

    Regards

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: