Think You Had A Tough Monday?

image description

Think You Had A Tough Monday?

by William DiPaolo

You gotta love a Monday like the one we just had.

We teed up today’s events with a huge change to our networking gear over the weekend. It involved new DNS routing as well as gobs of new equipment to increase speed, redundancies and security. Solid planning by Brian Gardner, our CTO made the whole thing happen in a lot less time than we anticipated. On Saturday night I got a call that all was good – and I enjoyed a stress free weekend.

Cogent Road’s main application, Funding Suite is a very complex application that connects to a multitude of third party mortgage systems. Unfortunately, our new Cisco firewalls had a firmware update that caused unanticipated issues when these systems tried to access Funding Suite. I’m not making excuses, maybe we could have tried simulating these connections in production – you know, hindsight and all the rest – but regardless our system was not operating fully.

What followed was a blur of texting, cell phone ringtones and cars pulling into the office all at the same time. Brian was already on the phone with Cisco hounding them for an explanation when I got in. I did what any good CEO would do – I set a Costco sized bottle of Excedrin on his desk and left.

The entire situation lasted about an hour…which seemed like two weeks. Seeing an engineer poke a smiling face and exaggerated thumbs up in your office door brings with it a special kind of relief only software folks can understand.

Then all hell broke loose.

Lights Out!

I had just begun a joint interview (I could not make this up) about Avail with our industry’s most veteran technology editor and one of our large clients. Just as I finished an initial comment – everything went dark. And quiet. No computers, no lights, no phone. Just immediate dark silence – all at once. And it was still Monday morning.

Turns out a transformer blew in San Diego which took out a good chunk of the La Jolla area. This meant we had no power and no telephones. Every customer that called was greeted with what I would argue is the most awful greeting a customer could ever hear – a busy signal. Our software was up and running – our datacenter runs on huge back up generators. But our phones were flatlined.

It wasn’t until two hours later that the power came back up and our heart started beating again. Customers didn’t seem to mind the inconvenience – and some even shared their own similar stories. But, in the end, I’m glad to see this day end.

Leave a Reply