Saturday, April 20, 2013

The $100k Door Stop

In my spare time, I enjoy decorating. My sons’ bedroom is decorated in a pirate’s theme and they love it. They are constantly running down the hall between the living room, their play room and their bedroom. At times, you just need to listen for the sound of little feet slamming on the floor or the sound of their bedroom door being flung open and make sure you’re out of the line of fire. I won’t tell you how many times I’ve had to jump out of the way to avoid being run over.

I was thinking the other day that it would be a good idea to get them a door stop for their bedroom door so there would be one less potential casualty in their mission to move as quickly from point A to point B with as few interruptions as possible. Well, I found one. I absolutely fell in love with this door stop. It’s a rope knot doorstop, sold by Ballard’s Design. For those interested, here is the URL:  http://www.ballarddesigns.com/rope-knot-doorstop/accessories/doorstops/10736?defattrib=&defattribvalue=&listIndex=0. The door stop’s price is $45. While that’s not an exorbitant amount of money, I think it’s a little pricey for a door stop, so I’m probably going to attempt a DYI project.

So, how does this in any way relate to technology? Let me tell you about a door stop that costs a tad bit more than $45.
A few years back, a peer, an Architect Manager walked into my office and delivered what I’m sure, he considered a landmine. “I just bought a server that your team will never be able to support.” Having been in the position for less than 6 months, I wasn’t sure exactly how to respond to the comment so merely said, “Great. Thanks.” He turned on his heel, seemingly disappointed at my lack of reaction. Internally, I was cursing life and this dysfunctional boob who was willing to sacrifice company stability for personal reasons.

In that particular instance, this Architect Manager purchased a system that no one in our company had ever supported and on top of that, the only requirement of any sort that was provided was that it be on an operating system that again, my engineers had never supported, Power Linux. That system became a very expensive door stop. It was delivered but no project plan had been developed for the system replacement. Once the business was engaged and a project plan created, it was found that the initial purchased needed an additional $100k or so of additional purchases to support the environment; storage, licenses, training, etc. The system sat for a year before any work was ever even started. 1/3 of the warranty period was eaten up waiting for teams to make plans and implement the solution. An additional six months went by before a timeline could be found to take a business outage. You see, this system was for a replacement of the support of the most crucial application in the company. It housed the largest database in the company. No small fete to replace. A three day outage would be necessary so Thanksgiving was the planned window. Teams across infrastructure, database, application support and the business were necessary for successful implementation.
The end of the story? A year later, the system was not performing properly, was impossible to update, the vendor did not provide reliable and experienced support and the business was again screaming about performance. A full system replacement was necessary in order to resolve the issues. The additional problem this time around? Our credibility was at risk.

This story is the perfect example of why IT purchases have to be scrutinized and considered from multiple views. At the time, no architectural standards existed for server environments. The only standards that existed were for desktops and those were sadly outdated with only a gig of RAM being the standard. You’re probably shaking your heads and wondering how it was possible to operate under such chaotic circumstances. The evolution to stability was not easy and was fraught with changes, both personnel and cultural. Otherwise, your IT purchases become extremely expensive doorstops and your IT department is taking away from your corporate stability and growth versus contributing.
·         First, you have to create, maintain and follow industry best practice supported architectural standards,
      o   By doing this, you insure decisions will withstand scrutiny
      o   You create an environment that is supportable and sustainable
            §  Purchasing systems that have the same core operating system or hardware is a key way    to reduce support cost
·         Next, a heck of a lot of planning needs to take place before hardware delivery is scheduled
      o   Who is going to provide the requirements?
      o   What is the system lifecycle going to be?
      o   How does this impact current business goals?
      o   What other projects are going on and how does this fit in with those?
      o   What is the priority?
      o   Is training necessary?
·         Engage business analysts and stakeholders whose involvement is key to the success of the project AND to the understanding of the application functions. (You may wonder why I added this bullet item because this may seem a natural approach. Some architects assume they understand the business well enough to avoid getting the business involved in the planning and implementation components.)
·         Schedule hardware delivery once you have established how much time is necessary for testing, configuration and training for your team
  o  Protect your precious warranty period (every day the system is in production past the initial warranty is at a costlier warranty level)
·         Allow your teams time for hardware burn-in (Hint:  less than a year)
·         Document baseline performance expectations
·         Document the Support Plan
·         Use both Architects and Support team engineers to plan these steps

I don’t know any business that wants to or can afford a $100k door stop. Maybe that $45 isn’t so bad after all.