The real problems with technology projects and technologists

I actually started writing this blog a week before Obamacare’s website snafus hit the media, but it further underlined my thoughts on what’s wrong with technology today – or rather, with technology projects and implementations.

I’ve had the privilege and honor to work with some extremely competent technologists over the years. Whether they were Microsoft Premiere teams, HP’s 3^rd level engineers, Verizon’s architects, support teams and DBA’s, or VMWare’s Implementation Engineers - these guys know their stuff.

To be as intelligent as we are though, we really know how to screw up a technology project. This has nothing to do with our “hard skills” and everything to do with the lack of “soft” skills.
·         Communication: Be honest with yourself. How well do you communicate? Specifically how well do you listen?
       o   Do you hear in 0’s and 1’s?
       o   Do you take the time to completely listen or are you already formulating your response?
       o   Have you ever sat down and watched a business person work through or on your support product?
     §  You would absolutely gain new respect for the business if you do.
Most techies would rather have their mouse taken away from them than talk with the business. Get over it. Start small by nodding and smiling as you walk by them in the halls.

·         Cross-Team Skills: Do you sit down with technologists from other teams and try to understand the limitations/abilities of the technology they support.
      o   When was the last time a DBA communicated with the application team that the adhoc report they needed put undue stress on the system if ran against production data?
      o   As an App support person, do you let the testing team know you are running “rough draft or dirty” on the newest release?
      o   Do you let a Project Manager know that you are concerned regarding release scope?
      o   Do you as Project Manager let the Business stakeholders know that you can’t meet proposed release dates with the resources available?
At the end of the day, you can only do what you can but if you don’t put yourself out there and talk with the other teams, it’s a great way to fail your team and your company.

·         Narrow versus broad: I’m not talking body style here but vision. What variables does your support area inject into the overall environment?
      o   You may be a great email administrator but do you necessarily understand data management requirements?
      o   Supporting messaging, BYOD admins are feeling the heat from the network guys because the network guys weren’t aware the messaging guys were going to start letting people use their mobile devices at the office.
There are no islands where technology is concerned. Keep in mind that OSI Model.

·         Think like a mortal:  (while I say that tongue-in-cheek) read that as – think like the business/customer/consumer
      o   I am going online to order something – is it easy? Is it obvious where/how I go about my task? Are there help tips to assist me?
           §  Is there an unnatural pause that makes me wonder if something is broken?
           §  Is there feedback that sets the pace for me?
      o   If your app is web-based, do you have sufficient web servers in place to support the potential audience?
           §  Do you consult with the network guys to confirm that the metrics match expectations?
           §  Do you work with the Virtual Environment support team in the event you need additional systems put into the load balancer?

·         Test like a mortal: You cannot test like a techy. We don’t work, think or act in the same manner that someone front-facing does. We don’t even type the same way. Accept it – we are not the business and therefore we need the business to test. Repeat after me: Just because I support the system does not mean I am smarter than or understand the business. Get the business to test.

I think that the traits that make us really good at what we do – also make us guilty of being arrogant – and therefore, unwilling or blind to accept that there are areas we don’t necessarily excel in. So now what? Look at it a little differently. Would you want to go to a doctor who had only received training in the skeletal system? Would you knowingly work in a building designed by an architect who didn’t understand the correlation between temperature change and materials used?  In order to better comprehend the whole, one has to understand not only their working piece of anything but also the impact to the whole.

The problem with technology training is that the focus is too narrow. The courses teach the HOW in the most narrow of circumstances but not the WHEN or WHY. This can be fixed though. Look at COBIT. Look at ITIL. These frameworks encourage communication, broad vision, working closely with the business, working closely with other technology teams. I cannot say enough about how a technologist can grow professionally once they consider the frameworks and their value.

No, it's not the network

I would be a rich person if I were able to count the number of times an IT person has claimed that the reason for slowness in an application or system is because of the network. As I type this, I can hear the cheers of network support teams across the world shouting “Hallelujah, somebody said it other than us.” Ok, so maybe not across the world but at least Tampa and Jacksonville.

The network is and always will be an easy target for system issues unless teams are given the tools to “easily” disseminate issues. A number of these tools are free and come with the operating system or database. What we’re really talking about here is training for IT staff. I don’t mean reading the books, studying for a test, passing and then become a “real” engineer. I mean understanding the OSI Model's Seven Layers.

·         Application

·         Presentation

·         Session

·         Transport

·         Network

·         Data Link

·         Physical

It is impossible to completely segregate each layer, so it is completely logical that one layer having issues can bleed into the others. This is really where problem management, thorough root cause analyses and careful incident management comes into play.

I love (not an exaggeration) troubleshooting. I have been asked repeatedly to help teach people how to troubleshoot. While this is possible, far more often, troubleshooting is an art; a skill culled over a course of many years.

·         Start with the obvious. This may sound silly but it’s not uncommon for engineers to “assume” that a system issue must be something complicated and exotic.

o   Has the issue happened before? If so, when? What is the failure frequency? To the same user/system or another?

§  If so, what resolved the issue then?

This is where strong incident management comes into play. Recording issues with resolutions allows for trending to point out repeated issues.

·         Are errors logged? If so, use resources to look up potential resolutions.

Windows, Linux, Unix operating systems ALL record errors. Applications can be configured to write errors to logs. Words of caution:  Massive error writing can cause massive log entries. Verbose logging should only be used for true errors, not application support.

                This is where the beauty of the internet and support agreements come into play. Internet sites such as eventid.net and vendor sites offering searchable databases can make fast work of troubleshooting.

·         Create a team across functional areas to troubleshoot and perform root cause analysis work.

o   Consider triage training for your teams (ITIL and MOF are excellent guidelines to follow)

·         Create a “no blame” environment

·         Track changes made to the environment – this should really be given your highest focus as most issues occur because of changes made to the environment.

o   Was a system having memory issues before the application was updated?

o   Was communication between environments an issue before a firewall change?

·         Track vendor releases for potential issue resolution.

o   My favorite catch phrase is “Trust, but confirm” – I would never have taken that tack in 2001 with Microsoft. Through the years however, Microsoft has heeded the message from customers that we would not accept crappy code any longer.

o   So, test, test and test again but UPDATE.

·         Is it the network? It could be but most likely it’s because of an architecture issue. Insufficient bandwidth was architected for the ever evolving needs of a mobile world. BYOD brings its own issues in that everyone connects – and they connect across the network. Is it the network? No, it’s the increased need for bandwidth.

As always with technology, communication is key. It’s not unheard of for an engineer to have a “back pocket of tricks” to resolve issues. This cannot be accepted by management. These steps, resolutions must be documented in order to make the overall environment a stronger and more successful one. Reward the guys with the bag of tricks but be sure they help those less capable troubleshooters. This is not about job security for an individual; it’s about the strength of the whole.

And, if you haven’t figured it out yet, it’s RARELY JUST about the network.

Consistent Inputs = Consistent Outputs (Critical message to IT Leaders)

I like consistency. I like knowing that every time I put the house key in my house’s lock, the door will open. I like knowing that if I keep batteries in the remote control, my television is going to turn on when I point the remote at it and click the power button. Of course, there are exceptions, but for the most part, that is everyone’s expectation. That when we do the proper “input”, we get the proper “output”.

In the car industry, most of the processes have been automated. Why? To get the same repeatable output each time a “product” is manufactured. This is the same across the rest of the manufacturing industry. When I was “much” younger, I worked in the Phosphate Industry as a Statistical Analyst. I loved it. I enjoyed the logic and predictability about inputs, additives, process flows and at the end of the day, a product.  A predictable product that was made up consistently of the same amount of product and effort – give or take a few percent points.

If that is the case, that you can create a predictable product by creating, following and perfecting process flows, then why are so many IT departments so resistant to implementing processes and procedures and documenting their expected end product? Isn’t that what we provide? A product?

The inputs are our knowledge, resources and efforts. The product is our problem resolution. The problem with this? You can’t count on the knowledge, resources and/or effort to be consistent. So, now what?

There are different ways to answer this question with most leading to automation or outsourcing --- but it doesn’t have to. Internal IT teams can level the playing field but don’t expect it to be without some effort and soul-searching on your part.

·         Take an honest look at the business you are supporting,
     o   Can you meet their needs with your current support model? (Worry about exceeding later – for now, just MEET expectations)
  o   How do you know?
          §  Ask the business. Face to face. Honest conversation. Don’t get defensive. (You’ll win serious brownie points for asking – as long as you follow through)
·        After you finish crying or punching out a wall in the men’s/ladies’ room, get your team together and go over your notes.
     o   Evaluate where you are falling short,
·         Create a Success Matrix (I’ve loaded one that works onto my website CGSOLUTIONSOFJAX.COM – feel free to use it – one less item for you to  have to worry about)
     o   Create a scale for Needs complete Reworking, Needs improvement, and Needs Attention.
     o   Be sure and have a column for Effort Required and another for Cost of Effort. This way you can prioritize each item based upon your resources.
·         Assign responsible persons to track progress and lead the effort.
·         Create checks and balances. It’s easy to focus on the low-hanging fruit while letting the back-breaking items wait. Don’t let this happen.
·         Ask business leaders to participate in meetings about the matrix – this way you have provided them feedback, asked for their participation which, by the way, means their tacit approval and buy-in.
     o   Be picky. If a mid-level manager has the authority to say – yep, that’s important to us, then he or she is the right person to select. If not, pick someone else.
     o   You could get turned down but you made the effort.
·         Celebrate and reward the progress.
·         Try hard not to focus on the negatives. All it will do is slow you down and discourage any positives. (I think this one is tough for people. I take my work ethic and progress personally and I get frustrated when I fail to meet my own objectives. Remember that you are only failing if you give up.)
·         Focus on creating a consistent work process. Teach it. Live it. Embrace it. (k, nuff said huh?)

Keys to Success:

·         Be realistic about what you can do with your existing team.
·         Breathe deeply frequently and stay positive (you may have no reason to be positive but if you let yourself get overwhelmed by the task at hand you’ll get overwhelmed and be of zero hope/help for your team or your business.
·         Be objective. If you can’t demonstrate the value of improving an item in business terms, focus elsewhere.

I’ve performed a fair number of technology and security assessments in the past two years and what I have found is “consistent” – although not consistency of the kind I’d prefer. The grass is NOT always greener in the Outsourced pasture.

The realities of internal versus external support:

·         Neither Internal nor external teams communicate well with the business.
·         Internal teams are not getting the training or support they need and are expected to both build and run their environments on shoe string budgets.
·         External teams are expected to do more with less just as frequently as internal teams are.
·         External teams get the training so that they can differentiate themselves from other teams.
·         External teams have executives who smooth the feathers and promise change in the event of issues. Obviously this can only go on for so long before changes are made there as well.
·         External teams are not perfect. The message from the President gets just as lost on its way down to the people on the floor as it does in an Internal Support team.

More often than not, when an internal team quits producing results, the business notices – more frequently than not – before IT management does. Close relationships with the business are key. They are not from Venus and you’re not from Mars or Pluto. You have to quit talking in bits and bytes and start talking in terms the business can relate to. Trust me, the external executives do.

How the NSA "Should" have prevented the Snowden fiasco

For a moment, let’s put aside the fundamental discussion about what information the NSA is or should be collecting. Let’s focus on the actual actions Edward Snowden seemingly took, according to what information is being released and how NSA should have handled the situation differently.

First, let’s look at who the NSA is and what they are responsible for. I went to the NSA’s website (http://www.nsa.gov/) and Wikipedia for guidance.  The NSA’s mission as stated on their website, “The NSA/CSS core missions are to protect U.S. national security systems and to produce foreign signals intelligence information.” I’m going to make an assumption here and state that the NSA does not have common restraints such as lack of resources or funds in order to protect the U.S.’s security systems.

If I were designing the infrastructure for the NSA, I would take a layered approach.

·         Implement an IDS (Intrusion Detection System). Your thought might be, well that’s great – that keeps the bad guys from getting in and while that is true, it can also prevents data from getting out without being detected as well.

·         Create zones for data based upon sensitivity.

·         Within the individual zones, separate servers from workstations into individual subnets so that data flow can be monitored and contained within the individual zones.

·         Limit open firewall ports to only those necessary and monitored.

·         Monitor typical traffic patterns between environments.

o   Report any irregularities and investigate.

·         Limit the subnets and IP addresses that can communicate with each.

·         LOG Entries.

o   Limit who can erase logs. This is a simple check mark in a group policy. Not rocket science.

o   Have a person NOT providing a specific function reviewing the logs and running correlations.

·         Implement Separation of Duty steps. Continued, daily “need to know or need to access” should be considered when advanced or privileged permissions are assigned. If that level of access cannot be confirmed, have the person open a ticket for access, get approval from a supervisor and then revoke within a short and reasonable period of time.

·         Limit screen capture ability.

·         Disable ALL USB drive access. Yes, USB devices make jobs easier for Admins but for the very reason that it’s small and can be used to remove data, it should not be allowed.

·         Create an Exceptions list for any access or transfers.

o   Have a responsible party reviewing and approving the exceptions lists.

·         Limit FTP (File transfers) from sensitive subnets.

·         Implement a data governance program that includes a risk matrix and timely reviews.

o   Provide reports to an audit function outside of technology. While this may prompt some needless questions or explanations, it also places scrutiny on the environment.

·         Eliminate generic accounts. DOCUMENT AND REVIEW ALL EXCEPTIONS.

·         Limit service accounts to running services and implement “DO NOT ALLOW LOGON” through Group Policies. Have regular reviews of the service accounts and their scope.

These solutions have NOTHING to do with the age of the systems at the NSA. That’s a whole other discussion around patching, maintenance, business continuity. All of the above items can be implemented with 2003 technology. The IDS is the only exception.

Next, let’s look at Snowden’s job. He was a System Administrator. By definition, system administrators are responsible for safeguarding the infrastructure systems. They do NOT own data. They are not system owners. They are not information owners. System Administrators are not even responsible for safeguarding the data. That role is held by a Database or Data Administrator.  I saw an article that said that as System Administrator, you ARE the auditor. That is simply not the case. That should never be the case in a regulated industry and that certainly should never be the case where our country’s secrets are concerned. That Sys Admin had a supervisor. That supervisor should have been alerted by any number of incidents that occurred.

·         Implement Role-Based Security. In this way, Sys Admins are restricted to the environments they regularly work. Implement policies and procedures to limit access.

·         It IS possible to logically limit what one can access and still do their jobs. Does it make their job more complicated? If the environment is not adequately architected, yes.

o   Restrict who can add who users to what security groups,

o   Restrict firewall people from servers from databases and visa versa,

Another article spoke of Snowden’s physical location being an advantage to him. He worked off-site. While this may seem to be logical, it simply isn’t true that that gave him an advantage. That means that his machine’s connectivity came from an external IP address. That information would have to pass through an additional device that logged the info. Why would someone, working externally, need to download Mbytes or Tbytes of data to his machine? Troubleshooting? NO. Absolutely not. I’ve worked in technology for over 30 years. There has never been an event that prompted me as an Engineer or Sys Admin to download data from a server to my own workstation. Never.

I have a mantra, “Trust, but confirm.” This isn’t because I don’t trust people. I don’t believe in putting temptation into people’s hands. When I started working in technology many years ago, there was an unspoken rule – “Never give anyone any reason to question your integrity”.  IT people, by the nature of the job that we do, have access to data. The best thing a security-minded manager can do is limit unnecessary access so that no one is tempted to breach their employer’s trust. If that means implementing more controls than an average business may have, so be it. Putting better controls and a review process in place COULD have prevented our country being unhinged by the scandal.

The bottom line for me is that the NSA has resources the average company cannot begin to compare to, but yet, there were insufficient reviews or controls to “watch the watchers”. The lack of proper security tools, processes, policies and procedures created this fiasco, not a cowboy sys admin. He simply took advantage of the lapses. Quit with the excuses and fix the environment.

If you aren’t necessarily confident that a layered security model is implemented in your environment, give CGSolutions of Jax a call. We have resources and partners who can help secure your environment. It isn’t rocket science.

It's the end of the Life Support for Windows XP

Windows XP was a good operating system. I fondly remember the stable Operating System that withstood a premature effort at retirement when Windows Vista was released. Unfortunately for Microsoft, Vista just was not well received in the business arena. The reason? It offered little to no benefit for the effort of an upgrade. That’s the difference between Vista and Windows 7 Professional. Windows 7 Professional has been a viable and stable Operating System pretty much from its release. It offered additional security features that forward-thinking IT departments were quick to focus on.

The bad news? Microsoft will no longer provide support for Windows XP past April 8, 2014.

The worst news? There are A LOT of companies that have yet to migrate to Windows 7.

Reasons for not yet upgrading?

·         Budgetary reasons,
·         Lack of security guidance,
·         Multiple, complex in-house developed applications that rely on Windows XP and an insufficient development budget,
·         Lack of priority,
·         Lack of lifecycle management program,
·         Just didn’t get around to it,
·         Lack of awareness as to the seriousness of this deadline.

Reasons why it matters?

·         No further support from Microsoft,
·         No further security updates,
·         Other software will cease to be updated with support provided for Windows XP

What can be impacted?

·         Ability to respond against targeted attacks
·         Compliance – GLBA, PCI, HIPAA

There are a lot of implications for a company that is still on Windows XP. It COULD be that there are insufficient IT resources to focus on upgrades and maintaining OS levels. What’s scary about that however is that if an IT department isn’t focusing on something as major as an operating system upgrade, where are they with other support questions?

Questions such as

·         Are parameter devices maintained at the appropriate level? One would hope so but isn’t it the same department that isn’t focusing on OS’s?
·         How about patching?
·         If the desktops aren’t being updated in a timely enough basis, what does that say about the servers? Is it possible there are still Windows NT machines sitting around?

So now what?

·         Train your support team on Windows 7,
·         Architect your next OS footprint,
·         Gather an inventory of all desktops,
·         Confirm that the hardware will work with Windows 7 requirements,
   o   If not, have the business order the machines necessary for replacements
·         Pick a business advocate out of each department to work with the IT department to sign off on the testing and the Disaster Recovery Plan
·         Reach out to software partners,
       o   Confirm that their systems are compatible with Windows 7. If they are not, find out what their plan is for support.
       o   You could well have to change software vendors
·         Reach out to device partners such as teller machines or credit card/debit machines,
       o   Confirm that the systems will work or plan on a secondary project to update these as needed
·         Begin departmental User testing to confirm current applications ARE compliant with Windows 7,
       o   Note: This does not mean IT people assigned to test applications. IT people do not test the same way end users do and end users know how their software is supposed to function, whereas IT people interpret what it is supposed to do.
       o   Note2: Schedule the testing over significant production cycles. Don’t overlook End-of-Month, Quarterly, Annual cycles. Software components that are key to the running of the business could only be used four times a year and elements would be easily forgotten and create catastrophes if there were a subsequent failure.

·         Train your business. This probably won’t require more than an hour of training (far less than moving from Office 2003 to Office 2007 did) but it will go a long way toward relieving stress.

The Plan:

As of today, you are one day shy of 7 months. I would shoot for a February 21, 2014 weekend deployment. This way, your accounting and sales departments hopefully have year-end out of the way. With sufficient IT and business resources, you could have your departmental desktop images built (30 days after feedback from partners) and pushed out to testers hopefully in time for them to get  30 days of testing under their belts. If you don’t have sufficient resources, there are IT resources willing to come in and assist (feel free to give me a call, I have mad skills). Leave the users alone in December and start back with a training schedule for January. Hopefully, you’ll be good to go for the first deployment weekend. Stage your upgrades across business units so that you avoid key dates. Understand that performing an upgrade under these circumstances will require more resources, manhours and financially.

And prayer, prayer is always good.

 

Education, education, education. Did I mention education?

·         Christmas Eve, 1997, it was a story of “Not a creature was stirring, except for the email administrator”. I had started a new job in October as an Applications Manager for an LLP to whom email was crucial. One of my challenges moving forward was to migrate the company from Groupwise mail to MS Exchange. The Groupwise system had been having a lot of issues, according to the CIO. He felt that the aging technology was causing undue stress on the company’s partners and, therefore, undue stress on him. Until such time as we were able to schedule the migration, it was my responsibility to stabilize the existing email system to reduce the noise about email. That afternoon, I remember reaming out an attorney for sending pictures of his daughter at a horse stable to over twelve relatives. His emails were slowing down the delivery of the dancing elves that others had sent. Stuck between those files were some crucial contracts that had an expiration date of midnight. The glut of files took time to clean out and caused mail to be backed up for hours. Thankfully, Exchange is easier to cleanup and bandwidth is cheaper.

·         In 2000, I was told by an older partner at the LLP, “You can’t fix stupid”. He in fact called the person who caused the 3^rd or 4^th outage due to an outbreak of the Iloveyou virus. I was standing in his office while he bellowed at the unfortunate person, “Are you insane or just stupid?” Although we had gotten quite proficient at our response to the breakouts, time down for attorneys meant less billable hours.  After that incident, we thought out of the box and came up with a solution to prevent a spread of the virus, before the antivirus companies came out with signatures, and over a year before we bought an email filter solution.

·         When Michael Jackson’s memorial service was streamed across the internet, we actually had to shut down streaming (possible only because of the technology we had already purchased) and access to social websites to keep end users from shutting down or slowing access to our revenue generating web-based applications.

·         Going over a corporate file server (inevitably looking for disk space), we first targeted media files. Over 30% of the directories had a variety of non-work-related music, video and photos. Before you ask, yes, there were policies that employees signed off on that notated there should be no such files saved on corporate systems.

·         Performing a social engineering “test”, employee after employee failed (including technologists who felt they should investigate a dropped memory stick in case it fell into the wrong hands) to follow corporate guidelines.

Looking back, end user behavior hasn’t significantly changed. Luckily, technology has evolved to manage these situations with more success than in the past. But is that what makes the most sense?

·         Email filter - There are multiple products and services that manage this better than any threatening phone call and far less expensive than dealing with a data breach.

·         Antivirus/Malware solutions are absolute MUSTS.

·         Disk Space management products - Data quotas can be set up at system creation but once a system is in place, it’s impossible to go back and perform cleanup without manual involvement. A good ole dos batch file scheduled to run once a week can go a long way toward keeping servers clear of media files that shouldn’t be there.

·         Patching solutions – I appreciate the simplicity and ease of patching with a managed solution but depending upon the size of a company, it may not be feasible. This has to be a cost benefit decision on the part of technology. How many hours and techs does it take to patch the environment? Can this be done manually in a timely enough basis that it prevents exploits? If the answer is no, then consider a management solution that offers sufficient flexibility so that patching can be managed in a risk adverse manner.

·         Intrusion Detection System – Consider this as a risk proposition. If you are a financial services company, the risk may be greater than the cost of reacting to a breach. With security industry prescribed firewall configurations and port blocking on subnets, it is possible to adequately defend your parameter. With other security restrictions in place, it IS possible to detect and defend against fraudulent insider behavior.

Creating a stable, structured and secure technology environment does not happen out of luck, not forever.

Taking a layered approach:

·         Begin with end user and technology team education,

·         Taylor your policies and procedures to support the risk footprint the corporation is willing to support,

·         Don’t jump to the conclusion that expensive software solutions will fix all of your problems. Without adequate processes and procedures in place, a team will fail, regardless of the tools provided.

·         Implement built-in support tools

o   Don’t ignore the value of logging,

o   Don’t ignore the value of team-led strategy sessions for issue reviews,

·         Invest in your team’s education and morale

·         Invest in solid vendor partnerships. They will be as interested in your success as you are.

 

And honestly, prayer never hurt.