Until August 2012, Knight Capital Group was a very successful American financial services company engaging in market making, electronic execution and institutional sales and trading. That was until 1st August, between 9:30 and 10:00 AM, when the companies trading software decided to buy high and sell low on 150 different stocks. In that short time, KCG had lost $440m, around $10m per minute. In a single day the company’s stock price dropped by 62%.
Whilst we don’t know exactly what went wrong at KCG, what is clear is that malfunctioning software systems can have a rapid and catastrophic impact on your business. Clearly not all IT quality issues are quite so dramatic, but many can still have a very serious effect on your business’ reputation. In this, the globally connected social age, you can expect half the planet to know about IT “glitches” almost instantaneously. For example, in June 2012, when NatWest customers couldn’t process payments after a problem introduced during normal maintenance, the twitter-verse went into overdrive, significantly damaging both their brand and that of their IT service provider.
Whilst DevOps means different things to different people, for me it is simple; DevOps is about QUALITY! It is the rigour, discipline, tools, processes and procedures that you should put in place to ensure maximum quality of your IT systems and software – thereby reducing the risk of IT change to your business. Unfortunately, as with most quality initiatives, implementing a strong DevOps strategy takes time, effort and know-how, and must be incorporated into any IT initiative from the beginning. Whether you are writing a sophisticated software solution from scratch, upgrading a server or rolling out a new software package, a strong DevOps strategy will help you succeed.
Like all quality initiatives, a good implementation usually requires the involvement of many departments. DevOps is not just an IT thing!!! IT systems are now at the core of almost all businesses – one mistake can cost you and your business dearly.
Do you know exactly what artefacts (code, scripts, configurations, etc.) are in a version of your software? This sounds incredibly basic but I am constantly amazed how many IT departments could not replicate a product build exactly – or even at all. One of the repercussions of this is that small changes/fixes take considerably longer to implement and test because the IT team needs to work out the starting point before they can make the necessary updates. “Working out” the starting point can be error prone and subject to human mistake and often introduces “bugs” into systems that previously worked well – so fixing one thing can in fact lead to an even more unstable solution.
Do your IT systems suffer from intermittent, seemingly random failures? Whilst not always the case, this is more often than not due to inconsistent system build and configuration. In modern computing environments a single application will tend to run across multiple servers (for load balancing, scalability etc.). Again, it seems very basic but very often these servers do not have the same configurations, and in fact sometimes not even the same exact version of the applications. This leads to hard to trace application issues.
Is your team slow at implementing change and in your opinion overly nervous of doing so? This can be a sign of a poor testing strategy and process. With the right testing strategy and tools, your team should feel confident, almost certain, that they can quickly make changes, test the result and deploy into production.
Decades of experience have shown that there is a common theme that runs through IT projects that deliver on time and to a high quality; whether they call it this or not, there is a strong DevOps ethos running through the entire team.
There is often a misconception that moving to the Cloud or consuming SaaS services and solutions reduces the need for DevOps – in fact quite the opposite is true. Whilst your service providers will carry out much of the system change activities for you, you will still need to ensure that all integration points are well tested and that you have the processes to manage “enforced” change from your service providers that sometimes happens without much warning.
The worst mistake of all, is not considering the DevOps strategy at the start of any IT project. This is often done because it is deemed to be an overhead and “we just want to get started”. In the long run, retrofitting good requirements management, configuration management and testing will give your projects the best chance of succeeding and help to protect your organisation from a potentially devastating mistake.