Enterprise software testing enables your business change, drives your software quality and focuses your remediation efforts. It can also slow down your projects and cost you way more than it should.
I’ve worked in enterprise software testing for the last 15 years and I’ve noticed 3 simple steps to stop you spending more than you should:
By testing sooner, later and less, you can maximise the gain and minimise the pain associated with testing.
You need to find the right defects at the right time; System defects in system testing, Integration defects in SIT. Often system defects aren’t found until integration, or even worse Performance or UAT.
How will this reduce cost? Defects are more expensive to fix the longer they remain undetected. Functionality is built around them and is dependent on them. Anyone who has worked in software development will tell you, unpicking those nested changes can be a real dog’s dinner.
The graph below shows the cost of change curve, as stated by Barry Boehm (Software Engineering Economics, 1981).
Nowadays projects are more complex, involving many interconnected and often disparate systems, and testing is carried out more widely throughout the SDLC. The knock-on effect being an escalating cost to fix within the traditional testing window. I’d conservatively estimate defects are roughly 10 times more expensive to fix in the next stage and 100 times more expensive 2 stages later. What this actually means is that the same system defect will cost:
- £100 to fix in system testing
- £1000 to fix in SIT
- £10,000 to fix in performance
I’m not saying you need to test everything early, far from it. Just test the right things at the right time, which leads to…
The flipside of the last point. Some testing activities really need to be held off, often later than you think. There are 2 prime candidates for this:
- Performance testing: There is little point in Performance testing an unstable system. Wait until you’ve completed and signed off SIT and ensure you have a window of exclusive access to an appropriate environment (QA/Pre-Prod) and you’ll save yourself a world of pain and cost.
- Test automation: It is incredibly inefficient to attempt automation while changes are afoot. I recommend holding off automation until you’ve gone live, come out of ‘hypercare’ or equivalent period and have an agreed regression pack ready.
It’s not only the cost you need to worry about if you do these activities early, it’s also the validity of the results themselves.
These phases can be incredibly important, just at the right time.
This is not about arbitrarily reducing testing windows, or removing test resources. This is about understanding the risks associated with your functionality, and understanding which of those risks are acceptable or not. When you’re deciding this, you may well hear, “it’s all high priority”. A good question to keep in your mind is, what would happen to our business if this bit fails?
Once you know the unacceptable risks, you need to create a lean pack that covers that functionality. Coverage matrices often highlight huge inefficiencies in testing (or even worse, huge gaps). This activity alone can easily cut the size of a test pack in half.
By testing sooner, later and/or less you will cut time and costs out of your projects. Not only direct testing costs but also:
- Each day saved cuts a day of all your project overheads; management, development etc
- More dynamic business with quicker changes in the future
- Business users aren’t frustrated by defects in UAT