The first thing as a reader of this article I encourage you to question, why would you automate a system in which the development is still in flight? Please bear with me on this as we will get to the why.
Yes, it is painful for the testers developing and redeveloping tests multiple times. Objects change and have to be updated, you just learn to live with it.
Functionality changes and I’ve even seen the meaning of a checkbox change from one to completely the opposite. Other annoyances are the changing of values between uppercase and lowercase in fields. Not only that, you will make a change to keep in line with the developments being rolled out only for the development team and business to decide it was better the previous way or worse continually flip flop between the two.
Surely there is no point to this automation if it requires maintaining so often? “Automation-in-flight” produces a Regression Test pack that has a few band aids applied to it thus it’s not perfect; but what if we are moving through multiple environments that are being refreshed every time? What if we have to prove that the environment we were handed actually worked in the first place? Then on top of all of this we still have to prove we have cutover our new functionality into the environment and that it works. That’s three times testing has to be performed in that environment, is that really worth all that investment in automation?
Well for one of our Customers it wasn’t just the one environment we performed this testing in. We started off creating our scripts in what eventually became the training environment. The next system acquired (Quality) required the 3 testing phases described above. We were there to make sure that there was a working system for System Testing and System Integration Testing to start. Of course at this stage our pack was made up of legacy and existing processes with no new functionality being tested. Consequently, the objective of the Automated Test pack was to prove the system would hang together and that new functionality hadn’t regressed previous functionality.
We then transitioned from Quality to the UAT environment, again running this multiple execution approach. This time however the early scripts that were ready for the new functionality were also executed validating not only the existing functionality works but also that some of the new functionality worked prior to anyone touching a manual script.
Finally, what we had all been waiting for, the final execution of this release – Regression. Again, we needed to validate the environment pre and post refresh, and with all the new developments. The full pack was executed and the results were in.
Surprisingly there were fairly few defects, had we missed them? Go-live, even now 6 months on there have been so few incidents that we can look back on the automation approach as a real success. Testing windows had been shortened, cutover and handover windows of environments had been reduced. The Return On Investment starts to be visible and the pain of so many script updates suddenly becomes worth the while.
If you are thinking the point of this article is about ROI then we had better discuss it. The 3 phases of execution saved approximately 3 weeks of execution in each of these 4 environments. Therefore 12 weeks of savings in total – what does that mean in monetary terms? This particular programme was running around £400,000 of costs a week. In total, savings were just over £4 million after automation delivery costs had been taken out of the equation.
Who can argue with these savings? Now the reason for “Automation-in-flight” is clear even if it makes the automation engineers life a little more difficult throughout the programme.
If you would like to find out more information on Experior’s approach to delivering Automation through our Automation Assessment service or learn how we can support the ongoing maintenance of Automated tests through our Regression Managed Service get in touch.