In the testing world we’re sometimes guilty of making the simple look complex maybe even more so with in Performance Testing!
However when you lift the bonnet on the dark arts of Performance testing you will find that it is just like any other type of testing. Its central purpose is that of risk mitigation and demonstrating that the solution delivered meets its requirements.
For Performance testing there are the Non-Functional Requirements (NFRs), those requirements that define ‘how a system operates the way it’s supposed to’ as opposed to the Functional Requirements that define ‘what a system is supposed to do’ but they are system requirements that define success of testing non the less.
So in what ways is Performance testing just testing, well successful Performance Testing will
- Focus on the key scenarios that impact your business
- Focus on the end process not individual applications
- Use testing tools to create testing efficiencies, an audit trail and repeatable assets and test results (We would recommend looking at the HP Testing Suite for SAP and for Performance Testing that is HP Performance Centre and HP Loadrunner)
- Uncover test defects (areas of the solution that do not meet the NFRs) and re-test following the resolution of the defect – this is the process of tuning
Like functional testing where different phases are used to identify various aspects of the solution that do not meet the requirements (Unit, System, System Integration, Regression, User Acceptance Testing (UAT)), different performance test cycles are executed iteratively (i.e. with fix & retest) to validate different types of requirement depending on the business goals:
- Component Tuning and Baselining – Individual system components interrogated with base loads to enable build, configuration and tuning of system parameters and establish baseline measurements
- Load – Representative scenarios of parallel running processes running at given loads (Normal, Peak, expected future) to demonstrate that the performance will be acceptable to the business under different circumstances
- Stress – load is steadily increased over time to find a break point in the system and validate system sizing
- Batch – test of (usually) overnight batch activities to determine if when executed with realistic volumes of data they complete in the required timeframes
All tests potentially yield results that can be measured against define acceptance criteria demonstrating coverage of NFRs from the perspective of both the End User (End to end response times) and the system (Technical Key Performance Indicators)
Defects in the live system are expensive, have an impact on the business performance and costly to fix – functional testing and performance testing both try to uncover these issues prior to go-live so they can be resolved more cheaply and before they impact the business. Where Performance Testing differs is that it can offer more than that….
For example, on a recent project a company was implementing a SAP UI5 solution as a front-end to their SAP CRM solution.
Their call centre handled 8 million processes a year with an average time of 67 seconds per process – 6246 mandays of effort every year!
Performance Testing was able to demonstrate that the UI5 implementation provided a 70% improvement in response times saving 2924 mandays per year.
But through Performance Testing and Tuning of the system, we were able to improve the response times of the system further to give an 80% improvement in response times saving a staggering 4054 mandays.
Meaning Performance Testing & Tuning was responsible for saving over 1000 mandays per annum or £400,000 per year every year.
As a tester, the skills required to successfully deliver performance testing are a combination of the process and solution understanding required for functional testing but also the technical understanding of the application, architecture and how to monitor and interpret the results.
It’s difficult to find people who can seamlessly combine both skillsets to work equally well on both functional and performance test engagements so good performance testers are highly valued as they can work equally well on both functional and non-functional testing assignments.
Testing is always a cost saving exercise and performance testing is no different, the opportunities to remove large costs from a business on an ongoing basis are perhaps surprising. Perhaps in my next blog, I will provide evidence of the surprising cost of defects in your live environment and show why investing in good testing whether it be Performance or Functional testing saves you money.