What is Performance Testing

What is performance testing and why should we bother with it? Whether a question or a statement – it seems a reasonable point to ask? For those at the sharp-end if performance testing or engineering, the answer is simple: it is about determining a system’s ability to process one or transactions concurrently to expected volume levels at given times of day within operating windows. It sounds simple, but for those not in the game, performance testing often looks like an esoteric subject they feel is best left to the technical experts. We all need an understanding of performance testing! Whilst I agree it is probably best left to the experts to conduct performance testing using specialised tools, there are many reasons that key stakeholders of a project should have a greater understanding of and need for performance testing, particularly when planning. Whatever, your role, make no mistake that performance testing is vital to assuring successful operations of both new and changed systems – before you send them to your production environment What’s the end goal of performance testing? So, just why is performance testing so important and what does it strive to achieve? I’ve seen many definitions, but for me performance testing is about determining if a system or process will:

• Meet the processing and turnaround needs of the business;
• Underpin any SLA in place for operations;
• Physical processing capacity is sufficient, both now and with planned future growth.

You might not agree with these points and say much more is needed, and you would be right at a detailed level. But, as a basic statement they serve to describe performance testing at a high level. Ok, so what’s involved in Performance Testing? As we said, performance testing is about many different things. To break it down, let’s look at the primary constituent parts of performance testing that your organisation might plan and execute to measure key acceptance criteria and provide key information that informs your decision-making; be that more testing, more tuning, a green light to launch, limited launch, a red flag or something else: Each of the different classes of performance tests in the diagram are long accepted by the industry as key to operational and non-functional success. Indeed, they have a clear definition of what they set out to achieve and the reasons why we should consider incorporating them into our corporate and programme level performance strategies and plans. Whilst some of the classes might look similar, each has its own unique and distinct function. Performance testing and the eight key measures: Performance testing can be used to measure many things, including what are now considered by many as the eight mandatory tests to measure, minimise and assure:

1. Network latency;
2. Memory leakage;
3. Database indexing;
4. Transaction response;
5. Transaction throughput;
6. Operating windows;
7. System breakpoints;
8. Recovery times.

Performance testing and its key test classes: Let’s take at them look at them individually so that you might consider how to provide assurance that the eight key measures are fulfilled to assure operational performance: Stress Testing. It is used to determine how your system or process reacts to and recovers after failure. In such a test, a system is loaded with a selection of different transactions and processes that will be used in the day-to-day business. Essentially, stress testing establishes the system breakpoint. Whilst the breaking point is clearly important, the test provides another key operational measure, and that is an analysis of the recovery time and pattern and actions to get it working normally again. You never want your systems to crash to an unrecoverable position, but you do need the data and demonstrable assurance that it will you can recover from unexpected peaks in mixed transaction throughput. Volume Testing exists to subject a system or process to a huge volume of data. Volume testing is performed to analyse the system performance by increasing the volume of data in the database. We use volume testing to determine the impact on transaction response time and system behaviour when it is exposed to a high volume of data base transactions. I.E. Not necessarily volumes that are intended to break the system, but volumes that are higher than would be expected in a normal business day. Load Testing exists to assert that your system or process can handle a significant number of the same transaction concurrently; as might occur when, say, lots of staff login to their systems simultaneously. Essentially, such a test is designed to target specific areas of a system where multiple instances of the same transaction are expected to determine that it can cope and not crash. The difference between Stress Testing and Load Testing is that Stress Testing uses a mix of transactions to test the overall systems and its architecture, whereas Load Testing is used to focus on a single area. Scalability Testing is a key test for any organisation or system that is expecting an increase in transactions of over a quantifiable period of time. This is a post-implementation safety that executes tests based on expected growth rates. It targets all areas of the system, including infrastructure, architecture and transaction throughput. This type of test can be used to extrapolate the point in time when, based upon business forecasts, the systems will no longer have the capacity to process the number of transactions and significant changes will be required. Endurance (Soak) Testing exists to assert that your system or process can perform under 100% load for an extended period. Such a test is used to determine how your system of processes perform under sustained stress (not load) and may well uncover holes in your system or architecture not found by the other classes of testing, such a such as memory leakage. Spike Testing is where we go over and above the maximum design capacity, just to see how the application can deal with a big spiking load.  Generally, what you’ll find will be a ripple effect but what we’re looking for is for the application not to fall over. Of the six classes, each may may be used to test individual process or measure and an entire day’s processing. They are not intrinsically linked and need not be conducted in a particular order. It is, though, important to consider performance testing in the context of a ‘characterised and profiled business day or days’. I.E. What is expected and what tests do we need to assert that that expectation can be met by the system. This might include a normal business day, month-end, start of day, end of day, etc. Typical Performance Measures: The graph following shows:

• a normal business day operating at an arbitrary 200 transactions per day;
• Growth over a sustained period that takes volume beyond the normal business day;
• Planned numbers for endurance testing, spike testing, load testing and stress testing.

The point being about the above is that it is vital that each different processing day be characterised and profiled in terms of time, transaction and volume, and that the six classes of tests are planned and conducted to:

• Determine that transactional and processing spike, load, volume and stress are manageable for sustained operations over and above the normal business day(s);
• The growth rate is manageable and at point in time;
• Any changes in profile are sustainable.

Can I elect not to use a class of testing? Of course, you can. However, I wouldn’t recommend missing any out, but it may be that your organisational strategy dictates which classes you must do and which, if any, are optional. You could, for example, consider missing out a phase if you can identify key reasons for doing so: The following suggests some possible reasons for omission: Summary In summary, performance testing is about your peace of mind that your new or changes system or process will underpin and continue to underpin your business. You might think that some classes of performance testing can be omitted – and that may be the case – but as a minimum we strongly recommend that:

• You understand the key operational parameters are of the business and service management team and any SLA;
• You characterise and profile your business days and plan tests that your system must meet, such as transaction response, system resilience, transaction throughput and system recovery time;
• You have the right tool to engineer and manage performance testing for you;
• You don’t try to do manage performance testing manually!

And whatever you do, make sure that you have the right:

• Training to turn your performance testing into the science of performance engineering
• Skills to manage the tool to its full extent to maximise defect or test failure analysis.

TSG Training offer a full programme of Performance Testing Training, spanning from a simple 1-day introduction through to complex tool usage over 5-days. Performance testing – it is not for the few, but for those who care about systems performance and company reputation.