In this article, we will be exploring the various outputs of a typical report using the JMeter performance test tool and how it helps in understanding the performance of an application. It is not an exhaustive list but some of the more useful visualisations are included. In a previous article Performance testing – Statistics we looked at the various statistical terms used in performance test reports, which may be a useful introduction to understanding the outputs of a performance test.
It is always much easier to understand a complicated set of data if it is represented in a visual or graphical format, as opposed to trying to comprehend tables of raw data – a picture does paint a thousand words – so it makes sense to use them in your reporting. However, it is also important to use graphs that are targeted so that they highlight issues that need to be investigated. This could mean that custom graphs are created that are filtered on time or granularity or compared with previous test results to show differences and trends. There is no one size fits all, and some thought needs to be applied when considering the information that needs to be highlighted and conveyed and the audience the report is intended for.
In a JMeter report, there are many default tables and graphs that are produced some of which we will cover here. The main table that is produced is the summary “Statistics” table which gives an overall impression of how well the application performed in the test. There is no time series related data here, consequentially it is not possible to know how the application behaved at different times of the test. This alone is not a useful indicator and should be used in conjunction with other information. An example of a summary table is shown below.
The response time distribution graph shows the number of occurrences for each response time collected, identifying the high response time transactions. It is desirable to have most of the occurrences shifted to the left of the graph as it indicates quicker response times. As the response times increase to the right of the graph there should, ideally, be fewer occurrences.
The graph below is probably the most common graph shown in a performance test report. In this example, it shows the average response times for each transaction at 1-minute intervals.
The advantage of this graph is that it is a time-series graph showing the performance over the course of the whole test so that any unexpected peaks can be identified and investigated.
Knowing at what times the peaks occur allows the investigation in the application, server, and database logs to be narrowed down and correlated. Other correlations such as CPU, memory, and network utilisation of servers at those times can also be done so that a fuller picture can be formed of what was happening to cause the peaks.
The percentile graph is a useful one, showing the percentage of the response times that come under a particular response time. In the example below, 80% of the transactions are 5 seconds or below for the yellow transaction and about 99% or below for the blue transaction. Most applications will have service level agreements for important transactions stipulating a certain percentage of transactions should come below a defined response time.
Related to the percentile graph, JMeter also provide the Response Time Percentile Over Time graph as shown below. In the example, the 90th and 99th percentiles response times are highlighted for all successful transactions in the test. This can be used in conjunction with the average response time graph and should follow the same peaks and troughs. In some ways, it is a more useful than the average response time graph especially if there are SLAs for a percentage of transactions to meet a response time criterion.
A useful graph to have is the http response codes over time. In the example shown below, the test had successful responses (http 200 responses) but if there were any http 400 or 500 failure codes then the times those occurred can be investigated further in logs etc. The peak and trough of the failure codes can be correlated with other activity that could have occurred during those times.
This graph shows the active threads (virtual users) over the course of test. This graph can be compared against the other graphs to show how response times change with the change of active threads. In other words, do the response times change as the load changes? This is a good comparison to make as it can show at what loads the response time becomes unacceptable or the application fails.
A related graph, Time Vs Threads shows the response time of transactions against the number of threads. In the example, the increasing load does not increase the average response time of the transactions.
This graph shows the number of bytes sent and received per second during the test. This is another way to evaluate how much load is generated on the application and how much data needs to be handled by the client.
It is a good idea to create dashboards that can show changes in the performance of an application for each test run using some of these graphs or custom graphs created from the test outputs. That way it easier to spot trends of response time behaviour improving or getting worse so that it can be investigated, and action taken. Things that could have changed the behaviour could be down to code changes, infrastructure changes, or other activity occurring on the test environment.
In conclusion, not one graph or table on its own will give an objective view of the performance of an application. It is important to thoroughly analyse everything to get an overall picture. Relying on one or two graphs would give a false impression which can be dangerous for an application going into production. Finally, when reporting to stakeholders, the appropriate information needs to be conveyed concisely so that good decisions can be made, as too much information can be overwhelming.
To see how SQA Consulting may assist your company in performance testing your applications, please contact us.
