Performance Testing Tools – Test Tool Analysis

In this article, we look at Test Tool Analysis and the steps to perform a proof of concept.

It may seem like an obvious question but when it comes to performance testing, do we really need a test tool?  Are you certain that purchasing a performance test tool is the best option to meet your requirements?

Firstly consider the following:-

1. Can you use a test harness? (sometimes known as an automated test framework) It may be cheaper to develop a dedicated test harness rather than purchasing a commercial tool.
2. If a test harness is not a suitable option, consider whether you want to buy or download a tool or even have it hosted externally. Hosting the tool externally may have advantages, such as: –

• It may be cheaper than purchasing a tool. (This may not be as cost-effective if purchase return is balanced over several years.
• No infrastructure maintenance is required.
• Usually, minimal setup/installation required.
• In most cases, this can be a good option for public-facing websites.

NB Performing a Proof of Concept on this hosted tool is still recommended.

If none of the above are suitable, then you are ready to perform a Test Tool Analysis.

Test Tool Analysis

How do you know which tool to use?

You need to look at the testing requirements, the system under test, and what the tool needs to do?

There are many influencers here in the equation when considering which tool is the ‘best fit’ as there are a multitude of different performance tools out there, not all of which are suitable.

To help determine the most suitable tool to meet your criteria the following questions may be considered: –

1. How many licenses will be required? (Different tools may have different limits and associated costs.)
2. Can the tool support the technology/protocol used by the system/application?
3. Can the tool run the tests required by the project?
4. Does the cost of this tool fit into the project’s budget?
5. Can the tool be installed within the existing infrastructure or wherever it is needed in this project?
6. Is the hardware required readily available or easily sourced?
7. Does the company have testers that are proficient in the use of this tool? If so, how many?
8. Are there resources available to perform the testing required in this project?
9. If not, how easily/quickly could a performance testing engineer without experience of this specific tool gain competence? (In other words, how steep is the learning curve?)
10. How quickly could a proficient user of the tool create scripts, prepare tests, gather and analyse results?
11. How useful are the reports generated by the tool?
12. Is there a results evaluation module in the tool?
13. Is monitoring built-in?
14. Does it cover/support the relevant parts of the system under test?
15. Can external monitor data be easily imported into the resultant data?
16. Can the tool assist you in drilling down into an issue and finding the original cause? For instance, Will it go down to the API level or the code level?

If the answer to most of the questions above is yes for a specific tool, you can then move to the Proof of Concept stage.

If more than one tool fits all the criteria, a tool comparison would be appropriate.  

Feel free to refer to the articles in our comparative testing tools series.

A Comparative Look At Performance Test Tools – JMeter V NeoLoad

A Comparative Look At Performance Test Tools – LoadRunner V Silk Performer

A Comparative Look At Performance Test Tools – Gatling V WebLOAD

A Comparative Look At Testing Tools – Eggplant Performance V The Grinder

Proof of Concept

Before deciding on which performance tool to use, it is important to carry out a Proof of Concept (PoC) to determine if the performance tool is suitable. i.e. will it work for your business and will it meet your requirements?

Why do a Proof of Concept?

Choosing a tool without conducting a PoC is a risk as no matter how good your assessment of a given tool is unless you have verified/used this tool against a similar system with the same technology and protocol and with the same kind of approach, you do not know whether the tool will fulfil your requirements.

How to perform a Proof of concept?

1. Install the performance tool. You can normally download trial versions of any of the commercial tools. for that very purpose.
2. Take 1 or 2 key business processes.
3. Transform the automation script(s) if it exists, to a performance testing script. If it does not exist record/create the script. Feel free to refer our article, Performance Testing Framework
4. Enhance the script by adding correlation, error handling, parameterization, and verification points to it.
5. Set up a couple of monitors.
6. Run a small load test (2-10 users) for 1 hour.
7. Analyse the results. a) If you get errors, drill down to understand what caused them. b) Find the transaction that takes the most time and find out where most of that time was spent (drill down to the lowest level available, (i.e. lines of code)
8. Produce a PoC report with a summary of findings.
9. Share the report with the Senior stakeholders to review who are ultimately the decision-makers.

In conclusion before selecting any performance test tool, some analysis should be done prior to selection and a proof of concept should be undertaken to prove it will work as you expect.

To see how SQA Consulting may assist your company in performance testing your applications, please contact us.