There are many types of web performance tests that ensure your application is fast and stable: load, stress, capacity, soak, smoke tests. However, importance of each test type, test scenarios, scale and environment vary from project to project. It will also affect the right tools to script out, execute and manage the tests.

1. Choosing the right tool

There are so many tools for the job (free, open source, commercial, etc.) that it’s hard to think of where even to start to choose one.

Perhaps, a reasonable starting point would be to find a tool that’s simple to use and has plenty of room to accommodate your future needs. Jumping to an expensive commercial solution would be premature at this point, so let’s pay attention to open-source options that are either free or have a freemium model:

Honourable mentions:

• Wrk - the fastest tool, but you gotta script the tests in Lua language (seriously, who’d do that?). Targets UNIX-like systems (available via brew, but no official docker image). Last released in 2018.
• NBomber - A lightweight framework (NuGet package) for writing load tests. It’s the only active project for C#/F# devs. Though, extensive testing might require deep knowledge of the underlining HttpClient and some work around bugs like this.
• Artillery - a single-threaded JavaScript tool that allows YAML tests only. Freemium. Paid versions has AWS support (no Azure or GCP).
• Siege - another Java tool (along with JMeter and Gatling) that’s poorly documented but keeps staying afloat for 7 years.
• Tsung - Powerful and scalable but needs XML-based config. Last released in 2017.

Many tools are off the race now, e.g. Microsoft officially retired their load testing tool in Visual Studio 2019 with its cloud service in Azure DevOps.

Regarding commercial options, I’d mention LoadNinja, Radview WebLOAD and StresStimulus.

2. Running tests in the cloud

Commercial cloud testing solutions may appear tempting at first glance, as they take managing the test infrastructure off your shoulders. It works if you’re OK with their limitations, like k6 offers only 17 geographic locations, Artillery works with AWS only.

JMeter and Gatling have excellent support among third-party commercial load testing services (e.g. RedLine13 and Flood.io). All of them support testing in the cloud and some also run in on-premise infrastructure. It can be a good mixture where you start with a free solution on your own and move to a paid service when your needs grow.

If you haven’t rushed to pay money and got here for not just a one-off test. Make it handy, integrate with the DevOps pipeline.

2.1. General concept

Running performance-oriented tests in the DevOps pipeline is tempting but would be highly unreliable due to underpowered resources. A better option would be

• to run tests on a capable cloud platform to avoid a bottle neck caused by insufficient resources;
• to deploy the test infrastructure closer to the targeting service to avoid latency fluctuations on requests.

It all leads us to a Load Testing Pipeline that conceptually would perform these steps:

1. Create & deploy the test infrastructure on-demand
2. Kick off testing and wait till completion
3. Collect the results and publish a report
4. Remove the test infrastructure on-demand

Regardless of the chosen tool, the test infrastructure would usually consist of

• a Docker container image with the test engine;
• a storage for
  • supplying test scenarios (if you can’t get away with just command line arguments)
    and
  • writing test results.

The actual implementation would depend on the cloud provider and the infrastructure of the targeting service, so it’s hard to give a generic advice.

2.2. Support from cloud providers

The major cloud providers have templates and instructions to run distributed load testing based on JMeter:

• AWS offers a bit overly sophisticated solution that even has a UI.
  If you’re ready to go all in and reliance on a dozen of AWS services is not a concern then it comes in handy.
  See the article and GitHub template/code.
• Azure solution is lean and offers integration with Azure Pipelines where it publishes test results. It’s easy to read and extend.
  See the article and GitHub template/code.

Both above are good to build a custom solution for your needs.

For building a custom testing solution check out my next post ”Load Testing. k6 + TypeScript + Azure DevOps“.

Thoughts? Leave a comment below, on Twitter, LinkedIn or Reddit.