Load Testing Best Practice : Using JMeter assertions

Context:

When Load Testing application, a critical and frequently overlooked task is to validate responses obtained under load.

A Typical issue is to have application return the wrong response to a request but doing so with a 200 response code. In the latter case JMeter (or any other tool) can not guess alone that the response is not the expected one.

It’s up to you to check some content in the response.

How to to that in JMeter:

Let’s see how to do that in JMeter, you do that using Assertions. There are many types of assertions in Apache JMeter, we’ll zoom on the ones that you will use in 99% of the cases.

You will usually add it as a child of the Sampler you want to check:

But be aware that you can add them at a higher level, thanks to JMeter scoping rules, it will then apply to all requests under this scope.

Response Assertion:

First of all, let’s examine Response Assertion which is the most powerful assertion.
It is composed of 4 Zones:

• Apply to : What Sample the Response Assertion applies on. You can apply assertion on :
  • Main Sample and sub-sample : For a web page for example, this means the assertion applies on a request and the downloaded resources (CSS, JS) if you’re using download embedded resource options.
  • Main Sample only : Apply assertion only on the Web Page, not the downloaded resources. This is the default in JMeter.
  • Sub-Samples only : The opposite of previous one
  • JMeter Variable : Apply the assertion on any JMeter variable. Note that in this case, “Response field to test” is ignored. We’ll see this later in examples.
• Response Field to Test: What zone of the Response we apply assertion on. You can apply it to:
  • Text response : It will be the HTML code for example in a web application. This is the default in JMeter.
  • Document : It would be an Excel file downloaded in a Web Application. JMeter will use Apache Tika to convert the binary data to Textual content you can then assert
  • URL Sampled : Apply it on the URL
  • Response Message : Apply it on the status message  for the response code (Page not found for example for a 404)
  • Response Headers : Headers of the response. This is for example useful to check a Cookie
  • Ignore Status : Checking this checkbox tells JMeter to not mark in error a page that returns a response code that it considers in error (Anything different from 2XX, 3XX). This is useful when you know a request should fail in a certain way and you want to check it does
• Pattern Matching Rules : this tell JMeter how it should interpret “Patterns to test”.  There are different options:
  • Contains : This means “Patterns to test” is a Regular Expression that JMeter will use to see if Response field to test contains the pattern
  • Matches : This means “Patterns to test” is a Regular Expression that JMeter will use to see if Response field to test matches the pattern. So the pattern applies on the whole response
  • Equals : This means “Patterns to test” is a text that JMeter will compare with Response field to test. If different assertion will fail.
  • Substring : This means “Patterns to test” is a text that JMeter will search in Response field to test. If not found, assertion will fail. This is the default in JMeter.
• The Not checkbox will apply a Negation on the previous rules, which will become:
  • Not contains
  • Not Matches
  • Not equals
  • Not substring

Let’s see some examples:

• The Text response from Main Sample only contains some text (<title>My Test</title>). The text is compared in a case sensitive way. Note you can have many texts checked.

• The Text response from Main Sample does not contain “ERROR”

• The Text response from Main Sample contains a text that matches regular expression “Hello (.*)” , for example “Hello M. Bond”

• The Headers from Main Sample only  contains a cookie mycookie with value “toto”.

• Check that Main Sample only response code is 403. We want in this case the Sampler to be marked as successful

• Check that variable ulp_lag_time (notice we only put variable name, not ${} around it) provided by Streaming plugin is equal to ulp_buffer_fill variable (works for plugin version before 7.0.0)

Size Assertion:

Another interesting assertion is Size Assertion.

This assertion is useful in many case where Response Assertion cannot be used:

• Response is empty
• Response does not contain a predictable pattern but you have indications on the expected size
• Response is binary (Video, Image)
• You want to compare a Variable to a value

As for Response assertion, we have the same zones:

• Apply to
• Response size field to Test

We have a new zone “Size to assert” which will tell JMeter how to compare the input value with the Variable or the selected zone size.

Let’s see some examples:

• Check that Full response size of Main Sample only is equal (=) to the value of variable sizeInBytes (which would come from a CSV file for example)

• Check Lag Time exposed by UbikLoadPack Streaming plugin as ulp_lag_time is higher than Buffer Fill Time exposed by plugin as ulp_buffer_fill (works for plugin version before 7.0.0)

Note that you can use any variable that you create during your test.

JSR223 Assertion:

Finally, the last type of assertion is more advanced and will be usually used when none of existing assertions can do the Job.

It relies on one of the most interesting features of JMeter, which is its extensibility with any Scripting language that provides a JSR223 bridge.

JMeter since 3.0 has Apache Groovy natively embedded, as it supports “Compilation” feature, it’s a good fit in terms of features and performance.

Example:

What happens here is the following:

• JMeter calls the code passing some Scripting variables:
  • vars that gives us access to Thread Variables (the ones we usually use with ${varName} syntax)
  • props which give us access to Properties (the ones we usually use with ${__P(propName)} syntax)
  • AssertionResult that we will modify to tell JMeter what the assertion result will be
  • log which lets use log some information for debugging purpose
  • some other variables we let you discover in documentation
• In the first line we get the value of variable ulp_play_time (exposed by the UbikLoadPack Streaming plugin, but you can apply this to any variable you create). We convert it to an Integer and compare it to 110000 (110 seconds).
• If comparison is true, we make the assertion fail by setting Failure boolean value and a Failure Message

This example is very simple but it shows you how this assertion works, you can use all the power of Groovy in it

NDLR

As a summary, we’ve seen in this blog is that checking responses in a Load Test is critical for load test validity.  JMeter makes this checking very easy and flexible.

As you have also seen, our Video Streaming Plugin integrates very smoothly with Standard JMeter and lets you create custom assertions on the metrics it exposes.

About the author:

Philippe Mouawad works as an Architect and technical expert for Ubik-Ingenierie where he leads among other things the development of UbikLoadPack a set of Commercial Plugins for Apache JMeter allowing to load test different protocols like HLS, GWT, Flex.
Philippe is a committer on the Apache JMeter project and member of the PMC since October 2011.
Finally he is a co-author of a french book on JMeter called Maîriser JMeter: Du test de charge à Devops.

About us:

• We provide professional services for Load Testing
• We’re strong contributors to JMeter and its eco-system
• Ubik Load Pack is a set of Enterprise Grade plugins for Apache JMeter. Our solutions are used by Big players
• Get a Free trial