JMeter Performance and Tuning Tips

on 22 July 2014 AJAX, JMETER, LOAD_TESTING, MEMORY, PERFORMANCE, PROFILING, TUNING and Tags: , , , , , with 4 comments

Context

You want to make a Load Test that will use hundreds to thousands of Threads and you’ve read on so many blogs that :

  • JMeter consumes a lot of memory…
  • JMeter generates OutOfMemory…
  • JMeter consumes a lot of CPU…
  • JMeter can only cope with 100 threads and not much more…
  • JMeter is fine for playing but once it becomes serious use another tool 🙂
  • …

This has become a kind of “Urban Legend” partly due:

  • to issues that have been fixed for a while now
  • and partly in my opinion to some default configuration parameters that lead to these issues (UPDATE 22th July 2014 : Defaults have been improved in versions 2.10 & 2.11)
  • misuse of elements that are to be used during scripting and never during Load Test

Hopefully, You’re in the right place, Everything you read is WRONG. 🙂

In this short article, we will see how to get the most out of JMeter.

JMeter Performance Basic concepts

Understand important concepts about JMeter:

  • The more it computes to create or process your samples, the less time it spends sampling, so when doing custom coding, ensure it is efficient
  • Listeners receive Sample Results and do some processing with it, this takes resources (memory, CPU) so during Load Testing, a very simple rule is to REMOVE ALL LISTENERS ! , you will ask How can I use my results then ? , the answer is, you will do it after the run. So NEVER EVER have one of these listeners in your Test Plan during a Load Test :
    • View Results in Table => OutOfMemory guarantee in GUI Mode
    • View Results in Tree => OutOfMemory guarantee in GUI Mode
    • Graph Results => Performance issues
    • Assertion Results => OutOfMemory guarantee
    • Comparison Assertion Visualizer
    • Distribution Graph (alpha) => Performance issues
    • Graph Results => Performance issues

JMeter Performance and Tuning Tips

Here are our tips to get the most out of JMeter:

Always use the last version of JMeter

JMeter performances and memory usage have been highly improved in last 3 versions and upcoming 2.9 version will bring new set of improvements regarding memory and CPU:

Use NON-GUI for Load Test

JMeter has 2 modes:

  • GUI mode is for creating the test plan, checking it, debuging it BUT NOT FOR MASSIVE LOAD TEST . AWT Event Thread will disrupt your Load Test.
  • NON-GUI mode is for massive load testing, it is as simple as:
<JMETER_HOME>/bin/jmeter -t <Path to Test Plan> -n -l <path to results>/results.csv

Configure JMeter Java options to meet your requirements

Default JMeter java configuration comes with 512 Mo and very little GC tuning.

First ensure you set -Xmx option value to a reasonable value regarding your test requirements.

Then change MaxNewSize option in jmeter file to respect the original ratio between MaxNewSize and -Xmx.

Finally try tuning GC options only if you master this domain.

Use CSV as output for SaveService

XML is verbose, it takes resources to be written (CPU and memory) and for analysis, CSV is great so forget about XML. Furthermore, for massive load tests there are many result data you don’t need.

So, in user.properties, add:

jmeter.save.saveservice.output_format=csv

jmeter.save.saveservice.data_type=false

jmeter.save.saveservice.label=true

jmeter.save.saveservice.response_code=true

jmeter.save.saveservice.response_data.on_error=false

jmeter.save.saveservice.response_message=false

jmeter.save.saveservice.successful=true

jmeter.save.saveservice.thread_name=true

jmeter.save.saveservice.time=true

jmeter.save.saveservice.subresults=false

jmeter.save.saveservice.assertions=false

jmeter.save.saveservice.latency=true

jmeter.save.saveservice.bytes=true

jmeter.save.saveservice.hostname=true

jmeter.save.saveservice.thread_counts=true

jmeter.save.saveservice.sample_count=true

jmeter.save.saveservice.response_message=false

jmeter.save.saveservice.assertion_results_failure_message=false

jmeter.save.saveservice.timestamp_format=HH:mm:ss

jmeter.save.saveservice.default_delimiter=;

jmeter.save.saveservice.print_field_names=true

Use Post-Processor and Assertion efficiently

Post-Processor and Assertions have a cost.

Ensure you use them when required and use the ones that consume the less memory and CPU:

Use Regular Expression Extractor

Use Regular Expression Extractor for extracting data BUT never ever check Body (unescaped), choose among:

  • Body
  • Headers
  • URL
  • Response Code
  • Response Message

Use efficient Regular expressions and extract as less data as possible

Avoid XPath Extractor when possible

XPath builds a DOM tree so it consumes CPU and memory, prefer Regular Expression extractor or CSS/JQuery Extractor (since JMeter 2.9) for HTML content

Use Response Assertion or Size assertion

These 2 implementations fit 99% of requirements.

Avoid costly ones as:

  • XML Assertion
  • XML Schema Assertion
  • XPath Assertion

Use JSR 223 + Groovy for Scripting

You have a lot of options to do scripting with JMeter:

  • Beanshell
  • BSF and all supported languages Javascript, Scala , Groovy, Java …
  • JSR223 and all supported languages Javascript, Scala , Groovy, Java …

Although you can be lazy and choose the language you know, FORGET ABOUT IT.

Use the most efficient option, which is JSR223 + Groovy + Caching (supported since JMeter 2.8 in external script and since JMeter 2.9 also supported with embedded scripts).

See:

Using Groovy is as simple as adding groovy-VERSION-all.jar in <JMETER_HOME>/lib folder.

But of course ensure your script is necessary and efficiently written, DON’T OVERSCRIPT

Generate Reports AFTER the Run

Always generate Graphs and Reports AFTER the Load Test and NEVER EVER DURING IT

You can use JMeter reports:

  • Aggregate Graph
  • Response Time Graph
  • JMeter Plugins graphs

Update 22th July 2014, we contributed this GraphsGeneratorListener, read our blog:

You can alternatively choose a SAAS solution to generate your report, see:

Distributed (Remote) Testing

Once you reach the limits of one machine, you can switch to distributed or remote testing.

But JMeter defaults before 2.9 were not fine for efficient remote testing, so in user.properties, add:

mode=StrippedBatch

Update 22th July 2014, this mode is now the default since JMeter 2.9.

This will:

  • remove some data from the SampleResults as the response body, but do you need response body during a High Load Test, NO, DEFINITELY NO !
  • will send Sample Results as Batches and not for every sample reducing CPU, IO and network roundtrips

NDLR

Following those recommendations you can reach easily up to 5000 threads on a standard basic machine, 10000 threads on a m3.xlarge …

When it’s not enough

  • Use one of the Cloud Solutions that support Apache JMeter (4 at least now)
  • Deploy it on the cloud yourself:

See also

About author:

Philippe Mouawad is a developer, committer and member of the JMeter Project Management Committee at Apache.

He is also the co-author of the book Master JMeter : from load testing to DevOps

He currently works as an Architect and technical expert for Ubik-Ingenierie where he leads among other things the development of UbikLoadPack a set of Commercial Plugins for Apache JMeter allowing to load test different protocols like HLS, GWT, Flex.

About us:

< Newer Post
Older Post >

4 Comments

Comments are closed.