UbikLoadPack : Advised tuning for injector when load testing Video Streaming servers

UbikLoadPack : Advised tuning for injector when load testing Video Streaming servers

Recently one of our customer asked the following question on Stackoverflow:

We thought that we should be answering this question with as much details as possible.

So here is the answer.

Always Calibrate

It is hard to answer such generic question as behaviour depends on at least following factors:

•  The type of Stream : HLS will perform better than MPEG-DASH and HSS, the 2 latest being more CPU consuming due to XML format.
• The duration of the Manifest: This is the total duration it holds, the biggest it is the more memory and parsing it will require
• The refresh period : This is the period at which a Live Stream manifest will be reloaded by player, the shortest it is, the more it will consume memory and CPU
• Whether stream is Live or VOD, Live will consume more CPU, VOD might consume more memory

So the best advice is to calibrate based on your Manifests, meaning to run this for 50 to 100 threads, see how much CPU and Heap it consumes and then extrapolate.

Physical Machine or Virtual Machine:

• Ensure you have enough bandwidth in your data-center between injector and target servers
• You should usually load test as close as possible to target to avoid network parasitizing your results. And only when you’re sure your server are correctly handling the load load test from Cloud servers. But you can do the latter if you can’t do the first step.
• Ensure your network card is correctly configured
• With “Average Manifest” , you should have no issue load testing 700 to 1500 Threads (Virtual Users) with machine of type m5.2xlarge (8 vCPU / 16 to 32 Go)

Operating System:

• We advise you to use Linux/MacOS instead of Windows OS as you will get more Threads on same virtual/physical configuration.
• But all OSes work since Java is used

For Linux apply following tuning:

 net.ipv4.tcp_syncookies=1
 net.ipv4.tcp_no_metrics_save=1
 net.ipv4.tcp_keepalive_probes=2
 net.ipv4.tcp_keepalive_intvl=20
 net.ipv4.tcp_fin_timeout=15
 net.ipv4.tcp_max_syn_backlog=5000
 net.ipv4.tcp_keepalive_time=300
 net.ipv4.tcp_tw_recycle=0
 net.ipv4.tcp_tw_reuse=1
 net.ipv4.tcp_sack=1
 net.ipv4.tcp_dsack=0
 net.ipv4.ip_local_port_range= 1025 65000
 vm.swappiness = 5
 net.ipv4.tcp_slow_start_after_idle = 0
 
 #Adjust those ones based on your machine specification:
 net.core.netdev_max_backlog=5000 
 net.ipv4.tcp_max_syn_backlog=10240 
 net.core.somaxconn=5000 
 net.core.rmem_max=25165824 
 net.core.wmem_max=25165824

Ensure you tune correctly the user that runs JMeter limits:

run this to check user limits:

• ulimit -a

Ensure you increase :

• open files (-n)
• max user processes (-u)

Edit /etc/security/limits.conf and add the lines:

*               soft    nofile          65535
*               hard    nofile          65535
*               soft    nproc          88000
*               hard    nproc          88000

Edit /etc/pam.d/login, adding the line:

• session required /lib/security/pam_limits.so

The system file descriptor limit is set in /proc/sys/fs/file-max. The following command will increase the limit to 65535:

• echo 65535 > /proc/sys/fs/file-max

You should then be able to increase the file descriptor limits using: ulimit -n unlimited  The above command will set the limits to the hard limit specified in /etc/security/limits.conf.

Java

• Ensure you use last minor version of Java 8 or 11. This is for reliability, security and performance
• Ensure you use Java 64 bits to be able to allocate more than 1.5/ 2GB memory. Check this by running

  • java -version

• Allocate enough heap based on your calibration, for example to allocate 6g Heap:

  •  -Xmx6g -Xms6g

• Note you should reserve around 400 Mb for your system and not allocate the full available memory
• If your calibration shows that JVM does not require the whole memory, then don’t allocate too big heaps
• Check in jmeter.log that the memory is really what you allocated, something like this, for 6g you would get:

  • 2018-03-14 21:55:09,662 INFO o.a.j.JMeter: Max memory =6442450044

• Some advised GC tuning:

  • -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -XX:G1ReservePercent=30 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -XX:MaxMetaspaceSize=256m -XX:G1HeapRegionSize=32m -Xss256k

• Run headless by adding this to JVM options:

  • -Djava.awt.headless=true

• To set those options on Linux, using JMeter 5.3:
  • create a file called setenv.sh and set:
    • GC_ALGO for GC tuning
    • HEAP for Memory tuning and other tunings

• JMeter

  As always use latest version of Apache JMeter as you’ll get:

  • Best performances
  • Best reliability
  • More features
  • More security

  Add to user.properties:

• httpclient4.time_to_live=300000
• httpclient4.validate_after_inactivity=10000
• ulp_streaming.sampler.player.thread_pool_size=<Max Number of Threads * 2>
• ulp_streaming.sampler.player.thread_pool_queue_size=<Max Number of Threads * 2>

Note you can pass the 2 latest on command line:

-Julp_streaming.sampler.player.thread_pool_size=<Max Number of Threads * 2> -Julp_streaming.sampler.player.thread_pool_queue_size=<Max Number of Threads * 2>

• Scripting

  Our solution does not require any scripting except if you want to export some variables, make some custom scripting or customize number of Thread for rampup.

  In this case, for your scripts :

  • ALWAYS USE GROOVY either with:
    • JSR223 Test Elements like JSR223 Assertion
    • __groovy function

  NEVER EVER USE the below languages as they will limit JMeter scalability in a huge way:

  • Beanshell
  • Javascript (functions or code in JSR223 TestElement)

  What is UbikLoadPack

  If you’ve never heard about our solution and are coming here for the first time for another reason, then you should know that UbikLoadPack is a commercial set of JMeter plugins that allow load testing complex protocols or technologies realistically and easily.

  Our solution is used by very big names in all fields and has been around since 2013.

  We offer solutions for load testing:

  • Video Streaming servers delivering HLS, MPEG-DASH, MS Smooth. Our solution test realistically simulating real players.
  • Complex enterprise solution like Oracle JD Edwards, Oracle PeopleSoft, Oracle Siebel, Vaadin…
  • GWT RPC
  • Java Serialization based applications
  • FLEX/AMF application

  Try it:

  So come and get a free trial version HERE.
  Trial is free for 10 days.

  Cloud Ready

  Our plugin can be used on Cloud providers like:

  • Redline13
  • Blazemeter
  • Floood.io

  About us:

  • Ubik Load Pack is a JMeter based solution for complex protocols.
  • Follow us on Twitter and StackOverflow.
  • Follow Apache JMeter on Twitter