Easy and realistic Load Testing of HTTP Live Streaming (HLS) with Apache JMeter

Updated on the 7th of may 2020

Easy and realistic Load Testing of HTTP Live Streaming (HLS) with Apache JMeter

Videos today are ubiquitous. Nearly every website shows videos, particularly those built for Mobile devices.

The most efficient way to serve video from website and even more for Mobile devices is to stream it as users don’t need to download the full video to start playing it.

Another motivation for streaming is performance and network bandwidth use. As a phone user for example, you don’t want to have to restart your full download because it was interrupted by a phone call or any other simultaneous activity. On the server side, having your users download “progressively” is a big performance benefit.

One of the most popular format for streaming was RTMP (by Adobe), but for iOS this format is not supported. Apple created HLS (HTTP Live Stream) which is becoming increasingly popular even outside of Apple world thanks to new services based on HTML5 Video.

Ensuring you serve videos correctly is a critical matter to ensure infrastructure performance and stability.

HLS (HTTP Live Stream):

HLS has become a standard nowadays for Apple devices, IOS devices but also Desktop. Solutions for playback of HLS on other platforms mostly rely on third-party plug-ins such as Flash or QuickTime. However, the recent development of THEOplayer by OpenTelly now also allows playback of HLS directly on HTML5 platforms without the need of any third-party plug-ins.

Apple initially created HTTP Live Streaming (HLS) to easily deliver audio and video content without server software. As an HTTP-based media streaming protocol, HLS uses regular web servers to send both live and pre-recorded content through a variety of streams at different bit rates, adapting to different network bandwidths. To do so, it breaks the media stream up into shorter sequential downloadable file components of their overall streams.

Being HTTP based, HLS can bypass firewalls or proxy servers and can easily be hosted by Content Delivery Networks.

You can read this overview for more in-depth understanding of protocol.

Load Testing of HTTP Live Streaming (HLS) with Apache JMeter:

Load testing HLS realistically is a challenge whatever the tool you use.

For Apache JMeter, whenever you want to load test a video infrastructure serving videos, you can find  blogs showing how to combine Regexp Extractor and HTTP Request to simulate load on servers:

• Load testing HLS with Ruby JMeter
• How to Load Test HTTP Live Media Streaming (HLS) with JMeter

Unfortunately, these methods have drawbacks or limits.

After the description of these limits, I will present an alternative method using Ubik Load Pack streaming Plugin for load testing HLS.

The challenges of an easy and realistic streaming load test :

Unrealistic simulation

Players have a complex behavior in order to reduce pause times (Lag) during the play.

Few examples:

• They select the playlist depending on the bandwidth
• They initially buffer few seconds before starting to play
• They download data while video is being played

Another problem is that your test won’t be able to limit the bandwidth for some of your users based on their network type.

None of the blogs you find tackle these challenges, handling those in JMeter is impossible unless you spend a lot of time on custom code.

Time consuming to create and maintain

Creating an “imperfect simulation” will take you some time and skill as you need to:

• Master Regular Expression Extractor
• Ensure you handle correctly pauses
• Select correctly the playlist depending on bandwidth

And of course you will need to customize it whenever playlist changes.

Finally note that these blogs don’t show how to Load Test playlists that use EXT-X-BYTERANGE to specify byte range media segments, while this tag is now part of the new style playlists and recommended by Apple as it makes it much easier to create playlists and drastically reduces the number of chunks and files to manage on server side.

Metrics and analysis

Once you have created and run your script you have regular Apache JMeter metrics like response time, latency, …

Unfortunately for videos, these metrics are too limited to ensure User experience is OK. Indeed it is very hard to have the following critical informations :

1. After how many seconds did player start to play video ? Do you want your customers to fly away because your video take too much time to load ?
2. Did users face pauses during the play ?
3. What format was chosen by your client ? HD, Medium size or only audio ?

Limited scalability

The described approaches usually use regular JMeter HTTP Request. This sampler is not made for video but for text responses, as a drawback it will store in memory each video chunk (which size can be quite big)limiting the scalability of your script and reducing drastically the number of Virtual Users you can simulate by each JVM.

Limited scope

URLs for HLS Streaming can be much harder to analyze than the illustrated examples, so you’ll spend a lot of time finding your regular expressions and building ones that covers all possibilities

UBIK LOAD PACK Video Streaming Plugin

The good news is that there is already a solution for this, and it’s called UbikLoadPack Video Streaming plugin.

Video Streaming plugin is a commercial plugin from Ubik Load Pack solution.

Current version used in this blog is version 7.0.0.

Easy installation:

1) Unzip bundle

To install plugin just unzip ulp_video-streaming-plugin-XXX.zip.

• Copy lib/ext/ulp_video-streaming-plugin-X.X.X.jar to jmeter/lib/ext folder

2) Configure License

When launching Apache JMeter for the first time after the installation of the plugin, a pop up will ask for the path of the license file. Navigate to the location of your license and click on the button “Open”.

ULP.videostreaming.licensepath=<LICENSE_FOLDER>/ubik-streaming-plugin.license

You’re ready to play !

Easy configuration and use:

To simulate video download you need to find the URL of the video and use ULP Video StreamingSampler:

In this example :

• video is https://devstreaming-cdn.apple.com/videos/streaming/examples/img_bipbop_adv_example_ts/master.m3u8
• Stream type is set to auto as the plugin will recognize the manifest as a HLS manifest
• In the “Play options” panel:
  • You can choose to play the video completely or to play a part of the video by setting the desired duration in seconds
  • You can select the language of the audio and subtitles tracks. If the Language is not set or if it is not in the manifest, the plugin will play the track labeled as default in the manifest.
  • Subtitles are optionals. You need to activate them in order to test subtitles track.
  • For HLS streams it is possible to play an alternate video track if the stream proposes different camera angles for example.
• In the “Network options” panel:
  • You can keep the “Auto” options and the plugin will play the track with the highest available bandwidth
  • Or you can select CUSTOM and enter a value in “Bandwidth”. Bandwidth can also be a variable (extracted from CSV for example ${bandwidth})
  • You can also select a network type among many presets.
• In the “Reporting options” panel:
  • Choose the desired freqency at which SampleResults with custom metrics will be generated.

Plugin will do all the complex stuff we described before for you 🙂

You’re nearly done

Rich metrics for User experience analysis:

The response time of the VideoStreamingSampleResult is the sum of the response time of each chunks and manifests downloaded during the period set in the VideoStreamingSampleResult frequency field.

Plugin computes and exposes 5 additional metrics.

These metrics are made available as JMeter Variables using syntax ${variable}:

These metrics are output in the 2 formats provided by JMeter:

• CSV
• XML

Validate your sampler

Plugin being fully integrated with Apache JMeter 2.12, you can use standard JMeter assertions to make checks on responses.

Size Assertion to check Size of downloaded data (Before version 7.0.0)

With this kind of assertion we can make checks on downloaded bytes.

In the below example, we check for example that size is equal to a variable ${sizeInBytes} which in this case is extracted from Csv DataSet:

Note of course you can check different “Type of Comparison”

Response Assertion to check one of the metrics exposed (Before 7.0.0)

In this below screen we check that ${ulp_lag_time} is equal to ${ulp_buffer_fill} which means we didn’t get any pause during play of video.

JSR223 Assertion using Groovy

Du to the nature of the new reporting method  introduced after version 7.0.0, the best way to use Assertion with the plugin, is to use JSR223 Assertions. In this below screenshot you can see a script that checks if the response time for manifest is below 500ms, if the response time for chunks is below 5000s and if that “ulp_lag_ratio_wo_bf” is > 0,5% and if not makes the sampler fail:

String sampleLabel = prev.getSampleLabel();
def failureMessage =""

if (sampleLabel.contains("Manifest")){
  if (prev.getTime() > 500) {
  failureMessage = "It took more than 0.5 seconds to download the chunk";
  }
} else if (sampleLabel.contains("Chunk")) {
  if (prev.getTime() > 5000) {
  failureMessage="It took more than 5 seconds to download the chunk";
  }
} else if (Float.valueOf(vars.get("ulp_lag_ratio_wo_bf")) > 0.5){
  failureMessage = "Lag ratio without buffer fill time was superior to 0.5%";
}

if (failureMessage?.trim()) {
  AssertionResult.setFailureMessage(failureMessage);
  AssertionResult.setFailure(true); 
}

Other features:

Realistic use of network

As you can see in screenshot, plugin let’s you input a bandwidth (bits/second) in HLS Sampler.

With this feature plugin ensures that a Virtual User with some kind of network (3G or EDGE) will not download at a speed of a User having a 4G Network making test realistic.

Using this feature and JMeter Throughput Controller,(in this below screenshot controller is configured to sample it’s children 30% of the time)

you can easily create a test where you split your users into different network populations for example:

• 50% using 4G netwok
• 30% using 3G network
• 20% using Edge network

CDN Compatibility

Thanks to JMeter 2.12 new DNS Cache Manager feature, plugin is able to behave realistically in a CDN based infrastructure.

You only need to add DNS Cache Manager to your script.

Clear results

Plugin will create sample results as the manifests and chunks are being downloaded. These sample results are name according to their type (Manifest/Chunk) and their content (Video/Audio/Subtitles)

The plugin also generate a SampleResult with the custom metrics at the frequency set in the “Reporting options” panel

You can see in below screenshot how plugin proceeded for downloading the video.

ND;LR:

As you can see in the previous examples, using Ubik Load Pack plugin for HLS lets you Load Test video infrastructure in few minutes:

• Comprehensive metrics for user experience analysis
• Realistic simulation (Player and network)
• No coding skills needed
• Performance and scalability
• “Apache JMeter” like philosophy which makes plugin very easy to use, no specific training needed
• Full compatibility with Core JMeter
• CDN compatibility thanks to JMeter 2.12 DNS Cache Manager
• Our plugin runs on any cloud provider(Amazon WS, CloudWatt, Google Cloud, Microsoft Azure). This can be useful when you need to make a massive load test that consumes important network bandwidth.
• Our plugin can also run on PAAS by our friends from Flood.IO, this can be useful when you need to make a massive load test that consumes important network bandwidth and don’t want to manage your own cloud servers.
• Very competitive pricing as you only pay the plugin, not the free core based on Apache JMeter

So try it ASAP here. Trial is free and lasts 10 days.

About the author:

Philippe Mouawad works as an Architect and technical expert for Ubik-Ingenierie where he leads among other things the development of UbikLoadPack a set of Commercial Plugins for Apache JMeter allowing to load test different protocols like HLS, MPEG-DASH, GWT, Java Serialization, Flex.
Philippe is a committer on the Apache JMeter project and member of the PMC since October 2011.
Finally he is a co-author of a french book on JMeter called Maîriser JMeter: Du test de charge à Devops.

About UbikLoadPack :

• Ubik Load Pack  is used by Big players in the Video streaming field
• We provide professional services for Load Testing
• Learn more about our streaming plugin
• Detailed features of UbikLoadPack Streaming Solution
• Get a Free trial

You’ll probably also like:

• Performance Testing Low Latency Dash servers with UbikLoadPack
• Performance Testing Low Latency HLS servers
• Load testing MPEG-DASH Video Streaming with Apache JMeter and UbikLoadPack
• Video Streaming: CMAF and Low-Latency
• Cases studies

3 Comments

Comments are closed.