Case Study from OTT sector: Load Testing video streaming services with UbikLoadPack, Redline13 and AWS Cloud

How to ensure the best quality of experience during the re transmission of a major sport event around the world ?

By Load Testing video streaming services with UbikLoadPack, Redline13 and AWS Cloud

Recently one of our customer had to ensure his OTT and CDN services would handle correctly the load incurred by re transmission of a major sport event.

He wanted to check multiple things:

1. Customers user experience (no lags, accuracy of stream, no delay)
2. CDN caching policy (aggressive caching of parent manifest, low caching of child manifest)
3. Origin performance delivering child manifests and chunk on time
4. Load Test his OTT video streaming servers with up to 100 000 Players

His event was a worldwide one where customers would be watching it from:

• USA
• Latin America
• Brasil
• Europe (Germany, France, Belgium, Austria …)

So he contacted our professional services to help him ensure he could have this test done in 5 days.

Which tools to use and why ?

His OTT service was composed of a CDN fronting an origin delivering Live HLS and MPEG-DASH manifests, and his CDN was spread accross the world.

Since we needed to run a massive scale Load Test,  we used:

1. UbikLoadPack Video Streaming plugin + Apache JMeter  as the tool to simulate users watching Live streams delivered in HLS and MPEG-DASH
2. Redline13 as the service to run at high scale and distributed accross the world the load test. As a bonus, JMeter and UbikLoadPack Video Streaming plugin are available Right Out Of The Box on their service which would ease the piece of work that was constrained by the very short delays

UbikLoadPack Video Streaming plugin fits customer requirements in the following fields:

• Reproduce realistically and very easily a player reading Live stream delivered in HLS,  MPEG-DASH, MS Smooth and HDS. There is no need to write complex and unmaintainable JMeter test plan to reproduce the player behaviour, knowing that the multitude of formats and possibilities of Live Streaming makes this very complex and time consuming.
• Provide custom metrics to analyze User watching video experience
• Ability to add additional metrics thanks to some groovy scripting to analyze CDN caching and accuracy in delivering the child manifests
• Ability to graph those metrics
• Availability both on Premise and from the Cloud. For the short term, customer wanted to run the test from the Cloud but in the longer term, he also wanted to run those tests as close as possible to his streaming servers in his own Data Center

Redline13 fits customer requirements in the following fields:

• Ability to easily deploy JMeter + UbikLoadPack with a single click from a web page on multiple servers in the AWS Cloud
• Ability to start those servers from all over the world using AWS datacenters from a simple Web Page
• Ability to get live feedback on load test
• Ability to get the merged results and metrics in one CSV accross all nodes
• Ability to get the JMeter report auto generated at end of test

Let’s see how simple it is to setup a test on Redline13 using UbikLoadPack.

The results:

Thanks to this load testing campaign , our customer was able to:

• Run a load test with 100 000 Players from 10 AWS locations accross the world
• Diagnose an issue in the caching policy that triggered lag in the playing. This was correlated thanks to the graphing of Ratio and the addition of Caching metrics that our professional services provided based on CDN headers
• Diagnose performance issues affecting customers in the USA and South America
• Diagnose 404 issues in the streams from certain locations
• Provide his end-user customer metrics showing that their services was delivering streams as per requested SLR

About UbikLoadPack :

• Ubik Load Pack  is used by Big players in the Video streaming field
• We provide professional services for Load Testing
• Learn more about our streaming plugin
• Detailed features of UbikLoadPack Streaming Solution
• Get a Free trial

You’ll probably also like:

• Performance Testing Low Latency Dash servers with UbikLoadPack
• Performance Testing Low Latency HLS servers
• Load testing MPEG-DASH Video Streaming with Apache JMeter and UbikLoadPack
• Video Streaming: CMAF and Low-Latency
• Cases studies