Apache JMeter vs Neoload vs HP Load Runner

on 10 January 2019 CLOUD, CONTINUOUS INTEGRATION, DEVOPS, HP LoadRunner, JMETER, LOAD_TESTING, LOW COST, MicroFocus Load Runner, Neoload, PERFORMANCE, PRODUCTIVITY, TUNING and Tags: , , , , , , , , with 0 comments

Apache JMeter vs Neoload vs HP Load Runner

We see a lot of unfair or very old comparison between Apache JMeter and commercial alternatives.

But times are changing, in december 2018, Microsoft recommended to use JMeter as Load testing tool in release notes of Visual Studio 2019 while announcing discontinuation of its tool:

We felt it was time to write a fair comparison with links to features.

We’ll be happy to take into account any remark regarding the other tools if you contact us by email.

Comparison

To convince you it is worth considering switching your load testing to Apache JMeter and UbikLoadPack (for more protocols support and new features sponsoring) let’s compare it with some Major actors.

Cost :

Criteria
LoadRunner (HPE)
JMeter + UbikLoadPack (Ubik-Ingenierie)
NeoLoad (Tricentis)
Commercial License
Yes
No (Free and Commercial Compatible License (Apache 2.0))
Yes
Yes
Core Price
Free up to 50 VUs then License is per number of Users.
Free whatever the number of users
Free whatever the number of users
Free up to 50 VUs.
License  depends then on number of Users, site deployment and other non public informations.
Additional Protocol Price
You pay per additional protocol + Core price
Not applicable
You pay ONLY for additional protocol but core (JMeter) is Free. Total Cost Of Ownership ends up being much lower than the competitors. You can either Rent (Daily, monthly or annual basis) or Purchase.
You pay for the Core  + per additional protocol + monitoring and additional feature
Cloud Load Testing Price
High
Low
Low
Medium

Features :

Criteria
HP Loadrunner
Apache JMeter
NeoLoad
Protocol Coverage
It has the widest protocols support (enterprise
software SAP, Oracle ERP…) but no support for Video Streaming testing
HTTP(S), Webservices (Soap, Rest), FTP, LDAP, and Java-related protocols (JMS,
JDBC, Neo4j).
NeoLoad has good support of many Protocols. It
supports enterprise software SAP (fully since version 6.6), Oracle Siebel etc.
But it has no support for Video Streaming testing
Record and Playback capability
Yes
Yes
Support / Training
Yes
 JMeter being an OSS provided by Apache there is no Apache support. But we (UbikLoadPack) provide support and training on the product. We are major contributor to Apache JMeter through features like JSON Extractor, the Web Report, lot of bug-fixes and enhancements and our Team leader being an Apache Committer and Project Member Committer of the project
Yes
Cloud Based Solution
Yes (Amazon , Azure, Google for the Controller)
Yes (Amazon WS, Google Cloud, Azure, IBM BlueMix, OVH with self install)
Yes (Custom)
APMs integration
Yes with APM Plugin from UbikLoadPack Monitoring plugin (Dynatrace, AppDynamics, NudgeAPM)
Yes (Dynatrace, AppDynamics, NewRelic, NudgeAPM)
Continuous Integration (JENKINS, Bamboo…)
Yes
Yes
Mobile Load Testing
Yes
Yes (Native + SAAS providers for more features)
Yes
Real Time Test Monitoring
Yes
Yes (With BackendListener based backend for InfluxDB, ElasticSearch, Datadog and others)
Yes
Script Customization
Hard (C language for all, Javascript added for Web in 12.5 version) and limited
Easy as JMeter is built on a Plugin architecture and made to be customized (JSR223 Elements allow you to use a lot of scripting languages (Groovy, Javascript, Java…)). This makes it the most customizable tool compared to others.
Medium to Hard (Javascript for extremely complex scenarios as per documentation quote
Result Reporting
Yes
Yes since JMeter 3.0 (APDEX, Summary and Error Tables, TOP5 Errors by Sampler, 15 graphs).
Results can also be sent to an Real Time Series thanks to this plugin or using third party tools like Influx DB, Datadog see real-time results.
Yes
Load Generation
Depends on Type of License
Unlimited Load Generation.
For UbikLoadPack, it depends on Type of License
Depending on Type of License
IPSpoofing
Yes
Yes search Spoofing in this link
Yes
Support of authentication schemes and client certificates
Yes
Yes (Basic, Digest, Bearer, Kerberos, …)
Yes
Flexible error handling and adjustable pass/fail criteria
Yes
Yes, see this and this
Yes
Browser Support
Yes
Yes through PAAS solutions  like Redline13, Blazemeter, Flood IO, …
Yes
Device Emulation
Yes
Yes by modifying User-Agent header in Header Manager
Yes
Network Emulation
Yes
Yes by modifying httpclient.socket.http.cps / httpclient.socket.https.cps, see this
Yes
Windows 7 and above
Yes
Yes as it’s based on Java
Yes
Mac OSX
No
Yes as it’s based on Java
Only for Scripting
Linux
Yes(Not all Load Generators)
Yes as it’s based on Java
Yes
Cluster (Scaling)
Yes
Yes (Horizontal through client/server mode or PAAS)
Yes
Number of Virtual Users per 8vCPU Machine/4Go RAM
1000
1000 to 1500
1000 to 1500
Load test as code
Yes for script code
Test Configuration as code through Taurus tool
Test Configuration as code since 6.7

NDLR

As you can see, Apache JMeter is no more a tool to “play” or to use only for limited and simple load testing.
It has grown up into a mature, professional tool, used by major organizations for complex load testing use cases.
Furthermore it is available on the Cloud at ultra competitive pricing.
On the protocols side, UbikLoadPack provides important additions to it and this will keep growing.

Besides being provided by Apache, you can find support for it in the market in the following fields :

  • Training
  • Consulting
  • SAAS and Cloud solutions

As a major contributor of JMeter, we also provide training and consulting around Load Testing and if you want to sponsor features you want to add to it, or have some custom protocols you can contact us.

Cloud Ready

Thanks to our partnership with Redline13 , our plugins are available from AWS cloud and ready to use in few clicks, read more :

We are also available on other SAAS Cloud providers:

About us :

< Newer Post
Older Post >