Performance Testing Basics – What is Throughput?

by Joe Colantonio on July 5, 2011

Post image for Performance Testing Basics – What is Throughput?

What is Throughput:

Basically, “Throughput” is the amount of transactions produced over time during a test. It’s also expressed as the amount of capacity that a website or application can handle. Also before starting a performance test it is common to have a throughput goal that the application needs to be able to handle a specific number of request per hr.

For example:

Let's Imagine

Let’s imagine that a gas station attendant fills up a car's gas tank using a gas pump. Let’s also say that it always takes the gas attendant just one minute to fill up any car, no matter how big it is or how low the car's gas tank is.

Let’s call this gas station “Joe’s Gas,” and envision that it only has three gas pumps. Naturally, if we have three gas pumps and three cars, it follows that Joe's attendants can only fill up three cars per minute. So, if we were to fill out a performance report for Joe's gas station, it would show that Joe’s throughput is three cars per minute.

Throughput no wait

This is Joe’s dilemma: no matter how many cars need gas, the maximum number that can be handled during a specific time frame will always be the same –three. This is our maximum throughput; it is a fixed upper bound constraint.

throughput

As more vehicles enter the gas pump line they are required to wait, thus creating a queue.

It is the same concept that applies if we are testing a web application. If a web app receives 50 requests per second, but can only handle 30 transactions per second, the other 20 requests end up waiting in a queue. When presenting performance test results, throughput performance is often expressed as transactions per second, or TPS.

Real performance test throughput results:

I use HP's LoadRunner (which comes with a throughput monitor) for performance testing. In a typical test scenario, as users begin ramping up and making requests, the throughput created will increase as well.
Once all users are logged in and processing in a steady state, the throughput will even out since the load each user makes stays relatively constant. If we wanted to find an environment’s throughput upper bound we would continue increasing the number of users. Eventually, after a certain amount of users are added, the throughput will start to even out, and may even drop. If the throughput enters this state, however, it is usually due to some kind of bottleneck in the application.

A look at a typical throughput results

Below are the LoadRunner throughput chart results for a 25-user test that I recently ran. Notice that once all 25 users are logged in and doing work, the throughput stays fairly consistent. This is expected.

Good Throughput Chart

Now notice what throughput looks like on a test that did not perform as well as the last example. All of the users log in and start working; once all users are logged in making request you would expect the throughput to flatline. But in fact, we see it plummet. This is not good.

Bad Throughput Chart

As I mentioned earlier, throughput behavior like the example above usually has to do with a bottleneck. By overlaying the throughput chart with a HP Diagnostics ‘J2EE – Transaction Time Spent in Element’ chart, we can see that bottleneck appears to be in the database layer:

Bad Throughput Chart with HP Diagnostics

In this particular test, requests were being processed by the web server, but in the back end work was being queued up due to a database issue. As additional requests were being sent, the back end queue kept growing, and users’ response times increased.

Recap

To recap: Throughput is a key concept for good performance testers to understand, and is one of the top metrics used to measure how well an application is performing.

Extra
For more detail info on performance testing make sure to grab a copy of Performance Analysis for Java(TM) Websites.

{ 43 comments… read them below or add one }

TM August 15, 2011 at 6:07 pm

Excellent. It helped me to understand with the simple examples and the visual graphs.

Reply

Joe Colantonio August 15, 2011 at 6:15 pm

TM » Thanks! I really appreciate your feedback. It helps me in planning what types of future post I should focus on. Cheers~Joe

Reply

Haris February 2, 2013 at 12:41 am

Hi Joe, thank you for the explanation on throughput. I really like the gas station analogy. I have another basics question though. How would you define SLAs when developing a performance test plan? How much throughput is good, what is acceptable CPU on servers etc. I know there is no one size fits all, but if you can give some examples similar to the gas station analogy, would appreciate it. Thanks.

Reply

Joe Colantonio February 3, 2013 at 1:06 pm

Haris » Thanks Haris! I’ll try to see what I can put together for an SLA post. Thanks for the idea ~Cheers Joe

Reply

shailesh December 19, 2011 at 6:06 am

Hi,
i would like to know…What is pacing?why pacing should be given between iteration or what if pacing is not given…

Thanks in advance.

Reply

Joe Colantonio December 22, 2011 at 3:08 pm

shailesh » Hi Shailesh – Good question – I think this quy can explain it better than me:. Hope it helps. Cheers~Joe

Reply

earnest January 18, 2012 at 12:13 pm

Shailesh,
Pacing is the time you wait between iterations.

BTW this is a great article JOE.

Reply

Joe Colantonio January 18, 2012 at 12:47 pm

Thanks Shailesh! I like your pithy definition for pacing :) Cheers~Joe

Reply

Paul January 11, 2012 at 7:48 pm

Thanks for the article. It’s really helpful !!

Reply

Joe Colantonio January 11, 2012 at 9:43 pm

Paul » Thanks Paul – I’m glad it helped you! Cheers~Joe

Reply

kumar February 22, 2012 at 2:09 am

Nice article. I do have one question.i was going through the Loadrunner training manual and in that it mentions “If throughput has flattened out while the number of Vusers increase, there is a likely a bandwidth issue.” But i feel your explanation is more correct. can you please shed some views on this?

Reply

Mathews April 4, 2012 at 9:24 am

Hi Joe, I woulld like to know that, what are techniques or methods that can be used to evaluate throughput in terms of computing ? [I am a Computer Science Honours student (South Africa), and currently doing something on throughput evaluation ]

Reply

Joe Colantonio April 9, 2012 at 8:34 pm

Mathews » Hi – First I would use a performance test tool like LoadRunner,SoaSta, Grinder or JMeter to place a load against the test system. I would start with one user and increase the user load by small increments until I found the throughput plateau. I consider a plateau to be the point where either the throughput really falls off or the response time is well over what is acceptable for my requirements.That’s it in a nutshell – this may be to basic for your needs.

Reply

fazal April 12, 2012 at 9:36 am

i want to know that what is throughput out and throughput in.
one think i must tell you, your way of explaining is toooooo good.

Reply

Joe Colantonio April 12, 2012 at 1:45 pm

fazal » Thanks Fazal.

Reply

Abhinav May 15, 2012 at 12:29 pm

Thanks Joe for this wonderful post, it helped me a lot.
However it is clear to me but I just wanted to confirm that throughput(It is also a term in Argeegate Report(Listener) of Apache Jmeter) is same as processing rate of requests, a high throughput means high performance, Right?

Reply

Joe Colantonio May 23, 2012 at 10:26 pm

Abhinav » Hi Abhinav yes the throughput on the JMeter Aggregate_Report is the same. Not sure I would go as far as to say that high thoughput = high performance. Its one factor in many that needs to be looked at when performance testing an application. Cheers~Joe

Reply

nikhil July 2, 2012 at 12:17 pm

Hi everyone…

What is ADHOC testing and when & why this testing is done?

Thanks,
Nikhil

Reply

Joe Colantonio July 7, 2012 at 11:22 pm

nikhil » Hi Nikhil – Ad hoc testing is a commonly used term for software testing performed without planning and documentation. Ad hoc testing is an informal test and usually is a test that is only run once

Reply

Abhishek August 21, 2012 at 5:05 am

Excellent Stuff. Finally I am able to understand throughput now.

Reply

Joe Colantonio August 22, 2012 at 10:53 am

Abhishek » Thanks Abhishek! Let me know if there are other performance concepts you want me to blog about. Cheers~Joe

Reply

Murali September 20, 2012 at 9:35 pm

Excellent article Joe it made things easy for me
how many users should i use if my Target is 0.7 TPS

and is it different if i call a webservice via soapUI

Reply

Joe Colantonio September 25, 2012 at 8:48 pm

Murali » Thanks Murali – its hard to tell without knowing how much wait time you have between transaction and the amount of time your response is for 1 user. You should try to get a handle on how many users the business expects and then try to create a realistic scenario emulating that to check your TPS. Good Luck~Joe

Reply

Dzmitry Kashlach September 27, 2012 at 8:49 am

Thank you, Joe! I like comparison with gas station)))).
BTW, what do you advise to start from when analyzing results?
Do you use the same approach as in the article from below?
http://blazemeter.com/blog/how-analyze-results-large-scale-load-test

Reply

waasay October 5, 2012 at 2:04 pm

good article. it is very helpful
thanqs joe.

Reply

Joe Colantonio October 9, 2012 at 8:50 pm

waasay » Thanks!

Reply

Sajid October 17, 2012 at 6:04 pm

HI Joe,
Thanks a lot for providing this nice information. Its the best article I have ever went through with very simple and elaborative examples. Its really helpful. Can you please publish few more similar articles on most important LoadRunner monitors like Hits per Seconds, Average Transaction Response Time etc.?

Reply

Joe Colantonio October 18, 2012 at 9:18 pm

Sajid » Thanks Sajid! I’ll try to write some more post similar to this. I usually only write about what I’m currently working on but I will try to focus more on LR again. Cheers~Joe

Reply

Shiv Raj Sharma December 19, 2012 at 1:12 pm

Hi Joe,

Very Nice article with lot of information. Please Provide more information on performance testing with LR.

Reply

Joe Colantonio December 20, 2012 at 10:45 pm

Shiv Raj Sharma » Thanks! I’ll try to write more post on LoadRunner :)

Reply

Anoop December 26, 2012 at 7:27 am

JOE>> This is a very good portal for performance testing. Well Done!!!!!!!!

Reply

Joe Colantonio December 28, 2012 at 7:49 pm

Thanks Anoop!

Reply

Vinod January 24, 2013 at 10:12 am

Appreicative – very informative :-)

I am using 9.1 loadrunner, how do we get – HP Diagnostics ‘J2EE – Transaction Time Spent in Element’ graph which show database/appserver in our case.

Reply

Joe Colantonio January 29, 2013 at 7:18 pm

Vinod » Thanks! Do you have a license for HP Diagnostics?

Reply

Shinto Abraham M.A January 29, 2013 at 11:43 am

Thanks a lot ….Joe’s gas gave a good understanding of throughput.

Reply

Joe Colantonio January 29, 2013 at 12:07 pm

Shinto Abraham M.A » Great!

Reply

Syed February 10, 2013 at 12:16 am

Amazing! you are doing great job for learner

Reply

Joe Colantonio February 11, 2013 at 12:55 pm

Syed » Thanks Syed!

Reply

Srikanth February 18, 2013 at 9:45 am

Thanks joe for the information pertaining to Throughput.
I have one query After the test is done we can get the total throughput value in bytes and my question is whether we can get the throughput value for a single request.(Example for a single login request how can we measure the throughput value.)
Thanks In Advance.

Reply

Mukundan March 6, 2013 at 5:12 am

Great Article Joe.

Reply

Joe Colantonio March 8, 2013 at 12:47 am

Mukundan » Thanks!

Reply

Ravi Ranjan April 26, 2013 at 10:39 am

Hi Joe,
Appreciate the way you explained Throughput.
I have one question:
What is the difference between Think Time and Pacing, if we give think time before “return 0″ in script then it will also work like pacing, then wats the difference?

Reply

Joe Colantonio May 3, 2013 at 3:21 pm

Ravi Ranjan » Thanks Ravi! Checking out Oliver Lloyds excellent explanation of pacing vs think time:
http://h30499.www3.hp.com/t5/LoadRunner-Support-Forum/Think-Time-vs-Pacing/td-p/2392288

Reply

Leave a Comment

{ 3 trackbacks }

Previous post:

Next post: