Last week I hosted a conference called PerfGuild – the first online conference dedicated to performance testing. It was a great time, and it yielded some awesome questions for our performance testing experts.
Here are the top 11 questions and answers from the LIVE Q&A portion of the event.
1.Why use LoadRunner when there’s a free solution like JMeter?
I chose this question because I think it’s a common one for folks who are deciding which tool(s) to use in their performance testing efforts. Shlomi Nissim admitted that he might not be the most reliable person to answer this question because he thought folks would perceive him as someone who has a clear preference for LoadRunner, but I believe he gave a very objective answer.
First, Shlomi said that he valued all the performance tools that are out there; because driving a load test is not a trivial exercise—and technology is involved—they’re not exactly simple to deal with.
If one is truly objective, however, he would have to admit that Load Runner–no matter how old it looks, or how much you like an open-source solution–is the world standard for load testing.
LoadRunner has years of experience built into it.
Also, many would probably agree that LoadRunner brings the most accurate virtual user simulation when compared with any other tool out there.
Since it’s a vendor tool, it is also kept up-to-date. LoadRunner also has a ton of technologies and supports and a wealth of functionality surrounding technologies that are driving the work load.
For example, the analysis tool is vibrant. You can import any data that you want, and customize things very easily. You can build your own monitors using virtual users’ scripts quite easily, and plug your other data into LoadRunner.
Although open-source tools like JMeter are perfectly fine tools and can meet your needs no matter how you look at it based on a number of benchmarks that were identified by some of HP’s customers while testing LoadRunner against other tools, they invariably prove that LoadRunner is more accurate.
So if that’s what’s most important to you, LoadRunner is still one of the top choices for that goal.
2.Should you have an established performance testing process before starting on performance engineering, or can you start both processes at the same time?
What he often finds is if that once you start incorporating some performance engineering practices, you’ll quickly begin revealing the value of doing things like performance testing or performance engineering. He feels that using some basic tips and tricks and getting started quickly—showing that fast value across technology and the business—is the best way to get started.
3.If you have ten virtual users and you know the transactions per second, and you’re trying to test for 100 users, can you run ten users doing more transactions per second to mimic 100 users? Is there an equation for this?
Ramya Moorthy (her session was on queuing theory and its application in performance testing) answered that everything starts with Little’s Law, which states that the average number “N” is equal to throughput multiplied by the response time.
So it is purely plugging in the values and what you have in Little’s Law you can figuring out whether your available number of Vuser’s licenses is enough.
But it’s not that easy – what happens when you start adding
Other layers of complexity to your test like think time? Increasing think time obviously would bring down your throughput.
How do you emulate a real user and how they will be accessing your application?
If your objective is to only stress the system to determine how your system is performing, it’s feasible–even if you have a small number of licenses.
But if your objective is to realistically certify that your application can support a hundred real time users, then you obviously need to have 100 virtual user licenses.
4.Is there one thing you see featured on News of the Damned that causes the majority of performance issues?
5.When testing an IoT platform, what is the advantage of using a Raspberry Pi versus a Windows or Linux machine as a load generator?
IoT performance testing was a hot topic this year, and Henrik Rexed gave a cool demo on how to use a Raspberry Pi as a tiny load generator.
When asked this question, Henrik replied that the primary purpose of using a Raspberry Pi is to simulate an endpoint and generate a message to a gateway, which the gateway will then pass on to the IoT platform.
In essence, you’ll be using the Raspberry Pi to simulate sensors communicating with the gateway–but you’ll never be able to adequately qualify the IoT platform through Raspberry Pi alone.
For example, if you were testing connected houses or buildings, you would need thousands or millions of gateways talking to the Cloud. For that type of test, it’s recommended to use standard Linux or Windows load generators.
But when you need to test the capacity of your getaways, using a Raspberry pie at that point would be better.REGISTER NOW!
6.When you’re load testing an application locally compared to the application being hosted in production, do you see much of a difference in response times?
Amber Race thinks that testing in production is the best thing to do.
For example, Amber talked about that she had been struggled for years to replicate the production environment in a lower environments. Many times she ended up testing against a lower powered environment and was forced to extrapolate the results and make a conclusion as if it was production. This never ended well.
Amber did warn that if you do tests in production always make sure that you already have monitoring capabilities in place to make sure that your not breaking production with your tests
But if you can keep an eye on things, testing in production is the best way to go especially for API Load testing.
7.We are an HP LoadRunner shop but want to push performance testing to an Agile scrum teams for Web apps. Is Gatling a better choice for this user case?
Stephane Landelle, a creator of Gatling, explained that he couldn’t compare it with Load Runner; however, his company has been helping more and more large organizations migrate from LoadRunner to Gatling.
One of the reasons for this is that Gatling is a good fit for propagating Load Test to the whole team and having all the different teams bring load tests into the continuous integration pipeline.
What he sees more and more of is having the project teams perform their performance tests in isolation, after which another team that specializes in load testing assembles components coming from different teams to build larger, full-scale load tests.
As a result, for a more Agile process, what’s needed is a tool that can you can use for all the different tests that you would perform across the organization.
Also having a load test written in the same language as the application code makes maintaining the test much easier–making a tool like Gatling a better fit for developer-centric performance testing efforts.
8.Where does performance testing fit in DevOps?
Richard said he believes in the mindset that says performance testing being the thing everyone does two days before “go live” is changing.
Nowadays he sees more and more performance testing activities happen as part of continuous integration pipelines.
You may not do a performance test for every check-in, but it’s feasible to run as a nightly task.
Richard has also seen that with newer technologies like containers and the Cloud, the ability to create an environment that is just like production has become a lot easier.
9.Would you say that Selenium is an optimal tool for performance testing?
Andrew Krug replied that for pure performance testing he wouldn’t use Selenium by itself because it’s hard to generate a suitable load for performance testing. He uses it mainly for the performance monitoring aspect of it.
Using this approach is also a good “poor man’s” performance test because you can use your Selenium tests’ overall run time trends as an indicator of whether there may be a potential performance issue that you need to dig into further.
But for straight performance testing, he doesn’t feel that Selenium is the tool for it because all the different users have to be simulated.
10.Why would someone want to use APIs for performance testing rather than another protocol?
Harsh Upreti believes that when planning a performance test it’s important to break your application down into its major areas before you actually start testing.
When you do this, you’ll be able to see that how your application performs is a combination of different factors. Many testers are testing slower, more complicated processes; many of which could be tested under the UI at an API level instead.
Also, since APIs are quicker and tend to be easier to create performance tests script for as opposed to UI tests, it makes sense to focus more of your performance testing efforts on them.
11.How much time do spend on analysis and retests versus test creation and initial runs?
Rebecca Clinard spends her valuable time in the analysis phase. The reason is that she’d rather analyze the results and really take her time to understand what’s going on.
This is critical to do before creating another performance test — especially if the previous performance testing results don’t make sense to you.
Rebecca’s best method of analyzing weird performance behavior if her performance SLA isn’t met is to start from first principles.
First – go basic! If you have a complicated performance script or scenario, reduce it to its most essential parts. For example, start off with a login. Once you’ve gotten consistent results with the login, add a more complex test like a one-click scenario.
Before redesigning a whole new performance test, be sure that your first initial baseline makes sense.
Another method Rebecca uses is what she called the “half and double” methodology. It goes like this:
Let’s say your goal for performance is to reach 5,000 concurrent users and the task falls over 2,000 users.
Well, number one, great job! You’ve identified the capacity or bottleneck, but you need to test those skills properly.
So your goal was 5,000 and started crashing over 2,000.
In this example, you would divide it in half, making it 1,000.
Now start at 1,000, which is your new goal. Then go to 2,000, and as you do so monitor your system to determine what deteriorates or becomes saturated.
Tune those areas and keep proceeding with the half/double tune until you reach your goal.
This should help you to get analyzable results.
Lots of Great Performance Testing Questions
We had more than 15 speakers and tons more questions, so you can imagine the performance testing awesomeness that took place during the conference.
There’s Still Time to Check out PerfGuild
If you don’t have anything to do with performance testing and didn’t make this year’s PerfGuild, I’ve got great news for you. Due to overwhelming demand, I’ve decided to keep registration open, since all the sessions are recorded and are still accessible. You can view them all, see all the chat and get access to the private slack channel.REGISTER NOW!