How to Choose Between Open-Source and Commercial Test Tools
One of the most important questions test engineers face when starting a new test automation project is what test tools to use. Nowadays, many automatically consider only open source tools. But is that always the correct choice? This is a topic that I recently covered in a tech beacon article recently.
Based on my experience, not always. The pendulum has swung from vendor to open source, but as is usually the case, the truth is found somewhere in the middle. Open source testing tools are awesome, but there are many instances when these tools just aren’t enough. Here's when to start looking for a commercial tool.
Testing More Than One Technology
When choosing a test tool it’s wise to first take a step back and clearly identify your testing needs. Open source may be a great choice or a commercial tool may be better; just be sure you’re making an informed decision before choosing one over the other.
The answer lies in making the decision to list all of the different technologies your organization will have to automate against. For larger companies, having one test tool that you can use across all teams and divisions can be a big help (and deliver cost savings).
In most cases, using an open source solution only—a Selenium-based testing framework, for instance—is not going to help you achieve the test coverage you need in this situation. Selenium is for browser-based automation only, so if you need to test non-browser-based applications, it's not your best choice.
If you're one of the many automation engineers who need to automate a broad set of technologies, such as REST, .Net, Java, Oracle Forms, SAP and mobile applications, vendor-based tools let you easily combine all these in one test.
Easy to Use Test Tool Requirement
Many open source tools are really not much more than an API. Creating a testing product using those tools requires having a team of engineers who are technically able to cobble together a custom test automation framework that meets their needs. In my experience, I’ve had a difficult time finding testers who have these types of skills.
Another benefit of using vendor-based tools is that good products simplify the test creation process. Out-of-the-box, they provide extra features such as built-in object spies, object recognition improvements, reports, monitoring and defect integration without you having to do anything. These features enable you to have a total application lifecycle solution in place.
Also, many companies find that, because of the architecture of vendor-based tools, the business has an easier time finding qualified test engineers, since these products generally require fewer programming skills.
Compliant Testing Tool Platforms
Some companies require engineers to get approval for the testing tools the want to use. I know one company that requires that any tool to be used for a given project be submitted to an architecture review board before teams are allowed to use them.
In situations like these, vendor-based tools may be a better fit, since they’ve usually gone through the rigors of meeting common compliance standards. In highly regulated industries such as healthcare and finance, choosing a tool that has proven to be safe, secure, and meets compliance standards is an attractive option.
Most vendor-based testing tools have been around for a long time, and while some test engineers see that as a weakness, others see it as an advantage. These commercial tools have been battle-hardened, and come with years of domain knowledge baked into them. And most of the bugs have been shaken out of these products after years of development. That leads to the next reason to consider vendor-based tools:
What Happens When You Have Bugs in Your Test Tool
What happens when you find a bug in open source testing software? You are at the mercy of the contributors when it comes to resolving open bugs. Yes, because it’s open source, you could theoretically fix it and commit the change yourself. But how many test engineers do you think have this ability?
Vendor-based technical support is better able to handle bug fixes or quickly provide a workaround to customers. They have a paid technical staff that works around the clock and is solely dedicated to customer issues.
With a paid product, you can plead your case to the vendor and the vendor will act with a sense of urgency to fix the issue— or at least add it to their product roadmap. (* This is a generalization — I've paid for support from vendors in the past and have received little support or poor response as well)
Try that with an open source project, and you might receive a snarky comment, or worse, encounter total indifference. It’s not that open source contributors don’t care, but most of the time they're supported by volunteers who don’t get paid for time spent solving your issues.
Open source is great, but many companies prefer to pay in order to have some measure of accountability.
As the saying goes, sometimes you get what you pay for. But don’t free tools save you money in the long run anyway?
Does Free Open-Source Test Tools Save You Money?
Open source tools are free, but do they really save you money? Not always. For example, consider cloud-based testing environment providers.
As more and more device types and operating systems arrive, with ever more complexity, the ability of your company to maintain it own test environments will consume more of your time, energy and money.
Do you want to spend all your time maintaining test environments and patching and fixing issues before you even get to the actual testing? Cloud-based testing providers take care of all the maintenance so that you can concentrate on what matters to you most — testing and development.
The biggest benefit of going with vendor-based tools is the reduced cost of ownership associated with easier implementation and less administrative overhead.
Think About The Business' Needs First
It’s tempting to always use the latest and greatest tools. But before you proceed, be sure it’s the right decision for your team and company, rather than just a way to build out your resume or satisfy your curiosity.
Professional engineers should think of automation projects, and test assets as part of the overall business. It's not about writing code or using the most fashionable technologies; it's about the fact that automation is a business solution.
How to Make the Right Test Tool Choice
I love Open source test tools. I’m a big proponent of using open source testing software. I truly believe in them and use many open source tools myself. But they're not always a total replacement for commercial testing tools. You just need to choose the right tools for your needs, whether they're open source or vendor-provided.
And you don't need to choose one or the other: Many vendor-based testing tools like LeanFT and SmartBear integrate with their open source counterparts, which enables to you get the best of both worlds.
So don't dismiss commercial tools—or open source tools—out of hand. Keep an open mind when evaluating your testing options, and you just might find that a mix of vendor-based and open-source tools is the best choice for your organization.