6 Open Source Test Automation Frameworks You Need to Know
Before we get started looking at the top open source test automation frameworks, it might be a good idea to talk about what a test automation framework actually is. I think of a framework as a set of best practices, assumptions, common tools and libraries that can be used across different teams.
The reason you need a framework is to help make your test automation code:
However, I’ve seen some teams take it to the extreme by building their own, elaborate automation framework from scratch. This drives me crazy, because they could have easily made use of existing open-source tools and libraries that would have met their needs without writing any code — in most cases with better results.
Before you fall into the “build your own framework” trap, be sure to check out these six open-source automation solutions.
Serenity (Formally Thucydides)
If you need a Java-based framework that integrates with behavior-driven development tools like Cucumber and JBehave, and want to keep your test scenarios at a high level with lower-level implementation details in your reports, Serenity might be the tool for you.
Serenity (formerly Thucydides) is an open-source tool designed to make writing automated acceptance and regression tests easier.
A key benefit of using Serenity is that it acts as a wrapper on top of Selenium WebDriver and BDD tools. This makes writing BDD and Selenium tests easier, because it abstracts away a lot of boilerplate code you sometimes need to write to get started.
Serenity proves a lot of built-in functionality like WebDriver management, managing state between steps, screenshots, running tests in parallel, Jira integration and more — all without having to write a single line of code.
What Serenity is really awesome at is creating unbelievable reports. Out-of-the-box, Serenity creates living documentation that can be used not only to view your Selenium BDD test results, but also as documentation for your application.
It also has an extra annotation called @Step. The @Step method gives you an extra level of abstraction that, when used correctly, can make your tests more reusable and maintainable.
Serenity takes care of many things you would normally have to code from scratch if you were creating your own BDD framework. I think it’s one of the better open-source frameworks available, but I might be a bit biased because it’s the tool I use for my day-to-day test automation efforts.
If you want to use Python for your test automation efforts, you can’t go wrong using the Robot Framework.
The Robot Framework is a mature solution that was created for testers and uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools you can use.
The Selenium WebDriver library is probably the most used external test library, but Robot Framework can also test things other than just web sites, like FTP, MongoDB, Android, Appium and more. In addition to all this open-source awesomeness, it has a lot of API’s to help make it as extensible as possible.
The Robot Framework is Python-based, but you can also use Jython(Java) or IronPython(.NET).
The keyword approach used by Robot Framework is great for testers who are already familiar with other vendor-based, keyword-driven test tools, making the transition to open source much easier for them.
If your team is made up of mostly testers, Robot Framework would be a great option for your automation framework.
RedwoodHQ takes a little bit of a different approach from the other tools on this list. It creates a website interface that allows multiple testers to work together and run their tests from one web-accessible location.
Tests can be written in Java/Groovy, Python and C# code to test web applications with Selenium, APIs or databases using their web IDE, which also includes built-in GIT integration.
You can leverage RedwoodHQ’s existing action keywords to get started creating and modifying tests quickly. To create a test script you just need to find the action you want to perform, drag it into your test case and enter in the parameter values it expects.
The built-in IDE also allows you to create and modify test cases and actions as well as the ability to execute your tests. You can also run your tests against predefined machines in parallel and see a running history of all your previous tests runs.
RedwoodHQ is tester friendly and has a lot of built-in features that allow you to create complete test automation and test management solutions for your teams.
Sahi has both an open-source and a pro version, and was developed for testing web applications. Sahi works as a proxy server that allows you to use it within a browser. From the Sahi dashboard, you can launch the browser you want to test.
The Sahi controller guides you through the recording and capturing of the interaction you want to perform against your application. When you hover over any element in the Sahi browser, for instance, it will show you all the actions you can perform against that particular element.
You can also use the controller to play your script back.
Sahi’s record and playback functionality makes it easy to get started with test automation of simple HTML applications, but beware of relying on record and playback; this approach has been proven to be a bad practice for creating maintainable, robust automation tests. It’s okay for helping create an initial test, but you’ll need to customize it to make it reliable and maintainable.
I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. However, it also comes in a pro version that lots of folks rave about.
*UPDATE I just interviewed the creator of Sahi Narayan Raman so if you want more info on Sahi open source or Pro version this is a must listen episode:
If your automation efforts are focused on User Experience Design (UX)/Layout testing, Galen might be a perfect fit for your needs.
Galen was created specifically for UX testing and has a special syntax and rules you can use to describe and verify your application’s layout. It also allows you to specify your browser size, then run tests against your application to verify your layout specifications. Galen tests also generate detailed html reports with screenshots, and a visual image comparison with a cool heat map feature.
Gauge is one of the newest open-source test automation tools for Mac, Windows and Linux, and was developed by the folks at ThoughtWorks — the same company that created Selenium.
Practitioners of TDD and BDD will appreciate Gauge’s focus on creating living/executable documentation.
Gauge automation tests called specs are written using a simple markdown language using Java, Ruby and C# within your existing developer IDEs like Eclipse and Visual Studio. Gauge’s functionality can also be extended with its support of plugins.
Gauge is pretty new and is still in beta, so if your team is into cutting edge test development you might want to give Gauge a try.
*UPDATE I was honored to have Prateek Baheti a developer at Thoughtworks to give a hands on demo of Gauge for the Automation Guild Conference 2017. The conference is over but you can still get all the sessions and recorded LIVE Q&A now ==>https://automationguild.com/
What to choose?
The preceding solutions are just a small sample of the types of open-source test automation frameworks you have available to you.
Before you write that first line of code, make sure there isn’t a library or framework you can leverage. Stop wasting your time reinventing the wheel and check out one of these automation frameworks first before investing anymore time and effort on creating your own framework from scratch.