6 Open Source Test Automation Frameworks You Need to Know


6 Open Source Test Automation Frameworks You Need to Know

Before we get started looking at the top open source test automation frameworks, it might be a good idea to talk about what a test automation framework actually is. I think of a framework as a set of best practices, assumptions, common tools and libraries that can be used across different teams.

The reason you need a framework is to help make your test automation code:

  • Reusable
  • Maintainable
  • Stable

However, I’ve seen some teams take it to the extreme by building their own, elaborate automation framework from scratch. This drives me crazy, because they could have easily made use of existing open-source tools and libraries that would have met their needs without writing any code — in most cases with better results.

Before you fall into the “build your own framework” trap, be sure to check out these six open-source automation solutions.


Serenity (Formally Thucydides)

If you need a Java-based open source test automation framework that integrates with behavior-driven development tools like Cucumber and JBehave, and want to keep your test scenarios at a high level with lower-level implementation details in your reports, Serenity might be the tool for you.

Serenity (formerly Thucydides) is an open-source tool designed to make writing automated acceptance and regression tests easier.

A key benefit of using Serenity is that it acts as a wrapper on top of Selenium WebDriver and BDD tools. This makes writing BDD and Selenium tests easier, because it abstracts away a lot of boilerplate code you sometimes need to write to get started.

Serenity proves a lot of built-in functionality like WebDriver management, managing state between steps, screenshots, running tests in parallel, Jira integration and more — all without having to write a single line of code.

What Serenity is really awesome at is creating unbelievable reports. Out-of-the-box, Serenity creates living documentation that can be used not only to view your Selenium BDD test results, but also as documentation for your application.

It also has an extra annotation called @Step. The @Step method gives you an extra level of abstraction that, when used correctly, can make your tests more reusable and maintainable.

Serenity takes care of many things you would normally have to code from scratch if you were creating your own BDD framework. I think it’s one of the better open source test automation frameworks available, but I might be a bit biased because it’s the tool I use for my day-to-day test automation efforts.


Robot Framework

If you want to use Python for your test automation efforts, you can’t go wrong using the Robot Framework.

The Robot Framework is a mature solution that was created for testers and uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools you can use.

The Selenium WebDriver library is probably the most used external test library, but Robot Framework can also test things other than just web sites, like FTP, MongoDB, Android, Appium and more. In addition to all this open-source awesomeness, it has a lot of API’s to help make it as extensible as possible.

The Robot Framework is Python-based, but you can also use Jython(Java) or IronPython(.NET).

The keyword approach used by Robot Framework is great for testers who are already familiar with other vendor-based, keyword-driven test tools, making the transition to open source much easier for them.

If your team is made up of mostly testers, Robot Framework would be a great option for your open source test automation framework.


RedwoodHQ takes a little bit of a different approach from the other tools on this list. It creates a website interface that allows multiple testers to work together and run their tests from one web-accessible location.

Tests can be written in Java/Groovy, Python and C# code to test web applications with Selenium, APIs or databases using their web IDE, which also includes built-in GIT integration.

You can leverage RedwoodHQ’s existing action keywords to get started creating and modifying tests quickly. To create a test script you just need to find the action you want to perform, drag it into your test case and enter in the parameter values it expects.

The built-in IDE also allows you to create and modify test cases and actions as well as the ability to execute your tests. You can also run your tests against predefined machines in parallel and see a running history of all your previous tests runs.

RedwoodHQ is tester friendly and has a lot of built-in features that allow you to create complete test automation and test management solutions for your teams.


Sahi has both an open-source and a pro version, and was developed for testing web applications. Sahi works as a proxy server that allows you to use it within a browser. From the Sahi dashboard, you can launch the browser you want to test.

The Sahi controller guides you through the recording and capturing of the interaction you want to perform against your application. When you hover over any element in the Sahi browser, for instance, it will show you all the actions you can perform against that particular element.

You can also use the controller to play your script back.

Sahi’s record and playback functionality makes it easy to get started with test automation of simple HTML applications, but beware of relying on record and playback; this approach has been proven to be a bad practice for creating maintainable, robust automation tests. It’s okay for helping create an initial test, but you’ll need to customize it to make it reliable and maintainable.

I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. However, it also comes in a pro version that lots of folks rave about.

*UPDATE I just interviewed the creator of Sahi Narayan Raman so if you want more info on Sahi open source or Pro version this is a must listen episode:

Galen Framework

If your automation efforts are focused on User Experience Design (UX)/Layout testing, Galen might be a perfect fit for your needs.

Galen was created specifically for UX testing and has a special syntax and rules you can use to describe and verify your application’s layout. It also allows you to specify your browser size, then run tests against your application to verify your layout specifications. Galen tests also generate detailed html reports with screenshots, and a visual image comparison with a cool heat map feature.

Tests can be written in a simple text file using the Galen syntax, JavaScript or Java.

*UPDATE I just interviewed the creator of Galen –Ivan Shubin so if you want more info on responsive design testing this is a must listen episode:


Gauge is one of the newest open source test automation frameworks for Mac, Windows and Linux, and was developed by the folks at ThoughtWorks — the same company that created Selenium.

Practitioners of TDD and BDD will appreciate Gauge’s focus on creating living/executable documentation.

Gauge automation tests called specs are written using a simple markdown language using Java, Ruby and C# within your existing developer IDEs like Eclipse and Visual Studio. Gauge’s functionality can also be extended with its support of plugins.

Gauge is pretty new and is still in beta, so if your team is into cutting edge test
development you might want to give Gauge a try.

Automation Guild Videos

*UPDATE I was honored to have Prateek Baheti a developer at Thoughtworks to give a hands on demo of Gauge for the Automation Guild Conference 2017. The conference is over but you can still get all the sessions and recorded LIVE Q&A now ==>https://automationguild.com/

What Open Source Test Automation Frameworks to choose? 

The precedingopen source test automation frameworks are just a small sample of the types of open source test automation frameworks you have available to you.

Before you write that first line of code, make sure there isn’t a library or framework you can leverage. Stop wasting your time reinventing the wheel and check out one of these automation frameworks first before investing anymore time and effort on creating your own framework from scratch.

Ady - May 11, 2016

Every-time I read your posts I learn something new and this time it was a newbie Gauge.
Can I say that you are my role model!! Whenever someone asks me what keeps you going? I say It’s someone like Joe with all the passion and knowledge for test automation.

Keep up the good work!


Jörg Sievers - May 11, 2016


“I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. However, it also comes in a pro version that lots of folks rave about.”

SAHI (Pro) is not only for “simple HTML applications” because it is possible to use the new HTML5 technologies like ShadowDOM and custom HTML. You are right that the record won’t help to create good automation tests but it is a nice start where you can select alternative detectors (!) and also add asserts you want to take care of. The best advantage of SAHI is the handling of AJAX because you can write on top of very complex pages test scripts without any timeout-patterns you need e.g. for Selenium.

We have saved 50% time in creating test cases/user scenario tests vs. Selenium and the test execution results on the CIS are very reobust.


    Joe Colantonio - May 12, 2016

    HI Jogi – thats good to know! I will try to get a pro license and really give it a proper eval

SreeCharan Shroff - May 12, 2016

is also one such open source test automation framework

Punee Varma - May 12, 2016

Hi Joe,

Were you a developer before you started doing testing? I have been doing testing for a long time but I don’t have the developer background. What open source tool do you suggest I should learn without knowing too much about programming. I have been using HP UFT for a while. I am trying to get away from this tool due not enough job out there like open source does.

    Joe Colantonio - May 12, 2016

    Actually no – I didn’t have a developer background. I went to school for music :) I had a friend who got me a job doing tech support for Microsoft Visual Studio even thought I knew nothing about programming. I learned on the job. I would check out learning Selenium with Python. TO me that would be the easiest to start with.

Divya - May 12, 2016

Nice Learning, Never heard about Gauge and RedwoodHQ, new learning , will try and explore them for sure.
Its always been a great experience working with you Joe, also learning new things from your Blog.

Please continue to share many more things about automation


Mahesh - May 12, 2016

I am very pleased to see this blog. I can sleep today with the feeling that I learnt something new today :-)

Thanks for sharing !!!

Thanking you,

Hari - May 12, 2016

Great information, Appreciate your efforts Joe.

Thank You,

saunak tarafdar - May 12, 2016

Great Article! I have worked with Robot Framework and I am in the middle of learning Galen and Gauge. Have you had a chance to explore Open2Test Framework?

Ram Malapati - May 12, 2016

Wonderful Article very informative. keep up the good work Joe.

I’m just wondering which framework is more suitable for WebServices RestFul API’s Automation in a .Net Stack based environment.

We’re use Postman plug-in but eager to know your thoughts on Web Services Automation frameworks.

Please let me know if there are any BDD kind of frameworks that can be helpful for us to write tests in readable format.

Sanjay Jain - May 13, 2016

Thanks for sharing these information Joe. Your every post have refreshed the concepts and have added the information on how it works. This post will help me to cut down the scope of evaluation for my current project.

Thanks a lot..! Good Work and Happy to be learn from your posts.


Kedar - May 13, 2016

Joe, your all posts and articles and the podcasts are full of awesomeness… Wish to work with a person like you…I watch all your Podcasts and they add a lot to my knowledge


Testing Bits – 5/8/16 – 5/14/16 | Testing Curator Blog - May 15, 2016

[…] 6 Open Source Test Automation Frameworks You Need to Know – Joe Colantonio – https://www.joecolantonio.com/2016/05/10/6-open-source-test-automation-frameworks-need-know/ […]

Compelling Sunday – 16 Posts on Programming and QA - May 15, 2016

[…] 6 Open Source Test Automation Frameworks You Need to Know – Joe Colantonio […]

Radek Bokuvka - May 16, 2016

I do like reading your articles Joe, and this is one of them. There is always something to learn!


GAGANDEEP SINGH - May 18, 2016

Nice article Joe!!!
Thanks for sharing the info….

Harshal - May 19, 2016

Joe, suggest to evaluate Open2Test. I have found it very good.

    Joe Colantonio - May 19, 2016

    Hi Harshal – I’ve never heard of it. Thanks for the suggestion I will check it out! Do you use it for Selenim of QTP? Looks like it supports both

HariCharan - May 19, 2016

Hi Joe,

Nice Article Man! Did you get a chance to evaluate Tellurium?


Any Comments on it?

Tej - May 24, 2016

Hi Joe,

I love your posts! Always get to learn something new.
I did a POC on Serenity and i have become a fan of that framework. I am implementing this in my current company for a project!!

It is so easy to use as well as the reporting is awesome, supports BDD as well as Junit style of tests. Parallel Testing ,DataDriven testing can also be achieved without any issues.


Aastha - June 7, 2016

Hi Joe,

Your articles are so informative.I have recommended few of my friends to read your awesome articles. Thanks!

I am currently using BDD framework along with Jersey to test Webservices.I tried hands on Rest Assured but decided to use Jersey as developers are using the same library in my project.

Sojan - December 29, 2016

Great Article! I am planning to learn Gauge. How can i automate UI and API using gauge?
can you help me in this?


Dan - January 18, 2017

Hi Joe,

Would you use Serenity for testing REST APIs too? If not, what automation framework would you recommend for this?


    Joe Colantonio - January 18, 2017

    Hi Dan use – Serenity also integrates with Rest-Assured for REST API testing. IN fact it would work for any java library that you want to use for testing your APIs since its a framework not a test tool you can use any valid Java library.

Chris Brown - January 30, 2017

I’ve used Sahi OS and Sahi Pro a fair bit during evaluation. There is quite a few differences between the two.
Compared to other commercial testing software I found Sahi to be much better due to its focus on scripting, interwoven with JS so it simple to learn and if anything is lacking you just write JS code to do the task.
It uses relational elements to find exactly what you want which is great for dynamic web pages.

Rajesh Shinde - February 28, 2017

Hi Joe,

Learned lot from this post. In my company we are evaluating different tools/frameworks for our upcoming framework. Our project involved mainly into the micro services which were developed in Java. We are going to test the API’s as well as the UI. We also wants to look after the service log file processing based on the various operations via API. Since different services were owned by different teams within our company we also needed to think about the adaptability as well as the ease of use by non-coder folks. In our last project we used TestNG but I may be thinking of using the ROBOT.

Could you please guide me about the pros & cons of both of these frameworks? Is the Gauge or SAHI are having any new features that the TestNG and ROBOT does not have?

Rajesh Shinde

Pritesh - June 21, 2017

Can we create dynamic script using Robot Framework?

Seth Khon - October 3, 2017

Is there anything wrong with MS Test Framework, which comes with Visual Studio 2017?

    Joe Colantonio - October 4, 2017

    Not if it works for you or your team :) Its not on this list because its not “open source”

Clayton - January 26, 2018

Robot framework can be used with any .net language (e.g c#) not just iron python!

Click here to add a comment

Leave a comment: