6 Open Source Test Automation Frameworks You Need to Know

Automation Guild Online Conference Top ad

6 Open Source Test Automation Frameworks You Need to Know


Before we get started looking at the top open source test automation frameworks, it might be a good idea to talk about what a test automation framework actually is. I think of a framework as a set of best practices, assumptions, common tools and libraries that can be used across different teams.


The reason you need a framework is to help make your test automation code:

  • Reusable
  • Maintainable
  • Stable


However, I’ve seen some teams take it to the extreme by building their own, elaborate automation framework from scratch. This drives me crazy, because they could have easily made use of existing open-source tools and libraries that would have met their needs without writing any code — in most cases with better results.

Before you fall into the “build your own framework” trap, be sure to check out these six open-source automation solutions.


120314_0034_BDDSerenity1.png

Serenity (Formally Thucydides)


If you need a Java-based framework that integrates with behavior-driven development tools like Cucumber and JBehave, and want to keep your test scenarios at a high level with lower-level implementation details in your reports, Serenity might be the tool for you.


Serenity (formerly Thucydides) is an open-source tool designed to make writing automated acceptance and regression tests easier.

A key benefit of using Serenity is that it acts as a wrapper on top of Selenium WebDriver and BDD tools. This makes writing BDD and Selenium tests easier, because it abstracts away a lot of boilerplate code you sometimes need to write to get started.

Serenity proves a lot of built-in functionality like WebDriver management, managing state between steps, screenshots, running tests in parallel, Jira integration and more — all without having to write a single line of code.

What Serenity is really awesome at is creating unbelievable reports. Out-of-the-box, Serenity creates living documentation that can be used not only to view your Selenium BDD test results, but also as documentation for your application.


It also has an extra annotation called @Step. The @Step method gives you an extra level of abstraction that, when used correctly, can make your tests more reusable and maintainable.

Serenity takes care of many things you would normally have to code from scratch if you were creating your own BDD framework. I think it’s one of the better open-source frameworks available, but I might be a bit biased because it’s the tool I use for my day-to-day test automation efforts.

robot

Robot Framework


If you want to use Python for your test automation efforts, you can’t go wrong using the Robot Framework.

The Robot Framework is a mature solution that was created for testers and uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools you can use.

The Selenium WebDriver library is probably the most used external test library, but Robot Framework can also test things other than just web sites, like FTP, MongoDB, Android, Appium and more. In addition to all this open-source awesomeness, it has a lot of API’s to help make it as extensible as possible.

The Robot Framework is Python-based, but you can also use Jython(Java) or IronPython(.NET).

The keyword approach used by Robot Framework is great for testers who are already familiar with other vendor-based, keyword-driven test tools, making the transition to open source much easier for them.

If your team is made up of mostly testers, Robot Framework would be a great option for your automation framework.

RedwoodHQ


RedwoodHQ takes a little bit of a different approach from the other tools on this list. It creates a website interface that allows multiple testers to work together and run their tests from one web-accessible location.

Tests can be written in Java/Groovy, Python and C# code to test web applications with Selenium, APIs or databases using their web IDE, which also includes built-in GIT integration.

You can leverage RedwoodHQ’s existing action keywords to get started creating and modifying tests quickly. To create a test script you just need to find the action you want to perform, drag it into your test case and enter in the parameter values it expects.

The built-in IDE also allows you to create and modify test cases and actions as well as the ability to execute your tests. You can also run your tests against predefined machines in parallel and see a running history of all your previous tests runs.


RedwoodHQ is tester friendly and has a lot of built-in features that allow you to create complete test automation and test management solutions for your teams.

Sahi


Sahi has both an open-source and a pro version, and was developed for testing web applications. Sahi works as a proxy server that allows you to use it within a browser. From the Sahi dashboard, you can launch the browser you want to test.

The Sahi controller guides you through the recording and capturing of the interaction you want to perform against your application. When you hover over any element in the Sahi browser, for instance, it will show you all the actions you can perform against that particular element.

You can also use the controller to play your script back.

Sahi’s record and playback functionality makes it easy to get started with test automation of simple HTML applications, but beware of relying on record and playback; this approach has been proven to be a bad practice for creating maintainable, robust automation tests. It’s okay for helping create an initial test, but you’ll need to customize it to make it reliable and maintainable.

I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. However, it also comes in a pro version that lots of folks rave about.

Galen Framework


If your automation efforts are focused on User Experience Design (UX)/Layout testing, Galen might be a perfect fit for your needs.

Galen was created specifically for UX testing and has a special syntax and rules you can use to describe and verify your application’s layout. It also allows you to specify your browser size, then run tests against your application to verify your layout specifications. Galen tests also generate detailed html reports with screenshots, and a visual image comparison with a cool heat map feature.

Tests can be written in a simple text file using the Galen syntax, JavaScript or Java.

Gauge


Gauge is one of the newest open-source test automation tools for Mac, Windows and Linux, and was developed by the folks at ThoughtWorks — the same company that created Selenium.


Practitioners of TDD and BDD will appreciate Gauge’s focus on creating living/executable documentation.

Gauge automation tests called specs are written using a simple markdown language using Java, Ruby and C# within your existing developer IDEs like Eclipse and Visual Studio. Gauge’s functionality can also be extended with its support of plugins.

Gauge is pretty new and is still in beta, so if your team is into cutting edge test development you might want to give Gauge a try.

What to choose? 

The preceding solutions are just a small sample of the types of open-source test automation frameworks you have available to you.

Before you write that first line of code, make sure there isn’t a library or framework you can leverage. Stop wasting your time reinventing the wheel and check out one of these automation frameworks first before investing anymore time and effort on creating your own framework from scratch.

28 comments
Ady - May 11, 2016

Every-time I read your posts I learn something new and this time it was a newbie Gauge.
Can I say that you are my role model!! Whenever someone asks me what keeps you going? I say It’s someone like Joe with all the passion and knowledge for test automation.

Keep up the good work!

Cheers,
Ady

Reply
Jörg Sievers - May 11, 2016

Hi.

“I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. However, it also comes in a pro version that lots of folks rave about.”

SAHI (Pro) is not only for “simple HTML applications” because it is possible to use the new HTML5 technologies like ShadowDOM and custom HTML. You are right that the record won’t help to create good automation tests but it is a nice start where you can select alternative detectors (!) and also add asserts you want to take care of. The best advantage of SAHI is the handling of AJAX because you can write on top of very complex pages test scripts without any timeout-patterns you need e.g. for Selenium.

We have saved 50% time in creating test cases/user scenario tests vs. Selenium and the test execution results on the CIS are very reobust.

Yours,
Jogi

Reply
    Joe Colantonio - May 12, 2016

    HI Jogi – thats good to know! I will try to get a pro license and really give it a proper eval

    Reply
SreeCharan Shroff - May 12, 2016

http://www.extensivetesting.org/
is also one such open source test automation framework

Reply
Punee Varma - May 12, 2016

Hi Joe,

Were you a developer before you started doing testing? I have been doing testing for a long time but I don’t have the developer background. What open source tool do you suggest I should learn without knowing too much about programming. I have been using HP UFT for a while. I am trying to get away from this tool due not enough job out there like open source does.
Thanks

Reply
    Joe Colantonio - May 12, 2016

    Actually no – I didn’t have a developer background. I went to school for music :) I had a friend who got me a job doing tech support for Microsoft Visual Studio even thought I knew nothing about programming. I learned on the job. I would check out learning Selenium with Python. TO me that would be the easiest to start with.

    Reply
Divya - May 12, 2016

Nice Learning, Never heard about Gauge and RedwoodHQ, new learning , will try and explore them for sure.
Its always been a great experience working with you Joe, also learning new things from your Blog.

Please continue to share many more things about automation

Cheers,
Divya

Reply
Mahesh - May 12, 2016

I am very pleased to see this blog. I can sleep today with the feeling that I learnt something new today :-)

Thanks for sharing !!!

Thanking you,
Mahesh

Reply
Hari - May 12, 2016

Great information, Appreciate your efforts Joe.

Thank You,
Hari

Reply
saunak tarafdar - May 12, 2016

Great Article! I have worked with Robot Framework and I am in the middle of learning Galen and Gauge. Have you had a chance to explore Open2Test Framework?

Reply
Ram Malapati - May 12, 2016

Wonderful Article very informative. keep up the good work Joe.

I’m just wondering which framework is more suitable for WebServices RestFul API’s Automation in a .Net Stack based environment.

We’re use Postman plug-in but eager to know your thoughts on Web Services Automation frameworks.

Please let me know if there are any BDD kind of frameworks that can be helpful for us to write tests in readable format.

Reply
Sanjay Jain - May 13, 2016

Thanks for sharing these information Joe. Your every post have refreshed the concepts and have added the information on how it works. This post will help me to cut down the scope of evaluation for my current project.

Thanks a lot..! Good Work and Happy to be learn from your posts.

Regards,
Sanjay

Reply
Kedar - May 13, 2016

Joe, your all posts and articles and the podcasts are full of awesomeness… Wish to work with a person like you…I watch all your Podcasts and they add a lot to my knowledge

Joe the CONTRIBUTOR

Reply
Testing Bits – 5/8/16 – 5/14/16 | Testing Curator Blog - May 15, 2016

[…] 6 Open Source Test Automation Frameworks You Need to Know – Joe Colantonio – http://www.joecolantonio.com/2016/05/10/6-open-source-test-automation-frameworks-need-know/ […]

Reply
Compelling Sunday – 16 Posts on Programming and QA - May 15, 2016

[…] 6 Open Source Test Automation Frameworks You Need to Know – Joe Colantonio […]

Reply
Radek Bokuvka - May 16, 2016

I do like reading your articles Joe, and this is one of them. There is always something to learn!

Regards,
Radek

Reply
GAGANDEEP SINGH - May 18, 2016

Nice article Joe!!!
Thanks for sharing the info….

Reply
Harshal - May 19, 2016

Joe, suggest to evaluate Open2Test. I have found it very good.

Reply
    Joe Colantonio - May 19, 2016

    Hi Harshal – I’ve never heard of it. Thanks for the suggestion I will check it out! Do you use it for Selenim of QTP? Looks like it supports both

    Reply
HariCharan - May 19, 2016

Hi Joe,

Nice Article Man! Did you get a chance to evaluate Tellurium?

https://www.linkedin.com/pulse/tellurium-gonna-new-era-test-automation-hari-charan?articleId=6136584880018030592

Any Comments on it?

Reply
Tej - May 24, 2016

Hi Joe,

I love your posts! Always get to learn something new.
I did a POC on Serenity and i have become a fan of that framework. I am implementing this in my current company for a project!!

It is so easy to use as well as the reporting is awesome, supports BDD as well as Junit style of tests. Parallel Testing ,DataDriven testing can also be achieved without any issues.

Regards
Tej

Reply
Click here to add a comment

Leave a comment: