11 Top Open Source Test Automation Frameworks (You Need to Know)

11 Top Open Source Test Automation Frameworks (You Need to Know)

Automation Testing Framework

Before we get started looking at the top open source automation testing frameworks, it might be a good idea to talk about what an automation testing framework actually is. I think of a framework as a set of best practices, assumptions, common tools and libraries that can be used across different teams. (I also used this content in my article for TechBeacon)

What is an Automation Testing Framework

The reason you need a framework is to help make your test automation code:

  • Reusable
  • Maintainable
  • Stable

However, I’ve seen some teams take it to the extreme by building their own, elaborate automation test framework from scratch. This drives me crazy because they could have easily made use of existing open-source tools and libraries that would have met their needs without writing any code — in most cases with better results. Many of these tools also have a proven built-in automation testing framework architecture.

Before you fall into the “build your own framework” trap, be sure to check out these ten open-source automation solutions.

120314_0034_BDDSerenity1.png

1) Serenity (Formally Thucydides)

If you need an open source automation testing framework selenium based solution that integrates with behavior-driven development tools like Cucumber and JBehave, and want to keep your test scenarios at a high level with lower-level implementation details in your reports, Serenity might be the tool for you. It comes in two flavors Java and JavaScript(SerenityJS).

Serenity (formerly Thucydides) is an open-source tool designed to make writing automated acceptance and regression tests easier.

A key benefit of using Serenity is that it acts as a wrapper on top of Selenium WebDriver, Appium and BDD tools. This makes writing BDD and Selenium tests easier, because it abstracts away a lot of boilerplate code you sometimes need to write to get started.

Serenity proves a lot of built-in functionality like WebDriver management, managing state between steps, screenshots, running tests in parallel, Jira integration and more — all without having to write a single line of code.

What Serenity is really awesome at is creating unbelievable reports. Out-of-the-box, Serenity creates living documentation that can be used not only to view your Selenium BDD test results but also as documentation for your application.
It also has an extra annotation called @Step. The @Step method gives you an extra level of abstraction that, when used correctly, can make your tests more reusable and maintainable.

Serenity takes care of many things you would normally have to code from scratch if you were creating your own BDD framework. I think it’s one of the better open source test automation frameworks available, but I might be a bit biased because it’s the tool I use for my day-to-day test automation efforts.

robot

2. Robot Framework

If you want to use a python test automation framework for your test automation efforts, you can’t go wrong using the Robot Framework.

The Robot Framework is a mature solution that was created for testers and uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools you can use.

The Selenium WebDriver library is probably the most used external test library, but Robot Framework can also test things other than just websites, like FTP, MongoDB, Android, Appium and more. In addition to all this open-source awesomeness, it has a lot of API’s to help make it as extensible as possible.

The Robot Framework is Python-based, but you can also use Jython(Java) or IronPython(.NET).

The keyword approach used by Robot Framework is great for testers who are already familiar with other vendor-based, keyword-driven test tools, making the transition to open source much easier for them.

If your team is made up of mostly testers, Robot Framework would be a great option for your open source automation testing framework python based choice.

3. Sahi

Sahi has both an open-source and a pro version and was developed for testing web applications. Sahi works as a proxy server that allows you to use it within a browser. From the Sahi dashboard, you can launch the browser you want to test.

The Sahi controller guides you through the recording and capturing of the interaction you want to perform against your application. When you hover over any element in the Sahi browser, for instance, it will show you all the actions you can perform against that particular element.

You can also use the controller to play your script back.

Sahi’s record and playback functionality makes it easy to get started with test automation of simple HTML applications, but beware of relying on record and playback; this approach has been proven to be a bad practice for creating maintainable, robust automation tests. It’s okay for helping create an initial test, but you’ll need to customize it to make it reliable and maintainable.

I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. However, it also comes in a pro version that lots of folks rave about.

4. Galen Framework

If your automation efforts are focused on User Experience Design (UX)/Layout testing, Galen automation testing framework might be a perfect fit for your needs.

Galen was created specifically for UX testing and has a special syntax and rules you can use to describe and verify your application’s layout. It also allows you to specify your browser size, then run tests against your application to verify your layout specifications. Galen tests also generate detailed HTML reports with screenshots and a visual image comparison with a cool heat map feature.

Tests can be written in a simple text file using the Galen syntax, JavaScript or Java.

Many testers confuse the purpose of a test automation framework. Some are designed for different purposes and excel in others. For example, a framework designed for UX design testing like Galen is much different than one that uses visual validation tools like Applitools. Check out my article on Applitools Vs Galen Framework What’s the Difference for more info.

5. Gauge

Gauge is one of the newest open source test automation frameworks for Mac, Windows, and Linux, and was developed by the folks at ThoughtWorks — the same company that created Selenium.

Practitioners of TDD and BDD will appreciate Gauge’s focus on creating living/executable documentation.

Gauge automation tests called specs are written using a simple markdown language using Java, Ruby, and C# within your existing developer IDEs like Eclipse and Visual Studio. Gauge’s functionality can also be extended with its support of plugins.

Gauge is also a tool agnostic test automation framework. It was developed from the start to be a BYOT (Bring Yout Own Tool) framework. So you can use Selenium or you can use anything else for driving your tests UI or API tests.

Gauge is now out of beta. If you want a readable non-BDD approach to automation you might want to give Gauge a try.

6. RedwoodHQ

RedwoodHQ takes a little bit of a different approach from the other tools on this list. It creates a website interface that allows multiple testers to work together and run their tests from one web-accessible location.

Tests can be written in Java/Groovy, Python and C# code to test web applications with Selenium, APIs or databases using their web IDE, which also includes built-in GIT integration.

You can leverage RedwoodHQ’s existing action keywords to get started creating and modifying tests quickly. To create a test script you just need to find the action you want to perform, drag it into your test case and enter in the parameter values it expects.

The built-in IDE also allows you to create and modify test cases and actions as well as the ability to execute your tests. You can also run your tests against predefined machines in parallel and see a running history of all your previous tests runs.

RedwoodHQ is tester friendly and has a lot of built-in features that allow you to create complete test automation and test management solutions for your teams.

Cypress Automation Testing Framework

7. Cypress.io

Compared to most of the other tools on this list, Cypress is a more developer-centric test automation framework that focuses on making test-driven development (TDD) a reality for developers. One of its design principles was to be able to package and bundle everything together to make the entire end-to-end testing experience pleasant and simple.

It has a different architecture than Selenium; while Selenium WebDriver runs remotely outside the browser, Cypress runs inside of it.

This approach delivers more consistent results since it’s able to understand everything that happens inside and outside the browser. It also gives you native access to every object without requiring you to deal with object serialization or over-the-wire protocols. You’re pulling your application into Cypress.

As a result, it can synchronously notify you of every single thing that happens inside the browser so that you have native access to every distributed object model (DOM) element.

Cypress also makes it easy to simply drop a debugger into your application, which in turn makes it easier to use the developer tools while you’re developing. Of all the automation testing framework tools listed here, Cypress is the one that I hear most developers prefer to use.

Katalon Mobile Automation Frameworks

8. Katalon Studio

Katalon Studio to a free automation testing tool developed by KMS technologies.

It is not open source but it is free, and this tool can be used to automate on front-end web applications, mobile and web services. So in mobile automation testing frameworks, this might be a good choice for you.

The great thing about this tool is it’s a complete package and framework. So you don’t have to download or install any other components when working with it. All you need to do is download the zip package, and then you unzip it. Start the executer, and you have a very clean and good-looking GUI that you can use to start creating automation test.

Another benefit of using this testing tool is the learning curve is very small, and you don’t need any programming skills to start. You also can use it to create a keyword test automation framework which is popular with some QA teams. If you don’t know a keyword-driven approach means that you write a test not with scripting (using a scripting language like JavaScript or any other scripting language), but as a sequence of keywords.

So testers who have no coding experience can, download it and start within 10 minutes to create automation tests cases.

9. TestCafe

TestCafe is an opensource tool built on Node.js by DevExpress.

If you work for a large enterprise and need to support customers that use older browsers, you should definitely give TestCafe a try.

Just like Selenium, it’s free and open source but you don’t need to install any other software or web drivers.

10. White Framework

Are you a QTP/UFT engineer who has wished there was a way to automate .NET and WPF applications using Visual Studio and C# rather than QuickTest Professional or Unified Functional Testing?

Do you need to automate a thick client application like WPF, Silverlight, Win32 or WinForms? If so, then White may be the automation framework for you.

White is an open-source library from Test Stack. Like CodedUI, it has a more narrow scope; it only targets desktop applications, not web applications. Both the Coded UI test automation framework and White are based on UI Automation Library for Microsoft.

There are not many automation testing tools that handle .NET applications so this is a valuable tool for folks that have to test such technologies. It’s also easy to get started using White.

Selenium IDE icon

11. Selenium IDE

OK, many folks would not consider Selenium IDE an “automation framework” but with the release of the latest version in contains many testing functionalities that you normally would need a test framework for.

For example, it now handles being able to run your scripts in parallel execution mode. It was also designed to run on modern continuous integration and continuous deployment environments. So you can easily pass it all kinds of capabilities from the command line. IN essences it takes care of a lot of things for you out of the box that usually required that you use a framework for.

See my hands-on video demo of all the latest feature in my Selenium IDE Getting Started Guide.

Automation Testing Framework

What Open Source Automation Testing Framework Should You Choose?

The preceding automated testing framework open source solutions are just a small sample of the automation testing framework types you have available to you.

What automation testing framework should you choose? There is no right answer. Its whatever one works for you and your team.

But before you write that first line of code, make sure there isn’t a library or framework you can leverage.

Stop wasting your time reinventing the wheel and check out one of these automation testing frameworks first before investing any more time and effort in creating your own framework from scratch.

Open Source Frameworks for API and Performance Testing

Are you looking to automate some no GUI based tests?

If so check out my other post for API, Performance and Visual Validation testing tools:

42comments
Ady - May 11, 2016

Every-time I read your posts I learn something new and this time it was a newbie Gauge.
Can I say that you are my role model!! Whenever someone asks me what keeps you going? I say It’s someone like Joe with all the passion and knowledge for test automation.

Keep up the good work!

Cheers,
Ady

Reply
Jörg Sievers - May 11, 2016

Hi.

“I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. However, it also comes in a pro version that lots of folks rave about.”

SAHI (Pro) is not only for “simple HTML applications” because it is possible to use the new HTML5 technologies like ShadowDOM and custom HTML. You are right that the record won’t help to create good automation tests but it is a nice start where you can select alternative detectors (!) and also add asserts you want to take care of. The best advantage of SAHI is the handling of AJAX because you can write on top of very complex pages test scripts without any timeout-patterns you need e.g. for Selenium.

We have saved 50% time in creating test cases/user scenario tests vs. Selenium and the test execution results on the CIS are very reobust.

Yours,
Jogi

Reply
    Joe Colantonio - May 12, 2016

    HI Jogi – thats good to know! I will try to get a pro license and really give it a proper eval

    Reply
SreeCharan Shroff - May 12, 2016

http://www.extensivetesting.org/
is also one such open source test automation framework

Reply
    Joe Colantonio - May 12, 2016

    Thanks SreeCharan! No, actually I’ve never heard of it. I will check it out.

    Reply
Punee Varma - May 12, 2016

Hi Joe,

Were you a developer before you started doing testing? I have been doing testing for a long time but I don’t have the developer background. What open source tool do you suggest I should learn without knowing too much about programming. I have been using HP UFT for a while. I am trying to get away from this tool due not enough job out there like open source does.
Thanks

Reply
    Joe Colantonio - May 12, 2016

    Actually no – I didn’t have a developer background. I went to school for music :) I had a friend who got me a job doing tech support for Microsoft Visual Studio even thought I knew nothing about programming. I learned on the job. I would check out learning Selenium with Python. TO me that would be the easiest to start with.

    Reply
Divya - May 12, 2016

Nice Learning, Never heard about Gauge and RedwoodHQ, new learning , will try and explore them for sure.
Its always been a great experience working with you Joe, also learning new things from your Blog.

Please continue to share many more things about automation

Cheers,
Divya

Reply
Mahesh - May 12, 2016

I am very pleased to see this blog. I can sleep today with the feeling that I learnt something new today :-)

Thanks for sharing !!!

Thanking you,
Mahesh

Reply
    Joe Colantonio - May 12, 2016

    Thanks Mahesh!

    Reply
Hari - May 12, 2016

Great information, Appreciate your efforts Joe.

Thank You,
Hari

Reply
saunak tarafdar - May 12, 2016

Great Article! I have worked with Robot Framework and I am in the middle of learning Galen and Gauge. Have you had a chance to explore Open2Test Framework?

Reply
Ram Malapati - May 12, 2016

Wonderful Article very informative. keep up the good work Joe.

I’m just wondering which framework is more suitable for WebServices RestFul API’s Automation in a .Net Stack based environment.

We’re use Postman plug-in but eager to know your thoughts on Web Services Automation frameworks.

Please let me know if there are any BDD kind of frameworks that can be helpful for us to write tests in readable format.

Reply
Sanjay Jain - May 13, 2016

Thanks for sharing these information Joe. Your every post have refreshed the concepts and have added the information on how it works. This post will help me to cut down the scope of evaluation for my current project.

Thanks a lot..! Good Work and Happy to be learn from your posts.

Regards,
Sanjay

Reply
Kedar - May 13, 2016

Joe, your all posts and articles and the podcasts are full of awesomeness… Wish to work with a person like you…I watch all your Podcasts and they add a lot to my knowledge

Joe the CONTRIBUTOR

Reply
    Joe Colantonio - May 13, 2016

    Appreciate that Kedar – thanks!

    Reply
Testing Bits – 5/8/16 – 5/14/16 | Testing Curator Blog - May 15, 2016

[…] 6 Open Source Test Automation Frameworks You Need to Know – Joe Colantonio – https://www.joecolantonio.com/2016/05/10/6-open-source-test-automation-frameworks-need-know/ […]

Reply
Compelling Sunday – 16 Posts on Programming and QA - May 15, 2016

[…] 6 Open Source Test Automation Frameworks You Need to Know – Joe Colantonio […]

Reply
Radek Bokuvka - May 16, 2016

I do like reading your articles Joe, and this is one of them. There is always something to learn!

Regards,
Radek

Reply
    Joe Colantonio - May 19, 2016

    Thanks Radek!

    Reply
GAGANDEEP SINGH - May 18, 2016

Nice article Joe!!!
Thanks for sharing the info….

Reply
    Joe Colantonio - May 19, 2016

    Awesome – glad it helped!

    Reply
Harshal - May 19, 2016

Joe, suggest to evaluate Open2Test. I have found it very good.

Reply
    Joe Colantonio - May 19, 2016

    Hi Harshal – I’ve never heard of it. Thanks for the suggestion I will check it out! Do you use it for Selenim of QTP? Looks like it supports both

    Reply
HariCharan - May 19, 2016

Hi Joe,

Nice Article Man! Did you get a chance to evaluate Tellurium?

https://www.linkedin.com/pulse/tellurium-gonna-new-era-test-automation-hari-charan?articleId=6136584880018030592

Any Comments on it?

Reply
    Joe Colantonio - May 26, 2016

    Thanks! No I have not taken a look at Tellurium yet. It is on my list :)

    Reply
Tej - May 24, 2016

Hi Joe,

I love your posts! Always get to learn something new.
I did a POC on Serenity and i have become a fan of that framework. I am implementing this in my current company for a project!!

It is so easy to use as well as the reporting is awesome, supports BDD as well as Junit style of tests. Parallel Testing ,DataDriven testing can also be achieved without any issues.

Regards
Tej

Reply
Aastha - June 7, 2016

Hi Joe,

Your articles are so informative.I have recommended few of my friends to read your awesome articles. Thanks!

I am currently using BDD framework along with Jersey to test Webservices.I tried hands on Rest Assured but decided to use Jersey as developers are using the same library in my project.

Reply
Sojan - December 29, 2016

Great Article! I am planning to learn Gauge. How can i automate UI and API using gauge?
can you help me in this?

Regards,
Sojan

Reply
Dan - January 18, 2017

Hi Joe,

Would you use Serenity for testing REST APIs too? If not, what automation framework would you recommend for this?

Daniel

Reply
    Joe Colantonio - January 18, 2017

    Hi Dan use – Serenity also integrates with Rest-Assured for REST API testing. IN fact it would work for any java library that you want to use for testing your APIs since its a framework not a test tool you can use any valid Java library.

    Reply
Chris Brown - January 30, 2017

I’ve used Sahi OS and Sahi Pro a fair bit during evaluation. There is quite a few differences between the two.
Compared to other commercial testing software I found Sahi to be much better due to its focus on scripting, interwoven with JS so it simple to learn and if anything is lacking you just write JS code to do the task.
It uses relational elements to find exactly what you want which is great for dynamic web pages.

Reply
Rajesh Shinde - February 28, 2017

Hi Joe,

Learned lot from this post. In my company we are evaluating different tools/frameworks for our upcoming framework. Our project involved mainly into the micro services which were developed in Java. We are going to test the API’s as well as the UI. We also wants to look after the service log file processing based on the various operations via API. Since different services were owned by different teams within our company we also needed to think about the adaptability as well as the ease of use by non-coder folks. In our last project we used TestNG but I may be thinking of using the ROBOT.

Could you please guide me about the pros & cons of both of these frameworks? Is the Gauge or SAHI are having any new features that the TestNG and ROBOT does not have?

Regards,
Rajesh Shinde

Reply
Pritesh - June 21, 2017

Can we create dynamic script using Robot Framework?

Reply
Seth Khon - October 3, 2017

Is there anything wrong with MS Test Framework, which comes with Visual Studio 2017?

Reply
    Joe Colantonio - October 4, 2017

    Not if it works for you or your team :) Its not on this list because its not “open source”

    Reply
Clayton - January 26, 2018

Robot framework can be used with any .net language (e.g c#) not just iron python!

Reply
    Joe Colantonio - January 26, 2018

    Thanks for pointing that out Clayton!! My bad :)

    Reply
Joe Hennessey - May 10, 2018

Joe,

Although not nearly as heavily used as for Web testing, Serenity also works well with Appium for mobile testing.

jph

Reply
Vivek - June 18, 2018

Hello Joe,
Does any of these frameworks supports AI & machine learning features

Reply
Suveer P - October 3, 2018

Hey Joe/team,

Thanks for taking the time in writing this up. Its great to see people being inspired by your work and efforts.

Joe, a lot of the tools here are focused on Web apps. I have been put into a situation where I need to test a windows app and man, am I finding it difficult to find a framework that works.

Now I’ve trialed Winium but its not mature enough, Auto-It is a no go because I don’t want my team to learn a language they will most likely not use again and Sikuli is way to high maintenance.

In your experience, have you worked with any windows app testing tools that you truly believe work, consistent and stable?

I appreciate your feedback.
Regards,
SP

Reply
Click here to add a comment

Leave a comment: