Selenium is awesome/don’t use selenium

PerfectoTopAd

BDD and Selenium

My company's engineering center of excellence has begun to promote using Behavior Driven Development (BDD). The good thing is that most of the teams have embraced BDD as part of their development process; the bad part is that I've noticed that most of the BDD tests are UI tests using Selenium.

I won't go into a rant on how BDD is not an automation framework – my main point is that as automation engineer, I believe we should try to avoid automating the user interface as much as possible.

Selenium is awesome/don't use selenium

 This sounds like a Zen koan (a succinct paradoxical statement or question used as a meditation discipline for novices). Yes. Selenium is great and can handle many web browser automated tests, but like any GUI automation tool, it has the three GUI automation causes of suffering:

  • It's slow to run
  • Hard to maintain
  • And hard to make reliable.

The goal should be to reduce the amount of Selenium test you create not more.

Before using Selenium Ask Yourself Why

Before using Selenium you should ask yourself why. Is there a better, faster, more reliable test type that can be used?


Browser UI automation should not be the only type of testing you are doing. You should try to test more things in isolation rather than a full blown end to end UI browser test. Touch the UI as little as possible. Focus on unit and service/API level test first.

How Google Test's Software

Google's automation rule of thumb is: 70% should be unit tests, 20% should be service integration testing and only 10% should be full blown GUI automation.

If you can test a feature without having to render a browser, you should automate that first.


To learn more about how Google test software check out my post on Google Testing Secrets Reveled.

Automation Mantra

So if you are responsible for maintaining test automation scripts your new mantra should be: Selenium is awesome/don't use selenium <–tweet this. Funny, since this tip was offered by more than one of the main committers to Selenium (including Simon Stewart the creator of WebDriver and project lead) at the Selenium Conference in Boston this year!

4 comments
Alex - November 13, 2014

I agree. The state of Selenium as an open source project is abhorrent. It’s borderline abandon ware. Bugs stay around for half decades. The person responsible only cares about selling VMS right now (sauce labs founder). We are currently seeing that even writing jquery selectors from scratch as automation is more reliable than selenium in general. Every week we get false reports of a bug, when the selenium driver is just random and unrobust in general.

Reply
    Joe Colantonio - November 20, 2014

    Hi Alex unfortunately I think this is an issue with any test tool. I’ve been hearing that the browser vendors will soon be responsible for maintaining their own implementations of WebDriver so that should help get issues fixed quicker — I hope. Also as far as i know none of the Selenium contributors get paid for their work so I think they are doing a fantastic job with the resources that they have.

    Reply
Liraz Shay - August 9, 2016

I don’t agree.
“it has the three GUI automation causes of suffering:
1. It’s slow to run
2. Hard to maintain
3. And hard to make reliable.”

1. It’s slow to run
It’s much faster than manual tester.
manual tester must check the system through the UI
it is very reasonable to replace the manual tests with automated.

2. Hard to maintain
“Test code is code”
If one says “UI tests are Hard to maintain”
that means he doesn’t do it well,
Please read about PageObjects
http://martinfowler.com/bliki/PageObject.html

3. And hard to make reliable.
It’s true that Selenium throws exceptions
but if you learn about those exceptions – you can avoid them all
for example:
“NoSuchElementException” can be solved by ImplicitlyWait
“StaleElementReferenceException”,
“ElementNotVisible\ClickbleException”, etc
those exception can be solved by retrying to perform the action
TryClick()
or by explicitly wait

In conclusion – if one is experienced with Selenium —
his tests will be reliable.

Reply
    Joe Colantonio - August 18, 2016

    Thanks Liraz but I stand by my statement compared to low-level testing like unit, integration , rest , micro -services testing Selenium or another GUI based functional test tool is slow, hard to maintain and hard to make reliable. In my 16+ years experience and after interviewing over 100+ other engineers I have seen this to be true. In fact both the main contributors to Selenium itself Simon Stewart and Jim Evans agree that if you can create a lower level test that doesn’t use Selenium then you should’t use Selenium. If you are a one man team working on a simple application sure you could make it somewhat reliable but it is not 100% reliable. Also most folks I talk too that use Selenium uses Page Object and still have issues. Even if you get them reliable if your doing continuous integration running a suite of automated Selenium test takes way to long to give your developers the feedback they need.

    Reply
Click here to add a comment

Leave a comment: