Selenium: Interview with Dima Kovalenko
I’ve gone into the vaults of some of my earlier, more popular TestTalks podcast episodes that didn’t have a transcript available at the time of publishing them.
#Selenium Nothing is more soul crushing than having a test suite that just fails randomly~@dimacus #TestTalksClick to tweet
Check it out:
Joe: Hey, Dima. Welcome to Test Talks.
Dima: Hello. Nice to be here.
Joe: It’s great to finally have you on the show and today I’d like to talk about your awesome new book Selenium Design Patterns and Best Practices but before we do, can you tell us a little bit more about yourself?
Dima: I started working in software development and software testing back in 2003 for a small, back then, company called Rosetta Stone and basically I had no formal education whatsoever, no classes in college or anything like that. Everything I’ve learned about software testing and development has been slowly acquired over time through self-education and just having really good coworkers who would spend the time and teach me. And that’s basically how I started.
And then I am currently working at Groupon. Part of the reason that I am enjoying working here is they’re allowing me to spend some time actually to work with Selenium and contribute back to the project itself so the problems that I find, which are making it difficult for me to accomplish something can be fixed in the code itself instead of just making a blog post so that makes me very happy.
I’ve been going to several Selenium conferences. I really enjoy going to those and meeting new people. I’ll be honest, first couple of times were completely scary but as you meet more and more people who are kind of excited about the project and about testing in general, it’s kind of fun to find like-minded people. So to me it’s a huge treat. It’s something I look forward to every year is to go and hang out with the people you see online and who write blogs and once in a while chat with them on Skype or something.
Joe: So why did you feel a need to write the “Selenium Design Patterns and Best Practices” book?
Dima: One of them was actually just a bucket list thing to do. I always wanted to write a book. I didn’t know what it was about. I was kicking around the idea ever since basically 2003 when I first started. As to what I am learning and as I am kind of picked up and solves problems that, in manual testing, I always had a monolog in the background going would this, should be a chapter about testing and the databases and this should be a chapter on testing this stuff and just putting down onto paper what I’ve learned. It’s kind of helped me also to arrange some of the ideas that I thinking about by putting them on paper.
So to me it was a two-fold thing. I really wanted to share but at the same time I wanted to crystallize some of the ideas that I’ve been kicking around that way. They can be kind of appreciated by me where it’s like, “Oh, yeah. That’s why I always believe in this and this is why I always do it in this way and it’s not just making up things as I go.”
Joe: Awesome. Yeah, I agree. For some reason just writing helps sometimes to clarify what we believe. I really enjoyed this book. I love the flow of it. And starting with the first chapter when you say, “Today is your first day on the job,” so you really ease in. You really go chapter by chapter and build up on a foundation. I think that’s going to be really, really a great resource for someone that’s starting new with testing in Selenium.
Dima: Absolutely. And that was my intention. Personally, my style of learning tends to be hands on. That’s the best way. I can go to class and read all the textbooks for days and days at a time but, honestly, I don’t really learn anything from it. But for me personally, just having a hands-on, real world experience carry you through and as you learn something, the challenge becomes more and more complicated. That is basically how I learn and I figured if I can put that down on paper, maybe somebody else who learns the same way that I do can take advantage of it.
Joe: Awesome. Yeah, I was reading some of the book reviews and one person, they rated the book very highly but they were disappointed that it was not in Java. But it really doesn’t matter because the principles you’re teaching here apply to anything. You’re just using Ruby as the way to teach people these principles but you could still apply it. I use Java, C#, same thing applies. So why did you choose Ruby out of all the language bindings to use?
Dima: My direction is that there are job examples. The website that is used for the testing under the code section, I did provide a link to a GitHub account where there’s examples both in Ruby and Java and I’m hoping as I learn new languages, maybe C# or Python, to also translate those examples into a given language or if somebody requests a new language such as Node or some other new language. I would like to actually translate the same examples into that language but put it in such a way of where it is easy to follow with the book but now, instead of looking at Ruby example, you are looking at Java example.
But part of the biggest reasons I did use Ruby instead of Java is because I just finished learning Java and I’m still not that good. Ruby is a lot more comfortable to me.
And on top of that, I honestly believe that Java might not be the best first learning language for people since majority of the people who are starting off with the automation tests are not necessarily developers to begin with. They are manual QA people, like myself. To just jump into Java is actually quite hectic hurdle to get over. So Ruby reads more like pseudo code to me and to a lot of people I know and after talking to them about it they seem to agree that maybe Ruby would be a little bit easier to read and follow through. And then have a similar example on a side in that language of choice.
Joe: I definitely agree. If I have my choice I would definitely use Ruby or Python but for some reason the companies I work for usually want you to use the same language that the developers use and the developers end up not doing automation anyway so it always confuses me.
Dima: I guess the other part it though is that the ideas and how you approach on the problem does not really matter what language you do it in because one of the great things about Selenium is that the APIs are pretty robust and similar, that pretty much every binding you choose will have similar click commands, similar type commands, how you find elements on the page is quite identical between all the different languages.
So it doesn’t really matter what language you’re reading with as long as you figure out that hey, this is how you will find the element on the page. You can easily translate it into any other programming language.
Joe: So in you book you actually show a side by side example of the language followed by the command to show how similar most of the language bindings syntax are. It’s pretty cool.
Dima: Yes, and that’s what draws me in to the whole idea of Selenium is that I can go and switch and get a job working for a .net shop and still be able to get up to speed and write Selenium relatively fast without having to learn a lot of .net first.
Joe: I know. That’s such a great point. One of the main questions I always get asked is, “I am trying to learn Selenium. What language should I use?” I’m like, “Pick any language you’re comfortable with and go with it because what you’re going to learn will apply to all the other language bindings and, in general, all the automation principles. So just go for it and start learning whatever you’re comfortable with now.”
Dima: Absolutely. It is very similar to a question that I hear a lot is what continuous integration system to use? Well it doesn’t really matter. Just pick one. If you don’t want to pick one then just write a simple shell script that runs every two hours. That’s still better than nothing and then you can kind of go any way you want afterwards.
Joe: Yeah. I think maybe because people are engineers they might over analyze but not actually get to the doing so I definitely agree. Just pick something and go with it. You can always change later but you’re learning and what you’re putting in place can only help you rather than hurt you, I think.
Dima: Absolutely. And just ideas, I guess. If we, there are problems that we tend to get bogged down too much in the details of the implementation and miss the forest for the tree. Is that how that? Being English second language all the idioms kind of go sideways for me.
Joe: So bringing up nationalities, of course being Italian, the first pan that caught my eye is the spaghetti AntiPattern. What is a spaghetti AntiPattern?
Dina: I think we should first talk what is an AntiPattern, which is basically the short answer to it is something that makes sense and seems like a good idea at the time but turns out to be a horrible, horrible idea later down the road.
So that’s an AntiPattern. Spaghetti code AntiPattern, it’s a very common AntiPattern used in software development where the code itself is kind of spread out and tied to each other in very weird and unexpected ways. Where some tests may be using some method calls inside of another test that is unrelated to the first one and that you have a strand of dependency that is looping all through your test suite.
Then if you step back and look at the whole test suite, all you see is these dependency strands that go all over the place and it is very difficult to make any sense of it because there’s no logical division into hey, this is for the registration page and this is for the purchase page and this is for, there’s no separation like that. It’s just all over the place. A very, very common example of a spaghetti code would be when people first start to learn to use Selenium you go into the Selenium IDE and you record a couple of test runs. You record a walk through the purchase flow and then you walk through the registration flow. And between those two there’s a lot of similar code and some of the code may be unrelated but it’s just a single line going together. Those two recorded tests are put next to each other.
There’s a lot of duplication between the two of them but since they are separate tests and there’s no logical framework, you do not see that duplication and you do not see the interdependencies right away. You just see one long strand of actions.
Joe: So it makes it hard debug basically, right? So if you had a dependency and a script and you don’t know about it, something’s hardcoded and you run it against a different environment, most likely it’s not going to run and it’s going to be hard to maintain.
Dima: Absolutely. It’s hard to debug. It’s hard to maintain. Recording and playing back a test is a very, very quick way to get started but when you have a bunch of tests that are just strings of individual actions.
What ends up happening is that every single time something changes on the website, all of the tests that are affected by that change are broken. So instead of having a registration action in one single place it is spread all over the test suite and now you have 10 or 20 tests that are breaking for no apparent reason and now you have to go and fix those 20.
Joe: If someone is using the Selenium IDE or they’re starting off brand new, would you recommend that they don’t use Selenium IDE and just go in and start using the language bindings directly?
So, if you’re new to Selenium, it’s a great way for you to record a quick test to show to your boss hey, look. We can run a test very quickly and show some examples to the manager and kind of convince them and push them into the direction of increasing the automated test suite size but if you plan to keep using it for the rest of your career, you’re going to hit a dead end really quickly because of inherited limitations that are built in to the Selenium IDE. So it’s good to try and then switch over to something more robust such as Java or Ruby.
Joe: Awesome. I definitely agree. Do you have any tips on how to refactor Selenium test code?
Dima: As a matter of fact, after I finished writing the book I realized that the whole book is just nothing but refactoring. Every single chapter begins with refactoring the previous chapter.
So to start talking about refactoring of the code, one of the biggest books that influenced me was the “Design Patterns: Elements of Reusable Object-Oriented Software” by Erich Gamma and that book talked about just looking back at any piece of software and breaking it into smaller and smaller parts and then building up the software, like a Lego house. In a similar way, when we’re starting to refactor a spaghetti pattern, we can take out and basically start off by removing all the duplicate functionality and grouping it into a single place. You have a couple of tests that use a method to fill out the registration form, well let’s move that out into a single method and have each of the tests call the method instead of having instance that code themselves.
Similarly speaking, once that is completed and once you’ve removed all the duplication, we can move on to improving the stability of the tests by grouping all of the methods that we pulled out into more sectioned situations. Thus you can go it for page object pattern where every single page is represented as a test code and it’s mimicked in the test and then individual parts of the page are stored in the single place as little modules that tests kind of builds together as a little Lego house and you have a whole page out of it.
Joe: Yeah, absolutely. And actually, just so the people know that, you have a whole chapter dedicated to refactoring tests in your book, Chapter Three. That goes over all the different principles and the different patterns and it’s really helpful. So we have our test, we refactor our test, now we have a bunch of tests. How do we manage that so it doesn’t grow into this monster test speed of thousands of tests? How do we know what to test, how to prune our test suite? How do we keep a manageable test speed I guess is what I am asking?
Dima: Eventually, no matter how controlled you grow your test suite, it will become a monster test suite. And that’s not necessarily a bad thing. More tests is always better than less tests unless those tests are flaky and unreliable but more good tests is always better than fewer good tests. But my approach in the past has been to try and cover everything in regression by adding a new test for every single regression action. However, my views have changed a little bit because that does turn into unmanageable situation where instead of testing new code, you’re spending all of your time writing a test for code that has not been touched in years. So my approach nowadays is to grow with the new features so as the new feature is being added to write a test that follows that feature and each new feature makes the previous test that you just wrote, it turns it into a regression automatically so you’re regression suite automatically grows.
My favorite approach of course is to attack the money path. That is a way the customer can give money to your company. You should have enough tests to prove that at no point the customer is not able to give you money. They will be able to forgive you if the upload picture button is broken or if the colors are wrong or if something doesn’t look good. But if they are not able to purchase the product that they want, your customers will get frustrated so that is always the first thing to attack. To make sure that all of the ways that increase the revenue are covered and always tested.
The next approach is to have your test suite grow with the features set so every time a new feature is added we write a new test for it. And next, I like to add a test for every new bug discovered to make sure that that bug does not recur when somebody accidentally reverts some code from last week or does a bad merge.
So having a test for every bug you find and then making sure that the fix is there. Between these three, that will cover majority of your free time unless you are in a situation where you have a lot of testers on your team and you can just spend time writing any other test you want. If you can cover those three approaches, the money path, the new features, and the bugs, your regression suite will slowly grow and grow and increase in size and usefulness. Few have resources to write tests for something in the past. Hey, good for you but maybe improving the existing test suite and making sure that your tests are more stable and more reliable and don’t give you any false negatives or false positives is a much better investment than trying to write regression tests for code that has not been changed in a while.
Joe: Awesome. Yeah, I definitely agree. Follow the money. If there’s a bug and there’s not a test to cover that, create it and if there’s a new feature, if you follow those three, I think your test suite would be in good shape. I guess what happens now though is they have a ton of tests.
Do you have any tips on how to manage your test environment? Because sometimes they get bogged down in setting up VNs and it’s almost like, rather than testing I get stuck doing the admin part of it. So do you have any tips around that?
Dima: That’s the million dollar question, isn’t it? I always felt that software testers are very perfect people to be in the DevOps field because we care a lot about the test environment and we care about stability of the test environment whereas the developers, they only want to write the code and send it out. Sysadmins, they only care about production so test environments, in general they tend to be a little bit neglected by every side.
Naturally, whether you are a manual tester or an automated tester, managing the test environment becomes a very high priority. That’s part of the reason I went and learned all the sysadmin skills I could find just to keep my staging instance up and running so that my team can keep testing and not have to run out every time the database needs to be rebuilt or new code needs to be deployed.
So there are multiple ways to test with the automation and it really, really depends on what environment you’re trying to test on. And based on that, the amount of control over that environment will come into hand. If you’re trying to test against production, for example, you have the least amount of control over the environment and over the data on that environment compared to testing it on your laptop.
My favorite approach is to have the tests run on a node in continuous integration cluster that is dedicated to, basically, when a developer pushes a change to your version control a node in Jenkins picks up that job and deploys the website locally and it sets up the database to have local test data. And then run your first suite against that.
And once that is passed, we can deploy the new build to step up to the next environment whether it is integration environment or staging environment and then have another run hopefully with the same set of tests against that environment. And that way, by running it first locally, you increase a lot of unpredictable things that happen in staging environments such as third party dependencies, integration with Facebook or Twitter, all these other things we can cut those out and just run the test against the new build of our application.
When we’re moving on to integration environment or staging environment, then we end up testing how the application works with all the other people and toys that are in our toolbox, I guess. So my approach is always start locally in continuous integration and then deploy to a higher end server.
And managing the test data, which is the next painful, painful product, becomes more complicated when you’re going from a local environment to a staging because with the local environment you can always tweak the data very quickly. Hey, these fixtures will have this user so I know which user to login with. So let’s talk real quick about testing locally on one of the Jenkins nodes. I use Vmware. It’s one of my favorite things to do.
At Groupon were using Docker right now and what that allows us to do is for every test we can spin up an environment, a test environment on the fly, test against it and then destroy that virtual machine and replace it with a brand new one within seconds. This way, every single time our test runs, we do not have problems with the database. We don’t have any problems with any test solution or anything like that.
If you’re using Vmware, such as VMware Fusion, you can do something similar where you have a base image and spin up a test environment in there to have your tests have a pristine condition every single time.
Joe: That was my next question, data management because we have all these environments. We have a local environment. We have a staging environment. We have a, where I work we have a ton of different environments and one of the hardest thing is the data management piece.
So when I’m doing a code review, I’ll look at some of their code and they’re looking for an element with an ID that’s only specific for that environment. So if it was to run in production, it wouldn’t work. How do you combat that?
Dima: The easiest way is of course to do it locally where you have the access to the back end database and you know what to compare your tests against. But working with higher environments in the food chain that are closer and closer to production things do get much more difficult. Previously, I’ve had, I was in one project where it was a Java project and the test environment and integration environment, we did not have access to it. We had no ability to deploy to it. It was all done by the sysadmin group. But they did give us access to the JMX portal for that environment. That way we could, before the test would start, we could go and create all of the usernames and all the products that need to be added to the purchase flow. So having some sort of external setup condition before the tests run can be one of the most useful things you can do.
And as always, the closer you get to production the less right access you have to these environments. Then you start to rely more and more on ability to read what’s going on in a database. And so, one of the approaches you could take is, if your website has a restful API to expose us to the public.
For example, you have a website and you have an iPhone application. The iPhone application will go and talk to the API and ask for a list of available products. You can harness that same API and have your tests. Before the tests start to begin, go through and interrogate the API and ask for what products exist on this website.
If that user that you expect to be there exists and just collect that for the test and then hand them off to the test, either in XML file or some other way that the test says, “Okay. This is the state of the environment right now and so we should be able to pull up this product so let’s go pull it up and test with it.”
Joe: I think that’s a great tip. If you have restful APIs or a web service that’s exposed to you in your application, you should make use of it. And if it makes your job easier, there’s no reason why you can’t use the functionality of your application to interrogate for data before you run your test. I think that’s a great, great tip.
Dima: One other possibility is that if you have ability to deploy the data in the database every time you deploy to a different environment and have a subset of data that you just want to, with expected users and what not. This way you deploy, you build to stage and then you flush out the database and rebuild it from a known state and then test against a known state of the database. That will solve the data issues.
Hopefully, you’re not in a situation where you’re not allowed to do that as testers, as a quality assurance individual you’re not allowed to have access to a staging database and if you can’t control that, that’s a very, very tough position to be in.
Joe: Is there one thing you see over and over again, in your experience, that most people starting to create Selenium scripts or frameworks are doing wrong?
Dima: Absolutely. The first one is the most common problem that I see is the desire to have quantity over quality of the test. You don’t necessarily need to have a test that uploads your avatar image in your profile before you have a test that tests the registration of a user. And so, going for 20 tests that do trivial things is less useful than having one test that attacks the money path and makes sure that there’s no problems to your money path of the application. That is the biggest one.
The next one is trying to solve some instabilities by adding weights and sleeps and thread that sleeps to make sure that hey, the website is running a little bit slow on staging so let’s just have the test sleep for 10 seconds and then maybe it’ll be fine, instead of trying to do something more intelligent and, say, wait for the page to fully load, wait for all the Ajax requests to finish, and then go on with the next step. That is the second most common thing.
And the third one is the, this is what I am very guilty of is, early on I would want to just get things done. Yeah, let’s write the test. Let’s do it. Let’s go. Let’s go. And never really stepping back and looking at what am I trying to accomplish? Am I trying to build a framework that is stable or am I just trying to write 10 tests so I can say that my test suite is now 20 tests long?
Having some sort of background and understanding of how the applications grow and how to make them grow in a manageable way is a huge, huge improvement over just starting out and writing a bunch of tests.
Joe: Great advice. So your book also covers testing the behavior of an application using tools like Cucumber, using behavior-driven development.
So do you find the overhead of behavior-driven development, do you find it as overhead or do you find it useful? When do you think it is appropriate to use BDD?
Dima: Some of the BDD tools such as Cucumber do have a very large overhead. Compared to them, something smaller and simpler like RSpec has a much lower overhead. So it really depends on what are you trying to accomplish? I’ve noticed that RSpec, for example, has a behavior driven syntax but it is closer to what developers are used to seeing. It’s written as code and not as natural language.
Compared to Cucumber, which is much more natural language but has a higher overhead in the back end because all the language needs to be parsed and needs to be correlated to actual actions that the test needs to perform.
So depending on what you’re trying to accomplish, sometimes Cucumber is a great tool, sometimes RSpec is a great tool, sometimes no behavior-driven tool is a good option and let’s take a look at a couple of examples.
If I’m writing a very, very small application that will only need 20, 30 tests and that need to run just really quickly, there is no reason to use a behavior-driven tool whatsoever because you’re going to spend so much time setting up the framework in the first place just to write 20 tests for a little plug-in application that will be written once and probably will not be updated for the next couple of years. Versus if you’re trying to write a large shopping cart or airline booking system, a BDD tool can help you put things into a framework.
However, having a behavior-driven tool can help you write a framework for a much larger application because it forces you to think about the application in smaller pieces, not necessarily as a set of actions. Cucumber might be a very, very great tool if your team is non-developers who can look at the natural language and understand what the test is trying to accomplish right away.
If your team is going to be development heavy, some other tool such as RSpec might be a lot more useful for you because it’s more programming style instead of natural language style with all the natural language pitfalls that might happen. Which approach really depends on what the end goal is.
Joe: This is probably a whole other episode but I wanted to get in to your excellent video on scaling and managing Selenium Grid. Just a quick, can you give us a quick blurb, a quick shout-out about what your Selenium Grid Extra open source package is all about?
Dima: Yes. To those who are unfamiliar with Selenium Grid, the project tries to spread out the execution of the test onto multiple machines. That way you can have a test that can be running in Mac Safari at the same time Internet Explorer but just run it from a single place but on multiple machines. So the idea is to kind of spread out the load and what I found myself and many people who start to use Selenium Grid is that there are certain irregularities, especially when using Internet Explorer, older Internet Explorer, especially Internet Explorer 7 and 8, that make testing with the Grid very difficult.
Sometimes the browser will crash, sometimes it will take down the system, sometimes something unexpected will happen or some feature such as multi-tier browsing where you have a page with mixed content such as http and https on a single page.
Sometimes those things are just disabled and they will make your test fail without giving you a good explanation as to why your test is failing. At one point, I was trying to, I was writing little scripts to mitigate that if Internet Explorer crashes and there’s a pop-up that says Internet Explorer crashed. Well, if you don’t close that pop-up, the next test that runs will act unpredictably because that pop-up does influence the page.
So I starting off writing some scripts to close any of these pop-ups if they happened and after each test to make sure and close the Internet Explorer browser because if you don’t close it, they will start to stack up and eat up all of your machine’s memory, making the 20th test down the road crash with unbelievable and unexpected results. But I’ve noticed that all I had is 20 or 30 of these scripts in AutoIt but they were difficult to maintain and they did not really solve a lot of the problems, new problems that I kept on running in to.
So I got permission from my boss to start a new project that would combine a lot of those little scripts that improve stability into a single management tool called Selenium-Grid-Extras. The idea there was to start developing the features that improve stability. Features such as recording a video as the test is running and saving that video. And then hopefully, eventually try to merge those features back into the Selenium Grid itself. That hasn’t happened yet but that’s, down the road, the goal of the project. But the overall idea behind it is to kind of take away some of the complexities that come with managing the Grid. By default, all you have to do is just run Java JAR Selenium-Grid-Extras. If it’s the first time you’ve run the application, it will ask you a bunch of questions such as is this a Grid hub or a Grid node? Do you have Internet Explorer here or do you have Safari here?
Things like that and it would collect some of the data from you and based on that would configure the Grid nodes with the appropriate timeouts and the appropriate maximum instance numbers, etc. for that given browser.
That way a lot of that experience in trying to figure out instability is documented in code and not on a blog. So it’s something that is actively trying to help you instead of trying to jump around everywhere and debugging things that you cannot understand.
Joe: Awesome. So I’ll have a link to this video, Scaling and Managing Selenium Grid, which goes into detail about this approach and how the Selenium Grid extras can help. And I’ll also have a link to the book at www.testtalks.com/30. So Dima, before we go, is there one piece of actual advice you can give someone to improve their Selenium automation efforts? And let us know the best way to find or contact to.
Dima: Best advice I can give is to slow down and never stop learning what is a good development practice. As you improve your development skills, you will improve your test writing skills because those two are related. And treating the tests as code and not as a set of instructions that are quickly slapped together by far is the best way to prove your test stability and just improve, overall, healthiness in your test suite. So as I mentioned before, there are several good design pattern books out there. It does seem like a waste of time to learn about design patterns and learn about how to best approach solving problems as a programmer because we just want to say, “Hey, we’re not developers. We’re just quality assurance people. We’re just testers. We’re not developers so why do I have to learn any of that?”
And that feels to me that we, as a group, as software testers we tend to sell ourselves short in that way. There’s no reason we cannot be as good at writing codes as developers are. There’s no reason for us to kind of take the backseat and think that it is okay to write poorly written code that tests the website and then get confused and sad when the poorly written tests are failing and we’re having a hard time managing them.
Instead we should just, in general, always keep trying to improve yourself, always keep trying to get better at what you do. And when you do that, all of a sudden, your tests will become better, your programming skills will become better, and you will just honestly start to enjoy writing tests a lot more.
It’s nothing is more soul crushing than having a test suite that is all over the place and just fails randomly, but when you build the test suite that is actually logical and well written and any developer can look at and be impressed by, it actually makes you feel good about yourself and about the job you’re doing. And I think that’s, by far, just striving for that perfectionist is what will make you a happier software tester in general.
The best way to reach me, as always, is through Twitter. My handle is @dimacus. Always can ping me there. Tend to reach out and answer. Similarly, the best just good place to come out and ask questions about Selenium and especially if you have a very difficult technical issue to solve is RC. The link to the RC chat room can be found on the seleniumhq.org. It’s right there at the bottom.
You’ll find a lot of the good developers who use Selenium there all the time. You will find Jim and Jim Evans. You will find Simon and all the guys who actually wrote Selenium there. And they are really, really great people and any time you have a technical question that is difficult to solve and you’ve wasted already five days on it and you’re about to throw your computer out the window, that’s a good place to go and ask that question because majority of the time the answer will be given really, really fast to you.