SELENIUM SUCKS! QTP RULES!

Selenium sucks! QTP rules! Got your attention, didn’t I? How many times have you heard this type of statement from overzealous automation engineers? Truth be told, I usually hear statements like this most often made by open source evangelists.

Test tool to rule them all

The fact is, however, there’s no such thing as a perfect test tool. There is no “Test tool to rule them all.” In lieu of that, you have to use the tool that works best for your team and your particular application. And just because you use one tool for one specific project does not mean you should use it for all your projects. Many test tools require an investment in not just dollars but also in learning and infrastructure – so I understand how tempting it can be to stick with one tool solution for all your testing needs. If you have a hammer, everything looks like a nail.

About my application dilemma

The main application I test in my current position has many layers of proprietary control wrapped in an active x control and displayed in a web browser (IE to be exact; it doesn’t run under Firefox). The application uses a hodgepodge of non-open source technologies like .NET, CACHE DB, and MUMPS. On top of all this is a Frankenstein-like build system that I couldn’t even begin to explain the workings of.

QTP

My company also has an enterprise license for HP’s automation tools. So which tool do you think I choose to use? That’s right — QuickTest Pro. Why, might you ask? Well first, it’s the standard tool used throughout the company. Most test engineers are already familiar with it so there’s no need for training and it works (albeit with several modifications) with our application. Also — it doesn’t hurt that I’ve been using the Mercury/HP toolsets for many years.

Selenium

Some of the groups in my company are moving to AGILE. One group in particular is involved in a new application that uses the latest and greatest web2.0 technologies. Also — unlike my group, most of the development is done using open source technologies and uses Hudson for daily builds. This team’s application can run under different browsers and the application is fairly straight forward. So in this case I would think a test tool like Selenium would be a better fit than QTP.

So what’s your point, Joe?

The point of my rant is that we should all have an allegiance to a particular test tool. As automation engineers, we should be focused on developing the quickest method with which to create reliable automation solutions that serve a specific business need!

Don’t be a test tool fanboy

This leads me to my last point. You should not align yourself with one company’s tool. Let me tell you a quick story. Long ago, in a cubicle far, far away lived a Win Runner automation engineer. How he loved the crisp white screen of a new Winrunner test! The engineer spoke the language of TSL. There was no business problem that he couldn’t solve with the right Winrunner script. Then, one day, Winrunner was discontinued. No longer would his favorite test tool get updates. Job postings for Winrunner experts became fewer and fewer. The engineer was heartbroken. Don’t become like him! Keep learning. Use the right tool for the job. Never become a Selenium or QTP (or whatever company’s technology) fanboy.

Agree/Disagree

So what do you think? Am I wrong? Leave a comment below and let me know!

Leave a comment:

125 comments
Anish - January 28, 2012

Completely Agree.. :)

Reply
    Joe Colantonio - January 28, 2012

    Anish » Thanks Anish – glad I’m not the only one :)

    Reply
Oriol Torres - January 31, 2012

+++++ 1 :)

Reply
saravana - February 1, 2012

Even in the Agile scenario QTP Rocks ! .. We also usong Hudson and we’re complete opensource (Development) group but since QTP supports all the challenges that we met ..

Reply
    Joe Colantonio - February 1, 2012

    saravana » Great to know. My company just started to use AGILE so I’m glad to hear that QTP worked for you in an Agile environment. For every build do you have Hudson run QTP scripts?

    Reply
arjun - February 1, 2012

So true….. As a tester quality is of at most priority and not the tool used.

Reply
    Joe Colantonio - February 1, 2012

    arjun » Well said Arjun!!

    Reply
    Omar Navarro - October 9, 2014

    I did run, however, into a project that used QTP because the licenses were readily available. However, due to some pretty nasty exceptions QTP was causing in the app (which crashed everytime you clicked on a button or requested an object property), we had to switch to other tools, and ended up with Visual Studio CUIT. You never expect to make such a swift change, but in the context of multiple custom object, layers and layers of ContentPresenters and above all, incompetent HP support (on this instance), you have to change.

    Reply
Kaniska - February 14, 2012

You are doing great…
thanks a lot…

Reply
    Joe Colantonio - February 14, 2012

    Kaniska » Thanks Kaniska!

    Reply
Aru - February 25, 2012

Considering the cost savings selenium definitely will have a great future and will over rule QTP in future for sure. Go to any job forums and you will see job postings on selenium more than QTP…. So i agree Joe, we will have to be really on par with pace of the automation trends.

Reply
    Joe Colantonio - February 26, 2012

    Aru » Good point Aru!

    Reply
      Saranya - September 11, 2013

      Hi Joe,
      I’m a ECE fresher wish to join software testing course. Having confusion in which to select, Selenium or QTP. please help me with that.

      Reply
        Joe Colantonio - September 11, 2013

        If I was starting all over again I would first learn Selenium with Java or Ruby. Make sure the training is not for Selenium IDE.

        Reply
          Saranya - September 16, 2013

          Hi Joe,
          Thanks. And the training is Selenium Java. Most of my friends are saying QTP, Really got confused which to choose.
          Thank u:)

          Reply
sam - February 27, 2012

Hi Joe,

My vote is to open source. i’m not fan of any specific tool but used wide varieties like rational, mercury/HP, WATiR, WATiN, Selenium, sahi etc. every tool has its advantages and drawbacks. Now a days after HP took QTP it lost its significance as i used QTP 6.5 in early days and i never had issues. Now i have some issues i posted in your forum still not resolved by HP SC. i’m exchanging emails everyday to get the problem solved. One of my friend in other group told me that HP is concentrating more on protecting unauthorized usage of QTP (licence areas  lol) than actual QTP functionality. I may not agree with it, but may be he is true with his exerience.
~Sam

Reply
    Joe Colantonio - February 27, 2012

    sam » Hi Sam – I’m with you, use the tools that works. I agree before HP took over I think Mercury was really just focused on creating killer test tools. With HP not so much. HP’s support is awful and that is suppose to be the big selling point of using a commercial tool rather then an open-source one. To bad Selenium is just for web applications.

    Reply
Hussain - May 23, 2012

wow… interesting article..! even my organization is also following complete agile mode of product development using open source technologies…still for automation, QTP is being used and 1000’s of test scripts are running in test CI :)

Reply
prasad - May 29, 2012

Thanks joe! & yours final choice

Reply
Julien - July 2, 2012

from my point of view, QTP is a poor choice since in 2012 it still use VBScript as the only available scripting language, which is kind of retarded if you ask me.
At least, Selenium let you choose your scripting language of choice.

Reply
nik - July 17, 2012

Hi joe.i am Completely new to automation testing.Wanted to know that ho the decision is made based on the application that which automation tool is to be used? To start with qtp would be good?Market has more jobs for automation than in manual testing ?

Reply
    Joe Colantonio - July 19, 2012

    nik » HI – If you are testing web based application than I would start with Selenium 2 WebDriver— it is free and I’ve been seeing more and more job posting that have it as a requirement. For testing web-services check out soapUI –which is also a free opens source tool.

    For other technologies like .NET,WPF etc.. I would try either HP’s UFT(QTP) or TestComplete.

    Before selecting any tool for a project I would recommend doing a proof of concept first to evaluate each option and choose the best one for your situation. There is no one tool that will be right for every company/project.

    Reply
Ravishankar - August 4, 2012

Hi Joe,

From past 3 years i m working on manual testing, i switched to automation, since i m very poor in coding, i found very difficult in learning selenium….But QTP was little easy….So if i continue in QTP, will there be a future for coming 10 years.. Please guide me.

Reply
    Joe Colantonio - August 6, 2012

    Ravishankar » The best way I found to stay current is to do a job search every few months and see what technologies are being listed as job requirements. I compile a list and in my spare time try to learn those skills. I personally have seen an increase in Selenium and Python as requirements for automation positions. Also I believe that all automation engineers need to know how to program and should know how to run/create performance tests. That’s just my opinion

    Reply
Shyam - September 5, 2012

I do Agree…. I love scripting in QTP

Reply
    Goutham - April 1, 2015

    I agree with you shaym..

    Reply
Neloy - September 8, 2012

My company also use QTP 11 and it is integrated with Hudson. It is working perfect.

Reply
Alex - September 24, 2012

Hi Joe,
Do you know how much license cost for QTP?
Thank you,
Alex

Reply
    Joe Colantonio - September 25, 2012

    Alex » Hi Alex – sorry I’m not sure. Its hard to tell since I think HP changes the prices based on the amount of licenses and what else QTP is bundled with.

    Reply
      Satya Mantena - February 8, 2013

      QTP Esstentials for web testing (includes HP QTP Essentials supports the following add-in environments:
      ActiveX, Java, Visual Basic, Web, .NET, WPF, and Silverlight. It also
      supports ASP, .NET, and AJAX through Web extensibility)
      and you can buy a seat license for $4K (approx)

      You can contact Checkpoint Technologies or any other HP Partner

      Good Luck
      -Satya Mantena

      Reply
    Vikas Kumar - October 5, 2012

    Hi Alex,

    Better you try opensource Selenium
    Selenium one of the popular software testing tools is a portable testing framework for web applications. This is an open source testing tool, developed by Jason Huggins and team and released under the Apache 2.0 license. Since it is an open source tool, it can be downloaded free of cost. Selenium provides a playback tool for authoring tests. Using the tool, the tester need not learn the test scripting language (Selenium IDE). Furthermore, it provides, a test domain specific language (DSL) Selenese that allows to write tests in various languages including C#, Java, Groovy, Perl, PHP, Python and Ruby. Then, these tests can be executed in many latest web browsers and works on everything including Windows, Linux, and Macintosh platforms.

    Reply
Dave P - September 25, 2012

I’ve been using QTP for 10 years, since version 6.5. I used to think it was the greatest thing since sliced bread. In the past couple years, my feelings have changed completely, especially since the HP buy-out of Mercury.

Although I’m no big fan of Selenium, its just too limited, I hate QTP! It is one of the most difficult to use, poorly designed, unstable, flaky and tempermental pieces of garbage software I’ve ever had the misfortune to use. Licensing is outrageously expensive for a piece of software that just doesn’t work. Our group is slowly phasing out QTP in favor of TestComplete. It can’t happen too quickly for me.

Reply
    Joe Colantonio - September 25, 2012

    Dave P » Hi Dave – Interesting we have an Enterprise license with HP, so the license is free for a group to use and we still have groups that go out and buy TestComplete instead. I really love soapUI by smartBear so I’m sure TestComplete is also a great tool. I agree – Mercury was awesome, HP not so much their to big to really focus on automation tools. Let us know how TestComplete works out for you. Cheers~Joe

    Reply
      Dave P - September 26, 2012

      Our software development team uses SoapUI to validate the structure of their webservice messaging. The way they have it set up is pretty slick. Hudson manages the build process, so whenever they check in code that touches any of the web service libraries, the SoapUI scripts run automatically to verify that they haven’t broken any of the messages.

      That works out really well for us in the QA department. I use TestComplete to generate web services messages from a data-driven library of test files. Then I walk through the application to validate the expected effects of those ws messages, e.g. creating new accounts, etc. The combination works out really well. I know that I don’t have to mess around with validating the structure of the message itself and can concentrate on deeper testing of the application.

      Smartbear has a way to go on some of their test management systems, but their support is awesome. They are very willing to listen to their customers and work to fulfill their needs.

      Reply
Alex - September 26, 2012

Thanks Joe for your responce!
I am in process of selecting automation tool for UI testing. As you know, every manager dreams to have test cases automated.
My team are not very eager to program. So, I am looking for a tool, which would automate but does no require heavy programming. Do you know of any?
Would be great to hear your recommendations!
Thank you,
Alex

Reply
    Dave P - September 26, 2012

    In my never to be humble opinion…

    I’ve been down the road a few times with different organizations trying to build automated testing without programming. Organizations seeking this goal are susceptible to the record-and-playback promises of almost every testing package out there. Record and playback is pure marketing BS. Any test script you build this way is nearly always obsolete within a revision or two. Its nearly impossible to modify the recorded scripts to match the application changes, so you have to record the scripts again. Eventually, you give up on automation entirely.

    Automated testing of any significance is going to require programming skills. Test scripts are software just like the application being tested. If you’re serious about test automation, you’re going to have to accept that fact. If you’re not willing to make that commitment, you will not have much success with test automation.

    Reply
    Jagath - October 16, 2012

    Alex,
    if you can tel me the technology stack and scope of your testing or testing activating which you want to automate , i can able to guide you the right tool.

    Reply
Alex - September 27, 2012

Thanks Dave!
Dave what do you currently use for automation?
Have you ever used ranorex?

Reply
    Dave P - September 27, 2012

    I’ve never used Ranorex, so I can’t tell you much there. I have some limited experience with Rational. Typical IBM software, overbloated and confusing. Our department is currently using a mix of Selenium and QTP. I haven’t worked with Selenium that much. It seems to work well enough, but it can only be used on web based apps. QTP is slowly being phased out and replaced by TestComplete. TestComplete is a far superior testing tool for several reasons:

    1. Multiple scripting language support, including JavaScript, C# script and VBScript. Object oriented languages are such an advantage in building test frameworks…

    2. Much more solid. I don’t have the problems with the IDE locking up during debugging like I do with QTP.

    3. Stronger support for distributed testing. The test engine is a separate module named TestExecute which can be installed on a remote machine to execute tests. It doesn’t require the complete IDE like QTP does.

    4. Substantially less expensive than QTP, about 2/3 the cost for the same level of functionality.

    5. Far better support. Lets be honest, HP bought Mercury so they could compete against IBM’s Rational. Now HP can say, “We have test automation too!” but beyond that, they don’t give a damn about the product line. True story: I once called HP’s support for a QTP question. The support tech literally asked me, and I quote, “QTP… is that one of our products?”

    That’s my two cents worth. Good luck on your automation efforts. No matter which system you ultimately choose, my advice would be:

    To begin with, write a few simple test scripts just to learn how the package works, its capabilities, idiosyncracies, etc. Whatever you develop at this early phase will likely all be thrown out once you become more familiar with how the test system works, so don’t spend a lot of time trying to build the ultimate testing system yet.

    Later, step back and figure out exactly what you expect to get from test automation. Plan it out like a software development project, because that’s exactly what it is. Then before you write any actual testing scripts, build a framework around your target application. The framework should be designed to handle things like launching the app, shutting it down, performing simple, commonly repeated validations, encapsulating common structures, etc.

    With your framework in place, thats when you can start writing real test scripts. If you follow this general plan, you will save yourself a lot of frustration and wasted effort.

    Once again, good luck.

    Reply
Jhenry - October 1, 2012

I started 5 years ago with Winrunner and hated it. When it was no longer supported we looked for a new tool and ended up with HP QC, QTP and BPT. The licensing cost is an issue but we found the framework to be the selling point. BPT makes writing tests fast, reuseable and an easy environment to maintain. The reporting out of QTP with its recording or screen shots an important tool for defect submission.

I cannot comment on Selenium but there are others in the company that want to go that route. And that is fine… but my tool of preference is the BPT world.

Reply
    Joe Colantonio - October 1, 2012

    Jhenry » Wow you are one of the few test engineers that I know that likes BPT :). I love the concept of BPT but I’ve had a hard time with its performance compared with QTP.

    Reply
Vikas Kumar - October 5, 2012

ThinkSys Inc’ team of Selenium and QTP experts have helped migrate many for our customers from QTP to Selenium in record time and at extremely low costs.

Using our unique, patent pending Selenium based framework KRAFT- Krypton Regression Automation Framework Testing, ThinkSys Inc has helped clients to create a low cost end-to-end automation solution for web sites and web based applications using Open Source tools.

Reply
Vikas Kumar - October 5, 2012

Helped a major Fortune 1000 client migrate over 8,000 test cases from QTP to Selenium in record time. This saved them hundreds of thousands of dollars in yearly costs and significantly reduced their QA cycle time. They now do multiple releases a week.

For a company recently gone public on Nasdaq (2012), using our KRAFT framework we created and automated about 1000 test cases in Selenium in 4 weeks.

Reply
    mark - November 13, 2012

    Your comments sounds like a sales pitch for a framework your company is intending to sell.

    Can you describe any problems with the KRAFT framework and what changes you made to resolve those problems?

    Reply
Shruti - October 15, 2012

Hi Joe,

gr8 article and comments… I am a manual tester with 6 years of experience and looking at current market i really feel that i should learn automation… but i am not sure how to start with…
Please guide, thanks in advance.

Reply
Jagath - October 16, 2012

Hi Joe – Completely agree on your thoughts.. I used to suggest my folks the same..
Also would like to thank you personally for a such wonderful job you are doing here and educating,suggesting,helping and motivating the guys who need your helping hands .. Great going .. keep up your same spirit always high…

Reply
srikanth - October 20, 2012

Hi Joe ,can you give suggestion for me to learn which tool is best QTP or Selenium

Reply
Mahboob Ali - October 23, 2012

Completely agree Joe:)

Reply
Denise - October 25, 2012

I agree with you Joe. In my job role, I actually have to used WinRunner, Selenium WebDriver, and QTP. They each have their advantages and disadvantages just as you said. We had a team that thought WATiN was going to solve everything and only to find out it was no longer being supported. They were frustrated and moved onto Selenium. They have put Selenium on a pedi stool although I’ve made the same argument that one is not superior over the other and could be discontinued at the drop of a hat. We live in ever evolving technology and I’m sure something “better” will come along within a year or two. In my 8 years of Automation Development, I’ve seen the end of life on two products that our company has consumed and I’m sure I will see the end of life on another before all is said and done.

Reply
Rob - November 10, 2012

I’ve used a variety of tools over the years but I started with WinRunner and then QTP. I have since used Webdriver, Concordion, FitNesse and Robot Framework/Selenium. And I’ve tried out many more in the course of finding the right tool for a particular audience. There is such a variety of tools available now, that it’s no longer a question of QTP vs Selenium, but QTP vs Selenium vs Telerik TS vs TestComplete vs Axe vs Ranorex vs Canoo etc etc. Determining the best tool for the job needs to take into account various factors such as who will be writing/running the tests, what languages the tool supports and whether they can be developed using a language and IDE that the development team is familiar with, how well the tool abstracts the intent of the test from its implementation etc. Things were certainly a lot easier when there were fewer tools to choose from but, as you say, limiting yourself to one tool is a quick route to redundancy.

Reply
kiran - January 5, 2013

Hi which tool gives more opportunity for freshers qtp or selinium. Please guide me

Reply
    Joe Colantonio - January 7, 2013

    kiran » I’d say start with Selenium. Every job post I’ve seen seems to have Selenium experience as a requirement.

    Reply
Vigneshwaran - January 17, 2013

Great Article Joe.

Reply
Kamal - January 17, 2013

Its a good debate on Automation Tool. Very Informative :)

Reply
Piyush - February 6, 2013

Hi Joe, I completely agree that there is no “best tool” as such, you have to select the one best suited to your requirement. I want to add to the Selenium v/s QTP debate. We were asked to do a small comparative report on tools in our project and we found some interesting insights:
a) Selenium out-of-the-box has very limited capabilities, being limited to web-testing and Selenese. But after integrating with opensource test frameworks and Java packs, its capabilities are unlimited. One of our development teams who are venturing into automated unit testing profess that Selenium supports Java and Java can do anything. I am a QTP guy myself (so far) but find that hard to contest. I cant say that about QTP, though of course QTP can be used to drive JARs.
b) One advantage of QTP over Selenium and other open-source packs is that the mundane automation tasks – reading test data from excels, maintaing test objects, identifying objects on the UI – are implemented as one-liner standard methods which are mature and bug-free. This means
b1 – you can get on with thinking about your test algorithm.
b2 – when debugging a failed test you can quickly check these and move on to checking the actual algorithm.
b3 – most QTP scripts are readable; you can take-over another person’s automation scripts with little or no handover process if you are reasonably experienced.
With Selenium or other open-source packs, your skill with the tool gets you only so far when reading other’s scripts as you need to learn how every little task was implemented.

Reply
Satya Mantena - February 8, 2013

Frist off great discussion and lot of good valid respected opinions
Funny analogy – So please dont take it personally
The ones that are Pro QTP are analogous to “the married and have decided to settle down”
Pro others are analogous to “either single or divorced, not literally, but similar thoughts to explore new and better, adventurous” feel like they made a bad decision to start with, newer the better, maybe because it is better, for which the Pro QTP’ers agree too, but have decided to be happy with what they have
That being said there are majority on the middle
My point being, sometimes its the ideology that makes the decisions more than pro’s n con’s

Reply
manjunath - February 11, 2013

hi good mrng to all..

Please tell me which tool is best to learn for feature use and good scope in feature..manual testing or automation testing or selenium.which is best…if you dont mind plz forward your suggestions to my mail id:-manjunath.manu3@gmail.com .please please do this needful for me..i will be waiting for your precious reply.

Reply
ndeans - February 20, 2013

My company spent big $ on QTP and we can’t even use it, LOL. The show stopper: QTP can’t handle the task of testing applications in a SSO environment using different user ids. This doesn’t seem like a strange or unusual request… test with multiple users, SSO… Hmm, and yet… QTP simply can’t do it.

Big huge glaring minus.

Reply
    murathp - March 1, 2013

    I have written a some functions that solved the SSO problem with QTP (vBScript). I was also able to create a framework where a PC woke up every morning at a set time, and started to execute a set of tests on 52 applications using a different username and password for both Windows Login and application login.

    This framework required a complex design though. It was pulling usernames and passwords from excel sheets and using other 3rd party tools on top of QTP to do login/logout.

    So unless I am mistaken, what you are thinking should be doable.

    Reply
      Mohammed - March 25, 2013

      Hi Murathp,

      Can you please eloborate on how did you accomplish
      “I was also able to create a framework where a PC woke up every morning at a set time, and started to execute a set of tests on 52 applications”…..
      Can you please mail me the details to mohds147@gmail.com
      It would be of great help.

      Reply
Sun Jay - February 21, 2013

Hi Joe, Wonderful discussion group over here. I have just switched over from restaurant management to testing( Hoping of having a regular life). I am taking classes for QTP automation testing. I was wondering if I should try to learn Sellenium also while I am picking up on QTP and what would be the best way to go about it. I noticed there are a lot of experienced people over here, if anyone has any suggestions please email me at: sjinalabama@yahoo.com.
As I can use all the help and guidance I can get, any feedback would be greatly appreciated. Thanks

Reply
kavi - February 22, 2013

Hi Joe,

Ur discussion forum is very much informative. I am planning to take up Selenium course.

Reply
    Joe Colantonio - February 23, 2013

    kavi » Thanks Kavi! Can you do me a favor – when your done taking the course can you let us know what course you took and if it was helpful. I get a lot of request selenium course recommendations so it would be good to know if you find one that you are happy with. Cheers~Joe

    Reply
      murathp - March 1, 2013

      Joe,

      I just discovered your site and I really like it. Information exchange and all of the contributors are great.

      I have a more general question though related to this topic. You see I picked up testing profession not by choice, but by mistake and economical circumstances of the time about 16 yrs ago. Started out as a manual test engineer, then picked up winrunner and QTP. I sure like to learn Selenium more and many other tools mentioned here, however, to be an expert in something open source, you need to learn something else…for example, Java, PHP, Ruby, C# and whole nine yards…so a test engineer eventually need to be as good as a software developer in my humble opinion if they want to do automation…am I wrong?

      Reply
        Joe Colantonio - March 1, 2013

        murathp » I agree – it looks like the direction automation is going with Selenium is that you do need to have solid programming skills to be successful. To get started with Selenium you need to know some programming language (Selenium IDE is not a valid option for serious automation). Also many Selenium books assume that you already know what junit and nunit is. On top of all this you have to also be aware of all the different CI build tools in order to integrated your Selenium test in your companies build process. So if you are coming from a vendor test tool background like QTP there is a bigger learning curve than most people like to admit.

        Reply
          Lukasz - March 1, 2013

          Hi Joe
          In fact the learning curve for Selenium is higher than for QTP, but it’s not a nightmare, it’s pure pleasure. You mentioned that a person who wants to work in test automation must know programming, so what’s the deal?
          I can share very fresh thoughts about Selenium2, because my main task for last 2 months was to develop the automation framework. I can say for sure that this time was not wasted and the set of components used for that framework allowed me to get what I want.
          For those who consider Selenium I recommend such components as: Jenkins – as CI environment, Java language for Selenium tests – biggest number of examples on the internet, ant – for facilitating flexible tests run, html reports; log4j or other logging framework, Selenium Grid 2 – fantastic thing! you can easily have your environement distributed; JUnit/TestNG. Community did fantastic job on the whole Selenium and Selenium Grid, kudos to you!!!
          I am still in process of recognition of other good practices and the tool set I have described may have slight modifications. But in general the overall result, effectiveness of the Selenium plus other components has impressed me very much.
          I was using QTP for 6 years and probably came to the high fence where no other progress can be made and the QTP limitation started limiting me.
          Selenium for Web is in my opinion obvious choice, don’t waste your time on crappy, slow, buggy QTP. Save your life!
          For desktop application I am waiting for Test Complete to appear in the company. The opinions are good in general, though I can’t write my own opinion since I didn’t work with that tool yet.

          Reply
          Joe Colantonio - March 1, 2013

          Lukasz » Hi Lukasz – I’m not against learning to program. I am a programmer and love doing it! It just that some people are not aware that automation is programming and that Selenium require it. I just accepted a new position in my company where I will be creating a Selenium framework work with BDD tools like SpecFlow and JBehave. I’d love to hear more about your framework. What programming language are you using? I would like to use Ruby but most of the developers want to use Java or .NET.

          Reply
          Lukasz - April 3, 2013

          Hi again
          I do not consider myself an expert in Selenium. I am still learning it. Of course the Internet is very helpful and provides many code snippets. Our role it to filter it out and often on the forums there are only questions and no good answer.
          My framework uses native Webdriver, so far I decided not to use any external libraries (Geb, Cucumber).
          The libraries are divided into packages. One package contains common methods that are generic and work for all other classes.
          People ask here about how to start with test automation. I can only say that of course you can start with automation having no knowledge about programming but you will never go beyond some horizon and your tests will more rely on tool functionality, wizards than on real code. This imposes a lot of test maintenance and sometimes test automation simply fails because manual testing would be cheaper.
          The best test automaton engineers, from my past experience, derive from software developers who wanted to develop test scripts.
          The community may be angry but majority of QAs who claim they automate, do crappy record-play tests which fail very often. That way companies may not believe in the power of test automation.
          Recently I had conversation with a VP who said automation is a waste of time. This is a result of low-quality test automation. I was hosting job interviews many times and saw small percentage of candidates have strong programming skills while in their CV only automation is being mentioned. This is sad.

          Reply
Lukasz - March 1, 2013

I am using Java for the tests.
I still think whether to use Page Factory or Geb, but I am cautious because often it is like the tools work well for presentations, on simple page.
But what will happen when the page is dynamic enough that number of objects is high, the DOM is big? Another concern is what happens when I don’t have unique id I can use but I have to find other query selector which not only has to find the object but be effective in IE (I have to support IE7/8/9 for cross browser testing).
I am sure people made many interesting solutions but not always want to share it. My framework desing not necessarily is the greatest but for a few tests currently done it works very well. The webpages I was to test are not the simpliest, there is dynamic content in the tables, but I can imagine more complex pages on the other hand.
I see you want to go the way of BDD, cucumber-like frameworks.

Reply
Pavan - March 11, 2013

Hi
My project is developing on java environment i want to know which automation tool (either QTP or Selenium) is best.Please suggest me..I have experience on QTP only……….

Reply
    Joe Colantonio - March 11, 2013

    Pavan » Hi Pavan – Is your application a browser based app? Selenium is only for browser automation. If it is a true java thick client app you need to use a different tool like QTP, TestComplete, or Jemmy

    Reply
krish - March 14, 2013

Why , why on the earth company buy QTP for web automation.
…because tester get very small learning time for programming , I guess….
Days are not far when QTP will die as VBS is going to be obsolete in future … TDD BDD and OOPS based test are the the future.

But QTP will not wash away for low salary jobs. It will be there with big company with non-IT core business…
— God bless Winrunner

Reply
Jean - March 21, 2013

Has anyone used Automation Anywhere? how it is comparable to UFT? I am evaluating some tools that can do the job without much programming. I was leaning towards QTP but all your comments here are throwing me off.

Reply
saramg - March 29, 2013

I have 5 years exp. In Manual Testing & I will shifted in Automation which will good for long term QTP or Selenium (my programming is not good)

Reply
    Joe Colantonio - March 29, 2013

    saramg » If I was starting to learn automation for the first time I would definitely start with Selenium. The great thing with Selenium is that you can use it with any programming language so if you are new to programming you can use a more user-friendly scripting language like Ruby or Python.

    Reply
      saramg - April 2, 2013

      hi joe ,
      thanks for good advise but one more question, is this Programming knowledge is compulsory & if compulsory then java script good or not.

      Reply
Rakesh - April 2, 2013

u got my vote. totally agree.

Reply
LiSyaoran - April 11, 2013

I agree.
No fan to become the tool.
I wish my company would understand, just think HP.
AND THAT BRINGS PROBLEMS.

Excellent note as always.
Greetings!

Reply
Sunil - April 16, 2013

Hi Joe,

Can i use VBscript in Selenium?

Reply
    Joe Colantonio - April 18, 2013

    Sunil » Hmm — I was going to say no until I just found a Selenium 2 VBScript webdriver . Is this what you are looking for?

    Reply
      Sunil - April 22, 2013

      Joe, what is the use of Selenium 2 VBScript webdriver?
      Can we use Selenium 2 VBScript webdriver for creating test cases in VBScript?
      and is there any specific difference between Selenium and Selenium 2 (Except for just the Version number)?

      Lots of queries!! Coz i am new to this automation thing…
      Hope you understand…. Thanks.

      Reply
Kevin Abel - April 22, 2013

This really is happening to me. I spent years learning to dance six count east coast swing style. I was getting to be one of the better dancers. Along came west coast swing. It was still dancing to the same music. But now all of the movements changed.
QTP and Selenium and now east/west coast swing. Once I get good at a skill, why can’t the target stop moving? I enjoy learning, but how about letting me enjoy the efforts of my hard work for a little while?

Reply
Uday - May 13, 2013

Hi,
I personally feel that using QTP or Selinium should be to the barest minimum. Imagine an enterprise level application where there is complex business logic involved, and they keep changing.

1) Can a manual tester be made to learn programming and maintain scripts build on build or release on release.
1 a) Say if only 40% of the suite of application, can only be automated imagine the effort on the programming front spent (time and money) is it worth the effort?
1 b) Most of the time will be wasted in reprogramming and negotiating and adapting to ever changing applications. Well manual testers can try various permutations and combination to test and uncover the defects in business logic, again is it worth the effort?

2) QTP poses huge licence and meager trail period(just about enough to get our hands on features), corporate are reluctant to try their hand on the tool.

Cheers!

Reply
SRay - July 2, 2013

You shouldn’t be religiously devoted to a particular tool set, but as with any form of development, you should try to pick 1 or 2 tool stacks and stick with them. As a result of efforts to, “get it done, ” test automation too often becomes a hodgepodge of duplicative code, living on non-inter-operative islands, at best patched together with a series of half-baked parsers and hacked together reporting solutions. If you can stick to a single stack, you can leverage this shared code base to implement rich running and reporting interfaces, and reduce maintenance costs in the long run by maximizing code reuse and knowledge portability between test suites and project teams. You shouldn’t only use Selenium any more than you should only use mongodb, but you also shouldn’t use 5 different tools any more than you should implement each table of your database in a different database engine.

Reply
raghava - July 8, 2013

QTP vs Selenium which is best.

Reply
Neha - August 21, 2013

H joe,
came across your website and looks like its very informative. I am going to dive into it.. I was curious to know which scripting language are you using with selenium.. Is it java or c# and have tried Selenium grid so far?

Reply
    Joe Colantonio - August 21, 2013

    Thanks Neha – we are using both C# and Java for our Selenium//BDD frameworks. We have started using Selenium Grid but are currently having lots of odd issues with it.

    Reply
      Neha - August 23, 2013

      I am at a point where I need to start looking into using Selenium Grid for distributing tests on different machines and think it takes bit of a work to make it work.I looked in to proprietary automation tools and they have very simple and neatly documented ways of achieving the same. Makes me wonder why not use a paid tool which is a lot hassle free when compared to selenium.Would you be able to talk about some of the issues that you are facing. I would also be interested to know the test runner tools used to make grid functionaly work. For example, i heard TestNg has great parallel attributes which helps working with selenium grid easier.What is the equivalent tools/framework used for selenium C# version?

      Reply
lipsha - October 18, 2013

Hii joe..
I am already join for qtp course of testing. I am really very much confuse that whether I am doing right or not. Plz say me whether there is any futer for qtp testing or not for fresher.

Reply
    Joe Colantonio - October 19, 2013

    there is not right or wrong answer for this. I can’t see into the future but if I was to guess I would say that the future belongs to open source tools like Selenium. Just like there are still jobs for COBOL programmers there will probably still be a small need for QTP engineers. But you should not focus on a specific tool – you should focus on automation testing in general and use what ever is the best tool for the job.

    Reply
lipsha - October 18, 2013

Plz give me suggestion… I need ur help.. wht m doing its wrong or right

Reply
Haris - November 1, 2013

For me, Selenium appears to be a developer testing tool while QTP (and such) are QA tools. But it looks like the QA world has shifted towards Selenium as well. Will programming then become the critical skill in QA? What about excellent analytical skills and penchant for breaking software.

Reply
    Joe Colantonio - November 1, 2013

    HI Haris – I agree – I think most companies are moving away from straight manual QA and towards more of a software developer in test type roles. My own company has started this transformation. So if you are in QA i think you must learn programming. Good or bad this seems to be a software industry wide trend

    Reply
sqavenky - November 20, 2013

Yes, I agree too. Never get too attached to tools. Just see what gets your job done.

Reply
jass - December 1, 2013

Agree

Reply
jass - December 1, 2013

Agree!!

Reply
stephen lynn - December 6, 2013

I love selenium! I’m a fan boy!

Reply
Arun Kumar - March 6, 2014

QTP/UFT rocks, i am a test manager from a company which is having employee strength of 465000, QTP providing big advantage for all over need.

Reply
Dave - April 20, 2014

Good post. I’d like to share some additional insights.

I think part of this Selenium vs QTP debate, when you cast aside the “just for web applications” aspect is really about web vs non-web applications testing. The problem is in the OSS/free tooling area, there has yet to be a popular widespread tool like Selenium for the desktop & non-web apps. Until there is, QTP will still have an advantage in some areas. There is some headway with things like AutoIt, Microsoft’s UI Automation support, the White OSS project that expands upon MS’s UIA, but none of those are near the level of commonplace use & test coverage capability of QTP/Selenium.

But I think towards the future, that gap can be closed by bridging those tooling into a Selenium WebDriver (JSONWireProtocol) format. Yes you don’t need one tool to rule them all, but one tooling format to rule them all sure would be nice. There is already things like Appium, ios-driver, etc. that bring Selenium API to mobile testing (for native, hybrid, and web apps). I don’t see why that can’t be expanded to non-mobile desktop, etc. apps as well utilizing a Selenium API but internally drives some other tool behind the scenes. Someone could retrofit a port of Selenium server or Appium server, etc. to interface to a desktop UI automaton tool. There already is such projects out there, just not very popular or gone very far, here’s one example:

https://code.google.com/p/twin/

And wanted to close with mentioning that if one wanted to have Selenium support from QTP/VBScript, you can do that without simply using shell/command line calls to some wrapper to Selenium. Get more integration with calling Selenium API in QTP through these:

https://code.google.com/p/selenium-vba/

http://htejera.users.sourceforge.net/vbswebdriver/

Reply
Nelson - May 10, 2014

This is all very interesting but the same things I hear from my company’s QTP fan boys who by the way do not have computer science degrees. They speak of all the tens of thousands of “test cases” programmed in QTP. I came in (one of the new Sr. QA mgrs) and designed a Python tool that auto-generates test cases using a combination of test rules coded in XML (7 variations of boundary value analysis, Cartesian products, relational algebra and decision table analysis) to produce a CSV file of multi-variable test cases that we THEN feed into ONE QTP script where even the test steps are under control of the CSV data, allowing the QTP script to vary its behavior. No need for lots of poor performing QTP scripts. Although sites like this have helped me to optimize our QTP function library, our fan boys don’t understand software design, don’t bother to benchmark QTP performance so don’t bother to research how to speed it up. So thank you Joe for taking the time to share your knowledge and experience!

Reply
John - June 10, 2014

Completely agree, although now I’m relearning Selenium (Webdriver, actually) after having been away from automation for a while. As well as learning Java more or less for the first time. Your blog rocks by the way – of all the heaps of technical books I’ve splurged on don’t contain a single example of an assert in Java (but your blog does!) Thanks for that, Joe.

Reply
    Joe Colantonio - June 16, 2014

    Cool – thanks John. For some reason when I was learning I really struggled with understanding jUnit assertions so I’m glad it helped you.

    Reply
Hemanth D - June 29, 2014

well said Nelson. love to see your blog for UFT updates :)

Reply
Bala - July 23, 2014

Hi joe,
Is there any book or code…for how to run the qtp scripts in Hudson/cucumber/bamboo for continuous integration tools

Reply
    Joe Colantonio - August 2, 2014

    You can run QTP/UFT test stored in ALM using Hudson in Hudson look under the Available tab, for the name HP Application Automation Tools.

    Reply
fahad - December 29, 2014

Totally agreed

Reply
Madhu - January 9, 2015

Hi Joe,
Do you have any idea about WATIN?.

Reply
Shaun - January 29, 2015

QTP is an expensive donkey and should be avoided if at all practicable.

Reply
Charanpreet Singh - March 19, 2015

I totally agree with you Joe. You should not be limited to one tool. But frankly speaking I worked on Winrunner, QTP/UFT, RFT, Testpartner, selenium for functional testing. I like QTP/UFT more because of ease to use. Lot of help available for any type of issue. User friendly, lot of support for technologies, good for any agile project as well and for many other reasons…QTP/UFT rocks…

Reply
Sunitha - April 14, 2015

Hi I completed my manual testing course.i’m a fresher.then which one is good in automation tools?like QTP or Selenium.plz provide me
Thanks

Reply
    Joe Colantonio - April 18, 2015

    If I was starting to learn automation for the first time I would definitely start with Selenium since its a free tool. The great thing with Selenium is that you can use it with any programming language so if you are new to programming you can use a more user-friendly scripting language like Ruby or Python.

    Reply
Ravi Shankar - April 15, 2015

Yeah! I am Completely Agree with Joe….

Reply
fahad - May 7, 2015

totally disagree with your thoughts .their is a lot of difference in 100000$ licences and a free of cost tool and the versatility in customization which gives edge to selenium over UFT. you can get every thing in Selenium too by integrating 3rd party open source tools like sikuli for image based recognition Robot for analog steps etc

Reply
    Joe Colantonio - May 7, 2015

    HA -I know you can develop anything from scratch using all kinds of open source libraries. The point is if you have to test legacy applications – for the typical test engineer – developing a tool from scratch it way outside their ability.

    Reply
Click here to add a comment

Leave a comment: