Welcome to Episode 82 of TestTalks. In this episode we’ll discuss what it take to be a software craftsman, and why quality and testing are the true signs of a professional developer. Also — if you’re an automation engineer or SDET (software developer in test) you’ll learn some awesome modern testing techniques that will expand your testing toolkit.
You might think that going into 2016, the days of developers not being particularly concerned with quality and believing that quality is the job of QA would be long gone. You would be wrong. Many SDETs still find themselves working in this kind of culture. This episode will give you the tools you need to strike back and get your revenge. Discover how leveraging test mutation frameworks and property-based testing can expose hard-to-find, edge case defects that many developers ignore. Scott’s enthusiasm and hustle around self improvement and making development efforts awesome really shine through in this can’t-miss episode.
Listen to the Audio
In this episode, you’ll discover:
- The difference between a Professional Debugger and a Professional Programmer
- Why having developers responsible for quality in an “Agile” environment can cause numerous issues
- How to become a software craftsman.
- Tips to improve your unit testing efforts.
- What test mutation frameworks and property-based testing and why you should use them.
Error 520 Ray ID: 4f748d56fd91c627 • 2019-07-16 14:09:07 UTC
Web server is returning an unknown error
There is an unknown connection issue between Cloudflare and the origin web server. As a result, the web page can not be displayed.
What can I do?
If you are a visitor of this website:
Please try again in a few minutes.
If you are the owner of this website:
There is an issue between Cloudflare's cache and your origin web server. Cloudflare monitors for these errors and automatically investigates the cause. To help support the investigation, you can pull the corresponding error log from your web server and submit it our support team. Please include the Ray ID (which is at the bottom of this error page). Additional troubleshooting resources.
Expose wannabe #developers with #test mutation frameworks and property based testing~@Bizmonger testtalks.com/82Click to tweet
Join the Conversation
My favorite part of doing these podcasts is participating in the conversations they provoke. Each week, I pull out one question that I like to get your thoughts on.
This week, it is this:
Question: Who should be responsible for software quality? Share your answer in the comments below.
Want to Test Talk?
If you have a question, comment, thought or concern, you can do so by clicking here. I’d love to hear from you.
How to Get Promoted on the Show and Increase your Kama
Subscribe to the show in iTunes and give us a rating and review. Make sure you put your real name and website in the text of the review itself. We will definitely mention you on this show.
We are also on Stitcher.com so if you prefer Stitcher, please subscribe there.
Read the Full Transcript
Joe: Hey Scott, welcome to Test Talks.
Scott: Hey Joe, thanks a lot.
Joe: It’s good to have you on the show today. Before we get into it, could you just tell us a little bit more about yourself?
Scott: My name is Scott Nimrod, born and raised on the mean streets of Cleveland. I started my career in software testing, and after I graduated, I transitioned from a co-op, working in software testing, to an application developer. I’ve never, ever ignored my itch to basically leverage QA controls, so it’s something that I hold very dear to my bosom. It’s something that I just wish other developers can appreciate just as much as I appreciate, which I’m trying really hard to change the mindset, but it’s a campaign, a [crystal 00:00:50] campaign that I’m on. In regards to languages, started off with C++, VB, then to C#, and now I’m really excited about F#. It’s showing a whole new world of possibilities, not just for cranking out applications, but also it opened up some other techniques that I just wasn’t aware of in regards to test automation. I guess perhaps I can provide some insight on my world of the perspective from the developer, the application developer, as it relates to test automation and some of the battles that perhaps SDATs and other members that participate in QA have to go through when they work with people like myself. Maybe we’ll touch on that, but that’s why I’m here. I’m trying to provide as much of my personal experience as possible, and also share what I’ve learned so far.
Joe: As a developer, how did you get into test automation? Most developers I know, when they even hear the word testing, their eyes glaze over and they just completely tune me out.
Scott: Yeah, you know what, that’s a shame too. That’s very unfortunate, but I got my associate’s degree in mainframe programming. I started off my career towards maybe in the middle- or no, I graduated high school during Y2K, but I really didn’t know what that was. I’m answering your question, it’s just taking a little while to get there. I ended up going to community college, and I ended up doing mainframe programming and I realized that there wasn’t an abundance of jobs out there for people to write COBOL. Someone made me a offer to work as a co-op, in which I would go to school and also put in 20 to 30 hours a week within their QA department, or really, they called it software testing department, in which I would write automated test scripts using Visual Basic 6.0 in a combination with XML to data drive those tests. That’s how I got introduced to test automation.
As a 22 year old that just wrapped up with community college, I get my foot in the door at a software company and I see this lab that they have of like, maybe 6 different machines that’s running from Windows NT to Windows Server 2000 to Windows XP. I’m in this lab and someone is showing me different parts of the company’s- I don’t know, floor plan or whatever, they’re giving me a tour, and I see these computers that are running these applications, but there’s nobody sitting in front of the computer. I look at it, and I’m like, it’s like ghosts are interacting with the software. How did you all do that? They’re like, “That’s our test automation,” and I’m watching menu controls come down and menu selections, and page navigation. I’m like, “Yo, I never saw that before, man … am I going to be doing that?” and they’re like, “Yeah, you are, that’s why we hired you. You better be doing that.” I’m like, “Okay.” That’s how I got introduced to it, and I immediately- I didn’t know exactly what I was looking at, but I knew it was cool. That was my step into test automation.
Joe: Awesome. As you transition more into development, do you still do test automation or what kind of testing do you still include in your life cycle?
Scott: Yeah. I’m all about RGR- red, green, refactor- and that’s an acronym if I can say that for performing test driven design or development. Before that, I would crank out a application, but in order to test my application I would have to launch the application every single time, which is time consuming. It’s really time consuming when you’re working within a mobile environment. Once I learned about test driven development and I started researching it, most of the hits were coming from Uncle Bob. I often say I’m a disciple of Robert Martin’s teachings. It started resonating with me, and I felt like I was focused. I felt like I was on a roller coaster and I was safe because I’m hugging the rails, and I’m not going to venture off the tracks because I’m locked in. It just keeps me focused. I feel like- I never shot a gun before, but if I had a Uzi or a AK, those tracer bullets, I know where I’m going. I know the destination that I’m trying to get to. That’s the way I feel about test driven development, or test driven design. Red, green, refactor- red, green, refactor- that’s the flow. I guess it’s the flow that I appreciate, that I’m addicted to.
Ever since I’ve learned that methodology of building systems, I get frustrated when I, as a contractor/consultant, I go into other shops in which- yes, in 2015, the majority of developers- I guess I can’t relate to Wall Street or Silicon Valley, but the majority of shops that I’ve been at, even in 2015 as we approach 2016, they’re still professional debugger-ers. They are still- in order to prove that their code works, they still feel as if they have to launch the entire application, spend several more seconds if not minutes putting the application to a specific state to step through the debugger and validate that their code works as expected or intended. The frustration is, they allow these companies, they fight change even though it should really be common sense.
In a way, I feel like a lot of these companies that I’m trying to help, they’re fighting common sense of what professionals should be doing. It’s kind of strange that- and remember, I’m speaking as a application developer that came from a test automation background- it’s weird how a lot of these shops, they view QA as kind of like a savior to what they’re doing. All they need to do is crank out code and then say that they did it. I think most developers- and this is me speaking- I don’t think they really think about cranking out code that’s proven, and then say, “Here QA, I want you to use your amazing brain and flush out bugs that really requires creativity to flush out. I think a lot of us, we just write code, and whatever is on the acceptance criteria- which I think is usually happy path scenarios- we say check, check, check, all right QA, and we expect QA to pretty much tell us what else we need to do, when a lot of thing we could be finding out on our own by just doing very straightforward sad path scenarios.
Joe: Awesome. You bring up a lot of points there Scott. One thing that’s kind of funny is I was in a code review the other day with some bugs that made it into production, and we’re just reviewing why it happened. The developer said, “Well, these tests were running flaky, the functional tests were running flaky in the CI environment,” so we kind of ignored it. When we found out what the issue was, it was just that they weren’t checking that an object array was being returned, which someone brought up then, shouldn’t that be a unit test? The guy said, “Yeah, I guess it should have been a unit test.” It’s funny. The developers just did a quick unit test, like you said, that’s fast and doesn’t rely on [inaudible 00:09:40]. The whole application and running into its integration every night, they would find it. How do we get to that point? How do we move a developer from what you call them, a professional debugger, to a professional software developer? Is it even possible? If someone just doesn’t have that drive like it sounds like you do, how do you get someone there?
Scott: Consider me a software craftsman militant. My views are pretty extreme compared to the people that I work with. I think this is the first step. I’m not going to hand out doctrine, but I’m going to tell you what I think is a first step, and this is the way that I operate. This is the standard that I hold myself to. Are you ready, Joe?
Joe: Yeah, lay it on me.
Scott: All right. I have reason to believe that within the agile movement- which you know, agile comes in a spray can, we spray it on ourselves and we say that because we do daily stand up, we’re agile- but I have reason to believe that within the agile movement, it’s popular belief that the development team is responsible for the quality of the code. That’s the belief system that I’ve come across. I think that system is flawed when it comes to DMDs, dark matter developers. This is something that Scott Hanselman had wrote a blog post about. Developers that are there to collect a check and do a good day’s work, but they’re not really obsessed about software. They don’t love building, loading, and delivering software.
When you say that everybody owns the code, and everybody’s responsible for the quality of the code, I have reason to believe that that results in no one taking ownership of the code. It has opposite effects. This doesn’t really apply for your top leading organizations. When you have people that are just gung-ho about building incredible software or [translated 00:12:11], when you have a team of software craftsmen, then I don’t think you have to worry about that because they take great pleasure in delivering and working and building what they build. I think when you take developers that aren’t as excited, when you take developers that the only thing they know about software is what their company has taught them and they’re not necessarily a servant of the industry, no. They’re a servant of their company’s practices, regardless of its effectiveness. When you have that type of environment, what happens is your agile team, they don’t own the quality of the code, and what happens is after several months or a year or two, it becomes harder and harder to build sustainable/accessible features on top of your code base.
Joe: Awesome. Scott, are there any books or resources that you recommend to someone to help them learn more about how to be a better software craftsman?
Scott: Yes, but they have to want to be a software craftsman. One book is xUnit Test Patterns. That changed the way that I write code. It’s xUnit Test Patterns. It’s a healthy book. It’s probably about 500 pages. It talks about code smells, and it’s pretty much design patterns in regards to the ecosystem of test automation of unit testing specifically. That book is a bible. I consumed that book 5 years ago- no, 3 or 4 years ago, and just last month I decided to just go through that book one more time, just to refresh my thoughts so that when I’m cranking out code, I can still reference certain ideas and then go back to the book to research it. xUnit Test Patterns is one book that I think is very, very vital for people to be mature when it comes to proving that their code works.
Here’s the thing, Joe. I think the majority of developers, I think they think that they write unit tests. I think when developers think that they write unit tests, I think that that belief is somewhat flawed because I don’t think unit tests are written. I think unit tests are engineered. This is what happens where developers will try to spray on their agile, they’ll take their agile can and spray it on, they’ll be like, “All right, fine, [CDD 00:15:15],” and they’ll start writing out a lot of useless tests. Then as the code base matures, they spend more time trying to maintain their tests, and then they give up on test automation, or they give up on test driven development. Why? Because they decided to use the company’s time and practice on company hours to make their mistakes, instead of being software developers, being professionals and realizing that it’s their job to stay up to date. Just like a boxer- I don’t know that much about boxing, but I don’t think boxers get paid to train outside of the ring. I think they get that purse after they perform inside the ring.
I think developers, they’re missing that. They have this expectation that we only practice on the clock, and it’s okay to make mistakes on the clock because we don’t want to make mistakes on our own time. I guess, make a long story short, I think that our heart has to be into it. Our heart has to be into unit testing. Our heart has to be into test automation. Then, they’ll learn about- instead of writing unit test, they’re engineering unit tests. They’re building a test API to support whatever mocking or subbing that they need for their test method clients.
I also wrote a blog post that I reference from Unit Testing in F#. That’s a book. I probably butchered the name of the book, but it’s F# unit testing, and in that book they speak of the ten commandments of unit testing. I can’t recite them all, but it’s something that I think a lot of developers should at least- I don’t know, look up, and just evaluate if those ten commandments really make sense or not. I think they do. In addition to that, for SDETs, if SDETs really want to expose the quality of unit tests out there, then I highly recommend that they at least research test mutation. Joe, are you familiar with test mutation, because I didn’t learn about it until like, two weeks ago.
Joe: No, what’s test mutation?
Scott: Okay. Again, when I started studying functional programming, because I feel like we’re, as application developers, everything is- we use object oriented design as a sledgehammer to solve all problems. This functional programming is opening up a world to me, not just for programming but for also testing. Test mutation is essentially this. Test automation, as I understand it- and I’m not a expert, I’m not the authority- but as I understand test mutation, you write tests or you execute tests, that test tests. You’re testing your unit tests.
What happens is this. My understanding is, in .NET, there’s a couple test mutation frameworks such as NinjaTurtles- hence Ninja Turtles are mutants, right- and there’s also VisualMutator. Java has several of them, the Java ecosystem. Basically what happens in test mutation, as I understand it- again, I’m not a authority on it- is you run a tool in which you point the tool to your unit test, and this tool will basically take the system under test, or the target of your unit test, and it will flip or swap operators within the function or the method that’s the target of your unit test. For example, if you’re unit testing a method that has some ‘and’ operators, your test mutation will take that DLL, or take that code, the function or method that’s being tested, and it will swap out that ‘and’ operator for a ‘or’, or it’ll swap out the ‘and’ operator for- it’ll put a negation on it. The same thing with plus plus operators, minus operators. It’ll take the plus operators and swap it out with minus.
What happens is this. The product or the benefit is you can run this tool and you can get a better understanding of what unit tests developers wrote that are BS. They just did it for code coverage constraints. I’m all about code coverage, and a lot of people say, “Scott, that’s stupid,” because developers will just do whatever they needed to do, regardless of the validity of that test, just to meet code coverage requirements. I’m like, “Okay, fair, fair. You’re absolutely right,” when you have developers that don’t have their heart in the quality. In SDET or another developer that’s a craftsman that is really sick and tired of this culture of application developers not really being concerned with quality, they believe that quality is the job of QA, then as payback, as revenge- and I wrote a blog called Revenge of the SDET- is they can leverage these test mutation frameworks, and they can run it on the developer’s unit tests and flag all the unit tests that pass regardless of the operators that are being used within their tests. Again, the logical operators- and, or, plus, minus, multiply, et cetera. If those tests still pass, then your test mutation test will fail those unit tests. It’s a way to, again, evaluate. It’s a tool for evaluating or examining the true quality of your tests.
We’re not even talking about code anymore. We’re not even talking about the deliverable of the application, we’re now scrutinizing- are the tests really tests, or are they just fudge that developers are just trying to just throw, just to satisfy management’s concerns, or management’s constraints, or gated check in constraints. Again, for every SDET or QA that has the technical [chops 00:22:24], this is how you can get back at them. Run some test mutation and expose them.
That might sound really political, but at the end of the day, I think the reason why we’re building software is for the client. Not only that, I really want to build software that is cheap and also has high quality. I believe that both can exist. I believe that you can reduce the cost of software if you just have the proper checks and balances that every other industry has, except for us. The more tools that we can leverage, if the developers aren’t leveraging these tools, then I think as QA and as SDETs that are working in the same house as QA, we owe it to ourselves to use these tools. I would never have ever found out about test mutations if I weren’t studying F#. There’s another one that developers can use as well. This is called- Joe, have you heard of property based testing?
Joe: No, you’re really laying some new ones on me today, no.
Scott: Again, if I never studied functional programming, I would continue my ignorance. Property based testing is this. Unit tests are really example based testing. What does that mean? As a developer, you are writing a test that is very specific to a case scenario. You’re saying with this specific value, or with these specific inputs as my setup I’m going to exercise this method or this behavior, and I expect this specific output. Property based testing is this. Are you ready?
Scott: Okay. Instead of putting in specific inputs and asserting specific outputs, we’re testing the properties or the characteristics of a function. What does that mean? Again, I’m not an expert on this. I’ve only been aware of this for like, two weeks. My understanding of property based testing is, you’re testing the behavior of a function, the properties, the characteristics of the function. We know that if we have a function called ‘add’, there are certain principles that this add function has. If we execute a add function and we provide two inputs for adding, like a variable that points to one, and then another variable that points to three, we know that a property of that function is- the result of that function when adding two numbers should never be less than the product of those two numbers. If I provide inputs one and three into a add function, then the result of that add function, in regards to the absolute values of those inputs, should never be lower than the product of those two inputs. I should not have a result- if one plus three is four, I should never have a result that’s less than four. That is a characteristic of that function based on absolute values.
Another behavior of a function is, if I provide inputs one and three for a add function, I should get the same output as if I swapped the inputs. If I first put in for the add function, one and three, and it equals four, if I swap the inputs and instead of one and three, I put three and one, I should still get the same result. That’s property based testing. What happens is this. The cool thing about property based testing- you’re probably like, okay, that sounds like specific inputs again, that sounds like example based testing, and you’re right, until I drop this on you. With property based testing, we’re not providing inputs at all. We’re not passing in literal values. We’re relying on the framework to provide a range of values. The framework is streaming in values from a range, like from -1000 to 1000, or whatever you specify. When you do property based testing, you let the framework pass the parameters, and it’s going to execute the same test over and over again with different values- negative values, positive values, really large values, zeros like edge cases. It’s going to do all that for you.
These tests don’t execute in ten milliseconds or twenty milliseconds. They’re not like unit tests. These tests, when you execute them, these property based tests, they may take 300 milliseconds. Depending on how you configure these tests based on the range of values that you want to test for, they can take four seconds, five seconds. They can take ten minutes based on how you configure it. The beautiful thing about property based testing is you’re letting the framework manage the edge cases for you. That’s the beautiful thing is you can tell- I think it complements your unit tests in which you leverage your property based tests to catch the edge cases or the special cases. That’s something that application developers can also adopt or take in to their arsenal of tools. [So far 00:28:38]
Joe: That’s awesome. It’s kind of funny or sad. I was actually last week at a conference talking to a developer of a popular test tool. He’s like, “Yeah, I’m trying to explore functional testing with automation,” I’m like “No one does that, I’ve never heard of anyone saying anything about functional testing,” and bam, the next interview I have is actually you talking about functional testing, so I’m so glad I had you on the show. It’s something I’ve heard about, but I didn’t think anyone was really exploring it, but it sounds like there’s a lot of benefits to functional testing and benefits that may benefit automation or testers to make your code quality even better.
Scott: Yeah. I think what we’re learning- and again, I have extreme views. I get really excited about things. I think we’re starting to learn, for people that are actually in touch with the industry and the ecosystem of the industry versus company XYZ, we’re starting to learn that, you know, maybe we are treating object oriented design as a sledgehammer. Maybe we should always leverage the principles of continuously mutating states as a way to build reliable software that delivers consistent results. Maybe building software that is based on modifying the state of a object, maybe that can get us in trouble. Maybe functions are a really good thing, because maybe functions really are deterministic, and maybe functions, if they don’t modify anything, they don’t have any side effects. If there’s no side effects, then maybe that provides a higher probability of higher quality. If functional programming doesn’t have any side effects, then maybe we can expect the same result over and over again versus imperative programming or object oriented programming where our programs thrive off of side effects. We’re always changing the properties of an object.
I look at functional programming as the next generation of programming languages that is going to still take a while for mainstream to adopt, but I look at it as a beautiful thing. Just as C was a beautiful thing potentially to assembly code, C++ was a beautiful thing to C. Generation to next generation to next generation. C#, people thought that was the next generation of C++, and I think that within .NET, F# is the next generation for C#, just as Scala or Closure potentially is the next generation of programming in Java.
I think that thousands of years ago when, I don’t know, tribes would attempt to build shields, it was probably a art. Maybe in certain parts of the world they just use sticks to create a vest to protect them from spears, and maybe in the times of Rome, they experimented with bronze and other type of metals. It was still a art. In this millennium, we’ve taken what’s been art and we really made it a science. We use carbon fibers. It’s no longer a mystery. It’s no longer experimentation, even though we can continue building on it, we know exactly what ingredients to use to get the desired result. I think that this is where we’re- it may be another hundred years off, or a thousand years off, but I think that sooner or later, I think that software development can really mature to be more of a science based on the progression of what has been proven to work, and what procedures have been proven to work. I’m really excited. Again, if I didn’t make a decision to investigate other ways of building software, I would have never, ever discovered the tools that I can leverage to help me deliver a higher quality code. I’m so thankful for it.
Joe: Awesome. Okay Scott, before we go, is there one piece of actual advice you can give us to improve our software craftsmanship skills? Let us know the best way to find or contact you.
Scott: This might sound really selfish, but I also think this is good advice. This is kind of my motto. The more I learn, the more I practice what I’ve learned, the more I demonstrate what I’ve learned, the more I’m worth. Even though I love building software, I also love increasing my worth. I love making more and more money based on what I’ve learned and what I’ve demonstrated I’ve learned to deliver software more effectively compared to the rest of my competition. I’m not sure if John Sonmez would appreciate this, what I’m going to say, but he’s advocated- John Sonmez is a software developer coach in regards to careers, and he’s advocated separating yourself from the herd, removing the herd mentality and differentiating yourself. With that said, based on the question that you’ve asked, I would encourage and I would challenge developers to chase the money, and by chasing the money, demonstrate how much more effective you are to the rest of your competition. In order to do that, you have to open yourself up to other ideas, and you have to experiment. I do not experiment on company time. Deliver on company time. Experiment in the morning, experiment in the evening, experiment and present your experiments at user groups, and refine your practice.
Software development is a practice. It’s not a job, it’s a practice. Refine it, refine it, practice makes perfect, and increase your worth, increase your worth so that you’re making $50,000, $100,000 more than the rest of your competition. I feel like that’s having your cake and eating it too. You build more on how to deliver more effective code, and you’re making more money as a result of it. That’s the best of both worlds. That would be I think what works for me, that I think other people should strongly consider. You can find me on my blog that I try to update every other week. It’s bizmonger.wordpress.com. I’m on Twitter, but I really don’t know what I’m doing on Twitter, so all I do is pretty much- I paste web links I find are really interesting in regards to software development, and I just put that on Twitter. I don’t really express my ideas on Twitter, I just share really good links.
Scott: You can message me on Twitter, and if you’re able to find my email address, which I’m not going to give out, but if you’re able to find my email address, and it shouldn’t be too hard to find, go ahead and reach out to me because I really do enjoy talking to passionate developers. Believe it or not, I don’t really have that opportunity to engage with people that really love building software, and I love learning from other people. That’s it.