BDD Gone Bad
I started in a new position several months back, and have been working on helping get a Behavior Driven Development process going. Because there are so many scrum teams involved in this new, company- wide initiative, one of the areas in our process that needs help is reviewing each team's BDD statements.
The biggest roadblock we've experienced is that people are treating BDD as an automation framework rather than as a way of having discussions to insure we're developing what the customer wants. A side effect of this is that some of the early BDDs running in the continuous integration environment are suffering from all kinds of random, flaky, unreliable behavior.
Fortunately, we're about to start a brand new project, and in order to try and get things right the first time around, I'm trying to find ways to make the new BDD process better. One method would be to create a high- level BDD code checklist that I'll try to have everyone use before their BDDs are accepted in the CI environment. This is what I have so far:
The BDD Code Review Check List
- Scenario has run multiple times locally, in a pass status
- Feature typically contains less than 20 scenarios. If more, you might want to make sure that your feature is not too broad.
- Scenario Names and Steps make sense
- Readability of G/W/T
- Not all tests are GUI tests (if API/web service is available).
- Scenarios should test critical boundary cases. For example: analyze each Assert() to understand all possible values.
- All scenarios have been discussed with all stakeholders –BDD is all about conversations.
- Limited use of technical and UI terms
- Describes behavior — not technical implementation (declarative style not imperative style used)
- Each scenario can be executed independently of any other scenario.
- None of the scenarios contain environment-specific, hardcoded test data (unless environment setup scripts are also provided as part of setup).
- All verification steps are in the Then steps — not the Given or When.
- None of the scenarios should be overly long. Typically, each should have less than 15 steps (not considering step multiplication by example tables). Rule of thumb is that an entire scenario can be viewed in the IDE without scrolling.
Should this even be a BDD test? Or would a lower-level test (Like a unit test) be better?
Most of the ideas found in the checklist were gleaned from my experience with previous projects, Google searches, The Cucumber Book, and advice from the BDD master consultants (Lance and Wes) at SolutionsIq.
Let me know what you think. Do you agree/disagree with the above, and if so, what do you feel should be on a BDD code review checklist?