August 14, 2019 | Robin Ninan

The Gauge Move

This blog is cross-posted from the authors original article

Is it worth taking the plunge from cucumber to gauge? I’ll narrate our story at Koodoo and let you decide.

The start of Quality Assurance at Koodoo was no easy step. We kicked off with an appetite for ensuring that we deliver high quality services. An appetite we have maintained thus far. This meant good automated tests every step along the way wherever possible. As important as tools, scripting language proficiency etc. all are, the glue that holds Quality Assurance together is truly communication.

Let me put forth 3 scenarios (Assuming Bob is our Automation QA engineer):

Bob gathers requirements and designs a test plan. He automates and tests stuff and finds a bug. However Bob fails to communicate the issue effectively with the relevant teams. If only the tests were human readable. Hmmm!
Bob misinterprets requirements and automates stuff. He fails to match up test plan coverage with requirements (through no fault of his own, given that test automation is all code, and Bob is good at reading code but not good at translating code into human readable format!).
Bob has a new graduate intern on team. Bob’s intern cannot understand the tests. They would have to wait till they have mastered the scripting/coding language before they can make sense of what Bob has done.

I can go on and on till the cows come home. It was the very essence of this problem that made way for BDD (Behaviour Driven Development) in the software development world.

The Rise of The Chosen One

In the test world we saw the rise of new BDD frameworks: Cucumber and Specflow were the popular ones. I for one am no stranger to the fame of Specflow / Cucumber or the test automation benefits of BDD.

Outlining some:

Human readable feature files which can be shared with product teams or business stakeholders.
Reusable steps and the ability to quickly spin up test using human readable texts.
Easy to relate tests to requirements and vice versa. This also means that it makes it easier to cross reference test coverage with product expectations.
A clear path to translating test scenarios into unit tests in the implemented code.
Community support and knowledge-base make it easy to learn.

Actually it’s not the framework itself but the concept of a BDD automation framework that i love.

From past and recent experiences I’ve learned that the above frameworks for BDD have not evolved with the times. What do I mean? Cucumber, for example: time and time again it fails to include improvements to enable test automation benefits. It focuses more on the readability and less on the ability to ease automation requirements. We have to remember that test automation with BDD is a two way street and writing an amazing step that is super readable serves little purpose for automation hurdles.

Let me just pick on two of the many ongoing issues.

State: A feature file in Specflow or Cucumber contains scenarios and scenarios contain steps.

Feature : Login
 Requirement summary...
  
Background: 
 Given a login page
  
Scenario: Login as Bob
 When user enters username "Bob"
 And user enters password "********"
 And user submits 
 Then user sees welcome page with title "Hi, Bob"

From the above example it is easy to see that under the scenario, we have a series of steps that results in actions on the screen. Each step is independent. This is fine for most test automation cases.

Let’s spice things up. Say the login process was complicated and Bob had to go through a 2 step authentication process.The rough and ready test would now look like this

 Feature: Login
 Requirement summary...

 Background: 
  Given a login page

 Scenario: Login as Bob
  When user enters username "Bob"
  And user enters password "********"
  And user submits and reads token
  And user uses the login token on login page two
  Then user sees welcome page with title "Hi, Bob"

Why is this a problem:

Behind the scenes of user reading the token and using the token in the next step there is a need to save the state/data generated by the test in runtime. We cannot pre-determine the token to be used. This raises the need for some sort of mechanism to save and share data between scenario steps. We can handcraft our own solution to this problem. Something as simple as a dictionary would suffice. This would mean that we also need to ensure that we clear out the data before or after every scenario. Small problems? Trust me, they add up.

Teardown: From the above examples we can see the use of a keyword called Background and this is a set of steps executed after every scenario. However, it baffles me that we have no tear down or clean up steps.

The BDD community has been asking the same question for years and we see the same response — readability. Are there ways around it? Yes, the most obvious one is to use before/after hooks to trigger a clean-up. In order to have different cleanup steps for different feature files we have to use a combination of hooks with feature tags. Not a neat solution by any means. And as the test pack grows, you will often find the growing need for more tags. This contributes to a very messy solution for teardown after test.

Furthermore we had niche challenges with Cucumber. This may resonate with a few. Our development follows the MERN (mongo, express, react, node) stack. Keeping inline with development and for easy interplay between development and automation, we decided to use Javascript as our base language. Naturally we made use of cucumberjs as our framework for BDD. It wasn’t long before we realised that the support for cucumberjs is far inferior to its Java counterpart. For reporting purposes, cucumberjs only supports json formats out of the box, which is a pain when trying to use reports within a pipeline as most pipelines prefer xml. Cucumberjs enforces the need for test directory structure. Although you can go against the grain here, you are technically on your own if you do. That brings me onto the next point: the documentation and support for cucumberjs is not adequate.

Let’s face it: there is a need for BDD frameworks to evolve with the times and most of them aren’t doing that. This is where we began soul searching. We were looking for a BDD framework that kept the core values of BDD intact, whilst respecting the need for enhanced automation support. After some research, reading and many youtube videos later, we hit gold. Drum roll ……

We found Gauge

Why Gauge?

Let me start with what it is. Gauge is a well thought out BDD centred automation framework developed by ThoughtWorks. Let me start with the good news:

Gauge supports the need for sharing state between steps. They provide a mechanism to deal with sharing data between steps within a scenario or even between scenarios.
Gauge supports the need for a teardown / cleanup at a feature level. No more messy hooks or tags to achieve this.
Gauge does not impose the need for implicit Given/When/Then statement initiators.
Gauge does a good job of updating their framework on a daily basis and listening to the growing needs of the automation community.
Gauge does not take away the BDD aspect of the framework and aims to keep the readability aspect intact.
Gauge has a fast growing support community.
Gauge supports living documentation through their plugin “Spectacle”.
Gauge provides a really well presented test report with their “HTML report” plugin.

What’s different

Gauge has a few different concepts that would require a bit of initial reading and wrapping your head around, but once that is done the rest is straightforward. Let me highlight a few notable differences:

For a start, Gauge has the concept of “spec” (specification) files instead of feature files which are essentially markdown files at the core. An example:

# Login
 Tags: simpleLogin
 Bob wants to login
 setup steps
 *  Given a login page
 ## Products with some properties as nulls
 *  When user enters username "Bob"
 *  And user enters password "********"
 *  And user submits and reads token
 *  And user uses the login token on login page two
 *  Then user sees welcome page with title "Hi, Bob"
 ___
 *  User cleans up browser

Titles are denoted by “#” while scenarios are denoted by “##” .
If you notice, in the above example we have a tear down step as well which will apply to all the scenarios in the spec file.
If we dig deeper into gauge documentation we find the DataStore feature which helps maintain data between scenarios, steps etc. Below is a scenario store example in Javascript:
```
// Adding value
gauge.dataStore.scenarioStore.put(key, value);
// Fetching Value
gauge.dataStore.scenarioStore.get(key);
```

There are a lot of cool features. Read more

The Challenge

We already had a working test framework we built on nodejs and cucumberjs. We were now faced with the dilemma of sticking with the old or taking a plunge into the new. We knew it would involve a rewrite of the step definitions and a rewrite of the feature files as markdown styled spec files. We were aware of the learning curve involved. Regardless, we couldn’t live with the fact that change may never happen. As a result we took the plunge for good.

The Journey

The logical step was to start a new project with Gauge as the test framework and start creating the required step definition functions. It was a smooth process as we found that more often than not the functions had fewer lines of code compared to the equivalent Cucumber step definition function.

Once that was done we moved on to adding new tests in Gauge instead of Cucumber. On the side we managed to migrate the existing feature files to spec files. Although this was time consuming, it was also a really enriching experience as it forced us to go through our feature files, validate their structure wording etc. before porting them over to spec files.

The whole process took a few days. Finally we had a working test framework in Gauge.

Was it worth it for us?

Absolutely worth it.