Oct 16, 2020 | Sai Krishna Vijaya Kumar

Adapting test automation to project architecture changes

As a tester, I’ve used Selenium, Cypress and Taiko on a few client projects. While there are challenges in using any tool, I prefer choosing a test automation tool that meets the needs of the project.

In this article, I share my experience of switching test automation tools on two projects I recently worked on. Both these projects changed it's initial architecture which required us to change our approach to testing.

Case study 1

In our first project for a retail giant, we started with Selenium. Selenium is widely used and open source. I love Selenium's community. It's very active, especially the core contributors community. I can easily connect to core contributors and get timely responses.

However, we hit a testing roadblock when the client decided to use a UI builder to build the application interface. With this new UI builder, the HTML structure of pages kept changing. As this was before Selenium introduced relative locators, the changing page structure made tests flaky and hard to debug.

For example, the application displayed a reason for cancellation at multiple locations on a page. This may seem visually redundant, but important to test, because the application had strict accessibility guidelines. Although the "reason text" was the same, sometimes we found that even though one of the reasons was displayed correctly, the other one had a display issue. The frequent structure changes caused our tests to fail intermittently.

With Selenium, this meant using parent/child XPath locators which are easily affected by HTML structural changes. The UI builder made adding class attributes or unique identifiers tough to work around this problem.

That is when we switched our tests to Taiko. Taiko supported only Chrome/Chromium during the initial versions. However, we did not have cross browser requirements as this was an internal application.

In Taiko we wrote tests by referring to elements visually. Like this example for verifying reason for cancellation using Taiko’s proximity selectors

textBox(below(text(“Reason for cancellation”)), toRightOf(“Cancellation type”))

Compared to HTML structure, the labels "Cancellation type" or "Reason for cancellation" rarely changed.

This also made the tests readable. Our team easily understood the element on which the action was performed on the screen.

Please note. Selenium 4 recently added relative locators which by definition is similar to Taiko’s proximity selectors.

Why Taiko worked

Taiko’s proximity selectors are independent of the HTML structure of the page. Proximity selectors made moving to Taiko more beneficial than we initially anticipated.

We also used proximity selectors to test positional features of the application. For example, testing the position of an icon on the screen. We had to ensure that the icon was near the pick up address closest to the car’s location. Proximity selectors caught crucial bugs around this feature which would’ve otherwise cost the business.

Case study 2

On my next project, Cypress was the initial choice. The team found it’s features like stacktraces, screenshots, highlight element on action, video recording
useful. Debugging test failures on our local machines was easy too.

But, we ran into the limitations of Cypress when the application architecture changed to use microservices and micro front ends.

The Login module was hosted in a different domain. In Cypress you cannot visit a domain of different origin in the same test.

As a workaround, we created a virtual DOM to fake and inject elements to the browser session. This worked until there were changes to the login module. The virtual DOM had to be constantly updated and this complicated debugging test failures.

Cypress does not support opening multiple browser tabs or windows.

The payment scenario in the application opened in a window or a tab. The payment scenario also used external gateways to process payment. Once the payment was complete, the window or tab was closed and the flow would continue in the original window.

As mentioned in the tradeoffs, cypress runs inside the browser which makes it hard to interact with external node libraries.

We had to use node libraries to communicate with the upstream and downstream systems of the application. All these systems communicated using Queues. For our end to end tests we read from the queue to begin the workflow and wrote onto the queue after the test run. As a workaround for getting node libraries to work with Cypress, we had to deal with promises within promises. This made the test failures very hard to debug.

These were significant blockers for using Cypress on that project.

So we moved to Taiko. Unlike Cypress, Taiko runs outside the browser as Node.js module and uses Chrome Devtools protocol to automate the browser. With the plugin architecture it was easy to build features missing in Taiko that were available to the team with Cypress.. Taiko-screencast and Taiko-screeny are examples to capture video and screenshots of every action respectively.

Why Taiko worked

Taiko works well with the Node js ecosystem. It works with any JavaScript test runner, assertion library or just any other node js library. Because of this we could easily use node libraries to communicate with the queues and address other project needs of node module interactions.

Taiko does not impose any cross-origin restrictions because it runs outside the browser. Taiko also supports opening and managing multiple tabs, windows or incognito windows.

Key insights

Aspects to consider before using Taiko in client projects

• Taiko is a node library for automating the browser. Although Taiko’s command line can run basic scripts you’ll need to use it with a Javascript test runner like Gauge, Mocha, or Jest for other features like reporting, running multiple tests etc.
• Taiko has a REPL, where we use Taiko’s API as commands to automate the browser.
• Taiko does not run on Browserstack or Saucelabs.
• Taiko tests can only be written in Javascript and Typescript.
• Taiko supports only the browsers that are CDP compliant, which at this point are Chrome, Microsoft Edge and Firefox.
• Taiko’s plugin architecture makes it easy to write plugins that suit your project requirements - examples of Taiko plugins.

As there were many open source testing tools, I was skeptical when I first heard of Taiko. Why create a new tool when we can contribute to existing ones?

I’ve come to realise that each tool has its strengths and opinions, some work for your project and some don’t.

After using Taiko on my projects, I started contributing features and bug fixes. I am a core committer now. I like Taiko's approach to test automation.

Please check out the plugins I’ve authored!

• taiko-android - A plugin to run web tests on android devices and emulators using Taiko.
• taiko-diagnostics A plugin for taiko which provides some diagnostics features like measuring speedindex, performance metrics of web pages.
• taiko-screeny A taiko plugin to capture screenshot on every action.

You can write your own plugin and become a contributor to the Gauge community.