Oct 16, 2020 | Sai Krishna Vijaya Kumar
Adapting test automation to project architecture changes
As a tester, I’ve used Selenium, Cypress and Taiko on a few client projects. While there are challenges in using any tool, I prefer choosing a test automation tool that meets the needs of the project.
In this article, I share my experience of switching test automation tools on two projects I recently worked on. Both these projects changed it's initial architecture which required us to change our approach to testing.
Case study 1
In our first project for a retail giant, we started with Selenium. Selenium is widely used and open source. I love Selenium's community. It's very active, especially the core contributors community. I can easily connect to core contributors and get timely responses.
However, we hit a testing roadblock when the client decided to use a UI builder to build the application interface. With this new UI builder, the HTML structure of pages kept changing. As this was before Selenium introduced relative locators, the changing page structure made tests flaky and hard to debug.
For example, the application displayed a reason for cancellation at multiple locations on a page. This may seem visually redundant, but important to test, because the application had strict accessibility guidelines. Although the "reason text" was the same, sometimes we found that even though one of the reasons was displayed correctly, the other one had a display issue. The frequent structure changes caused our tests to fail intermittently.
With Selenium, this meant using parent/child XPath locators which are easily affected by HTML structural changes. The UI builder made adding class attributes or unique identifiers tough to work around this problem.
That is when we switched our tests to Taiko. Taiko supported only Chrome/Chromium during the initial versions. However, we did not have cross browser requirements as this was an internal application.
In Taiko we wrote tests by referring to elements visually. Like this example for verifying reason for cancellation using Taiko’s proximity selectors
textBox(below(text(“Reason for cancellation”)), toRightOf(“Cancellation type”))
Compared to HTML structure, the labels "Cancellation type" or "Reason for cancellation" rarely changed.
This also made the tests readable. Our team easily understood the element on which the action was performed on the screen.
Why Taiko worked
Taiko’s proximity selectors are independent of the HTML structure of the page. Proximity selectors made moving to Taiko more beneficial than we initially anticipated.
We also used proximity selectors to test positional features of the application. For example, testing the position of an icon on the screen. We had to ensure that the icon was near the pick up address closest to the car’s location. Proximity selectors caught crucial bugs around this feature which would’ve otherwise cost the business.
Case study 2
On my next project, Cypress was the initial choice. The team found it’s features like stacktraces, screenshots, highlight element on action, video recording
useful. Debugging test failures on our local machines was easy too.
The Login module was hosted in a different domain. In Cypress you cannot visit a domain of different origin in the same test.
As a workaround, we created a virtual DOM to fake and inject elements to the browser session. This worked until there were changes to the login module. The virtual DOM had to be constantly updated and this complicated debugging test failures.
Cypress does not support opening multiple browser tabs or windows.
The payment scenario in the application opened in a window or a tab. The payment scenario also used external gateways to process payment. Once the payment was complete, the window or tab was closed and the flow would continue in the original window.
As mentioned in the tradeoffs, cypress runs inside the browser which makes it hard to interact with external node libraries.
We had to use node libraries to communicate with the upstream and downstream systems of the application. All these systems communicated using Queues. For our end to end tests we read from the queue to begin the workflow and wrote onto the queue after the test run. As a workaround for getting node libraries to work with Cypress, we had to deal with promises within promises. This made the test failures very hard to debug.
These were significant blockers for using Cypress on that project.
So we moved to Taiko. Unlike Cypress, Taiko runs outside the browser as Node.js module and uses Chrome Devtools protocol to automate the browser. With the plugin architecture it was easy to build features missing in Taiko that were available to the team with Cypress.. Taiko-screencast and Taiko-screeny are examples to capture video and screenshots of every action respectively.
Why Taiko worked
Taiko does not impose any cross-origin restrictions because it runs outside the browser. Taiko also supports opening and managing multiple tabs, windows or incognito windows.
Aspects to consider before using Taiko in client projects
- Taiko has a REPL, where we use Taiko’s API as commands to automate the browser.
- Taiko does not run on Browserstack or Saucelabs.
- Taiko supports only the browsers that are CDP compliant, which at this point are Chrome, Microsoft Edge and Firefox.
- Taiko’s plugin architecture makes it easy to write plugins that suit your project requirements - examples of Taiko plugins.
As there were many open source testing tools, I was skeptical when I first heard of Taiko. Why create a new tool when we can contribute to existing ones?
I’ve come to realise that each tool has its strengths and opinions, some work for your project and some don’t.
After using Taiko on my projects, I started contributing features and bug fixes. I am a core committer now. I like Taiko's approach to test automation.
Please check out the plugins I’ve authored!
- taiko-android - A plugin to run web tests on android devices and emulators using Taiko.
- taiko-diagnostics A plugin for taiko which provides some diagnostics features like measuring speedindex, performance metrics of web pages.
- taiko-screeny A taiko plugin to capture screenshot on every action.
You can write your own plugin and become a contributor to the Gauge community.