August 23, 2019 | Soumya Swaroop Gupta, Nivedha Senthil

Taiko Comparison

How Taiko compares to other browser automation tools

ThoughtWorks has been writing free and open source automated browser testing tools for over 15 years. Selenium RC was released in 2004, and WebDriver in 2007. Today, there are a lot of free and open source tools to automate browser based tests. Unfortunately, problems like flakiness and the cost of test maintenance have plagued this category of tests for a long time.

Over time, as users, we have opted for workarounds to deal with some of these issues. But sometimes all it needs is a different approach.

This year, the Gauge (by ThoughtWorks) team released Taiko 1.0, a simple NodeJS library to address some of the problems of browser-based automated tests.

In this post, we’ll compare Taiko with five popular open source browser automation tools

  • Selenium
  • Webdriver IO
  • Testcafe
  • Cypress
  • Puppeteer

Since we are a part of the team that built Taiko, our intention to compare is to discuss our rationale behind each point of comparison (along with examples) and how we think Taiko’s approach helps in each instance.

While analysing, we considered these parameters for a holistic comparison

  • Test maintenance
  • Reliability (reduced flakiness)
  • Performance
  • Test frameworks integration
  • Ease of test failure analysis
  • Cross Browser support
  • Language support

1. Test Maintenance

The goal: Tests should change only when there is a change in functionality. A change in the structure of a web page should not impact a test validating functional correctness.

However, most browser automation tools create tests that are hard to maintain. That’s because they use locators like XPath, ID's, CSS selectors to identify and perform actions on web page elements.

“Maintaining locators must be calculated as part of test maintenance cost.” - FireFox Test Engineering blog

These locators depend on the structure of a web page. The underlying structure can (and does) change frequently throughout the development process. Even minor changes in page structure break such tests. So, it increases the test maintenance costs.

Taiko’s approach

Taiko allows you to select elements on the page based on what you see. It's not based on the specific underlying structure that only a developer sees.

This means, to click that "Purchase" button you can use click( "Purchase"). It's based on the text on the screen and not based on the specific style, location or hidden identifier.

Multiple matches can be resolved with proximity selectors or a combination of them. Yet, if this is unsuitable, Taiko has fall back options. You can choose to select an element on the web page via XPath or CSS selector.

Benefits

Taiko’s API treats the browser like a black box. Tests written in Taiko are resilient to page structure changes as they minimise the need of using complicated locators.

Compare with examples

This example compares scripts automating the TODO MVC application. The tests currently pass for the React flavor of the TODO MVC. However, only tests written in Taiko pass when they are run against the AngularJS flavor of TODO MVC. This is because Taiko focuses on testing functionality and not the underlying page or framework.

Key Findings

2. Reliability (reduced flakiness)

The goal : Eliminate the root cause of flakiness in browser based tests.

In modern web applications, elements on the page can dynamically appear and disappear based on user actions. For example, imagine clicking on a user's icon to display their photos.

However, many libraries expect the test code to handle the wait time to perform an action. Other tools provide some granular and low level controls to explicitly wait for elements or time however, it's hard to learn and use these controls.

“1 in 7 of the tests written by our world-class engineers occasionally fail in a way not caused by changes to the code or tests.” - Google Testing blogs

Explicit waits make test code unpredictable. Improper configuration, inappropriate use of APIs along with ineffective handling of waits by the tool makes tests flaky.

Taiko’s approach

As shown below, both Taiko and TestCafe have very good implicit wait mechanism before performing actions. This reduces flakiness in tests. However, unlike TestCafe, Taiko also ensures that the performance of the test run remains good!

In case you want finer control, Taiko also has good fallback options to override the default behavior. You can define explicit wait for conditions for a given action in the test code to suit your needs.

Benefits

Modern test libraries "await" the element to appear instead. This makes your tests run more resilient to things like slow network connections. Implicit waits increase predictability.

Compare with examples

In this example only Taiko and TestCafe don’t need any explicit waits. To observe flakiness, comment out all the explicit wait conditions in tests of other tools and run it a few times. A detailed comparison is available in the readme file inside the example.

Key Findings

3. Performance

The goal: Get fast and reliable feedback on test failures

“Usually the concern is application performance, which can have a direct impact on the bottom line. Testing performance, on the other hand, has a direct impact on developer productivity.” - Why your choice of software testing suites matters

Slow test suites increase feedback time and time to fix, when something breaks. Teams generally tend to ignore slow test suites.

Taiko’s approach

Taiko explicitly chooses reliability over extreme performance. If this tradeoff is unsuitable, you can choose to override it to improve performance. Explicit waits can be defined for a given action.

Benefits

  • Fast and more importantly, reliable feedback helps improve developer productivity. Flaky tests can be difficult to work with. You not only waste time analysing them but also genuine failures are re-run to check for flakiness. So as important as it is to be fast, it’s important that tests are reliable.

  • Run tests in parallel for free. Running parallel tests in Cypress is a paid service. Unlike Cypress; Selenium, Puppeteer and Taiko tests can integrate with any test framework like Gauge, Mocha, and Jest. These can then use the framework’s parallel execution feature for free.

Compare with examples

In this example, we ran all the tests sequentially, multiple times, to compare performance across tools. Machine and other details of the run are available in the readme file of this example.

A shorter time and lower CPU indicate better performance. Here are some results of benchmark tests run in cpu with 8 core.

Key Findings

Testcafe’s execution time could be 16 seconds if there were no reloads between tests. Reloading added an additional 4 seconds to the test suite.

Some observations

  • We added waits for a time period in Selenium tests while measuring the performance. We found that it required a lot more effort to write optimized code (wait for required conditions) to reduce flakiness. Further optimisations can reduce the time taken for execution.
  • Since it is difficult to integrate TestCafe and Cypress with other test frameworks, they have to depend on their inbuilt parallelization support. WebdriverIO supports parallelization.
  • Running parallel tests in Cypress is a paid service.
  • Further optimisations can reduce the time taken for execution in Taiko. Users can choose to override implicit waits for a given action and define it explicitly in test code.

4. Test framework integration

The goal: A testing library like Taiko must be able to take advantage of the rich testing oriented feature set provided by modern test frameworks like Gauge, Mocha, and Jest.

Taiko's approach

Taiko is a simple NodeJS library that can easily integrate with any Javascript test frameworks.

Benefits

  • Avoiding vendor lock-in and increasing the portability of tests.
  • Available for free. E.x., Gauge’s reporting, parallel execution, IDE support, integration with CD workflows, and more.

Key Findings

5. Ease of test failure analysis

The goal: Tools must make it easy to collect contextual data to analyse test failures.

“The analysis part is normally done manually and if the analysis is not done correctly, there could be genuine failures that are overlooked or masked by other issues.” - 5 reasons why automated tests fail to find regression bugs

Failure analysis feature allows testers and developers to quickly find issues and fix test failures. Contextual data allows easier test failure analysis.

Taiko’s approach

Apart from the default data available on failure, more contextual data can be collected with custom plugins.

Taiko’s plugin architecture allows you to extend it in ways that suit your requirements.

Benefits

Users can use/build a plugin to suit the requirements to collect relevant data. Example:- screencast plugins capture video of the current run.

Key Findings

  • Since Selenium, Taiko and Puppeteer can integrate with any test framework they can take advantage of the framework’s debugging, IDE, CI/CD support and more.

  • The paid services of Cypress have more options for test failure analysis in CI/CD environments.

6. Cross browser support

The goal: Ensure that tests run consistently across modern browsers.

Running tests across browsers adds significant overhead to both resources and test run times. We can avoid this overhead by choosing what to test. Modern Javascript applications, frameworks, shims and transpilers like babel do a good job of ensuring cross browser compatibility.

Browsers are now starting to adopt standards for better web compatibility and less fragmentation of underlying web platforms

Taiko’s approach

Like Puppeteer, Taiko also uses the excellent Chrome DevTools Protocol(CDP) to automate browsers. Both work seamlessly with Chromium based browsers.

Benefits

The next version of Microsoft Edge is adopting the Chromium open source project. The Firefox team is working on adding support for CDP.

Taiko scripts are standards compliant and portable once browsers adopt these standards.

Below are the details of our key findings of the browsers supported with various tools.

Key Findings

7. Number of languages supported to author test code

The goal: The language available for authoring tests should allow users to write tests expressively.

All the tools we’ve chosen for comparison, support either one or more programming languages to write tests. With first class support for commonly used programming languages, these tools can leverage the team’s capabilities as well as IDE support.

Taiko’s approach

Taiko only supports Javascript to author test code, because it’s built on top of CDP (Chrome DevTools protocol) and the best library for working with CDP is in JavaScript.

Benefits

  • It works well with the Node.js ecosystem and frameworks like React, Angular and Vue.js
  • The team doesn’t have to learn a new language if they are building web applications
  • Taiko can automate any part of the browser

Key Findings

Other Observations

  • Cypress has a paid service that gives you access to recorded tests when running Cypress tests from your CI provider. The dashboard provides insights about test runs.
  • WebDriverIO has support for integration with cloud platforms to aid cross browser testing under its cloud services.
  • Puppeteer has the best performance. However, testing is not the focus of Puppeteer. It is widely used for web scraping. When used to write tests, the learning curve is high.
  • Taiko unlike a UI recorder, Taiko’s REPL takes instructions given in the terminal and performs the action on the browser. The user can continue to give instructions to complete a workflow. Only successful actions are recorded as a script. This generates human readable code and keeps the learning curve low.

Summary

Here is the report of the tools comparison

Taiko Infographic

Check CompareBrowserAutomationTools, a GitHub repository to validate our claims made in this blog.

We hope you find this comparison summary useful! Through various examples listed above we can deduce that Taiko is a reliable, cost effective browser automation tool with a good performance. There will be pros and cons of using any tool. If you’ve come across a tool that isn’t in the comparison but solves a problem better, leave us a comment.

As always, the team welcomes any feedback that helps improve Taiko. You can install Taiko and explore more!