Cross browser testing with non-functional tests

Ensuring cross browser compatibility is a non functional requirement (NFR). Cross browser requirements do not define the application's behavior but defines where the application will be used.

A functional requirement defines what a system is supposed to do. A NFR specifies criteria that can be used to judge the operation of a system, rather than specific behaviors. - wikipedia.

In my experience as a developer and tester, I've seen a lot of teams hoping to uncover cross browser issues by only running their automated UI tests across browsers.

Not all cross browser compatibility issues are caught by functional tests. In this post we look at ways to ensure cross browser compatibility without relying on functional tests.

Deal with Non Functional Requirements earlier

The plan for implementing non-functional requirements is detailed in the system architecture, because they are usually architecturally significant requirements. - wikipedia.

Cross browser requirements have to be taken into account by developers before the development begins as it’s architecturally significant.

Most probably everyone uses some kind of framework which already checks for compatibility. Javascript frameworks like ReactJS, Vue.js, JQuery, and UI frameworks like Bootstrap, Tailwind test for cross browser compatibility.

"React implements a browser-independent DOM system for performance and cross-browser compatibility. We took the opportunity to clean up a few rough edges in browser DOM implementations." - React JS Docs
"Vue.js supports all browsers that are ES5-compliant (IE8 and below are not supported)" - Vue.js Browser Compatibility
"Bootstrap supports the latest, stable releases of all major browsers and platforms. On Windows, we support Internet Explorer 10-11 / Microsoft Edge."
Bootstrap browser devices

It is better to invest in the right architecture for your application than waiting for issues to be reported.

Write non functional tests for Cross browser compatibility

There are three main components to how a browser renders a web page.

HTML - handles the static structure of the page
CSS - is the way we present the information with styles and visual layout
Scripts - add behaviour to UI elements.

The functionality of the application is the same across browsers. To ensure that the application works as expected in the browser/s you have to ensure that the CSS, HTML and JavaScript features work across browsers. This can be done by static analysis of front end code without running the application.

“Static program analysis is the analysis of computer software that is performed without actually executing programs, in contrast with dynamic analysis, which is analysis performed on programs while they are executing.” - wikipedia

We can utilise some tools to cross-reference our code against the compatibility databases, targeting only the browsers and versions we are interested in. - cross browser testing, without the browsers

Let’s look at visual regression analysis and scripting analysis to understand better!

Visual regression

The visual regression’s intention is to ensure the app appears as its designers intended. Browser automation frameworks used for functional tests ignore layout issues or UI regression.

Look at this UI. How did this happen in production? My guess is that they have test automation for this. But guess what…all of the expected text is there. So, the tests are passing because traditional assertions would never catch this. - Tweet by Angie Jones

Crossbrowser testing with Non-functional tests - Tests pass with UI

This example may seem exaggerated, but possible, because the functional tests pass by performing actions on the structure page (HTML selections using XPath) even if there are visual defects or if the app is visually unusable. Clearly functional tests do not ensure CSS and HTML browser compatibility.

Static code analysis reduces visual regression defects. CanIUse has gained popularity among front end developers. The incompatibilities are when front-end developers want to adopt new features. If you are not aware of the browsers to be supported, it's better to use features that are tried and tested.

CSS Checking with Stylelint, ESLint for CSS

Stylelint-no-unsupported-browser-features is a plugin that allows configuration of target browser/s and errors are reported with the list of browsers that do not support a feature. It uses doiuse to detect browser support. Stylelint is a peerdependency of this plugin.

Crossbrowser testing with Non-functional tests - CSS plugin

Scripting

When functional tests are used to ensure cross browser compatibility, it increases the amount of time needed to triage and analyse the failures.

Functional testing tools act as mere aggregators of test results from various browsers and leave it up to the end user to get actionable insight. Functional tests fail when a test script cannot not perform an action, but it won't tell you the root cause of failure. For instance, it will not tell you that the failure is because some scripting feature used in the application code isn’t supported by a relevant browser.

To get that specific feedback, plugins like ESLint-plugin-CanIUse can help.

Checking JavaScript with esLint against caniuse db

The ESLint-plugin-CanIUse plugin allows configuration of target browser/s, percentage of feature supported. Errors are reported with the list of browsers not supported and the percentage of the feature.

Crossbrowser testing with Non-functional tests - ESLint plugin compat

Transpilers

Another option is using a transpiler like Babel. Babel allows you write code in the latest version of JavaScript and transpile to a version that’s backward compatible for browsers that don’t support newer features.

Impact of standards and consolidation of browsers

In the early days, JavaScript features weren't standardised across browsers like IE and Firefox. We find various articles talking about the support for older browsers. There is a lot of information available to address these issues.

Testing frameworks were created to specifically target this issue. However, in recent times browsers follow standards. Polyfills, shims and Modernizr are only used in projects where old browser versions need to be supported.

By the time we started to build Taiko, browser vendors agreed to standards. With browsers adhering to standards, using functional tests to check cross browser compatibility is not just costly, but also redundant.