Sep 10, 2020 | Soumya Swaroop
Cross browser testing with non-functional tests
In my experience as a developer and tester, I've seen a lot of teams hoping to uncover cross browser issues by only running their automated UI tests across browsers.
Not all cross browser compatibility issues are caught by functional tests. In this post we look at ways to ensure cross browser compatibility without relying on functional tests.
Deal with Non Functional Requirements earlier
Cross browser requirements have to be taken into account by developers before the development begins as it’s architecturally significant.
- "React implements a browser-independent DOM system for performance and cross-browser compatibility. We took the opportunity to clean up a few rough edges in browser DOM implementations." - React JS Docs
- "Vue.js supports all browsers that are ES5-compliant (IE8 and below are not supported)" - Vue.js Browser Compatibility
- "Bootstrap supports the latest, stable releases of all major browsers and platforms. On Windows, we support Internet Explorer 10-11 / Microsoft Edge."
- Bootstrap browser devices
It is better to invest in the right architecture for your application than waiting for issues to be reported.
Write non functional tests for Cross browser compatibility
There are three main components to how a browser renders a web page.
- HTML - handles the static structure of the page
- CSS - is the way we present the information with styles and visual layout
- Scripts - add behaviour to UI elements.
“Static program analysis is the analysis of computer software that is performed without actually executing programs, in contrast with dynamic analysis, which is analysis performed on programs while they are executing.” - wikipedia
We can utilise some tools to cross-reference our code against the compatibility databases, targeting only the browsers and versions we are interested in. - cross browser testing, without the browsers
Let’s look at visual regression analysis and scripting analysis to understand better!
The visual regression’s intention is to ensure the app appears as its designers intended. Browser automation frameworks used for functional tests ignore layout issues or UI regression.
Look at this UI. How did this happen in production? My guess is that they have test automation for this. But guess what…all of the expected text is there. So, the tests are passing because traditional assertions would never catch this. - Tweet by Angie Jones
This example may seem exaggerated, but possible, because the functional tests pass by performing actions on the structure page (HTML selections using XPath) even if there are visual defects or if the app is visually unusable. Clearly functional tests do not ensure CSS and HTML browser compatibility.
Static code analysis reduces visual regression defects. CanIUse has gained popularity among front end developers. The incompatibilities are when front-end developers want to adopt new features. If you are not aware of the browsers to be supported, it's better to use features that are tried and tested.
CSS Checking with Stylelint, ESLint for CSS
Stylelint-no-unsupported-browser-features is a plugin that allows configuration of target browser/s and errors are reported with the list of browsers that do not support a feature. It uses doiuse to detect browser support. Stylelint is a peerdependency of this plugin.
When functional tests are used to ensure cross browser compatibility, it increases the amount of time needed to triage and analyse the failures.
Functional testing tools act as mere aggregators of test results from various browsers and leave it up to the end user to get actionable insight. Functional tests fail when a test script cannot not perform an action, but it won't tell you the root cause of failure. For instance, it will not tell you that the failure is because some scripting feature used in the application code isn’t supported by a relevant browser.
To get that specific feedback, plugins like ESLint-plugin-CanIUse can help.
The ESLint-plugin-CanIUse plugin allows configuration of target browser/s, percentage of feature supported. Errors are reported with the list of browsers not supported and the percentage of the feature.
Impact of standards and consolidation of browsers
Testing frameworks were created to specifically target this issue. However, in recent times browsers follow standards. Polyfills, shims and Modernizr are only used in projects where old browser versions need to be supported.
By the time we started to build Taiko, browser vendors agreed to standards. With browsers adhering to standards, using functional tests to check cross browser compatibility is not just costly, but also redundant.