What browsers and devices should you test during experiment QA?
May 2019
QA is a boring, time-consuming, essential part of experimentation. How can we make it as efficient as possible while minimizing the risk of breaking the internet, or (even worse) getting unreliable test results?
Use the right tools
To do QA right, youâll need 3 things:
- Chrome DevTools Device Mode. This allows you to simulate mobile devices and test different screen sizes right within your browser.
- A real device testing platform. Unless you have 20 mobile phones laying around your office, a service like BrowserStack is crucial.
- The mighty checklist. Outlining exactly what youâre looking for and verifying during QA will reduce errors, and opens up the possibility of delegating or collaborating on this onerous task.
Choose the right devices
Using Google Analytics, determine the following:
- What proportion of visitors are on Desktop, Tablet, and Mobile? If you have very little traffic on mobile, you might be better off excluding it from tests. This will speed up development and QA. Most of us wonât have this luxury, though.
- On Desktop, what browsers do you have to check in order to cover ~80% of users?
- On Mobile, what devices do you have to check in order to cover ~80% of users?
Your purpose here is to identify the browsers and devices you should always check (because lots of your visitors use them), and hopefully a few devices and browsers you donât need to check (because very few visitors use them).
An example of browser/device triage
QA in the right order
Our hope during QA is that everything will be perfect. But our goal is to discover any and all defects as quickly as possible. For that reason, consider the following process:
- QA in Chrome, and Chrome DevTools Device Mode. Try different device sizes, and run through your entire QA checklist. If thereâs an error you can catch in a single browser, thereâs no sense opening up 10.
- QA in âproblem browsersâ. This means Internet Explorer first (if itâs on your list), followed by one each of iOS and Android. If thereâs an IE compatibility bug, or an issue that pops up on mobile only (but not in Device Mode), youâll catch it here.
- Complete QA in the remaining browsers.
Thatâs it! Thereâs no secret trick to cut QA time in half, but by being selective and systematic, you can shave off minutes per session while shipping fewer broken experiments.
(If you have any tips to make QA quicker, please hit reply and tell me about it. Everyone I know is struggling with this topic. Letâs save each other some time so we can work on exciting things.)