👈 Home

What browsers and devices should you test during experiment QA?

QA is a boring, time-consuming, essential part of experimentation. How can we make it as efficient as possible while minimizing the risk of breaking the internet, or (even worse) getting unreliable test results?

Use the right tools

To do QA right, you’ll need 3 things:

Choose the right devices

Using Google Analytics, determine the following:

Your purpose here is to identify the browsers and devices you should always check (because lots of your visitors use them), and hopefully a few devices and browsers you don’t need to check (because very few visitors use them).

An example of browser / device triage in Google Analytics

An example of browser/device triage

QA in the right order

Our hope during QA is that everything will be perfect. But our goal is to discover any and all defects as quickly as possible. For that reason, consider the following process:

  1. QA in Chrome, and Chrome DevTools Device Mode. Try different device sizes, and run through your entire QA checklist. If there’s an error you can catch in a single browser, there’s no sense opening up 10.
  2. QA in “problem browsers”. This means Internet Explorer first (if it’s on your list), followed by one each of iOS and Android. If there’s an IE compatibility bug, or an issue that pops up on mobile only (but not in Device Mode), you’ll catch it here.
  3. Complete QA in the remaining browsers.

That’s it! There’s no secret trick to cut QA time in half, but by being selective and systematic, you can shave off minutes per session while shipping fewer broken experiments.

(If you have any tips to make QA quicker, please hit reply and tell me about it. Everyone I know is struggling with this topic. Let’s save each other some time so we can work on exciting things.)

    © 2024 Brian David Hall