Stop guessing at devices "We tested on twelve devices" tells nobody anything. Twelve could be generous or negligent depending on which twelve. A device matrix built from your own traffic turns coverage into a number you can defend, and it usually costs less than the scattergun approach it replaces.
1. Start with your analytics, not a top-ten list
Pull the real breakdown for the last ninety days: browser and version, operating system and version, device model or vendor, screen width buckets, and the split by country. Google Analytics, your app store console or your server logs all carry enough. Compare it with worldwide market share only as a sanity check, never as the source, because your audience is not the world's average.
2. Rank two axes separately
One axis is platform version, stacked by share until you reach your coverage threshold. The other is vendor or browser engine, ranked by share in the regions you care about. Keep them separate; combining them too early hides the fact that one vendor's skin behaves differently on the same version.
3. Tier by risk, not by count
- Tier 0: the top two platform versions crossed with the top two vendors or browsers. Roughly four combinations, usually covering forty to fifty per cent of sessions on each axis. Always real devices. A failure here blocks the release.
- Tier 1: the next three or four pairs plus one deliberately older device. Together with tier 0 this normally passes seventy per cent of your traffic. Emulators are acceptable here for pure layout and logic checks.
- Tier 2: form factors rather than share. A tablet, a foldable, a low-memory budget phone, and anything specific to a region you sell into.
Write the percentage next to each tier. That single habit converts an argument about device counts into a conversation about acceptable risk.
4. Where an emulator is honest and where it is not
Emulators and simulators are fine for layout, business logic, API-level behaviour and quick reproduction of a known bug. They cannot be trusted for vendor firmware behaviour, permission dialogs, background execution and battery optimisation, real camera and sensor input, biometric prompts, push delivery, genuine network conditions, or performance and memory pressure. If a bug only matters on a real handset, an emulator will tell you everything is fine.
5. Browsers deserve the same treatment
Three engines cover the web: Blink, WebKit and Gecko. Testing Chrome and Edge together is largely testing Blink twice. Real Safari on a real iPhone is not optional if you have iOS traffic, because iOS Safari is the engine behind every browser on that platform. Add one older major version if your analytics show a meaningful tail.
6. Refresh the matrix on triggers, not on a calendar
Rebuild it when a version crosses ten per cent of your traffic, when a platform deadline forces an upgrade, when crash reports spike on something untested, or when a flagship launches in your main market. Twice a year is a reasonable floor.
Both of our QA options work from this matrix: the manual team runs the tier 0 and tier 2 devices by hand, and the automated suite sweeps the browser grid on every release. See software testing and QA or cross-browser and cross-device testing.
Quick checklist
- Pull ninety days of your own browser, OS, device and country data
- Rank platform versions and vendors on separate axes
- Build tier 0, tier 1 and tier 2 and write the coverage percentage beside each
- Run tier 0 on real devices, never on emulators
- Include one deliberately old device and one low-memory device
- Test real Safari on a real iPhone if you have any iOS traffic
- Rebuild the matrix when a version crosses ten per cent of your traffic
Frequently asked
How many devices is enough?
Coverage is the answer, not a count. State the percentage of your own sessions the matrix covers. Twelve to sixteen well-chosen devices usually passes seventy per cent of traffic for a consumer product.
Can we just use a cloud device farm?
For browsers and for reproducing known issues, yes. For permission dialogs, background behaviour, battery, camera and real network conditions, a physical device in a tester's hand still finds things a farm does not.
Do we need to test old browser versions?
Only if your own analytics show a meaningful tail. Testing versions nobody uses costs the same as testing versions everybody uses, and finds nothing.