But have you ever heard of the Mobile Testing Pyramid?
I hadn’t either.
Until I recently spoke with Kwo Ding about his take on the Testing Pyramid as it relates to his experiences with automated mobile testing.
So what is a mobile testing pyramid?
Kwo came up with this concept after years of performing mobile testing. He had noticed some patterns he thought could help testers think better about their mobile testing strategy.
Like Mike Cohn’s pyramid, the Mobile Testing Pyramid is made up of three layers: Browsers, Simulators/Emulators, and Real Devices.
The Bottom Browser Layer
The base, or bottom layer is comprised of desktop browsers like Chrome, Firefox, Safari, etc. These are great to help you simulate some mobile behaviors like responsive design. This approach also allows you to inject user agent string so that you can pretend you’re a mobile device.
Because it’s the bottom layer (just like with Unit Tests in the Cohn pyramid), it means they’re the fastest tests that offer the quickest feedback.
Also, this approach allows you to scale up your tests, since you can run multiple test instances on one machine.
Middle Layer: Simulators/Emulators
The middle layer of the Mobile Testing Pyramid is Simulators and Emulators. Kwo says this one of the most overlooked layers of the pyramid. If you remember back in the early days of mobile testing simulators and emulators were very unstable. Nowadays, however, they have become much more mature and stable.
This layer is a great place to validate some touch interactions your mobile app might have as, well as visual validation testing.
Another plus to using simulators and emulators is that they tend to be easy to set up and run against.
Top Layer: Real Devices
The top layer is made up of Real Devices. This segment should be the smallest percentage of your overall mobile testing suite. The best types of tests to create at this layer are ones that focus more on usability or real-life condition tests.
Risk: Does Everything Need to Run Against a Real Device?
I’ll be honest; this is not how I’ve been accustomed to thinking about mobile testing. I always assumed it would be better to run tests on a real device. But Kwo reminded me that this is the same fallacy that we ran into early in Web testing when everyone just started automating from the UI.
Now, thankfully, we know that many of these tests are better handled and more reliable when covered at the unit or integration level. GUI tests should be the last resort for automation.
This is the same with a mobile device test. Always start at the base first and only use real devices for tests that actually require a real device.
Remember Faster Test Feedback is Key
Running tests on real devices can be the slowest type of mobile testing on the pyramid. As we move toward CI/CD, it’s important that we give feedback to our developers as soon as possible.
Focusing more of your mobile testing on the Mobile Testing Pyramid outlined by Kwo is a great place to help get you to meet that goal.