Mobile App Testing Strategies

To put any idea into a practical action plan, you need a strategy. Same applies for software testing too. Software application development for mobile devices has increased immensely in the last decade and is expected to continue growing at a profound pace.

Mobile devices have become so ubiquitous the number of expected people with a device is expected to reach 5.5 Billion by 2021 (70% of the expected population). All these users of mobile devices — like smartphones, tablets, and even e-readers, create a never-ending demand for more and more unique and useful mobile applications.

These users have grown accustomed to a high level of quality mobile applications on these devices that just one nagging bug or usability issue can spoil the entire user experience. With so much competition in the mobile app market, ensuring the quality of your product is imperative to retaining market share and to avoid watching your users flock to the competition.

To ensure the delivering of quality of mobile applications to these users in fulfilling their necessity and needs, testing these applications in a strategic manner is key. Testing a mobile application is often challenging as compared to web application mainly due to the various factors affecting the behaviour of the application. Some of the factors including operating system versions, different screen size and network availability.

Test scripts or test cases are critical to the overall success of any testing strategy. In one of my meetings I once heard a manager ask, “What so difficult about testing? Just make sure everything works.” This is a perfect example of a complete lack of understanding of testing and how it needs to be done. You can’t simply “test everything” — it’s not the way things are done. As you’re improving or implementing a feature or releasing a new piece of functionality, you need to have clear understanding of what exactly is implemented or being released to ensure testing is done in a logical and systematic manner. Test scripts helps in providing clear understanding of feature to the team. Please make sure we have test scripts in place before start testing the application.

Another strategy which make sures the users are very happy with the applications is “Functional testing”, it tries to answer two basic questions: “Can a user complete a task?” and “Does this feature really work?” A tester will go through the flow that is being changed to ensure that the user can go back and forth through all the steps in the particular flow that is being built. The goal is to ensure that — from a purely functional point of view — all of the steps work as expected and there are no flaws in the process. For example, ending up on a mobile page without any way to return to either the previous page or to exit the flow is considered a defect in functional testing.

Another critical step in mobile apps testing is to ensure that the user experience that was built by the UX engineer and the designer matches its final implementation. I cannot recall a single feature that I’ve managed throughout my career which passed this test right off the bat. For various reasons (some intentional, some not), developers will frequently fail to implement the functionality in the exact way that the designers outlined. In many cases it takes highly-skilled testers to catch these discrepancies. The variations between specifications and the working software are fairly small, but still noticeable: there may be different fonts between visuals and the final code, different styles, error handling that doesn’t match the designs, incorrect copy, etc. An experienced tester must have an eye for this type of detail and must be able to catch these issues immediately.

The simplest rule of performance testing is, with the addition of a feature, you want to make sure that you haven’t added any delay in the overall performance of the flow that is being changed. In the context of mobile apps, what you are monitoring is the overall speed and responsiveness of the app when a new feature is added. You are comparing these metrics against the current production flow (before/after feature performance analysis) and determining whether the new feature has any impact on the overall device and the battery performance of the device.

Even the smallest code change can have unintended consequences for the overall application, often in ways you wouldn’t have dreamed possible until you actually see it in action. Regression testing makes sure that where a small change is made, the overall flow still works as expected once the development of the feature is completed. As an example, even a small change in an hybrid application shouldn’t effect the feature working on other platforms. That’s why regression testing is critical — to ensure that the end-to-end flow that is being touched by a developer still works after a change has been made.

Functional testing, Load testing, Performance testing, Regression testing — all of these test types can be fully automated, and it’s great. A computer won’t recognize if there’s a discrepancy between a designer’s specs and the final experience built by a developer, but it can easily test and document anything in the flow that is not working as expected. When a developer is done making a change they can easily run the testing script overnight and return to work to see a list of issues that the testing software identified. Personally, I strongly recommend doing research and choosing an automated test solution. It saves time and money, it improves accuracy and it can help developers identify and fix issues before they ever release a build for testing. Though certain testing tasks will never be replaced by a computer / software, there’s an awful lot of work that can be done using automated testing tools.

There exists a lot of mobile devices with a different type of screen sizes and operating systems. Stating some of the statistics, there exist about 24,000 devices which have different screen sizes but the same OS, i.e., Android. Android devices literally come in all shapes and forms, so deciding which devices and what screen sizes an application should be optimized for is critical to the overall success of the testing strategy. In any organization, it is next to impossible to make an application optimized for all of these devices. Therefore, an organization should set a list of the devices it is going to support. The device selection should be strictly related to your target market. Plan to test on those devices that have high coverage in the market you want to distribute your app in. You can use different criteria to find the most suitable devices to test on. You can analyze the most sold devices in that particular market or the most used screen sizes and OS versions. When you have collected all those information, you can start by choosing 5–10 devices that are most relevant. You can always expand the testing devices later on. This makes the testers follow the rules and ensure that the application supports all of the listed devices.

What’s annoying about mobile apps testing is not limited to the fact that the device market is incredibly fragmented. It’s also that both iOS and Android have a significant number of operating system versions that exist in the market. Granted, iOS is significantly better than Android in this regard and has only 3 versions of the OS that are supported — which is one of the many reasons why we suggest that companies develop their first application on iOS (source). By contrast, Android has at least 7 different OS versions that are still on the market in the US alone. To be fair, since Apple is both the manufacturer and the software provider for the iPhone they have full control of the number of versions available to their users, where Android doesn’t have full control of these versions. Instead it relies on manufacturers to push out an update. From a testing perspective this is a total nightmare, and I’m not even talking about the manufacturers’ versions of the operating system with their tweaks. This means that it is very important for your company to have a testing policy in place that clearly calls out which versions of Android / iOS are supported so that testers can check every new functionality in each of the operating systems.

As we all know, various carriers provide different connectivity options to their customers: Anything from 3G to LTE. So one additional thing testers need to take into account is how the performance of a feature may differ from one carrier to the next. For example, app performance testing shouldn’t exclusively be conducted on WiFi. You need to specifically test your app on actual 3G, 4G and LTE mobile communication standards to see how the application performs. Additionally, you want to make sure that you test what happens when connectivity is lost or suddenly shifts from 4G to 1G. Does the app has offline support? Any limited features when offline? How does the application behave? What is the user expected to do? Covering all these scenarios keep your app users happy and stick to the app.

The question that arises is what exactly are interrupt conditions? Interrupt conditions is referred to any condition when there’s an incoming call or SMS while you’re using the application that needs to be tested. The user is doing something on your app, and through no fault of theirs (they weren’t trying to leave the app), something interrupts its operation. The question becomes “How should the app behave when the interruption is over?” (e.g., I hang up the phone). Testers should always check what happens when interruptions occur, and they should report these issues to the product teams to ensure that a proper course of action to prevent customers from getting frustrated.

Security testing on a mobile application is a must-have these days, as the stats are scary. 90% of the applications has tested as part of their methodology had at least 2 critical vulnerabilities. 50% of organizations have zero budget dedicated to security testing. Following are the minimal things to check with respect to security testing.

This article provides few best testing strategies that can be employed in the process of testing a new feature on a mobile application. Though this is not the clean work, having a robust and mature testing strategy is key to your application’s success. At the end of the day, the reality is that most mobile users are unforgiving. If they see a mobile app constantly taking more than 3 seconds to load they will abandon it or if a flow is broken and prevents them from proceeding with the task at hand, they will thrash your app. In their mind, you are only as good as your app’s ability to allow them to complete the tasks they had in mind when they tapped on your icon. That’s why QA testers are the hidden heroes of mobile app development. They make sure that your shiny new application can actually deliver on the promise you make to your customers in a smooth and convenient manner. Follow above strategies and you’ll have 99.99% bug free mobile apps.

Related posts:

Why do they words we need seem to escape us just at the moment we need them? What would happen if we were brutally honest? Speaking our minds without the self censoring that we often use to make…

Posting a book on ForgeFiction is easy, but making it appealing for the readers and potential co-writers can be a tough job. Sometimes your book can get lost among others, but it shouldn’t discourage…

Blockchain-powered idle role playing game Brave Frontier Heroes is having a very successful launch with already over 168 thousand dollars in trading volume. This number is from twelve hours after…