Angrib med taxonomier og heuristikker: nye overvejelser om mobiltest

Foredrag, konferencer og networking giver oftest megen inspiration, og i dag fik jeg en smule af slagsen efter at have overværet Local Rockstars til et en-dags Internet Week of DK inspiration event i Aarhus. Specielt QA & UX spor har været interessante, ikke mindst i lyset af fokus på udvikling til mobile platforme.

Som QA har jeg haft fokus på kvalitetssikring af mobile apps, og det var derfor interessant at høre et par andre bud på hvad er vigtigt at have fokus inden for dette. Der blev snakket om flere gode ideer og specielt erfaringer fra firmaer som beskæftiger sig med udvikling og test af mobile apps. Mange mobile defekter skyldes antagelser (det vil virke på alle andre devices), forglemmelser eller manglende indsigt i mangfoldigheden af situationer som brugere kan stå i når mobile løsninger bruges. For at gøre op med dette, bliver erfaringer med udvikling og test af mobile løsninger tit delt i form af tips, tjeklister og best practices. Dette var også tilfældet, i hvert fald på QA sporet af Lokal Rockstars, og jeg blev mindet om at bestræbelserne på at opnå højere kvalitet tit har haft denne form for vidensdeling.

Ganske for nyligt fokuserede jeg selv på fejlbaseret tilgang til kvalitetssikring af mobile apps, da jeg skrev (på engelsk) Basic patterns for QA of mobile apps. Denne kan betragtes som en en taxonomi og flere før mig har gået i sporet af “How to break software” bogen i forsøg på at skabe noget tilsvarende ud fra mobiltestvinklen. Bl.a er der Software Test Attacks to Break Mobile and Embedded Devices som jeg er blevet opmærksom på siden jeg skrev blogindlægget, og som er kommet på min ønskeliste (det er jo snart jul)

Fejlbaserede angreb med fokus på brugeradfærd er en god start, men god kvalitetssikring kræver mere end det. Den stigende fokus på sikkerhed og user experience (UX) er ikke til at overse, og disse er tit temaer når der snakkes om mobiltest. At netop disse er blevet toneangivende kunne skyldes at det er i sig selv en konkurrenceparameter at have styr på netop de to områder når det gælder mobiludvikling. Funktionaliteten med andre ord ikke er nok hvis der ikke styr på fortrolige data eller brugeroplevelsen er ringe. Hvordan forholder man sig som tester til de to aspekter?

User Experience

UX er jo et domæne for sig selv, som tit bliver ledet af UX eksperter og designere. Som QA har man en god mulighed for at hjælpe med at finde UX problemer ved hjælp af sin nysgerrighed, struktureret tilgang (når man besidder sådan en) og andres erfaringer.

  • Som QA, er man også en (avanceret) bruger som kan finde alle afkroge og steder hvor navigationer mangler vigtige elementer
  • Ved hjælp af eksplorativ tilgang kan man finde alle de steder hvor det er uklart hvad brugeren forventes at gøre
  • Kendte heuristikker, som 10 Usability Heuristics for User Interface design kan man tage udgangspunkt i for at finde inspiration til mangler at lede efter
  • Tit findes der design guidelines, som udarbejdes af UX eksperter til udviklingsteamet. Disse kan bruges som tjekliste, og hvis man sætter en streg hver gang et punkt i dette dokument findes overtrådt og bekræftet som UX fejl kan man over tiden få en taxonomi over typiske fejl (som kan være et fint input til selvevaluering for hele teamet.

QA review af prototyper og tidlige versioner kan være en effektiv måde at forbedre brugeroplevelsen. (på lige fod med det arbejde som UX eksperter gør)

Sikkerhedstest af mobile løsninger

Typisk er sikkerhedstest set som noget der kræver ekspertise, og særlig talent for at finde sårbarheder. Som QA har man derfor mulighed for enten at udlicitere det til andre eller bruge kendte metoder og sikkerhedsfejlangreb.

Test af sikkerheden kan splittes i API/Backend delen på serversiden og App-kode som eksekveres på devices.

API kald kan være udsat, hvis de returnerer for meget data i tilfælde hvor de ikke burde. Værktøjer som Charles og Wireshark, Postman-udvidelsen til Chrome (og sikkert mange andre lignede) kan bruges til monitorering og fiflen med netværkskommunikation — således at systemet bag app’en kompromitteres. Hacking er en kunst i sig selv, og langt ud over scope af denne blog 🙂

Test af sikkerheden i App-delen af den mobile løsning kan også være et spørgsmål om brute-force søgning af huller. App-filer kan blive udsat for reverse-engineering, med mindre der er blevet gjort forsøg på at modvirke dette med obfuscation. I så fald det lykkedes den onde part at få læsbar adgang til koden, kan hemmeligheder som login informationer, nøgler og API URLs læses og bruges i API angreb.

Kommunikation er en vigtig aspekt af sikkerheden i Apps og ting som manglende brug af HTTPS hvor dette kræves, certificate and public key pinning og flere andre kendte “angreb” kan danne baggrund for endnu en taxonomi til sikkerhedstest af apps. ( Kilde )

Brug af værktøjer kræver stadig lidt teknisk snilde, men med rette hjælp fra udviklingsteamet kan man få sat et miljø op og prøve at lege med sikkerhedskritiske (informations og funktionsbærende) dele af systemet


Kvalitetssikring af mobile løsninger har mange sider og tilgange. Fra en simpel prototype til udrulning i App Store som massevis af brugere er der en vej med flere valg undervejs. Iblandt aspekter som er værd at overveje er test automatisering, device cloud, regressionstest, UX- og sikkerhedstest og ikke mindst det at lære din bruger at kende: undersøg adfærdsmønstre, konfiguration af hardware og OS, og måden at bruge apps på. Det er vejen til at forståelsen af brugeren; hvordan vil man ellers lave apps som vil blive elsket, hvis ikke man sætter sig ind i deres sko?

Som Hans-Henrik Olesen udtrykte det i dagens præsentation:

Don’t forget to move (your users do)

Reklamer

Basic patterns for QA of mobile apps

I’ve been a QA on mobile platforms for a couple of years now. The speed of changes in technology and constantly changing ways of appliance are just incredible! Mobile apps are usually considered to be small pieces of software in mobile environment created to perform specific functions. Yet, these apps are typically part of larger systems, consisting of API and integrations to other systems, special hardware (like Smart Things or WeMo) and ecosystem of other apps and services that evolve and change through time. It seems that quality assurance of these apps won’t be trivial. There can be lots of testing situations to take into consideration and it can be rather hard to test everything.

To cope with complexity, testing can be user-centered, focusing on cases and scenarios that most users will find eventually. High quality is seldom written in requirement specifications explicitly. Yet, the users will expect it and you might be surprised by the feedback and reviews that you will get if not taking care of situations affecting core functions of your core users.

Experience can be brought to help you with defining good tests. A good example is James A. Wittaker’s book “How to break software”, which can be a good  inspiration in finding alternative scenarios to try in testing. General concepts from this book work very well for mobile testing: you might test input and output capabilities using basic patterns and classic test design techniques. You may also explore ways to corrupt internal data and find inconsistencies in computations as all these are basic testable properties of software.

Besides general patterns, mobile testing needs to look at platform and technology specific error sources. I’ve been missing patterns, specifically related to the new advances in mobile technology. The new ecosystem (application stores like Google Play and Apple AppStore and their upgrading mechanisms) and mobile operating systems specifics affect the way we should test mobile apps. Based on my experience and other sources, I have put together some patterns I have found useful in mobile testing. This is not a checklist, but rather an inspiration to do better mobile testing.

If you prefer to get these as printable cards, you can get them here: downloads.

1. Apply real user’s navigation patterns

Purpose: discover and apply real user behavior, that is far more complex than simple test scenarios during the development. The specific method depends on functionality, specifically on what is possible within the app.

Patterns for navigation:

Test navigation elements in realistic and differentiated ways, that are not covered by sunshine scenarios in specs. Users press everything and usually not in the order you intended:

Technically speaking, this approach relates to deallocation of memory and asynchronous execution of code expecting something that is no longer there — execution may take too long while the user has already moved on to the next screen.

Things to play around with could be menu items, window tabs, links and buttons.

How to work with it:

  • Explore forth-and-back paths: explore in deep the longest paths to leaves of the navigational tree and go back all the way to the root
  • Try switching between navigation elements in varied ways: reverse the order and randomly select the next element to toggle
  • A more systematic way: use Process Cycle Test from T-Map to achieve optimal coverage guaranteed by the principle of this technique.

Patterns to apply for testing lists of items:

  • Try scrolling up and down the list. Look for: content inconsistencies, wrong pictures (could occur if  eg. picture containers are reused in code)
  • Open many items one after another, coming back to the list every time. Look for: memory issues leading to missing resources or a crash. Also, scrolling position on the list is usually supposed to be remembered,
  • Edit/delete list items and refresh the list to fetch content from the storage. Look for: inconsistencies, leading to wrong items being edited/deleted

2. Check upgrade scenarios

Purpose: upgrade an older version of your app to the newest one.

This includes:

  • Testing upgrade of the most used app versions out there. Usage statistics, such as those provided by Google Analytics can give you that information.
    • Testing a trivial upgrade scenario is not always enough!
    • Test both fresh install on clean device, and one with previously installed earlier version of the app (some data may be remaining)
    • Consider cases where user has a previous version with some local app data related to that version: settings, saved data pointers (like searches and favorites)

      screenshot_2016-11-03-17-10-10_f
      While testing upgrade scenario seems pretty obvious, I’ve seen this failing more than a couple of times. Here, after upgrading the IFTTT app on Android: it is not possible to proceed from the welcome screen because pressing Continue always results in an error message.
  • Testing API requirements: if the next App requires new version of API you may check if old versions work with the new API. If the new API version is to be released gradually (beta testing or experiments), you may check that the next app works with the current API as well.
  • Testing other related integrations

    • Integration with other Apps, such as Facebook login. New version of your app might brake integration protocols.
    • Hardware and it’s firmware versions (users may not upgrade it!) — new version of your app might not be compatible with older firmware versions
  • Testing an upgrade scenario that fails.
    • Upgrade scenarios may occur within the first launch of an updated app. It may convert persisted internal data to new formats, create or copy files and download additional data required from external sources. All of these operations can fail due to changing system and external conditions.
  • Try uninstall/reinstall scenario
    • If the app make changes to system settings or shared resources, these must be restored after uninstallation
    • Local / shared data might need to be removed as a part of uninstallation

3. Try different start conditions

Purpose: simulate different states in which the app can be started. This includes starting the App using URL Schemes (special registered  URLs that can open specific screens and load particular data using paths and identifiers within the URL). This is not something you can trigger yourself as a user, app developers need to code/enable this feature.

What to look for: starting the App in different states can introduce a variety of scenarios leading to functional deviations, infinite loops and even crashes.

  • Launch a URL scheme opening a particular screen, while the app is not already running in the background (cold start).
  • Launch a URL scheme while the app is launched and showing a modal screen: for instance accept changed user terms screen, requiring explicit user consent to proceed. You may need to clarify the expected behavior.
  • Other apps may be able to launch your app with arguments, while it is already running and executing a task, or even showing a screen that contains user data not yet submitted. Interrupting that may cause loosing data, and requiring to start from the beginning.

4. Check different screen sizes

There exist plenty of devices with varying screen sizes. Fortunately, developers are aware of this and handle this using resolution modes. iOS and Android apps can be enabled to run at higher resolutions, including the tablet mode

  • Check the app on supported resolutions. Android in particular has a history of layout issues on smaller screens, which do not appear on regular bigger screen devices. I found Galaxy Mini models to be good to have around, since these were also widely used small screen devices.
  • Check screen rotation in supported resolutions: landscape mode may still look good on your tablet, but what if you put it in vertical mode and back again? Check any places that allow rotation; picture galleries are a typical place to look for resolution bugs.
  • Install custom keyboards that use more vertical space than the native ones; this is possible on both iOS and Android – SwiftKey, Go Keyboard can be configured in ways that make them “taller”: screens with many input fields may assume default keyboard height. For example: two apps, showing input field on the first screen with the same keyboard toggled:

    Sometimes important controls, such as action buttons can be hidden behind the keyboard which impacts usability.

5. Check other OS versions

This speaks for itself, check supported OS versions on all supported devices 🙂 That may be a lot, so you need a good strategy to be efficient:

  • Determine widely used OS versions with analytics or some public statistics. The border line could be 1%, 5% or even 10% of users using a particular version. The choice may be depending on the spread.
  • Define which devices should be supported in the same way
  • You may reduce the amount of devices by spreading most used OS across most used devices. This introduces the risk of not knowing whether it is the device or the OS version affected by a bug, and thus requires more work to narrow the conditions later.
  • Use test automation in the cloud which allows to run concurrent test execution on all desired configurations: this requires an investment of money and time, both on development and maintenance of such setup but definitely saves testing time.

6. Simulate changing system conditions

Purpose: simulate a working environment, in which the app will be used to the real users to find conditions leading to unexpected behavior.

This pattern is rather a never-ending checklist of relevant (for your app) system events that could cause unexpected behavior. The assumption behind is that the app executes in a host environment, while host operating system can be considered as an actor itself (which relates back to James Whittaker’s IO patterns).

In mobile context, conditions widely applied to the majority of apps can be checked:

  • Switching between WiFi networks
  • Switching from WiFi to Cellular connection and back
  • Set the phone in airplane mode
  • Turn off Bluetooth, GPS, NFC or any other phone features
  • Turn screen rotation ON, rotate the phone: both after and before launching the app
  • Turn screen rotation OFF
  • Send app Background, and bring it back. Check the app state is the same as before (for example the user is still logged in)
  • Lock and then unlock the screen to resume using the app
  • Hard-close the app by swiping it away, and start the app. Check the app state is the same as before
  • Let the phone run out of power while app is running. Start the app and check the state of user data in the app
  • Receive other system notifications, which can be interrupting popups and interrupting apps — phone calls, alarms, system error messages (see example). Do this while your app is processing, uploading or downloading data to check whether resume will succeed
screenshot_2016-11-08-19-34-34
An error screen saying: “Hmm, something went wrong. Try again later” (after switching from WiFi to 3G)

7. Introduce time as a testing parameter

Purpose: simulate real user situations in which the app is used during longer periods of time.

Reasoning behind: typical developer testing applies to local builds living for a couple of minutes (before new code changes trigger a new build). Real users would definitely use the application for longer time:

nuxpg
Don’t keep activities option in Android settings
  • Let the app run for a while phone being in sleep mode, and then resume it after a couple hours or even a day. Be aware that this may not be sufficient for home screen only, but also screens that use and fetch resources or do some processing in the background. Memory leaks is what you are looking for.
  • For Android: turn on Developer Settings and check “Don’t keepactivities” setting on. This simulates situations when memory is overloaded, and the Android OS closes activities to free the memory. If not properly handled within the app, this situation can lead to a crash
  • Test on older (and therefore usually slower) devices: features introduced in newer versions of OS can appear slower on older devices. This might even lead to race conditions, which are hard to simulate unless you know there is a risk for it

8. Introduce concurrency as a testing parameter

Purpose: simulate real user situations in which the app is used to do simultaneous tasks or concurrently with other software.

Reasoning behind: concurrent execution of tasks and software in modern mobile OS introduces the same variety of testing situations as in desktop and similar software. Switching between apps, screens and tasks may be implemented differently on mobile platforms, thus calling for discovering patterns which can lead to unexpected situations:

  • If the app has an ability to start background jobs, such as upload, download or batch processing of files: attempt to alter the files in some way while the job is still running. For instance, try to delete the file you are uploading.
  • Explore the boundaries of concurrency: start download, upload or batch processing of  as many files as possible
  • If there are more than one screen that allows to start background jobs: attempt to execute them all at the same time.
  • Launch a URL scheme telling the app to open another screen/activity while processing is happening on the screen currently loaded.

9. Simulate changing external conditions

Purpose: discover what happens when external parts of the system are changing. While this is an activity of integration testing, some situations might not be taken care of in system design considerations. For example:

  • Check the situation in which an external account (fx. facebook or google)  that is used for authentication gets disabled / deleted. What should your app do in attempt to use such account first time ever? What should it do if the user is already authenticated, and the account gets disabled?
  • Service used to retrieve structured data (XML, JSon etc) returns Server Error 500 response in HTML.
  • Compatible integrated hardware disconnects or stops responding due to error or upgrade process. For example: Google Chromecast integration within android apps might experience the device to become turned off or busy / hanging.

10. Investigate efficiency

Draining battery. Slowing network communication. Takes lots of space! These might be some of the comments from your users rating and reviewing your app.battery

What you are looking for using reasoning behind this pattern is inefficiency. It may been there for a while, or just introduced in the latest build after an upgrade of 3rd party packages.

  • Check battery consumption while using your app with typical settings for brightness
    and connectivity (Wi-Fi, Bluetooth etc). There are tools to monitor battery usage over time which can be used for that purpose.
  • Monitor payload size of communication with your API and external services. Your app can appear slower if payload size increases dramatically
  • Monitor application size, relative to previous builds. You don’t have to define a threshold but you should be able to spot jumps in size
    • This includes both what user downloads from the store, and in combination with data your application generates/downloads and stores on the system. Modern mobile OS can show the user which apps fill up the previous space.
  • Check CPU and memory usage, which also could affect speed

What else is out there?

There exist plenty of tips and checklists on mobile testing written on the net. Some of these I’ve found to be good inspiration:

T-Map has a good Mobile App Testing Checklist, identifying test conditions for device (system), network connection, user interface and app store release procedure.

Some advice to best mobile QA practices can be found in “How to Break your App – Best practices in Mobile App Testing” by Daniel Knott who wrote “Hands-on mobile app testing“. If you don’t want to but the book, check some tips from the author in Top 10 Best Practices for Mobile App testing.

Mobile Testing Checklist from Lee Barnes: