Break It Before You Buy It! A Recipe to Improve Your Application Testing
Sometimes a sports team will have an “easy” schedule: A string of weaker opponents that lead to easy wins. That can make the team look a lot more talented than they actually are. How many times has this happened to a baseball or basketball team, coasting into the postseason only to be crushed by the first truly talented team they meet?
The same can happen when testing applications. If all you do are “easy” tests—tests that merely prove that the application can work as expected under typical or ideal conditions—your product will rack up a string of easy wins. That will create a false sense of security and confidence in the product. When reality comes crashing down, it might be too late.
In my time as a developer, I’ve created something of a “recipe” for testing. The main idea in my recipe is to get through the easy wins as quickly as possible, and then move on to a more rigorous schedule. That way, your testing becomes a valuable contribution to the overall development effort, creating a better product at each iteration. My hope is that my recipe will help both developers and testers think a little more rigorously about what makes a testing regimen truly useful, and thereby avoid developing products in a vacuum.
Allison’s Recipe for Robust Application Testing
My recipe for testing is simple:
- Test with the end in mind
- Test in a “natural” setting
- Do the unexpected
- Spiral your testing
Test with the End in Mind
Every application has a purpose. For example, a simple application might present the user with a timesheet to fill in so they can report their work hours. The goal seems fairly straightforward in this case, but all we really have is a broad function for the application.
A goal goes beyond just the broad function and makes clear what acceptable end-states would be. For example, would the application be of any use if it took 30 minutes to enter a timesheet? Would the user abandon the process before it was complete? Or what if the user entered their timesheet in under 3 minutes, but it was riddled with errors? What if they tried to complete the timesheet, but were left with a ton of unanswered questions? As the product is being developed, take the time to hone in on what the specific goal is, not just the broad function of the application.
That said, developers and testers can err on the side of being too specific, as well. For example, in a given iteration, a developer might focus solely on fixing certain bugs. That’s not a bad thing to do, but it’s worth asking: Is effort being spent just fixing bugs, or fixing the actual problem? Every application is built to address certain problems, and each iteration of the application will be valuable only insofar as it makes incremental steps toward solving those problems.
In other words, testers should always be thinking about what the application should allow the user to do, and what the experience of that is like. That mindset is the north star guiding every round of testing.
Test in a “Natural Setting”
I’ve seen many developers test their apps using their own machines and their own credentials. But how many users will have that same environment? That same set of access permissions? Or, for that matter, the same understanding of how the application should work?
When testing, you want to set the stage so that the test is as close to the actual users’ situation as possible. For example, if an application is going to be used by employees working in the field on iPads, you had better test that app on iPads! Likewise, the tester should have a privilege set that matches the end users and not, say, admin or developer access.
It’s also important to test applications without giving the tester a robust set of instructions for how to do their tests. You read that right: Think IKEA instructions (minimal), not LEGO instructions (so simple a four-year-old could follow them). Testers should understand the main goal (see above), but the developer should not over-explain what is meant to happen or in what order. This forces the tester to move through the system the way that a user would, and reveals whether a user can, unprompted, achieve the goals set forth for the application.
Do the Unexpected
Developers and testers alike tend to test applications by seeing if they do what they are supposed to do. Again, that can give them a false sense of security. It’s just as important to verify that an application doesn’t do what it’s not supposed to do.
This means getting a little creative and purposefully trying to “break” the application. For example, what happens if a user makes a mistake or gets frustrated, and so just randomly hits the “esc” button? If that drops the user out of the application entirely, or leaves data in an incomplete state, that could be frustrating now, creating data quality issues later! Another set of examples: What happens when a user enters text into a number field? Or enters a decimal when a whole number is expected? Or a value of zero? There are both obvious and not-so-obvious ways to misuse and thereby break an application.
The more ways in which testers do the unexpected, the more robust the application can become. Take our timesheet example. If the goal is to enter a set of time data, one would expect that, once the user hits “submit,” there would be some entries of time durations. What if there are none? Why did that happen? The application should probably prompt the user at this point with a message: “It looks like you don’t have any time entries. Is that right?”
Ideally, an application should give the user feedback throughout its use. If the user is on the right path, it should indicate that (“Timesheet submitted successfully—thank you!”). If something has gone wrong, the application should prompt the user (“Are you sure you worked 235 hours last week?”).
Defining that correct path requires knowing where users could trod—that is what your testing needs to reveal. Developers should act something like the university that wanted to place walking paths on their new campus: Instead of having the architect dictate where the paths would go, the university simply grew grass everywhere. After a few months, natural walking paths emerged, and the university simply placed their paved paths over those. The result was not only aesthetically pleasing, it was functional precisely because the paths naturally fit where people wanted to go.
Spiral Your Testing
In education, teachers will sometimes give students a “spiral review” that tests not only what the students just learned, but tests previous concepts as well to ensure the students are retaining what they’ve learned.
Testing needs to do the same thing, to a point. Developers often call this recursive testing or regression testing, but the idea is the same: re-running functional and non-functional tests to ensure that any changes that were made have not altered the performance of the software. For example, maybe the data validation on that timesheet worked fine last time, but now that you’ve introduced a new feature, the validation step does not trigger as it should. In this kind of scenario, we say that the software has regressed.
The farther into development an application gets, and the longer the list of found defects, the larger the suite of regression tests grows. That suite can start to get unwieldy after a while. This is where test automation comes in handy. Developers might also opt to do a change impact analysis to determine an appropriate subset of tests that look at components most likely to have been impacted by a change. Clusters of these can be tested and fixed together.
Final Thoughts on Rigorous Application Testing
Applications are meant to get users from point A to point B quickly and without fuss. If your testers are following the main road and are armed with a map and a compass, they will of course succeed (or at least they should find it very hard to fail). What testing should do is reveal whether a user can get to point B when they don’t have a map, have a malfunctioning compass, and frequently wander off the road
The recipe here is meant to serve as a set of reminders about how to do just that. The feedback you’ll get from this style of testing is invaluable if you want to create a product that is truly successful.
