What is excellent software testing?

18 characteristics of excellent software testing.

This story is about Alice.

Alice is an excellent tester. Excellent testing is hard to describe.

You only find out what excellent testing is when you see it.

Here’s what I have seen.

★ Alice acts like a sociologist.

Alice knows that testing is a social science. She knows that the quality of the software is inherently subjective. She knows that quality is value to some people who matter at some time (Gerald Weinberg). She understands that a bug is something that threatens this value. So, she realizes that a bug is simply something that bugs somebody (James Bach). She recognizes that a bug isn’t necessarily in the software but rather represents a relationship between a person and the software (Cem Kaner). Hence, she accepts that quality cannot be measured but only assessed. So, she acts like a sociologist who looks for different problems (that matter) to different people (who matter), because she knows that different people will perceive one and the same software as having different levels of quality.

★ Alice acts like a psychologist.

Alice knows that our thinking errors can become software errors when we aren’t aware of our cognitive biases (Andrew Brown). Therefore, she looks for bugs in the mind of people. So, she acts like a psychologist who delves into the mind of people to analyze their thinking, because she knows that most of their knowledge (including their opinions, assumptions, expectations, desires, hunches, feelings, experiences) about the software is in their heads.

★ Alice acts like a controller.

Alice knows that automated repetitive checking is important because she knows that she cannot trust the software. She doesn’t trust the software because the software continually changes its “mind”. The software and the (business, technical) domain in which it is situated constantly and rapidly evolve. So, she automates checks to make the process of comparing the current state with some past state of the software faster and less error-prone. In doing so, she thinks of regression checking as an airbag, not as a substitute for good driving, because she understands that repeatability in software testing is not as big a deal as adaptability is (Michael Bolton). Therefore, she doesn’t overfocus on confirmation in testing because she is aware that the risk lies in the part that hasn’t been explored yet. In short, she knows that checking lies in the domain of the known knowns, while testing lies in the domains of the known unknowns and unknown unknowns (Del Dewar). She has also learned that there’s no such thing as free lunch when it comes to automation. She knows that automation requires maintenance. So, she treats automation like cake, since she knows that a small amount can taste great but too much can make her sick. So, she doesn’t just automate some checks, she automates checks that matter. She automates checks that repeatedly reveal useful information about the software. So, she acts like a controller who carefully controls the benefit-to-cost ratio of automated checks to not get drown in maintenance efforts.

★ Alice acts like a leader.

Alice knows that software testing is not a mechanical, one-dimensional activity but a creative, multi-dimensional, and human-centric activity. She sees human testers and their skills at the center of testing, not machines. She is a master in programming but she doesn’t feel the need to artificially elevate her role by inventing new names for it (e.g., SDET, automation tester, test automation developer). She knows that saying tester is enough. She understands that she is actually being paid to test, not to code. She sees tools as something that can accelerate, extend, amplify, and simplify testing (Michael Bolton). She knows that testing is no more about “test tools” than mathematics is about rulers. She knows that she can test without tools. For that reason, she doesn’t see tools as something that eliminates the need for human thinking in testing. She knows that it is silly to blame gravity for falling flat on the face. So, she doesn’t blame tools for the poor tests she performs. She knows that only poor testers blame tools for the poor tests they perform because she understands that blaming a “test tool” for her poor testing is like blaming Microsoft Word for a meaningless article. So, she acts like a leader who is leading by example by holding herself responsible for her testing actions and accountable for the results of her testing actions.

★ Alice acts like a chameleon.

Alice knows that testing is context-driven. She is aware that there aren’t any best testing practices, only good practices in context (Cem Kaner). She knows that the value of any practice depends on its context (e.g., project, product, people, environment). She knows that she wouldn’t test an autopilot of an airliner in the same way as a website (Iain McCowatt). Of course, sometimes, instead of finding and learning how to use a chisel, she turns to the hammer she is already holding (Matt Parker). But, in hindsight, she becomes aware of that. She keeps equipping herself with a variety of testing techniques (e.g., heuristics) and has them ready in different forms (e.g., mindmaps, cheat sheets, checklists) to design, perform, prioritize, and analyze tests in different contexts. She doesn’t shy away from making this effort because she considers herself a good tester who strives to become an excellent tester (Katrina Clokie). She knows that testing is neither just a talent nor a process. She considers testing as a set of skills that can be learned (Isabel Evans). She also relates testing to other disciplines (e.g., physics, mathematics, history, philosophy, social science) to broaden her skill set (e.g., problem-solving, modeling, speaking, critical thinking). She understands that as technology advances so must her testing approaches (Angie Jones). So, she acts like a chameleon that has the ability to adapt to new, unknown, or unfamiliar environments fast.

★ Alice acts like a critic.

Alice knows that the set of models (e.g., risk models, requirements models, architecture diagrams) she is using in testing are just a set of ideas about the software. She knows that these models aren’t the software; they are abstract representations of the software. She is using these models to understand certain aspects of complex software in simple terms. She is using these models to form conjectures, direct her testing, make decisions, and communicate her conclusions. She is aware of the fact that these models are simplifications of the software and understands that simplicity doesn’t precede complexity but follows it (Alan Perlis). So, she reflects on her conjectures, decisions, and conclusions by considering alternative models. She knows that these models are probably flawed representations of the software based on her flawed mental model of the software. So, she acts like her own best critic who questions her own models in testing to not get fooled by them.

★ Alice acts like an astronomer.

Alice knows that testing software is more than just checking the specification of the software. So, she doesn’t just compare the actual software to its specification, she also compares the idea of the software to the actual software and to the specification of the software. Alice knows that the purpose of the software is to solve someone’s problem. So, she evaluates the purpose of the software too, since she knows that testing the purpose of the software is one purpose of software testing. She does so because she understands that conformance with the specification has little to no value if the software solves the wrong problem. So, she acts like an astronomer who not only uses a microscope to evaluate the software from a small-scale (technical) perspective but also uses a telescope to judge the value of the software from a large-scale (conceptual) perspective.

★ Alice acts like a historian.

Alice knows that those who cannot learn from history are doomed to repeat it (George Santayana). So, she looks back to the mistakes she has made, identifies patterns of failures, and extracts the lessons she has learned. She doesn’t deny her mistakes and so she doesn’t hide them. She documents these mistakes and shares them with other people (e.g., testers, developers) by conducting regular retrospective sessions (e.g., monthly). In doing so, she learns from other people’s mistakes and she enables other people to benefit from her lessons learned. So, she reflects on mistakes before she takes on new challenges (e.g., project, release, sprint) to prepare herself to not repeat these mistakes again and again. She knows that prevention is better than cure. So, she acts like a historian who builds a learning culture by analyzing her past testing activities to continuously improve her future testing activities.

★ Alice acts like a guide.

Alice understands that having a testing specialist on the team doesn’t mean to restrict the responsibility of testing to that single person (Trish Khoo). She is aware that excellent testing is never done by one single person in isolation but requires the collective brainpower of a group of people. She understands that great (testing) teams are teams of diverse talents cooperating. She knows that diversity makes testing powerful. She knows that the idea of one person having all the answers is deeply flawed. She knows that knowledge comes in many different flavors. So, she looks for sources of potential support through collaborative activities (e.g., pair testing, mob testing). So, she acts like a guide who actively involves, guides, and advises other people (e.g., product owners, developers) in testing by stimulating them to continuously think critically about the software they envision, design, and build.

★ Alice acts like a realist.

Alice knows that the time needed for testing is always infinitely larger than the time available (Cem Kaner). So, she prioritizes her testing activities, because she knows that testing has to stop although it actually never ends (Michael Hunter). She is aware that she cannot prove the software right, only wrong. She knows that she has to think of falsification instead of verification. She accepts the fact that she cannot answer the question “Did we find all bugs?”. She understands that she cannot quantify the number of absent bugs. She knows that testing can only prove the presence of bugs, not their absence (Edsger Wybe Dijkstra). She is aware that only infinite testing can find every bug, but (most) testing budgets are not infinite (Dorothy Graham). So, she acts like a realist who knows that testing is undecidable because she understands that the problem of finding all bugs in the software is undecidable.

★ Alice acts like an experimentalist.

Alice knows that testing is learning. Whenever she runs out of test ideas, she suspects that she probably hasn’t learned enough about the software. She keeps exploring, investigating, and experimenting to constantly evaluate this conjecture. She never stops learning about the software and the (technical, business) domain in which the software is situated. She knows that testing is experimental science. She sees testing as an adaptive investigation rather than a factory process of creating, automating, and executing test cases. She knows that testing is more than just checking things out (Karen Johnson). She understands that her test success is not defined by the number of her test cases but by the quality of her test ideas. So, she acts like an experimentalist who doesn’t confuse testing with test cases, since she considers tests as experiments to challenge hypotheses, assumptions, and opinions about the software.

★ Alice acts like a stakeholder.

Alice knows that testing means comparing the invisible to the ambiguous (James Bach). She knows that the stakeholder’s knowledge, goals, and experiences are not identical to hers. She knows that the universe of stakeholders is vast, but she doesn’t get tired to examine risks for both internal and external stakeholders. She considers how internal stakeholders (e.g., sales, marketers, product owners, developers) would look at the risks she is uncovering. She ties risks to their objectives, needs, and goals. She examines risk from the perspective of external stakeholders (e.g., customers, partners). In doing so, she avoids saying and even thinking things like “No one would ever do that!”. She knows the more advanced she gets in the software, the harder it becomes to relate to true beginners (e.g., end users). So, she is trying to wear the end user’s shoes to not forget what it’s like to be a beginner. She is aware that it isn’t just the software but the product (e.g., software, people, environment) the end users depend on (Lisa Crispin, Janet Gregory). So, she steps out of her role as a tester to take on the perspective of different stakeholders by looking at the risks from their vantage points. She does so because she knows that these different viewpoints allow her to look at the value of the software, and the impact of a threat against the software, in a variety of ways. So, she thinks like a stakeholder to enrich the way she assesses and prioritizes risk to get a broader picture of the overall risk profile.

★ Alice acts like a kid.

Alice knows that she can’t expect answers to questions she didn’t ask. So, she isn’t afraid of looking “stupid” by asking simple questions. She understands that tests are questions she asks the software to reveal useful information about the software (Cem Kaner). She knows that assumptions (“unexamined beliefs”) are the termites in software testing. She suspects that she probably has answers that might be wrong and beliefs that might not be true, but she isn’t afraid of not knowing things (Richard Feynman). So, she doesn’t get tired of applying this crazy method called “asking” before she assumes something. For that reason, she judges herself not only by the answers she gives but also by the questions she asks (Simon Sinek). She knows that curiosity fuels testing. So, she acts like a curious little kid who never stops asking, because she knows that asking great questions is the source of great testing.

★ Alice acts like an accountant.

Alice knows that if she doesn’t have any questions about the risks in the software, then there’s no reason to test at all. If she has at least one such question, then she will ask: Will these tests cost more to perform than their answers will be worth? So, she acts like an accountant, since she understands that good testing involves balancing the need to mitigate risk against the risk of trying to gather too much information (Gerald Weinberg).

★ Alice acts like a skeptic.

Alice knows that she shouldn’t believe everything she hears and only half of what she sees (Edgar Allan Poe). She knows that there are no facts, only interpretations (Friedrich Nietzsche). She knows that certainty is absurd. So, she thinks critically about what she has been told about the software. She is skeptical about what she knows about the software, and she knows that she will never know everything. She is aware of the fact that she isn’t Laplace’s Demon. She focuses on what she doesn’t know about the software and doesn’t just settle with what she currently knows. She keeps thinking despite already knowing. Hence, she sees testing as a thinking activity and she treats it like one. She knows that she has to change her thinking to change her testing. She knows that software is a cunning deceiver that is always trying to fool her in one way or another. For that reason, she remains cautious, curious, and critical to not get fooled over and over again. So, she acts like a cosmic skeptic, because she knows that excellent testing is just the by-product of continuous critical thinking.

★ Alice acts like a friend.

Alice knows that the primary goal of testing is to make other people (e.g., product owners, developers) aware of problems in the software. In doing so, she doesn’t break the software, she breaks people’s illusions about the software (Maaret Pyhäjärvi). She knows that making people aware of these problems is a socially difficult task, because of people’s natural desire to avoid trouble (Michael Bolton). She understands that problems are something negative at a first glance. She considers a problem as a deviation from what is perceived and what is desired. The bigger this gap the bigger the problem. The bigger the problem the more it might threaten the software’s value. She is rational enough to know that different people react differently, and sometimes irrationally, to potential threats. She knows that there are people who welcome critical feedback, and there are people who don’t. She knows that there are people who value the perspective of others, and there are people who don’t. She knows that there are people who hold themselves responsible and accountable for their mistakes, and there are people who don’t. So, she knows that the nature of humans is diverse. So, she studies people’s personalities and their relationships among each other to make them aware of problems in a dispassionate, deliberate, and personal way. She understands that if she isn’t making people aware of problems, these problems will eventually find them. So, she communicates these problems right away with modesty and empathy. She is direct, critical but not judgmental. She doesn’t confuse being nice with being kind. She knows that just being nice doesn’t help, serious feedback does (Alexandra Schladebeck). So, she talks about negative things in a positive way because she sees problems as an opportunity to improve. She knows that the software doesn’t have to behave the way she wants it to behave. She is aware that she doesn’t design, code, or sell the software. She tries to help other people to understand the software they have got, so that they can decide whether it’s the software they want (Michael Bolton). So, she acts like a friend who reduces people’s ignorance about the software without embracing arrogance.

★ Alice acts like a storyteller.

Alice understands that software testing is the headlights, quality is the journey, and business outcomes are the destination (Anne-Marie Charrett). So, she lights the way for other people by describing the status of the software in terms of what it does, how it works, and how it might not work. She describes her testing activity in terms of what has been tested, what has not been tested yet, and what probably won’t be tested due to several constraints (e.g., time, resource, budget). She describes the quality of testing in terms of the successes she achieved and obstacles that impeded her testing. So, she not only looks for problems that threaten the value of the software, she also looks for problems that threaten testing. She understands that problems in the process of looking for problems in the software (aka testing) can prevent her from finding problems in the software. In summary, she provides a 360° narrative on testing packed with useful and actionable information instead of bombarding her stakeholders with vanity metrics (e.g., test case counts) packed into shiny but useless dashboards. So, she acts like a storyteller who understands that testing requires doing and describing testing.

★ Alice acts like an assistant.

Alice knows that she cannot assure quality in the same way a doctor cannot assure health (Michael Bolton). She understands that her primary objective is to collect quality-related information (e.g., risks) about the software to enable other people (e.g., product owners, developers) to make better-informed decisions (e.g., shipping decisions, fixing decisions) based on the information she provides. In short, she tests, others decide. She understands that testing is an information-based science (Keith Klain). So, she considers herself as an assistant who neither does quality assurance nor quality control but rather quality assistance.

This list could go on and on and on. It’s not complete. It never will be. You get the point. Alice is a cocktail of testing excellence.

Her testing is inexpensive, fast, credible, and accountable. That’s the difficulty of doing excellent testing. Ergo, anyone can test but not anyone can test well.

Here is a secret. Alice is real but only in a metaphorical sense. Alice is a fictional character that symbolizes the collective testing excellence of a large and diverse group of inspiring software testers.

This short story stands on the shoulders of these giants. Here are 99+ of them. They know the way, go the way, and show the way of excellent testing.

Namasté, my software testing friends.