Testing Terminology

Abstract:Another brief post, reacting to a culmination of bad posts I have seen lately in the testing community surrounding the testing terminology we use to define our craft. With language comes power, and this terminology tells others what we think about our craft. Much of what I have seen lately casts testing in a negative light – one of button clickers and script writers, rather than intellectual explorers and light casters. I am the latter, so I must delineate and separate myself from the former. My post below is mainly a reaction to this uTest forum post, the straw that broke the camel’s back.

Recently, I joined the uTest community, simply because I wanted to get some more raw testing time in, as well as see what kind of testing caliber is available in one of the most popular crowd-sources testing services in the world. After browsing around for a few days, I found that while the intentions in many blog posts, articles and links were probably good, much of the information related to definitions is outdated and contribute to misleading new testers, which keeps the testing craft in the dark – preventing it from advancing. Here are a few corrected definitions of the poorly defined words that I saw while going through some of the forums on uTest…

Testing: evaluating a product through exploration and experimentation for the purpose of informing our stakeholders on risk.

Note: “The purpose of testing is to cast light on the status of the product and it’s context, in the service of our stakeholders.” James Bach. We are not Product Management, we do not make the go/no-go calls on releases – we are simply the lighthouse, that points out risk and potential problems so that our stakeholders can make much more informed decisions about how to run the business.

More on testing here: https://www.utest.com/articles/what-is-testing
More on testing vs checking: http://www.satisfice.com/blog/archives/856
More on the difference between Testing and QA: http://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/

Bug: anything that threatens the value of the product.

Note: Anyone can log a bug report. This does not necessarily mean that it is a “defect”, which is in fact a term that the context-driven software testing community is moving away from. More on CDT here: http://context-driven-testing.com/

Quality: Quality is value to some person, who matters.

Note: This is incredibly important, because it highlights the extremely subjective nature of what product quality really is. We must realize that the objective is not to test “everything”; however, it is to test what matters in the time allotted. “The goal is to reach an acceptable level of risk. At that point, quality is automatically good enough.” – More on that here: http://www.satisfice.com/articles/gooden2.pdf

Test Case: formally structured, specific, proceduralized, explicit, documented, and largely confirmatory test ideas – and often, excessively so.

Note: A test case is not a test, any more than a recipe is a meal, or an itinerary is a trip. Open your mind to the fact that heavily scripted test cases do not add the value you think they do. If you are reading acceptance criteria, and writing test cases based on that, you are short-circuiting the real testing process and are going to miss an incredible amount of product risks that may matter to your client. More on the value (or lack thereof) of test cases here: http://www.developsense.com/blog/2017/01/drop-the-crutches/

Quality Assurance (QA): Ah, yes. The most abused title/phrase in the testing world…No one person does this, and anyone who has a title “QA” is fooling themselves. “The quality assurance role in the company resides with the management and the CEO (the principal quality officer in the company), since it was they—and certainly not the testers—who had the authority to make decisions about quality.”

Notes: Again, more on the difference between Testing and QA here: http://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/

Test Plan: The test plan is the set of ideas that guide or represent the intended test process. Often those ideas are only partially documented, spread across multiple documents, and subject to change as the project evolves.

Notes: More here… http://www.satisfice.com/tools/tpe-model.pdf

Perhaps you meant one of these instead?…
Test Plan Document: A test plan document is any document intended to convey test plan information. However, test plan documents are not the only source of information about the test plan. Test plan information is also contained in the oral tradition of the project and the culture of the company.

Test Strategy: The test strategy is the way tests will be designed and executed to support an effective quality assessment. Test strategy is the plan for what parts of the product will be covered by tests and what test techniques will be used. Test strategy is distinct from the logistics of implementing the strategy. Test strategy is essentially the “brains” of the test process.

I will stop there for now. I could go on about how we should replace the phrase “verify that” with “challenge that” in our vocabulary, or how standards from IEEE and ISO (e.g. ISO29119) are archaic and detrimental to software testing/development, but I could go on for pages on that, so I’ll save it for another time.

Crossposted: uTest – Testing Terminology

2 thoughts on “Testing Terminology

Leave a Comment

Your email address will not be published. Required fields are marked *