Thoughts about quality engineering and scientific literacy
It's better to be precise than to be not wrong
People, understandably, do not like being wrong, especially if they have an intellectual profession since
they may see not being wrong as part of their job. One way to avoid being wrong is being right, but being
right is difficult, being vague is another approach and is very easy to do, that's why most people go with
vagueness. Consider how experts define software testing,
in an interview titled What is Software Testing?: A Conversation with Michael Bolton,
software testing consultant, trainer, and avid conference presenter
Michael Bolton (not the singer) is asked "What is software testing?" this is his answer:
So they've been lots of definitions over the years, one definition that we used in Rapid Software Testing for
quite a while was "questioning a product in order to evaluate it" another example would be a Jerry Weinberg's
assessment of it, he never actually said it explicitly, he kind of hedged and avoided saying something too specific,
but the closest he came to in his book Perfect Software and Other Illusions About Testing was "gathering information
in order to inform a decision" Kim Keiner who is one of the most admirable figures in the whole history of
software testing developed a sort of consensus definition with a bunch of people who attended some conferences
wherein he said and I am going to try to remember this exactly "which is software testing is an empirical technical
investigation of software done on behalf of stakeholders with the intention of revealing quality related information
of the kind that they seek how about that" and can you tell Kevin was a lawyer? But that covers all the bases...
- Michael Bolton
These definitions may indeed "cover all the bases" but what they don't do is give you any useful standard for determining
what is testing and what isn't. Many things are empirical analyses but are not testing, arguably every part of the
software development lifecycle include an empirical analysis, is everything developers do testing?
A precise definition is more useful not because it is always going to be right but because every time it is shown to be wrong
it can be improved and definition that is so vague that is meaningless cannot be shown to be wrong and thus will never be amended to be
more accurate.
It's not aliens, it's never aliens
The other day I was watching a "debate" between a UFO believer and podcaster named
Kent Bye
and former game developer turned skeptic Mick West.
The debate was a good example of people (in this case Kent Bye) not understanding
the concept of evidence-based reasoning. I will explain it here and use the debate to illustrate.
What is evidence?
First, we have to discuss what the word evidence means in the context of science. The meaning of this word, just like other words, is determined
by how people use it, dictionaries just like all output of linguistics are
descriptive, not prescriptive.
Consider a poorly-designed, non-placebo-controlled study in which the experimenter measures the participants' blood pressure, gives them a medication, and then measures the blood pressure again and finds that
it is lower after the medication. As you probably know this is not a good experiment because it lacks a control group.
But why, specifically, is a control group necessary? What does it accomplish?
It is necessary because it eliminates other explanations of the results, for example, a critic might point out that in our hypothetical non-controlled
experiment the lower blood pressure could have been the result of the placebo effect or the fact that they started caring about their health more as a result of
participating in the study. Notably, the critic of this experiment does not have to prove any of the alternative explanations, the critic of the experiment
merely needs to point out that a possible alternative explanation has not been ruled out. It's the experimenter's responsibility to rule out alternative explanations.
This is because it would be impossible to make decisions if the decision maker had to disprove every relevant claim, in order to have
a way of making decisions wherein decisions can actually be made, people have long ago decided that
the onus for providing evidence is on the person making the affirmative claim.
As is hopefully clear from this simple hypothetical, evidence is not merely an observation that is consistent with a hypothesis,
it is an observation that is not consistent with any explanations other than the hypothesis.
What does this have to do with aliens?
In the debate between Kent Bye and Mick West, Bye focuses on the idea that West cannot be certain of his alternative explanations
of the alleged UFO footage, eg at 18:50, after summarizing a list of West's alternative explanations for a specific claimed UFO video:
I don't think this is a good debunking and I would argue that what you're doing is adding more noise.
-Kent Bye
Here Bye is implying that because West did not prove his alternative explanations he is just adding more "noise" or unsupported claims.
This view is wrong because it puts the people making the affirmative claim (that this is a video of a UFO) on par with the person
offering alternative explanations and as we have discussed they do not have an equal burden of proof. West does not have to prove his alternative explanations,
he just has to show that they are possible and that the people making the affirmative claim have not conducted their research in a way that would rule them out.