Ivan Kouznetsov

Senior Engineer


Thoughts about quality engineering and scientific literacy

It's better to be precise than to be not wrong

People, understandably, do not like being wrong, especially if they have an intellectual profession since they may see not being wrong as part of their job. One way to avoid being wrong is being right, but being right is difficult, being vague is another approach and is very easy to do, that's why most people go with vagueness. Consider how experts define software testing, in an interview titled What is Software Testing?: A Conversation with Michael Bolton, software testing consultant, trainer, and avid conference presenter Michael Bolton (not the singer) is asked "What is software testing?" this is his answer:

So they've been lots of definitions over the years, one definition that we used in Rapid Software Testing for quite a while was "questioning a product in order to evaluate it" another example would be a Jerry Weinberg's assessment of it, he never actually said it explicitly, he kind of hedged and avoided saying something too specific, but the closest he came to in his book Perfect Software and Other Illusions About Testing was "gathering information in order to inform a decision" Kim Keiner who is one of the most admirable figures in the whole history of software testing developed a sort of consensus definition with a bunch of people who attended some conferences wherein he said and I am going to try to remember this exactly "which is software testing is an empirical technical investigation of software done on behalf of stakeholders with the intention of revealing quality related information of the kind that they seek how about that" and can you tell Kevin was a lawyer? But that covers all the bases...

- Michael Bolton

These definitions may indeed "cover all the bases" but what they don't do is give you any useful standard for determining what is testing and what isn't. Many things are empirical analyses but are not testing, arguably every part of the software development lifecycle include an empirical analysis, is everything developers do testing?

A precise definition is more useful not because it is always going to be right but because every time it is shown to be wrong it can be improved and definition that is so vague that is meaningless cannot be shown to be wrong and thus will never be amended to be more accurate.

It's not aliens, it's never aliens

The other day I was watching a "debate" between a UFO believer and podcaster named Kent Bye and former game developer turned skeptic Mick West. The debate was a good example of people (in this case Kent Bye) not understanding the concept of evidence-based reasoning. I will explain it here and use the debate to illustrate.

What is evidence?

First, we have to discuss what the word evidence means in the context of science. The meaning of this word, just like other words, is determined by how people use it, dictionaries just like all output of linguistics are descriptive, not prescriptive.

Consider a poorly-designed, non-placebo-controlled study in which the experimenter measures the participants' blood pressure, gives them a medication, and then measures the blood pressure again and finds that it is lower after the medication. As you probably know this is not a good experiment because it lacks a control group. But why, specifically, is a control group necessary? What does it accomplish?

It is necessary because it eliminates other explanations of the results, for example, a critic might point out that in our hypothetical non-controlled experiment the lower blood pressure could have been the result of the placebo effect or the fact that they started caring about their health more as a result of participating in the study. Notably, the critic of this experiment does not have to prove any of the alternative explanations, the critic of the experiment merely needs to point out that a possible alternative explanation has not been ruled out. It's the experimenter's responsibility to rule out alternative explanations.

This is because it would be impossible to make decisions if the decision maker had to disprove every relevant claim, in order to have a way of making decisions wherein decisions can actually be made, people have long ago decided that the onus for providing evidence is on the person making the affirmative claim.

As is hopefully clear from this simple hypothetical, evidence is not merely an observation that is consistent with a hypothesis, it is an observation that is not consistent with any explanations other than the hypothesis.

What does this have to do with aliens?

In the debate between Kent Bye and Mick West, Bye focuses on the idea that West cannot be certain of his alternative explanations of the alleged UFO footage, eg at 18:50, after summarizing a list of West's alternative explanations for a specific claimed UFO video:

I don't think this is a good debunking and I would argue that what you're doing is adding more noise.

-Kent Bye

Here Bye is implying that because West did not prove his alternative explanations he is just adding more "noise" or unsupported claims. This view is wrong because it puts the people making the affirmative claim (that this is a video of a UFO) on par with the person offering alternative explanations and as we have discussed they do not have an equal burden of proof. West does not have to prove his alternative explanations, he just has to show that they are possible and that the people making the affirmative claim have not conducted their research in a way that would rule them out.

That's all for now!