Until the Craze Fades

There was the hula hoop. More recently, as anyone who frequents small children knows, the craze was spinners. In research, too, crazes come and go. The current one in software engineering (and some other fields of computer science, as noted by Jeff Ullman) is quantitative evaluation. In the top venues Normal 0 false false false EN-US X-NONE X-NONE —note that I am cowardly publishing this while everyone else in the field is away at ICSE this week, so that I can hope they will miss or forget this comment—it has become essentially impossible to publish a novel idea without including the results of a quantitative assessment study.

In the first and second parts of an earlier article, I marveled about the remarkable progress of empirical software engineering. It sometimes happens that an approach that once struggled for recognition (as when Vic Basili, Walter Tichy, Barry Boehm, Dieter Rombach, and others were exhorting computer scientists to become experimentalists too) triumphs. As the empirical/experimental/quantitative style reaches that stage, we should remember that it is not the only form of software engineering research. Others, such as conceptual research, have their place too.

It is partly a matter of which hackneyed quote from a famous scientist you choose. To Lord Kelvin's Normal 0 false false false EN-US X-NONE X-NONE "when you cannot measure [what you are speaking about], when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind," one may retort with "not everything that can be counted is worth counting" (attributed, although without serious evidence, to Einstein).

Experimental evaluation risks crossing the bridge from good idea to tyranny. Like the hula hoop in its time, this particular craze will fade, and some reasonable middle point will emerge. Until then, as a public service to the community, I hereby propose a standard section 5 (this is about where in your paper the referee expects the evaluation part) that you may include, royalty-free, in any of your submissions.

Section 5. Quantitative evaluation

No animals or undergraduate students were harmed during the preparation of this work. In fact, we performed no evaluation whatsoever. Our idea is so clever anyway, and our presentation so well-written, that we do not need any further justification.

A shorter version, on which, however, I do not hold the copyright, is "frankly speaking, my dear, I don't give a damn."