Home → Magazine Archive → March 2016 (Vol. 59, No. 3) → Repeatability in Computer Systems Research → Abstract

Repeatability in Computer Systems Research

By Christian Collberg, Todd A. Proebsting

Communications of the ACM, Vol. 59 No. 3, Pages 62-69

[article image]

In 2012, when reading a paper from a recent premier computer security conference, we came to believe there is a clever way to defeat the analyses asserted in the paper, and, in order to show this we wrote to the authors (faculty and graduate students in a highly ranked U.S. computer science department) asking for access to their prototype system. We received no response. We thus decided to reimplement the algorithms in the paper but soon encountered obstacles, including a variable used but not defined; a function defined but never used; and a mathematical formula that did not typecheck. We asked the authors for clarification and received a single response: "I unfortunately have few recollections of the work ... "

Back to Top

Key Insights


We next made a formal request to the university for the source code under the broad Open Records Act (ORA) of the authors' home state. The university's legal department responded with: "We have been unable to locate a confirmed instance of [system's] source code on any [university] system."


CACM Administrator

The following letter was published in the Letters to the Editor of the May 2016 CACM (http://cacm.acm.org/magazines/2016/5/201586).
--CACM Administrator

Christian Collberg and Todd A. Proebsting deserve our gratitude for their article "Repeatability in Computer Systems Research" (Mar. 2016) shining sunlight the best kind of disinfectant, according to Supreme Court Justice Louis D. Brandeis on the very real problem of lack of repeatability in computer science research. Without repeatability, there is no real science, something computer science cannot tolerate.

Alex Simonelis
Montral, Canada

Displaying 1 comment