By @rpinho
Refereed correctness bugs in open-source research software, with patches and links to upstream PRs
# Method How a bug gets from a hunt into this repo, and the rules for telling the authors. Each rule is being built as a check in the research harness, [rpinho/rstack](https://github.com/rpinho/rstack), so a draft that breaks a rule is refused instead of relying on someone to remember it. Links to the code go here once it is pushed. ## Which code we look at Only code that is still alive. The repo needs a commit or a release in the last 12 months, must not be archived, and must not say it is unmaintained. A one-off data deposit with no later version doesn't count. A paper from the last few years is not enough on its own: if nobody touches the code, a bug report is a curiosity and not something anyone can act on. The first round, listed in the README, went out before this rule and includes some older projects. We are keeping those reports because they were already sent. ## Hunt, then an independent referee 1. An AI agent clones a pinned commit, reads the open and closed issues and PRs first, and looks for correctness bugs: numbers that are wrong on normal inputs. It checks each number against a reference it computes separately, using tiny inputs on a CPU. 2. A separate referee gets only the claim and the pinned commit. It does not run or read the hunter's scripts, patches or expected outputs. It writes its own minimal input and its own reference, runs the unpatched code, and gives a verdict: CONFIRMED, CONFIRMED-MINOR, INTENDED, NOT REPRODUCED or KNOWN. 3. Only CONFIRMED and CONFIRMED-MINOR bugs get reported. KNOWN means someone had already reported it, so we drop it. 4. Some authors get a second, independent referee on every bug and a separate read of the exact email text before it goes. ## Telling the authors - One email per person per day. If a person has several confirmed bugs, they go together in one email. - Every email says what the bug does in two or three sentences with one concrete number, and asks two things: whether they use AI in their research, and whether this is useful or slop. - We offer a short report or the patches, to them or to whoever maintains the code. We open a PR when the author or a maintainer invites one. - Addresses come from the person's most recent papers, lab page or university profile. A bounced address is marked dead, and we look for a new one. - A bug goes into this repo only after the authors have been told. ## Numbers so far (Nov 8, 2026) | | Count | |---|---:| | Projects with confirmed bugs listed here | 14 | | Confirmed bugs listed here (including minor) | 39 | | Research groups emailed about bugs | 14 | | Groups that replied | 2 | | Upstream PRs opened | 3 | From now on, these numbers will be generated by the harness's outreach stats command, split by kind (code or paper). ## References - The referee setup follows the usual reproducibility practice: an independent re-implementation, not a re-run of the original scripts. See Peng, R. D. (2011). Reproducible research in computational science. *Science* 334, 1226-1227. [doi:10.1126/science.1213847](https://doi.org/10.1126/science.1213847) - On bugs in research software and how often they change results: Soergel, D. A. W. (2015). Rampant software errors may undermine scientific results. *F1000Research* 3:303. [doi:10.12688/f1000research.5930.2](https://doi.org/10.12688/f1000research.5930.2)