By @rpinho
AI for biology and drug discovery: Anthropic x Adaptyv protein design, Arc Virtual Cell Challenge, Virtual Embryo Challenge, open drug-target discovery, and refereed bug reports on scientists' published software
# Independent referee verdicts: ExOrthist B01–B17 16 claims survive: **11 CONFIRMED**, **5 CONFIRMED-MINOR**. **B12 is INTENDED** as a variant-selection policy, with a separate zero-regulated-input crash reproduced and distinguished below. No claim is classified KNOWN; no matching public report was found. This is a software referee result, **NEW: no** under the friends lane's scientific-result definition. Every row was checked against its own fresh clone at `83185dcf04c4bda7c765a6bcf87fb1b48b36f329`. [referee.py](referee.py) constructs independent inputs and runs only unpatched source; each case contains a hand-derived `expected.txt`, cited source lines in `source.txt`, commands, return codes, stdout and stderr. No hunter reproduction, expected-output file, script, or patch was loaded. The hunter's BUGS.md was used only to identify the claims. | ID | Verdict | Independent repro | Effect on real ExOrthist outputs | |---|---|---|---| | B01 | CONFIRMED-MINOR | [repro/B01](repro/B01/) | C3→C4→C5 keeps 3 pairs instead of 2: nested exons at 980–1120 and 1010–1070 receive different overlap IDs; redundant variants survive in `-NoOverlap.tab` at coordinate digit boundaries. | | B02 | CONFIRMED | [repro/B02](repro/B02/) | The module's literal `-sp1/-sp2` command exits 2; the same inputs with `-s1/-s2` succeed. Optional `--orthopairs` exon reclustering cannot complete once its gene input is available. | | B03 | CONFIRMED | [repro/B03](repro/B03/) | Six genes in three disjoint pairs become only two genes in one component: `header=0` discards the first orthopair and first cluster-file gene; documented headerless reclustering silently loses records. | | B04 | CONFIRMED | [repro/B04](repro/B04/) | Two pairs absent from exon clusters share fabricated `Family.01nan`; a D2-discarded exon also reproduces this. Latent reclustered-output corruption behind B02/B16. | | B05 | CONFIRMED | [repro/B05](repro/B05/) | D1-generated input with genuine reverse edges makes unpatched D2 stop with `the condition has length > 1`; a qualifying gene OG aborts its clustering process/chunk. | | B06 | CONFIRMED | [repro/B06](repro/B06/) | A returned four-exon cluster reports total 3; the hub's membership is 1.12 instead of 0.90. Reintroduced exons inflate membership scores in final cluster tables. | | B07 | CONFIRMED | [repro/B07](repro/B07/) | Explicit zero sequence/length cutoffs drop a pair that passes independently calculated zero thresholds; tiny positive cutoffs retain it. User-specified filters are silently replaced by defaults. | | B08 | CONFIRMED | [repro/B08](repro/B08/) | Two 90-nt exons pass on `contig7` but fail on `contig-7`, with numeric warnings. Legal hyphens in sequence names can remove valid exon matches. | | B09 | CONFIRMED | [repro/B09](repro/B09/) | A 6:1 OG is omitted from both ≥3/≥4 categories; a 3:4 OG yields `Other OGs = -1 (-100%)`. The documented descriptive OG statistics are wrong. | | B10 | CONFIRMED-MINOR | [repro/B10](repro/B10/) | A three-exon plus-strand transcript is plotted in order 2,3,1, with incorrect downstream phases; affects plot coordinates/phase labels at digit boundaries, not inferred main-pipeline clusters. | | B11 | CONFIRMED | [repro/B11](repro/B11/) | Requesting `-min_dPSI 40` still announces 15 and calls both dPSI 24 and 65 regulated, instead of only 65; requested regulated sets and resulting conservation statistics are wrong. | | B12 | INTENDED | [repro/B12](repro/B12/) | Default variant filtering is explicit in source/help, and the code retains the first row; the retained call therefore depends on order. The resulting empty regulated set crashes, as does a one-row nonregulated control. | | B13 | CONFIRMED | [repro/B13](repro/B13/) | An unrelated species C exon at an identical coordinate overwrites the A/B pair's suffix `.003` with `.009`; latent wrong/merged exon-cluster labels behind B02/B16. | | B14 | CONFIRMED-MINOR | [repro/B14](repro/B14/) | Changing one gene ID from `oak` to `oak#extra` makes D2's input parser abort; unusual IDs can kill a chunk. The tested embedded-apostrophe variant succeeds, so the quote claim is not generalized. | | B15 | CONFIRMED-MINOR | [repro/B15](repro/B15/) | Seeds 1 and 2 swap `.01/.02` for the same two correct components; labels differ across fresh runs, but biological membership is unchanged. No demonstrated Nextflow `-resume` failure. | | B16 | CONFIRMED | [repro/B16](repro/B16/) | The documented headerless 3-column gene file raises `ParserError`; 4 columns run and 5 columns raise `KeyError`. Optional gene reclustering fails on the documented format. | | B17 | CONFIRMED-MINOR | [repro/B17](repro/B17/) | A valid OG identifier `0` makes the stats helper ignore all its genes and divide by zero; renaming only that identifier restores the correct one-OG summary. | ## Ranked author shortlist 1. **B16 — documented gene-cluster format breaks optional reclustering.** A three-column file is the README's normal input. This is an upstream blocker to the advertised `--orthopairs` workflow. 2. **B02 — exon-reclustering module supplies invalid arguments.** An independent blocker even when D3.1 has compatible input. B16 and B02 both need correction for the advertised optional workflow to finish; the latent B03/B04/B13 defects should be kept alongside that repair. 3. **B05 — ordinary D1-produced graphs can abort core clustering.** This affects the default clustering path and stops the whole process handling that part, rather than merely one OG. Its prevalence in real annotations is unmeasured. 4. **B11 — requested regulation threshold is ignored.** This silently changes the scientific comparison whenever a nondefault dPSI threshold is requested through the documented analysis helper. 5. **B06 — core output reports wrong cluster sizes and inflated scores.** A five-node, seven-edge input gives four final exons but total 3 and membership 1.12 versus 0.90. This affects the documented membership definition without involving the optional reclustering path. ## Reachability and independent arithmetic - **B01:** README algorithm C (lines 394–404) calls for collapse to one representative of overlapping exon variants. `modules/local/exorthist/collapse_matches.nf:15–17` wires C3 straight into C4 then C5, with no numeric sort. The valid reproduction is `runs/count-2`, `overlap-2`, `collapse-2`. The earlier exploratory C5 run omitted the header and is not used for the verdict. Interval intersection requires one oak overlap group; 980–1120 has two hits versus one, so the two distinct target genes require exactly two representative pairs. - **B02/B03/B16:** README lines 176–188 and 258–267 document headerless gene groups (3 columns) and orthopairs (2 columns). `params.yaml` defaults `orthopairs` to null; both modules run when a file is supplied. B03 deliberately uses four group columns to isolate header loss from B16, then headers one input at a time: only the corresponding first-record loss disappears. B16's headerless three-column run matches the actual documented convention. The 5-column failure is additional compatibility evidence, not the basis of the documented-input verdict. - **B04/B13/B15:** `subworkflows/local/exorthist/cluster.nf:36–48` supplies D3.2 with the full best-hit pair table and the filtered exon-cluster table. D2 explicitly removes some exons as unclustered, so a missing coordinate lookup is reachable. B04 `runs/pipeline-derived` uses B06's actual D2 output: discarded node08 and retained node06 both acquire `Family.01nan`. Coordinate strings contain no species namespace: identical coordinates in different genomes are not identical exons. For B13 the input explicitly assigns oak/elm to `Family.003`; adding an unrelated species C entry must not relabel that pair. B15's disjoint edges fix the two memberships independently; only the arbitrary suffix assignment changes. These scripts are tested directly using their defined flags, without claiming that the currently broken optional pipeline completes. - **B05/B06:** `modules/local/exorthist/format_input.nf` calls D1 and `cluster_exons.nf:12` calls D2; `orthogroupnum` defaults to 500. Inputs start as gene groups and exon pairs, and **unpatched D1 computes all reciprocal flags**. Every TRUE edge has its reverse, and every node is a different gene/species, so no same-species match is required. README lines 408–420 define the membership score. In B06 the four-member hub has in-degree 3, out-degree 3, and 3 reciprocal neighbors: numerator 9. There are 5 original genes, 1 same-species gene and 1 same-species exon, so the denominator is `2*(4-1)+(5-1)=10`; the correct score is 0.90. D2 uses stale total 3, denominator 8, and prints 1.12. B05's final eight-node, 21-edge witness gives initial communities of sizes 6,1,1. node07 first moves to singleton node09; the next stale singleton entry then contains both exons and crashes (see `independent-partition.txt` and `runs/D2-4`). B06's final outputs are in `runs/D2-2`. A graph-only B05 shrinking heuristic failed to preserve the actual error; it was reverted before using direct D2 execution to shrink the witness. No ExOrthist code was patched. - **B07/B08:** README lines 200–203 documents configurable similarity and a length ratio in [0,1]; `filter_matches.nf:39–41` passes them directly to C2. The default nonzero distance parameters avoid B07. `nextflow_schema.json:100–113` accepts these parameters as strings and imposes no pattern excluding zero. A 120:40 exon-length pair has ratio 1/3, so it passes a zero minimum. Sequence score 0.001 passes a zero minimum but fails the substituted defaults (0.04 for the exon and 0.03 for its neighbors). B08's exon lengths are both `389-300+1 = 789-700+1 = 90`; changing the scaffold name cannot change that ratio. No blanket assertion that every hyphenated-coordinate pair is dropped is supported. - **B09/B17:** These are a documented standalone helper (README lines 427–456), not a Nextflow process. The helper accepts `-main_output` and optional explicit gene/exon files. Fixtures contain the annotation-overlap catalog and exon clusters, so counts are ordinary pipeline-shaped inputs. B09's category answers follow directly from integer comparisons of constructed species counts; the negative Other count is invalid regardless of how the category labels are interpreted. README imposes no nonzero restriction on ClusterID, so B17 is a valid but unusual custom identifier case. - **B10:** `modules/local/exorthist/plot/generate_fake.nf:17–24` calls the script in the documented plot workflow. One GTF transcript explicitly numbers the CDSs 1,2,3 at starts 970,1120,1320 and gives phases 0,1,2. The independently expected table is `expected.tsv`: fake starts 0,30,60 and downstream phases 1,2,last. Output instead places the first exon last and gives it `DownPhase=last`. - **B11/B12:** `compare_exon_sets.pl` is the documented standalone module (README lines 541–547), rather than a Nextflow process. Its help explicitly documents `-min_dPSI` and `-allow_overlap`. B12's comment says to avoid counting exon variants, and the help says overlapping entries are filtered by default; it supplies no maximum-dPSI representative rule. `-allow_overlap` retains the regulated variant in the tested single-list case. Thus the hunter's selection-policy patch is not justified as a correctness fix. The unguarded zero-regulated denominator is a real, separate defect: the single low-dPSI-row control crashes at line 1176 without variant ambiguity. The LIST1b source also omits the overlap bypass, but the verdict does not claim an independently reproduced two-list failure. - **B14:** D2 reads D1's tabular IDs with default R comment syntax. `#` has no documented prohibition for gene IDs, and `A0_check_input.pl` checks their presence in the GTF rather than excluding this character. Prior stages also use identifier delimiters, so this is not evidence that every unusual-ID dataset reaches D2 unchanged. The confirmed defect is the hash parser failure; the apostrophe-only control did not fail. ## Upstream check and limits Checked on **2026-11-06** using `gh api`: all **42 issue/PR entries**, open and closed, **102 issue comments**, and the pull-review-comment endpoint (**0 entries**). Raw responses are in [upstream](upstream/). No matching report for these 17 claims was found. Issue [17](https://github.com/biocorecrg/ExOrthist/issues/17) concerns an older igraph/undefined-object failure, not B05/B06; issues [31](https://github.com/biocorecrg/ExOrthist/issues/31), [33](https://github.com/biocorecrg/ExOrthist/issues/33) and [36](https://github.com/biocorecrg/ExOrthist/issues/36) discuss species naming or earlier alignment problems, not the reproduced coordinate/gene parser defects. This is a bounded public-history check, not a guarantee nobody knows privately. The API reports master at `83185dcf04c4bda7c765a6bcf87fb1b48b36f329` and v2.0 at `09c26df9235d139cc298f2f3250cb298b5512654`. **Both have identical root tree `799cdc0a7a878e297e7e19c71bd9a0ac06c79177`**, so none of the cited source has been fixed on either branch. [branch-check.txt](upstream/branch-check.txt) lists the per-claim blob comparisons. The README, modules and defaults are identical too. Runtime: supplied Python 3.11 with pandas 1.5.3; supplied R 4.5.3 with igraph 2.3.4 and hashmap; system Perl. Repository configuration uses pandas 1.5.2 and its R image Dockerfile starts from R 4.4.2 with unpinned igraph. The R community witnesses are confirmed in the provided runtime; community partitioning can depend on igraph version/ties. No container or full-genome pipeline was run, and no prevalence estimate is inferred from these small examples. Runs used nice 10 and one numerical thread; no source modifications, public writes, or contacts were made. No predictive performance claim is made, so the predictive-baseline/SBS rule is inapplicable. Reproduce a case with `python3 referee.py B01` (or any B01–B17); reruns get new directories. Clones B02–B17 are fresh local shared clones from the initial fresh public clone, checked out at the full pinned revision with sparse working trees; their source is unmodified. Inputs are independently authored; current file arguments are copied beside their runs, while directory-based fixtures remain under their case folders. Graph-search and shrinking scripts are included to explain how the independent R witnesses were obtained. The hunter's suspected list is outside the requested B01–B17 adjudication.