By @rpinho

AI for biology and drug discovery: Anthropic x Adaptyv protein design, Arc Virtual Cell Challenge, Virtual Embryo Challenge, open drug-target discovery, and refereed bug reports on scientists' published software

e28bba7 / friends/salathe-covid-twitter-bert/VERDICTS.md

Raw
# Referee verdicts

All six failures reproduced from independently downloaded, unpatched pinned source. Three are **CONFIRMED**, two **CONFIRMED-MINOR**, and one **KNOWN** because its EOF-loss mechanism and empty-list crash have upstream history. The five non-KNOWN findings survive as software defects within the limits below. **NEW: no** under Ricardo's paper-result definition: no printed score was beaten, open scientific problem settled, or prediction tested on new data.

| ID | Verdict | My repro | Effect on users' results |
|---|---|---|---|
| CT001 | **CONFIRMED** | [driver](repro/CT001/reproduce.py), [result](repro/CT001/result.json), [batch input](repro/CT001/mixed.txt) | Text exceeding the sequence budget produces inconsistent tensor lengths; an ordinary mixed text-file batch fails before prediction. |
| CT002 | **CONFIRMED-MINOR** | [driver](repro/CT002/reproduce.py), [result](repro/CT002/result.json) | The documented, default-enabled So-removal option leaves residual non-emoji symbols in model inputs; downstream prediction changes are unmeasured. |
| CT003 | **CONFIRMED** | [driver](repro/CT003/reproduce.py), [result](repro/CT003/result.json), [traceback](repro/CT003/numeric-traceback.txt) | Numeric TSV labels make fine-tuning data generation crash before any training records are written. |
| FR001 | **CONFIRMED** | [driver](repro/foodrepo.js), [result](repro/FR001/result.json) | Product-detail calls, including the bundled demo's call, omit the ID and cannot select the requested product resource. |
| FR002 | **CONFIRMED-MINOR** | [driver](repro/foodrepo.js), [result](repro/FR002/result.json) | A scalar include option throws before any request; arrays work, and scalar support is described only in a helper comment in this alpha example. |
| PV001 | **KNOWN** | [driver](repro/PV001/reproduce.py), [result](repro/PV001/result.json), [current upstream comparison](repro/PV001/upstream-result.json) | A one-iteration log loses both output rows and crashes CSV export; this is an inherited Caffe EOF-loss edge case with prior reports, and the archived full log exports normally. |

## Ranked for Ricardo to read before any author contact

1. **CT001 — prediction input construction.** A single over-budget line can abort a whole mixed prediction batch. Reproduced at the shipped 96-token setting, not just an artificial small budget. It also appears unchanged in the README-linked TensorFlow notebook's prediction helper. This affects the repository's TensorFlow input code; it does not establish a failure of Hugging Face's tokenizer or model.
2. **CT003 — numeric-label data preparation.** Numeric class labels are a normal representation in the documented TSV schema, which imposes no alphabetic-label restriction. The actual main/processor/converter fails with `KeyError`, whereas the matching textual-label control writes valid records. This blocks preparing a user's dataset, rather than silently changing a paper score.
3. **FR001 — product IDs discarded.** Every nonempty-ID detail call takes the same path, including the demo linked from the API docs. This is a direct failure of the example's principal lookup operation. Its reach is limited to this bundled alpha JavaScript client; no server-side API defect is established.
4. **CT002 — ineffective default cleaning option.** The promise to remove Unicode category So is explicit in CLI help and the README command. The comparison is always false. Common emoji are converted earlier, so the independently demonstrated residual symbols are a narrower case; no checkpoint-quality improvement is asserted.
5. **FR002 — scalar include throws.** Reproducible through the public `Endpoint.product` method, consistent with the helper's stated value-or-list contract, but absent from the bundled demo and with a simple array workaround. Report alongside FR001 if the example is to be maintained.

**PV001 is not on the author shortlist.** Its one-iteration failure is real and would otherwise be CONFIRMED-MINOR, but known upstream EOF-loss and empty-list-export reports must accompany it. No inference is made about whether Marcel would care; Ricardo reads first, then the friend decides. Nobody has been contacted.

## Independent observations and normal-use checks

**CT001.** At [run_predict.py:77](https://github.com/digitalepidemiologylab/covid-twitter-bert/blob/86909698f104525b13d8ab0e22ae666e264ac5c6/run_predict.py#L77), the slice adds two to the sequence budget before two special tokens are added. The correct 96-position layout is `[CLS]`, at most 94 wordpieces, `[SEP]`, then padding if needed. My independent vocabulary-line oracle uses the word `patient`: 3 and 94 words give correct lengths `[96,96,96]`; 95 give `[97,97,96]`; 140 give `[100,100,96]`. The original file generator with a short and long line throws TensorFlow `InvalidArgumentError` while stacking word IDs. CLI help explicitly supports arbitrary `--input_text`, one-example-per-line `--input_txt_files`, and interactive input; the preparation default and README example use 96. The [notebook helper has the same AST](evidence/ct001-notebook.json). No model weights or full classifier run were needed to demonstrate the malformed inputs and batch crash.

**CT002.** At [utils/preprocess.py:69](https://github.com/digitalepidemiologylab/covid-twitter-bert/blob/86909698f104525b13d8ab0e22ae666e264ac5c6/utils/preprocess.py#L69), a one-character category prefix is compared with the two-character `So`. By hand, `Near⌂home ∑ $ 12 café!` must become `Nearhome ∑ $ 12 café!`; the helper returns it unchanged. Actual default preprocessing produces `near⌂home ∑ $ 12 café!`, still retaining HOUSE, and retains BLANK SYMBOL in `A␢B`. Both characters have category So and survive the preceding emoji conversion. A plain accented/math/currency/punctuation control agrees with its manually specified expected output. CLI help at create_finetune_data.py:176 explicitly promises So removal, with default True. This is a real contract violation, not evidence that removing these symbols improves a trained model; training/inference cleaning would need to stay consistent.

**CT003.** At [create_finetune_data.py:145](https://github.com/digitalepidemiologylab/covid-twitter-bert/blob/86909698f104525b13d8ab0e22ae666e264ac5c6/preprocess/create_finetune_data.py#L145), pandas infers numeric labels. My three training rows use labels `2,7,2`, with two development rows. Stdlib CSV independently yields the label names `["2","7"]` and training IDs `[0,1,0]`; the original main instead writes mapping `[2,7]`, and the pinned converter looks up string `'2'` in an integer-keyed map and crashes. The training TFRecord is empty. A textual control with `class_a,class_b,class_a` succeeds and its real serialized labels are `[0,1,0]`. The README documents this id/label/text TSV layout and arbitrary custom datasets. No documented requirement rules out numeric class names. The provided COVID Category dataset uses text names; there is no claim that its paper experiment crashed or used this numeric representation.

**FR001.** At [foodrepo_api_v3.js:114](https://github.com/digitalepidemiologylab/foodrepo_api/blob/61b437d948e6c21acfedcc49aa8ff6dde2a1b945/v3/code/javascript/foodrepo_api_v3.js#L114), a nonempty ID appends only `/`. My independent calls for IDs 731 and 902 both record `/api/v3/products/?`; the cloned curl docs specify `/api/v3/products/{ID}`, so their requested resources differ. A normal listing still records page number and page size. The v3 README links the JavaScript example; index.html loads this file; helper_v3.js:73 calls `endpoint.product(product0.id, null, ...)`. Thus the trigger is in the bundled normal demo, not just a speculative caller. Transport was recorded locally; whether the live server answers this malformed detail call with a collection or an error remains untested.

**FR002.** At [foodrepo_api_v3.js:27](https://github.com/digitalepidemiologylab/foodrepo_api/blob/61b437d948e6c21acfedcc49aa8ff6dde2a1b945/v3/code/javascript/foodrepo_api_v3.js#L27), scalar normalization is assigned to an unused variable. `product(731, 'nutrients', callback)` throws `TypeError: values.map is not a function` and sends zero requests; `['nutrients']` sends a request with `include=nutrients`, matching an independent URLSearchParams construction. The helper comment explicitly says value or list; the exposed Endpoint accepts includes, but has no separate scalar-form docs and the demo passes null. The file labels itself a draft/ALPHA. This warrants a minor classification. FR001 is independent and still affects the array-form detail URL.

**PV001.** At [parse_log.py:132](https://github.com/digitalepidemiologylab/plantvillage_deeplearning_paper_analysis/blob/90dd7555352a77e427918c124306780e0951f46d/googLeNet/color-80-20/finetune/results/parse_log.py#L132), parsing finishes without appending pending train/test rows. The hand-written four-line fixture explicitly contains a solver start, an iteration/lr line, a training output and a test output: expected 1/1 rows, observed 0/0; CSV export then raises `IndexError` at line 219. Adding a second iteration yields 2/2 rows and valid CSVs. All 62 parser copies are byte-identical by independently computed SHA-256. The README's normal result workflow invokes this parser, whose help imposes no minimum-iteration restriction, so a short/interrupted run is a legitimate edge case. The shipped full log yields 602/31 rows and exports normally; this is not evidence of an error in the published results.

The parser's stated upstream source has a [2014 EOF-row-loss report](https://github.com/BVLC/caffe/pull/1547#issuecomment-67324321) and [maintainer acknowledgement](https://github.com/BVLC/caffe/pull/1547#issuecomment-67342408). The later fix handles normal multi-row logs but leaves this single-iteration remainder. That precise fixture was not shown in the old discussion, so KNOWN here means an inherited, already-reported mechanism, not proof that the lab received this exact report. Current Caffe reproduces the same 0/0 row loss; [Caffe #3292](https://github.com/BVLC/caffe/pull/3292) separately fixed empty-list CSV crashing upstream. The lab copy lacks that guard. Calling the combination a fresh lab discovery would omit material prior history.

## Current status, authorship, and limits

[Source and paper references](SOURCES.md) establish Salathé as coauthor for all three associated papers and the paper-to-repository connections. The affected file:line statements above were checked against fresh pinned source; numbered excerpts are in [cited-source-lines.txt](evidence/cited-source-lines.txt).

Read-only `gh api` checks covered all visible open/closed issues and PRs with discussions, recent commits, every current repository branch's affected files, and every open PR's changed files. All three tested pins remain default-branch heads. None of the five non-KNOWN defects is described or fixed in those retrieved histories/branches; none of the repos publishes a newer GitHub release. Full details and scope limits are in [SOURCES.md](SOURCES.md); private knowledge is untestable.

The documented CT-BERT TensorFlow 2.2 install fails on this Mac. The independent tests use real TensorFlow 2.16.2 with the original pinned models converter/tokenizer; full preprocessing restores only a removed HTML-unescape API. PlantVillage/Caffe source uses Python 2, so only print syntax and tab indentation were adapted in memory. No source patch was applied. [Reproduction instructions](repro/README.md) describe every substitution and expected-answer construction. These observations support local correctness verdicts, not an unqualified reproduction of the historical cloud/TPU/Caffe environments. No model training, checkpoint download, authenticated API call, public write, or contact occurred.