By Research tools contributorsSourceGitHub

rstack: the stack for running AI research agents — a bounded research harness, a file-based agent knowledge base, and the layer map that keeps every rule tied to something that enforces it.

Fork
Clone
git clone https://github.com/rpinho/rstack.git
Cite
Version link
https://openversions.com/rpinho/rstack/commit/da66d23a5671a71e0a1e15989da51558c2424ea4
Version identifier (commit hash)
da66d23a5671a71e0a1e15989da51558c2424ea4

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Earlier version

Checks complete

Refresh model outcome leaderboard

Committed

Claims

Findings

In an earlier line of work, running about 38 workers for two days produced 528 claims of which the reviewer passed 98%, yet none of it was any good despite every check being green.

Out of 12 guards examined in the author's system, 5 crashed instead of refusing, which allowed hooks to let them go through.

The author opened 129 PRs on other people's repositories, of which 15 are merged (6 bounds in Tao's optimization constants table and 9 bug fixes), 106 are still open, and 8 were closed, as of a November 18, 2026 snapshot.

Planted false claims caught a failure mode on 11-09 where GPT referees found the real bug and still said yes to the fake claim, which was fixed by adding one line: judge the claim as it is written.

Four reports based on NASA TESS data sent to Research Notes of the American Astronomical Society (RNAAS) were all corrected after re-auditing the numbers, with corrected versions on Zenodo crediting two earlier public sources for some of the signals.

For the MecCog pipeline on APOE4 hypotheses (H1, H2, H3), a second model checked every row without human checking, and in a 5-row check of H2 by Claude, all 5 matched while also finding 2 bad rows that the code now catches.

131 of the 356 rules in rstack have something enforcing them, while 225 are still prose.

Methods

Cross-family bug refereeing: One agent finds bugs and writes them up as claims, and a separate referee agent from a different model family clones the code at the same commit without access to the hunter's scripts, outputs, or reasoning, writes its own input, and assigns a verdict such as CONFIRMED or REFUTED.

Source excerpts

“One agent finds bugs and writes them up as claims. The referee gets the claims and clones the code at the same commit. It reads the code, but it does not get the scripts, outputs or reasoning of the agent that found it. It writes its own input and works out the right answer by itself. Then it gives each claim a verdict, like CONFIRMED or REFUTED.” · Show in paper

“The referee has to be a different model family than the hunter. If it's the same family, nothing in the verdict counts.” · Show in paper

Planted false claims: Every bug-hunt batch includes at least one fake claim (a real claim with observed and expected values swapped, or the cause in the wrong place) unknown to the referee; verdicts without a fake claim or where the referee approves the fake claim are refused.

Source excerpts

“So in the bug hunt, every batch has at least one fake claim in with the real ones. It's a real claim with the observed and expected values swapped, or with the cause in the wrong place. The referee doesn't know which one it is. A verdict with no fake claim in its batch is refused. If it says yes to the fake one, nothing in the verdict counts.” · Show in paper

Guard testing and poteto's hierarchy: Rules are enforced according to poteto's hierarchy (architecture > compiler and tests > CI and lint > bot rules > prose), and every guard is tested to ensure it refuses when it should rather than crashing.

Source excerpts

“The order is poteto's hierarchy, from Lauren Tan's pstack: architecture > compiler and tests > CI and lint > bot rules > prose.” · Show in paper

“Every guard here has a test that shows it says no when it should. And a guard has to say no, not crash.” · Show in paper

Egress gate and clean-room execution: egress-gate runs as a global git pre-push hook to scan for secrets or blocklisted words, and public-bound work runs in clean-room with an empty identity.

Source excerpts

“`egress-gate` scans a folder, a text or a push, and says no if it sees a secret or a word on your list. On my machines it runs as a global git pre-push hook.” · Show in paper

“Work that goes public runs in `clean-room`, with an empty identity.” · Show in paper

Codex spend wrapper: A wrapper around codex on PATH counts calls and stops a lane at 60 calls per hour or 3 points of the weekly allowance in a day, supplemented by a scheduled job to detect missed runs.

Source excerpts

“Now `codex` on the PATH is a wrapper that counts every call it sees. By default it stops a lane at 60 calls an hour, or 3 points of the weekly allowance in a day.” · Show in paper

Research files

Commit by Research tools contributors

Code files

Other files

rstack

rstack

rstack is the harness I run research agents on. One agent searches, and a separate one referees. This is how it works, what it found, and what still doesn't work. Every number has a source at the end. No journal has accepted any of this, so all of it is a candidate.

The problem

Agents do a lot of stuff. The hard part is to know if you can trust it.

You will not trust the AI from day one, so you have to start small. At first it does stuff bad, and then it starts doing stuff better, and your trust meter goes up. But that only works if something checks the work, and it can't be the same agent.

And the problem is not that it does nothing. It's that it does a lot, and everything looks fine. One line of work here ran about 38 workers for two days and wrote 528 claims. The reviewer passed 98% of them. None of it was any good, and every check we had was green. So the question is not if it did the work. It's if someone else can check it, and if they can say no.

How it works

It has a yes or no gate for a lot of stuff that I do. There are five parts, and they are all in this repo.

1. One agent searches, a different one checks

This is the bug hunt. One agent finds bugs and writes them up as claims. The referee gets the claims and clones the code at the same commit. It reads the code, but it does not get the scripts, outputs or reasoning of the agent that found it. It writes its own input and works out the right answer by itself. Then it gives each claim a verdict, like CONFIRMED or REFUTED. A bug email that points at a verdict fails the lint until that verdict is accepted. https://github.com/rpinho/rstack/blob/main/harness/templates/bug_referee.md

The referee has to be a different model family than the hunter. If it's the same family, nothing in the verdict counts. The family comes from the launcher's receipt, because on 11-09 a Gemini referee said it was GPT. If there's no receipt, the harness uses what the referee says about itself and flags it. Batches from before we kept the hunter's family aren't checked, and get a flag too. https://github.com/rpinho/rstack/blob/main/docs/outreach.md

2. Planted false claims

A different family is not enough by itself. It can still say yes to everything. So in the bug hunt, every batch has at least one fake claim in with the real ones. It's a real claim with the observed and expected values swapped, or with the cause in the wrong place. The referee doesn't know which one it is. A verdict with no fake claim in its batch is refused. If it says yes to the fake one, nothing in the verdict counts. The harness counts how many each referee caught. The fakes are made from the real claims, so a careful referee can sometimes tell which one it is. They catch a referee that says yes without running anything.

Same thing for exact checkers. A checker has to say no to broken certificates before we trust it.

3. Guards with tests, by poteto's hierarchy

A rule that is only written down is as good as the agent that reads it. The same rule as a hook, a lint or a test works the same way every time.

poteto's hierarchy

The order is poteto's hierarchy, from Lauren Tan's pstack: architecture > compiler and tests > CI and lint > bot rules > prose. Architecture makes the wrong thing impossible. Prose is just a note, and the agent might not read it. When something breaks, we move it up one. https://github.com/rpinho/rstack/blob/main/rstack/principles/architecture-not-prose.md

Every guard here has a test that shows it says no when it should. And a guard has to say no, not crash. If it crashes, most hooks just let it go through. We looked at 12 of my guards, and 5 did exactly that. Now we test for that too. https://github.com/rpinho/rstack/blob/main/rstack/principles/a-guard-must-refuse-not-crash.md

4. The egress gate

Agents make public stuff on the same machine as my private stuff. egress-gate scans a folder, a text or a push, and says no if it sees a secret or a word on your list. On my machines it runs as a global git pre-push hook. You have to install the hook yourself, and anything that doesn't go through git has to be scanned by hand. Work that goes public runs in clean-room, with an empty identity. https://github.com/rpinho/rstack/blob/main/rstack/tools/README.md

5. The Codex spend wrapper

On 11-17 a Python loop called Codex 1,568 times, and the launcher never saw it. Now codex on the PATH is a wrapper that counts every call it sees. By default it stops a lane at 60 calls an hour, or 3 points of the weekly allowance in a day. It's not a wall. A call to the real binary by its full path, npx, or another machine goes around it, and one long run can go over the limit before the next call is checked. A scheduled job looks for the runs it missed. https://github.com/rpinho/rstack/blob/main/rstack/tools/codex-spend.md

What it found

Merged PRs

I opened 129 PRs on other people's repos, and 15 are merged. 106 are still open and 8 were closed. That's from a snapshot on November 18, 2026, with the query in it: https://github.com/rpinho/rstack/blob/main/docs/snapshots/2026-11-18-prs.md The 6 in Tao's table are bounds, and the other 9 are bug fixes.

Planted claims

Evals are built in: exact checkers must reject corrupted certificates, and cross-family referees get planted false claims. The math checkers had to say no to 4 broken certificates for C86 and 18 for C68. https://github.com/rpinho/rstack/blob/main/harness/README.md

The plants caught something real. On 11-09 the GPT referees said yes to fake claims. They found the real bug and said yes to the fake one anyway. The fix was one line: judge the claim as it is written. https://github.com/rpinho/rstack/commit/ee2c52c

A catch is not the whole story. A referee can catch every fake one and still say yes to every real claim. That doesn't mean it would say no to a real one that's wrong. The harness warns about it, but it's only a warning.

RNAAS

We sent four reports based on NASA TESS data to Research Notes of the American Astronomical Society (RNAAS). All four were corrected after the numbers were re-audited, and the corrected versions are on Zenodo: https://doi.org/10.5281/zenodo.23895484, https://doi.org/10.5281/zenodo.23824790, https://doi.org/10.5281/zenodo.23836131, https://doi.org/10.5281/zenodo.23895440. The corrections credit two earlier public sources for some of the signals. https://github.com/rpinho/rstack/blob/main/harness/README.md

MecCog

For MecCog, a pipeline finds the papers on an APOE4 hypothesis and gets each finding out. A second model checks every row. No person checked them.

PassFixRejectPapersRows
H1516898820
H248645171372
H340591632161

H2 is the first round. Claude checked 5 rows, and all 5 matched. It also found 2 bad rows, and the code catches those now. All of this is from the pipeline's README at its November 18 commit: https://github.com/rpinho/meccog-literature-pipeline/blob/2ffa2ab105e7df6d51ce07008c9bb8e270de37f8/README.md

What still doesn't work

Most of the rules are still prose. From the README: "131 of the 356 rules have something enforcing them; 225 are still prose." I think that's fine. We only add a guard after something actually broke.

The harness can't tell if anyone was honest. From its README: "It is not an access-control sandbox, a competition platform or an independent judge of scientific truth."

The referees are models too. No human expert has refereed any of this.

References

Sources