Docker 29.6.2 is up. Checking host resources, then writing the sandbox images (egress allowlist proxy + non-root sandbox).

SDK shape is clear. Now writing the core modules: config, state DB, spend ledger, tracing, and the LLM client.

No secrets in the hook file. Committing.

While batch a finishes, I'm picking CPU-friendly dev papers for the v2 sanity check. Hand-picking is fine for dev, which exists for tuning. The held-out set will be run strictly in queue order.

Overall status is PARTIAL. Everything I can do is finished and pushed to the now-public repo, https://github.com/isHeSatoshi/paperci. The remaining items need a person: the cfo.ai plan, the demo video, the posts, the trace share links and the final submission. Each has exact steps in HUMAN_TASKS.md.

Held-out results (same 5 papers and sealed claims, v1 baseline vs v2 self-improved builder):

The honest gain is speed, cost and one validated task, not more faithful reproductions. When I checked the v2 builds by hand, 2 of the 3 were not faithful. One replayed a stored results file, and the same paper did this in v1 too. The other scored a formula the agent wrote itself instead of running the repo's code. The verifier now flags both cases. I re-checked every stored run with the stricter rules, and 3 packages were moved to registry/rejected/.

What changed this session:

• Oracle bug: the faithful held-out build first failed its own oracle check. The value it needed was split across two log lines, and a number in a section header was read as the result. After the fix its oracle earned reward 1.0. The v1 build of the same paper still fails its oracle and was rejected.
• Failure and recovery test: I stopped the network proxy on purpose before a repo download. The download failed, the pipeline restarted the proxy, the retry worked, and the failure was recorded as recovered. The agent then correctly gave up in 5 turns for $0.09 because the paper's data isn't released.
• Disk incident: drive C: filled up and Docker died during an oracle run. You freed space and I deleted the stale PaperCI build images. One dev build image alone was 17 GB. The pipeline now removes crashed containers, warns when the disk is low, and retries container crashes instead of counting them against the task.

Deliverables:

• Entire checkpoints (links are in hackathon.md):
  • 01M4KP3DNR0TF8TWQE611MJJYV on commit e70be0f
  • 01M4MKYQ3938RM8FJ5K9ETQ55A on commit e3b748d
• neatlogs traces:
  • Before/after pair on the same paper: 294209f0b2d1b60bfc1509fdbece2f1b (v1) and 4e85d604ac7a5d343950ee923f573502 (v2).
  • Recovery test: 14ed5fe7f083b2b1ff4c98fbbf705223. neatlogs failed to finalize this trace (status dlq), so it may not display. The recovery is also recorded in docs/evidence.md.
• Docs:
  • Generated from measured data: docs/evidence.md, docs/unit-economics.md (includes the cfo.ai entry sheet) and docs/dashboard_public.html (anonymized; no paper is named next to a verdict).
  • Filled in with the measured numbers: docs/posts.md and docs/demo-script.md.
  • Updated: hackathon.md, PROGRESS.md and HUMAN_TASKS.md.
• Spend: $5.44 of the $20 key. My ledger matches the spend the proxy reports.

Checks: the offline unit tests pass. The secret scan found 0 hits across the working tree, the full git history including checkpoint refs, and the session transcript before I attached it to Entire. I made the repo public only after that scan.

Caveats:

• n=5 is a small sample.
• Both validated tasks are reduced-scale runs that Haiku solved 0 of 2 times.
• C: still has only about 10 GB free because Docker's disk file doesn't shrink on its own. Optional steps to reclaim space are in HUMAN_TASKS.md item 0.

I tried to do the neatlogs dashboard work, but your CLI login can only read data, so I couldn't label or share traces. Everything else from this round is committed and pushed. Entire saved this conversation as checkpoint 01M4MNN3X23K0YACPFB290WFRG on commit 13388bd.

neatlogs dashboard

• What worked: I read every trace back through your CLI login. The numbers and trace IDs are in docs/evidence.md.

• Labels and comments: I tried to tag and rename the before/after/recovery traces so they're easy to find. neatlogs answered FORBIDDEN because the login only allows reading.

• Share links: the CLI can only check whether sharing is on, not turn it on. Public share links have to be made on the website. Making traces public would also show verdicts next to real paper names, which is why I didn't plan to publish them.

• neatlogs is having problems right now: its own health check (doctor --probe) passed all 8 checks yesterday and now fails ("stuck at pii_redaction"). 16 traces never finished processing (status dlq), and the newest run's trace hasn't shown up at all.

• What to show judges: you don't need links. Use the trace IDs in docs/evidence.md:

  • Before: 294209f0b2d1b60bfc1509fdbece2f1b
  • After, same paper: 4e85d604ac7a5d343950ee923f573502

  I also wrote docs/neatlogs-feedback.md, a ready-made bug report for the neatlogs Discord. Posting it is good sponsor engagement.

• Second recovery test: I ran the proxy-kill test again on another paper. It recovered again, and this time the failure got the new proxy_unreachable label I added after the first test.

What's built

PaperCI is "CI for arXiv": a bot that checks whether a new paper's code actually reproduces its numbers.

1. Watches new arXiv and Hugging Face papers that link a GitHub repo.
2. Filters out papers a CPU machine can't run: GPU-scale models, paid API keys, huge downloads.
3. Seals the paper's headline numbers, so the agent doing the work never sees what it has to hit.
4. Builds: an agent (Claude Sonnet) gets the repo's code running inside a locked-down Docker sandbox. It runs as a non-root user and can only reach approved package sites.
5. Verifies: a separate model family (Amazon Nova) reruns the code with no network and compares the results to the sealed numbers. It also checks for cheating: typed-in numbers, copied result files, replaying stored results, and code that never uses the repo.
6. Packages each faithful run as a Harbor-format task. A run counts only if its reference solution passes its own hidden test. A cheap model (Haiku) then tries the task to see how hard it is.
7. Improves itself: it learns rules from its own failures, enforces a turn budget, and adds packages it keeps installing to its base image. All of this is gated, and the judge, sealed numbers and held-out papers are off limits to it.
8. Heals itself: it restarts a dead proxy, cleans up crashed containers and retries infrastructure failures. The two recovery tests show this.

What we're pitching

• Problem: most papers' code is never re-run by anyone. Agent benchmarks go stale and leak into training data.
• Product: reproduction becomes a cheap, continuous, auditable signal, and every fresh paper can become a fresh, verified task for testing or training agents.
• Self-improvement:

• Honesty story: I audited our own wins. 2 of the 3 v2 builds were not faithful, so I hardened the verifier and re-judged every run. "Faithful reproductions: 1 → 1" stays in the pitch. Judges tend to trust this more than a perfect number.
• Business: a reproduction attempt costs about $0.21. About $0.18 of that is measured LLM spend; the compute part uses an assumed $0.17/hour CPU price. That's about $0.91 of LLM spend per verified held-out task, from one task, so treat it as rough. Pricing is a hypothesis: you could charge per verified task to labs and eval teams, or offer badges and audits to authors and venues. Possible customers are in docs/buyer-map.md, and the cfo.ai numbers are in docs/unit-economics.md.

Scale and scope (honest)

• Funnel:

• Cost and time: $5.66 of the $20 budget. One machine, CPU only; a run takes minutes per paper.
• Not done yet: GPU papers, full-scale runs and a large sample. The before/after comparison is only 5 papers, so it's evidence, not proof.
• Next: more held-out papers, a GPU worker class, and an improvement generation that learns from the new cheat flags.

Still yours

• cfo.ai plan
• Demo video (script in docs/demo-script.md)
• Posts (drafts in docs/posts.md)
• Final X submission on Mon 12 Oct, 7:00–11:55 PM IST
• Optional: post docs/neatlogs-feedback.md in the neatlogs Discord

PaperCI now works like CodeRabbit for papers: one command takes a paper and produces a pull-request-style review. I tested it end to end on a fresh paper: 11.9 minutes, $0.15. Everything is pushed with an Entire checkpoint (01M4MQWGPTEW7NN60B0TDQ1AZP).

What you get

• One command: python -m paperci review <arxiv-id> --anonymize --open runs the whole pipeline with live, CI-style output in the terminal. You see the paper's claims get sealed, each agent turn and command, the learned controller stepping in ("stop reading, run code"), the dry run with the network off, three offline reruns, the oracle replay and the cheap-solver attempts.
• The review page (HTML plus Markdown):
  • a pipeline strip of pass/fail steps;
  • a walkthrough of how the environment was built;
  • a claims-vs-measured table: what the paper says, what the code produced (mean ± spread over 3 reruns), the tolerance, and a status ("Reproduced", "Reproduced at reduced scale", "Differs", "Not measured", "Run not trusted");
  • CI-style checks, including anti-cheat;
  • suggestions for the authors, such as missing dependencies, data that isn't released, or numbers no script prints;
  • "reproduce it yourself", with the exact run script and every command the agent ran.
• Catches cheating: when a run replays stored results or never calls the paper's code, every claim is marked "Run not trusted". This is a real case from our own agent and makes a strong demo moment.
• For authors: a README badge (SVG) and a GitHub Action template that runs the review on each release. I haven't run the Action on GitHub yet; the file says so.
• README: rewritten around "CodeRabbit for papers", including your vision of moving from deterministic code to GPU work and then simulators (robotics, physics, chemistry). It has real, anonymized screenshots of the review page, the caught-cheating case and the live terminal.
• Browsable reviews: docs/reviews/index.html lists every review as anonymized pages. I checked them for paper IDs and titles and found none.

Stronger numbers

I ran the other 5 held-out papers so the before/after now covers all 10. Both versions are judged by the same strict rules:

The integrity checks threw out 3 runs in each version. Ten papers is still a small sample, so the docs call these signals, not proof.

Also fixed

• Disk: built Docker images are now deleted after each run (one had been 17 GB), and a batch stops if drive C: drops under 5 GB. C: never filled up again this round.

Total spend is $7.77 of the $20 budget.

For you

1. Record the demo with docs/demo-script.md (3:30, rewritten around the review). Run paperci review <fresh dev id> --anonymize --open while recording and speed it up. Pick an ID with python scripts\show_queue.py dev review 5.
2. Show only anonymized pages on screen, so no verdict appears next to a named paper.
3. Optional: turn on GitHub Pages for the docs/ folder so judges can click through the reviews online.
4. Still yours: the cfo.ai plan, the posts (docs/posts.md has a new "CodeRabbit for papers" post) and the X submission on Mon 12 Oct, 7:00–11:55 PM IST.

Installed. Now upload support in sources and claims.

PaperCI now has a web app where an author talks to an agent, instead of filling in a form and waiting. I tested it end to end in a browser on two real papers, and everything is pushed with an Entire checkpoint (01M4N5BQQ8TXMGNB3AK1WHD2A4).

What an author does

1. Open the site, enter an email, then paste an arXiv link or upload the paper. Uploads can be PDF, LaTeX (a .tex file or the source .zip/.tar.gz), Word .docx, HTML, Markdown or plain text. Add the GitHub repo if the paper doesn't link it, tick consent, and press "Start verification".
2. They land on a private page: a chat with the agent on the left, and on the right the progress steps, the hosts the agent connected to (blocked ones in red) and a live sandbox log.

How the agent behaves

• Reads, then talks first. It pulls out the paper's headline claims with their values and seals them, looks through the repo, and messages the author: what it found, its plan, and only the 1–3 questions that would change the plan. Examples: "which command produces this number?", "your data is on a university drive my sandbox can't reach, is there a mirror?", "which claim matters most?". If the author doesn't reply within 15 minutes, it starts on its own plan.
• Uses the answer. It turns the reply into instructions for the engineer agent (which claims to do first, the command to run, data links) and hides any of the paper's numbers before passing them on.
• Narrates while it works, in plain language: "Cloned your repo into a fresh container", "I've read enough, switching to running it", "Your code ran end to end with the network switched off".
• Comes back when stuck. If data is missing, keys are needed or it can't find the command, it asks the author exactly that (in the chat, and as an email), then retries with the answer, up to 3 attempts.
• Finishes and follows up. It writes a plain summary with the claim-by-claim review below it, then offers to watch the repo and re-verify on every new commit.

What the two test runs showed

• arXiv paper:
  1. The agent asked its questions, and I replied as the author.
  2. Attempt 1 got stuck, so the agent asked for the command that produces the number.
  3. Attempt 2 found the data is only on a university drive it can't reach, and said so.
  4. I asked for the write-up; it gave an honest summary with two fixes, and I accepted the watch offer.
• LaTeX source upload: it extracted 5 claims from the .tar.gz, ran the code and correctly found that a dependency needs CUDA. It suggested adding a CPU fallback.
• Problems these runs exposed, now fixed:
  • A blocked download was being mislabeled as a broken proxy.
  • PyTorch's new download host was missing from the sandbox's allowed list.
  • The agent once copied a claim value into the engineer's instructions. The hiding step caught it, as designed.

Limits you should know

• It runs only on your machine (127.0.0.1). Making it a public website needs hosting.
• Emails are saved as files in registry/web/outbox/ but not sent; no mail service is connected.
• It handles CPU-scale code only. GPU papers end with a clear explanation, not a result.
• Each run costs about $0.15–0.30. The app stops taking new papers once total spend passes $15. Spend so far is $8.33 of $20.

Try it

Then open http://127.0.0.1:8000. For the demo video, record this flow; docs/demo-script.md is rewritten around it. Add ?redact=1 to a conversation's address to hide the paper's identity, which is how I made the README screenshots.

PaperCI now runs by itself on AWS. The public live board is at https://44-215-70-124.sslip.io/board, and the author app is at https://44-215-70-124.sslip.io. The server will power itself off on Fri Oct 16 at 23:59 IST.

What it does on its own

• Autopilot: every 90 minutes, when no author is waiting, it checks arXiv, triages and seals claims of new papers, reproduces the freshest paper no earlier run has tried, and posts an anonymized review to the board.
  • The first cycle already ran. It correctly gave up on a paper that needs a GPU and a gated dataset, for $0.20.
  • Today there were no new arXiv papers (it's Sunday), so cycles work through about 24 accepted papers that were never attempted.
• Nightly learning (03:00 IST): it learns new recovery rules from the last two days of failures, kept only if they pass the existing gate. Only dev-split papers feed it, so the held-out comparison stays clean. The board shows the build rate for each rule-set version.
• Chaos drill: each night it breaks itself on purpose. The first drill killed the egress proxy; the agent detected it, classified it as proxy_unreachable, and restored sandbox internet access in 3.6 s.
• Watchdog (every 3 minutes): it repairs Docker, the egress proxy, leftover sandboxes, a full disk and the web app, and logs each repair on the board.
• Verified suggested changes in reviews: like a CodeRabbit review, each review now offers the repo patch the agent needed, a reproduce.sh script and a pinned requirements.lock.txt. All are checked by offline reruns and downloadable as paperci.patch. Lockfiles appear only on new runs.

Safety and spend

• Submitting a paper needs an invite code, and each network is limited to 3 submissions per hour. A wrong code was rejected over public HTTPS. Codes: REDACTED and REDACTED (they are not in git).
• LLM spend is capped at $1.50/day, counted from deployment, on top of the existing caps. When the cap is hit, the autopilot pauses and says so on the board. Today's server spend is $0.20.
• AWS cost should be about $12 for the window.

The account forced three changes

• Encrypted launches fail because the account's default disk encryption key is broken. The first instance died on launch, so the disk is unencrypted. Only .env (mode 600) is sensitive on it.
• The account has no free Elastic IPs. The public IP stays the same unless the instance is stopped and started.
• CloudFormation is blocked, so deploy/aws/provision.py (boto3, with up/down) serves as the infrastructure-as-code record.

LLM credits to request: +$30 on the same LiteLLM key, same models, valid through Fri Oct 16. At $1.50/day the agent reproduces about 5 papers a day. The full schedule costs about $4–5/day, at about $0.18–0.25 per attempt. Once the credits are added, tell me and I'll raise the caps on the server.

Checks: all 7 offline tests pass, including new ones for the patch extraction and the daily cap. The secret scan found 0 hits. HTTPS, the health check, the invite gate and a path-traversal attempt on the board all behaved as expected.

Commits dfe3f84 and d2ea7da are pushed and deployed. Neither has an Entire checkpoint yet; it should attach on the next commit, as before. Details are in docs/cloud.md. Your to-dos (credits, invite codes, backup with scripts/pull_cloud.ps1) are at the top of HUMAN_TASKS.md.

We make sure by keeping the healer simpler than what it heals, putting each layer under a simpler supervisor, and attacking those layers on purpose. Five drills against the live server all passed.

I checked the live system first and found six ways the healing layer itself could fail or cause damage. All six are fixed and deployed:

1. A hung agent looked healthy. The health check only proved the web server answered. Now the worker sends a heartbeat and each job type has a hard time limit, so a frozen or stuck worker fails the health check and gets restarted.
2. Bad jobs could loop forever. A crashing job used to be retried on every restart. Now the schedule moves on before a job starts, author jobs are retried at most twice, and autopilot never picks the same paper twice. Overdue jobs are killed along with everything they started.
3. The healer could cause damage. Disk cleanup could delete the image a running verification needed. It now skips images in use and never clears the build cache while a job runs.
4. Restart storms. The watchdog now restarts the app at most 3 times an hour. After that it rolls back to the last good commit once, then stops and logs that it needs a human.
5. A bad deploy could break the healer too. If the Python watchdog crashes or hangs, a plain shell script that imports nothing from the repo takes over. Deploys now go through deploy/deploy.sh, which installs dependencies, checks imports, runs the tests and checks health, and rolls back automatically if anything fails. Today's docs update went out this way (2c41883).
6. Nothing watched the VM. CloudWatch alarms are blocked in this account, so instead:
   • The kernel reboots the server itself on a panic, and every service starts at boot.
   • EC2's built-in recovery covers hardware failures.
   • A GitHub Actions job checks the health endpoint from outside AWS every 30 minutes. If a check fails, GitHub emails you. It hasn't run yet; GitHub picks up a new schedule within about 30 minutes.

Drills on the live server (5/5 passed):

The results are logged on the board in a new "Who heals the healer" section. You can rerun the drills any time with bash deploy/healer-drills.sh on the server; it refuses while a paper is running.

All repair actions come from a fixed list of safe steps, such as restarting a service or rolling back a commit, and no LLM decides what to repair. All 8 offline tests pass locally and on the server; the new one covers the hung-worker states.

Not covered:

• I didn't drill the kernel-panic reboot, because it would reboot the live server.
• Actions on the AWS account itself, like the organizers stopping or terminating the instance, can't be healed from inside it.
• If the uptime email goes unread, a dead server stays down.

The full table of layers is in docs/cloud.md.

PaperCI is now live at https://paperci.vercel.app: the app at / and the live board at /board. Vercel serves the public URL and passes every request through to the EC2 agent, which still does all the work. PaperCI itself can't run on Vercel, because it needs Docker sandboxes, jobs that run for hours and a background worker.

The VeriFix Lambda isn't ours. Our project is PaperCI, and its live app is the EC2 server.

• The two verifix-hyperx-* Lambdas sit in the shared hackathon AWS account. Other teams have about 12 servers running there too, so these are almost certainly another team's. Nothing in our repo or in my work references them.
• The "Your live app is VeriFix" message came from a different Droid session in this same folder (11a73a45, on the deepseek model). It saw those Lambdas and wrongly assumed they were yours.
• That session also left proxy files for them in runs/verifix-live/ and runs/verifix-proxy/. They're git-ignored, so they were never pushed. I left them alone; you can delete them.
• The paperci project on your Vercel team is ours; your only other projects are the two karnsheet ones.

Checked after the switch:

• /healthz, /board and / all return 200 through Vercel.
• A 6 MB paper upload goes through the proxy.
• The submission rate limit now uses each visitor's real IP from a Vercel header. Behind the proxy, the server would otherwise see one Vercel address for everyone.
• Email links and the external uptime check now point at the Vercel URL.
• The server deployed 4876fc7 through the safe deploy script, with all 8 tests passing.

I made one slip along the way: a PowerShell edit garbled ± characters in the README and docs/cloud.md. I restored both from git and redid the edits before committing.

What you need to do: run /mcp in Droid, select vercel, and finish the sign-in in your browser. Once you say it's done, I'll run the guide's two checks (a documentation search and the read-only team list).

The Vercel MCP entry is removed, so Droid will stop asking you to sign in. The Vercel CLI is still logged in and is all deploys need. To redeploy the Vercel front, run vercel deploy --prod --yes from deploy/vercel.

Before planning around prompt management, I'll check that our SDK key can actually read prompts (a read-only call).

Neatlogs and Entire are now both used more fully, and the changes are live on https://paperci.vercel.app. Health is green, and the new telemetry field shows no dropped spans. The site is ready for your friend's stress test, with one fix explained below.

neatlogs: what changed

Before this, we only used the basics: wrapped LLM calls, typed spans, guardrail spans, tags and logs. New:

• Prompt versioning. Each distinct builder system prompt is saved in neatlogs as a version of paperci-builder-system. A new version appears when the playbook generation or the run schedule changes, labelled gen-N plus the variant. Each builder trace links to the exact version it ran with. That lets the dashboard compare cost, turns and outcome across what the agent learned. Version 1 (gen 2) now exists in neatlogs.
• Sessions and users. Every trace has a session ID: one web job, one autopilot day, or one eval batch. It also has an anonymous user ID: a hash of the author's email, or autopilot. Email addresses never reach neatlogs.
• Delivery checks. /healthz now reports telemetry_problems. If a run drops spans, that shows up as an incident on the board. It is reported only and can never take the service down.

The prompt API returned 503 errors for a few minutes and then recovered. I made saving a prompt best-effort (15-second limit, retried on the next run), so it never blocks a reproduction. I added this to docs/neatlogs-feedback.md.

I skipped linking the web job and the pipeline into a single trace. It would need a parent span held open for up to 2 hours, and sessions already group them. Evaluators, code detections and session replay are dashboard features you can turn on in the neatlogs UI.

Entire: what changed

• Why recent commits had no checkpoint. commit_linking defaults to prompt, which quietly skips linking when nobody is there to answer. Checkpoints are also only saved at the end of a turn. I set it to always.
• Redaction before the public push. Email redaction is on. A local-only rule (in a file kept out of git) redacts the invite codes and your home IP.
• Attached this session. I attached it as checkpoint 01M4NMK8AM1VY3RET702J4WD7X on the new commits. Before pushing, I scanned it: no leaked invite codes or IP, and 0 hits for the real API keys.
• Still missing. The five earlier commits (dfe3f84 to 4876fc7) are already pushed, so they keep no checkpoint. Linking them would mean rewriting public history.

Entire features worth showing in the demo: entire search, recap, dispatch, why/blame, and review.

Stress-test readiness

• Fixed: the $1.50/day cap was shared with the autopilot, so your friend's jobs could have ended with "budget" by mid-day. Author submissions now skip the daily cap. They still count toward it, so the autopilot pauses for them.
• Still in place: invite codes, 3 submissions per IP per hour, the web spend cap ($17), the global hard stop, and the per-paper cap.
• Spend: $8.54 so far. That leaves room for roughly 30 more paper runs before the hard stop.
• One job at a time. Jobs run one after another, each up to about 15 minutes. A burst of submissions will queue, and the friend should expect waits. Every job has its own time limit and the watchdog guards the worker, so a hung job can't stall the queue.

Checks: all offline tests pass, secret scans are clean, and both deploys went through the safe deploy script (tests plus health check, with automatic rollback).

Self-healing agents: the interview answer

1. The healer must be simpler than what it heals. A smart healer is one more thing that breaks. Our healers are fixed, deterministic scripts with no LLM in the repair path. The model can do the work; it never decides how to fix the system.
2. A chain of supervisors. systemd restarts the web app, the watchdog checks the app, a plain shell script backs up the watchdog, and an outside uptime check watches all of it. Each layer is smaller and dumber than the one it watches.
3. "The process is up" is not "the system is healthy". Liveness alone lies: a frozen worker still has a PID. Heartbeats with per-job deadlines catch a worker that is alive but stuck.
4. Retries have limits and escalate. Retry a bad job twice, then quarantine it. Allow 3 restarts an hour, then roll back to the last good commit, then call a human. Never loop forever.
5. Do no harm to work in flight. Cleanup checks whether a job is running first. Fixes are small, reversible and logged.
6. Prove it with drills. We kill, freeze and break things on purpose: 5 of 5 drills passed live, a chaos test runs nightly, and a deliberately broken commit was rolled back automatically.
7. Learning stays separate from judging. The agent improves its playbook only from dev papers, behind a gate, and is scored on held-out papers it never learns from. Every incident is logged honestly on the public board, including the ones the system did not fix.

Short version: self-healing is disciplined restarts with evidence, not an AI fixing itself.

Your first evolution round is running on the server now, with live progress on https://paperci.vercel.app/board. The result isn't in yet: the six benchmark builds run one after another and should take about 2–3 hours.

What was wrong before

The old nightly learner adopted new rules after only plausibility checks. It never tested whether the agent actually got better with them, and it could only edit text.

What it does now

Each night, one round:

1. Proposes two changes. One is a new playbook (lessons plus turn limits). The other is a rewritten control policy: paperci/stages/builder_policy.py, the one code file the agent may change. That file decides when to force the builder to act, the late-turn nudge, and extra prompt text.
2. Screens them before running anything.
   • The policy code runs in a restricted sandbox: no imports, no file or network access, no hidden internals, no unbounded loops, and only a small fixed set of state fields.
   • That state never contains the sealed answers, so the agent can't aim at them.
   • The code is then exercised on 240 simulated builder situations.
3. Tests them for real. Each candidate and the current version actually build and verify (offline) the same two dev papers.
4. Adopts only clear winners. A candidate must score at least 1 point higher, no paper can go from "built" to "not built", and no cheating flag can fire. A flagged candidate is rejected however well it scores.
   • Playbook winners go live automatically.
   • Code winners wait for you: the pending and approve commands are in HUMAN_TASKS.md, section B2.
5. Re-checks after adoption. An adopted change is compared against live autopilot papers it never saw, and rolled back automatically if build rates drop.
6. The recursive part (depth 1). The improver's own strategy is picked by how much its past proposals actually helped. A fixed meta prompt writes new strategies from that scoreboard.

What the agent can never change: the verifier, sealed claims, tolerances, cost limits, the gate, and the loop itself. These are fingerprinted at the start of each round, and any change voids it.

Tonight's round

• Both candidates passed the static screening. So far $0.16 of the $1.50 budget is spent, on the proposals.
• The policy candidate has a real flaw. It tells the builder that PyPI and Hugging Face are blocked, but they are allowed during setup. The benchmark should score it below the current version, and it can't go live without your approval anyway. Either outcome will show whether the testing works.
• Cause: my proposer prompt never told the model the network rules. I've fixed and pushed that, but deploying it mid-round would void the round. The server will deploy it on its own once the round ends and the worker is idle.

Checks: all 10 offline tests pass, including new ones that try to sneak cheating or escape code past the sandbox. The secret scans are clean, and the Entire checkpoint was checked for leaked codes before pushing.

Limits

• Two benchmark papers is a noisy signal. That is why adoption needs a margin, your approval for code, and the live re-check.
• The sandbox is a layer of protection, not a guarantee. Your approval step exists because of that.
• Money: spend is now up to about $3/day (autopilot $1.50 plus evolution $1.50), plus whatever the stress test uses. About $8.70 of the $16 hard stop is used, so that is roughly 2 more days. The credit request in HUMAN_TASKS.md section A matters more now.

Full write-up: docs/evolution.md.

No, it doesn't wait for you to type a question. Your paper is moving: the agent has read it, sealed its claims, and posted its plan and questions in the chat. It's now waiting for your reply. If you don't answer, it starts building with its own plan about 10 minutes after asking. Answer or click "Go ahead with your plan" to start sooner.

Why it sat at "Queued Prepare" at first

That was my bug, not your input. A background job was holding the worker: one benchmark build for tonight's self-improvement round. "Authors go first" only applied between background jobs, never during one, so a submission could wait up to about 75 minutes.

Fixed and live: background jobs (autopilot cycles and benchmark builds) now check every 5 seconds whether an author is waiting. If so they stop, remove their sandboxes and resume later. An interrupted benchmark build is redone from scratch and doesn't count as a failed try. Your friend's stress test won't hit this wait.

Your submission also broke tonight's round

The round's anti-tampering fingerprint covered the sealed claims of every paper. When your paper was sealed, the fingerprint changed and the round voided itself as if it had been tampered with. It now only covers the benchmark papers. A real change mid-round now stops the round early and still decides on the runs that finished before the change. I recovered tonight's round that way.

First round result: nothing adopted

The playbook made things worse. The old nightly learner would have adopted those 4 rules automatically; the new loop tested them and said no. The policy did worse on its one finished run, and it never reached you for approval. The strategy that produced both proposals now carries a negative score, so the next round will try a different one.

Spend

The round cost $1.07 of its $1.50 budget. Your paper's run is separate and comes out of the web budget.

All 11 offline tests pass, and the secret scans were clean before each push.

The agent says "I'll write up the review now", but the job never leaves "Awaiting Author". I'll look at how replies are handled.

You were right, three things were broken. All are fixed and live on https://paperci.vercel.app.

1. Your replies were ignored. During the question step, the agent can either "start" or "write up the review now". It chose "write up", which does nothing before anything has been built, so the job stayed on "Awaiting Author". "Go ahead with your plan" also went through that same step instead of always starting. Now:
   • The button and plain replies like "go", "ok", "start" or "do everything" always start the run.
   • "Write it up as it is" during that step now runs the build first, then writes the review, with unverifiable claims marked as such.
2. The timer stuttered. It only updated when the page asked the server, every 2 seconds. The browser now counts each second itself.
3. The final chat summary gave the wrong reason. It said "the blocker was infrastructure setup". It now reports the builder's actual stop reason.

What happened to your paper: it did run. Your replies were ignored, but the job auto-starts 10 minutes after the agent asks its questions. The builder stopped after 6 turns with a real finding: the linked repository, romilbhardwaj/cilantro, is the code of a different paper (the Cilantro scheduler from OSDI 2023). It has nothing for the HotelReservation or SocialNetwork experiments in "CPU-Limits kill Performance". So nothing could be measured. That matches what the agent told you in chat: the repo doesn't contain those experiments. If the authors published the right code somewhere, submit again with that GitHub URL.

About the sandbox view: it stayed empty because nothing had started yet. While a build runs, it's the collapsible "Live sandbox log" panel at the bottom right.

All 12 offline tests pass, and the page script passed a syntax check after the change. I didn't re-test the reply flow end to end on the live site. A fresh submission is the way to confirm it: click "Go ahead with your plan" and the build should start within seconds.

Here is how self-healing works today, which parts are actually agentic, and what I'd do to turn this into a product.

How the self-healing is built

It is layered. Each layer is smaller and simpler than the one it watches, and none of the repair steps uses an LLM.

1. Inside a run (the agent recovers from its own errors)

• Every failed command is sorted into a failure type: missing module, blocked host, out of memory, GPU required, and so on. The builder then gets matching advice from the playbook.
• If the same failure happens 3 times, a stall breaker forces a change of approach or an honest give-up.
• When the builder submits, the harness runs its script with the network off. If that fails, the builder is told why and tries again, up to 3 times.
• Every pipeline stage saves its result, so a crashed run resumes where it stopped. A stage that fails is retried up to 2 times. If Docker or the egress proxy broke, they are repaired first.
• If an evolved control policy misbehaves mid-build, the builder switches back to the baseline policy for the rest of that build.

2. Per job (one bad job can't take the agent down)

• Every job has a hard time limit; past it, the whole process tree is killed.
• A job that keeps crashing the worker is retried at most twice, then marked failed.
• The schedule moves forward before work starts, so a crashing job is skipped rather than retried in a loop.
• Author submissions now pause background work within seconds.

3. Per process (is the agent alive and actually working?)

• systemd restarts the web app if it dies, and the OS protects it from being killed when memory runs low.
• The worker writes a heartbeat with a deadline for its current job. /healthz returns an error when the worker is stuck, even if the process is still running.

4. Per machine (the watchdog, every 3 minutes, as root)

• It checks Docker, the sandbox network and proxy, leftover sandboxes, free disk, and web health.
• Web restarts are capped at 3 per hour. After that it rolls back to the last good commit, then escalates.
• If the watchdog itself breaks, a plain shell-script fallback takes over.
• Deploys roll back automatically if tests or the health check fail.
• The kernel reboots on panic.
• A GitHub Action checks the site from outside every 30 minutes, in case the whole machine is gone.

5. Proof that it works

• Nightly chaos drills deliberately kill the proxy or leave a dead sandbox behind.
• Drills against the healers themselves passed 5 of 5 live: SIGKILL, frozen process, watchdog crash, Docker stop, and a deliberately broken deploy.

6. Behaviour (the evolution loop)

• Changes to the agent are tested before adoption, re-checked on live papers, and rolled back if build rates drop.

What it does not cover. Today you found three bugs (the job stuck in "Awaiting Author", the falsely voided round, and the wrong summary). None of the layers above caught them. They handle crashes and hangs, not a system that is up but doing the wrong thing. Also, the board says "AWS reboots the VM if it stops responding". I can't confirm any alarm for that: CloudWatch was denied in this account, so only the kernel panic reboot is certain. I should soften that sentence.

What's actually agentic

In one sentence: the agents do the open-ended work (getting unknown code to run, talking to authors, improving themselves), and fixed rule-based code does the judging and the repairing.

What I'd do to make it a product

1. Catch "up but wrong", not just "down".

• Submit a synthetic paper every hour and check it reaches "done" within an expected time. This would have caught all three of today's bugs.
• Alert when a job hasn't changed state for N minutes.
• Send real alerts to Slack, email or a pager; today incidents only appear on the board.

2. Real infrastructure.

• A proper job queue with many sandbox workers instead of one VM.
• Stronger isolation: Firecracker or gVisor microVMs instead of plain Docker.
• A paid GPU tier.
• Postgres and object storage instead of SQLite and local disk.
• Everything defined as infrastructure-as-code (not possible in this hackathon account).

3. Meet users where they work.

• A GitHub App that posts a reproducibility check on every push and adds a badge to the README.
• Plug into conference artifact evaluation (ACM badges) and OpenReview.
• A pre-submission check for labs, plus an API.

4. Make the judgement trustworthy.

• A benchmark of 50+ papers with human-labelled verdicts.
• Several seeds and paired statistics for evolution decisions instead of 2 papers.
• A dispute flow for authors.
• Never publish a verdict about a named paper without consent (already the rule).

5. Cost.

• Cache the working environment per repo, so re-verifying after a new commit costs cents.
• Predict the cost of a run before starting it.
• Send easy steps to cheaper models.

6. Business model. Two revenue lines:

• Verification as a service for conferences, journals and labs, priced per verified paper.
• The by-product: every verified paper becomes a packaged, replayable environment with a difficulty label. AI labs pay for those as training and eval tasks.

7. Accounts and safety. GitHub sign-in instead of invite codes, per-organization spend caps, abuse limits, and audit logs.

If you want one next step before the deadline, I'd build the hourly synthetic submission. It's about an hour of work and closes the gap that today's bugs went through.

Yes, this works, and it's better than what I proposed. The project is Hugging Face's ICML 2026 Open Reproductions challenge. In July, 1,221 people used coding agents to reproduce 2,226 ICML papers claim by claim. Every verdict was frozen in a public dataset (ICML-2026-agent-repro/verdicts), and the full paper list is in ICML-2026-agent-repro/challenge. I downloaded both and measured them:

• 6,885 logbooks (one write-up per reproduction attempt), with 35,908 claim verdicts: 17,167 verified, 9,425 inconclusive, 7,617 toy, 1,699 falsified. These are almost exactly our verdict labels.
• Paper-level consensus, counting only papers with 2 or more medium- or high-quality logbooks and an arXiv ID:
  • 47 papers verified by independent teams
  • 69 falsified by independent teams
  • 430 contested (teams disagree), which is the most interesting set for us
• The most common area is Theory (213 of those), then Deep Learning (83).

How the eval would work (your idea):

1. Pick ICML papers where Hugging Face's teams agreed (verified or falsified) and that have code and pass our CPU triage.
2. PaperCI runs them blind, without seeing their verdicts.
3. Score our agreement per paper and per claim. A cheap model matches our numeric claims to their text claims.
4. Every disagreement goes to a human review queue: you decide who was right, using both sides' evidence. Those decisions become human-verified labels, so the eval improves every time it runs.
5. The same set becomes the evolution benchmark, replacing my 2 dev papers.

Caveats:

• Their labels come from a model (GLM-5.2) reading agent write-ups, so they are strong signals, not ground truth. Only the roughly 35 falsifications Hugging Face re-checked adversarially are close to human-verified, which is exactly why step 4 matters.
• They ran on GPUs; we run on CPU only. Many papers will come out "blocked" or "toy" for us, so we filter by our triage first. Theory papers with small numerical checks fit our sandbox best.
• Our rule still holds: our verdicts on named ICML papers stay private. Public outputs only show aggregate agreement and anonymized IDs.
• On "instead of synthetic papers": a real paper with a known outcome makes a better daily golden check, but each run costs about $0.20, so hourly would cost about $5 a day. I'd keep a free, no-LLM hourly check for "is the pipeline flowing", and run one real ICML paper as a daily golden test.

Cost:

• Screening all 546 candidates (rule filter, then cheap triage and claim sealing): about $2–3.
• Each blind reproduction: about $0.20, so 10 papers is about $2.
• About $7 is left before our $16 hard stop, so the credit request matters now.

A few decisions before I build:

Screening is at 6 runnable out of 42 checked. While it runs, I'll commit the docs.

The first DeepSeek run is still building: 14 builder turns so far, about $0.06 by my estimate. Screening is paused until GitHub's rate limit resets in about 27 minutes, then it resumes on its own.

What's left is all running on the server:

• Screening the remaining ~80 papers takes about 5–6 hours because of the GitHub limit.
• Each newly runnable paper is queued automatically when screening ends.
• Each blind run takes 30–60 minutes, so results will come in through tomorrow.

Ask me anytime and I'll check the results and write up the comparison: agreement with the Hugging Face consensus, coverage, and why each disagreement happened. A read-only GitHub token would cut screening to about 30 minutes.

The icml_eval.py change was my own splice script, and git shows no other edits. Next I'll add the token redaction and put the token on the server through stdin.

Tests pass. Next I'll see how scripts/reverify.py re-judges stored runs, so the false-flagged ICML run gets corrected.

I'll finish the verifier fix rollout first, then rework the README based on your feedback.

The README is done, at about the same length as before. While the demo job runs, I'll check the evolution round, which the verifier deploy should have stopped early.

The meme works. Next, the README text around it: no jab at reviewers, and a matching alt text.

That error is only a missing favicon. Next: commit the README, memes and screenshot, then scan and push.

Everything you asked for is done, and the agent will keep improving on its own until 18:00 IST. My monitor now checks the server every 2 hours (2:05, 4:05, …) and fixes what breaks.

Learning from community logbooks. This was the idea you didn't follow; here it is in plain terms:

• What it does: Hugging Face's ICML dataset has thousands of logbooks where people wrote down how they got a paper's code running. Each night the improver reads 8 of these next to PaperCI's own repeated failures. It then writes general fixes, such as "download the data during setup and check the files exist before the offline run".
• Why it stays fair: it only reads logbooks for papers outside our ICML set, so no benchmark or eval paper is ever seen. A gate also rejects any rule that names a paper, repo, logbook or result number.
• How a fix gets kept: each fix competes in an evolution round alongside the agent's own proposals. It is adopted only if it scores higher in real benchmark runs.
• Dry run: 4 good rules came out of 8 logbooks and passed every check.

Improvement tonight

• Evolution rounds now follow each other with a 30-minute gap instead of once a night. At 18:00 IST it goes back to nightly by itself, which covers "self-healing on its own from tomorrow".
• I found and fixed a verifier bug. It flagged honest runs as "never used the repo's code" when the agent wrote a small driver script that imports the repo. One ICML result changed from inconclusive to toy-scale after the re-judge.
• Evolution and ICML runs now take turns on the worker. A deploy no longer burns a run's attempt.
• GitHub token: it's on the server only, and the GitHub limit is now 5,000 requests per hour. It's redacted from Entire checkpoints; I scanned every checkpoint and found 0 leaks. Revoke it after the event.

ICML eval. 20 papers are runnable: 2 benchmark papers plus 18 eval papers, all queued. Three runs have finished so far, all built at reduced scale.

README

• Framed as "CodeRabbit for research papers: claim verification", with the paperci.vercel.app link at the top.
• The price comparison uses Enago's published $272, $535 and $799 for 1, 2 or 3 reviewers. It's worded so human review is valued, not mocked: reviewers can't rerun the code in a few hours.
• A vision section covers simulators beyond code and verified papers becoming benchmarks.
• About the same length as before. The first meme now says "PaperCI's agent verifies each claim in your paper by rebuilding the repo in a sandbox", with no CodeRabbit mention.
• Two agent-chat screenshots sit side by side:
  • Not verified: the earlier screenshot, where the agent got stuck on unreleased data.
  • Verified: a real run I made tonight through the live site on the ICML paper "Are Two Datasets Close Enough…". All steps are green and 3 of 5 claims were reproduced at reduced scale.
• TECHNICAL_WRITEUP.md from earlier is still in the repo. I made no further changes to it. Say if you'd rather I remove it.

Tests pass (15 offline checks). Everything is pushed and deployed.

When you wake up: if an evolved control-policy change scores higher on the benchmark, it waits for your approval. Run python -m paperci.evolve pending on the server.

Night watch, 02:05–02:25 IST: the server is healthy and I fixed three problems.

Status: health check OK, watchdog fresh, 40 GB disk free, Docker up. 5 of 18 ICML eval runs have finished: 4 ran at reduced scale and 1 ran out of turns twice. A new evolution round starts after the current ICML step. Nothing is waiting for your approval.

Fixed:

• Hugging Face downloads were blocked. Their download CDN (us.aws.cdn.hf.co) wasn't on the allowlist, and it was refused in 4 runs, up to 159 times in one. I allowed it, plus bare openml.org, then rebuilt the proxy. A test request through it gets 200, and hosts off the allowlist are still refused.
• Proxy updates silently didn't apply. The allowlist is baked into the proxy image, which deploys never rebuilt, and deploy.sh could run a mix of old and new code when it updated itself. Both are fixed, and the new script passes a syntax check on the server.
• ICML scoring was misleading. Reduced-scale runs counted as disagreeing with the Hugging Face verdict, which pushed agreement to 0%. They're now shown as "reduced scale only", with their claim matches counted separately.

The verifier fix earlier stopped the previous evolution round before any candidate was scored, so no candidates were judged. Tests pass, the checkpoint scan found 0 secrets, and everything is pushed and deployed. Spend over the last 6 hours, as estimated in USD for Factory credits: about $1.85 in total, $1.14 of it evolution. Notes are in data/night_watch.md.

Night watch at 04:05–04:20 IST. The server is healthy; I fixed one builder problem and caught a near-leak before it was pushed.

ICML eval so far: 13 of 18 papers done.

None reached a full-scale verdict yet. That's expected on a CPU-only server.

Fixed:

• Run scripts lost at the turn limit. One builder wrote a working run script but ran out of turns before submitting it. The harness now submits an existing script once by itself, through the same offline dry run. It is always marked reduced scale, so it can never produce a "verified". It's tested, and I deployed it before any run in the current evolution round had finished, so the round stays fair.
• Token fragment in a checkpoint. A new Entire checkpoint contained a 10-character piece of the GitHub token, picked up from my own scan commands; the full token stayed redacted. It was never pushed. I added a redaction rule for it to the local settings, and pushed the fix without that checkpoint.

Evolution: round 0210 is evaluating, with the current version and three candidates (pb, pol, lb) each on the 2 benchmark papers. My deploys cut the two earlier rounds short. Nothing has been adopted and nothing needs your approval.

Spend, last 2 hours (Factory estimates): about $0.77 on ICML runs and $0.13 on evolution.

Still open, for the next watch: why two runs gave up early at turn 3–4, and the run with no target metric. The read-only exploration case is left to evolution.

Details are in data/night_watch.md.

Night watch at 08:10–08:30 IST: healthy, and the agent got measurably better overnight. Evolution adopted 2 playbook changes, and both are now on canary watch against live papers.

Evolution: rounds ran back to back all night. The integrity tripwires rejected two candidates (one logbook, one policy) whose benchmark runs never called the repo's code. All policy candidates scored below the current version, so nothing needs your approval.

ICML pass 1 is finished (18 papers):

The 11 that built are 7 reduced-scale runs (8 of 16 measured claims matched) and 4 inconclusive. None reached a full-scale verdict, which is expected on CPU.

New this watch: a second blind pass. The same 18 papers are rerunning with the improved agent: the adopted playbooks, the auto-submit fix and the Hugging Face CDN fix. The agent never learned from these papers, so pass 1 vs pass 2 is a clean before/after for the submission. These runs go first in the queue, and /icml will show both passes. Expect it to finish in 7–8 hours, around 16:00 IST.

What broke: one deploy rolled back automatically. A test depended on the new queue-priority setting on the server. I fixed the test and redeployed.

Spend, last 2 hours (Factory estimates): about $0.82, all on evolution.

When pass 2 finishes, I'll put the before/after numbers in the README.

Plan is up-to-date.

The login client is still waiting, so neatlogs hasn't recorded the approval for this code yet.

The ICML job is 11 minutes into a build (paper 2509.19771). I'll keep waiting for it to finish; the deploy fires automatically right after.

PaperCI now uses neatlogs properly, but the server is still on the old code. A deploy is queued there and runs by itself as soon as the current ICML pass-2 job finishes, so that run isn't cut off. I've stopped checking on it; the 2-hourly loop will confirm the deploy and that new traces show input, output and detection hits.

What was wrong

• Odd trace inputs: spans had no input or output set, so neatlogs showed the first log line as the trace input. That line was "prompt registration skipped".
• Prompt registration failed on every run: neatlogs allows only one label per prompt version, and the builder sent two. It now sends one label (the variant) plus a tag for the playbook generation.
• "Execution Failed" never fired: failed runs were recorded as successful (OK) spans.

What changed (pushed as f22b3c7 and d7b8157)

• Traces: each run and stage now has an input and output (the paper goes in, the verdict comes out). Factory LLM calls now carry token counts.
• Failed runs: a run marked failed now shows as ERROR in neatlogs, so "Execution Failed" fires. That covers turn limits, sandbox errors, integrity flags and a failed offline run. An honest give-up (for example, the paper needs a GPU) still counts as a correct answer.
• Six new PaperCI detections: builder ran out of budget, integrity flag, proxy refused a host, rejected dry run, honest give-up, and claims verified. They only score new traces, so they start working after the deploy.
• Incidents: a new paperci-server service account can only touch this project, and its token expires in 7 days. The token went straight into the server's .env and was never printed. After each job, every builder failure category that didn't recover in the last 24 hours becomes a native neatlogs incident linked to its traces. A category that stops failing gets a "quiet" update.
• First live publish: 11 incidents, including turn budget exhausted, missing data, egress denied and missing module. Please check they appear on your Incidents page; the CLI can't list incidents.
• Docs: I rewrote the README's neatlogs section to say only what the code does. I also updated hackathon.md, added the new variables to .env.example, and added an Entire redaction for the new token format.

Limits and side effects

• The server can't post trace feedback or comments. neatlogs only accepts those from a logged-in user, not a service account.
• Incident "findings" written by an outside tool are rejected, so the analysis goes in the incident description.
• One token failed to capture during setup; I revoked it and minted a replacement.
• A test incident called "probe" may show on the Incidents page with no occurrences left. It came from working out what the API accepts, and you can ignore it.
• The deploy changes pipeline.py, which the evolution round locks. The round that has been waiting behind the ICML priority will stop early and decide on what it has; fast mode keeps starting new rounds until 18:00 IST.

All 18 offline tests pass, and the commits scanned clean for secrets. The pass 1 vs pass 2 numbers go into the README after pass 2 finishes (around 16:00 IST).

For you after the event: revoke the paperci-server neatlogs token, along with the Factory key and GitHub token.

Zero hits in the new checkpoint and in HEAD. Pushing:

I'll pull the official judging weights and evidence rows, so the page quotes them exactly rather than copying the other team's version.

Now a real-data test. I'll render the page on the server from a temp copy, then view it locally in the browser.

The judges page is live at https://paperci.vercel.app/judges, and you now have a neatlogs story to point at: three real bugs neatlogs flagged, each fixed, and two of its incidents already closed with the fixing commit named. Everything is deployed (9c53a8f) and all 21 offline tests pass locally and on the server.

What neatlogs found and what each flag turned out to be. All 10 "Execution Failed" hits came from 5 traces:

The page also shows the tracing fix working: since it went live, 8 of 8 paper traces carry token counts and 5 failed runs are marked ERROR. In the 12 hours before, it was 2 of 91 and 0.

The judges page is a PaperCI review of our own submission:

• Each of the 7 official evidence rows is a claim, re-checked live against git, the neatlogs API and the run database.
• Each claim has a command a judge can run to verify it, and a note on what it doesn't show.
• It includes the triage table above, with live "since the fix" counts, plus a real neatlogs trace drawn as a waterfall that needs no login.
• judges.json lets a judge's own agent check the same data.
• Current status: 5 pass, cfo.ai partial, building in public missing. Those last two stay that way until you add links.

Entire: every fix commit carries the checkpoint of the session that wrote it, and the triage table shows it next to each fix.

Also found and fixed:

• Playbook reset: every deploy since 08:10 had reset the learned playbook to the committed generation, so ICML pass 2 ran without the overnight improvements. Deploys now keep it, and generation 4 is restored.
• Orphan sandbox: a sandbox left over from a killed job ran for an hour next to a benchmark run.
• ICML pass 2: flat against pass 1 (11 vs 11 built, 0 full-scale verdicts). The README now says so and gives the playbook reset as the reason.

For you before submitting:

1. Share three traces in neatlogs (open each and click Share): a3018a6313414b1ad77c8db639be57f0, 29c2dee743f80c15c5e43f9f2b231bdb, 59a1c806c9921f4fe1723f14518702da. Send me the public links and they'll show on the page.
2. cfo.ai plan: send the link and that row turns green.
3. #neatHack posts: drafts are in docs/posts.md. Building in public is 15% of the score, and that row stays red until there are links.
4. Demo video: the script and shot list are in docs/demo-script.md. It's a 3-minute story built around the neatlogs triage, with exact trace IDs and pages for each shot.

The rules don't set a number of posts. The building-in-public row only asks for "links to what you shared during the hackathon, tagged #neatHack", and that category is judged on how you shared while you built. So 7 isn't required, and one thread of 5 is fine. Posting it today still counts, because the hackathon runs until submission closes. Once it's up, send me the link and the "You built in public" row on /judges turns green.

Your outline had 7 parts, so to fit 5 I merged neatlogs with the self-improvement story, and the vision with the limitations. Each post is under 280 characters (links count as 23). I used plain names for neatlogs, Entire and cfo.ai; swap in their @handles if you know them, and recheck lengths after that.

1/5 PaperCI: CodeRabbit, but for research papers. Pre-submission review costs $270-$800 per paper, and reviewers rarely rerun the code. PaperCI rebuilds your repo in a sandbox, reruns the experiments and checks every headline claim, for cents. paperci.vercel.app #neatHack

2/5 How: claims are sealed (sha256) so the builder agent never sees the numbers it must hit. It rebuilds the env behind an allowlist proxy; a different model family reruns it 3x offline and flags shortcuts. Runs 24/7 on AWS: it heals itself and rolls back bad deploys. #neatHack

3/5 Real test: 18 ICML 2026 papers the Hugging Face community already reproduced, run blind (the agent never sees their verdicts). On a 2-CPU box it built 11, got reduced-scale evidence on 9, and 0 full-scale verdicts yet. All public: paperci.vercel.app/icml #neatHack

4/5 neatlogs shows what fails; an evolution loop writes a fix, tests it on a fixed benchmark and keeps it only if it scores higher. neatlogs caught 3 bugs in one morning; each fix closed its incident. Held-out papers built: 3→5 of 10. Entire checkpoints every fix. #neatHack

5/5 Limits, honestly: CPU only, minutes-scale code; most ML papers need GPUs, so many stop at triage, and numbers only in figures are missed. Next: GPU runs, then papers whose evidence comes from simulators. Pricing and runway planned in cfo.ai. Try it: paperci.vercel.app #neatHack

Post 5 mentions cfo.ai, so finish the plan before posting, or drop that sentence.

Plan is up-to-date.

I raised the server's spend limits so they won't get in the way. I kept them as guards against a runaway loop or abuse, not as budgets.

The daily pacing was the one actually holding the agent back: it paused the agent's own arXiv cycle each day once it hit $1.50. Now that cycle can run every 90 minutes all day. The per-paper limit of $0.60 and 3 submissions per network per hour are unchanged.

The service restarted healthy, and I logged the change in hackathon.md (pushed as 6dba6b4).

With LLM spend no longer a constraint, the limit is the machine itself: 2 CPUs, one sandbox job at a time. Public submissions still go ahead of the background work.

I made the change, and the rate limit is now live at 10 submissions per network per hour. cfo.ai still needs you to log in once before I can use it.

Rate limit

• The limit used to be a fixed 3 per hour in the code. It is now a setting, PAPERCI_SUBMITS_PER_HOUR, with a default of 10 (paperci/web.py:43). The server has no value set, so it uses 10.
• The $0.60 per-paper cap and the spend ceilings are unchanged.
• I updated docs/cloud.md, hackathon.md and .env.example to match.
• The offline tests pass. I deployed commit e07057f with an Entire checkpoint attached, and the health check reports OK.
• The secret scan before pushing flagged one item: the checkpoint transcript holds the short display prefix of the first neatlogs token. That token was revoked this morning, and a prefix can't be used to log in, so I pushed.

cfo.ai The cfo.ai server is added to your Droid setup, but it is still waiting for login. I can't do that part myself because it needs your browser:

1. Run droid mcp auth cfo-ai in a terminal, or type /mcp in this chat, pick cfo-ai and choose Authenticate.
2. Approve access in the browser window that opens.
3. Tell me when that's done.

After that, I'll start a short cfo.ai chat about PaperCI's costs, pricing and runway, using the numbers in docs/unit-economics.md. Then I'll take screenshots and put the plan link in docs/judges-links.json so it appears on /judges.

The three trace shares, the 5-tweet thread and the demo video are still waiting on you.

Nothing is broken. I found one problem and fixed it in code. The fix (9428ddb) is pushed, but it is not deployed yet: a script on the server will deploy it automatically when the current evolution round finishes.

• Health: The service, watchdog, disk (36 GB free) and Docker are all fine.
  • The two restarts this morning were my own .env edits. Each one killed a benchmark run, and the round automatically ran both again.
  • Leftover sandboxes were cleaned up automatically, and no stray droid processes remain.
  • The 10-per-hour submission limit (e07057f) is live, and the learned playbook is still at gen 4.
• What broke: In 2 runs on the benchmark paper 2602.05395, the proxy blocked Google Drive's download host (doc-*-docs.googleusercontent.com). Both runs then used up their turns without finishing. My earlier Drive fix had missed this redirect host.
• Fix: I added that host pattern to the proxy allowlist. Tests pass, and the secret scan of the commit and its Entire checkpoint found nothing.
  • The allowlist is a frozen file, so deploying it now would cut round 1309 short with 2 of 8 runs left.
  • The script checks every 10 seconds and deploys once the round stops evaluating, for up to 3 hours. As of 14:50 IST the round was still running.
  • The next night watch will confirm the deploy landed.
• ICML eval: Pass 2 is finished (18 of 18) and nothing is queued, so the tally is unchanged: 11 built, 9 reduced-scale, 8 of 22 claims matched, 0 full-scale.
• Evolution: Two playbook adoptions (0210-pb and 0350-pb) are still under canary watch. No policy candidates are waiting for your approval.
• Spend: $0.58 in the last 2 hours (almost all evolution), $8.40 in the last 24 hours, and $16.17 all time.

The summary is added to data/night_watch.md.

Nothing is broken now. One builder bug turned up and is fixed and deployed (42beb3a), and the earlier Google Drive allowlist fix is live.

• Health: The service, watchdog, disk (36 GB free) and Docker are all fine. The waiting script deployed the Drive fix (9428ddb) right after round 1309 finished, and the proxy now allows Drive's download hosts.
• What broke: In round 1426, a builder gave up saying the sandbox was empty and had no network. In fact, all 6 of its commands had run fine.
  • The cause is that every droid exec call adds the CLI's own working folder (a temp folder holding prompt.txt) to the model's context. The model mistook that folder for its sandbox.
  • Tool access was not the problem. A probe confirmed the model can't read files there.
• Fix: The CLI now runs from an empty folder, with the prompt file kept outside it. A short system-prompt note tells the model the CLI's own environment is unrelated to the task.
  • I added an offline test, and all tests pass.
  • A live call on the server with the new flags worked: the model no longer makes up a file listing.
  • llm.py is not a frozen file, so this deploy didn't cut the round short.
• Evolution: Round 1426 adopted a new playbook as gen 5 (+2.04). It is under canary watch alongside the two earlier adoptions.
  • The round's policy candidate was rejected by the no_repo_code_path tripwire. It ran notebook functions it had copied out of the repo, so rejecting it is cautious but fair.
  • Round 1524 is evaluating. Nothing is waiting for your approval.
• Author submission: Someone submitted 2510.10747, a paper that depends on Kubernetes. The builder wrote its own microbenchmark instead of running the repo's code. The verifier flagged no_repo_code_path and returned "inconclusive", which is the correct call.
• ICML eval: Pass 2 is finished and nothing is queued.
• Spend: $0.76 in the last 2 hours, $8.60 in the last 24 hours, and $16.94 all time.

cfo.ai: The cfo-ai entry is gone from your MCP config. A new server, nfoai-neathack, is there and shows "needs authentication". If that's the cfo.ai one, run /mcp, pick nfoai-neathack and log in. Once it's connected I can start the PaperCI chat and take the screenshots.

The summary is added to data/night_watch.md.

There was no error on your demo job. Your post is live on /judges (the row reads "1 #neatHack post, PASS"), and HUMAN_TASKS.md now lists only what's left. Both are deployed.

1. Why "Resolved" shows 0 in neatlogs. The neatlogs API and CLI can only publish incident snapshots; they can't change an incident's status. When a fix landed, PaperCI sent an empty snapshot, and neatlogs moved those 6 incidents to "Investigated". Marking them Resolved has to be done by hand in the dashboard. That takes about 2 minutes and is a nice 5-second shot for the video. Leave the 3 Open ones open: they are live agent-behaviour problems that feed the evolution loop.

2. The demo job. "CPU-Limits kill Performance" is the paper's title, not an error. The run finished in 3 min 29 s.

• The paper's main results need a Kubernetes cluster, which the sandbox doesn't have. The agent wrote its own microbenchmark instead.
• The verifier refused to trust numbers that didn't come from the repo's code and returned "inconclusive". That's the correct call, but it's a weak demo.
• For the recording, submit arXiv 2507.12843. The repo is found automatically, it takes about 3–5 minutes, and last time it matched 3 of 5 claims within tolerance. Submit it once now so you have a finished page to fall back on.

3. That row on /judges. neatlogs flagged a trace as Execution Failed and grouped it into the turn_budget_exhausted incident. That trace was an evolution candidate on test paper 2602.05395 that used all 18 turns without submitting. The row says this is the agent's behaviour rather than a bug in PaperCI, so it goes to the evolution loop instead of getting a code fix (hence no fix commit). "35 in the last 24 h" is the live count the next evolution round learns from.

4. What's left tonight, apart from the video and cfo.ai (details in HUMAN_TASKS.md):

1. Share 3 neatlogs traces, about 3 minutes: a3018a63…, 29c2dee7…, 59a1c806…. Open each one, click Share, and send me the links so I can put them on /judges.
2. Mark the 6 "Investigated" incidents Resolved (optional, about 2 minutes).
3. Check Entire, about 2 minutes: on entire.io, open a recent commit in isHeSatoshi/paperci and confirm its checkpoint loads.
4. Final submission, 7:00–11:55 PM IST: quote the official neatlogs post on X with the team, "PaperCI", the repo link, the demo video link and https://paperci.vercel.app/judges.

Send me the video link when it's uploaded, and the cfo.ai plan link if you make one, and I'll add both to /judges.

After the event: revoke the Factory key, the neatlogs token and the GitHub token. The AWS server stops itself on 16 Oct.

The traces do exist. The API returned all three from your project. They are part of the evolution round 1108 run (session run:evo-20261012-1108) in workflow paperci-reproduce-paper, and all three have status ERROR. The dashboard search most likely doesn't match raw trace IDs, or its filters left these out.

Here is how to find them on the Traces page:

1. Set Workflow to paperci-reproduce-paper.
2. Set the time range to All time (or today, 11:30–12:00 IST).
3. Filter Status = Error, or filter by Session run:evo-20261012-1108.
4. Look for these start times (IST):
   • 59a1c806… started at 11:38
   • a3018a63… started at 11:54
   • 29c2dee7… started at 11:59

Open each trace, click Share, and send me the links. I can't get share links myself because the server's neatlogs token isn't allowed to read or create them.

If they still don't show up, check that the project picker at the top left says Aditya's project, since that's the project these traces are in. You can also send me a screenshot of the Traces page with its filters showing.

The three trace links are now on https://paperci.vercel.app/judges, under the neatlogs trace section, and the change is deployed.

These links aren't public, though. When I opened one while logged out, neatlogs sent me to its sign-in page, so a judge without access to your project can't open them. That's why the page labels them "neatlogs dashboard links (sign-in required)" and doesn't call them public.

To make them public:

1. Open each trace in neatlogs and click Share.
2. Turn on public or anyone-with-the-link access, then copy the link it gives you.
3. Send me those links, or just tell me "public" if the URLs stay the same.

Once a logged-out browser can open them, I'll flip one setting and the page will say "Public neatlogs links".

Everything is ready for submission except the video and cfo.ai. The repo is cleaned up and pushed, the live site is deployed and working, and nothing secret is anywhere in git.

What I removed from the repo: 8 files that weren't part of the submission. I added them to .gitignore so they can't be committed again, and your local copies are untouched:

• PROGRESS.md, which held stale notes for resuming a session.
• docs/posts.md, the post drafts.
• Six one-off dev scripts that nothing referenced: smoke_llm, run_bg, show_queue, inspect_db, reset_triage and nl_label_demo.
• Your local agent files (AGENTS.md, .factory/mcp.json, .factory/skills/, data/night_watch.md) were never committed and are now ignored too.

What remains is the app, its tests, deploy and sandbox setup, the docs and screenshots the README uses, the learned playbook, hackathon.md and HUMAN_TASKS.md.

What I updated in hackathon.md:

• Removed "(working name)".
• Filled in Team as "Aditya K (@isHeSatoshi), solo". Change it if that's not how you want it to read.
• Corrected the stack to the current models.
• Updated the spend figure and added the ICML numbers, the recent Entire checkpoints, the live links and your X post.

Checks:

• Secrets: I scanned every tracked file, all 20 Entire checkpoints and the full git history for keys, tokens and invite codes. All scans found 0 matches.
• Tests: All offline tests pass.
• Live site: It is running commit 199f95e. /, /judges, /icml, /board, /healthz and /judges.json all return 200, and the health check is OK.
• Evidence rows on /judges: 6 of 7 pass. The one marked partial is "You used cfo.ai", and it will pass once there's a plan link.

Your last steps:

1. cfo.ai: send me the plan link and I'll add it to /judges, which turns that row to pass.
2. Video: send me the link and I'll put it on /judges and in hackathon.md.
3. Submit before 11:55 PM IST: quote the official neatlogs post on X with:
   • Team: Aditya K (@isHeSatoshi)
   • Project: PaperCI
   • https://github.com/isHeSatoshi/paperci
   • your demo video link
   • https://paperci.vercel.app/judges

The README already says the cfo.ai plan link is on /judges. If you end up skipping cfo.ai, tell me and I'll reword that line so it stays accurate.

Nothing is broken, and nothing needed fixing this round.

• Health: The service, watchdog, disk (35 GB free) and Docker are all fine, and the live site is on the cleanup commit 199f95e. The 18:00 deploy restart left one verification sandbox running, and the automatic cleanup removed it within 2 minutes.
• ICML eval: Pass 2 is finished and nothing is queued, so the tally is unchanged.
• Evolution:
  • Two rounds finished and adopted nothing. In one, a candidate playbook was caught by the no_repo_code_path integrity check. In the other, all three candidates scored below the current version.
  • Round 1734 is still evaluating.
  • The three earlier adoptions are still under canary watch (gen 5 is live), and nothing is waiting for your approval.
• Failures: Everything in the last 2 hours came from the two benchmark papers the evolution rounds use, and it's the agent's own behaviour (missing data, running out of turns). There's no sandbox or allowlist gap to fix.
• Spend: $0.65 in the last 2 hours, $9.25 in the last 24 hours, and $17.59 all time.

I won't deploy anything during your recording and submission window unless something breaks. The summary is added to data/night_watch.md.

The installer failed on Windows because the skills CLI isn't found. The workflow skill is already installed in this environment, though, so I'll load it directly.

I have the positions. Copying the screens into the project and looking at the main ones to plan cursor moves and zooms.

The demo video is done. Upload this file to X: D:\Project\paperci-video\videos\paperci-promo\renders\paperci-demo-x.mp4. It runs 2:19.5 and is 22.7 MB, at 1920×1080, 30 fps, with H.264 video and AAC audio.

• Why there are two files: the full render, renders/video.mp4 (2:20.1, 38.7 MB), is 0.1 s over X's 2:20 limit for accounts without Premium. The upload copy trims 0.6 s from the end card and fades the audio out over the last second. Nothing narrated is lost.
• What I checked: npx hyperframes check passed with 0 errors and 0 warnings. I reviewed frames from every scene after the layout fixes; the stamp, the X badge, the chips, the highlights, the self-healing loop and the end card are all placed correctly. I also confirmed the final file's resolution, frame rate, duration and audio track. I haven't watched the whole thing at full speed, so play it once before posting.
• Other narrator voice: Pip currently uses the Sarvam voice amelia_en_conversational. If you'd rather hear sophia, that means regenerating the narration and re-rendering, about 5 minutes. Tell me if you want it.

Once the video and the cfo.ai post are live, send me both links. I'll add them to docs/judges-links.json as demo_url and cfo_url, run the secret scan, push and deploy. Your final X submission is due before 11:55 PM IST.

Both changes are done, and the new video is ready to upload: D:\Project\paperci-video\videos\paperci-promo\renders\paperci-demo-x.mp4. It runs 2:19.3 (under X's 2:20 limit) and is 40.6 MB, at 1920×1080 with H.264 video and AAC audio. I deleted the old 2:20.1 render so you can't upload the wrong file by mistake.

Video changes

• Entire: the self-healing scene now ends with Pip saying: "GitHub keeps the code. Entire keeps the agent session behind every fix: why it changed, and where to rewind." A card on screen shows the same comparison: a dimmed "GitHub · keeps the code" row, then a highlighted Entire row and the real checkpoint 01M4QVCY... on commit 42beb3a.
• Ending: it now says "claims made" (struck through), then "claims verified", then "All in a sandbox." with a lock icon. Pip says: "Because claims made should mean claims verified. All in a sandbox."
• Staying under 2:20: the new line added about 3 s. To make room, I tightened the simulations line and the cfo.ai line, read the self-healing scene slightly faster, and shortened the pause on the end card.
• Small fix: the speech bubble used to sit empty for about a second between scenes. Captions now stay up until the next one starts.

npx hyperframes check passed, and I reviewed frames of the changed scenes. I haven't watched the full video, so play it once before posting, mainly to hear how the new Entire line sounds.

Dashboard change (live on paperci.vercel.app)

• The field label now reads "Code repository (optional)".
• Inside the box, the hint scrolls slowly so it is no longer cut off: "Found automatically if your paper links its code · or paste https://github.com/you/your-paper-code". It disappears when you click into the box or type. For visitors whose system is set to reduce motion, it stays still and is shortened with "…".
• I previewed the form in a browser and the offline tests pass. The change is committed and pushed (3bb7b57) and deployed to the server, and the live page shows it.

Once the video and the cfo.ai post are up, send me both links. I'll add them to docs/judges-links.json and deploy.

I stopped the night-watch loop. All three Entire bonus points now have live evidence, and the /judges page links to every piece.

Copy-paste answer for the bonus section:

Checkpoints (progress and key decisions). 45 of PaperCI's 60 commits carry an Entire checkpoint. Each one links the commit to the agent session that wrote it: the requests, the decisions and the code changes. The whole build is one continuous session, from the mission prompt to the final submission fixes. Secret redaction is checked before every push.

• Latest checkpoint: https://entire.io/gh/isHeSatoshi/paperci/commit/d45b31299303d76301837dcecf0eed7fd5082514 (ID 01M4R9H0R27FDDHVV8HW2GSVYJ)
• Example fix found by the self-healing loop: https://entire.io/gh/isHeSatoshi/paperci/commit/42beb3a1e5c9aad2a8290ee122f88e3c8cd22534 (ID 01M4QVCYNAD0ST9WYSM1QMPG3V)
• Full session: https://entire.io/gh/isHeSatoshi/paperci/session/bf4a7d1a-7a3a-45c9-b8bd-1b2c021e7bc7

entire-graph (finding code and assessing risky changes). We ran entire graph diff on all 44 checkpointed commits it covers. Each commit is labelled by risk to PaperCI's self-healing loop:

• 17 touched the frozen referee (the verifier, sealed claims and sandbox allowlist that the evolution loop scores against).
• 4 changed an interface that many other parts of the code use.
• 23 were contained.

The repo's agents are also set up to search with the graph before falling back to plain text search.

• Report: https://github.com/isHeSatoshi/paperci/blob/main/docs/entire-graph.md
• Agent guide: https://github.com/isHeSatoshi/paperci/blob/main/.entire/graph-agent.md

Open-source build on Entire's CLI and Graph. paperci/entire_guard.py is a referee guard for a self-healing agent. Its gate command runs entire-graph on a change before deploy and refuses it if it touches the referee, so the scoring rules never change during an evolution round. Its report command links every commit to its checkpoint. It has an offline test.

• Code: https://github.com/isHeSatoshi/paperci/blob/main/paperci/entire_guard.py
• Live evidence: https://paperci.vercel.app/judges (the "You used Entire" row)

Checks:

• All offline tests pass, including the new guard test.
• The new checkpoint data had 0 hits on the secret scan.
• The commit (d45b312) is pushed and deployed. The live /judges page shows the new row, and entire.io shows the new checkpoint.

Limits to know:

• The gate runs on your machine before a deploy, not on the server, because the server doesn't have the Entire CLI. The report also leaves out the commit that added it, which is why it covers 44 commits, not 45.
• The commit hook added a checkpoint ID but never saved the session data. So I edited that one commit before pushing it, removed the empty ID, and attached the session again.
• The demo video and cfo.ai links are still empty in docs/judges-links.json. Send them over and I'll add them.

Below are the three answers, ready to paste. I checked every link and each one opens. The neatlogs trace links need a neatlogs login, which is why the neatlogs answer points to /judges first: it shows a trace without a login.

cfo.ai

You still need to fill in your cfo.ai plan link here. Send it to me as well and I'll put it on /judges.

Entire

neatlogs

My trace links might not match the before/after pair you meant. /judges names that pair (traces 553880c6… and 80481abc…), and docs/evidence.md lists all the trace IDs. If you want those exact traces linked, I'll swap them in. As a fallback, every number above can be checked on /judges.

Anyone can now run PaperCI on their own machine with their own LLM, including a local LiteLLM proxy. It's live on the dashboard at https://paperci.vercel.app/local and in the README. It is the same agent as the cloud server: same sandbox, allowlist, sealed claims, verifier and dashboard.

How a user runs it

The script sets up Python, copies a ready-made .env, builds the sandbox images, checks everything, and opens the dashboard at http://127.0.0.1:8000. Users need Docker and Python 3.11+.

Choosing the LLM

• In .env, any role can use a model written as local:<model>. It works with Ollama, LM Studio, vLLM, llama.cpp, or a local LiteLLM proxy.
• Local LiteLLM, kept simple: deploy/local/litellm.yaml defines two names, a builder and a verifier, each from a different model family. The user starts it with litellm --config deploy/local/litellm.yaml --port 4000 and points .env at port 4000. Each name can point at Ollama, vLLM, or a paid API using their own key.
• If the proxy goes to a paid API, two price settings in .env keep the per-paper dollar limits working. Truly local models cost nothing, and turn and time limits cap each run instead.
• If a model doesn't support tool calls, PaperCI switches to its own JSON format for them automatically.
• Hosted keys (LITELLM_API_KEY or FACTORY_API_KEY) still work.

The preflight check (python -m paperci.localrun check) tests the setup for real before anyone submits a paper:

• Docker is running and both sandbox images exist.
• The LLM server answers and serves every configured model.
• Each model makes a real tool call.
• Each model can take a 16,000-token prompt.
• It warns if the builder and verifier are the same model.

Where people see it

• A "Run locally" link in the site's top menu, and a "Server busy?" line under the submit form.
• When someone's run is waiting behind others: "N run(s) ahead of you · skip the queue: run it on your machine".
• The README links to it at the top and has a "Run it yourself" section. The full guide is docs/run-locally.md.

What I tested

• All 24 offline tests pass, including a new one for the local model connection.
• I ran a fresh copy of the repo with no cloud keys, using LM Studio and a small 1.5B model, and submitted a real arXiv paper. The tool-call check passed for real.
• A problem the test found: the 1.5B model fell over on the first run. LM Studio's default 8,000-token limit rejected PaperCI's 9,500-token claims prompt, and Ollama's default would silently cut it short instead. I added the long-prompt check above, and the docs say to load models with a 32,000-token limit. After reloading at 32,000 the check passed and the run went through to the claims step.
• The 1.5B model found no claim it could back with an exact quote from the paper, so the run stopped before the sandbox build. That is the expected safe behavior, and the docs recommend models of 7B or larger. I have not run a full sandbox build with a local model. The sandbox code is the same as on the cloud server; only the LLM connection is new.
• The referee guard rated the change "contained", so it doesn't interfere with the self-improvement rounds.
• The secret scan found nothing. Commit 7b07ada is pushed and deployed, and the live site is healthy.

One server change: your public IP changed (.119 to .129), so SSH to the server was blocked. I updated the server's firewall rule to allow SSH only from the new single IP and removed the old one.

I removed 9 files from GitHub. They're still on your machine and now listed in .gitignore, so they can't be published by accident again. The change is pushed and deployed, and the site is healthy.

Removed from GitHub:

• HUMAN_TASKS.md: your personal to-do list.
• docs/demo-script.md: the video script and shot list.
• docs/neatlogs-feedback.md: the feedback notes for the neatlogs team.
• docs/buyer-map.md and scripts/crustdata_buyers.py: the Crustdata list of target companies, plus the script that built it.
• scripts/pull_cloud.ps1: the server backup script, which had the server IP and your SSH key path in it.
• .factory/settings.json: your local Droid/Entire hooks with Windows paths.
• docs/review.html: the old mock "review my paper" page, which the real site replaced.
• docs/dashboard_public.html: an old generated page that was out of date.

I also fixed the places that pointed to these files: hackathon.md, docs/unit-economics.md, and scripts/economics.py. In docs/cloud.md I replaced your personal SSH commands with the server-side deploy commands.

Kept public, because something needs it:

• The code, the tests, and requirements.txt.
• deploy/: the server pulls its configuration from git, so deploys would break without it.
• The evidence files that /judges reads (docs/judges-links.json, docs/neatlogs-loop.json, docs/entire-graph.*, docs/evidence.md).
• The docs the README links to.
• docs/reviews/: these are served as "Earlier runs" on the site.
• hackathon.md: the build log the hackathon asked for.
• CLAUDE.md and .entire/graph-agent.md: Entire's setup asks you to commit these, and they back your Entire bonus answer.

I checked that nothing on the site or server uses the removed files. All 24 offline tests pass, and the secret scan found nothing.

Two things to know:

• Old commits still contain these files. Anyone browsing older commits on GitHub can still open them. The only way to erase them is to rewrite the git history and force-push. That would change every commit ID, which would break the Entire checkpoint links in your submission and on /judges. I'd wait until judging is over. None of these files contain secrets; the worst is the server IP, which is public anyway.
• The Entire checkpoints are public and contain our whole chat, including your messages. API keys were automatically replaced with "REDACTED". If you'd rather keep the chat private, you can change who can see the repo in Entire's settings after judging.