Build Trueup AI Tracing Verification Framework

You are building Trueup, an entry for neatHack (a 48-hour hackathon run by neatlogs, 10-12 Oct 2026). This session is recorded by Entire and published with the repo, so work only inside this repository, and never mention anything outside it. Do not run git commit or git push; the owner commits.
What Trueup is (README first line, word for word): "Trueup finds where your AI agent's traces disagree with what really happened, fixes the tracing code, and proves the fix."
Plain idea: teams watch their AI agents through tracing libraries (neatlogs, OpenInference, OpenLLMetry). These wrap the AI provider's SDK and record each call. When the wrapper gets a case wrong, the dashboard shows a crashed run as OK, misses a refusal, or shows double the real cost. Trueup runs the agent against a fake provider (truth), captures what the trace says, compares, and when they disagree finds the cause in the tracing library, writes a fix with a test, and proves it. Workers: planner, runner, comparer, checker, deduper, fixer, verifier, reporter. The language model proposes; code decides.
Your task now is only the skeleton:
- A uv project, Python 3.12, src layout, package
trueup, console scripttrueup(argparse is fine). Dependencies kept minimal: httpx; dev: pytest. Runuv syncanduv run pytest -q; uv is at ~/.local/bin/uv. - Pure core in
src/trueup/core/with no I/O: the data model as dataclasses (Finding, Run, Mismatch, Severity enum HIGH/MEDIUM/LOW, Observation holding truth and trace) with to_dict/from_dict round trip. Effects live in other modules later. - CLI subcommands audit, sweep, fix, upstream, verify, report, doctor. Only
doctorworks now: it reports Python version, whether NEATLOGS_API_KEY and TRUEUP_MODEL_KEY are set (never print values), and exits 0 or 2. The rest exit 2 with "not built yet". Exit codes: 0 no mismatches, 1 mismatches found, 2 could not run. - Tests for the data model round trip, severity ordering, and doctor (including that it never prints a key value).
- Files: LICENSE (MIT, copyright 2026 Jennycruzy), README.md (the first line above, then a short plain-language "what it does", "status: early build", and how to run tests), CHANGELOG.md, CONTRIBUTING.md, SECURITY.md, THIRD_PARTY_NOTICES.md (list httpx, pytest), .gitignore (include .env, .venv, .trueup/, .mcp.json), hackathon.md (a dated build log; first entry: 2026-10-10, skeleton), and .github/workflows/ci.yml running uv sync and pytest on push.
- Writing rules: plain words, no marketing filler ("seamless", "leverage", "robust" and similar are banned), no claims of results that do not exist yet.
Finish by running the tests and printing a short summary of the files you made and the test count.
The skeleton is built and all 14 tests pass (uv run pytest -q). I didn't commit or push anything.
Files made:
- Project setup:
pyproject.toml(Python 3.12, src layout, httpx as the only dependency, pytest for dev,trueupconsole script) and.python-version.uv syncalso createduv.lock. - Data model:
src/trueup/core/model.pyholdsSeverity(sorts LOW < MEDIUM < HIGH),Observation,Mismatch,RunandFinding, each withto_dict/from_dict. It does no file, network or environment access. - CLI:
src/trueup/cli.pyand__main__.py.doctorprints the Python version and whetherNEATLOGS_API_KEYandTRUEUP_MODEL_KEYare "set" or "not set". It never prints the values.audit,sweep,fix,upstream,verifyandreportprint "not built yet" and exit 2.
- Tests:
tests/test_model.py(6 tests): round trips, including through JSON, and severity order.tests/test_cli.py(8 tests): doctor with keys set and unset, including that the secret values never appear in any output, plus one test per unbuilt command.
- Docs and repo files:
README.md(starts with your line word for word),LICENSE(MIT, 2026 Jennycruzy),CHANGELOG.md,CONTRIBUTING.md,SECURITY.md,THIRD_PARTY_NOTICES.md(httpx, pytest),.gitignore,hackathon.md(2026-10-10 skeleton entry) and.github/workflows/ci.yml(runsuv syncand pytest on push).
I also ran the CLI by hand: uv run trueup doctor exits 0, and uv run trueup audit exits 2 with "not built yet".
doctor only exits 2 when Python is older than 3.12; missing keys are reported but don't make it fail. Exit code 1 (mismatches found) is defined but nothing uses it yet.
Two connectors in this environment, claude.ai Paybox and claude.ai arvix mcp, need authorization in your claude.ai connector settings. They weren't needed for this task.