Most "AI agents" are a chat log with a goal and a prayer. I write about the other kind.
// from apps/zoltar-be/src/session/session.tools.ts
// What Claude returns each turn.
const submitGmResponseSchema = z.object({
playerText: z.string(), // narration shown to the player
stateChanges: z.object({ /* deltas */ }).optional(),
diceRequests: z.array(z.object({ /* notation, purpose */ })).optional(),
gmUpdates: z.object({ /* npc state, notes, canon */ }).optional(),
adventureMode: z.enum(['freeform', 'initiative']).nullable().optional(),
});
An AI game master where Claude runs tabletop RPG sessions without fabricating dice rolls or losing the plot.
The model can ask for a roll. It cannot fake one. The dice happen at the table or in the backend, never in the prompt.
Game state lives server-side and is never in the model's context. Claude proposes changes via tool calls; the validator decides what's real.
Rulebooks are ingested as hash-pinned chunks in pgvector. The model retrieves passages by query; nothing about the rules lives in the prompt.
Every turn produces a structured delta that goes through one validator pass. If it's invalid, Claude gets one re-prompt to fix it — no further retries.
After chucking the first version of the eval harness for Zoltar, I started over with a minimal version that I could easily understand. I've learned that an LLM judge can tell me if a pass rate improved, but it can't yet tell me if the pass rate is good enough.
I didn't realize I was vibe-coding my project until I tried to write an article about its testing infrastructure and couldn't. The solution turns out to be writing documentation by and for humans.
Claude Code logs every session locally in an undocumented JSONL format. I wrote cctx to turn those logs into markdown transcripts you can commit and review alongside the code, so the reasoning behind a decision is preserved and not just its outcome.
You can't stop an AI feature from hallucinating, but you can keep nonconforming data out of production documents by storing state in an authoritative database, structuring updates with tool-calls, and validating data before it reaches the database.
Building a reliable system around an LLM means dealing with its weird failure modes. The conventional wisdom on retries doesn't fit — a second failure isn't a model problem. It's a signal that the validator, prompt, or schema needs work.