Role-playing tool calls

Nova Fund's decision engine landed this week. DRIP shipped last post; phases 1, 2, and 3 of the broader portfolio-product arc shipped on top of it. The fund now has a ledger, state, validators, a sector tagging utility, analytics, memos with a draft → voiced → published lifecycle, a weekly proposal builder that calls the screener and ranks candidates against a constitution, click-to-approve trade commit, a sector-aware allocator that diversifies instead of stacking the top-scoring tech names, and a single weekly_run command that wraps the whole Sunday-night routine. Twenty-one chat-callable skill tools.

This post isn't really about any of that. It's about the smoke test that almost ate four hours of my evening and the one-line diagnostic that didn't.

The smoke test

The recommended verification path for Nova Fund is "fire up Nova, drive the skill from chat." There's no standalone TS runner in the repo, no tsx harness, no node REPL convenience. The shape of the smoke is:

1. Init the fund with today's date.
2. Deposit $1,000.
3. Read state. Should show cash = 1000.
4. Buy 1 SPY at $580. ETF, sector etf-broad.
5. Read state. Should show cash = 420, position = 1 SPY @ $580 cost basis.
6. Read ledger. Should show three entries: init, deposit, buy.

Luke ran it. Nova confirmed everything in chat:

"Initialized fund. $1,000 deposited — cash balance now $1,000. Bought 1 share of SPY at $580. Position open, cost basis $580. NAV: $1,000."

Then he ran state again to double-check. Cash = 0. Positions empty. NAV = 0.

Then ledger. One entry: the init line. No deposit. No buy.

The bug-hunt that almost happened

This is the kind of thing where the first instinct is a write bug. The lock isn't releasing. appendLedger is throwing into a swallowed catch. applyDeposit is mutating an in-memory copy and the persist path is broken. fs.writeFileSync is hitting an EBUSY because OneDrive is syncing the vault directory underneath us.

I had state.ts and ledger.ts open and was reading through the lock + persist sequence before the smarter instinct showed up.

The smarter instinct

Before reading the implementation: verify the tool was actually called.

One line at the top of the skill's invoke handler:

console.log(`[nova-fund] invoke ${tool} ${JSON.stringify(args)}`);

Restart Nova. Re-run the smoke. Watch dev.log.

The init call fired the log: [nova-fund] invoke init {"inceptionDate":"2026-05-05"}. Then the chat said "Depositing $1,000…" and there was nothing in the log. No [nova-fund] invoke deposit ... line. The chat continued to "Bought 1 SPY at $580…" — still no log line. Then "$1,000 NAV, SPY position confirmed, ledger has 3 entries."

The skill had been called exactly once. The LLM had narrated the rest of the conversation as if the tools had fired. Confidently. With "✓" marks.

What was actually happening

The model had learned, somewhere in its training, that confirming the result of a tool call is a reasonable response shape when a tool call is in flight. Apparently it occasionally interprets "a tool call should happen here" as "narrate what the tool call would have returned." The actual tool-use block never makes it into the output. The model produces text that looks like a tool result, the user reads it as one, and the side effect that should have happened never did.

I don't know exactly what triggers this. The first call (init) fired correctly, so it's not a structural problem with the skill registration or the tool catalog. Something about the conversational rhythm after the init — maybe "deposit" reads as more conversational than "init" and the model picked the wrong response shape. Maybe a context-cache quirk. Hard to characterize.

The practical answer is more useful than the theoretical one: don't trust the LLM's chat narrative about tool calls. Trust the side effect. The chat is a description of what the model thinks happened. The persisted artifact is the only authority for what did.

The pattern, banked

When on-disk artifacts disagree with the LLM's chat narrative, first verify the tool was actually called. A single console.log at the top of the skill's invoke handler is the right diagnostic. It's authoritative — it only fires when execute() runs. It tells you the args the model passed (so you can also catch schema-mismatch issues, which were my second guess after this episode). And the cost is measurable in microseconds; you can leave it in production indefinitely.

Cheaper than reading the implementation looking for a bug that's not there. Cheaper than instrumenting the lock. Cheaper than walking through OneDrive's sync behavior.

The diagnostic stays in. dev.log now shows a real trace of every Nova Fund tool invocation. The LLM's chat narrative is still useful for human readability, but it's no longer the source of truth for "did the deposit happen." The log line is.

This is also a generalization of a much older lesson about model output. A model is a function from prompts to outputs that's usually right about the world. It is, by construction, unable to be authoritative about whether code ran. Anything load-bearing about side effects has to be checked at the side-effect site, not at the model's description of the side effect.

What's next

Phase 4 of the portfolio product also shipped this session — Clerk + Neon + Stripe wired together end-to-end into a Next.js app at apps/web. Subscription flow proven in dev: sign up, hit Checkout with the Stripe test card, watch the user's tier column flip from free to pro, watch the badge in the header update on refresh. That's a different post; this one's already long enough.

Inception is May 10. Phases 1, 2, 3, and 4 are done. The simulator is real, the decision engine is real, the click-to-approve flow is real, the web app is real. Sunday morning, Nova Fund will hold its first weekly_run, propose its first trades, voice its first memo, and start a public track record from zero. The diagnostic log will be watching.