Panos development log

Receipts for your AI workflows

Ordinary engineering visibility for agent sessions: skills, failures and timings.

Published · 613 words · View as Markdown

Contents
  1. Adding some determinism
  2. Harness logs
  3. Ichnos 🔍
  4. More thoughts

2021, 2022, 2023, 2024, 2025, now it’s that time of the year for a new blog post, and since we are in an AI psychosis period, it’s no surprise that I’ll be dumping my thoughts related to it.

“Software engineering is changing”, “Humans are the bottleneck”, “Software engineering is dead”………. this post is not about that.

What I’ve been thinking about, since the start of all the AI hype, is how we can apply common software engineering practices, and common sense(that took a big hit in the AI era😅), to our AI workflows and systems. One thing everyone who isn’t a vibe coder agrees on, it’s that visibility and measurements on how our AI systems actually work is a must.

Adding some determinism#

Take fixup commits. When your agents tries to make fixup commits on some code you can leave it to chance and do it’s own, or you can create a skill goes beyond the “Create fixup commits” that invokes a script that tries to add some determinism to that operation.

In my case the flow goes something like this:

  1. Check the repo’s state
  2. Find eligible commits
  3. Snapshot the diff
  4. Draft a plan and flag conflicts

then the AI comes in:

  1. Reviews the plan
  2. Re-targets or settles what blame couldn’t decide

after that the code

  1. Validate the plan and refuse it if anything would conflict
  2. Creates the fixup commits

The above skill involves running actual code. However, at some point I noticed that one of the scripts was failing. Since LLMs are goal oriented, instead of stopping, the agent was identifying the failing part and running some form of it by itself.

The fixups were made, but the part “I wrote” to be deterministic wasn’t anymore, and nothing told me.

So… some visibility would be nice there.

Harness logs#

One tool software engineers were using long before LLMs is… logging.

If you’re working with a local harness, like Claude Code, you have access to some kind of logs, claude codes stores its session logs under

~/.claude/projects/-some-project/some_session_id.jsonl

You can point another Claude Code session at that file and ask questions about your session, or /fork a session and ask questions directly there after you finish, to deepen your understanding of what the agent did.

Ichnos 🔍#

Ichnos session view
One session in Ichnos

We are living in the age of personal software. Everyone has their own slop-written code, tailor prompted for their own needs.

Mine is called Ichnos. It hooks into Claude Code’s callbacks and writes Claude’s events to a local SQLite database, one per session. That gives you an overview of:

  • which skill was triggered
  • which command failed
  • time logs, to identify slow operations

then with it’s accompanied skill the agent can query /ichnos sessions-id the logs

then we can go back to the AI tools and do adjustments and optimizations.

More thoughts#

If you are in the camp of “LLM all the things” and token burning then you have no use for the above, “it will be next models problem” to do better.

If understand that confidence is not the same as competence then you want receipts, logs and numers of what the agent did and did not do, not just a summary. There is saying “You can’t improve what you don’t measure” Once you can see what’s happening, then you can start measuring(more on that later) whether your changes made it better or not.

We live in an era were we play with words in .md files, we prompt LLMs to modify already prompted instructions with the hopes that “make it smaller, faster, better, no mistakes” will actually work, without concreate evidence we at best gambling.