Own your intelligence: an honest scorecard for my AI agent stack

Owning your intelligence does not mean building every layer of your AI stack from scratch. It means controlling the specific parts that decide how that intelligence behaves, what it learns from, and whether it gets better with time. Access to a smart model is not the moat. The moat is what you do with the layers around it.

That is the case Harrison Chase, CEO of LangChain, made in a long post this past July: in the next few years, a company either runs its critical work on AI or ships AI as part of what it sells, and either way generic intelligence stops being enough. His analogy is a supply chain. A retailer does not build its own trucks, but it controls what gets ordered, how inventory moves, and how demand gets forecast. Nobody expects a grocery chain to manufacture axles, and everybody expects it to know what sells. His post closes with ten yes or no questions meant to test whether you actually own your intelligence or just rent access to somebody else's. What follows is the framework in three pieces, then all ten questions, scored against my own setup without flattering it.

• The three-layer test. Model, harness, and context are the parts of the stack Chase says you actually have to control, not the whole stack end to end.
• The second axis. Owning the cost, the quality bar, and where the system needs a human matters as much as owning the layers themselves.
• The compounding test. Whether session one hundred starts smarter than session one is the piece most setups fail without ever noticing.
• Ten questions, scored straight. Chase's checklist runs to ten yes or no questions, checked directly against the source, and more of them land on no or partial than I expected going in.
• Seven honest gaps. Seven of the ten land on a no or a partial, and most of them trace back to the same missing piece.

What owning your intelligence actually means

Three layers of the system itself

Chase's first axis is control over the agent system, and he splits it into three layers that fail in different ways.

The model layer is the one everybody thinks about first: which provider, which checkpoint. Owning this layer does not mean training your own model. In practice, it means the model call sits behind an interface you control, so replacing one provider's model with a better one is a configuration change, not a rewrite of how the task works.

The harness layer is the orchestration code: routing, tool definitions, the steps a task goes through before it counts as done. This is where most of the company-specific behavior actually lives, and it is the layer people give up first, because a hosted product will happily do the orchestrating for you. In practice, owning it means the logic that decides what your agents do next lives in a repo you control, not inside a vendor's product you can query but never edit.

The context layer is documents, policies, tools, and memory: everything the model sees beyond the question typed into it that minute. Chase's point about this layer is blunt: without owning it, you are not accumulating anything, because every session reasons from scratch. In practice, owning context means the material your agents draw on is stored and versioned somewhere you control, so what your system "knows" about your own operation does not evaporate when a session ends or a vendor changes how it retrieves things.

The economics, the quality bar, and the risk boundary

The second axis is less about architecture and more about discipline: do you own the cost, the quality bar, and the line between what the system can do alone and what needs a person.

Cost has to be tracked at the level of a user, an org, or an agent, not discovered once a month in an invoice. Quality has to be measured every time a prompt, a tool, or a model changes, not assumed because the last few outputs looked fine. And the boundary between autonomous action and supervised action has to be written down in advance, not improvised in the moment something goes wrong. That third piece is close to the whole subject of my production-readiness checklist: twelve gates, each one a place where an agent either has a pass-or-stop test on file or does not, because a rule that lives only in an operator's discipline is a rule that survives exactly until that operator has a bad week. Underneath cost, quality, and boundaries sits a fourth piece, observability: whether anyone, human or automated, can reconstruct why the system did what it did after the fact. That one is not a nice-to-have sitting next to the other three. It is what makes them checkable instead of assumed.

Whether the intelligence compounds

The third piece of the framework is the one most setups quietly fail without noticing: does the system get better the more it gets used. Chase's bar for this is specific. Your hundredth interaction with the system should be more valuable than your first, and it should get there through an ordinary loop: traces of what happened, feedback on what worked, evals that stop a change from undoing a fix you already made. The sharper requirement, the one that separates a real answer from a hopeful one, is that this loop has to survive a model swap. If switching providers wipes out everything the system learned, you were never really compounding anything. You were running the same session over and over under a different name.

Ten questions, scored honestly

The checklist that closes Chase's post is ten yes or no questions. Here is where my own setup lands on each one, in order.

Can you swap your model for whatever ships next?

Yes. Nothing in my day-to-day setup is wired to one provider by name. In practice, the model call sits behind a boundary, so moving a task to a better model is a configuration change, not a rewrite of the logic around it. That matters more than it sounds like it should: the moment swapping providers means touching core code, you have quietly agreed to whatever that provider does to its pricing or its roadmap next, without ever holding a meeting about it.

If your provider pulled the model tomorrow, could you self-host it?

No. My setup assumes a hosted API is always reachable from one provider or another, not that I could stand up and run a model myself if every provider I use stopped offering it. That is a real dependency, and naming it plainly is the point of running this checklist.

Do you actually know what reaches the model at every step?

Partial, and I want to be honest about the shape of the gap rather than round it up to a yes. For any single run the trace answers it exactly, after the fact. What I do not have is a standing, always-current answer to that question across every agent at once, before anything runs. Nothing about that gap is dramatic, which is exactly why it is easy to leave alone.

Can you run your agent identically across clouds or on a local machine?

No, and I have not actually tested this, which is its own answer. Everything I run assumes the same handful of hosted services are reachable. I have never tried moving a task to a different cloud or to a local model and checking whether it behaves the same way, so "identical" is not something I can currently claim even as a maybe.

Can you monitor and control AI spend on a per user basis?

Partial. Budget alerts and a cost estimate before anything ships are already part of how I work, the same discipline behind my production-readiness checklist, where anything that spends wants both an alarm and a cap. What exists today covers the alarm. The view is per project, not per user, because there is normally one person running the queries, and there is no hard cap stopping a run once it starts, by choice rather than by oversight.

Can you produce the full trace behind a decision?

Yes. My monitoring field notes treat a full trace as the non-negotiable layer underneath everything else: every run gets one, not just the ones that look interesting in hindsight. Dashboards, alerts, and status lamps are derived from that trace, not a substitute for it. If a decision needs explaining three days later, the trace is already there, because it was never optional in the first place.

Do you have evals that catch a regression before it ships?

Yes. My notes on LLM evaluation lay out when the suite runs: a change to a prompt, the model, the retrieval logic, a tool, or the output parser triggers it before that change ships. Sampled audits and traces sit on top of that as a second layer, not a replacement for the suite itself. The test cases do not move just because the code around them did.

Is your hundredth run better than your first?

Partial. Auto memory carries a session's corrections into the next one, and a mistake clear enough to name gets written into CLAUDE.md so it does not repeat, so the lesson does move forward. What is missing is not that channel, it is a measurement: nothing checks today whether the hundredth session actually performs better than the first, or by how much. Whether the fleet of sessions is getting smarter over time is a question nothing currently answers, which is a different and quieter problem than any single session being wrong. A system that does not measure compounding cannot tell the difference between compounding and coincidence.

Is what you learn portable to a different system?

Partial. Portability asks whether a lesson learned in one setup would survive a move to a different model or a different harness entirely. The text of a lesson already travels: my rules are plain markdown, and the other coding agents I run, on other models, load an instruction file that sends them to the same rules and a copy of the memory index at the start of every session. The machinery does not travel: auto memory is a Claude Code feature, and the hook that blocks production changes there becomes a plain instruction in the other runtime. Nothing checks today whether a ported lesson works the same way on the other side.

Do you control how your agent actually learns?

Partial. Controlling how an agent learns means having a defined process that turns one session's outcome into an input for the next. Part of that is mine: the rules in CLAUDE.md are ones I choose to add, like the one about re-reading a file before editing it. Auto memory is different: Claude decides on its own what it keeps there. The loop that runs by design on every reviewed failure is monitoring, where the failure turns into a permanent check the next run inherits. Ordinary sessions have no standing process like that, only the rules I add when a mistake is clear enough to name.

The verdict

Seven of the ten answers above are not a clean yes. A model I could not host myself, an environment I have never tested outside a handful of hosted services, a context picture I only have one task at a time, a cost view that stops at the project level instead of the person, and lessons that move forward, some by my choice and some even into another agent, with nothing that checks today whether they helped: strip away the specifics, and most of these share one root. Pieces of the loop Chase describes are there, but nothing checks today whether the pieces are actually working.

The hundredth-run question, the portability question and the question about controlling how the agent learns point at the same root. Lessons get carried forward, some by my hand and some by auto memory, and some already cross to another agent, but nothing afterward checks whether the hundredth session actually benefited from them. Fix the measurement, and the portability answer gets its missing half too: the same check would show whether a ported lesson still works on the other side.

So that is where I would start, and it is a smaller fix than seven open gaps make it sound. Just a standing way to check, after the fact, whether the lesson a session carried forward actually made the next one better. Traces already tell me what happened, on every run. The eval suite already tells me whether a change was safe before it ships. Neither one currently tells me whether the system as a whole is getting better with use, and that is the gap worth closing first.

Frequently asked questions

What is LLM evaluation, and which metrics matter?

LLM evaluation is testing a model or agent's output against representative cases before and after a change, using deterministic checks, human review, and a calibrated model judge where rules cannot capture the answer. Which metric matters depends on the task: exact accuracy for factual work, pass or fail against written criteria for agent behavior, and consistency across repeated runs for anything with real variance. My notes on LLM evaluation go through the stack in more detail.

What is an AI agent framework?

An AI agent framework is a library your code calls to build an agent: it supplies the pieces for planning, tool calls and state across steps, so that plumbing does not get rebuilt for every new task. Frameworks range from thin libraries that just wire up tool calls to fuller systems that also manage memory, retries, and coordination between multiple agents. A framework can be part of a harness, but it is not the whole of it.

Build vs buy: when should you build your own AI system?

Build when the harness or the context layer is where your actual advantage lives, because that is the part a vendor's default assumptions will not fit. Buy or use an existing framework for the genuinely generic pieces, like basic tool-calling plumbing, and spend engineering time on the parts that make the system different from a template.

What is an AI agent harness?

An agent harness is the orchestration logic wrapped around a model: how it picks the next tool, how it tracks progress toward a goal, and what makes it stop. It is distinct from the model and from the raw prompt, and it is usually where the most business-specific behavior in an AI system actually lives. My glossary entry on agent harnesses covers the term on its own.

What is AI vendor lock-in?

AI vendor lock-in is what happens when a system is built in a way that makes switching providers, models, or platforms costly or risky, even once a better option exists. The usual cause is baking provider-specific assumptions into the harness or the data format, instead of keeping a clean boundary that lets the model get swapped without rewriting the logic around it.