In Don’t Automate the SDLC, I argued that we should recover the need behind each software development artefact and redesign how that need is met. Backlogs serve memory and coordination, teams provide belonging and accumulated understanding, code ownership creates accountability, and roadmaps give customers confidence. We need those things, but we don’t necessarily need their current forms.
I’d replace the familiar lifecycle with a continuous system that compares the world we have with the outcome we want, then makes bounded interventions to close the gap. It observes what happens, updates its beliefs and decides what to try next. There’s no natural point at which the outcome becomes done.
Continuous delivery is moving from software changes to product outcomes
We moved from yearly golden releases to continuous integration and deployment because smaller changes, faster feedback and routine releases were safer than accumulating everything into one enormous event. Growth teams then applied a similar idea to customer behaviour, running experiments continuously and using evidence to decide what to retain.
AI lets us extend that model into product development itself. Agents can explore different implementations, produce working versions, evaluate them and adapt them quickly enough that choosing the intervention no longer has to be a long planning exercise followed by a long build.
Growth experimentation is the closest existing model, although it doesn’t transfer perfectly. Growth experiments are often narrow, reversible and disposable. A message, landing page or onboarding sequence can lose an experiment and disappear. Features create longer-lived contracts with customers. They affect documentation, support, integrations, data, learned behaviour and the overall coherence of the product.
Applying the growth loop to features therefore needs stronger controls around compatibility, customer disruption and what the product is allowed to become. The eventual intervention also needs to converge into one coherent product. I don’t expect customers, Sales or Support to accept a world where every account has its own permanently generated version of the software, and I don’t think they should have to.
The outcome and context have to remain live
The system needs persistent state rather than a trail of documents produced at different stages. The outcome is the first part of that state: the customer behaviour we want to change, the expected business effect, the character of the experience, the boundaries we won’t cross and the counter-effects we need to watch.
It also needs to retain uncertainty. “Customers who do X retain better” doesn’t prove that making more customers do X will improve retention. The system should store the causal belief and the evidence behind it, rather than quietly turning a correlation into an instruction.
Product still has an important role here. Someone needs to steward which outcomes deserve attention, how they connect to customer and business value, and whether the emerging interventions still form one coherent product. Product moves away from maintaining the queue and specifying the solution, and towards holding the intent, causal belief and evidence together.
Product can’t define the outcome alone. Company strategy gives it legitimacy, customer evidence keeps it real, and design, engineering, operations, security and legal contribute boundaries that Product can’t trade away. Somebody also needs to check that local improvements still add up to a product that makes sense.
Alongside the outcome sits the context: customer research, commercial data, brand principles, technical constraints, previous experiments, current commitments and decision history. This is where some of today’s backlog can survive as agent context. The queue of speculative solutions can go.
Context needs active maintenance. A giant collection of stale and contradictory material would recreate the backlog in a more expensive format. The system has to know which information is current, where evidence conflicts and which decisions no longer apply.
Interventions should compete on value, evidence and risk
Instead of selecting one solution and moving it through delivery, the system can maintain several possible interventions. Some might involve product changes, while others change pricing, documentation, support or an operational process.
Each intervention has an expected effect, an information value, a cost and a potential downside. A useful intervention may produce direct business value, reduce uncertainty or do both. That matters because a system focused only on immediate outcome movement will favour things that look good quickly. Conversion can increase while trust falls, engagement can rise while the experience gets worse, and adoption can grow by attracting customers who won’t stay.
The system therefore needs to consider counter-effects and longer-term signals alongside the headline outcome. Kohavi, Tang and Xu’s Trustworthy Online Controlled Experiments calls these guardrail metrics, and Twyman’s law warns that any figure that looks interesting or different is usually wrong. It also needs to choose interventions that teach us something when the causal belief is weak. A small experiment that rules out an attractive but wrong theory may be more valuable than a larger change built on confidence we haven’t earned.
Evaluation is part of the persistent system too. Technical tests matter, but the evaluation also needs to cover behavioural effect, business value, experience quality, safety, compatibility, supportability and any customer promises involved. Some criteria can be deterministic, some statistical, some model-based and some need people.
Jessica Kerr gives a useful example in Our Limits Are Not Our Imagination: an engineer using agents with property-based testing, formal verification and architecture checks in a greenfield system. Cheaper implementation can make stronger checks practical too. People still have to decide which properties matter and whether the checks provide enough evidence for the proposed exposure.
There is a serious risk of recreating the PRD here. If the evaluations specify every expected screen, interaction and implementation detail, we have produced machine-readable requirements and left the system no room to find a better intervention. Evaluations should describe what an acceptable result looks like and what must remain true. That includes the outcome signal, unwanted effects, customer promises, brand qualities, safety boundaries and operational limits.
Each evaluation should be traceable to something it protects, such as an outcome, known risk, customer promise or previous failure. It should also have a scope. Some evaluations are permanent product invariants, some apply to a capability and some belong only to the current outcome bet. Without that discipline they will accumulate forgotten decisions in much the same way backlogs do.
I don’t think the disguised-PRD problem is solved. The tension is useful because it forces us to ask how much intent a system needs before we have simply specified the answer.
Release becomes progressive exposure
A release in this model is an increase in real-world exposure. An intervention might begin with generated examples and synthetic users, move into simulated or replayed production conditions, then reach employees, design partners, a small customer cohort and eventually broad availability.
Those aren’t compulsory stages. Changing a piece of copy and changing a payments architecture shouldn’t follow the same route. The evidence required depends on the possible harm, reversibility and uncertainty of the intervention.
Experimental cohorts also shouldn’t create permanent product fragmentation. They let us gather evidence before choosing the canonical behaviour, with any real product tiers or variations made explicit. Once the decision has been made, customers and the people supporting them need to know what the product is, how it works and what has changed.
A mature capability doesn’t have to stop changing. It has stable behavioural contracts, known failure modes, useful observability, compatibility guarantees, safe migrations and predictable deprecation. Those mechanisms allow it to change without surprising the people who depend on it.
An untouched capability may be mature, or it may be sitting still because nobody understands it well enough to risk a change.
There is also no useful “done” state for an outcome. The system may scale an intervention, modify it, reverse it, continue observing, stop investing or reframe the outcome because the original belief was wrong. A capability can move into a lower-attention operating state when it behaves within acceptable bounds and further investment has less value than work elsewhere. Agents can continue monitoring it and bring people back when something materially changes. Will Larson found his agent loop most useful for exactly this, checking on projects after release.
Customer-operated software creates a second delivery loop
SaaS makes progressive exposure easier because the vendor controls the runtime. Customer-operated software is different. The vendor can generate and evaluate an intervention, but the customer decides when and how it enters their environment.
That creates two connected loops. The vendor explores, evaluates and certifies an intervention, then the customer evaluates and adopts it within their own constraints. The release package is the transfer between those loops, with a compatibility claim, support position and adoption plan attached.
Asking customers to describe their complete setup manually won’t be enough. The strangest details are often the ones nobody remembers to mention until the upgrade fails. A better approach would combine a machine-readable environment profile, privacy-conscious telemetry or diagnostic bundles, customer-run preflight checks and reference environments generated from the configurations we see most often.
The system also needs an explicit support envelope. Tested, supported, observed and unknown configurations aren’t the same. “We have never seen this combination before” should affect the recommendation the customer receives, even when every individual component is technically supported.
Progressive exposure still works, but the mechanism changes. Internal environments can be followed by reference installations, design partners, preview channels, stable releases and long-term-support versions. Customers retain control of the final adoption step.
Documentation, notice, migration, rollback and support readiness belong inside the intervention. A technically sound release that customers can’t safely absorb has failed part of its evaluation.
Agents can earn authority within defined boundaries
Every agent needs explicit limits on which systems it may change, what data it may access, which customers it may affect and how much it may spend. Those limits should also say what is prohibited and when a person must decide. An intervention can proceed within them; a larger, less reversible or less familiar action needs further authorisation. That is the leash again: long for the reversible and low-risk, short for the rest.
Agents should be able to earn wider authority over time, based on evidence from their decisions and outcomes. Imprint already does this for humans, as I described in the previous post: a change can only skip review if, among other conditions, its author has ten or more merged changes in that service. That authority has to remain specific to the capability and kind of action involved. A strong record making low-risk onboarding changes says very little about whether an agent should alter authentication or payments.
Changes to the agent matter too. A different model, instruction set, tool collection or context may mean some authority has to be earned again. Trust in this setting belongs to a configured system operating in a particular environment, rather than to an agent name that happens to remain the same. The six tests I published in October are mostly questions about whether a platform can hold that envelope at all: identity per tool call, spend per run, policy per step, approval at a named step.
The agent can propose a change to its own evaluations or authority, but it shouldn’t approve that change unilaterally. Proposal, evaluation and authorisation need separation, with independent systems handling low-risk cases and humans making consequential decisions. Authority should also expire or shrink when the surrounding environment changes.
Human involvement should depend on the decision being made. People need to enter when the outcome is contested, values conflict, the evidence supports several interpretations, the intervention creates a meaningful customer promise, the brand judgement is novel or the possible harm exceeds the system’s authority.
That includes interventions which perform well commercially but feel wrong. The experience character and non-negotiable boundaries should be part of the outcome before broad exposure. Taste review also needs to happen early enough to prevent a locally successful but manipulative intervention reaching most customers. If the problem only becomes visible in real use, a human should still be able to stop it without having to invent a quantitative countermetric to justify the decision. Control that is cheap to invoke and biased towards stopping is the pattern I borrowed from biology for agent systems generally.
Auditability can’t depend on exact generative replay
Earned authority requires us to understand what an agent did and why the surrounding control system allowed it. Exact reconstruction of a generative execution is a hard problem, and in many cases may be impossible.
Keeping the model name, prompt, parameters and context doesn’t guarantee the same output. Hosted models change, retrieved information moves, tool responses depend on external state and providers don’t expose every part of their infrastructure. Capturing all context indefinitely would also create a fairly spectacular security and data-retention problem.
The audit target should be the consequential action path. This is what the audit and replay test in my six tests was asking for: reconstruct the external state changes and verify the artefacts and controls from stored inputs, tool results and outputs, without calling the model. We need to know which configured system acted, what authority it had, which evidence and policies were available, which tools it called, what those tools returned, which evaluations passed, what external actions occurred and what changed as a result.
We already audit human decisions without recreating the person’s thoughts. We examine the available evidence, authority, recorded decision, applicable controls and outcome. AI needs stronger records because it can act faster and at much greater scale, but a model-generated explanation of its own reasoning isn’t a safe substitute. It may produce a convincing rationale without giving us a faithful account of why it acted. Anthropic’s own research on reasoning models found they often leave out of their stated reasoning the hint that drove the answer.
For each material intervention, the system should retain a bounded decision record. That would include the outcome and causal belief at the time, references to the relevant context, the versions of policies and evaluations, the agent/model/tool configuration, proposed and executed actions, human decisions where required, the final artefact, the exposed environments or cohorts and the observed result. Customer-operated software would add the compatibility claim, support envelope, build provenance, release signature and software bill of materials.
The control system should also record why an action was permitted: which conditions were met, which evaluations passed and why the remaining risk sat inside the authority envelope.
Higher-consequence domains may need frozen models, retained weights, versioned execution environments and more deterministic surrounding components, and in the EU the AI Act’s record-keeping rules for high-risk systems will settle some of this for you. That will be expensive and may not be possible with many hosted systems. Lower-risk and reversible work can rely more on scoped permissions, observable actions, independent evaluation and reliable rollback.
What doesn’t exist yet
I haven’t found anything that joins agent identity, authority, context provenance, evaluations, human decisions, external actions and outcomes into one properly governed record yet. I said in March that this layer barely existed, and the list of what it needs has only got more specific since. Tracing is the most complete piece. OpenTelemetry’s conventions for agents already define a span per agent invocation and one per tool call, with arguments and results recorded where privacy allows. Authority, evaluations and human decisions have nothing equivalent, and the integrated layer feels quite far away. It’s also the problem closest to my own day job, which is partly why I care about it.
Until we have that layer, autonomy beyond low-risk, reversible work will be difficult to justify.