Nick Hoff

The Compiler Is Only Half the Programmer

March 11, 2026

A compiler can produce a binary without ever seeing it fail. A programmer cannot build a serious application that way.

The programmer runs the system, pokes it, watches it, changes the experiment, and rewrites the program. Designing only a language for an AI programmer is like designing C without a terminal, debugger, test runner, browser, or computer.

The language is the agent's pen. A programming system also needs eyes, hands, instruments, memory, and a judge.

Part 3 extended the semantic program into deployment and provider state. That gives us something complete enough to run. It does not tell the model what to run, how to interact with it, what to observe, or whether the result satisfies the requirement.

The compiler analogy ends too early

The feed-forward story is attractive:

intent → model → semantic program → compiler → running system

It is also missing most of development. After a failed run, an agent has to decide whether the cause is code, fixture state, environment drift, a provider event, an expired session, a race, or a mistaken requirement. It needs to alter one condition, rerun from a meaningful checkpoint, collect evidence, and update its working theory.

The operative loop is:

write → compile → run → observe → evaluate → learn → rewrite

Manipulation cuts across the middle of that loop. A developer does not merely watch a program. They resize the viewport, expire a session, delay a webhook, seed a database, deny a permission, kill a process, or freeze time. These are experiments: controlled changes intended to distinguish hypotheses.

The strongest objection: agents already have tools

Coding agents already use shells, browsers, Playwright, emulators, debuggers, database clients, and cloud CLIs. Why invent another protocol?

Those tools should remain. The problem is not their capability; it is the boundary between them.

A Playwright click, SQL result, process log, provider event, and trace span use different identifiers and authority models. Their outputs do not automatically map back to the semantic declaration that caused them. The agent receives subsystem-specific text and spends tokens reconstructing chronology and ownership.

A common experiment protocol can wrap existing tools without replacing them. It gives actions and observations stable identities, timestamps, semantic provenance, authority metadata, and a shared evidence model. The browser adapter still uses Playwright. The database adapter still speaks SQL. The protocol says how their results belong to the same run.

Experiments should be programs

An experiment is a structured sequence with an environment, fixtures, actions, manipulations, observations, assertions, checkpoints, and a replay policy.

scenario AnnualPlanPurchase {
  environment = preview

  device = Browser {
    viewport = 390x844
    locale = "de-DE"
    network = slow_4g
  }

  fixture {
    user = Storefront.customers.test_user {
      email = "buyer+scenario@example.com"
    }
    payment = Storefront.billing.test_payment("succeeds")
  }

  run Storefront.web

  act navigate("/pricing")
  act sign_in(user)
  act tap(ui.button("Buy annual plan"))
  act complete_payment(payment)

  observe [
    screen,
    accessibility_tree,
    browser.console,
    network,
    trace(Storefront.begin_checkout),
    Storefront.billing.events,
    Storefront.commerce.Subscription
  ]

  expect {
    Storefront.billing.checkout.count == 1
    Storefront.billing.verified_event.count == 1
    Storefront.commerce.Subscription(user).status == active
    eventually ui.text("Annual plan active") within 10s
    network.no_server_error
  }
}

This is not a brittle recording of cursor coordinates. ui.button("Buy annual plan") resolves through an accessibility role or stable semantic identity. The scenario describes what matters: a signed-in customer, one checkout, one verified event, one active subscription, and an eventual visible state.

Playwright's ARIA snapshots are a useful concrete example. They expose the accessible structure of a page as a tree that can be asserted independently of pixel coordinates. The experiment protocol should combine that semantic view with pixels, not choose one.

An ad hoc debugging experiment can later become a regression scenario. That promotion is valuable: the repair leaves behind a reproducible artifact rather than only a successful final patch.

Actions change the evidence available

The runtime needs semantic actions for several classes of environment.

For browsers and phones: navigate, tap, type, drag, change viewport, rotate, change locale, grant or deny permissions, background an app, and select through accessibility semantics.

For processes and services: throttle a network, advance virtual time, inject or reorder events, kill a process, change a feature flag, seed or corrupt data, deploy a preview, or bind a provider interface to a simulator.

The action vocabulary is domain-extensible. A graphics runtime can move a camera. An ML runtime can resume from a checkpoint. A systems runtime can inject an interrupt. The universal layer records the action's identity, target, authority, time, result, and relationship to the experiment.

Observation must be structured and multimodal

A screenshot is necessary because it shows what a user could perceive. It is insufficient because it does not say which semantic node owns a label, which request populated it, or what state changed underneath it.

A useful browser observation might correlate:

• Screenshot and viewport.
• Accessibility tree and focus state.
• DOM or platform UI hierarchy.
• Console and process events.
• Network requests and responses.
• Application trace.
• Database and application-state deltas.
• External provider events.
• Semantic source identities.

Distributed tracing already demonstrates part of this mechanism. OpenTelemetry context propagation carries trace and span identifiers across service boundaries so separate operations can be assembled into one trace. A model-facing runtime needs similar correlation across UI actions, generated code, database writes, and provider adapters.

The model should not receive every artifact by default. It should query the evidence bundle: show requests caused by this tap; show state that changed after this event; show the source declaration that produced this handler.

Observation is not evaluation

This loop is bad:

run → dump logs → ask the model what it thinks

It mixes evidence, interpretation, and success criteria in one probabilistic step.

A better loop is:

run → structured evidence → evaluate against explicit criteria

The screenshot may show "Annual plan active." Evaluation asks whether a verified event exists, whether the entitlement belongs to the correct user, and whether a duplicate event changed state twice. Visual success is one observation, not the definition of correctness.

The evaluator also needs some independence from the repair process. Otherwise the easiest repair is to weaken the assertion. Part 5 will make that boundary explicit.

Replay the meaningful state, not just the clicks

Debugging asynchronous systems requires control over nondeterminism where possible:

• Stable random seeds.
• Recorded external events.
• Virtual time.
• Database and fixture restoration.
• Semantic checkpoints.
• Explicit declarations of what cannot be reproduced.

Perfect determinism is not a realistic requirement. External providers, schedulers, networks, and hardware introduce behavior the runtime may not control. The goal is to bound and expose nondeterminism rather than hide it behind flaky retries.

experiment DiagnoseDelayedEntitlement {
  replay AnnualPlanPurchase from step.complete_payment

  manipulate {
    Storefront.billing.delay_event(CheckoutCompleted, by = 30s)
    clock.freeze()
  }

  watch ui.node("account.subscription-status")
  probe value Storefront.commerce.Subscription(user)

  expect {
    before event(CheckoutCompleted) {
      ui.text("Payment processing")
      Storefront.commerce.Subscription(user).status != active
    }

    after event(CheckoutCompleted) {
      eventually ui.text("Annual plan active") within 5s
    }
  }
}

The checkpoint is semantic: after payment completion, before event delivery. That remains meaningful even if generated test code changes.

The browser return is not the payment event

The customer completes payment and the browser returns to the application. That navigation is not authoritative proof of payment. The application can show a processing state while it waits for a signed provider event. The event handler verifies the signature, performs a transactional and idempotent state transition, and only then exposes an active entitlement.

The runtime should be able to duplicate the event deliberately:

variant DuplicatePaymentEvent from AnnualPlanPurchase {
  manipulate Storefront.billing.deliver_event(duplicate = 3)

  expect {
    Storefront.commerce.Subscription.count_delta == 1
    Storefront.billing.checkout.count == 1
  }
}

This experiment makes a distributed-systems obligation executable. It also creates better training material than a comment saying "webhooks may be delivered more than once."

Observation is domain-specific; evidence is not

Different domains need different instruments:

| Domain | Useful observations | |---|---| | ML | Loss curve, gradient distribution, sample predictions, accelerator memory | | Spreadsheet | Rendered sheet, changed cells, formula dependency graph, schema violations | | Systems | Kernel log, interrupt trace, registers, memory state, crash dump | | Graphics/game | Rendered frame, scene graph, physics contacts, frame time, input trace |

The core protocol does not standardize gradient_statistics or physics.contacts. Domain extensions define those observation types and their adapters. The core standardizes how an observation is requested, scoped, timestamped, correlated, redacted, and compared with an assertion.

The useful training unit is no longer only intent → code. It is the verified trajectory through failed runs, hypotheses, experiments, and repair.

The runtime can now return screenshots, traces, state, and events. That is still not a good debugging interface. The next problem is turning a pile of evidence into a causal, repairable explanation without letting the agent redefine success.

Previous: The Browser Is the Uncompiled Part of Your Program. Next: Debugging Software No Human Wrote.