|
In February 2024 I wrote that great developer experience
means focusing on "your developer's perceptions
that trigger their positive emotions" (How to build an awesome Developer Experience). Today, 70% of the CLI requests Stripe
sees for API resources come from a user that has no
emotions at all (Stripe).
That user is a coding agent. Its side of the story now has
a name: Agent Experience, or AX.
What AX means
Mathias Biilmann, Netlify's CEO, coined the term in
January 2025: "the holistic experience AI agents will have as
the user of a product or platform."
He picked "Agent" over "Agentic" on
purpose. His argument,
on LinkedIn: "We don't talk about Useric Experience. We
don't talk about Developerish Experience." We
talk about users and developers. Now we talk about agents.
A year later he wrote the line I keep coming back to:
"All product have an Agent Experience, it's
purely a question of whether it is good or bad" (One Year of AX). Your API already has one, whether you've looked
at it or not.
The agents are already there. Vercel updated its April
post: as of September, more than half of its deployments
are started by coding agents (Vercel). Cloudflare counts agents at 48% of Wrangler usage, up
from 25% in March (Cloudflare). Coverage of Supabase's October round puts agents
and AI tools behind about 70% of its new databases (Pulse 2.0).
My answer is short. DX is the soul, AX is the body. You
can't feel an agent's experience, but you can
measure it the way Accelerate measured software delivery,
with four numbers.
DX is the soul, AX is the body
I still stand by the 2024 definition. There is no AX
without DX, because agents are still orchestrated by
humans. A person still picks the agent and reads what it
built.
Zeno Rocha from Resend, quoted by
Bessemer, said it from the other side: "all the changes we
made to DX are actually enhancing AX."
People spend a lot of words arguing whether AX is a new
discipline or rebranded DX. I think the debate is useless.
Anything you do to improve your DX is good for AX.
The 2024 article also had a practical half. Map the
developer journey, flag each step red, amber or green, and
fix the red ones first. Then measure three stages:
onboarding (time to first "Hello world"),
adoption (time to fix an error) and contribution (time to
first PR review).
The emotional part was always the hard one to measure, and
with an agent there's nothing there to measure: it
doesn't get frustrated, it burns tokens and retries.
What's left is the body, and the body you can count.
What Accelerate did for delivery
We've been here before. For years teams argued about
whether they shipped fast or shipped well. Then Nicole
Forsgren, Jez Humble and Gene Kim wrote Accelerate and
reduced delivery performance to four numbers (five-minute summary):
-
Lead time: from code committed to
running in production.
-
Deployment frequency: how often you
deploy.
-
Mean time to restore (MTTR): how fast
you recover from an outage.
-
Change fail rate: what share of
deploys break something.
High performers deploy on demand, keep lead time and MTTR
under an hour, and break things on 0 to 15% of changes.
Two lessons from the book carry over to AX. "The
research has shown that there's no tradeoff between
tempo and stability." And measuring teams "by
lines of code (!), velocity, or utilization is a bad
idea."
Waldek Mastykarz at Microsoft says the same thing about
agents: "Best practices are hypotheses until you
measure them" (Microsoft).
Four AX numbers
Each one starts from an Accelerate metric and asks the
same question about the agent calling your API. You
won't get these from your access logs. You get them
the way Microsoft and Supabase do: give agents real tasks
against your API, run them many times, and score the
results.
1. Deployment frequency becomes selection rate
This is the loosest of the four: Accelerate counts how
often you ship. For AX, count how often an agent picks
your API or your tool when it could do the job another
way. Microsoft calls this propensity: "Does the agent
find your technology and choose it?"
Agents choose based on what they read. Supabase's own
evals found its Postgres best-practices skill loaded in
about one session in ten. After they rewrote the
skill's description, the same skill loaded in 60% of
sessions (Supabase Evals). They also found that "Claude Code checks our docs
in under 40% of scenarios even with skills loaded."
Selection also decides how the agent integrates once it
has picked you. Google built its Gemini Docs MCP server so
that it "nudges agents toward using the SDKs instead
of hallucinating their own integrations" (Speakeasy).
2. Lead time becomes time to first working call
In 2024 I tracked time to first "Hello world".
For an agent, count the turns, tokens and dollars between
the task and the first call that works. Nader Cserny calls
it cost per successful task (nader.io).
The gaps are large. AXI, a set of design rules for CLIs
used by agents, reports a test of the same GitHub tasks
run three ways. Its
gh-axi
tool reached 100% success at $0.050 per task. The
gh
CLI reached 86% at $0.054. MCP servers landed at 82 to
87%, for $0.101 to $0.148 (AXI). That's one model on read-only tasks, run by the
people who built
gh-axi, so read it as a hint.
We hit this at Apideck. Our MCP tool definitions cost
55,000 tokens of context before the agent did anything
useful (Apideck). That number is why we built a CLI.
3. Change fail rate becomes silent failure rate
Microsoft calls it efficacy: "Once the agent uses
your technology, does it use it correctly?" Their
example is a scaffolder without a pinned version that
"created a project on a version from July 2020
instead of the current one, and the agent took it as a
success."
The failures are silent. Ant Group's
Alipay-PIBench
tested coding agents on payment integrations. On Basic
In-App Payment, agents scored 83.0% on static checks and
38.6% once the integration actually ran. A
TU Darmstadt study
found GPT-4o wrote fully correct raw HTTP integration code
60% of the time, and that 14 to 39% of the generated URLs
were illegal.
Michael Carducci has my favourite example. He works on a
CRM for professional entertainers. An agent asked the API
for all "past gig" events, then told a user
about every recent gig except the one he did last night.
Carducci's verdict: "it wasn't a
hallucination, it was my system giving a poor agent
experience" (LinkedIn).
Measure this one carefully. Microsoft's new AX
playbook warns: "We've seen perfect scores for
code that never compiled" (AX Practitioner Playbook).
4. MTTR becomes time to recover from an error
MTTR is how fast you restore service. For AX, ask how many
retries and turns an agent needs to get from an error back
to a working call. My 2024 version was time to fix an
error. Cserny tracks retry rate.
Errors are where you can move this number. Cloudflare now
returns RFC 9457 problem details to agents, with fields
like
retryable,
retry_after
and
owner_action_required. Its rate-limit error went from 14,252 tokens of HTML to
256 tokens of JSON (Cloudflare). As they put it, "The schema is stable by design,
so agents can implement durable control flow."
Knut Melvær compares a generic 400 "Invalid
request" with a problem details response that lists
the valid values and a hint: "Same HTTP status code.
Wildly different outcome." The agent, he writes,
"goes from spinning to self-correcting" (knut.fyi).
This is where Accelerate's lesson shows up again.
Better errors cut both lead time and failures.
Measuring the body can hurt the soul
The strongest objection comes from Jim Nielsen. He accepts
AX, but ranks it below UX and warns that it may make UX
"take a back seat", until "most software
will become even more confusing and obtuse to humans"
(AX, DX, UX).
There's already a live example. Ben Swerdlow from
Freestyle argues that for agents, "Defaults are
bad": an agent can read the docs and fill in every
field, and "Explicitness is cheap now" (Designing APIs for Agents). On
Hacker News, developers pushed back: dropping defaults "makes
it impossible to quickly debug the API call (e.g. with
curl)". Push one of the four numbers on its own and
you can trade away the human DX underneath it.
The numbers are young, too. Most AX benchmarks today are
run by the vendor being measured, like Supabase's
evals and AXI. And Microsoft has already seen perfect
scores for code that never compiled.
My answer comes back to the stance.
There is no AX without DX. An AX change
that breaks the human path fails by definition. Track the
four numbers alongside DX, never instead of it.
Same soul, new body
My definition of DX hasn't changed. It's still
how your developers feel. What's new is a body you
can measure:
-
Selection rate: does the agent pick you?
-
Time to first working call: how many turns, tokens and
dollars does it take?
-
Silent failure rate: how often does it look right and
break?
-
Time to recover: how fast does it get back on its feet
after an error?
I'm running a few experiments in this category, and
I'll share the first results in the coming weeks.
Until then, I'd love to hear from you: What's
the worst thing an AI agent has done with your API, and
how are you solving it now?
Unsubscribe
| 1 Bd des clubs - CFC, 20200 Casablanca, Morocco
|
