--- type: "article" title: "Spec-driven Testing for AI Agents: Now in Early Access" summary: "Safe Intelligence is launching a new AI Agent validation and monitoring platform named /Spec27. Signup is free for early access." newsletter: "Steampunk AI" newsletter_handle: "steampunkai" newsletter_url: "https://usecommune.com/n/steampunkai" author: "Steven Willmott (@njyx)" published: "2026-05-07T21:38:57.000Z" canonical_url: "https://usecommune.com/n/steampunkai/a/spec-driven-testing-for-ai-agents-now-in-early-access" markdown_url: "https://usecommune.com/n/steampunkai/a/spec-driven-testing-for-ai-agents-now-in-early-access.md" chat_url: "https://usecommune.com/n/steampunkai/a/spec-driven-testing-for-ai-agents-now-in-early-access/chat" source_url: "https://www.steampunkai.com/spec-driven-testing-for-ai-agents-now-in-early-access/" body_source: "imported" likes: 3 replies: 9 body_words: 565 --- # Spec-driven Testing for AI Agents: Now in Early Access I try not to use this newsletter for much self-promotion, but I hope you’ll forgive a short expedition in that direction this week... I'm super proud of the new product we've been incubating at Safe Intelligence, and now that it has launched in early access, I'm excited to share more. In a nutshell: Spec27 helps teams reliably test AI Agents in an automated, infrastructure-agnostic way. Think [unit, integration, and security testing for AI Agents all rolled into one](http://spec27.ai). Wait what? Explain more. Ok, let's rewind... As you'll know if you follow these posts (or have opened any type of media in the last 6 months), AI is increasingly turning to AI functionality delivered in the form of agents. These range from public-facing support bots to internal agents for analysis, all the way to the full-on life-organizing, [open-claw-type](https://openclaw.ai/) personal agents. As someone who used to have a lead role in a multi-agent systems lab, I find this amazing, bemusing, and also scary in equal measure! No matter what form agents take, the idea is that they take on tasks on behalf of users with only partial or sometimes no supervision. This is an incredible boon, if... You can be sure they'll behave as expected when running in the real world! To help solve this problem, the team at [Safe Intelligence](http://safeintelligence.ai/) began applying many of the principles we've developed for AI model validation to language-model-driven agents of different types. The result is an approach to AI Agent testing that takes a different tack from exciting approaches. Specifically: - **Automation**: we're heavily focused on automatically generating and running test cases so that users don't have to. More specifically, we take small sets of examples, goals, and rules and run them automatically, but also generate new robustness tests in principled ways. We also take this input data to generate red-team security tests. - **Spec-driven repeatability**: from the beginning, a key question we asked ourselves was "how do you specify how an agent is supposed to behave?" It turned out there were no great answers to that question. This led us to develop the notion of agent validation specifications. Each spec defines a capability via an input data set, context, and what an agent fulfilling a spec needs to be robust to. By building specs over time, you have the building blocks of long-term repeatable benchmarking. - **Infrastructure independence**: there are now hundreds of frameworks for building agents from extremely hands-on, like LangChain, Google Vertex, OpenAI Frontier, and many others, to very turnkey, such as agents for customer support. Our aim from the beginning was to test them all, so we took an approach that doesn't require deploying SDKs or AI gateways. Spec27 tests from the outside as a user would and tests the whole stack, including guardrails and filters. So if you're building or deploying any kind of agents, we'd love to talk. The system is free to use and is in early access. There are [early-access sign-up buttons](https://www.spec27.ai/) all over the [Spec27 site](https://www.spec27.ai/), so feel free to give it a spin. There's also a handy walkthrough video here to help you understand the system. A huge hat off to the fantastic team at Safe Intelligence working on this. There are a lot more features to come, and with your feedback, we'll get better even faster. Normal service will resume over the weekend with the next weekly links email! *** ## Discussion (9 replies) Numbering shows the reply tree: `2.1` is a reply to `2`. Message text is reproduced verbatim as posted. ### 1. Steven Willmott (@njyx) - 2026-05-08T05:40:21.494Z I don't normally do self-promotion, but I'm excited to share this. Building "testing for agents" seems simple enough - turns out the non-determinism of LLMs bites you not only on the agent build side but also on the testing side. Add to that, there's also no existing way to really specify desired behavior. Curious to hear how people are doing this already. ### 2. Fran Mendez (@fmvilas) - 2026-05-10T11:43:37.915Z Damn, I'm experiencing this problem myself with Claude all the time. It's usually a lack of context but not only. Would have to save some time to look deeply at this 👀 ### 3. Fran Mendez (@fmvilas) - 2026-05-10T11:44:12.044Z Oh! And congratulations on the launch! 🎉 Best of luck with this Steve! ### 4. Steven Willmott (@njyx) - 2026-05-10T12:52:54.250Z With this we're not o focused on individual AI or coding use since every query is different there (hard to benchmark) but we are thinking about it. At the moment more focused on cases where you make an agent that is supposed to do a fixed range of tasks at scale without supervision! ### 5. Steven Willmott (@njyx) - 2026-05-10T12:53:15.865Z Indeed context management is the new "clean up your room kid!". ### 6. Joyce Stack (@joycestack) - 2026-05-11T16:13:57.582Z I watched the video for this. It looks soooooooo cool. All the best with it. Speaking of context management - I'm trying to think about non-tech analogies for managing context and the side effects. Anyone got some nice examples? I'll credit you in my work presentation. Claude is awful ### 7. Steven Willmott (@njyx) - 2026-05-11T18:01:26.678Z Thanks Joyce! Been thinking about the same problem - AI for code works so well because you have context (docs, the code base, the system), a closed loop (testing and CLI) and a lot of architectural patterns. For knowledge work more generally, you might have the last (sometimes), but generally don't have the others. Plugging into google drive doesn't help a lot! ### 8. Joyce Stack (@joycestack) - 2026-05-11T18:26:45.902Z > Quoting Steven Willmott (@njyx): Thanks Joyce! Been thinking about the same problem - AI for code works so well because you have context (docs, the code base, the system), a closed loop (testing and CLI) and a lot of architectural pa… I started with a whiteboard example…(this may also be lost on younger folks)…thinking back to the days of redrawing the sprint board when a new sprint starts. One would carefully write your index cards and place them on the board. Every card had a purpose. Every card had a place. But nobody ever cleaned the board between sprints. Cards got added. Old ones stayed up.Some work got squeezed into the margins. Nobody knew what was missing. This is the AI context - CLAUDE.md (or whatever your poison is) is the board that never gets cleared, gets things stuck to, linked to other cards with bluetac or tape. You never know when the board is full. Cards fall off it. You'll never know. ### 9. Joyce Stack (@joycestack) - 2026-05-11T18:27:46.466Z You'll only ever know there was an issue with your board when you get to production.