Cisco Open-Sources Specification For Building AI-Powered Security Evaluation Systems

Cisco has released an open-source specification designed to help security teams build structured, AI-driven systems for evaluating software vulnerabilities, moving beyond ad hoc use of large language models towards repeatable and auditable security assessments.

The project, called Foundry Security Spec, provides a model-agnostic and infrastructure-neutral blueprint for constructing what Cisco describes as an “agentic security evaluation system” – one that wraps AI models in defined roles, orchestration layers and guardrails rather than relying on unstructured prompts.

Omar Santos, Distinguished Engineer in AI Security Engineering at Cisco, framed the release as a response to a common problem facing security teams experimenting with frontier AI models.

“Every security team with access to a frontier LLM has tried the same thing at least once: toss a repo at the model and ask it to ‘find the bugs,'” Santos wrote. “The result is usually a wall of unbounded, unverifiable output that mixes sharp insights with hallucinated findings, with no way to know what was missed or when you’re actually done.”

The problem with unstructured AI security testing

The specification addresses what Cisco identifies as a growing gap between the potential of AI models for security work and the reality of how most organisations are deploying them.

According to Santos, organisations investing in AI-assisted security are frequently getting back “hallucinated findings, false positives at scale, and no coverage signal.” The Foundry Security Spec is intended to provide the scaffolding that turns those experiments into something defensible.

“The difference is stark – one is an interesting demo; the other is a security evaluation system you can defend in front of your CISO and your auditors,” Santos explained.

The specification aims to deliver four outcomes: a bounded and prioritised set of verifiable findings; a clear “done” signal based on operator-defined coverage floors and economic yield thresholds; an auditable chain of provenance from detection through triage, validation and publication; and safety guardrails that constrain AI model behaviour at the system level rather than through prompts alone.

Eight roles and eleven principles

Foundry Security Spec is published as two main artefacts. The first is the specification itself, which defines eight core agent roles, five extension roles, a finding lifecycle, a coordination substrate and roughly 130 functional requirements. Each requirement includes an inline rationale explaining why it exists.

The eight core roles are Orchestrator, Indexer, Cartographer, Detector, Triager, Validator, Coverage-Guide and Reporter. Each has defined inputs, outputs and functional requirements, but Cisco has deliberately left implementation choices open.

“You can implement them as subprocess loops, as graph-based pipelines, as serverless functions, as a bespoke harness,” Santos wrote. “The shape is what transfers; the implementation is yours.”

The second artefact is a “constitution” document containing eleven principles that Cisco describes as inviolable. The company indicated that every one of them encodes a production failure that was shipped, diagnosed and fixed internally.

The constitution is designed to be read by AI agents such as Claude Code or Codex to build the infrastructure, but Santos noted it is also “deliberately written as prose aimed at the human builder and maintainer,” with each principle explaining the specific failure it prevents so that engineers understand the cost of weakening any rule before they do so.

Integration with Project CodeGuard

The specification is designed to work alongside Project CodeGuard, a separate Cisco open-source project that was previously donated to the Coalition for Secure AI (CoSAI). CodeGuard provides security rules and agent skills that guide AI coding agents to generate more secure code.

Together, the two projects form what Cisco describes as a “self-improving detection-to-prevention flywheel.” CodeGuard rules sweep a codebase for known vulnerability patterns, while Foundry’s exploratory agents hunt for issues no existing rule describes. When exploration uncovers something the rules missed, the gap is recorded, generalised into a new CodeGuard rule and added to the corpus.

Because CodeGuard rules are portable, they can also be loaded into developer coding assistants as secure-coding rulesets. The result, according to Cisco, is that vulnerability classes discovered during one evaluation are prevented at the keystroke in every developer’s editor before the next evaluation runs.

“Every turn of the loop improves detection here and prevention everywhere,” Santos wrote.

A blueprint, not a product

Cisco was explicit that Foundry Security Spec is not a turnkey scanner or managed service. It is a specification intended to be adapted to each organisation’s environment, threat model and objectives.

“Every place where the seed could either dictate a choice or leave it open, we left it open and explained the trade-off,” Santos wrote.

The company explained its decision to release a specification rather than source code by pointing to the tight coupling of its internal implementations to Cisco-specific infrastructure including its LLM gateway, issue tracker and private cloud. Open-sourcing that code would have produced something that runs in one environment and does not transfer, Santos argued.

“What transfers is the design: which roles you need and why, what each must guarantee, how findings flow from detection to publication, what ‘done’ means for an evaluation, where the quality gates go, and which shortcuts will hurt you six months in,” he wrote.

Designed to outlast current models

Cisco also addressed the question of whether the specification will become obsolete as AI models evolve. Santos argued the spec was designed to avoid that fate by building on functional requirements and roles rather than specific model parameters.

“Whether you are using today’s frontier models or the more complex reasoning agents of tomorrow, the need for an orchestrator, a detector, and a validator will remain constant,” he wrote. “The spec is designed to be the stable harness that keeps your security evaluation consistent, regardless of the ‘engine’ under the hood.”

The specification is compatible with GitHub’s spec-kit, an industry set of spec-driven development workflows that can be used with different AI agents.

Cisco has published the Foundry Security Spec on GitHub and invited security teams and developers to build on, adapt and contribute to the project.

Last Updated on May 12, 2026 by Nick Ross