Microsoft Unleashes Text-Driven AI Behavior Testing for Devs

While AI researchers and laboratories have made significant strides in evaluating AI models for a broad spectrum of criteria, from safety and compliance to alignment and avoiding sycophancy, a distinct and pressing need has emerged for companies and developers: ensuring their AI systems consistently perform as intended for their specific products or services.

To streamline this crucial testing process, Microsoft unveiled ASSERT on Tuesday, an acronym for Adaptive Spec-driven Scoring for Evaluation and Regression Testing.

This open-source framework, according to Microsoft, simplifies the evaluation of application-specific AI behavior. It leverages AI to transform high-level, natural-language descriptions of desired goals, policies, or behaviors into comprehensive, scored tests that are fully investigable.

ASSERT operates by processing plain-language descriptions of an AI model's expected conduct and policies, converting them into a structured framework of acceptable and unacceptable behaviors. It then generates relevant problem scenarios and test cases, executes them against the target system, and assigns scores to the outcomes. Furthermore, it can meticulously record the pathways taken by the AI system, including intermediate actions and tool calls, empowering developers to pinpoint precisely where failures occur.

Developers also have the flexibility to provide system context, specific tools, and operational constraints to further tailor the scope of these evaluations.

For instance, a developer could specify that an AI agent designed for document research must not send emails to external contacts, restrict confidential information access solely to C-level executives, and generate concise summaries while considering prior context. ASSERT would then utilize these precise rules to generate ongoing test cases, verifying the system's adherence to these stipulations.

Microsoft asserts that this framework effectively bridges a critical gap that broader, more generalized evaluations cannot address, particularly when AI models' intended behavior is intricately shaped by an application or product's unique context, policies, and integrated tools.

“One of the things we’ve learned is that evaluations are absolutely critical to making good decisions,” stated Sarah Bird, Chief Product Officer of Responsible AI at Microsoft. She elaborated, “Because if you don’t understand the behavior of the AI system, it’s really hard to know if it’s meeting your organization’s bar […] What we found is that if you really want to have a trustworthy system, you should evaluate many more dimensions that are application-specific.”

Bird highlighted ASSERT's versatility, noting its applicability for evaluating systems during their development phase, post-deployment, and even for continuous monitoring.

This release aligns with a broader, evolving trend within the AI industry. As AI models become increasingly sophisticated, researchers are intensifying their focus on repeatable testing and robust regression checks. This shift is evident in initiatives like Stanford’s HELM, MLCommons’ AILuminate, and various evaluation groups such as METR, all of which are establishing benchmarks to meticulously measure how models perform under diverse conditions.