AI laboratories have built their reputations on model benchmarking, a system designed to objectively measure the superiority of one AI product over another. However, given this reliance, can current testing methods be trusted?
“Trust is such a weighted term,” Solaiman explained. “In terms of whether we can trust results, one big part is whether you can reproduce an evaluation. Do you have the necessary level of access to the model itself? This is where open weights play a significant role,” she added.
Reproducibility in model evaluations has consistently been one of the most challenging aspects of benchmarking and remains a subject of intense industry debate.
Beyond individual test validity, there is also the question of how different evaluations compare to one another.
“They are an evolving science where we don’t know how to meaningfully compare them to each other,” Solaiman noted.
She added that these tests tend to become ineffective relatively quickly, and the industry does not yet have a thriving ecosystem. “There is a current hunger for an embedded evaluator ecosystem, but the qualifications that determine who is an effective evaluator across various tasks require much more standardization,” she stated.
The concept of embedded evaluators—where supposedly “neutral” third parties are placed inside AI organizations—has gained popularity recently as labs try to address a wave of safety incidents, such as the infamous incident targeting Hugging Face.
Instead of relying on active government regulation to identify and mitigate safety concerns, labs have pushed for third-party solutions. However, doubts persist regarding the neutrality and effectiveness of these evaluators. For instance, the backlash against Anthropic hiring Accenture as its first evaluator highlighted concerns about whether a consulting firm is qualified for the task.
Evaluations are now evolving from merely testing the model itself to assessing the entire system, including the harness and agent. The harness, the tool a user employs to access the model, significantly impacts performance, and a weaker model with a better harness can outperform a frontier model, Solaiman shared.
“Agentic evals are really hard and really expensive,” Solaiman remarked. “We have more evaluations for models than we do for harnesses, and we are developing better evaluations for the holistic agent because that is where users are deriving the most value,” she added.
In short, the evaluation space is currently a free-for-all. If benchmarking is to remain a useful tool for industry leaders, a more uniform, industry-wide approach—where everyone is aligned—may be necessary.
The Editorial Staff at AIChief is a team of professional content writers with extensive experience in AI and marketing. Founded in 2025, AIChief has quickly grown into the largest free AI resource hub in the industry.