AI benchmarks are often used to compare models, but their results are only useful when the testing methods are reliable and can be repeated. Irene Solaiman, chief policy officer at Hugging Face, says the field still needs more consistent practices.
Reproducibility is a key part of assessing whether an evaluation can be trusted. Access to a model, including its weights, can help other researchers repeat tests. But even when individual evaluations are repeatable, comparing results across different tests remains difficult.
Assessing the whole AI system
Solaiman says evaluator qualifications and testing methods need greater standardization, including for third-party evaluators embedded within AI organizations. Assessments are also expanding beyond models to include the surrounding tools, or "harnesses," and the broader AI agents people use. These agent-focused evaluations can be costly and challenging to conduct.
A shared approach could make benchmarks more useful across the field. As evaluation methods mature, clearer standards may help people compare AI systems and understand how they perform in real-world use.
