Computer Scientist Moritz Hardt States AI Tests Are in Crisis
Moritz Hardt, director at the Max Planck Institute for Intelligent Systems, argues that AI benchmarking tests are broken and that this crisis has deepened in the era of large language models.
Computer scientist and Max Planck Institute for Intelligent Systems director Moritz Hardt stated that standardized benchmarking tests used in artificial intelligence are fundamentally flawed and that this situation has led to a crisis in the era of large language models.
The Role of AI Tests
Computer scientist and Max Planck Institute for Intelligent Systems director Moritz Hardt argues in his new book that standardized tests for artificial intelligence are fundamentally broken but can be fixed.
The computer scientist describes the tests as the original sin that brought machine learning into existence, noting that these metrics shaped the AI winter of the 1980s and the revolution triggered by ImageNet in the past.
Large Language Models and Leaderboards
New benchmarking tests are constantly being created and solved for large language models such as OpenAI and Anthropic.
Hardt emphasizes that the benchmarking ecosystem is broken because machine learning systems are able to manipulate leaderboards.
Reliability Crisis and Search for Solutions
Because model developers can strategically curate data, the best model does not always win anymore.
To solve this crisis, Hardt draws inspiration from athletic altitude training and suggests that models undergo a standard preparation phase prior to evaluation.