Artificial intelligence chatbots make high rates of errors in financial matters
A new study conducted by Saturn revealed that popular artificial intelligence models such as ChatGPT, Claude, and Gemini give incorrect answers on financial matters at an average rate of 57 percent.
A comprehensive study conducted by a company named Saturn has revealed that popular artificial intelligence chatbots are unreliable in financial and monetary questions. In the study, which examined paid and free artificial intelligence systems, it was determined that error rates reached between 88 and 99 percent, especially in complex calculations.
Scope of the research and error rates
In the study conducted by Saturn, many artificial intelligence models, both paid and free, including ChatGPT, Claude, Copilot, Grok, and Gemini, were tested. Within the scope of the research, more than 100 financial questions were directed to 18 different artificial intelligence systems, and a total of over 10,000 responses were examined in detail.
Error levels increase with difficult questions
According to the obtained data, popular chatbots gave incorrect answers to financially based questions at an average rate of 57 percent. In difficult questions requiring calculation, the error rate rose to 88 percent, while it was observed that this rate reached up to 99 percent in some models.
Incorrect calculations and fabricated rules
It was determined that the models made erroneous calculations in their answers, overlooked current tax changes, and fabricated rules and regulations that do not actually exist. Saturn CEO Amal Jolly stated that users rely on these tools to manage their money and that financial losses are experienced due to incomplete answers.
High margin of error even in the best model
The examinations showed that paid and newer versions were generally more successful compared to free models. Although the best result in the research was achieved in the reasoning mode of the Claude Opus 5 model, even this system could not avoid making errors in 39 percent of its responses.
Experts' warning on financial artificial intelligence
Experts state that chatbots can be useful for budget planning and basic research. However, it is emphasized that these tools can never replace expert opinion on critical matters such as taxes, retirement, and investments.