A recent study by researchers at Friedrich Schiller University Jena has investigated how powerful modern AI models such as GPT-4 are in Chemistry and how they perform in comparison to human experts. Using a newly developed test method called "ChemBench"", the team led by Dr Kevin M. Jablonka was able to show that AI models are convincing in certain task areas, but also have clear weaknesses. The team reports this in the scientific journal "Nature Chemistry".

"The possibilities of artificial intelligence in Chemistry are attracting increasing interest - so we wanted to find out how good these models really are," explains Jablonka, head of the Carl Zeiss Foundation Junior Research Group "Polymers in Energy Applications" at Friedrich Schiller University Jena and the Helmholtz Institute for Polymers in Energy Applications (HIPOLE) Jena. At the centre of the study was "ChemBench", a tool developed by the researchers that was compared with the skills of chemists.

More than 2,700 tasks compared between humans and machines

To test the AI's abilities, the team at the University of Jena developed a special test procedure that uses real-life tasks encountered in modern chemistry. More than 2,700 questions from various areas of Chemistry - from organic to analytical chemistry - were integrated into the "ChemBench" tool. They cover both basic knowledge and challenging problems and are based on typical Chemistry curricula. The performance of the AI models was compared with that of 19 experienced experts working on the same tasks.

While the humans were allowed to use aids such as Google or chemical programmes for part of the study, the AI models had to manage without such external resources. "The models were therefore able to draw their knowledge exclusively from training with existing data," explains Jablonka. "We also tested two AI agents with access to external tools - but these could not keep up with the best models," adds the chemist. In addition to the accuracy of the answers, the researchers also assessed how well the AI rated its own response reliability.

AI is faster and more efficient, humans are more reflective and self-critical

The results of the study show a mixed picture, reports Jablonka: "For even very demanding textbook-type questions, some AI models proved to be more efficient than humans." However, while the chemists openly admitted in some cases that they could not answer a question with certainty, the best AI models showed an opposite tendency: they often gave answers with great confidence - even if the content was incorrect.

"Incorrect answers with high conviction can lead to problems"

"This was particularly noticeable with questions on the interpretation of chemical structures, such as the prediction of NMR spectra," says Jablonka. Here, the models seemed to provide clear answers, even if they sometimes made fundamental errors. The human experts, on the other hand, hesitated more often and questioned their own conclusions. "This discrepancy is a decisive factor for the practical applicability of AI in chemistry," Jablonka categorises, because: "A model that provides incorrect answers with high conviction can lead to problems in sensitive areas of research."

"Our research shows that AI can be an important addition to human expertise - not as a replacement, but as a valuable tool that supports the work," summarises Kevin Jablonka. "Our study thus lays the foundation for closer collaboration between AI and human expertise in Chemistry."