Can ChatGPT Conquer Accounting? Not Yet.

Can ChatGPT Conquer Accounting? Not Yet.
Gregory Dawson, clinical professor of accountancy, discovered this with his co-authors in their paper, “The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?” Published in the Issues in Accounting Education journal, it examined how artificial intelligence performed when confronted with typical questions for aspiring accountants.
Specifically, Dawson and his colleagues found that students scored an average of 76.7%, while ChatGPT scored 47.4% when only completely correct answers were counted (ChatGPT’s average improved to 56.5% when given partial credit for answers).
ChatGPT’s performance varied by topic area. It performed better in accounting information systems and auditing, which are less mathematically based, than in tax, financial, or managerial accounting. Its performance also varied depending on the type of question. ChatGPT performed better on true/false and multiple-choice questions than on problem-solving and short-answer questions.
“It does well with general-theory questions but doesn’t know how to apply them. It was not capable of ‘thinking,’ ” Dawson says. “Questions that were very fact-based, like, ‘What does IRS Regulation 2503 say?’ it knew that very well. And, of course, we know that Gen AI (generative AI) is terrible at math because we’ve never trained it for that.”


Clinical Professor of Accountancy
Besides addressing a new technology, the researchers used an uncommon methodology—crowdsourcing. The lead author recruited more than 300 other accounting educators from 186 schools in 14 countries, asking them to cull questions from their classroom work, enter them into ChatGPT, and compare its answers with their students’ answers. The group developed 28,000 questions and provided details, including the question topics, students’ average scores, and grade levels. Crowdsourcing allowed the group to quickly gather data shortly after ChatGPT came out on Nov. 30, 2022, and publish its paper in Issues in Accounting Education within a few months.
Dawson says the research broke new ground in that it was the first to study ChatGPT and artificial intelligence in a setting like accounting, where the rules are very well known. As new versions of ChatGPT come out and “learn” from more training, the researchers say the performance gap between artificial intelligence tools and students will likely close.
Dawson has several takeaways from the research. First, artificial intelligence is reshaping every discipline in business schools, and professors must train students to use AI ethically and appropriately. Second, students who use AI will need to memorize fewer facts but must become better at understanding and interpreting information.
“We in education have to recognize this is the future,” Dawson says, “and position ourselves to create students capable of living in this brand-new world.”