ChatGPT took an MBA exam. Here’s how it did

A professor at Wharton, University of Pennsylvania has given a theoretical B-grade pass to ChatGPT after marking the answers it generated for a final exam in a typical MBA course. 

Christian Terwiesch, a professor of operations management at Wharton, said ChatGPT does an “amazing job at basic operations management and process analysis questions”. 

“Not only are the answers correct, but the explanations are excellent,” he writes in a new white paper. 

Also: How to get started using ChatGPT

But he added the chatbot makes “surprising mistakes in relatively simple calculations at the level of 6th grade Math”, and that it currently can’t handle more advanced process analysis questions. On the other hand, the current version of ChatGPT can modify its answers in response to human hints to arrive at the correct solution.

“Considering this performance, Chat GPT3 would have received a B to B- grade on the exam,” writes Terwiesch. 

Several of Terwiesch’s questions aimed to test whether ChatGPT could identify a bottleneck in a processing operation with multiple machines that have different throughput capacities.  

ChatGPT, however, “made a significant mistake of a massive magnitude” when calculating with medium level arithmetic. 

Interestingly, ChatGPT initially incorrectly answered one of Terwiesch’s question regarding queuing analysis. He prompted it with a hint for a better answer and ChatGPT improved its answer. The following day, the professor asked it the same initial queuing question without the hint — and ChatGPT answered correctly on its first attempt.    

“It either is capable of learning from past feedback or I just got lucky,” he notes, adding that there seemed to be some randomness in the quality of its answers. 

Terwiesch also found ChatGPT was able to formulate clever and humorous questions that he could use in future exams. However, the chatbot also introduced subtle flaws in some questions that made them impossible to answer. 

Also: ChatGPT ‘lacked depth and insight,’ say prestigious science journal editors

Terwiesch warns others to be mindful of ChatGPT’s capabilities and limitations. He says he “feels in love” with ChatGPT after reading its answer to his first question, but warns it “made major mistakes in some fairly simple situations”.  

“We are still far from an A+ for complex problems and we still need a human in the loop,” he writes. 

“In my view of education, an elementary school student still needs to learn that 7 x 7 = 49 and that the capital of Pennsylvania is Harrisburg, even though calculators have been widely used for over 50 years and students can use Google Wikipedia to find answers for most factual questions. It is the nature of foundational skills that they are required to comprehend more advanced topics.”

Terwiesch agrees that educators should be concerned that K-12 students might use ChatGPT to cheat on assignments and exams. For example, New York City Department of Education recently banned the chatbot because reliable tests are important in teaching, while skill certification shouldn’t be compromised because of new technology. A student using ChatGPT is like being able to call a friend with “average academic competence” to complete the exam for them, he notes.   

But he also argues ChatGPT and similar technologies have the ability to play the role of a “smart consultant” — one who produces elegant but oftentimes wrong answers, which he believes is the “perfect training ground” for developing skills among MBA students who need to critically evaluate suggested alternatives.