New York City schools ban access to ChatGPT over fears of cheating and misinformation

The New York City Department of Education has blocked access to ChatGPT on its networks and devices over fears the AI tool will harm students’ education.

A spokesperson for the department, Jenna Lyle, told Chalkbeat New York – the education-focused news site that first reported the story — that the ban was due to potential “negative impacts on student learning, and concerns regarding the safety and accuracy of content.”

“While the tool may be able to provide quick and easy answers to questions, it does not build critical-thinking and problem-solving skills, which are essential for academic and lifelong success,” said Lyle.

ChatGPT was released last November by San Francisco-based AI company OpenAI, and has sparked a huge debate over the potential effects of machine learning on everything from education to misinformation and the world of work.

ChatGPT’s most revolutionary quality is its open-access user interface and ability to answer questions in human-like language. The tool is not only able to discuss a wide range of topics using data scraped from the internet, but can perform a number of linguistic tricks, like writing in different styles and genres — from medieval quatrains to sitcom scripts.

However, ChatGPT also suffers from failures common to all of the most recent AI language systems (known as large language models, or LLMs). Because it’s trained on data scraped from the internet, it often repeats and amplifies prejudices like sexism and racism in its answers. The system is also prone to simply making up information, from historical dates to scientific laws, and presenting it as accurate fact.

It’s this combination of traits — fluency, accessibility, and misinformation — that makes the tool particularly worrisome for educators. Many teachers have said software like ChatGPT essentially makes it impossible to test students’ ability to write essays at home. Why bother writing an assignment if ChatGPT can do the job for you in seconds? And although tools to detect AI-generated writing already exist, it’s unclear how accurate these systems will be, or if students will be able to outwit them with simple alterations to AI-generated text.

Others, though, argue that the education system will simply have to adapt to the appearance of this technology — just as it has adapted to earlier disruptive technologies like Google Search and Wikipedia. New testing standards could focus more on in-person examinations, for example, or a teacher could ask students to interrogate the output of AI systems (just as they are expected to interrogate sources of information found online).

But such adaptations will take time, and it’s likely that other education systems will ban AI-generated writing in the near future as well. Already some online platforms — like coding Q&A site Stack Overflow — have banned ChatGPT overs fear the tool will pollute the accuracy of their sites.