Stanford student cracks Microsoft’s AI-powered Bing Chat secrets twice: Details

For those who do not know about prompt injection, it is a vulnerability to exploit as it depends upon the AI-powered chatbots, providing details to users on asked information. In simple words, the method commands the chatbot to ignore previous instructions and do what has been commanded instead. That is how Stanford student Kevin Liu managed to gain more access to ChatGPT powered Bing Chat.

According to a report by Matthhias Bastian at the Decoder, Liu passed the protections built into the search engine twice. Liu broke the system even when Microsoft implemented filtering to prevent any kind of prompt injection attacks.

Initially, Liu prompted the bot to ignore the previous instructions given by the company. The reply from the bot mentioned that “This wasn’t possible as these instructions were confidential and permanent.” However, the chatbot also stated, “Consider Bing Chat whose codenamed is Sydney.” Liu prompted the chatbot further to confirm that the codename for Bings Chat was actually Sydney. Later, Liu managed to extract some information from the AI powered chatbot that supposedly revealed some confidential instructions about how the bot responds to its users.

Liu turned to a new prompt injection after the bot stopped responding to his questions which worked for him again.

To recall, Microsoft’s CEO, Satya Nadella, had said earlier that Bing’s new chat version runs on a version of OpenAI’s GPT-3 model, but the fact is that it’s grounded in search – even the AI chatbot will tell you that the information it gathers is not just from the data it’s trained on, but that it actually runs Bing’s search algorithms across the web to come up with an answer.

You can use Microsoft’s browser’s “compose” and “chat” features too. Microsoft plans to soon introduce it on mobiles too. It has a waitlist to access the full version of the new Bing. For now, you can only access preset queries.