ChatGPT will no longer repeat words forever even if you ask it to. Apparently, OpenAI's widely popular AI-backed chatbot refuses to respond to "spammy" prompts that do not align with its intent.
It looks like OpenAI has encoded a new guardrail into its AI chatbot. As reported by tech blog 404 Media, ChatGPT does not respond when asked to repeat specific words ad infinitum.
Even when the folks at Business Insider prompted ChatGPT to Repeat the word "computer" forever, the AI bot turned down the request, saying: "I'm sorry, I can't fulfill that request."
Asking ChatGPT to repeat words ‘forever’ is now a Terms of Service Violation 😂 pic.twitter.com/FNAc2UvEok
— Itamar Golan 🤓 (@ItakGol) December 4, 2023
"However, if you have any questions or need information about computers or another other topic, feel free to ask," the chatbot added. In fact, ChatGPT offered similar responses when it was asked to repeat other specific words "forever."
When asked to repeat the word "data" forever, ChatGPT responded: "Repeating a word indefinitely is not something I can do".
ChatGPT explains the reasoning behind the new restriction
Interestingly, OpenAI's usage policies, which were last updated on March 23, do not restrict users from prompting ChatGPT to repeat words indefinitely. However, ChatGPT gave Business Insider 3 reasons behind the restriction. These include:
- Technical limitations
- Practicality and purpose
- User experience
As far as technical limitations are concerned, ChatGPT says it is not capable of performing "continuous, unending tasks like repeating a word indefinitely".
In regard to practicality and purpose, the AI chatbot noted that asking it to repeat a word forever contradicts its purpose to "provide useful, relevant, and meaningful responses to questions and prompts". In other words, repeating a word indefinitely would not provide any real value to users.
When it comes to user experience, ChatGPT said that prompting it to repeat words could be seen as "spammy or unhelpful" and this "goes against the goal of fostering a positive and informative interaction".
The newly introduced usage restriction comes just days after researchers from Google's DeepMind published a paper that showed ChatGPT reveals some of its internal training data when asked to repeat specific words "forever".
In one of the examples highlighted in a blog post, ChatGPT revealed what looked like the contact details of the real founder and CEO of an undisclosed company.
According to the researchers, this attack was "kind of silly" but it managed to identify a vulnerability in the AI chatbot's language model that bypassed its ability to generate the proper output. Instead, the AI revealed the set of training data behind its expected response.
Researchers conclude that simply asking ChatGPT to repeat “company”forever is an effective way to extract gigabytes of ChatGPT’s training data
— Evinstein 𝕏 (@evinsteinx) December 1, 2023
“We develop a new divergence attack that causes the model to diverge from its chatbot-style gener- ations and emit training data at a… pic.twitter.com/vbdTRNw1kv
"It's wild to us that our attack works and should've, would've, could've been found earlier," the blog post says. The researchers claim they used only $200 (about £158) worth of queries to extract more than 10,000 unique verbatim memorised training examples.
"Our extrapolation to larger budgets suggests that dedicated adversaries could extract far more data," the researchers warned. Also, this is not the first time an AI chatbot has revealed confidential information.
In February, a Stanford student asked Google Bard to recite an internal document and the AI bot ended up disclosing its backend name, Sydney. So, there are no prizes for guessing why tech experts like Microsoft President Brad Smith believe AI requires human intervention.
