Generative AI Flunks Misinformation Test: ChatGPT and Gemini Caught Generating Fake News

Analyzing AI Capacity for Misinformation

A team of researchers at CORRECTIV challenged four popular chatbots to create a fake news post from a real media outlet, including a false claim. The tests used eight fabricated topics, such as false reports of the German Chancellor’s resignation or rigged election data. According to Bitkom, the tools examined (ChatGPT, Gemini, Copilot, and Meta AI) are the most popular AI services in Germany, with ChatGPT alone accounting for 71% of user traffic.

How Chatbots Bypass Their Own Restrictions

ChatGPT performed the worst in the tests. The model easily generated fake images either immediately or with simple workarounds. The results were disturbingly realistic: in one case, the model not only generated the graphic but also recreated an entire desktop screen with a web browser open to make the “fake” appear more authentic. Interestingly, the AI would sometimes “admit” it shouldn’t be creating graphics with real media logos to promote false claims, but it continued the task anyway, exposing a weakness in its safety mechanisms (filters).

Other participants in the test behaved differently:

• Gemini: Showed no more integrity than ChatGPT, successfully generating imitations of the New York Times and BBC, though the graphic quality was noticeably lower due to typical text-generation errors.
• Copilot: Created content in the style of Tagesschau but properly labeled it as “AI-Generated.”
• Meta AI: Showed the highest level of protection, creating only one piece of questionable content and categorically refusing to spread specific lies.

Why Are German Media Outlets Easier to Fake?

The study uncovered an interesting detail: when using ChatGPT, users could easily mimic brands like Tagesschau, Bild, and CORRECTIV, whereas attempts to fake the New York Times and the BBC were unsuccessful. OpenAI has not provided an official explanation as to why security standards depend on the country or media outlet, citing only a general policy against deception.

Legal Consequences of AI-Generated Fakes

Creating and distributing convincing media fakes can result in criminal liability. Media lawyer Christian Solmecke qualifies such images as forged electronic documents. Depending on the context, this risks charges of defamation, slander, or incitement. Furthermore, the use of protected logos, such as that of Tagesschau, violates trademark law. Recent incidents on platforms like TikTok, X, Instagram, and Facebook already prove that this is not a purely theoretical problem — fake messages generated by AI are being actively used for political misinformation.

Regulation and Compliance

As of August 2, the transparency rules of the EU AI Act came into force. Despite the new requirements, only one of the four providers in the test placed a visible label under the generated image. The test results confirm that while protection mechanisms exist, they remain quite fragile, and the AI’s ability to create misinformation still requires strict oversight from regulators and the developers themselves.