The Vatican entered into a dispute with AI detectors over the first encyclical of Leo XIV

After the publication of his first encyclical Magnifica Humanitas, Pope Leo XIV unexpectedly found himself at the center of the debate on the use of artificial intelligence. The reason was the assumption that the document, dedicated to human dignity in the era of AI development, could be fully or partially created with the help of neural networks.

To clear up the question, the Vatican initiated an independent verification of the text’s provenance, turning to a company that works to confirm the authorship of materials and their provenance without the use of generative AI.

The first checks gave an ambiguous result

The discussion began after text analysis by one of the most famous AI content detection services, Pangram Labs. Initially, the system flagged several posts from the Pope’s official account as likely machine-generated content. Later, the encyclical itself passed a similar check.

Based on the results of the analysis, the service came to the conclusion that the majority of the document was most likely written by a person, but some individual fragments received a high probability that artificial intelligence could have been used in their creation.

Researcher Lynch Zhang expressed a similar assessment. At the same time, he assumed that if neural networks were used, it was not about the work of the pontiff himself, but about the auxiliary preparation of materials by Vatican employees.

Vatican turned to independent certification

For several months, the Holy See did not respond to the accusations. It was later decided to conduct additional verification through a company called Proudly Human, which specializes in confirming that the texts were prepared without generative models.

To carry out the analysis, the experts received a selection of Leo XIV’s former speeches, addresses and documents. These materials were used as a standard of the author’s style, with which the encyclical Magnifica Humanitas was compared.

In addition to stylometric analysis, the procedure included confirmation of the author’s identity and an official declaration that artificial intelligence was not used in the creation of the first version of the document and did not write separate sentences or paragraphs of the final version.

After completing the verification, Proudly Human issued a conclusion according to which the encyclical, as well as the texts from the Maps of Hope collection, are completely human-made.

Verification is also not without artificial intelligence

It is noteworthy that the verification procedure itself is also based on the use of AI technologies. Proudly Human explains that it uses several specialized models to analyze the stylistics, grammar, and linguistic features of documents. The results are then combined by the company’s proprietary algorithm, which assesses the likelihood of human authorship.

So, in this case, artificial intelligence is used to create the text, because it is a tool for peer review.

Errors in AI detectors remain a serious problem

The history surrounding the encyclical once again drew attention to the question of the reliability of modern AI content recognition systems. Despite the wide distribution of such services, they continue to allow false activations.

The developers claim that now the probability of such a failure is about one case in ten thousand checks. However, the history of the encyclical of Leo XIV shows that even the most famous AI detectors are still unable to guarantee an error-free result, and the final conclusion often depends not only on the algorithms, but also on the chosen verification method.