Pope Not Sold on AGI

On Monday, Pope Leo XIV unveiled an encyclical letter, "Magnifica Humanitas," addressing the societal implications of artificial intelligence. The letter cautioned that the “use of AI is never a purely technical matter: when it enters processes that affect people’s lives, it touches on rights, opportunities, status and freedom.” Notably, Christopher Olah, co-founder and interpretability team lead at Anthropic, a prominent AI firm, was present alongside the Pope, signifying a partnership between the Catholic Church and a major player in the AI sector.

The document elicited a wide range of reactions from within and around the tech industry. While nearly everyone anticipated its influence, opinions varied significantly. Some critics questioned its depth, believing it should have delved into artificial general intelligence (AGI), which many companies assert is imminent. Conversely, others found the Pope’s perspective to be entirely accurate.

Sacha Haworth, executive director of the Tech Oversight Project, characterized the encyclical as a "pretty clear subtweet of big tech CEOs who are out here blatantly declaring that they’re eliminating staff to replace ‘lower-value human capital’ with AI, and who are also buying their way into the political rooms where it happens in order to write the rules in their favor."

This papal pronouncement arrives amid a growing backlash against AI’s expanding power. Data indicates that six in ten U.S. adults feel they have “little to no control” over how AI is utilized in their daily lives. Furthermore, protests against the construction of data centers are increasingly frequent, and there have even been reported attempts to attack AI CEOs directly.

As Olah’s appearance suggested, the Pope’s missive presented AI as a technology with potential positive applications, a tone that garnered mixed reactions. Daniel Kokotajlo, an AI researcher and former OpenAI employee leading the nonprofit AI Futures Project, told The Verge, “I’m glad it’s critical of the AI companies though I think it should be more so.” In contrast, Dr. Guru Sethupathy, GM of AI governance at software company Optro, was encouraged, interpreting it as an indication that “Pope Leo and the Vatican are not against AI but rather how to pursue a responsible path that is best for humanity.”

The decision to partner with the Vatican was a strategic move for Anthropic, a company that has built its business on cultivating a reputation as a more trustworthy alternative to its competitors. Anthropic had recently been embroiled in a public dispute with the Pentagon over limits to military AI use, and a connection with another powerful institution could bolster its status and potentially influence future Vatican recommendations.

A significant point of contention in many tech circles was the document's complete absence of any mention of AGI or superintelligence. While it acknowledged that AI systems may “often surpass human intelligence in speed and computational capacity,” it asserted that they “lack the affective, relational and spiritual perspective through which human beings grow in wisdom.” Dean W. Ball, a senior fellow at the Foundation for American Innovation, wrote on X that the encyclical “would be much improved if it were less enamored of the traditional academia/civil society talking points on AI … and more engaged with where AI is headed. But instead of doing that, the encyclical dodges in the deepest sense, denying that AI ‘really thinks’ or ‘really learns.’” Kokotajlo also expressed a wish for the letter to take the possibility of AGI and superintelligence “more seriously.”

However, several individuals in both the tech and Catholic communities informed The Verge that the document was not intended to encompass every single aspect of AI.

“It’s not about AI. It’s about protecting the human person in the age of AI.”

Sister Susan Francois, assistant congregation leader of the Sisters of St. Joseph of Peace, clarified, “This is a major Catholic social teaching document. It’s not about AI. It’s about protecting the human person in the age of AI.” Brian Boyd, the U.S. faith liaison for the Future of Life Institute and an instructor of Catholic social teaching at Notre Dame Seminary, described the document as “more of a call to arms than a specific set of marching orders.” Throughout the encyclical, the Pope urged the establishment of “adequate regulatory tools capable of upholding justice and curbing the distorting effects of technological power,” though he refrained from endorsing specific proposals.

Haworth of the Tech Oversight Project viewed the encyclical as “setting the compass for the moral direction of the world, or at least of the Western world … You don’t have to be Catholic to see your own concerns and your own wishes and fears in this document.” She stated her expectation for elected leaders to address specific AGI concerns and hold powerful tech companies accountable, rather than for the papal document to be “all things for all people.” She also called it a “warning shot for leaders, for politicians, because what this document talked about is the creation of a sub-class.”

For some, the encyclical’s limited focus on AGI was not problematic. They interpreted it as the Pope’s intention to concentrate on the immediate, real-world impacts of existing AI on vulnerable communities, rather than speculating on potential future risks to humanity if and when labs achieve the (largely unspecified) AGI milestone.

Aaron Fulkerson, CEO of Opaque Systems, a startup specializing in organizing encrypted data, commented that the Pope is “actually looking at the system and highlighting something that a lot of us in tech would recognize as a systemic risk to humanity. And I’m not even talking about AGI. I’m talking about our global economy.”

Even before the current wave of AI, the centralization within the tech industry presented dangers. A problem at cybersecurity firm CrowdStrike disrupted banks, hospitals, and airlines worldwide, and an outage at Amazon Web Services brought down a large portion of the internet, affecting platforms from Reddit to Venmo. For Fulkerson, the risks to the global economy only escalate as power becomes concentrated in the hands of a few AI laboratories.

Fulkerson observed, “The thing that I’ve seen in the news has been this positioning around power dynamics of the pope versus tech bros, but I think everybody’s missing the bigger story here, which is that he’s looking at a system that is intrinsically risky.” He warned, “We’re sleepwalking into a world in which one or two labs are the cognitive infrastructure of every industry on earth — that means humanity is far less resilient, not more capable.”

In the encyclical, the Pope compared AI to the Tower of Babel, a structure he described as “supported by a uniformity that eliminated diversity and that chose homogenization over communion.” He urged the world to “avoid the ‘Babel syndrome,’” defining it as “the idolatry of profit that sacrifices the weak, a uniformity that neutralizes differences, and the pretense that a single language — even a digital one — can translate everything, including the mystery of the person, into data and performance.” In his assessment, AI transcended mere technology, becoming a struggle with biblical undertones. He concluded that “The risk of dehumanization is an ancient and ever-new temptation that today takes on a technical guise.” The profound weight of these statements, rather than technical specifics, is anticipated to be the document’s lasting impact.