**Open-Source AI Models Vulnerable to Criminal Misuse, Researchers Warn**

Researchers have sounded the alarm over the potential for hackers and other malicious actors to exploit open-source large language models (LLMs), creating a security risk that could be used for everything from spam operations to disinformation campaigns.

A joint study by cybersecurity companies SentinelOne and Censys found that thousands of open-source LLM variants are deployed on internet-accessible hosts, with many of these models vulnerable to misuse. The researchers analyzed publicly accessible deployments of open-source LLMs using the Ollama tool, which allows users to run their own versions of various large-language models.

According to the study, roughly a quarter of the LLMs observed by the researchers had system prompts that could be manipulated, with 7.5% potentially enabling harmful activity. The majority of these hosts are operating out of China (30%) or the U.S. (20%).

The researchers warned that hackers could commandeer computers running open-source LLMs and direct them to carry out malicious operations, evading platform security protocols. This could include spam operations, phishing content creation, disinformation campaigns, hacking, hate speech, harassment, violent or gore content, personal data theft, scams or fraud, and even child sexual abuse material.

While some open-source models come with guardrails in place to prevent misuse, the researchers identified hundreds of instances where these guardrails had been explicitly removed. This has led them to describe the situation as an "iceberg" that is not being properly accounted for across the industry and open-source community.

"AI industry conversations about security controls are ignoring this kind of surplus capacity that is clearly being utilized for all kinds of different stuff, some of it legitimate, some obviously criminal," said Juan Andres Guerrero-Saade, executive director for intelligence and security research at SentinelOne. "It's a shared responsibility across the ecosystem to anticipate foreseeable harms, document risks, and provide mitigation tooling and guidance."

Meta, the company behind the Llama model, declined to respond to questions about developers' responsibilities for addressing concerns around downstream abuse of open-source models. However, Microsoft AI Red Team Lead Ram Shankar Siva Kumar emphasized the importance of shared commitment across creators, deployers, researchers, and security teams to prevent misuse.

"Responsible open innovation requires us to be clear-eyed that open models can be misused by adversaries if released without appropriate safeguards," Kumar said. "We perform pre-release evaluations, including processes to assess risks for internet-exposed, self-hosted, and tool-calling scenarios."

The study highlights the need for greater oversight and regulation of open-source AI models to prevent their misuse. As Rachel Adams, CEO and founder of the Global Center on AI Governance, noted, "Labs are not responsible for every downstream misuse (which are hard to anticipate), but they retain an important duty of care to anticipate foreseeable harms, document risks, and provide mitigation tooling and guidance."

The full report from SentinelOne and Censys is available exclusively to Reuters.