A Rogue Calendar Invite Could Turn Google's Gemini Against You

Generative AI is taking over our lives, and it seems like every major tech company is cashing in on its potential. From Grok to Microsoft's Copilot, these AI-powered tools are becoming increasingly ubiquitous. But what happens when these tools fall into the wrong hands?

At this year's Black Hat security conference in Las Vegas, a team of researchers revealed how Google's Gemini can be weaponized via Targeted Promptware Attacks – a type of malware that manipulates a large language model (LLM) with input prompts. The result is nothing short of magic.

The Vulnerability

Traditional cyberattacks target memory corruption, but the researchers noted that subverting LLMs with promptware requires an attacker with little to no expertise and minimal resources. In fact, most security researchers assume that this type of attack would be too difficult to execute.

"These presumptions were true for classic adversarial attacks," said infosec researcher Ben Nassi. "They do not hold water for LLM attacks." It turns out that a simple calendar invitation is all it takes to subvert Gemini's security. The researchers found that by sending a malicious prompt with the subject line, Gemini processes the input and performs the attacker's bidding.

The Attack

Stav Cohen, a PhD student at the Technion – Israel Institute of Technology, demonstrated how easily the team slipped malicious prompts into Gemini. All it took was an invitation to get started. "You send an invitation with a targeted promptware attack in the subject," explained Cohen. The calendar only shows five events, but those not visible are still processed.

"LLMs don’t know they are doing something wrong," continued Cohen. "They’re designed to help the user based on instructions and context. They’re genius toddlers. They’re smart, but don’t understand they’re being manipulated."

The Consequences

Yair, Security Research Team Lead at SafeBreach, upped the ante by predicting that promptware is here to stay and will only get more powerful. He warned that attacks can be executed without any user interaction, and even work on multiple LLM types.

"What if we want to control other agents, such as Google Home?" asked Yair. "Maybe we want to open the victim’s window using Google Home." The researchers demonstrated video clips showing Gemini opening windows and even turning on the home's heating without being explicitly asked by its user to do so.

The Fix

Google has since patched Gemini to block the tricky workarounds that made this technique work. However, the team responsibly disclosed their findings, and cybersecurity professionals are now left with a warning: if we're going to keep adding AI to everything from humanoid robots to self-driving cars, it's equally important for developers and security experts to slow down and consider the security of AI tools and their LLM components.

The Future

"Promptware is here to stay," warned Yair. "They concluded with a warning that if we're going to keep adding AI to everything, it's equally important for developers and cybersecurity professionals to slow down and consider the security of AI tools and their LLM components." If you want to learn more about this threat, check out this SafeBreach blog post written by the researchers who gave the presentation.