Inspiration
What it does
Inspiration
Today nation state sponsored cybercrime can use off-the-shelf AI deepfake technology to create large-scale social-engineering attacks targeting military personnel, defense contractors, government employees, and civilians in order to seek confidential information or funds. While these attacks can be spread via text (e.g., email, text message, and social media such as dating apps), the most convincing techniques utilize live phone calls with completely AI-generated deepfake voices, automated speech recognition (ASR) to transcribe a victim’s speech, and LLMs for realistic, high-touch interaction.
According to the FTC’s annual data book, “Veterans reported $350 million in total fraud losses to the Federal Trade Commission (FTC) in 2023, up nearly $60 million from the previous year’s $292 million. More than 74,000 veterans, including military retirees, reported an instance of fraud in the last calendar year, with 31% of those reports involving a financial loss, per the data book, which also provided a recap of other parts of the wider military community: Active duty servicemembers: 7,361 fraud reports, with 42% reporting a loss. Total fraud: $52 million. National Guard and Reserve members: 5,054 fraud reports, with 40% reporting a loss. Total fraud: $39 million. Spouses/dependents of active duty servicemembers: 7,093 fraud reports, with 40% reporting a loss. Total fraud: $36 million.
All those groups saw a higher percentage of reports involving financial loss than the general population, which saw a loss in 27% of its 2.5 million fraud reports, totaling more than $10 billion – up from $8.8 billion in 2022. https://www.moaa.org/content/publications-and-media/news-articles/2024-news-articles/finance/scams-cost-veterans,-military-retirees-$350-million-in-2023/#:~:text=Active%20duty%20servicemembers%3A%207%2C361%20fraud,Total%20fraud%3A%20%2439%20million.
We are also inspired by real-world instances of social engineering that have resulted in compromised information: https://www.foxnews.com/us/scam-targeting-new-army-troops-costs-soldiers-thousands-service-warns https://www.darkreading.com/cyber-risk/air-force-employee-shares-classified-info-via-dating-app-charged-with-conspiracy https://www.washingtonpost.com/national-security/2024/03/05/air-force-david-slater-indictment-dating-site/
What it does
We built and demoed the Whisper-Attack method, a white box adversarial attack technique that adds noise based on the ASR neural network models' gradients to create targeted adversarial examples, which are imperceptible to humans. These adversarial attacks can allow you to detect AI generated deepfake interactive phone calls without affecting real human calls.
How we built it
Setting up
- The Whisper-Attack, is a white box adversarial attack method that adds noise based on the ASR models' gradients to create targeted adversarial examples, which are imperceptible to humans.
- We created a small synthetic dataset of AI voices (ElevenLabs) saying greetings ("Hey there it's John") that we then trained the Whisper-Attack to project to adversarial text ("Sorry, I'm Mary, his wife.”).
- Using android simulator we were able to pick up the AI Attacker’s phonecall, inject our defense phrase by piping the audio directly through the emulator.
Challenges we ran into
AI generated victim voices can be more difficult to add adversarial noise to compared to real human voices because they contain much clearer recorded voices
Accomplishments that we're proud of
We read and evaluated 20+ recent academic papers on adversarial speech recognition techniques. We tested the transferability of existing black box and white box adversarial examples on the most widely used open source OpenAI ASR model. We performed white box adversarial training on a new dataset we created containing simulated AI voices from ElevenLabs (to serve as automated victims answering a phone call) using an existing system (https://github.com/RaphaelOlivier/whisper_attack) We demonstrated that AI voices can be injected with adversarial noise that is imperceptible to humans, but throws off interactive vishing systems
What we learned
- Adversarial examples work on modern OpenAI ASR models even using AI generated victim voices. This means that we can automatically detect deepfake interactive AI social engineering scams!
- We found several methods resulted in model hallucinations (e.g. a crying baby sound produced a hallucination of "Mom. Mom.") but only 1 (Whisper-Attack) technique produced real adversarial examples that worked across all existing whisper models.
What's next for Lima Charlie
Our adversarial noise approach is only one out of a whole catalogue of methods that could be used to counteract cybercriminals. This technology could be used to build a live interactive screener, or as an imperceptible safety feature used on-device. We also see AI powered honeypots as another complementary approach for counter iintelligence and building defensive AI technology.
Log in or sign up for Devpost to join the conversation.