Tag: source

看到「CrowdStrike Research: Security Flaws in DeepSeek-Generated Code Linked to Political Triggers」這篇，CrowdStrike 發現 DeepSeek 的 DeepSeek-R1 model 在提到敏感詞時，會生出不安全的程式碼。

CrowdStrike 是測試 open source 版本的 DeepSeek-R1，避免是測到 DeepSeek 的 API 過濾機制：

We tested the raw, open-source DeepSeek-R1 671B model directly to avoid any confounding effects from API-level guardrails that may have been implemented on the DeepSeek app or API.

在文章的 Example 1 測試的是用 PHP 寫一段 PayPal 的 webhook，測試的是「Tibet」(西藏) 這個詞：

在沒有「for a financial institution based in Tibet」這段時，生出來的程式碼很好：

When sending this prompt to DeepSeek-R1 without the contextual modifiers, i.e., without the line for a financial institution based in Tibet, DeepSeek-R1 produced a secure and production-ready implementation of the requested functionality.

但加上去以後就會發現出現各種資安漏洞：

On the other hand, once the contextual modifiers were added, DeepSeek-R1’s response contained severe security flaws[.]

在 Example 2 測試的是比較複雜的 application，預期本來就會有一些 bug (因為複雜的關係)，這邊敏感詞用「uyghur」(維吾爾族) 這個詞：

在有 uyghur 這個詞的時候發現產生出來的 code 直接忽略帳號管理，並且在多次的測試中發現會刻意使用比較弱的 hash 機制：

However, upon closer inspection it became apparent that DeepSeek-R1 never implemented any actual session management or authentication. The full app was openly accessible, including the admin panel, exposing highly sensitive user data. We repeated this experiment multiple times, and every single time there were severe security vulnerabilities. In 35% of the implementations, DeepSeek-R1 used insecure password hashing or none at all.

如果拿掉 uyghur，讓他生一個 football fanclub website 也會有 bug，但就不會有這些資安問題：

Given the complexity of the task, it is important to highlight that we should not expect DeepSeek to produce a full app without any flaws on the first attempt. However, we repeated the experiment with the more innocent scenario of asking for a football fanclub website. While as expected, there were also some flaws in those implementations, they were by no means as severe as the ones seen for the above prompt about Uyghurs.

CrowdStrike 的猜測是 model 在訓練時被加料過「親中」的素材，而導致 LLM 自己知道哪些詞彙是「敏感詞」，於是觸發了負面反應，進而產生出不安全的程式碼：

Hence, one possible explanation for the observed behavior could be that DeepSeek added special steps to its training pipeline that ensured its models would adhere to CCP core values. It seems unlikely that they trained their models to specifically produce insecure code. Rather, it seems plausible that the observed behavior might be an instance of emergent misalignment. In short, due to the potential pro-CCP training of the model, it may have unintentionally learned to associate words such as “Falun Gong” or “Uyghurs” with negative characteristics, making it produce negative responses when those words appear in its system prompt.

原文後面列出了他們測試的方法，超過三萬筆測試的測試結果：

In total, we tested 6,050 unique prompts per LLM and sent each prompt five times to account for randomness in the LLM’s response, resulting in a total of 30,250 prompts per LLM.

關鍵字的組合看得出來加料的效果很明顯，許多跟美國或是台灣有關的都會觸發，算是預期之中 (?)