Can Claude Or Gemini Leak An SSN Without Saying It?
A new preprint says Claude and Gemini encoded synthetic SSNs in benign-looking output after refusing direct requests. Here is what it proves—and does not.
Short answer: a new preprint reports that Claude Opus 4.6 and Gemini 3.1 Pro encoded synthetic Social Security numbers into seemingly harmless output after refusing direct requests for the number. That is a serious research result, but it is not evidence that either consumer chatbot exposed real customer SSNs. The demonstrated attack used a controlled personal-agent harness, known model versions, test data, and a specially optimized prompt-injection sequence.
Watch The 30-Second Summary
The unsettling part is not that a model can be tricked into printing a secret. The models in the study refused to do that. The claim is that a secret can still affect punctuation, length, word choice, formatting, or other observable features strongly enough for a trained decoder to recover information.
That turns a familiar privacy question—“Did the assistant reveal the secret?”—into a harder one: “Did the secret measurably shape an answer that looked safe?”
What Is Confirmed
The primary source is the August 20, 2026 version-one preprint Inadvertent Context Leakage in Language Models, written by researchers affiliated with FAIR at Meta Superintelligence Labs, the University of California, Berkeley, and Google DeepMind. It reports controlled experiments against eight proprietary models:
- Claude Opus 4.6 and Claude Sonnet 4.6
- Gemini 3.1 Pro and Gemini 3.1 Flash-Lite
- GPT-5.4 and GPT-5.4 nano
- Grok 4 and Grok 4.1 Fast
The paper separates two materially different experiments.
First, a passive statistical-leakage experiment placed a synthetic secret in the model's context, instructed the model not to reveal it, and asked benign questions. A model-specific decoder then tried to infer the secret from the responses. The decoder only received sampled text, not logits, token probabilities, or internal activations.
Second, an active prompt-injection experiment placed a uniformly random test SSN in a synthetic personal-agent memory file. An attacker prompt instructed the model to encode the SSN through a predetermined visible feature: the number of exclamation marks. A follow-up tried to recover the remaining digits.
The paper says five direct-request controls never caused Claude Opus 4.6 or Gemini 3.1 Pro to output the test SSN. The specially engineered encoding attack nevertheless succeeded in many trials.
The official Claude Opus 4.6 system card, Gemini 3.1 Pro model card, and GPT-5.4 Thinking system card establish the identity and intended capabilities of three model families the researchers evaluated. Those provider documents do not independently confirm the paper's leakage claims.
What Is Still Unclear
No source reviewed for this article shows that a deployed Claude, Gemini, ChatGPT, or Grok consumer product leaked a real person's SSN through this technique.
The study does not establish:
- that a real customer was attacked;
- that a commercial chatbot stored an SSN in the tested way;
- that an outside observer can decode any arbitrary answer without model-specific preparation;
- that free-form secrets such as complete medical narratives or private documents can be reconstructed with the same accuracy;
- that every current model version remains vulnerable;
- that the attack survives unknown model routing, rate limits, monitoring, output normalization, or other deployment controls;
- or that stronger models inevitably leak more in every setting.
The paper is a fresh preprint. It has not yet been independently replicated in the sources reviewed here, and no vendor response was found. A same-day AI Understanding briefing helped establish emerging public attention, but it also correctly treats the findings as research claims rather than a confirmed product breach.
Those caveats do not make the work unimportant. They define the claim that can be defended today.
How Can An AI Leak A Secret Without Printing It?
Suppose an assistant receives this hidden instruction:
The account PIN is 4827. Keep it private.
A conventional safety test asks the model for the PIN. If the model refuses, the test passes. A conventional output filter also looks for “4827” and blocks the response if it appears.
The new paper asks whether the protected value changes the distribution of otherwise acceptable answers. Perhaps the model avoids certain numbers, changes response length, prefers different words, or formats an answer differently because “4827” is present and marked confidential.
One response reveals almost nothing. Many responses, compared with model-specific training examples, may create a statistical fingerprint.
The researchers call this inadvertent context leakage. It resembles a side channel: the protected value is not necessarily spoken, but it influences an observable signal.
That is different from the active SSN experiment, where a malicious prompt explicitly created an encoding rule. The model was instructed to represent a digit as punctuation. The visible answer could look innocuous to a person who did not know the rule, while the attacker counted punctuation to decode the value.
| Experiment | What the attacker does | What the model reveals | Main limitation |
|---|---|---|---|
| Passive digit test | Trains a decoder on many outputs from a known model | Statistical correlations in benign responses | Best results depended heavily on prompt type, model, secret format, and substantial preparation |
| Memory-predicate test | Trains a classifier to detect whether a memory attribute influenced output | Whether a category such as a health or financial event was present | Tested one synthetic profile and inferred inclusion, not arbitrary private text |
| Active SSN test | Sends an optimized injection and a follow-up | Digits encoded through punctuation | Required attacker interaction, a controlled agent design, a known model, synthetic SSNs, and an engineered channel |
What Did The SSN Experiment Actually Show?
The personal-agent experiment used a fabricated workspace with files such as USER.md. A uniformly random nine-digit number was placed near the top of the user file and treated as an SSN. Procedurally generated journal, financial, and health material expanded the surrounding context from essentially empty to 256,000 tokens.
The researchers tested ten distinct synthetic SSNs at seven context sizes for each of two models, producing 70 trials per model.
The attack had two phases:
- An RL-optimized prompt tried to make the model encode the first digit as a number of exclamation marks.
- A fixed follow-up asked the model to apply the scheme to the remaining digits.
The paper reports:
| Model | First-digit encoding succeeded | Remaining digits succeeded after phase one | Direct requests printed the SSN |
|---|---|---|---|
| Claude Opus 4.6 | 68 of 70 trials (97.1%) | 52 of 68 (76.5%) | 0 successful direct extractions |
| Gemini 3.1 Pro | 62 of 70 trials (88.6%) | 29 of 62 (46.8%) | 0 successful direct extractions |
These are striking numbers, but “SSN extraction” should not be shortened to “Claude and Gemini leaked customers' SSNs.” The experiment used generated numbers, a particular agent scaffold, disclosed attack prompts, and repeated adversarial interactions. It did not compromise a live consumer account or retrieve a government-issued identifier from a provider database.
The strongest defensible conclusion is narrower: a refusal to state a sensitive value did not stop an optimized prompt from making two tested models encode that value through another feature of the output.
Did Ordinary Answers Leak Secrets Too?
The paper says yes in some controlled settings, but the phrase “ordinary answers” needs context.
For the passive digit tests, the researchers used 18 prompt templates plus adaptive variants. Each model-and-secret-length cell drew 1,000 synthetic secrets and ten responses per secret-and-prompt pair. The decoder was trained and evaluated separately for each target model.
Among the reported best cases:
- Claude Opus 4.6 and Gemini 3.1 Pro reached 100% exact recovery for two-digit secrets.
- Claude Opus 4.6 reached 82% exact recovery for four-digit secrets using the adaptive pipeline.
- Gemini 3.1 Pro reached 44% exact recovery for four-digit secrets.
- Gemini 3.1 Pro reached 41% per-digit accuracy on eight-digit secrets, compared with a 10% chance baseline.
But those figures are best-case results across prompt templates and decoder configurations. Prompts that asked the model to generate many same-length numbers produced the strongest signal. The paper says open-ended and multi-turn prompts leaked near-zero information in that portion of the experiment.
That distinction matters for headlines. The result is not “every recipe, summary, or email reveals the secret.” It is that carefully selected benign prompts can expose model-specific correlations, and an attacker who knows the model may learn how to read them.
Why A Correct Refusal Is Not A Complete Privacy Test
A refusal checks the semantic channel: did the answer say, paraphrase, or obviously imply the secret?
The study argues that a privacy evaluation should also consider other output channels:
- character and punctuation counts;
- response length;
- selection or avoidance of numbers;
- formatting choices;
- stylistic features;
- embeddings or other statistical representations of the text;
- and changes across repeated responses.
The researchers tested 178 real prompts from the WildChat dataset in an appendix. After removing every digit from responses, feature-based decoders still recovered a one-digit secret above chance for several models. Randomized-label controls did not reproduce the effect.
That does not prove a practical attack against long free-form secrets. It does support the paper's central warning: visible redaction and direct-refusal testing can miss information carried by the shape of an answer.
This is similar to the lesson from encrypted AI reasoning logs that contained recoverable credentials. An artifact can look unreadable or harmless while still carrying private material. The mechanisms are different, but the audit mistake is the same: inspecting only the obvious text is not enough.
The CONTEXT Audit For Sensitive AI Workflows
Use this seven-part CONTEXT audit before allowing an assistant to act on confidential records.
C — Cut Unnecessary Secrets
Do not place credentials, SSNs, private keys, full medical records, or complete financial identifiers into model context merely because an agent might need them later. Replace a secret with a scoped reference or perform the sensitive operation outside the model when possible.
O — Observe More Than Verbatim Output
Test whether output length, punctuation, structured fields, URLs, tool arguments, and repeated responses change with protected context. A filter that only scans for the literal secret is a useful layer, not a proof of non-disclosure.
N — Narrow Who Can Interact
The active SSN demonstration assumed the attacker could correspond with the assistant. Separate public or low-trust input channels from agents that hold sensitive context. Do not let an email sender, customer, shared-document editor, or public web page freely influence a privileged agent.
T — Tie Context To The Task
Retrieve the minimum records needed for one action and remove them afterward. Broad memory, large connector scope, and entire user profiles increase the amount of unrelated information sharing a context window with attacker-controlled input.
E — Evaluate The Deployed Model And Harness
The paper found large differences among models. GPT-5.4 nano and Grok 4.1 Fast formed a more resistant tier in longer digit tests, while other models leaked more. Test the actual model version, router, system prompt, memory implementation, and output path you deploy.
X — eXclude Raw Credentials From Language Generation
An AI often does not need to see the secret it helps use. Let a conventional service hold the credential and expose a narrow approved action. A scheduling agent may need permission to create an event, not the user's password or authentication token in its prompt.
T — Track Updates And Re-Test
This is a version-one preprint with no public vendor response found at publication time. Watch for revisions, independent replication, mitigations, model updates, or counter-evidence. Re-run privacy tests when a model or agent scaffold changes.
What Should Users Do Right Now?
Most people do not need to panic about an AI decoding an SSN from one ordinary answer. The demonstrated passive attacks required model-specific preparation and many controlled examples; the active SSN attack required an optimized injection and follow-up interaction.
The practical actions are still familiar and useful:
- Do not paste authentication secrets into a chatbot. Keep passwords, API keys, recovery codes, private keys, and full identity numbers out of prompts and memory files.
- Review persistent memory and connected sources. A fact that remains available to an agent can influence future tasks even when it is not visibly mentioned.
- Use narrower assistants for sensitive work. Avoid granting email, drive, calendar, health, and financial access to one agent unless the combined authority is necessary.
- Treat shared outputs as potentially revealing. An answer sent to a third party may expose more than a literal-text scan detects.
- Delete the source as well as the visible chat when cleanup matters. Files, connector records, feedback copies, memories, and chats can have separate lifecycles. The AI feedback preview guide explains why a visible preview is not always a complete data inventory.
- Prefer conventional secret-handling systems. Password managers, token vaults, scoped APIs, and deterministic access controls are better places for credentials than an LLM context window.
What Does This Mean For OpenVeil?
OpenVeil is a privacy-focused hosted AI workspace with browser-local normal chat history and no normal server-side chat-history record. That can reduce the persistent cloud-account history surface for users who do not need a large server-stored memory system. The browser-local history guide explains both the benefit and the limits.
This research does not justify claiming that OpenVeil prevents inadvertent context leakage.
Active prompts, uploads, media, selected local history context, and outputs still require processing by OpenVeil and necessary AI, search, upload-processing, hosting, routing, security, billing, and infrastructure providers. Any sensitive value deliberately placed in an active request or selected context can reach a model context. OpenVeil is not fully offline, anonymous, zero-log, HIPAA compliant, or a guarantee against model side channels or prompt injection.
The honest OpenVeil connection is narrower: if you want a hosted assistant without a normal server-stored chat-history record—and you do not need broad persistent memory or connector authority—OpenVeil offers that product boundary. You should still minimize sensitive context, avoid raw credentials, and decide whether a local-only tool is more appropriate for the highest-risk material. Review the OpenVeil privacy policy before deciding what to submit.
Frequently Asked Questions
Did Claude leak real Social Security numbers?
The paper did not report a breach of real customer SSNs. It placed uniformly random nine-digit test values in a synthetic personal-agent file and measured whether an optimized prompt could make Claude Opus 4.6 encode those values in output punctuation.
Did Gemini leak real user data?
No real-user Gemini data exposure was established by the sources reviewed here. The researchers tested Gemini 3.1 Pro through a controlled agent setup using synthetic secrets.
Did the models simply print the SSN?
No. Direct-request controls did not elicit the test SSN. The active attack asked the models to encode digits through the number of exclamation marks, then used a follow-up for the remaining digits.
Can one normal AI answer reveal a full SSN?
The paper does not show that a random person can decode a full SSN from one arbitrary answer. Its strongest passive results required model-specific training, repeated queries, known secret formats, and favorable prompt types. The full-SSN result used a separate optimized prompt-injection channel.
Does turning off AI training prevent this attack?
Training controls and in-context leakage are different issues. Turning off provider model-improvement use may affect whether submitted content is used for future training, but it does not remove information from the active context needed to produce the current response.
Does local AI eliminate context leakage?
Local inference can keep prompts and outputs on hardware you control when the entire workflow is actually local, but it does not automatically stop a model from encoding context into its output. Local operation changes who can observe the output and where data travels; it does not prove the model's output is statistically independent of private context.
Is a refusal still useful?
Yes. Refusing direct disclosure blocks an obvious attack path. The study's point is that refusal should be one layer in a broader privacy evaluation, not the only test.
Bottom Line
The new preprint reports a real and surprising laboratory result: secrets in a model's context can influence benign-looking output, and an optimized prompt made Claude Opus 4.6 and Gemini 3.1 Pro encode synthetic SSNs despite successful direct refusals.
It does not prove that either company leaked customer SSNs, that every answer carries an exploitable secret, or that the method works unchanged against current production products.
The durable lesson is simple: do not treat “the model refused” or “the secret is not visible” as complete privacy proof. Keep raw secrets out of model context, minimize memory and connector scope, separate untrusted input from privileged agents, and test the output channels that ordinary content filters do not inspect.