The model remembered what it was fed. What it was fed was you.
Also known as Memorization and extraction attacks
A model reproducing, in its output, specific content from the data it was trained on, including personal, private, or copyrighted material that was never offered for that use. The system ingests a vast corpus, and under the right prompt it can regurgitate fragments verbatim: a name, an address, a passage, a face. The person whose data was absorbed never consented and was never asked. Their words or image became part of the machine's memory without their knowledge, and now surface in outputs they cannot control. The extraction happened at training time, invisibly, and the harm surfaces later, one leaked fragment at a time.
Truth-adjacency
Truth-independent: the pattern works regardless of whether the claim is true
Where it shows up
Platforms and algorithms
How it works
The phrases and tells that mark this pattern in the wild:
a model reproducing specific text or data it was trained onpersonal details surfacing in output that were never meant for itcopyrighted passages appearing verbatim in generated contentno consent given and no way to have the data removedthe model treating private material as if it were public knowledgeThe tell is the specificity. A model that has learned general patterns produces general output. A model that has memorized produces the particular: a real name, an exact passage, a recognizable face. Watch for output that reproduces identifiable material rather than reflecting a style, and for the absence of any path to consent or removal. Also watch the timing of the harm. The extraction happened long before the output, at training time, invisibly and at scale. The person whose data was taken learns about it only when a fragment surfaces. If a model produces something specific and private that was never offered to it, the data was taken, and there is no way to give it back, because it is no longer stored. It is learned.
You call “training data extraction” on output that reflects general patterns the model learned, without reproducing anything specific or private. Resemblance is not always reproduction. The pattern requires the model to reproduce identifiable material, personal, private, or copyrighted, ingested without consent and surfacing in output the source cannot control. If the output is a new synthesis that does not reproduce a specific fragment, you are looking at ordinary learning. The distinction matters, because extraction is a violation of a specific person or work, while generalization is how these systems learn at all.
One of these two real scenarios is Training data extraction. The other is a different pattern entirely. Which one is which?
The tell
A model producing output that reflects general patterns it learned, without reproducing specific private or protected content, is generalizing, not extracting. The pattern requires the model to reproduce identifiable material from its training data, personal, private, or copyrighted, that was ingested without consent and now surfaces in output the source cannot control. If the output is a new synthesis that does not reproduce a specific fragment, you are looking at ordinary learning, not the regurgitation of someone's data.
A corpus is assembled at a scale that makes asking impossible. Whatever was reachable is included, and reachable is not the same as offered.
Because the harm completed at training time, invisibly, and the remedy would mean rebuilding the system rather than correcting an output. Everyone in the chain can point to a step at which nothing unusual was done.
Nobody refused consent, because nobody was asked, and the silence got counted as agreement.
Field notes where this pattern was identified:
How this pattern gets misused
Someone treats any model output that resembles its training data as extraction, including the ordinary case of a model learning general patterns and producing something similar but not copied. The term becomes a way to claim theft whenever generated content echoes a source, which blurs the line between learning a style and reproducing a specific, private, or protected fragment. Indiscriminate use makes the genuine violation, the verbatim leak of private material, harder to isolate.
What it looks like when you're wrong about it
A model producing output that reflects general patterns it learned, without reproducing specific private or protected content, is generalizing, not extracting. The pattern requires the model to reproduce identifiable material from its training data, personal, private, or copyrighted, that was ingested without consent and now surfaces in output the source cannot control. If the output is a new synthesis that does not reproduce a specific fragment, you are looking at ordinary learning, not the regurgitation of someone's data.
Not sure? Describe the situation to someone outside it. If they do not see the pattern, pause before you name it.
The name is designed to spread. The hook is designed to stick. If you recognized something, share the name.
Seen a real example of training data extraction? Suggest it for the Register →