Which GPU created each open model? Reading the weights of 265 LLMs

🌐 Languages: 中文 · English

Today Will DePue of Unemployment Labs published Checkpoint forensics, which shows that an open model’s weight file can reveal what kind of GPU created its first weights. I rebuilt the method from scratch, without his code, and ran it on every open-weight LLM I could download. It holds up. Results come first, then an explanation for readers who have never thought about GPU random number generators.

TL;DR

PyTorch fills a new model with random numbers using every thread the GPU has, and the thread count differs between GPU generations. Weights that training never touched still carry that count.
I scanned 265 open models and got a reading for 185. Those 185 releases come from 125 distinct initialisations, because labs re-release the same weights and fine-tune each other’s models.
25 releases read as Blackwell-class: 16 initialisations by 12 developers. On the GB200/GB300 thread count: DeepSeek V4-Pro, Kimi K3, NVIDIA Nemotron-3 Ultra and Super, Thinking Machines Inkling, Ant Ling-3.1-flash, IBM Granite-Switch-4.1. On B200/B300: DeepSeek V4-Flash, MiniMax M3, Qwen3.8-Flash-Next, LLaDA-MoE-v2, LG K-EXAONE-2.0, SK Telecom A.X-K2, Trillion Labs Gravity-30B.
The rest: 128 Hopper (H100/H200/H800), 27 A100, three on AMD accelerators, one V100, and one Kimi model that blends two thread counts.
Every reading is labelled with its evidence tier: 152 from untouched rows, 28 from trained rows, 4 from rows where only the signs survive, 1 from a sibling release.
The original article and I report the same GPU class for 65 models, and for 9 more neither of us finds a GPU fingerprint. Nothing contradicts. I add one reading (Qwen3.8-Flash-Next) and explain one difference (Qwen3-235B).
A reading is the thread count of the device that ran the initialisation. It doesn’t identify a chip, the place the model was trained, or the owner of the hardware.

Blackwell-class and AMD readings, by developer

Results

The figure above lists every Blackwell-class and AMD reading; two more below cover Hopper, A100 and V100. No mark on a name means untouched rows, the strongest tier: the published numbers are the original random numbers and I regenerate them. A dagger (†) means trained rows, a double dagger (‡) rows that kept only their signs, and a section sign (§) a sibling release of the same model. A tick means the original article reports the same class. The figures show the 169 original releases (124 initialisations); the text counts all 185, fine-tunes of other developers’ models included.

Blackwell. GB200/GB300 is 311,296 threads and B200/B300 is 303,104.

DeepSeek. V4-Pro reads GB200/GB300 and V4-Flash reads B200/B300, each from about 460 untouched rows. Every earlier DeepSeek model I can read is Hopper or A100.
Moonshot. Kimi K3 and K3.1-Pro share one initialisation and read GB200/GB300 from untouched rows. K2 through K2.6 read Hopper, and K2.7-Code is a blend (see below).
MiniMax. M3 and M3.1-Flash, also one initialisation, read B200/B300 from untouched rows. M1 through M2.7 read Hopper.
NVIDIA. Nemotron-3 Super reads GB200/GB300 from 862 untouched rows, and Nemotron-3 Ultra and 3.5 Super read the same from trained rows. NVIDIA’s technical report names GB200 for part of Ultra’s training, which makes that one a sanity check, if an indirect one, since training hardware only hints at what ran the initialisation. Nemotron-3 Nano and 3.5 Lightning read Hopper.
Thinking Machines. All three Inkling models read GB200/GB300, from about 1,000 untouched rows each.
IBM. Granite-Switch-4.1-30B reads GB200/GB300 from trained rows. Granite-3.3 reads Hopper.
Ant Group. Ling-3.1-flash reads GB200/GB300 from 139 untouched rows, as in the original article. Ling-1T, 2.6 and 3.0 read Hopper.
Alibaba. Qwen3.8-Flash-Next reads B200/B300 from trained rows, where the original article lists “no signature”. The trace is strong in the output layer (365σ, with the next thread count near 1σ) and weaker in the embedding (18σ). Qwen3.5-397B and Qwen3.8-2.4T read Hopper from trained rows and their output layers.
Renmin University and Ant. LLaDA-MoE-v2 reads B200/B300 from 367 untouched rows. It shares a training stack with Ling.
LG, SK Telecom and Trillion Labs. Three Korean models read B200/B300 from untouched rows: K-EXAONE-2.0, A.X-K2 and Gravity-30B. Gravity reads only in the first rows of its vocabulary; something I haven’t identified filled the last rows later. LG’s EXAONE-4.5 and SK Telecom’s A.X-K1 read Hopper.

The caveats matter most for the Chinese labs. A thread count is not a chip, initialisation is not training, and nothing here says who owned the hardware, where it was, or whether anyone broke a rule.

DeepSeek is the cleanest example: its seed has never changed, and only the thread count under it moves.

DeepSeek models over time, by GPU class

AMD. Three models read as AMD accelerators. Zyphra’s ZAYA1-74B reads 622,592 threads, a full MI300X, from trained rows, and Zyphra has said it trained on MI300X. AI2’s OLMo-7B-Twin-2T reads 225,280 from untouched rows, half of an MI250X, the chip in the LUMI supercomputer where AI2 trained it. OLMo-1B reads the same count from trained rows. I wrote no AMD code. My emulator of the NVIDIA path reproduces these weights, so PyTorch on AMD evidently lays out its random numbers the same way.

Hopper and A100. Most models read Hopper: 128 releases from 82 initialisations. They include Meta’s Llama-3 through Llama-4, Mistral from Mistral-7B to Mistral-Small-4, AI2’s later OLMo models, StepFun, Upstage’s Solar, Jais and Sber’s GigaChat3.5. The original article already has Llama-3.1, Mixtral, OLMo-2 and -3 and the Step models. Llama-3, 3.3 and 4, most of the Mistral line, Solar, Jais and GigaChat are new here. I read the Llama models and DBRX from public mirrors of the same weights, because I couldn’t read the official repositories. The 27 A100 readings are mostly older models such as Pythia, BLOOM, OPT, OLMo-7B, Yi and the first Qwen models. RedPajama-INCITE-7B reads 163,840 threads, a V100, and it was trained on Summit, which is built from V100s.

Hopper readings, by developer

A100 and V100 readings, by developer

By upload date, the move to Blackwell-class thread counts starts in 2026.

Readings by developer and upload date

Seeds. The same search finds the random seed, and the seed turns out to identify the training code.

3952 appears in 60 of the 124 initialisations I can attribute to their original developer, across 28 developers, from GPT-NeoX-20B in 2022 to K-EXAONE and Nemotron in 2026. It is 1234 + 2718: the default seed of NVIDIA’s Megatron-LM framework plus a constant Megatron adds.
2760 (42 + 2718), 6125 (3407 + 2718) and 2718 (0 + 2718) come from the same code path with a different base seed. 3407 is a popular seed, yet all seven initialisations that carry 6125 are Qwen’s, from Qwen-7B to Qwen3-235B. Seed 2718 is Meta’s, in Llama-3 and Llama-4. Meta’s OPT used 2719.
953049583 is in all 10 DeepSeek initialisations since 2023, across four thread counts. Kimi’s main seed is 20256586, though Moonlight used 20233441. AI2’s early OLMo models used 6198, then 12536 for OLMo-2-32B and 0 for Olmo-3. LG’s EXAONE-4.0 and 4.5 use 17 (K-EXAONE-2.0 uses 3952), and Upstage uses 1997.

Recovered seeds and how many initialisations share them

Fine-tunes inherit the fingerprint. Sixteen models carry the initialisation of another developer’s model. Cogito-671B, RakutenAI-3.0 and Nex-N1 match DeepSeek-V3, and Nex-N2.5-Max matches DeepSeek-V4-Pro. INTELLECT-3 matches GLM-4.5-Air and Atria-Dawn matches GLM-5.2. Hermes-3 and Hermes-4 carry Llama-3.1-405B’s rows, and three more carry Llama-3.1-70B’s or 3.3-70B’s. Intern-S2, Baichuan-M3, DeepSeek-R1-Distill-Qwen-32B and KAT-Dev-72B match Qwen3.5-397B, Qwen3-235B, Qwen2.5-32B and Qwen2.5-72B, in that order. A shared seed can’t decide the parent, because two labs on Megatron’s defaults with the same tokenizer start from identical tables. I compare trained rows instead: KAT-Dev’s have a cosine of 0.998 with Qwen2.5-72B and 0.61 with Qwen2-72B. Post-training leaves the fingerprint alone, so the GB200/GB300 reading on Nex-N2.5-Max belongs to DeepSeek-V4-Pro’s initialisation, not to Nex.

What I could not read. 80 models gave no fingerprint. None of them is evidence about hardware.

25 have no GPU layout in their rows. Ten Gemma models, Grok-1 and GPT-J come from JAX, whose random numbers don’t depend on the device; GPT-J’s 143 unused rows regenerate on a CPU from the public training code. ERNIE-5.5 uses an older PaddlePaddle function that is also device-independent. TeleChat2.5 and TeleChat3 were trained on Huawei Ascend chips, per their model cards. Two Jamba models and AliceAI were initialised on a CPU. Seven Qwen models (Qwen3-8B, 14B, 30B and 32B, Qwen3.5-9B and 27B, Qwen3.8-27B) had their unused rows overwritten by a CPU after training, which leaves their real initialisation unknown (see “Rows written by a CPU”).
11 have no rows I can use.
44 have rows that look untouched, but no seed I searched regenerates them. gpt-oss-120b is one: 922 unused output rows of clean Gaussian noise, identical to those in gpt-oss-20b, matching nothing. DeepSeek-V4.1-Flash is another, with 428 untouched rows that match neither DeepSeek’s usual seed nor any other I tried; the original article found nothing there either. Llama-1, Llama-2, Phi-3 and the newest Mistral models are in this group. So is ERNIE-4.5, which uses Paddle’s ordinary, device-dependent GPU fill and should be readable once someone finds its seed.

265 models: fingerprints by class, and why the rest gave none

Against the original article. We report the same GPU class for 65 models, and on 9 more neither of us finds a GPU fingerprint. For DeepSeek-V4-Flash I get 461 of 468 rows at 1,384σ, and the article reports 461 of 469 at 1,385σ. Qwen3.8-Flash-Next is new on my side. For Qwen3-235B the original reports a CPU initialisation and I report Hopper (§), and both are right about what they looked at. The April upload’s unused rows were overwritten with CPU noise after training, while Qwen’s 2507 releases of the same model keep the original rows: 441 in Instruct-2507 and 415 in Thinking-2507 regenerate on the Hopper thread count. Qwen3-32B and Qwen3.5-9B, two of the 9, are the same story without a surviving copy. The original reports CPU there, and I find the same CPU rows but read them as overwritten after training.

How it works

If you already know how PyTorch seeds a CUDA generator, skip to “Threads write in stripes”.

A model’s weights start as random numbers. The part that matters here is the embedding table, with one row per token in the vocabulary. DeepSeek-V4-Flash has 129,280 rows of 4,096 numbers, each drawn from a bell curve centred on zero with standard deviation 0.006.

The numbers come from a generator, a fixed recipe that turns one starting number, the seed, into a long sequence. The same seed always gives the same sequence, so if I can guess the seed and the recipe, I can recreate a model’s starting weights.

Most generators are chains, each number computed from the one before. PyTorch on GPUs uses Philox, which is counter-based: a formula takes the seed and a position and returns the number at that position directly. Many threads can draw from one seed at once, each on its own range of positions. A guess is also cheap to check, since I only compute the handful of positions I want to compare.

Every GPU has a fixed thread count

An NVIDIA GPU is built from streaming multiprocessors (SMs), each hosting a fixed number of threads. The GPU’s total is SMs × threads per SM.

Threads per GPU for each GPU class

The original article talks in SM counts: 132 for Hopper, 148 for B200. A100, Hopper and Blackwell datacenter GPUs have 2,048 threads per SM, but many other cards have 1,536 or 1,024, so I use the total, which is what shows up in the weights. Some chips share a total. H100 SXM, H200 and H800 SXM all have 270,336 threads (the PCIe H100 and H800 have fewer), B200 and B300 have 303,104, and GB200 and GB300 have 311,296. The method can’t separate chips within such a group, so I call each group a GPU class.

Threads write in stripes

To fill a large tensor with random numbers, PyTorch launches as many threads as the GPU can hold. Call that number T. Thread 0 writes elements 0, T, 2T and so on. Thread 1 writes 1, T+1, 2T+1. Each thread draws from its own Philox stream, set by the seed and the thread’s index.

T threads taking turns to fill a tensor

Thread 2 produces the same numbers on any GPU, but its second number lands at position T+2, which is 270,338 on an H200 and 303,106 on a B200.

The same seed laid out by two different thread counts

PyTorch reads T from the hardware; nobody chooses it. In DeepSeek-V4-Flash’s table one stripe covers exactly 74 token rows on a B200 and 66 on an H200. The first T elements are the same on every GPU, since each thread is writing its first number. The information is in what comes after.

I don’t own a B200, so I reimplemented PyTorch’s CUDA fill with T as a parameter. On my own card, an RTX PRO 6000 with 288,768 threads, the emulator matches real PyTorch bit for bit.

Some rows are never trained

An embedding row gets a gradient only when its token appears in the training text, and every vocabulary has tokens that never appear: reserved placeholders like <reserved_12>, and padding rows that round the vocabulary up to a convenient size. Weight decay may shrink these rows by a constant factor, but their pattern survives.

They are easy to find. A row of fresh Gaussian noise has a predictable length, σ × √(row width), which for DeepSeek-V4-Flash is 0.006 × √4096 = 0.384. Trained rows are much longer and vary from token to token. The untouched ones sit in a flat line at the predicted value.

Row norms at the end of DeepSeek-V4-Flash's vocabulary

Regenerate and compare

The untouched rows are fixed by the seed and T, so I regenerate them under each candidate T and compare with the published weights. For DeepSeek-V4-Flash with seed 953049583, the B200/B300 thread count reproduces the rows and the Hopper thread count gives unrelated noise.

Published weights against regenerated values under two thread counts

I test 19 thread counts, from a T4 to AMD’s MI300X, and one of them fits. Of the 468 candidate rows, 461 match, and over all 468, 1.9 million numbers, the score is more than 1,300 standard deviations above chance. A wider sweep of 822 thread counts over the Blackwell-class readings from untouched rows found no second fit, apart from the AMD case under Limits.

Match score for every candidate thread count

The seed comes from brute force. Seeds that people type are almost always below 2³², about 4.3 billion, and with Philox a single GPU tests all of them in about a minute. Given 64 numbers from one untouched row of DeepSeek-V4-Flash, the search returns 953049583 and nothing else, the seed the original article reports. A wrong T finds no seed at all.

Other fills, other layouts

My first scan left about a hundred models with clean untouched rows and no seed. Many shared a cause. Mistral, Meta, AI2, Ant, Upstage and others initialise with PyTorch’s trunc_normal_, which never produces a value beyond a cutoff the caller sets, about 3σ in these models. It draws uniform numbers, which on a GPU land in the same stripes, and bends them into a bell shape with erfinv. My search was looking for the plain normal fill.

The rows give it away: slightly flatter than a Gaussian (excess kurtosis −0.17 instead of 0), with nothing beyond 3σ. The fix is to bend each value back through erf, which recovers the uniform numbers, and search for those.

Untouched rows from a truncated-normal initialisation, mapped back to uniform draws

That one change read Mistral, Llama-3, OLMo, Ling, LLaDA, Solar, Jais and Gravity. StepFun’s rows are simpler still, a plain uniform fill.

Large models are also rarely initialised as one tensor on one GPU, and each framework splits the work its own way.

A seed per GPU. Megatron gives each GPU a slice of the vocabulary and seeds it with seed + 2718 + rank. Llama-3 is filled this way with seeds 2718, 2719 and up, as are XVERSE, OPT, Stockmark and the tail of Qwen2. MiniMax-M2 is a four-way split with its own rule, 1001102 + 1021 × rank. Mistral’s newer models seed each block of rows or columns with 0, 1, 2 and so on.
A seed per pipeline stage. Megatron adds 100 for each stage, so the output layer of Qwen3.5-397B has seed 4252 where its embedding has 3952.
One seed, different stream positions. PyTorch’s distributed tensors give every GPU the same seed and a different starting point. Inkling is cut into 64 to 256 column blocks this way, and OLMo-2 and Granite into row blocks. A plain offset is the same idea: tensors initialised earlier have advanced the generator, so Kimi K3 sits at offset 3,776 of Moonshot’s seed and Qwen3.5-397B’s embedding at offset 828.

PaddlePaddle’s ordinary GPU fill writes the same stripes, which I checked against real Paddle on my card.

Rows written by a CPU

Some untouched rows match no GPU thread count because a CPU wrote them. PyTorch’s CPU generator is a single Mersenne Twister (mt19937) stream that hands out numbers one after another, so it has no thread layout and says nothing about hardware. It can still be proven without knowing the seed. On a bf16 tensor, the CPU fill uses only the low 8 bits of each generator output and writes values in pairs (a Box–Muller transform), and the angle of each pair gives those bits back. A few rows yield about 30,000 equations over GF(2) in the generator’s 19,937 state bits. Zero contradictions proves a CPU stream, and the solved state walks back to the seed.

Jamba is torch.manual_seed(23) on a CPU, which reproduces 1,019 untouched rows, and AliceAI is seed 1234. InternLM2-1.8B is seed 1001, and InternLM’s public training code, InternEvo, does allocate its embedding on the CPU. The surprise was Qwen, where the “untouched” rows of many models are CPU noise written over the unused rows after training. Qwen3-8B, 14B, 30B, 32B and the April Qwen3-235B each have a seed of their own, and the 8B, 14B and 30B rows reproduce bit for bit. Qwen3.5-9B and 27B and Qwen3.8-27B and 2.4T all use seed 42. Qwen3.8-2.4T still reads Hopper from its trained rows and output layer. The untouched rows of Qwen3.5-35B, 122B and 397B are not a CPU stream. So the original article’s “Qwen3 on CPU” readings describe overwritten rows. Where an earlier copy survives, in Qwen3-235B’s 2507 releases, the real initialisation reads Hopper.

Trained rows keep a trace

Some models have no untouched rows. But training nudges a row without replacing it, so a weight that started high usually stays on the high side.

Weights before and after training in a small model

In a single row of a large model the trace is faint. For Xiaomi’s MiMo-V2.5-Pro-Base, whose seed I know from its untouched rows, the median trained row has a cosine similarity of only 0.034 with its regenerated starting value. Summed over 16,384 rows, that is 488 standard deviations for the Hopper thread count and at most 5 for any other. This is how I read the 28 daggered models, among them Nemotron-3 Ultra, Granite-Switch, Qwen3.8-Flash-Next, Llama-4 and ZAYA1. I count such a reading only if it holds on a second, disjoint set of rows and no other thread count comes close.

The tier is weaker than untouched rows because I can’t regenerate the published numbers, only show that they correlate with one candidate and with no other. It also needs a seed to test, usually from a sibling model or from the seeds other labs use.

Sign-only rows (‡). In Qwen2-7B, Qwen2.5-7B and Qwen2-57B, weight decay has shrunk the unused rows to the smallest number the storage format can hold, about 10⁻³⁷. Every value has the same size and only the signs are left, but they are the original signs. The vocabulary is split four ways, and the signs match seed 6125 + rank on the Hopper thread count in more than 99% of positions, against about 50% on any other, which is chance. The fourth such reading is NVIDIA’s Nemotron-Nano-9B-v2. Its embedding is a column-pruned copy of Nemotron-Nano-12B-v2’s, whose decayed rows agree in sign with seed 3952 on Hopper in 100% of positions.

Kimi-K2.7-Code: two GPUs in one row

The original article describes Kimi-K2.7-Code as about 80% Hopper values inherited from K2.5 and about 20% a B200-class draw. My first scan saw only Hopper.

The 254 newest rows of its vocabulary were initialised as a small tensor of their own, with seed 244. Against the Hopper thread count they score 0.97, where untouched rows should score 1. Subtract 0.8 × the K2.5 row from each K2.7 row, and the remainder matches the same seed on the B200/B300 thread count at 0.9998.

Kimi-K2.7-Code's new rows as a blend of two thread counts

The ratio is the same in almost every row to within a fraction of a percent, which looks like a weighted average of two checkpoints, as the article suggested. In the first 37 rows, inside the first stripe where both GPUs write identical numbers, the blend is invisible. All it shows is that a device with 303,104 threads ran this one small initialisation a second time. I count K2.7 apart from the Hopper and B200 totals.

Checking against known answers

Disclosed hardware. The best test is a model whose developer has said what it was trained on. I have 25 of those, on six thread counts, and the reading matches every time.

Disclosed hardware against the class read from the weights

AI2 is the most useful case, because OLMo moved between three kinds of hardware and said so each time: A100s for OLMo-7B, AMD MI250Xs on LUMI for OLMo-7B-Twin-2T, H100s after that. The readings follow. I also reproduce Llama-3.1, the original article’s known-answer case.

One class is missing. No model that reads 303,104 has disclosed its hardware. The B200/B300 label, like the GB200/GB300 one, rests on the original article, whose author ran the fill on a physical B300 and a rented GB200.

My own model. I trained a small model on my RTX PRO 6000 whose last 256 tokens never appear in the training text. After training, those rows point in exactly the same direction as before (cosine 1.0000, scaled by 0.86 from weight decay). From the trained checkpoint alone, the search returns the right seed and 288,768 threads, and it gives the same answer from the trained rows only.

Recovering the seed and thread count of a model trained locally

Pythia. EleutherAI documents A100 training and publishes the checkpoint from before training began. For Pythia-70M the tool finds seed 3952 on the A100 thread count and regenerates all 25,755,648 starting embedding values, an fp16 fill, bit for bit. No other thread count does. It gets the same answer blind from the untouched rows of the final checkpoint.

Limits

A thread count is not a chip. It can’t tell an H100 from an H200 or H800, a B200 from a B300, or a GB200 from a GB300.
AMD can produce NVIDIA’s numbers. An MI300X has 304 compute units. Split into two groups of 152, each half presents exactly 311,296 threads, the GB200/GB300 count. Limited to 148, it presents 303,104, the B200/B300 count. The original article confirmed this on real hardware, and my AMD readings show such machines do initialise models. Nemotron-3 Ultra ties 311,296 to a GB200 in one partly disclosed case. For the other 311,296 readings, and for all ten B200/B300 readings, an MI300X set up this way would look the same.
It can be imitated. Software can emulate any thread count, as my emulator does. The original article describes a separate numerical check that favours NVIDIA hardware for V4-Flash, which I haven’t replicated.
Initialisation is not training. Weights can be created on one machine and trained on another.
It says nothing about ownership, location or legality.
No fingerprint means no information. My search is not exhaustive, and this round showed how many real fingerprints an incomplete search misses.

Reproduce it

git clone https://github.com/bojieli/checkpoint-forensics
cd checkpoint-forensics && pip install -e .

ckptf verify DeepSeek-V4-Flash  # ~4 MB of public weights
ckptf verify all                # every fingerprint card
ckptf scan <hf-repo> --quick    # search any model from scratch
ckptf selftest                  # train a small model, then recover it

138 of the 152 untouched-row readings ship as fingerprint cards: repository, tensor, rows, seed, offset and thread count. verify downloads only those rows and regenerates them under every thread count. Any NVIDIA GPU will do, because the other thread counts are emulated. The trained-row readings and the ones that needed a special layout have no cards yet; each has a script and a log in the repository, and needs gigabytes of download. If you think a reading is wrong, the card or the script gives you what you need to show it, and I’d like to hear about it.

The method and the original readings are from Checkpoint forensics by Will DePue, Unemployment Labs. This is an independent clean-room implementation and replication by Bojie Li. Code, cards and full results: github.com/bojieli/checkpoint-forensics; every reading, model by model: READINGS.md.