Method · testing our own work

Auditing our own claims

The atlas measured how far the US Geological Survey moves its own production figures between the first printing and the revision: a median 1.5% for copper's world total, 11.6% for antimony's, and once 42.5%. That measurement was about other people's numbers. This page pushes those revisions back through our own concentration study and asks which of its claims they can erase. Three claims, tested on the same draws: that the materials already called critical before the window mostly diversified; that cobalt concentrated while those older materials came off monopoly highs; and the full-set median that gets quoted most and that our own study says is substantially an artifact of what was added to critical-materials lists during the window.

What the test is

The concentration finding says that between 1995–2004 and 2015–2024, the typical critical material's production became more concentrated — a median rise of +0.046 in the Herfindahl index across 23 materials. Every number behind it is a country's reported production, and every such figure is an estimate that its source later revises.

So the test is not a comparison, it is a propagation. For each material we rebuild the country-by-year production the finding is computed from, multiply every value by (1 + a revision drawn at random from that commodity's own measured revisions), and recompute the finding. Not a modelled distribution — the actual revisions observed in the USGS editions, resampled. 2,000 draws, seed 20260924, both fixed in the filing before the test ran, along with the verdicts: robust if the finding keeps its sign in at least 95% of draws, fragile between 50% and 95%, not supported below.

The three claims, and what noise does to them

1. The materials already critical before the window mostly diversified. This is a threshold pass, not a survival. Across the 6 of 7 materials on the EU's 2011 list that can be tested, the median change is -0.068 and stays negative in 96.3% of draws — clearing the filed 95% line by more than the noise in 2,000 draws, so the label holds — but the band runs -0.100 to -0.003, which is as good as zero. And the six split three and three: antimony, graphite and rare earths down; tungsten, fluorspar and cobalt up. The median is negative because it averages a large, stable decline with tungsten, whose own sign survives only 87.1%. "Mostly diversified" is not what an even split shows. Under the size-stratified draw the same claim runs 99.65% with a band of -0.090 to -0.026, well clear of zero — so which test you believe decides how strong this claim looks, which is itself the finding.

One correction to an earlier draft of this page, because it flattered the result: it said that excluding the platinum-group metals made this a conservative test. It does not. With seven materials the published median is the fourth ordered value, and that value is the platinum-group change (-0.095). Removing it does not stress the published -0.095 at all — it tests the midpoint of rare earths and tungsten instead, a different and closer-to-zero statistic. The reviewers caught that; it was not caught here.

2. Cobalt concentrated while the older export-controlled materials came off monopoly highs. This is the claim furthest from its sign boundary. Requiring all three of antimony, graphite, rare earths to stay negative and cobalt to stay positive in the same draw, the pattern holds in 99.85% of draws. That is not because the test is demanding — it is because the margins are wide: cobalt's band starts at +0.269 and antimony's ends at −0.185, so these bands never come close to meeting. An earlier version of this page reported the weaker version of this test (the median of the three, which holds unless two of them turn) and called it joint. It was not. Caveat this one carries: graphite is inside it, and graphite is where the two agencies disagree most — a median 30.7% apart on the world total against 0.6% for copper. This test perturbs BGS figures with USGS-measured revisions; it cannot speak to the two agencies disagreeing about who produces.

3. The full-set median keeps its sign — which is informative about measurement error and says nothing about selection. Across the 19 testable materials of 23 the median is +0.043 (published +0.046 across 23), positive in 99.95% of draws, band +0.017 to +0.046. Revision noise does not explain this number away. But the concentration study's own note says it "is substantially an artifact of materials ADDED to lists during the window", and that is a selection problem which no perturbation of the measurements can address. The audit clears this number of one charge and leaves the more serious one untouched.

Why the sign test is a weak bar, stated plainly. "Keeps its sign" is what was filed, and it is a low bar for a median across materials: lithium (+0.504) and cobalt (+0.310) cannot plausibly flip, so the median stays positive unless several mid-sized materials turn together in the same draw. The band is the informative part, not the percentage — and for the full set the band's lower end, +0.017, is well short of the published +0.046. The claim that survives is directional: the typical tested material's concentration rose. The published magnitude is not what this test certifies.

What this test is not. An earlier draft of this page argued that independent draws must understate uncertainty, so a pass was weak and a failure strong. That is withdrawn: an HHI is a function of shares, so a revision common to every country in a year cancels exactly, and the structure named as widening the band in fact annihilates it. Meanwhile the pool is unweighted, so small producers' revision magnitudes get applied to the dominant producers who actually move an HHI — the smallest quartile of producers revises around 9% against about 5.5% for the largest tenth. Those two biases run in opposite directions and their net is unknown, so neither a pass nor a failure here is a bound. Both are diagnostics.

The size objection was tested rather than argued. Drawing each country's revision only from similarly sized producers changes no verdict at all: 19 of 19 materials keep their label, the full-set median holds at 100% (band +0.018 to +0.046), and the two materials it moves most — molybdenum to 91.1% and tungsten to 94.3% — stay short of the filed 95% line. That run is post-hoc and is published beside the filed one, not in place of it.

Material by material

What revision noise does to each material's change. The dot is the published change in the concentration index; the bar is the 5th to 95th percentile of that change once measured revisions are resampled into the underlying production; the number at the right is the share of draws in which the change keeps its sign. Amber marks the materials whose sign does not survive at the filed 95% threshold.

Material	published change	5–95 band under noise	sign survives	stage read	pool median absolute revision	revisions in the pool	USGS–BGS world gap	verdict
Lithium	+0.504	+0.391 to +0.547	mine (Lithium minerals)	100%	8.2%	201	not measured	robust
Cobalt	+0.310	+0.269 to +0.335	mine (Cobalt (mine production, metal content))	100%	6.7%	259	12.7%	robust
Fluorspar	+0.158	+0.110 to +0.199	mine (Fluorspar)	100%	8.3%	272	not measured	robust
Vanadium	+0.100	+0.061 to +0.126	mine (Vanadium (mine production, metal content))	100%	6.1%	91	not measured	robust
Phosphate rock	+0.089	+0.069 to +0.107	mine (Phosphate rock)	100%	5.8%	320	not measured	robust
Nickel	+0.076	+0.060 to +0.089	mine (Nickel (mine production, metal content))	100%	6.2%	344	not measured	robust
Feldspar	+0.060	+0.042 to +0.081	mine (Feldspar)	100%	5.0%	429	not measured	robust
Tungsten	+0.046	-0.020 to +0.102	mine (Tungsten (mine production, metal content))	87.1%	8.3%	245	6.9%	fragile
Chromium	+0.046	+0.019 to +0.071	mine (Chromium ores and concentrates (gross weight))	99.6%	7.5%	93	not measured	robust
Zinc	+0.044	+0.037 to +0.050	mine (Zinc (mine production, metal content))	100%	4.2%	229	not measured	robust
Manganese	+0.024	+0.007 to +0.041	mine (Manganese ore (gross weight))	98.8%	10.4%	193	not measured	robust
Molybdenum	+0.010	-0.031 to +0.045	mine (Molybdenum (mine production, metal content))	71.2%	6.5%	327	not measured	fragile
Titanium	-0.004	-0.030 to +0.017	mine (Titanium minerals)	73.3%	9.4%	178	not measured	fragile
Copper	-0.031	-0.037 to -0.026	mine (Copper (mine production, metal content))	100%	3.0%	306	0.6%	robust
Tin	-0.044	-0.080 to -0.007	mine (Tin (mine production, metal content))	97.2%	11.1%	325	not measured	robust
Barytes	-0.077	-0.106 to -0.039	mine (Barytes)	100%	9.8%	316	not measured	robust
Rare earths	-0.182	-0.265 to -0.088	mine (Rare earth oxides)	100%	5.0%	179	5.7%	robust
Graphite	-0.190	-0.260 to -0.078	mine (Graphite)	99.7%	6.7%	327	30.7%	robust
Antimony	-0.344	-0.370 to -0.185	mine (Antimony (mine production, metal production))	100%	20.0%	209	13.1%	robust

Source: the atlas's own published figures (the concentration measurement, from BGS World Mineral Statistics) tested against the revision record measured at how firm is this year's production figure? (USGS Mineral Commodity Summaries, 1996–2026). Computed values: out/self_audit.json.

16 of 19 materials are robust. three are not: molybdenum (71.2%), titanium (73.3%), tungsten (87.1%). Those three state changes that the revision record alone could erase, and this page is the reason to stop quoting them without that caveat. Tungsten is the instructive one: its published change is +0.046, the same rounded size as the full-set median, and its sign survives only 87.1% of draws — while chromium, whose published change is also +0.046 and whose revision pool is no quieter, survives 99.6%. The difference is in the share vector, not the size of the number. Size is not sufficient: everything at 0.06 or more here is robust, but among the smaller changes the published number does not tell you which survive.

Titanium's row is the weakest of the 19 and was flagged as such before it was run: the atlas series is BGS titanium minerals while the USGS chapter prints mineral concentrates, and the atlas has already found that titanium splits three ways on the ilmenite-versus-slag definition. Its published change is -0.004 — a claim of almost nothing — so "fragile" here means the data cannot tell a tiny move from none, which is not the same as a finding being overturned.

What we could not test, and why that matters

Material	published change	why it is not tested
Lead	+0.130	the atlas series is refinery (BGS 'Lead, refined') and the USGS chapter prints no refinery table to measure revisions on - perturbing one stage with another would be a category error
Magnesite	+0.076	in the older editions the reserve columns are headed where this parser reads a year, so reserve values would enter the panel carrying years - our limit, not a gap in the source
Bismuth	+0.074	same reserve-column fault as magnesite in the 2022 edition - our limit, not a gap in the source
Platinum group metals	-0.095	the USGS chapter prints platinum and palladium side by side, each with its own pair of years, so reading it needs handling this study does not have - our limit, not a gap in the source

An earlier run of this audit could test only 10 of the 23 materials, and those 10 had a median change of −0.004 against +0.060 for the 13 it could not reach. In other words the testable half was almost exactly the half that had not concentrated, and any verdict computed on it would have been a verdict about a different population. That is why the panel behind the revision measurement was widened from fifteen commodities to twenty-five before this page was written, rather than publishing the convenient number. The filing records the whole sequence.

What this cannot say

Neither verdict is a bound, and this page has already had to withdraw the argument that one of them was. A robust verdict means the sign held up under this perturbation, which is too harsh in one respect (it shocks dominant producers with small producers' revision magnitudes) and too gentle in another (it draws independently across years, while real revisions persist). A fragile verdict means the sign did not hold up under the same imperfect model. Every row here is a proxy row: the concentration finding rests on BGS production, while the revision record is measured on USGS editions, because no BGS vintage history exists to measure — the two agencies' world totals themselves differ by a median 0.6% for copper and 30.7% for graphite. The audit covers production-based claims only; the atlas's trade-based findings rest on sources whose revision behaviour has not been measured, and are listed as out of scope in the filing rather than quietly passed. And the revision pools are unequal — from 91 revisions to 429, printed beside every row — and a median drawn from the smaller pools is a thinner instrument than one drawn from the larger.

Method and filing

The filing. Measure, population, bands and pre-commitments, committed before any comparison was computed, then changed in 6 dated deviations — among them the discovery that the headline was not testable on the original panel, and the decision to widen the panel rather than report the subset: self-audit/PREREGISTRATION.md.
The code, committed before it was run: self-audit/analysis.py.
The finding under test. Concentration over time, from BGS World Mineral Statistics as held in the atlas's harmonised cube.
The revision record. How firm is this year's production figure?, from USGS Mineral Commodity Summaries 1996–2026.

Built by build_self_audit.py from out/self_audit.json, written by self-audit/analysis.py. Guarded by check.py, which fails if the audit's copy of a published figure stops matching the study it came from.