Does the product space predict the climb? An honest control
Key findings
- The raw signal looks convincing — and is nearly all diversity. Capability-type products score AUC 0.855 (2002→2024). A pure diversity baseline scores 0.847. Density and diversity are correlated 0.95–0.98 across materials: they are almost the same variable.
- The residual collapses. Regress density on diversity and score only what is left, and the AUC falls to 0.63 (long window) — barely above the 0.50 coin-flip, and driven by the very few, very diversified entrants.
- Placebos pass the same test. Whisky (0.76), wine (0.81), cotton (0.72) and salmon (0.70) — products with no product-space relationship to critical materials — land in the same 0.70–0.81 band as the real signal, and wine actually outscores the capability residual. A test that “confirms” whisky and wine is not confirming anything specific to capability.
- Report the null, keep the tool. The product space is still a useful map of what sits near what; it is not a validated forecaster of critical-material capability on this evidence. Where the atlas names who could break a chokepoint, that estimate is labelled as adjacency, not destiny.
1 · The test
We build the binary export-capability matrix (Balassa RCA ≥ 1, with world-share and value floors) for every country and product in each year of the CEPII BACI HS2002 panel — one consistent nomenclature from 2002 to 2024, so there is no vintage join to contaminate the result. For a set of downstream critical-material products, we ask a strict emergence question in the spirit of Hidalgo & Hausmann:
We score that ranking with ROC-AUC. Crucially, we score three things side by side: density itself; a pure diversity baseline (how many products a country already exports competitively); and the residual of density after diversity is regressed out. And we run the identical test on placebo products chosen to have no product-space link to critical materials, as a control on the whole procedure.
2 · Results
| Window | Product tier | density AUC | diversity AUC | residual AUC | n products | n entrants |
|---|---|---|---|---|---|---|
| 2002 → 2024 | Capability-type | 0.855 | 0.847 | 0.627 | 10 | 53 |
| Commodity-type | 0.692 | 0.685 | 0.570 | 5 | 39 | |
| 2016 → 2024 | Capability-type | 0.873 | 0.861 | 0.699 | 9 | 32 |
| Commodity-type | 0.719 | 0.717 | 0.539 | 5 | 25 |
Two things jump out. First, in every row the density AUC and the diversity AUC are within ~0.01 of each other — density adds almost nothing over simply asking “is this a diversified economy?” Second, the residual (what density knows beyond diversity) drops to 0.54–0.70, with the commodity tier at essentially chance. The signal is real but it is a diversity signal, not a product-space one.
3 · The placebo control
If the test were measuring something specific to product-space proximity, products with no such proximity should fail it. They do not.
| Placebo product | AUC 2002→2024 | AUC 2016→2024 |
|---|---|---|
| Whisky | 0.76 | 0.74 |
| Cotton | 0.72 | 0.56 |
| Wine | 0.81 | 0.89 |
| Fresh salmon | 0.70 | 0.62 |
| Placebo average | 0.75 | 0.70 |
Whisky is not something a country “climbs to” through nearby manufacturing capability; it is a story about being Scotland. That it scores 0.76 — right on top of the real capability tier — is the whole point. A test any diversified economy passes for whisky is not evidence that the product space forecasts critical-material capability.
4 · Why publish a negative result
An earlier version of this atlas reported the 0.86 figure as a validation of the product-space method. It was not one. Under a two-engine adversarial review the number was re-run, the diversity baseline and placebos were added, and the claim did not survive. The honest conclusion is a boundary, not a confirmation — and the atlas is more trustworthy for stating it. A decision layer that tells a policymaker who could realistically build the missing capability is only as good as its knowledge of its own error bars. This report is that error bar, stated in the open.
A methodological caveat compounds the story: under the ~0.5–5% base rates of these emergence events, ROC-AUC is optimistic; precision-recall (reported in the data) is the fairer lens and is weaker still.
Method
Binary capability matrices from CEPII BACI HS2002, 2002–2024. Balassa RCA with world-share (≥0.1%) and value ($500k) floors; density computed from the product-space proximity matrix φ (co-occurrence normalised by max ubiquity). Emergence = RCA crossing 1 between t₀ and t₁ among non-holders at t₀. AUC by Mann–Whitney; residual AUC by regressing density on diversity (and ECI) and scoring the residual. Placebos: HS 220830 (whisky), 220421 (wine), 030212 (fresh salmon), 520100 (cotton). Products split into capability-type (specialty/manufactured downstream) and commodity-type (bulk smelting/ferroalloys) tiers. Exploratory throughout; reported as a null-ish control, not a confirmation. Full methods are open; all figures are in avalidate.json.
@techreport{cma_report03_2026,
title = {Does the product space predict the climb? An honest control},
author = {{Critical Materials Atlas}},
institution = {Critical Materials Atlas},
number = {Report 03},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.21948855},
url = {https://criticalmaterialsatlas.org/report-product-space.html}
}