Global food trade data¶
Pablo Rosado • June 22, 2026 (last edit) • Feedback
Introduction¶
This is a technical companion to our article "How does food get traded around the world?". It examines FAOSTAT's Detailed Trade Matrix (TM), and explains the methodology and choices behind our visualizations.
Before we delve into the data, let's first define a few key concepts.
An item is a food product, either a primary commodity such as wheat or bananas, or a processed product such as meat (fish products are not included in the dataset).
An item flow is a trade exchange, namely an import or an export, of an item between two countries. Each flow is measured as a quantity (in tonnes, or number of animals) and as a monetary value (in dollars). Here we will analyse quantities of traded food (in tonnes).
Each entry in the TM dataset corresponds to a trade flow reported by a certain country and year, with a given partner country. For example, in one entry, Brazil reports exporting ~4 million tonnes of soya beans to Argentina in 2023.
Because each flow links a specific pair of countries (the reporter and its partner), this is known as bilateral trade data: it describes trade between two named countries, not a country's total trade with the rest of the world. The same flow can therefore appear twice, reported independently by the exporting country and by the importing country.
A flow can be reported by the country itself or by its trading partner; the partner's version is called mirror data. When a country does not report a flow, this mirror is often the only record we have of it.
Set up¶
This section explains a few steps needed to be able to run the code in this file and replicate its results. If you are only interested in reading the outcomes, feel free to skip this section.
Required libraries¶
Whether you run this code locally or on Google Collab, there are some libraries you may need to install.
%%capture
# Whether you run this code in your local computer or on Google Colab, you first need to install the required dependencies:
# * owid-catalog: a library that lets you explore and load datasets from OWID.
# * plotly: a library that lets you create interactive visualizations.
%pip install --upgrade owid-catalog plotly
Import necessary libraries and define some common variables.
import contextlib
import io
import numpy as np
import pandas as pd
import plotly.express as px
import plotly.graph_objects as go
from owid.catalog import fetch
# Load published tables from OWID's public catalog, so this notebook runs anywhere (e.g. Colab)
# without a local ETL checkout. fetch() returns an indexed Table; reset_index() gives plain columns.
# fetch() also prints an animated spinner that nbconvert captures as many lines, so silence stdout while loading.
with contextlib.redirect_stdout(io.StringIO()):
tb = fetch("garden/faostat/2026-05-07/faostat_tm/faostat_tm").reset_index()
tb_scl = fetch("garden/faostat/2026-02-25/faostat_scl/faostat_scl").reset_index()
tb_food_trade = fetch("garden/faostat/2026-05-07/food_trade/food_trade").reset_index()
reg = fetch("garden/regions/2023-01-01/regions/regions").reset_index()
# Map country names to ISO-3 codes, used by the reporting-rate choropleth (regions table from the catalog).
country_to_iso3 = dict(
zip(
reg.loc[reg["region_type"] == "country", "name"].astype(str),
reg.loc[reg["region_type"] == "country", "iso_alpha3"].astype(str),
)
)
# A few selected items combine several FAO codes (a primary commodity plus a mechanically-derived
# form whose trade FAO reports separately). The food_trade step sums them; this mirrors the
# multi-code entries in food_trade.items.yaml. Maps display name -> {FAO code: short form label}.
COMBINED_ITEMS = {
"Beef": {867: "bone-in", 870: "boneless"},
"Pork": {1035: "bone-in", 1038: "boneless"},
"Almonds": {221: "in-shell", 231: "shelled"},
"Cashews": {217: "in-shell", 230: "shelled"},
"Walnuts": {222: "in-shell", 232: "shelled"},
"Hazelnuts": {225: "in-shell", 233: "shelled"},
"Brazil nuts": {216: "in-shell", 229: "shelled"},
"Groundnuts": {242: "in-shell", 243: "shelled"},
"Rice": {31: "milled", 32: "broken", 28: "husked"},
"Sugar": {162: "raw", 164: "refined"},
}
# Map each selected FAO code to its display name. Combined items appear in the published table under
# a synthetic id (100000 + first code), so we expand them back to their underlying FAO codes here,
# letting analyses that join on the trade matrix's own codes still cover every selected item.
selected_code_to_display = {}
for code, item in dict(zip(tb_food_trade["item_code"].astype(int), tb_food_trade["item"].astype(str))).items():
if item in COMBINED_ITEMS:
for member in COMBINED_ITEMS[item]:
selected_code_to_display[member] = item
else:
selected_code_to_display[code] = item
Basic data exploration¶
The TM dataset has ~52 million entries, and over 500 unique items. Many of those are very specific food items, e.g. "Edible offal of pigs, fresh, chilled or frozen", or uncommon ones, e.g. "kola nuts". To avoid clutter, we created a selected list of items. However, in most of the present analysis we consider all items, except when we explicitly mention that we use the selected list.
We select only entries corresponding to item quantities (in tonnes, not dollars or animals). We also drop rows of self-trade, where reporter and partner countries coincide (which are just a ~0.1% of the data). Around 12% of rows correspond to trade of zero tonnes (which we keep in the data).
# Total number of entries
assert len(tb) > 52e6
# Basic data preparation: restrict to quantity reports in tonnes (this drops the
# alternate-unit rows used for live animals — 'An', '1000 An', 'No' — and the
# value-in-USD rows),
tb = tb[(tb["unit"] == "t")].reset_index(drop=True)
assert len(tb) > 25e6
# Drop self-trade rows (~0.1% of the data, where reporter
# and partner are the same country).
self_trade_mask = tb["reporter_country"].astype(str) == tb["partner_country"].astype(str)
assert 0.1 < 100 * len(tb[self_trade_mask]) / len(tb) < 0.2
tb = tb[~self_trade_mask].reset_index(drop=True)
# Share of rows with zero tonnes reported.
assert 100 * len(tb[tb["value"] == 0]) / len(tb) > 12
FAOSTAT flags each quantity row with its provenance. In general, quantity (in tonnes) and value (in dollars) are independently collected in the TM dataset.
About 98% of rows in the TM dataset are flagged "Official figure"; the remaining ~2% are flagged as imputed, estimated, or sourced from an external organization.
We keep all rows regardless of flag.
# Check that there are more than 500 items.
assert len(tb["item_code"].unique()) > 500
# Share of rows flagged "Official figure" across all years (claimed: ~98%).
official_share = (tb[tb["element"].isin(["Import quantity", "Export quantity"])]["flag"] == "Official figure").mean() * 100
assert 97 < official_share < 99
Yearly coverage¶
We would ideally show food trade figures for the most recent year available in the data. However, the latest year may be more affected by incomplete reporting.
The following chart helps us decide which year to choose. It shows the number of rows, as well as the number of distinct reporting countries, for each year in the dataset.
def plot_coverage(tb):
"""Show two bars per year: number of rows and number of distinct
reporting countries. When both drop together a year is partially
reported; when only rows drop it might be due to an actual trade contraction."""
grouped = tb.groupby("year", observed=True)
rows = grouped.size().sort_index()
reporters = grouped["reporter_country"].nunique().sort_index()
years = rows.index.astype(int).tolist()
fig = go.Figure()
fig.add_bar(x=years, y=rows.values, name="Rows", yaxis="y1", opacity=0.8)
fig.add_bar(x=years, y=reporters.values, name="Distinct reporters", yaxis="y2", opacity=0.8)
fig.update_layout(
title="Yearly data coverage",
xaxis=dict(title="Year"),
yaxis=dict(title="Number of rows", side="left"),
yaxis2=dict(title="Number of distinct reporters", side="right", overlaying="y", showgrid=False),
barmode="group",
legend=dict(orientation="h", yanchor="bottom", y=1.02, xanchor="right", x=1),
)
fig.show()
plot_coverage(tb)
We can see that, as of the day when this analysis was created, the latest year (2024) shows a significant drop in coverage with respect to the second latest.
So, in the rest of the analysis, we focus on the latest year with a reasonably complete coverage, namely 2023.
# Assert current maximum year and define the one we use for the rest of the article.
assert tb["year"].max() == 2024
# Year to use for year-specific analyses from now on.
YEAR = 2023
Reporting issues¶
Every trade flow has two sides. When Brazil ships soya beans to Argentina, Brazil can record it as an export, and Argentina can record it as an import. So you might expect the two countries to tell the same story: the same shipment, the same number of tonnes.
And sometimes they do. Brazil reports exporting around 4 million tonnes of soya beans to Argentina in 2023, and Argentina reports importing roughly the same amount from Brazil that year.
However, this is very often not the case.
- Item flows are often reported only one way; sometimes the importer, and sometimes the exporter.
- When both countries report the same flow, there are often significant disagreements in the reported quantities.
These differences can arise even when both countries follow the same international guidelines, due to timing, partner country attribution, confidentiality, and other factors. FAOSTAT explicitly notes in their Detailed Trade Matrix methodology note that "the trade matrix data are un-reconciled".
For example, in 2023 Ukraine reports exporting ~2 million tonnes of wheat to Romania, but Romania only reports importing ~200 thousand tonnes (a factor of ~10 apart).
According to FAOSTAT's Food Balance Sheets and Supply Utilization Accounts Resource Handbook 2025, section 6.1,
"[...] imports are typically documented more thoroughly and verified more rigorously than exports".
This claim cites the United Nations Statistics Division's International Merchandise Trade Statistics. That document mentions, in section 3.24:
Because of the greater customs scrutiny to which imports are subjected by customs in most countries, it is usually more feasible to derive estimates of exports from counterpart imports.
Then, in section 9.44:
"Import data were, in general, considered to be of better quality than export data because imports are reported in sufficient detail to allow customs to apply duties, taxes or other regulatory controls."
Finally, in section 15.15:
"Also, the customs administration is generally more interested in the quantity information for imports than in that for exports, since quantity information is, in some cases, utilized to determine both import duties and the unit values used to validate the price and value information declared by the importers."
However they also note, in section 9.44, that
[...] for certain commodities and in some countries, export data were viewed as being more accurate for the same reasons.
With all this, we therefore adopt the following simple criterion:
- If only one country reports the flow, we use that country's quantity.
- If both sides report the flow, we use the importer-reported quantity.
But note that there is unfortunately no simple way to reconcile reported quantities in bilateral trade flows. For broader context on why bilateral trade reports disagree, see our topic page on trade and globalization, specifically the section on trade data discrepancies.
One-sided versus two-sided reporting¶
For each item flow in a year, we count how many times that flow was reported both ways ("matched") or only one way ("exporter-only" or "importer-only"). The result is shown in the following chart.
def plot_reporting_coverage_by_year(tb):
"""Stacked bar chart per year showing what fraction of bilateral flows
are reported by both sides (matched), only by the exporter, or only by
the importer."""
qty = tb[tb["element"].isin(["Export quantity", "Import quantity"])][
["reporter_country", "partner_country", "item_code", "year", "element"]
].copy()
for col in ("reporter_country", "partner_country"):
qty[col] = qty[col].astype(str)
exp_keys = qty.loc[
qty["element"] == "Export quantity",
["reporter_country", "partner_country", "item_code", "year"],
].drop_duplicates()
imp_keys = (
qty.loc[
qty["element"] == "Import quantity",
["reporter_country", "partner_country", "item_code", "year"],
]
.rename(columns={"reporter_country": "partner_country", "partner_country": "reporter_country"})
.drop_duplicates()
)
merged = exp_keys.merge(
imp_keys, how="outer", indicator=True,
on=["reporter_country", "partner_country", "item_code", "year"],
)
by_year = (
merged.groupby("year", observed=True)["_merge"]
.value_counts(normalize=True)
.unstack(fill_value=0.0)
.rename(columns={"both": "matched", "left_only": "exporter-only", "right_only": "importer-only"})
)
by_year = by_year[["matched", "exporter-only", "importer-only"]].reset_index()
long = by_year.melt(id_vars="year", var_name="status", value_name="share")
fig = px.bar(
long, x="year", y="share", color="status",
title="Most trade flows are reported by only one country<br><sup>Share of bilateral flows reported by both sides, only the exporter, or only the importer, by year</sup>",
labels={"year": "Year", "share": "Share of item flows"},
category_orders={"status": ["matched", "exporter-only", "importer-only"]},
)
fig.update_layout(barmode="stack", yaxis_tickformat=".0%")
fig.show()
plot_reporting_coverage_by_year(tb)
We can see that the reporting inconsistency is quite severe. Most item flows are reported only one way (only the importer, or only the exporter). In 2023, only around 40% of item exchanges were reported both ways.
But flows are not all the same size. The chart above counts each flow equally, regardless of how many tonnes were traded. In the chart below, we weight each flow by tonnage (and, when both countries report, we assume the importer's quantity). Now we see that most of the traded tonnes are reported by both sides; around 82% in 2023.
def plot_reporting_coverage_tonnage_by_year(tb):
"""Same as the reporting-coverage chart, but weighting each flow by the importer-reported
tonnage (falling back to the exporter's when the importer is silent, the rule the final flow
table uses) instead of counting flows equally."""
qty = tb[tb["element"].isin(["Export quantity", "Import quantity"])][
["reporter_country", "partner_country", "item_code", "year", "element", "value"]
].copy()
for col in ("reporter_country", "partner_country"):
qty[col] = qty[col].astype(str)
exp = (
qty[qty["element"] == "Export quantity"]
.groupby(["reporter_country", "partner_country", "item_code", "year"], observed=True)["value"].sum()
.reset_index().rename(columns={"reporter_country": "exporter", "partner_country": "importer", "value": "v_exp"})
)
imp = (
qty[qty["element"] == "Import quantity"]
.groupby(["reporter_country", "partner_country", "item_code", "year"], observed=True)["value"].sum()
.reset_index().rename(columns={"reporter_country": "importer", "partner_country": "exporter", "value": "v_imp"})
)
flows = exp.merge(imp, on=["exporter", "importer", "item_code", "year"], how="outer")
has_exp, has_imp = flows["v_exp"].notna(), flows["v_imp"].notna()
flows["tonnage"] = flows["v_imp"].fillna(flows["v_exp"])
flows["status"] = np.where(has_exp & has_imp, "matched", np.where(has_exp, "exporter-only", "importer-only"))
year_total = flows.groupby("year", observed=True)["tonnage"].sum()
by_year = flows.groupby(["year", "status"], observed=True)["tonnage"].sum().reset_index()
by_year["share"] = by_year["tonnage"] / by_year["year"].map(year_total)
fig = px.bar(
by_year, x="year", y="share", color="status",
title="But most traded tonnage is reported by both countries<br><sup>Share of traded tonnage reported by both sides, only the exporter, or only the importer, by year</sup>",
labels={"year": "Year", "share": "Share of traded tonnes"},
category_orders={"status": ["matched", "exporter-only", "importer-only"]},
)
fig.update_layout(barmode="stack", yaxis_tickformat=".0%")
fig.show()
plot_reporting_coverage_tonnage_by_year(tb)
# Simple code to double-check that result for 2023 alone.
exporter_reports = tb[(tb["year"] == YEAR) & (tb["element"] == "Export quantity")].rename(columns={"reporter_country": "exporter", "partner_country": "importer"})
importer_reports = tb[(tb["year"] == YEAR) & (tb["element"] == "Import quantity")].rename(columns={"reporter_country": "importer", "partner_country": "exporter"})
exporter_reports_set = set(exporter_reports["exporter"].astype("string") + exporter_reports["importer"].astype("string") + exporter_reports["item"].astype("string"))
importer_reports_set = set(importer_reports["exporter"].astype("string") + importer_reports["importer"].astype("string") + importer_reports["item"].astype("string"))
# Complete set of item flows reported (by importers, exporters, or both).
all_flows_set = importer_reports_set | exporter_reports_set
# Share of all flows that are reported by importers only.
assert round(100 * len(importer_reports_set - exporter_reports_set) / len(all_flows_set)) == 31
# Share of all flows that are reported by exporters only.
assert round(100 * len(exporter_reports_set - importer_reports_set) / len(all_flows_set)) == 27
# Share of all flows that are reported by both importers and exporters:
assert round(100 * len(exporter_reports_set & importer_reports_set) / len(all_flows_set)) == 42
This flip is down to how concentrated trade is by weight. A handful of bulk staples make up most of the tonnage, and those large flows run between major traders that report them well (we look at which items in the next section).
Reporting by item¶
Of the 500+ items in the dataset, a handful of staples carry most of the trade by weight:
def plot_item_tonnage_concentration(tb, year, top_n=15):
"""Horizontal bar of the items with the largest share of all traded tonnage in a year.
A handful of bulk staples dominate, even though 500+ items are traded. Per-item tonnage uses
the importer-first flow value, consistent with the rest of the analysis."""
yr = tb[(tb["year"] == year) & tb["element"].isin(["Export quantity", "Import quantity"])].copy()
for col in ("reporter_country", "partner_country"):
yr[col] = yr[col].astype(str)
exp = (
yr[yr["element"] == "Export quantity"]
.groupby(["reporter_country", "partner_country", "item_code"], observed=True)["value"].sum()
.reset_index().rename(columns={"reporter_country": "exporter", "partner_country": "importer", "value": "v_exp"})
)
imp = (
yr[yr["element"] == "Import quantity"]
.groupby(["reporter_country", "partner_country", "item_code"], observed=True)["value"].sum()
.reset_index().rename(columns={"reporter_country": "importer", "partner_country": "exporter", "value": "v_imp"})
)
flows = exp.merge(imp, on=["exporter", "importer", "item_code"], how="outer")
flows["ton"] = flows["v_imp"].fillna(flows["v_exp"])
per_item = flows.groupby("item_code")["ton"].sum().sort_values(ascending=False)
total = per_item.sum()
names = dict(zip(yr["item_code"].astype(int), yr["item"].astype(str)))
top = per_item.head(top_n)
da = pd.DataFrame({"item": [names.get(int(c), str(c)) for c in top.index], "share": top.to_numpy() / total * 100}).iloc[::-1]
# Trade is highly concentrated: assert the top 10 items carry roughly half the tonnage.
assert 0.40 < per_item.head(10).sum() / total < 0.60
fig = px.bar(
da, x="share", y="item", orientation="h",
text=[f"{v:.1f}%" for v in da["share"]],
color="share", color_continuous_scale="Blues",
labels={"share": "Share of all traded tonnes (%)", "item": ""},
)
fig.update_traces(textposition="outside", cliponaxis=False)
fig.update_layout(
coloraxis_showscale=False, height=560, margin=dict(l=10, r=40, t=70, b=10),
title=dict(text=(
f"A few staples dominate food trade by weight ({year})<br>"
f"<sup>Share of all traded tonnes, by item. Just 10 of {len(per_item)} items make up "
f"{per_item.head(10).sum() / total:.0%} of the total.</sup>"
)),
)
fig.show()
plot_item_tonnage_concentration(tb, year=YEAR)
Reporting completeness also varies widely by item. Here we show the 15 best- and 15 worst-covered items, by the share of their flows that both countries report:
def plot_reporting_coverage_by_item(tb, year, n_worst=None, n_best=None):
"""Horizontal bar chart of the share of each selected item's bilateral flows that are
reported by both sides ("matched"), for the given year.
A low matched share means most of that item's flows are reported by only one country.
By default all selected items are shown, sorted worst-first. Pass n_worst and/or n_best
to keep only the n items with the lowest and/or highest matched share (e.g. n_worst=15,
n_best=15 shows the two extremes with the middle omitted)."""
code_to_display = selected_code_to_display
sub = tb[
(tb["year"] == year)
& tb["item_code"].isin(code_to_display)
& tb["element"].isin(["Export quantity", "Import quantity"])
].copy()
# Collapse to display-item level so combined items (e.g. bone-in + boneless beef) form a single bar.
sub["item"] = sub["item_code"].map(code_to_display)
sub["reporter_country"] = sub["reporter_country"].astype(str)
sub["partner_country"] = sub["partner_country"].astype(str)
cols = ["reporter_country", "partner_country", "item"]
exp_keys = sub.loc[sub["element"] == "Export quantity", cols].drop_duplicates()
# Swap reporter/partner on the import side so an export R->P matches the importer P reporting it.
imp_keys = (
sub.loc[sub["element"] == "Import quantity", cols]
.rename(columns={"reporter_country": "partner_country", "partner_country": "reporter_country"})
.drop_duplicates()
)
merged = exp_keys.merge(imp_keys, how="outer", indicator=True, on=cols)
matched = (
merged.groupby("item", observed=True)["_merge"]
.apply(lambda s: (s == "both").mean())
.sort_values(ascending=True)
)
total_items = len(matched)
# Keep the requested extremes (worst at the bottom, best at the top), or all items.
if n_worst or n_best:
parts = ([matched.head(n_worst)] if n_worst else []) + ([matched.tail(n_best)] if n_best else [])
shown = pd.concat(parts)
shown = shown[~shown.index.duplicated(keep="first")]
else:
shown = matched
if n_worst and n_best:
title = "Food items most and least often reported by both countries"
subtitle = f"From our selected list of items, {year}"
elif n_worst:
title = f"The {len(shown)} food items with the most one-sided trade reporting in {year}"
subtitle = "Share of each item's flows reported by both countries."
elif n_best:
title = f"The {len(shown)} food items with the most two-sided trade reporting in {year}"
subtitle = "Share of each item's flows reported by both countries."
else:
title = f"Share of each food item's trade flows reported by both sides in {year}"
subtitle = None
fig = px.bar(
x=shown.values,
y=shown.index,
orientation="h",
color=shown.values,
color_continuous_scale="RdYlGn",
range_color=(0, 1),
labels={"x": "Share of flows reported by both sides", "y": ""},
)
fig.update_traces(text=[f"{v:.0%}" for v in shown.values], textposition="outside", cliponaxis=False)
fig.update_layout(
title=dict(text=title if subtitle is None else f"{title}<br><sup>{subtitle}</sup>"),
xaxis_tickformat=".0%",
xaxis_range=[0, min(1.0, float(shown.max()) + 0.08)],
coloraxis_showscale=False,
height=max(400, 22 * len(shown)),
)
# When showing both extremes, mark the omitted middle with a dotted line.
if n_worst and n_best and (n_worst + n_best) < total_items:
fig.add_hline(y=n_worst - 0.5, line_dash="dot", line_color="gray")
fig.add_annotation(
x=float(shown.max()) * 0.6, y=n_worst - 0.5, yanchor="bottom",
text=f"{total_items - n_worst - n_best} items in between, not shown",
showarrow=False, font=dict(color="gray", size=11),
)
fig.show()
plot_reporting_coverage_by_item(tb, year=YEAR, n_worst=15, n_best=15)
Reporting by country¶
For each country, we can compute its reporting rate: the number of item flows reported by a country as a share of all item flows the country is involved in (as either exporter or importer).
As the chart below shows, in 2023, 46 countries traded food but reported nothing at all. This means that everything we know about their food trade comes from what their partners report. Examples of those countries are Russia, Bangladesh, Algeria, Iraq, Chile, and Iran.
Only one country reported more than 90% of their item flows (Netherlands).
And it is reasonable to expect that some important trade flows will be entirely missing from the dataset. For example, any trade between countries that report nothing will be ignored, e.g. any flow between Russia and Iran.
def country_reporting_rate(tb, year):
"""Per-country reporting rate for the given year, both by share of flows and by share of
tonnage. For each bilateral flow a country is involved in (as exporter or importer) we record
whether the country reported its own side, and the flow's displayed tonnage (the importer's
quantity, falling back to the exporter's). Computed over all items, not the selected list.
Returns one row per country with rate (by flows) and rate_tonnage."""
qty = tb[(tb["year"] == year) & tb["element"].isin(["Export quantity", "Import quantity"])][
["reporter_country", "partner_country", "item_code", "element", "value"]
].copy()
for col in ("reporter_country", "partner_country"):
qty[col] = qty[col].astype(str)
exp = (
qty[qty["element"] == "Export quantity"]
.groupby(["reporter_country", "partner_country", "item_code"], observed=True)["value"].sum()
.reset_index().rename(columns={"reporter_country": "exporter", "partner_country": "importer", "value": "v_exp"})
)
imp = (
qty[qty["element"] == "Import quantity"]
.groupby(["reporter_country", "partner_country", "item_code"], observed=True)["value"].sum()
.reset_index().rename(columns={"reporter_country": "importer", "partner_country": "exporter", "value": "v_imp"})
)
flows = exp.merge(imp, on=["exporter", "importer", "item_code"], how="outer")
flows["by_exp"] = flows["v_exp"].notna()
flows["by_imp"] = flows["v_imp"].notna()
flows["ton"] = flows["v_imp"].fillna(flows["v_exp"]) # displayed importer-first tonnage
# Aggregate from each country's perspective, as exporter and as importer.
ev = (
flows.assign(self_ton=flows["ton"].where(flows["by_exp"], 0.0))
.groupby("exporter", observed=True)
.agg(n_flows=("ton", "size"), n_self=("by_exp", "sum"), ton=("ton", "sum"), ton_self=("self_ton", "sum"))
.reset_index().rename(columns={"exporter": "country"})
)
iv = (
flows.assign(self_ton=flows["ton"].where(flows["by_imp"], 0.0))
.groupby("importer", observed=True)
.agg(n_flows=("ton", "size"), n_self=("by_imp", "sum"), ton=("ton", "sum"), ton_self=("self_ton", "sum"))
.reset_index().rename(columns={"importer": "country"})
)
pc = pd.concat([ev, iv]).groupby("country", as_index=False).sum()
pc["rate"] = pc["n_self"] / pc["n_flows"]
pc["rate_tonnage"] = pc["ton_self"] / pc["ton"]
return pc
def plot_country_reporting_rate_distribution(tb, year):
"""Histogram of country reporting rates for the given year.
For each country C, the reporting rate is the fraction of bilateral
flows involving C (as either exporter or importer) for which C itself
reported its own side of the flow. A rate near 0% means we know about C's
trade only because C's partners report it; a rate near 100% means C
publishes its own side of almost every flow it's involved in.
The "0%" bar is given its own bucket so the silent group stands out
from the rest of the (0, 100%] distribution. Bars are shaded red (low,
barely self-reporting) to green (high), and the two extremes are
annotated."""
pc = country_reporting_rate(tb, year)
# Build 11 bins: an exact-0% bucket plus ten 10%-wide buckets covering
# (0, 100%]. A tiny epsilon separates the 0% bucket from "(0, 10%]".
eps = 1e-9
bin_edges = [-eps, eps, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 + eps]
bin_labels = ["0%", "0-10%", "10-20%", "20-30%", "30-40%", "40-50%",
"50-60%", "60-70%", "70-80%", "80-90%", "90-100%"]
pc["bin"] = pd.cut(pc["rate"], bins=bin_edges, labels=bin_labels, include_lowest=True)
counts = pc.groupby("bin", observed=False).size().reset_index(name="n_countries")
# Representative rate per bin, used to shade each bar from red (low) to green (high).
bin_rate = dict(zip(bin_labels, [0.0, 0.05, 0.15, 0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85, 0.95]))
counts["rate"] = counts["bin"].map(bin_rate).astype(float)
# Numbers behind the callouts, computed so they stay correct across years.
n_silent = int((pc["rate"] == 0).sum())
# Only one country reports more than 90% of its own flows; assert it so we catch any change.
top_reporters = sorted(pc.loc[pc["rate"] > 0.9, "country"])
assert top_reporters == ["Netherlands"], f"Expected only the Netherlands above 90%, got {top_reporters}"
fig = px.bar(
counts,
x="bin",
y="n_countries",
text="n_countries",
color="rate",
color_continuous_scale="RdYlGn",
range_color=(0, 1),
category_orders={"bin": bin_labels},
labels={"bin": "Share of its own flows the country reports", "n_countries": "Number of countries"},
)
fig.update_traces(textposition="outside", cliponaxis=False)
fig.update_layout(
title=dict(text=(
f"Most countries report only part of their own food trade ({year})<br>"
f"<sup>Countries grouped by the share of their trade flows they report.</sup>"
)),
coloraxis_showscale=False,
showlegend=False,
bargap=0.15,
)
# Call out the two extremes, parking the text in the empty space beside each bar.
fig.add_annotation(
x="0%", y=n_silent,
text=f"<b>{n_silent} countries report nothing</b><br>we rely entirely on their partners",
showarrow=True, arrowhead=2, ax=40, ay=-15, xanchor="left", align="left",
bgcolor="rgba(255,255,255,0.85)",
)
fig.add_annotation(
x="90-100%", y=1,
text="<b>Only the Netherlands</b><br>reports over 90%",
showarrow=True, arrowhead=2, ax=-140, ay=-100, xanchor="center", align="center",
bgcolor="rgba(255,255,255,0.85)",
)
fig.show()
plot_country_reporting_rate_distribution(tb, year=YEAR)
# List of countries involved in at least one item flow, that have zero reported flows in 2023:
assert len((set(tb[(tb["year"] == YEAR)]["partner_country"]) | set(tb[(tb["year"] == YEAR)]["reporter_country"])) - set(tb[(tb["year"] == YEAR)]["reporter_country"]))
Everything above measures reporting by share of flows. Weighting by tonnage instead, as we did for overall coverage, changes the picture: once a country reports at all, it usually reports the bulk of its trade by volume. The large majority of countries report over 90% of their trade by tonnage, versus just one (the Netherlands) by share of flows, so the reporting gap is concentrated in the fully-silent countries rather than in partial reporting.
def plot_country_reporting_rate_by_tonnage(tb, year):
"""Same as the reporting-rate histogram, but each country's rate is by traded tonnage: the
share of its own trade tonnage (importer-first) that it reports itself, not the share of its
flows. The fully-silent countries stay at 0%, but most others move to the right."""
pc = country_reporting_rate(tb, year)
eps = 1e-9
bin_edges = [-eps, eps, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 + eps]
bin_labels = ["0%", "0-10%", "10-20%", "20-30%", "30-40%", "40-50%",
"50-60%", "60-70%", "70-80%", "80-90%", "90-100%"]
pc["bin"] = pd.cut(pc["rate_tonnage"], bins=bin_edges, labels=bin_labels, include_lowest=True)
counts = pc.groupby("bin", observed=False).size().reset_index(name="n_countries")
bin_rate = dict(zip(bin_labels, [0.0, 0.05, 0.15, 0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85, 0.95]))
counts["rate"] = counts["bin"].map(bin_rate).astype(float)
n_silent = int((pc["rate_tonnage"] == 0).sum())
n_over_90 = int((pc["rate_tonnage"] > 0.9).sum())
# By tonnage the large majority of reporters clear 90%; assert that so we catch any change.
assert n_over_90 > 100, f"Expected most countries above 90% by tonnage, got {n_over_90}"
fig = px.bar(
counts, x="bin", y="n_countries", text="n_countries",
color="rate", color_continuous_scale="RdYlGn", range_color=(0, 1),
category_orders={"bin": bin_labels},
labels={"bin": "Share of its own traded tonnage the country reports", "n_countries": "Number of countries"},
)
fig.update_traces(textposition="outside", cliponaxis=False)
fig.update_layout(
title=dict(text=(
f"By tonnage, most countries report nearly all of their own food trade ({year})<br>"
f"<sup>Countries grouped by the share of their traded tonnage they report.</sup>"
)),
coloraxis_showscale=False,
showlegend=False,
bargap=0.15,
)
fig.add_annotation(
x="0%", y=n_silent,
text=f"<b>{n_silent} countries still report nothing</b><br>(the same silent group as before)",
showarrow=True, arrowhead=2, ax=40, ay=-20, xanchor="left", align="left",
bgcolor="rgba(255,255,255,0.85)",
)
fig.add_annotation(
x="90-100%", y=n_over_90,
text=f"<b>{n_over_90} countries</b> report over 90%<br>of their traded tonnage",
showarrow=True, arrowhead=2, ax=-120, ay=-30, xanchor="center", align="center",
bgcolor="rgba(255,255,255,0.85)",
)
fig.show()
plot_country_reporting_rate_by_tonnage(tb, year=YEAR)
This also has a clear geography. Mapping each country's share of trade reported by tonnage shows where the real gaps are: most of the world reports its own trade, with the fully-silent countries standing out in red.
def plot_country_reporting_rate_map(tb, year):
"""Choropleth of each country's reporting rate by tonnage: the share of its own traded
tonnage it reports, over all items. Most of the world is green; the fully-silent countries
stand out in red. Countries not in the dataset are left grey."""
pc = country_reporting_rate(tb, year)
pc["iso3"] = pc["country"].map(country_to_iso3)
# Bin the rate into five intervals, shaded red (low) to green (high).
eps = 1e-9
bands = ["0-20%", "20-40%", "40-60%", "60-80%", "80-100%"]
pc["band"] = pd.cut(pc["rate_tonnage"], bins=[-eps, 0.2, 0.4, 0.6, 0.8, 1.0 + eps], labels=bands, include_lowest=True)
colors = px.colors.sample_colorscale("RdYlGn", [0.1, 0.3, 0.5, 0.7, 0.9])
fig = px.choropleth(
pc,
locations="iso3",
color="band",
hover_name="country",
custom_data=["rate_tonnage"],
category_orders={"band": bands},
color_discrete_map=dict(zip(bands, colors)),
projection="robinson", # match OWID's world-map projection
)
fig.update_traces(hovertemplate="%{hovertext}: %{customdata[0]:.0%}<extra></extra>")
fig.update_layout(
title=dict(text=(
f"How much of their own food trade each country reports, by tonnage ({year})<br>"
f"<sup>Share of a country's traded tonnage for which it reports its own side, over all items.</sup>"
)),
margin=dict(l=0, r=0, t=70, b=0),
legend=dict(orientation="h", yanchor="top", y=0.05, xanchor="center", x=0.5, title_text="Share reported (by tonnage)"),
)
# Crop Antarctica, as OWID's maps do.
fig.update_geos(showframe=False, showcoastlines=False, lataxis_range=[-58, 85])
fig.show()
plot_country_reporting_rate_map(tb, year=YEAR)
Quantity agreement¶
For item flows that were reported both ways, how closely do the two reported quantities agree? Let's define an agreement ratio as:
100 x min(exported quantity, imported quantity) / max(exported quantity, imported quantity)
This means that 100% is a perfect match (i.e. both countries report exactly the same quantity), and 50% means one country reports twice the other.
The following chart shows the agreement ratio for the top food trading countries.
def plot_quantity_mismatch_by_reporter(tb, year=None, top_n=20, include_unmatched=False, by_tonnage=False):
"""Stacked horizontal bar showing how each country's reports distribute
across quantity-agreement bands, for the top-N reporting countries.
A *matched* flow is one where both the exporter and the importer filed a
report. Agreement = min(exp_qty, imp_qty) / max(exp_qty, imp_qty). 100%
= perfect match, 50% = one side reports 2x the other. Bins into five bands.
The chart aggregates *by the country that filed the report* (the FAOSTAT
`reporter_country` column). Each matched flow therefore contributes two
data points — one to the exporter's distribution and one to the importer's
— both carrying the same agreement value.
When include_unmatched=True, flows reported by only one side land in a
sixth Unmatched band, attributed to the single reporting country."""
if year is None:
rows_per_year = tb.groupby("year", observed=True).size()
year = int(rows_per_year[rows_per_year >= 0.9 * rows_per_year.max()].index.max())
sub = tb[(tb["year"] == year) & tb["element"].isin(["Export quantity", "Import quantity"])][
["reporter_country", "partner_country", "item_code", "element", "value"]
].copy()
for col in ("reporter_country", "partner_country"):
sub[col] = sub[col].astype(str)
# Canonical flow key: (exporter, importer, item_code) for every row.
# Export rows: exporter = reporter, importer = partner.
# Import rows: exporter = partner, importer = reporter.
is_imp = (sub["element"] == "Import quantity").to_numpy()
rep = sub["reporter_country"].to_numpy()
par = sub["partner_country"].to_numpy()
exporter = np.where(is_imp, par, rep)
importer = np.where(is_imp, rep, par)
item_code = sub["item_code"].astype(str).to_numpy()
sub["flow_key"] = [f"{e}|{i}|{c}" for e, i, c in zip(exporter, importer, item_code)]
side_counts = sub.groupby("flow_key").size()
matched_keys = side_counts[side_counts == 2].index
matched_rows = sub[sub["flow_key"].isin(matched_keys)].copy()
# For each matched flow, the agreement is the same for both of its rows
# (since min/max is computed per flow_key).
matched_rows["v_min"] = matched_rows.groupby("flow_key")["value"].transform("min")
matched_rows["v_max"] = matched_rows.groupby("flow_key")["value"].transform("max")
with np.errstate(divide="ignore", invalid="ignore"):
agreement = np.where(matched_rows["v_max"] > 0,
matched_rows["v_min"] / matched_rows["v_max"], 0.0)
matched_rows["agreement"] = agreement
matched_band_labels = ["<25%", "25-50%", "50-75%", "75-90%", ">=90%"]
matched_band_colors = ["#d73027", "#fc8d59", "#fee08b", "#91cf60", "#1a9850"]
matched_rows["band"] = pd.cut(
matched_rows["agreement"],
bins=[-0.001, 0.25, 0.50, 0.75, 0.90, 1.001],
labels=matched_band_labels,
)
if include_unmatched:
unmatched_keys = side_counts[side_counts == 1].index
unmatched_rows = sub[sub["flow_key"].isin(unmatched_keys)].copy()
unmatched_rows["band"] = "Unmatched"
all_rows = pd.concat([
matched_rows[["reporter_country", "band", "value"]],
unmatched_rows[["reporter_country", "band", "value"]],
], ignore_index=True)
band_labels = ["Unmatched"] + matched_band_labels
band_colors = ["#67000d"] + matched_band_colors
else:
all_rows = matched_rows[["reporter_country", "band", "value"]].copy()
band_labels = matched_band_labels
band_colors = matched_band_colors
all_rows["band"] = pd.Categorical(all_rows["band"], categories=band_labels)
# Rank reporters by total tonnes they reported this year (matched + unmatched-by-them rows).
# This matches the natural intuition of "biggest traders" better than report count.
reporter_tonnes = all_rows.groupby("reporter_country", observed=True)["value"].sum().sort_values(ascending=False) # ty: ignore[missing-argument]
top = reporter_tonnes.head(top_n).index.tolist() if top_n is not None else reporter_tonnes.index.tolist()
sub_flat = all_rows[all_rows["reporter_country"].isin(top)]
grouped = sub_flat.groupby(["reporter_country", "band"], observed=True)
shares = (grouped["value"].sum() if by_tonnage else grouped.size()).unstack(fill_value=0)
shares = shares.reindex(columns=band_labels, fill_value=0)
shares = shares.div(shares.sum(axis=1), axis=0)
shares = shares.sort_values(">=90%", ascending=True)
long = shares.reset_index().melt(id_vars="reporter_country", var_name="band", value_name="share")
title = f"Quantity agreement among matched flows{', by tonnage,' if by_tonnage else ''} in {year}"
if by_tonnage:
x_label = "Share of the country's matched-flow tonnage"
else:
x_label = ("Share of all reports filed by the country" if include_unmatched
else "Share of matched-flow reports filed by the country")
fig = px.bar(
long, x="share", y="reporter_country", color="band", orientation="h",
title=title,
labels={"share": x_label, "reporter_country": "Reporter"},
category_orders={"band": band_labels, "reporter_country": shares.index.tolist()},
color_discrete_sequence=band_colors,
)
fig.update_layout(
barmode="stack",
xaxis_tickformat=".0%",
height=max(400, 18 * len(shares)),
yaxis=dict(tickmode="linear", dtick=1),
)
fig.show()
plot_quantity_mismatch_by_reporter(tb, year=2023, top_n=30, include_unmatched=False)
For Brazil, for example, we see that only around 43% of those matched flows agreed with each other well (the reported quantity matched between 75 and 100%).
This disagreement is surprisingly large and common.
# Simple code to double-check that result for 2023 alone.
# Define a unique identifier of each item flow, namely [importer country][exporter country][item]
t = tb[(tb["year"]==YEAR)].reset_index(drop=True)
t["id"] = ""
mask_imports = (t["element"] == "Import quantity")
t.loc[mask_imports, "id"] = t[mask_imports]["reporter_country"].astype("string") + t[mask_imports]["partner_country"].astype("string") + t[mask_imports]["item"].astype("string")
t.loc[~mask_imports, "id"] = t[~mask_imports]["partner_country"].astype("string") + t[~mask_imports]["reporter_country"].astype("string") + t[~mask_imports]["item"].astype("string")
assert t["id"].notnull().all()
# Select item flows reported by both countries in 2023.
t = t[t.groupby("id")["id"].transform("count")==2].reset_index(drop=True)
# Add columns for the minimum and maximum values of each item flow.
t["value_min"] = t.groupby("id")["value"].transform("min")
t["value_max"] = t.groupby("id")["value"].transform("max")
# Define the quantity agreement ratio.
t["ratio"] = t["value_min"] / t["value_max"]
# Percentage of item flows reported by Brazil that have an agreement ratio >=90%:
mask_country = t["reporter_country"] == "Brazil"
# print(f'<25%: {100 * len(t[(mask_country) & (t["ratio"]<0.25)]) / len(t[mask_country]):.0f}')
# print(f'25%-50%: {100 * len(t[(mask_country) & (t["ratio"]>=0.25) & (t["ratio"]<0.50)]) / len(t[(mask_country)]):.0f}')
# print(f'50%-75%: {100 * len(t[(mask_country) & (t["ratio"]>=0.50) & (t["ratio"]<0.75)]) / len(t[(mask_country)]):.0f}')
# print(f'75%-90%: {100 * len(t[(mask_country) & (t["ratio"]>=0.75) & (t["ratio"]<0.90)]) / len(t[(mask_country)]):.0f}')
# print(f'>=90%: {100 * len(t[(mask_country) & (t["ratio"]>=0.90)]) / len(t[(mask_country)]):.0f}')
Weighting each flow by its tonnage, the large flows agree far better. So the severe disagreement is concentrated in small shipments; most of the tonnage matches closely.
plot_quantity_mismatch_by_reporter(tb, year=2023, top_n=30, include_unmatched=False, by_tonnage=True)
Estimating production and supply¶
To put trade figures in context, we can express them in relative terms:
- exports as a share of what a country produces, and
- imports as a share of domestic supply (the amount of tonnes that are available for consumption in the country).
However, production and domestic supply are not available in the TM dataset. So we need to combine it with other FAOSTAT datasets, of which there are three candidates:
- Crops and livestock products (QCL) is FAOSTAT's headline production dataset, but it only contains production-side elements (production, area harvested, yield, animal numbers). It has no trade information, so domestic supply cannot be computed from it at all. It also mostly tracks primary production: for 2023 it covers only 43% of the items traded in TM, accounting for 63% of traded tonnage.
- Food Balances (FBS) conveniently provides production, domestic supply, imports, and exports directly.
However, its items are aggregated groups that do not map cleanly to TM or QCL item codes.
For example, FBS' "Cocoa Beans and products" bundles several items (
661 Cocoa beans, 662 Cocoa paste, 665 Cocoa powder & cake, 666 Chocolate products), and properly disaggregating FBS commodities is not trivial (see e.g. Zhao et al. (2025)). - Supply Utilization Accounts (SCL) contains production, imports, exports, and stock variation for each item, country and year. It covers 78% of the items traded in TM, accounting for 91% of traded tonnage, and conveniently uses the same item codes as TM.
We therefore estimate production and supply by combining the TM and SCL datasets. The main downside of SCL is country coverage: it omits ten countries entirely (including Japan and Singapore) that QCL does cover.
# Double-check the QCL vs SCL coverage numbers quoted above (kept as silent assertions).
tm_year = tb[tb["year"] == YEAR].copy()
tm_year["item_code"] = tm_year["item_code"].astype(int)
tm_tonnage_by_item = pd.Series(tm_year.groupby("item_code", observed=True)["value"].sum())
item_codes_tm = set(tm_tonnage_by_item.index)
total_tonnage = tm_tonnage_by_item.sum()
item_codes_scl = set(tb_scl[tb_scl["year"] == YEAR]["item_code"].astype(str).astype(int))
with contextlib.redirect_stdout(io.StringIO()):
tb_qcl = fetch("garden/faostat/2026-02-25/faostat_qcl/faostat_qcl").reset_index()
# QCL has no trade elements: production-side only (its "Stocks" element is live-animal headcounts).
assert not {"Import quantity", "Export quantity", "Stock Variation"} & set(tb_qcl["element"].astype(str).unique())
item_codes_qcl = set(tb_qcl[(tb_qcl["year"] == YEAR) & (tb_qcl["element"].astype(str) == "Production")]["item_code"].astype(str).astype(int))
def _coverage(item_codes):
share_items = len(item_codes_tm & item_codes) / len(item_codes_tm)
share_tonnage = tm_tonnage_by_item[tm_tonnage_by_item.index.isin(item_codes)].sum() / total_tonnage
return share_items, share_tonnage
qcl_items, qcl_tonnage = _coverage(item_codes_qcl)
scl_items, scl_tonnage = _coverage(item_codes_scl)
# Verify the coverage figures quoted in the text above: QCL ~43% of items / ~63% of tonnage, SCL ~78% / ~91%.
assert 0.40 < qcl_items < 0.46 and 0.55 < qcl_tonnage < 0.70
assert 0.75 < scl_items < 0.81 and 0.85 < scl_tonnage < 0.95
Domestic supply is not directly available in the TM or SCL datasets, but we can compute it from SCL ourselves, via the Food Balance Sheet identity (the same accounting framework that underlies FBS):
Production + Imports − Exports − Stock variation
Here, Stock variation is defined as closing minus opening stocks, so stock accumulation reduces the supply available for domestic use.
Consistency between TM and SCL trade reports¶
Our visualization combines the two datasets: trade flows come from TM, production comes from SCL, and domestic supply is calculated based on SCL data.
We'll now check whether the two datasets agree where they overlap.
Both report each country's total imports and exports per item, so we compare SCL against each country's own TM reports (what the country itself declared, summed over all partners).
# Build the comparison: own-reported TM totals vs SCL, per (country, item), for YEAR.
tm_own = (
tb[(tb["year"] == YEAR) & tb["element"].isin(["Import quantity", "Export quantity"])]
.groupby(["reporter_country", "item_code", "element"], observed=True)["value"]
.sum()
.unstack("element")
.reset_index()
.rename(columns={"reporter_country": "country", "Import quantity": "tm_imports", "Export quantity": "tm_exports"})
)
tm_own["country"] = tm_own["country"].astype(str)
tm_own["item_code"] = tm_own["item_code"].astype(int)
scl_trade = tb_scl[
(tb_scl["year"] == YEAR)
& (tb_scl["unit_short_name"] == "t")
& tb_scl["element"].astype(str).isin(["Import quantity", "Export quantity"])
].copy()
scl_trade["element"] = scl_trade["element"].astype(str)
scl_trade["country"] = scl_trade["country"].astype(str)
scl_trade["item_code"] = scl_trade["item_code"].astype(str).astype(int)
scl_trade = (
scl_trade.groupby(["country", "item_code", "element"], observed=True)["value"]
.sum()
.unstack("element")
.reset_index()
.rename(columns={"Import quantity": "scl_imports", "Export quantity": "scl_exports"})
)
trade = tm_own.merge(scl_trade, on=["country", "item_code"], how="outer")
# Restrict to individual countries (drop FAO region aggregates).
trade = trade[~trade["country"].str.contains(r"\(FAO\)|World|countries|Union", case=False, regex=True)].reset_index(drop=True)
Where both datasets report a figure, they agree closely: within ~10% for about 94% of import tonnage and 97% of export tonnage.
def tm_scl_agreement(trade, side):
t, s = trade[f"tm_{side}"], trade[f"scl_{side}"]
both = t.notna() & s.notna() & (t > 0) & (s > 0)
pair = pd.concat([t[both], s[both]], axis=1)
ratio = pair.min(axis=1) / pair.max(axis=1)
weight = pair.max(axis=1)
return both, ratio, weight
# Verify the claims quoted above: TM and SCL agree within 10% for ~94% of import tonnage and ~97% of export tonnage.
_, ratio_i, weight_i = tm_scl_agreement(trade, "imports")
_, ratio_e, weight_e = tm_scl_agreement(trade, "exports")
assert 0.92 < weight_i[ratio_i >= 0.9].sum() / weight_i.sum() < 0.97
assert 0.95 < weight_e[ratio_e >= 0.9].sum() / weight_e.sum() < 0.99
# Visual check: own-reported TM imports vs SCL imports per (country, item), log-log.
sample = trade[(trade["tm_imports"] > 0) & (trade["scl_imports"] > 0)].sample(5000, random_state=1)
fig = px.scatter(
sample,
x="tm_imports",
y="scl_imports",
hover_data=["country", "item_code"],
log_x=True,
log_y=True,
opacity=0.3,
title=f"Imports per (country, item) in {YEAR}: own TM reports vs SCL (sample of 5,000 pairs)",
)
fig.add_trace(go.Scatter(x=[1e-2, 1e8], y=[1e-2, 1e8], mode="lines", line=dict(dash="dash", color="gray"), name="y = x"))
fig.show()
Where they disagree, it is almost always because SCL has no figure at all. Items SCL does not track (around 9% of own-reported tonnage on each side), and ten countries it does not cover at all (including Japan and Singapore, two large importers).
The reverse also happens: SCL reports trade for many country-item pairs that are not in TM, because SCL fills non-reporters with partner (mirror) data.
scl_items = set(scl_trade["item_code"])
scl_countries = set(scl_trade["country"])
# Verify the decomposition of TM tonnage that has no SCL counterpart (claims quoted above).
for side in ["imports", "exports"]:
t, s = trade[f"tm_{side}"], trade[f"scl_{side}"]
total = t.fillna(0).sum()
blank = (t > 0) & s.isna()
item_untracked = blank & ~trade["item_code"].isin(scl_items)
element_blank = blank & trade["item_code"].isin(scl_items) & trade["country"].isin(scl_countries)
# Items SCL doesn't track account for ~9% of own-reported tonnage; tracked-pair blanks are negligible.
assert 0.06 < t[item_untracked].sum() / total < 0.12
assert t[element_blank].sum() / total < 0.005
# Ten TM reporters are entirely absent from SCL, including Japan and Singapore.
tm_reporters = set(trade.loc[trade["tm_imports"].notna() | trade["tm_exports"].notna(), "country"])
absent = sorted(tm_reporters - scl_countries)
assert "Japan" in absent and "Singapore" in absent and len(absent) == 10
This matters for our flow table, which uses the importer's quantity and falls back to the exporter's when the importer is silent. Those fallback ("mirror-only") flows are trade the importing country never declared anywhere, so SCL has no record of them either: about a quarter of flows, though only ~9% of tonnage. When relating these flows to SCL-derived supply, a blank on the SCL side must therefore not be read as a zero.
# Mexico barley (item_code 44): own reports vs mirror reports.
mexico_own = tb[
(tb["year"] == YEAR)
& (tb["reporter_country"].astype(str) == "Mexico")
& (tb["item_code"].astype(int) == 44)
& (tb["element"] == "Import quantity")
]["value"].sum()
mexico_mirror = tb[
(tb["year"] == YEAR)
& (tb["partner_country"].astype(str) == "Mexico")
& (tb["item_code"].astype(int) == 44)
& (tb["element"] == "Export quantity")
]["value"].sum()
assert mexico_own == 0 and mexico_mirror > 400_000
# Mirror-only share of the food_trade table (the dataset produced by our garden step).
tb_ft = tb_food_trade
importer_reported = tb[
(tb["year"] == YEAR) & (tb["element"] == "Import quantity") & (tb["value"] > 0)
].copy()
importer_reported["importer"] = importer_reported["reporter_country"].astype(str)
importer_reported["exporter"] = importer_reported["partner_country"].astype(str)
reported_keys = set(zip(importer_reported["importer"], importer_reported["exporter"], importer_reported["item_code"].astype(int)))
# A display item may combine several FAO codes (see COMBINED_ITEMS), and the food_trade table
# carries only the synthetic id for those, so map each item back to its underlying FAO codes. A
# flow counts as importer-reported when any of the item's codes is reported for that (importer,
# exporter) pair.
display_to_codes = {}
for code, item in selected_code_to_display.items():
display_to_codes.setdefault(item, set()).add(code)
is_mirror = pd.Series(
[
not any((importer, exporter, c) in reported_keys for c in display_to_codes[item])
for importer, exporter, item in zip(
tb_ft["importer"].astype(str), tb_ft["exporter"].astype(str), tb_ft["item"].astype(str)
)
],
index=tb_ft.index,
)
share_flows = is_mirror.mean()
share_tonnage = tb_ft.loc[is_mirror, "value"].sum() / tb_ft["value"].sum()
assert 0.15 < share_flows < 0.35 and 0.05 < share_tonnage < 0.15
Validating domestic supply against trade¶
Because we estimate domestic supply from SCL's components, and imports are one of them, a country that under-records its imports ends up with too low a supply, and imports look like an impossibly large share of it. Mexican barley in 2023 is a clean case, SCL records production but no imports (supply ~0.9 million tonnes), while partners report ~0.5 million tonnes shipped in, so trusting SCL would put imports at 58% of supply instead of the true ~37%. The trade matrix is an independent read on each country's imports, so we keep SCL's supply only where its recorded imports broadly match the inbound trade we observe, and blank it otherwise; the trade data decides whether to trust the supply, it never changes it.
# Build the bilateral flow table for the chosen year: importer's report vs exporter's report per flow.
fy = tb[tb["year"] == YEAR]
cols = ["reporter_country", "partner_country", "item_code", "value"]
exp_f = fy[fy["element"] == "Export quantity"][cols].rename(
columns={"reporter_country": "exporter", "partner_country": "importer", "value": "v_exp"}
)
imp_f = fy[fy["element"] == "Import quantity"][cols].rename(
columns={"reporter_country": "importer", "partner_country": "exporter", "value": "v_imp"}
)
flows = exp_f.merge(imp_f, on=["exporter", "importer", "item_code"], how="outer")
# Coverage of domestic supply that survives the trade cross-check, as a function of the threshold.
flows["value"] = flows["v_imp"].fillna(flows["v_exp"])
fl = flows[flows["value"] > 0]
tm_in = fl.groupby(["importer", "item_code"], observed=True)["value"].sum().reset_index().rename(columns={"importer": "country", "value": "tm_in"})
tm_out = fl.groupby(["exporter", "item_code"], observed=True)["value"].sum().reset_index().rename(columns={"exporter": "country", "value": "tm_out"})
comp = ["Production", "Import quantity", "Export quantity", "Stock Variation"]
sc = tb_scl[(tb_scl["year"] == YEAR) & (tb_scl["unit_short_name"] == "t") & tb_scl["element"].astype(str).isin(comp)].copy()
sc["country"] = sc["country"].astype(str)
sc["item_code"] = sc["item_code"].astype(str).astype(int)
sc["element"] = sc["element"].astype(str)
sw = sc.groupby(["country", "item_code", "element"], observed=True)["value"].sum().unstack("element").reset_index()
sw["supply"] = sw["Production"].fillna(0) + sw["Import quantity"].fillna(0) - sw["Export quantity"].fillna(0) - sw["Stock Variation"].fillna(0)
sw = sw.merge(tm_in, on=["country", "item_code"], how="left").merge(tm_out, on=["country", "item_code"], how="left")
sw["tm_in"] = sw["tm_in"].fillna(0.0)
sw["tm_out"] = sw["tm_out"].fillna(0.0)
sw = sw[sw["supply"] > 0]
si = sw["Import quantity"].fillna(0).to_numpy(float)
se = sw["Export quantity"].fillna(0).to_numpy(float)
tin = sw["tm_in"].to_numpy(float)
tout = sw["tm_out"].to_numpy(float)
thr = np.linspace(0, 1, 51)
def kept(scl_flow, observed, t):
return (observed == 0) | (scl_flow >= t * observed)
rows = []
for t in thr:
ki = kept(si, tin, t); ke = kept(se, tout, t); kb = ki & ke
for label, mask in [("imports", ki), ("exports", ke), ("both", kb)]:
rows.append({"threshold": t * 100, "check": label, "tonnage kept": tin[mask].sum() / tin.sum() * 100})
cov = pd.DataFrame(rows)
fig = px.line(cov, x="threshold", y="tonnage kept", color="check",
title=f"Domestic-supply coverage surviving the trade cross-check ({YEAR}, all items)",
labels={"threshold": "SCL flow must cover ≥ this % of observed trade", "tonnage kept": "% of inbound tonnage with supply"})
fig.add_vline(x=90, line_dash="dash", line_color="gray", annotation_text="chosen: 90%")
fig.update_yaxes(range=[0, 101])
fig.show()
We set the bar at 90%: where both sources are essentially the importer's own customs data they agree within about 10% anyway, so a larger gap signals a genuine hole rather than ordinary noise. We test imports only, since for a big net importer a trivial export mismatch would otherwise veto a crucial supply figure (China imported ~103 million tonnes of soybeans in 2023 but exported just 71 thousand).
Combining items¶
A certain food item is sometimes traded in different derived forms, e.g. bone-in vs boneless meat, in-shell vs shelled nuts, paddy vs milled rice, raw vs refined sugar. For this reason, the TM dataset often splits one common product, e.g. almonds, across several items, e.g. 'Almonds, in shell', 'Almonds, shelled'.
But this split is often a technicality, rather than a meaningful distinction. We therefore combine these derived forms to be able to have a complete picture of the trade of those food items.
To do that, we sum traded quantities across all their derived forms. However, for these combined items we do not estimate their production or domestic supply, because adding up the production of derived forms could lead to double-counting.
The charts below show the items we have combined, and the relative share of each derived forms across all item flows.
# Group the combined items by commodity family so each chart shares one set of FAO forms.
ITEM_GROUPS = {
"Meat (bone-in vs. boneless)": ["Beef", "Pork"],
"Nuts (in-shell vs. shelled)": ["Almonds", "Cashews", "Walnuts", "Hazelnuts", "Brazil nuts", "Groundnuts"],
"Rice (milled, broken, husked)": ["Rice"],
"Sugar (raw vs. refined)": ["Sugar"],
}
# Global trade per FAO code (2023): the larger of the two reported directions, summed to the world total.
year_t = tb[(tb["year"] == YEAR) & (tb["unit"] == "t")]
world = year_t[year_t["element"].isin(["Export quantity", "Import quantity"])]
by_code = world.groupby(["item_code", "element"], observed=True)["value"].sum().unstack("element").max(axis=1)
def split_frame(items):
rows = []
for item in items:
members = COMBINED_ITEMS[item]
total = sum(float(by_code.get(code, 0.0)) for code in members)
for code, form in members.items():
rows.append({"item": item, "form": form, "share": 100 * float(by_code.get(code, 0.0)) / total,
"mt": float(by_code.get(code, 0.0)) / 1e6, "primary": code == next(iter(members))})
return pd.DataFrame(rows)
for group, items in ITEM_GROUPS.items():
frame = split_frame(items)
# Order items by the primary code's share, so the split reads as a gradient.
order = frame[frame["primary"]].sort_values("share", ascending=False)["item"].tolist()
fig = px.bar(
frame, x="share", y="item", color="form", orientation="h",
category_orders={"item": order},
hover_data={"mt": ":.2f", "share": ":.1f", "primary": False},
labels={"share": "Share of the commodity's traded tonnes (%)", "item": "", "form": "FAO form"},
title=f"{group}: how trade splits across FAO codes, {YEAR}",
)
fig.update_layout(height=200 + 55 * len(items), width=850, barmode="stack", legend_title_text="FAO form")
fig.show()