LocaleWatchWhat breaks in Shopify translations: 27,140 product pages measured
We compared 27,140 product translations against their source text across 48 real multilingual Shopify stores, then verified samples by hand on the live storefronts. This page is the public record: what actually breaks, how we kept the false-alarm rate down, and — just as useful — which suspected defects turned out not to exist. If you write about Shopify localisation, these numbers are free to cite with a link.
Method
Shopify storefronts expose their catalogue per language; for each store we read every published language and compared each product's translated title and description against the source text, at one polite request per second. Every defect class below carries thresholds tuned on this corpus — each threshold exists to kill a specific false alarm we met — and findings were sampled and re-checked on the live pages before we trusted a store's results. The sampling matters more than it sounds; see "the feed can disagree with the page" below.
The defect classes and their thresholds
| Untranslated product | title and body both identical to the source; only counted when the source body is ≥200 characters, so short texts cannot match by accident |
| Blank translation | source ≥200 characters, stored translation under 20 — the shopper gets an empty description |
| Translated title, untranslated body | body identical to source, title translated; no ratio threshold — whoever translated the title meant to localise this product (measured separately: 129 cases) |
| Partial or stale translation | source ≥300 characters and the translation under 60% of its weighted length |
| Source outgrew the translation | weighted length ratio above 1.8 — the source was edited, the translation never caught up (why Shopify's own flag can miss this) |
| Structure loss | three or more structural elements — list items, table rows and cells, headings, images, links — present in the source and missing from the translation |
| Lost measurements | more than half of the number-plus-unit tokens missing from the translation, and at least two of them |
Each guard exists because the naive version fired falsely on real stores. One extra paragraph tag is not corruption, so only structural containers count and the floor is three. Decimal separators differ by locale — 3,5 cm in French is 3.5 cm in English — so tokens are normalised before "lost measurement" may fire.
Raw character counts lie across writing systems
The subtlest false alarm in the corpus: a complete, faithful Chinese translation of an English description is roughly a third of its character count, because one Han character carries about as much text as three Latin ones. Compared by raw length, complete Chinese translations screamed "partial": we hand-read 42 full CJK pairs, and by raw length all 42 would have been flagged. We used those pairs to calibrate weights (Han 3.0, Kana 2.0, Hangul 2.4, everything else 1.0) and every length comparison now runs in that unit. Scripts we have not measured keep weight 1.0: no threshold without a measurement.
What never happened
Two defects we expected to find, and did not:
- Broken translated URLs. Translating product handles changes URLs, so we assumed some would break. We found 763 translated handles and live-tested 72 of them: all 72 answered 200. Shopify percent-encodes even ® and Japanese characters and serves them. A syntax check is not a brokenness check; we removed the class entirely.
- SKU corruption. Zero cases in 27,140 comparisons.
Negative results are the cheapest thing to leave unpublished and the most expensive for everyone else to re-discover. These two cost us real measurement time; take them for free.
The feed can disagree with the page
In an early 12-store sample, 3 stores showed untranslated descriptions in the catalogue feed while the live pages were translated: their themes render the description from app-managed metadata rather than the standard body field. This is why every scan verifies on the live storefront before it reports. The store-level pattern is strong: in a later 24-store scan, every store where sampled verification reached a decision came out either 100% confirmed or 0% — 16 stores of 16, none in between. A theme renders the body field or it does not; so we sample a few pages per store and trust or discard the store's whole block accordingly.
Assorted numbers a scanner has to respect
- In 27 of 70 stores, at least one language lives at a path that is not
its language code —
/pt/serving Brazilian Portuguese, for example. Guessing/{language}/404s on real stores; the store's own hreflang map is the only honest source of language URLs. - hreflang inflates with country variants — one store carried 200+ hreflang rows for its languages. Deduplicate by language path before counting anything.
- Roughly a quarter of the stores we scanned (~26%) do not expose a public catalogue feed at all; an outside-in scan has to say "could not read" there, which is a different statement from "clean".
- A language that exists but is not published returns an empty translation set through the API — byte-identical to "nothing translated". We measured a case where scanning unpublished languages would have manufactured 26 false findings on one store. Filter to published languages first.
- A language can be published and attached to no market — it then serves nobody and leaves no external trace at all (measured separately).
- Blank translations are real but rarer than untranslated ones: 39 findings across the 24-store scan.
The single most useful sentence
Almost everything above is invisible from inside the admin, because Shopify's fallback design makes a missing translation render as a normal page in the wrong language (the fallback, explained). A mismatch is evidence, not a verdict — but you cannot weigh evidence you have never seen, and nothing in the default tooling shows you this particular kind.
Related measurements
Check your own store in one minute.
Our free scanner reads your public storefront in every published language and lists the products where the translation stopped matching the source — with links to your own pages, so you judge the evidence yourself. No install, no login.
To hear about it first — a weekly watch with an email when a translation breaks — that's LocaleWatch on the Shopify App Store.