What Machine Translation Does to a Psychological Test Item

A localization pipeline can be green on every check it knows how to run and still be shipping the wrong test. That is not a hypothetical for us. Our item sets in German, Spanish, French, Portuguese and Ukrainian were produced by machine translation with review — that is their origin, and it has not changed. For a long stretch the only evidence we had that those locales were sound was the kind a build script can produce: every item id had a string in every language, no keys missing, nothing falling back to English at render time. In September 2026 we did the thing a script cannot do and read them. Every item in fourteen test banks was read in each of the six languages it ships in — 624 items, 3,744 recorded reads — and what came back was not a list of clumsy phrasings. In three separate banks, the translated item was asking a different question from the English one.

The French Social Skills items were asking something else

The worst finding was the cleanest. On 35 of the 36 items in our social skills test, the French text was asking a different question from the English. Not stiff, not unidiomatic, not a register mismatch — a different question. A respondent answering the French form in good faith was answering honestly about something the instrument was not built to score, and their answer was then fed into the same scoring path as everyone else's. An item that asks something else is not a degraded version of the item; it is a different item wearing the same id, and every statistic computed over it inherits the substitution silently.

What makes this defect so dangerous is that it is invisible from every direction a normal engineering process looks. The string was present, fluent and grammatical. It rendered correctly, it fit the layout, it passed the completeness check, and a French-speaking reader skimming the page would have found nothing obviously wrong — the item reads as a perfectly reasonable question. It was simply not the question the scale was measuring. If you have ever argued that "we have translations for every item" and "the test asks the same thing in every language" are the same claim, this is the counterexample.

Grammatical gender decided who the test was talking to

The second defect was structural rather than semantic, and it came from the grammar of the target languages rather than from any single mistranslation. In the Spanish, French and Portuguese versions of our Enneagram test, 21 of 55 items asserted that the reader was a man. Romance-language adjectives and participles have to agree with the subject, machine translation resolves an ungendered English "you" by defaulting to the masculine, and the result is an item that grammatically addresses a man. A woman taking the test in Spanish was being told, in the text of the item itself, what she was — 21 times over a 55-item form.

That is a content defect and also a measurement one. Self-report items work by asking someone to recognise themselves in a statement, and an item that misidentifies the reader in its first three words is interfering with exactly that mechanism. You cannot know in advance whether the effect is a shrug or a systematically different answer, and that uncertainty is the problem: the item no longer behaves the same way for every respondent. Note too that English hid this entirely. There is nothing to find in the source language, so no amount of care with the English items would have surfaced it — the defect is created by the translation and lives only in the target locales.

Second person broke a first-person instrument

The third defect ran through an entire bank. All 288 items in our character strengths test were phrased in the second person — "You ..." — where the instrument requires the first person: "I ...". Across the full read, 295 strings were corrected in total. A first-person item asks you to endorse a statement about yourself; a second-person one asks you to agree with someone else's description of you. Those are different tasks, they invite different response styles, and switching between them is not a stylistic choice a localizer gets to make. Every one of these three defects was fixed.

Why no parity check could have found any of it

Here is the part worth taking away if you work on i18n rather than on psychometrics. Automated parity checks verify that a translation exists for every item. They can check that the key is present, the string non-empty, the placeholders resolved, the length plausible, nothing silently fallen back to the source locale. They cannot see that the string asks something else. Presence, well-formedness, and equivalence are three different properties, and tooling only reaches the first two:

• present: a string exists under this item id in this locale
• well-formed: it is fluent, grammatical, correctly interpolated, correctly rendered
• equivalent: it asks the same question the source item asks

A translated string can satisfy the first two completely and fail the third completely, which is precisely what happened on 35 French items. So i18n parity is not completeness. If your coverage dashboard is green, what it is telling you is that you have no missing strings — a genuinely useful thing to know, and not evidence about meaning. The only way we found these was by reading every item in every language against the item it was supposed to be.

Be clear about who did the reading

The reader was an AI model working at the platform owner's instruction, and every recorded row says so. This was not a native-speaker sign-off and it was not external expert review. Neither of those has happened. We state that plainly because the failure mode here is a review quoted later as stronger than it was; "reviewed" is a word that gains authority each time it is repeated. What the read actually produced is a defensible, dated, per-item record of a careful pass that caught three serious defects — which is a great deal better than the parity check it replaced, and less than a professional translation review.

One engineering detail matters for keeping that record honest. Each recorded read is hashed against the item text that was read. If someone later makes a small copy edit to a German item, that item's review does not silently carry over; the hash no longer matches and the review marks itself stale. That is by design, because the alternative is an approval record that drifts away from the text it approved and becomes a claim about nothing. If you build review tracking of any kind, hash it to the content — an approval that outlives its subject is worse than none, because it still looks like coverage.

What is still missing

No differential item functioning or measurement-invariance analysis has been run across our locales. None. And where a test has norms, those norms are computed on the pooled sample rather than stratified by language. So the honest summary of where we are is this: an item now says the same thing in six languages, which is a precondition for comparing results across those languages, not a substitute for having done the comparison. Treat any cross-language comparison of our results as unverified, because it is.

The reason this is not pedantry follows directly from the French defect. If a French item asks something different, a French respondent's score on that scale is not comparable to an English one, and any cross-language norm built on both samples is measuring two things at once and reporting a single number. That stays true for the whole period the defect is live. It is also the mechanism by which localization debt turns into measurement error rather than merely into awkward copy — which is why we would rather publish the finding than the reassurance. The same caution applies across the shelf, the DISC test included: these are tools for self-understanding, not clinical or diagnostic ones, and a result read across languages carries a caveat a result read within one does not.

If you want the longer version of what these instruments are and are not claiming to measure, what personality tests actually measure covers the ground, and our methodology page records what has been checked, when, and by whom — including this read and what it did not establish.