In short: A rating scale can lose comparability in translation even when every word is "correct". The labels shift in intensity ("strongly", "completely"), the midpoint changes meaning, and people answer differently in different languages: in a 26-country study, English-language questionnaires drew more middle answers and native-language questionnaires more extreme ones (Harzing, 2006). To keep a multilingual study comparable, translate the labels as a set rather than word by word, keep the same number of points and the same order, have a fluent reviewer check every label and option, store answers against one master version, and compare languages with care. Read the reasons behind the numbers before you call a gap between markets real.
Running a study in several languages this week? Jump to the per-language review checklist. The rest of this guide explains why each step matters.
You run the same satisfaction question in English, German and Spanish. Germany comes out lower. Is the product worse there, or did "very satisfied" and "sehr zufrieden" set a different bar? Unless you designed for comparability, you cannot tell. Open questions survive translation reasonably well, because you read what people meant. Scales and answer options are harder: each label carries an intensity, each option draws a boundary, and the numbers that come out look precise whether or not they mean the same thing.
Four ways a translated scale drifts
Survey methodologists have studied this for decades. Most problems fall into four groups.
1. Intensity words
"Strongly agree", "extremely likely" and "completely satisfied" do not have exact equivalents in every language. A translator may pick a word that is a little stronger or a little weaker, and the scale's end point moves. The European Social Survey's translation quality checklist for Round 11 tells translators that intensifiers such as "extremely" and "completely" in answer scales must be translated as extreme endpoints, so that "nothing can go beyond it", and that identical scales used throughout the questionnaire are translated consistently (ESS R11 Translation Quality Checklist, 2024). If one of Europe's largest cross-national surveys writes that down as a rule, it is worth following in a product study.
2. The midpoint
An English midpoint like "Neither agree nor disagree" can become "partly agree, partly disagree" in another language. Those are different statements: one is indifference, the other is a mixed view. Decide what your midpoint means, write that meaning down for the translator, and check that every language says the same thing.
3. Order and polarity
Keep the same number of points, the same direction (negative to positive, or the reverse) and the same kind of scale (one that runs from "not at all" to "extremely", or one that runs from "very dissatisfied" to "very satisfied") in every language. Do not let a translator turn a 5-point scale into 4 points because one label is awkward, or flip the order to match a local convention. The Dolnicar and Grün study on "translating" between survey answer formats (Journal of Business Research, 2013) starts from the problem that the sheer variety of answer formats makes results nearly impossible to compare across studies. Changing the format between languages creates the same problem inside one study.
4. Response style by language
Even a perfect translation does not guarantee that people use the scale the same way. In a study across 26 countries, Harzing (2006, International Journal of Cross Cultural Management) found large national differences in acquiescence (agreeing regardless of content) and extreme responding, linked to cultural traits such as power distance and collectivism. The language mattered too: English-language questionnaires drew more middle responses, and questionnaires in the respondent's own language drew more extreme ones. So if you run one market in English as a shortcut, you can create a gap that is only about the language.
What helps, according to the research
- Translate in the respondent's own language. The Harzing result is a reason not to make non-native speakers answer in English to "keep it simple".
- Consider ranking for comparisons. In two studies with 1,965 undergraduate and 1,714 MBA students in 16 countries, Harzing and colleagues (International Business Review, 2009) found that both a 7-point scale and a ranking format reduced response and language bias compared with a 5-point scale, and that ranking generally did better. If what you really need is "which of these matters most?", a ranking question may travel better than five separate rating scales.
- Use a team process, not back-translation alone. The TRAPD approach (translation, review, adjudication, pretesting and documentation), used by the European Social Survey, puts a reviewer and a pretest after the first translation. We explain why back-translation alone misses problems in translating research questions with TRAPD.
- Do not assume every difference is a problem. A GESIS survey experiment (Schick, Behr, Neuert and Lechner, Survey Research Methods) tested four German versions of a five-point agree-disagree scale, including the versions used by two large international survey programmes, which differ in polarity and in the intensity of their end labels. It found no systematic differences in the answers or in response quality. The authors call for replication with other samples and other language pairs. The lesson is to check, not to panic.
Answer options: the other half of the problem
Choice questions have their own traps, separate from the scale labels:
- Categories that do not exist in every market. Job titles, education levels, company types and payment methods differ by country. "Community college" or "GmbH" needs a local equivalent, not a literal translation.
- Bands that cross a different line. Income or price bands in one currency may cut through a meaningful local threshold in another. Set bands per market where it matters, and keep the analysis plan in mind.
- Options that get longer. German and Finnish labels can run much longer than English ones. Check that options still fit and still read as parallel.
- Order effects. If option order matters, shuffle it for everyone, in every language, so no market sees a different first option.
- One master answer. Whatever the participant reads, store the answer against one master option, so "Sehr zufrieden" and "Very satisfied" count as the same answer in your chart.
A review checklist for every language
Run this with a fluent speaker, ideally someone who knows the product category, before you launch.
- Same number of points, in the same order, in every language.
- End labels are equally strong: "extremely" and "completely" are true end points.
- The midpoint means the same thing (indifference or a mixed view, not one in one language and the other elsewhere).
- Identical scales are worded identically everywhere they appear.
- Every answer option has a real local equivalent, or a local replacement.
- Bands (income, price, company size) make sense in each market.
- Options read as parallel and none is noticeably longer or more attractive.
- Answers are stored against one master version of each option.
- A short pilot in each language reads as natural, and people explain their answers the way you expected.
Comparing results across languages
Even after a careful translation, treat a gap between markets as a question, not a finding.
Photo by Vitaly Gariev on Unsplash
- Look at the distribution, not just the mean. If one market uses the end points far more, the averages can differ while the order of items is the same.
- Compare within a market first. "In Germany, onboarding scores lower than support" is safer than "Germany scores lower than Spain".
- Read the reasons behind the numbers. If German participants who chose 3 say the same things as Spanish participants who chose 4, the gap is in the scale, not the product.
- Test formally when the decision is big. For high-stakes comparisons, check measurement invariance before comparing group means. Our guide to measurement invariance explains when it matters.
How Koji handles translated scales and options
In Koji, you choose the interview languages for a study, and when you publish, Koji translates every question into each language: the question text, the answer options, the scale labels and the ranking items. Participants see the whole study in their own language, by text or voice.
Two details matter for comparability:
- Answers are stored against the original options. A participant reads the translated label, but the answer is recorded against the master option, and a scale answer is stored as its number. Charts, exports and screener results are comparable across languages without any recoding.
- You review every language. The machine translation is a first draft, like the first T in TRAPD. A Viewing in toggle in the Research Brief shows the whole study in any enabled language, so a fluent reviewer can check each label and option against the checklist above and edit the wording. Hand edits are kept. If you later change or reorder a source question, its translations are marked out of date and the original wording shows until the next publish translates it again.
Voice and phone interviews use the reviewed wording. And because every structured answer can come with a follow-up in the participant's own words, you can read why people in each market chose what they chose, which is the most direct check that a gap is real. This works on every plan, including Free. For the full setup, see running research in multiple languages.
Machine translation is not certified translation. For regulated, legal or clinical wording, use a professional translator and keep the Koji review step for the final check.
The short version
- Translate the scale as a set, with the intensity of each label pinned down.
- Keep the point count, order and kind of scale identical in every language.
- Localise answer options where categories differ, and store answers against one master version.
- Ask people in their own language, and consider ranking when the comparison is the point.
- Have a fluent reviewer check every language, then pilot.
- Before you call a market gap real, read the reasons behind the numbers.
Try it: create a free Koji account, add a second interview language to a study, publish, and use Viewing in to review every scale label and option with a fluent colleague before you launch. Planning a new market? Read research a new country before you translate next.
