How to Prevent Duplicate Segments in Translation Memory

The source is the same. The target is the same. And the TM still has it twice. Sometimes three times: translator role, reviewer 2, then a TMX you imported “to be sure.” Lookup still works. The TU count lies. QA reports an inconsistency on identical text. The next alignment import makes it worse.

This is a different ticket from conflicting entries (same source, different targets). Conflicts need a master translation. Exact duplicates need a setting that stops the second write. Using the conflict workflow on identical pairs wastes the afternoon and does not stop the next import from writing them again.

This page is a prevention checklist for LSP TM admins and PMs. It is not a vendor feature list.

What “duplicate segment” means here

Teams use “duplicate” for four different objects:

  • Exact pair. Same source text, same target text. Pure bloat. Safe to merge or delete once you keep one unit’s metadata.
  • Role copy. Same pair, different workflow role (translator / reviewer 1 / reviewer 2). memoQ treats role like context: a second confirm creates a new unit instead of overwriting.
  • Context copy. Same pair, different previous/next segment or ID. These are often intentional 101%/102% matches. Do not mass-delete them unless you decided this TM should not store context.
  • Conflict. Same source, different target. Not a duplicate. Resolve it on the conflicts guide.

Prevention starts by picking which of the first three you refuse to store. The tool cannot guess.

How the second copy gets written

memoQ’s own duplicate-filter help lists the usual generators: confirming the same translation in different contexts, a TM that allows several translations for one source, a TM with no context, TMX imports full of identical sources, and confirming non-text (numbers, dashes). Add two operational ones: a second import of the same file, and two writable TMs on one project.

1. The TM was created to allow multiple translations

In memoQ, Allow multiple translations is a create-time checkbox. Vendor docs say to leave it cleared unless you are importing a foreign TM that already has several targets per source. If context storage is on, multiple translations are usually unnecessary: different targets almost always have different context.

You cannot flip the checkbox later. The workaround is export to TMX, create a new TM with the box off (and with context storage if this is a working TM), then import once. Confirming into the old TM while you “think about recreating it” keeps writing extras.

Trados does not use that checkbox. It uses the update mode on confirm and on import (below). The same failure mode is Add new translation units left as the default.

2. Role storage was on before anyone confirmed

memoQ support is explicit: with default settings (store context, multiple translations off) you still get one unit per context and per role. Translator confirms. Reviewer 2 confirms the same segment in the same context. A second unit appears instead of an overwrite.

Turning role storage off later does not collapse the copies you already have. Confirming again after the change can add a fourth unit with role None. If you do not need per-role history, turn role storage off before the first confirm. Then filter the existing role copies with “Sources equal, roles different” and keep the reviewer unit.

3. Confirm and import are set to “add new”

Trados Studio’s TM update and import dialogs offer four treatments when the incoming source already exists:

  • Add new translation units — writes another row. This is how a refresh import doubles a clean TM.
  • Overwrite existing translation units — replaces the stored pair (and most fields) with the incoming one.
  • Leave existing translation units unchanged — skips the incoming row. Use this when the live TM is the source of truth.
  • Keep most recent translation units — overwrites only if the incoming change date is newer.

RWS documents Merge translation units as the default and recommended option for updating TMs on confirm: replace the unit only when context matches; leave other same-source units alone. That prevents a new exact pair without deleting a context variant you still want.

Pick one policy and write it on the TM, not in Slack. “Add new” is the setting that creates the ticket you are reading.

4. The same TMX was imported again

memoQ’s TMX import dialog warns that the TM may have duplicates after import when multiple translations are allowed or the TM uses double context. A second import of the same export does not “refresh” lookup. It writes another copy of every pair the policy does not overwrite.

If matches disappeared after a tool move, that is a migration / index problem, not a reason to import again. Reindex first.

5. Two writable TMs, or an alignment dumped in as new units

A project TM and a main TM both marked for update will store the same confirm twice. Alignment results imported with “Add new” do the same for every segment the aligner already had in the TM. One writable TM per project. Treat alignment output as an import with leave-unchanged or keep-most-recent, then spot-check—do not pour it on top of a production TM.

Prevention: set these before the next project

  1. Decide context policy. Working / master TM: store context, refuse multiple translations. Reference-only TM: no context is acceptable; do not confirm live work into it (memoQ will try to save context and you will lose it).
  2. Create the TM with multiple translations off. If the box is already on, recreate via TMX. Do not keep confirming into the old file.
  3. Turn role storage off before the first confirm unless you have a written reason to keep translator vs reviewer units.
  4. Set confirm update to merge-on-matching-context or overwrite — never add-new as the house default.
  5. Set import to leave-unchanged or keep-most-recent when the destination TM is already in use. Use add-new only for a brand-new empty TM.
  6. Import a given TMX once. If lookup is empty after the import, check language codes and reindex before you import again.
  7. One writable TM per project. Attach other TMs for lookup only.

What to do with units that are already doubled

Prevention does not delete the copies you have. Clean them once, then lock the settings.

In memoQ, open the TM editor → Operations → Filter for duplicates. For this ticket use Sources and targets both equal. Tighten tag strictness if formatting-only copies should count as the same pair. Merge metadata into one master (concatenate notes) and delete the rest. Use Sources equal, roles different for role copies; keep the highest workflow role.

If the filter returns groups whose targets differ, stop. That is a conflict, not a duplicate.

Do not export-and-reimport as a cleanup. A second import without the settings above is how the pile grew.

What this usually is not

  • “The TM is corrupt.” Duplicate pairs are a write-policy problem. The file is doing what the checkbox asked.
  • “We should allow multiple translations so we don’t lose work.” Different targets belong in a conflict review or in context variants—not in an unbounded list on every lookup.
  • “Drop the match threshold so translators see both copies.” They already see both. That is the complaint.
  • “Re-import the client TM every Monday.” That is a duplicate factory unless overwrite/leave-unchanged is on.

Fixes that help — and fixes that do not

Help:

  • Recreate a memoQ TM with multiple translations cleared, then import once
  • Turn role storage off before anyone confirms
  • Trados merge / overwrite / leave-unchanged / keep-most-recent, chosen on purpose
  • One writable TM; others lookup-only
  • One exact-pair cleanup pass after the settings change

Do not help, or help only once:

  • Importing the same TMX again “to refresh”
  • Turning role storage off after T/R1/R2 copies exist, then confirming again
  • Mass-deleting context variants because the source text matches
  • Treating different targets as duplicates and keeping both “just in case”

The limit a duplicate filter cannot repair

A TM answers one question: have I stored this source string before? Deduplicating the rows does not tell the matcher which wording was right for this audience, or why a reviewer changed a confirm. You can have a perfectly unique TM that still preserves the wrong translation with full confidence.

That is why we built Aquilla around living memory instead of string lookup. Suggestions come from the project’s own work and the team’s documented translation philosophy—not from how many identical rows survived an import policy. You can still run a conventional TM. This page exists because that workflow is real, and duplicate segments have mechanical causes.

Summary: duplicate-prevention checklist

  • ☐ Separate exact pairs from conflicts (different targets) and from context variants
  • ☐ Create / recreate the TM with multiple translations off
  • ☐ Turn role storage off before the first confirm if you do not need per-role units
  • ☐ Set confirm update to merge-on-context or overwrite — not add-new
  • ☐ Set import to leave-unchanged or keep-most-recent on a live TM
  • ☐ Import each TMX once; reindex instead of re-importing
  • ☐ One writable TM per project
  • ☐ Filter sources+targets equal; merge metadata; delete extras
  • ☐ Send different-target groups to the conflict workflow
  • ☐ If stored translations look wrong, not merely doubled, audit the TM

Related Resources