Deduplication

Deduplication reduces repeated representations, but the definition of same depends on the collection. Two posts can contain the same image while remaining distinct historical instances. Two records can describe one event with different titles. Exact text matching is a useful signal, not a complete identity rule. Compare source identifiers, object content, dates, context, and provenance. Keep a review list for ambiguous pairs instead of automatically deleting the shorter record. When merging, preserve useful fields, source links, and old identifiers as aliases where appropriate. Document conflicts rather than resolving them by arbitrary field order. Reconcile the before-and-after counts and explain every removed row. A successful merge makes the archive clearer while preserving the evidence needed to understand why separate records were judged equivalent.

Editorial review: 2026-09-23. New evidence may change this account.

Report a problem with this record