Merge research data without erasing its history

Combining catalogs is more than copying rows. Different files can reuse the same identifier for different objects, describe the same object with different fields, or attach contradictory dates. A careful merge preserves the source records and makes each transformation explainable. 1. Preserve the inputs. Keep the original files unchanged and record their filenames, sizes, source, and acquisition dates. If you calculate hashes, store them with the inventory. Work on copies. This gives you a way to inspect what an import changed and to repeat the process after fixing a rule. 2. Read the schemas. Identify required fields, data types, date conventions, identifier rules, and missing-value meanings. In JSON, check the distinction between an empty string, null, an absent property, and an empty array. They can represent different things in a particular application. Avoid duplicate object property names because parsers can handle them inconsistently. 3. Map meanings explicitly. Write a field map from each source schema to the destination. Do not assume two fields named date describe the same event. Preserve unsupported source fields in documented provenance or a retained source file rather than silently discarding them. 4. Resolve identity carefully. Equal titles do not prove duplicate objects, and different titles do not prove different objects. Use source identifiers, exact content, and contextual evidence. When old identifiers collide, assign a new stable identifier and retain the original as an alias with its source context. 5. Test a small batch. Include an ordinary record, a missing field, a duplicate candidate, a long text, unusual characters, and an invalid URL. Check how the destination displays and exports them. A technically valid JSON file can still contain semantically wrong or unusable data. 6. Reconcile the results. Compare input and output counts, list merged duplicates and rejected rows, and inspect representative records. Every missing row should have an explanation. Keep a merge report containing the rules, known limitations, and a way to return to the original inputs. For a personal archive, the practical result is a clean content pack plus a short provenance note. Do not insert invented dates or authors merely to satisfy a required field; adjust the representation or flag the record for review. A future researcher should be able to distinguish source content, editorial normalization, and newly added interpretation. That distinction is what makes a larger archive trustworthy rather than simply larger.

Editorial review: 2026-09-23. New evidence may change this account.

Report a problem with this record