Guide / Data integration
Merge datasets.
Keep their provenance.
A useful data merge must answer two questions: which source records refer to the same entity, and how can someone trace the consolidated result back to the original records?
Inspect both data models before joining
List source files, record identifiers, column definitions, units, category systems and relevant timestamps. A field named “name” might refer to a product label in one source and a registered legal name in another. Treating those as equivalent can create misleading output.
Define the target dataset before mapping columns. Specify which entities the final records represent and what the data will be used for, then map each source field to that purpose.
Match with evidence and preserve decisions
Reliable shared identifiers can support deterministic matching. When no common key exists, compare combinations of fields under agreed rules and review thresholds. A similar title, a different spelling or approximate location alone does not prove two records refer to the same object, person or company.
Maintain a crosswalk: the target record ID, source system and source record key for every accepted link. Flag candidates requiring human review and retain a record of rejected or uncertain links when useful.
An illustrative catalogue example
Supplier A uses SK-104 and 0.8 kg; supplier B uses sk104 and 800 g. Case normalisation and an explicit unit conversion may reconcile the identifier and weight. However, “Sand” and “Beige” might refer to different variants. Put this conflict into the review queue rather than inventing a definitive value.
A reasonable handover includes a field mapping, source-to-target identifier crosswalk, the prepared catalogue and a list of unresolved conflicts. These data are fictional and serve only to illustrate the process.
Reconcile before declaring success
Check input and output record counts, unmatched records, key uniqueness, invalid values and the preservation of required source references. Count records at a clearly defined entity level: a catalogue item, inventory event and supplier listing may legitimately appear multiple times.
Document which mapping rules were agreed, which matches were reviewed, which conversions were made and where uncertainty remains. A data integration engagement can define these checks for a particular dataset.
Ready to discuss your data?
Start with a description of the records and what you want to achieve. We can then agree an appropriate next step.
Start an enquiry