BlogData migration
Migrating data without duplicates: keys, conflicts and import review
Similar names do not prove that two customers are the same. Set identity rules before moving orders, balances or permissions.
Test worksheet: Data migration: duplicates, conflicts and import review
Similar names do not prove that two customers are the same. Set identity rules before moving orders, balances or permissions. Record inputs, expected outcome, evidence, owner and observed result.
Define what each record represents first
For a distributor, “North Trading” may mean a company, a branch or a contact who purchases for several legal entities. Define those entities before cleaning the file. A shared purchasing email does not identify a company on its own; neither does a recycled phone number. Keep original values beside normalized values so every decision can be explained.
Commercial products already have merge tools. Microsoft Dataverse lets you choose a primary record and the fields to retain, with limitations by record type. HubSpot warns that a merge cannot simply be undone. Those tools help, but you still need to check relationships and identifiers used by other systems.
Fictional example: two names and an unresolved identity
Suppose a file contains 12 accounts. C-014 says “North Trading Ltd” and C-088 “N. Trading,” sharing a tax identifier the business has verified. Both have open orders. C-091 shares their email but belongs to another company. The proposed decision merges C-014 and C-088 into N-014 and keeps C-091 separate. These synthetic records support a rehearsal; they are not a customer case or measured savings.
The expected result is 11 accounts, rather than arbitrary row deletion. A mapping table retains C-014→N-014 and C-088→N-014. Every order keeps its amount, currency, date and origin. If credit limits differ, the importer should not automatically choose the larger one: the sales owner must decide. A merge record must explain the reduced account count.
| Situation | Proposed decision | Who confirms |
|---|---|---|
| Verified key and same entity | Merge selected fields and retain both origins | Data owner |
| Only similar name or email | Flag a possible match; keep separate during review | Sales |
| One company, different branches | Keep branches and their company relationship | Operations |
| Missing or invalid identifier | Hold for review; never invent a tax identity | Administration |
Test the complete set, not just the customer list
Import into a rehearsal environment using an authorized copy first. Review accounts, addresses, contacts, open orders and associated documents together. Document totals by currency and status must remain consistent; adding balances in different currencies does not make them reconcile. Record accepted rows, merges and pending rows, including a reason for each outcome.
| Case | Expected result | Evidence |
|---|---|---|
| C-014 and C-088 merged | One N-014 account; both old identifiers remain searchable | Mapping and resulting record |
| C-091 shares an email | Separate account without another customer's orders | Account search and order list |
| Conflicting credit limits | Field held pending a named decision | Conflict list |
| Order references an unknown customer | Visible rejection without an orphaned document | Error file |
| Rerun the exact batch | Same accounts and orders; balances not added twice | Before and after counts |
| Corrected file after rehearsal | Traceable new version; authorized changes only | Batch version and comparison |
Handle changes made during the review
A cleanup lasting several days needs a reference time. If sales updates an address after export, importing an old copy must not erase that change. Decide whether to pause entry temporarily or apply a second change batch. Use data synchronization conflict rules when two sources can change the same field.
Stop the production transition if you cannot reconstruct relationships, find orphaned documents or lack an owner to confirm a match. Available tools, export permissions and merge restrictions change the scope. For stock and movements, use the inventory migration guide; for the change window, use a cutover and rollback plan.
Prepare a review that ends in a decision
Download the worksheet and replace fictional identities with anonymized examples. Include one valid account, a questionable match and a dependent order. Leave observed results empty until the rehearsal runs. In a free consultation for software consulting, we can review migration needs and what your current tools can resolve before considering custom development.
Frequently asked questions
Use it to suggest matches, rather than merging automatically. Abbreviations, namesakes and branches need other evidence. Agree with the business which combination of identifiers establishes one entity, and hold cases that do not meet that standard for review.
Before merging, demonstrate where their relationships will remain and how the original identifier can be found. The treatment depends on the product and document type. A general cleanup rule should not delete or reassign closed documents.
Run the same file twice during rehearsal and compare accounts, documents and amounts by currency. The second run must recognize imported records through stable keys. If another order or balance is added, a safe rerun rule is still missing.
Not necessarily. Test the system's detection, merge and export tools first. Custom development may help with rules those tools cannot cover, but the specific conflict and dependencies should be documented before justifying it.
Sources
- Detect duplicate records and mergeMicrosoft
- Merge recordsHubSpot
Last updated: