Entity Resolution & Record Deduplication

The problem

Companies record the same customer more than once: a slightly different spelling, a new email address, a second form submission, and the same person becomes three rows. Entity resolution is the task of working out which records refer to the same real-world entity, across datasets and within them, so that duplicates can be removed before any analysis. It matters most in large public-sector and federal datasets in health, transport, finance and law enforcement, but the client's need was the ordinary business version: clean customer data.

What I built

• An evaluation of JedAI, the open-source entity-resolution toolkit, which I rejected because its compute time was too high for the client's datasets.
• A deduplication pipeline on top of dedupe, an open-source Python library that uses machine learning to learn which records match.
• A labelled training set for the library's active-learning loop, so the model learned from real examples of matching and non-matching pairs rather than hand-written rules.
• An analysis of the model's output to confirm it was finding the right duplicates before recommending it.

Outcome

The dedupe-based approach did a good job on the client's data: output analysis showed it matched duplicate records accurately at a compute cost that JedAI could not reach, and it was adopted as one solution to the problem. Cleaner, deduplicated records mean every downstream analysis counts each customer once.