Structured data

Entity resolution

deduplicate customers, match products across suppliers, link records

Decide which records refer to the same real-world thing.

Probability & Bayes

Odds & log-odds

Foundations

Classic record linkage is a Bayesian likelihood-ratio test per pair.

Frame the problem

Frame

Precision or recall: merging two different customers is usually worse than missing a duplicate. Define the entity.

Exploring the data

Cleaning the data

Labeling & data collection

Data & labels

Normalise first: case, whitespace, phone formats, addresses. Half the duplicates disappear here. Label pairs, not records; active learning is unusually effective because the uncertain pairs are exactly the informative ones.

Entity resolution

Create features

Represent

You cannot compare all pairs. Blocking keys, or ANN over embeddings, cut candidates to a tractable set. Then string-similarity features per pair.

Split the data

Split

By entity cluster, so no pair straddles train and test.

Logistic regression

Gradient boosting

Metric learning

Resolve

Pairwise decisions must be made transitive. Connected components or clustering over the match graph.

Entity resolution

Clustering

Model

Pairwise classifier on similarity features. Learned embeddings when text is messy.

Train

Nothing unusual here.

Classification metrics

Error analysis & significance

Evaluate offline

Pairwise precision and recall, then cluster-level purity. Inspect the largest clusters by hand; that is where the errors are.

Evaluate online

Nothing unusual here.

Serving & release

Monitor & retrain

Ship & monitor

Incremental matching for new records. Keep merge decisions reversible.

React Flow