Introduction: Why Duplicate Detection Matters in Python

Data quality plays a crucial role in data analysis, machine learning, and business decision-making. However, as datasets grow and information is collected from multiple sources, duplicate records become an increasingly common problem. These duplicates can lead to inaccurate analysis, misleading insights, unnecessary storage costs, and inefficient data processing.

For example, imagine a customer database containing thousands of records collected from different applications. The same customer might appear multiple times with slightly different names, email addresses, or phone numbers. While some records may be exact copies, others may contain spelling mistakes, inconsistent formatting, or missing information, making them harder to identify.

This is where duplicate detection in Python becomes essential.

Duplicate detection is the process of identifying records that represent the same entity within a dataset or across multiple datasets. It helps developers, data analysts, and data engineers maintain accurate, consistent, and reliable information.

Python provides a powerful ecosystem of libraries and tools for identifying duplicate records, ranging from simple exact matching to advanced fuzzy matching and machine learning-based approaches.

For instance, Pandas offers built-in methods such as duplicated() and drop_duplicates() to identify and remove exact duplicates efficiently. For more complex scenarios, libraries like RapidFuzz can detect similar records containing spelling variations or typographical errors. Meanwhile, tools such as Record Linkage Toolkit, scikit-learn, and Sentence Transformers enable more advanced duplicate detection using multiple attributes, text similarity, and semantic matching.

The choice of duplicate detection technique depends on the nature of the data. Exact matching works well for identical records, while fuzzy matching is useful for identifying near duplicates. Machine learning and semantic similarity techniques can help detect records that represent the same entity even when their text values differ significantly.

Implementing an effective duplicate detection process in Python offers several benefits:

• Improved data accuracy: Identify redundant records and reduce inconsistencies in datasets.
• Better analytical insights: Prevent duplicate entries from distorting reports, statistics, and business metrics.
• Enhanced machine learning performance: Reduce duplicate or overlapping training examples that may bias model evaluation or contribute to data leakage.
• Increased processing efficiency: Minimize unnecessary data storage and repeated computations.
• Reliable data integration: Identify overlapping records when combining information from multiple databases or external sources.

However, duplicate detection is not always straightforward. Two records may look similar but represent different entities, while records with different values may actually refer to the same person, product, or organization. Therefore, effective duplicate detection requires more than simply comparing rows. It often involves data cleaning, normalization, similarity scoring, and carefully selected matching rules.

In this guide, we’ll explore how to detect duplicates in Python, starting with basic techniques using Pandas and progressing to fuzzy matching, record linkage, and machine learning-based approaches. Through practical examples and Python code snippets, you’ll learn how to identify exact duplicates, detect similar records, and build a scalable duplicate detection workflow for real-world datasets.

What Is Duplicate Detection?

Duplicate detection is the process of identifying records within a dataset that contain identical or similar information and may represent the same real-world entity. These entities can include customers, products, organizations, transactions, or documents.

In Python, duplicate detection involves comparing records using different techniques, ranging from simple equality checks to advanced similarity algorithms and machine learning models. The goal is to identify potential duplicates while avoiding incorrect matches between records that represent different entities.

For example, consider the following customer dataset:

In this dataset, records 101 and 102 contain identical names and email addresses, making them exact duplicates based on those fields. Record 103 contains a slightly different name but shares the same email address, suggesting that it may also represent the same customer.

Python libraries such as Pandas, RapidFuzz, and Record Linkage Toolkit can help identify these duplicate records using different matching strategies.

Duplicate Detection vs. Deduplication

Although duplicate detection and deduplication are closely related, they serve different purposes in data processing.

Duplicate detection focuses on identifying records that may refer to the same entity. It compares data values, calculates similarities, and flags potential duplicate records for further analysis.

Deduplication, on the other hand, involves resolving those duplicates by removing redundant records, merging information, or consolidating multiple entries into a single record.

For example, Python’s duplicated() method identifies duplicate rows in a Pandas DataFrame, while drop_duplicates() removes duplicate rows based on specified criteria.

import pandas as pd
df = pd.DataFrame({
"name": ["John", "John", "Jane"],
"email": [
"john@example.com",
"john@example.com",
"jane@example.com"
]
})
# Detect duplicate records
duplicates = df[df.duplicated()]
print(duplicates)
# Remove duplicate records
df_cleaned = df.drop_duplicates()

import pandas as pd

df = pd.DataFrame({
    "name": ["John", "John", "Jane"],
    "email": [
        "john@example.com",
        "john@example.com",
        "jane@example.com"
    ]
})

# Detect duplicate records
duplicates = df[df.duplicated()]

print(duplicates)

# Remove duplicate records
df_cleaned = df.drop_duplicates()

In this example, duplicated() identifies the repeated record, while drop_duplicates() creates a DataFrame without the duplicate.

However, removing duplicates automatically is not always appropriate. In real-world datasets, records may contain complementary or conflicting information that requires validation before merging or deletion.

Types of Duplicate Data

Duplicate records can appear in different forms depending on the structure and quality of the dataset. Understanding these types helps determine which Python techniques are most suitable.

Exact Duplicates

Exact duplicates occur when two or more records contain identical values across the fields being compared.

For example:

These records are identical and can be detected using Pandas methods such as duplicated() .

Exact duplicate detection is generally straightforward and computationally efficient, making it suitable for structured datasets where values are consistently formatted.

Near Duplicates

Near duplicates are records that refer to the same entity but contain small differences caused by spelling errors, abbreviations, formatting variations, or incomplete information.

For example:

Although the names are not identical, the records may refer to the same person.

Exact matching methods may fail to identify these duplicates because they require values to match precisely.

To address this problem, Python developers can use fuzzy matching techniques that measure the similarity between strings.

Libraries such as RapidFuzz provide algorithms including Levenshtein distance and other string similarity measures to identify near-duplicate records.

Semantic Duplicates

Semantic duplicates occur when records use different words or expressions but refer to the same underlying concept or entity.

For example:

These descriptions may refer to the same product or closely related products, even though their wording differs.

Traditional exact matching and basic fuzzy matching may struggle to identify these relationships because the strings contain different words.

Python libraries such as scikit-learn and Sentence Transformers can help identify semantic similarities using techniques such as TF-IDF, cosine similarity, and text embeddings.

However, semantic similarity alone does not guarantee that two records represent the same entity. Additional attributes, such as product identifiers, brands, or model numbers, may be needed to confirm a duplicate.

How Duplicate Detection Works in Python

Duplicate detection in Python generally follows a series of steps:

1. Data loading: Import records from CSV files, Excel spreadsheets, databases, or APIs using libraries such as Pandas.
2. Data preprocessing: Clean and standardize values by handling missing data, converting text to consistent formats, and removing unnecessary whitespace.
3. Record comparison: Compare records using exact matching, fuzzy matching, or semantic similarity techniques.
4. Similarity scoring: Assign similarity scores to potential duplicate pairs based on their matching attributes.
5. Duplicate classification: Apply rules or thresholds to classify records as duplicates, possible duplicates, or unique records.
6. Duplicate resolution: Review, merge, remove, or flag identified duplicates according to business requirements.

The complexity of this process depends on the dataset size, the number of attributes being compared, and the types of duplicates being detected.

Why Different Duplicate Detection Methods Are Needed

No single duplicate detection technique works perfectly for every dataset.

For example, exact matching is effective when records contain identical values, but it cannot reliably detect spelling variations. Fuzzy matching handles minor text differences but may incorrectly match records with similar names. Semantic matching can recognize related meanings, but it may identify conceptually similar records that are not true duplicates.

Therefore, real-world duplicate detection systems often combine multiple techniques to improve accuracy.

For instance, a customer duplicate detection pipeline might use exact matching for email addresses, fuzzy matching for customer names, and additional validation rules for phone numbers and addresses.

By combining these approaches, Python developers can build more reliable duplicate detection systems capable of handling both structured and unstructured data.

Setting Up Your Python Environment

Before implementing duplicate detection in Python, it’s important to set up a development environment with the necessary libraries and a sample dataset. Python offers several powerful tools for identifying exact duplicates, detecting similar records, and performing advanced similarity comparisons.

In this section, we’ll cover the prerequisites, introduce commonly used Python libraries for duplicate detection, install the required packages, and create a sample dataset that we’ll use throughout the tutorial.

Prerequisites

To follow the examples in this guide, you should have:

• Python 3.10 or later: A modern Python installation for running the examples.
• pip: Python’s package manager for installing external libraries.
• A code editor or IDE: Such as Visual Studio Code, PyCharm, or Jupyter Notebook.
• Basic Python knowledge: Familiarity with variables, functions, lists, and loops.
• Basic Pandas knowledge: Understanding DataFrames and common data manipulation operations will be helpful.

Python Libraries for Duplicate Detection

Python provides several libraries that support different duplicate detection techniques. The choice of library depends on the type of duplicates, dataset size, and matching requirements.

Pandas: Detecting Exact Duplicates

Pandas is one of the most widely used Python libraries for data manipulation and analysis.

It provides built-in functions such as:

• duplicated() to identify duplicate rows.
• drop_duplicates() to remove duplicate records.
• value_counts() to count repeated values.
• groupby() to group records based on common attributes.

Pandas is particularly useful for detecting exact duplicates in structured datasets, including CSV files, spreadsheets, and database tables.

RapidFuzz: Detecting Near Duplicates

RapidFuzz is a Python library designed for fast string matching and similarity calculations.

It is useful when records contain minor variations such as spelling mistakes, abbreviations, or inconsistent formatting.

RapidFuzz provides functions such as:

• fuzz.ratio() to calculate string similarity.
• fuzz.partial_ratio() to compare partial string matches.
• fuzz.token_sort_ratio() to compare strings regardless of word order.
• process.extract() to retrieve the most similar strings from a collection.

For example, RapidFuzz can help identify that “Jonathan Smith” and “Jon Smith” may refer to the same customer.

Record Linkage Toolkit: Matching Records Across Datasets

The Record Linkage Toolkit is designed for identifying records that may refer to the same entity within a dataset or across multiple datasets.

It supports techniques such as:

• Candidate pair generation.
• Blocking and indexing.
• Exact and approximate comparisons.
• Multi-attribute matching.
• Rule-based and statistical classification.

This library is particularly useful for matching customer records across different databases or combining information from multiple data sources.

scikit-learn: Text Similarity and Machine Learning

scikit-learn is a popular machine learning library that can also be used for duplicate detection.

It provides tools such as:

• TfidfVectorizer to convert text into numerical features.
• cosine_similarity() to measure similarity between text representations.
• Classification algorithms for learning duplicate matching patterns.

These techniques are useful when detecting duplicate product descriptions, documents, or other text-based records.

Sentence Transformers: Semantic Duplicate Detection

Sentence Transformers provides pretrained models that convert text into numerical embeddings representing semantic meaning.

These embeddings can be compared to identify records that use different words but express similar meanings.

For example, semantic similarity can help identify relationships between descriptions such as “wireless headphones” and “Bluetooth audio headset.”

Sentence Transformers is especially useful for duplicate detection involving unstructured text, although semantic similarity should be combined with additional checks when exact entity identity matters.

Importing the Libraries

After installing the packages, you can import the libraries into your Python script or Jupyter Notebook.

import pandas as pd
from rapidfuzz import fuzz, process
import recordlinkage
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from sentence_transformers import SentenceTransformer

import pandas as pd

from rapidfuzz import fuzz, process

import recordlinkage

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

from sentence_transformers import SentenceTransformer

These imports provide the functionality needed for exact matching, fuzzy matching, record linkage, and semantic similarity.

You don’t need to use every library in every project. For example, Pandas alone may be sufficient for datasets containing only exact duplicates.

Creating a Sample Dataset with Duplicate Records

To demonstrate duplicate detection in Python, let’s create a small customer dataset containing exact duplicates, near duplicates, and unique records.

import pandas as pd
data = {
"customer_id": [101, 102, 103, 104, 105, 106, 107],
"name": [
"John Smith",
"John Smith",
"Jon Smith",
"Alice Johnson",
"Alice Jonson",
"Robert Brown",
"Emily Davis"
],
"email": [
"john@example.com",
"john@example.com",
"john@example.com",
"alice@example.com",
"alice@example.com",
"robert@example.com",
"emily@example.com"
],
"city": [
"New York",
"New York",
"New York",
"Chicago",
"Chicago",
"Boston",
"Seattle"
]
}
df = pd.DataFrame(data)
print(df)

import pandas as pd

data = {
    "customer_id": [101, 102, 103, 104, 105, 106, 107],
    "name": [
        "John Smith",
        "John Smith",
        "Jon Smith",
        "Alice Johnson",
        "Alice Jonson",
        "Robert Brown",
        "Emily Davis"
    ],
    "email": [
        "john@example.com",
        "john@example.com",
        "john@example.com",
        "alice@example.com",
        "alice@example.com",
        "robert@example.com",
        "emily@example.com"
    ],
    "city": [
        "New York",
        "New York",
        "New York",
        "Chicago",
        "Chicago",
        "Boston",
        "Seattle"
    ]
}

df = pd.DataFrame(data)

print(df)

The resulting dataset looks like this:

This dataset contains several potential duplicate records:

• Records 101 and 102: Exact duplicates when comparing name, email, and city, although their customer IDs differ.
• Records 101 and 103: Potential near duplicates because of the variation between “John Smith” and “Jon Smith.”
• Records 104 and 105: Potential near duplicates because of the spelling difference between “Alice Johnson” and “Alice Jonson.”
• Records 106 and 107: Unique records with no obvious matching entries in this sample.

These examples will help demonstrate how different duplicate detection techniques work.

Saving and Loading the Dataset

In real-world projects, datasets are often stored in CSV files, Excel spreadsheets, or databases.

To save our sample dataset as a CSV file, use:

df.to_csv("customers.csv", index=False)

df.to_csv("customers.csv", index=False)

This creates a file named customers.csv in the current working directory.

You can load the dataset later using:

df = pd.read_csv("customers.csv")
print(df.head())

df = pd.read_csv("customers.csv")

print(df.head())

The read_csv() function loads the file into a Pandas DataFrame, allowing you to apply duplicate detection techniques directly.

Performing a Quick Duplicate Check

Before moving on to advanced duplicate detection methods, we can perform a simple check using Pandas.

# Detect duplicates based on name, email, and city
duplicates = df[df.duplicated(
subset=["name", "email", "city"],
keep=False
)]
print(duplicates)

# Detect duplicates based on name, email, and city
duplicates = df[df.duplicated(
    subset=["name", "email", "city"],
    keep=False
)]

print(duplicates)

Expected output:

   customer_id        name             email      city
0          101  John Smith  john@example.com  New York
1          102  John Smith  john@example.com  New York

Here, the duplicated() method checks whether multiple records contain identical values in the specified columns.

The keep=False parameter marks all occurrences of matching records as duplicates rather than excluding the first occurrence.

Notice that records 101 and 103 are not identified as duplicates because “John Smith” and “Jon Smith” are different strings.

This demonstrates an important limitation of exact matching: it cannot detect records containing spelling variations or other inconsistencies.

To identify these types of duplicates, we’ll need more advanced techniques such as fuzzy matching and similarity scoring.

Preparing Data for Accurate Duplicate Detection

Before applying duplicate detection algorithms, it’s good practice to clean and standardize the dataset.

For example, differences in capitalization or unnecessary whitespace can prevent identical values from matching correctly.

Consider these customer names:

• “John Smith”
• “john smith”
• ” John Smith “

Although these values refer to the same name, a case-sensitive exact comparison treats them as different strings.

We can normalize the values using Pandas:

df["name_clean"] = (
df["name"]
.str.strip()
.str.lower()
)
df["email_clean"] = (
df["email"]
.str.strip()
.str.lower()
)

df["name_clean"] = (
    df["name"]
    .str.strip()
    .str.lower()
)

df["email_clean"] = (
    df["email"]
    .str.strip()
    .str.lower()
)

This code removes leading and trailing whitespace and converts text to lowercase.

For example, all three variations of “John Smith” would be standardized to:

john smith

john smith

Data preprocessing improves the reliability of duplicate detection and reduces unnecessary mismatches caused by formatting differences.

However, normalization should be applied carefully. Some attributes, such as identifiers, addresses, and international names, may require specialized cleaning rules to avoid losing meaningful information.

Detecting Exact Duplicates Using Pandas

One of the simplest and most effective ways to perform duplicate detection in Python is by using Pandas. Pandas provides built-in methods for identifying, counting, and removing duplicate records from structured datasets, making it an excellent starting point for data cleaning and deduplication.

Exact duplicate detection involves finding records that contain identical values across all columns or a selected set of columns. This approach is particularly useful when working with customer databases, transaction records, product catalogs, and CSV files.

In this section, we’ll explore how to detect exact duplicates using Pandas, identify duplicates based on specific columns, count duplicate records, and remove duplicates safely.

Finding Duplicate Rows Using duplicated()

The Pandas duplicated() method identifies rows that contain identical values. It returns a Boolean Series indicating whether each row is considered a duplicate.

Let’s start with a simple dataset:

import pandas as pd
data = {
"name": [
"John Smith",
"Alice Johnson",
"John Smith",
"Robert Brown",
"Alice Johnson"
],
"email": [
"john@example.com",
"alice@example.com",
"john@example.com",
"robert@example.com",
"alice@example.com"
]
}
df = pd.DataFrame(data)
print(df)

import pandas as pd

data = {
    "name": [
        "John Smith",
        "Alice Johnson",
        "John Smith",
        "Robert Brown",
        "Alice Johnson"
    ],
    "email": [
        "john@example.com",
        "alice@example.com",
        "john@example.com",
        "robert@example.com",
        "alice@example.com"
    ]
}

df = pd.DataFrame(data)

print(df)

Output:

            name               email
0     John Smith    john@example.com
1  Alice Johnson   alice@example.com
2     John Smith    john@example.com
3   Robert Brown  robert@example.com
4  Alice Johnson   alice@example.com

Now, let’s identify duplicate rows:

df["is_duplicate"] = df.duplicated()
print(df)

df["is_duplicate"] = df.duplicated()

print(df)

Output:

            name               email  is_duplicate
0     John Smith    john@example.com         False
1  Alice Johnson   alice@example.com         False
2     John Smith    john@example.com          True
3   Robert Brown  robert@example.com         False
4  Alice Johnson   alice@example.com          True

By default, duplicated() marks the first occurrence of each record as False and subsequent identical records as True .

This makes it easy to identify repeated entries without modifying or deleting the original data.

Understanding the keep Parameter

The duplicated() method includes a keep parameter that controls which occurrences are marked as duplicates.

1. keep="first"

Marks all duplicate occurrences except the first one.

df.duplicated(
subset=["name", "email"],
keep="first"
)

df.duplicated(
    subset=["name", "email"],
    keep="first"
)

2. keep="last"

Marks all duplicate occurrences except the last one.

df.duplicated(
subset=["name", "email"],
keep="last"
)

df.duplicated(
    subset=["name", "email"],
    keep="last"
)

3. keep=False

Marks every occurrence of a duplicated record, including the first.

df.duplicated(
subset=["name", "email"],
keep=False
)

df.duplicated(
    subset=["name", "email"],
    keep=False
)

The keep=False option is particularly useful when you want to review all records belonging to a duplicate group before deciding which records to retain.

For example:

duplicates = df[
df.duplicated(
subset=["name", "email"],
keep=False
)
]
print(duplicates)

duplicates = df[
    df.duplicated(
        subset=["name", "email"],
        keep=False
    )
]

print(duplicates)

This returns all records that share identical names and email addresses.

Detecting Duplicates Based on Specific Columns

In real-world datasets, records often contain unique identifiers such as customer IDs, transaction IDs, or database-generated primary keys.

As a result, two records may represent the same entity even though their entire rows are not identical.

Consider the following dataset:

data = {
"customer_id": [101, 102, 103, 104, 105],
"name": [
"John Smith",
"John Smith",
"Alice Johnson",
"Robert Brown",
"Alice Johnson"
],
"email": [
"john@example.com",
"john@example.com",
"alice@example.com",
"robert@example.com",
"alice@example.com"
]
}
df = pd.DataFrame(data)

data = {
    "customer_id": [101, 102, 103, 104, 105],
    "name": [
        "John Smith",
        "John Smith",
        "Alice Johnson",
        "Robert Brown",
        "Alice Johnson"
    ],
    "email": [
        "john@example.com",
        "john@example.com",
        "alice@example.com",
        "robert@example.com",
        "alice@example.com"
    ]
}

df = pd.DataFrame(data)

If we check for duplicates across all columns:

print(df.duplicated().sum())

print(df.duplicated().sum())

Output:

0

0

No duplicates are detected because every record has a unique customer_id .

However, if we compare only the name and email columns:

duplicates = df[
df.duplicated(
subset=["name", "email"],
keep=False
)
]
print(duplicates)

duplicates = df[
    df.duplicated(
        subset=["name", "email"],
        keep=False
    )
]

print(duplicates)

Output:

   customer_id           name              email
0          101     John Smith   john@example.com
1          102     John Smith   john@example.com
2          103  Alice Johnson  alice@example.com
4          105  Alice Johnson  alice@example.com

Here, Pandas identifies two groups of duplicate records based on the selected fields.

This demonstrates why choosing the right columns is essential for accurate duplicate detection.

For example:

• In a customer database, email addresses or combinations of contact details may be useful matching attributes.
• In a product catalog, SKU or product identifiers may help identify duplicate listings.
• In a transaction dataset, transaction references may be more reliable than customer names.

Best practice: Avoid relying on a single attribute unless it is sufficiently reliable and unique for your use case. Matching multiple fields can help reduce false positives.

Counting Duplicate Records in Pandas

Before removing duplicate records, it’s useful to understand how many duplicates exist in your dataset.

Pandas makes this easy using duplicated() together with sum() .

Count Duplicate Rows

duplicate_count = df.duplicated(
subset=["name", "email"]
).sum()
print("Number of duplicate rows:", duplicate_count)

duplicate_count = df.duplicated(
    subset=["name", "email"]
).sum()

print("Number of duplicate rows:", duplicate_count)

Output:

Number of duplicate rows: 2

Number of duplicate rows: 2

This counts duplicate occurrences beyond the first record in each matching group.

Count All Records Involved in Duplicates

If you want to count every record belonging to a duplicate group, use keep=False .

duplicate_records = df.duplicated(
subset=["name", "email"],
keep=False
).sum()
print("Records involved in duplicates:", duplicate_records)

duplicate_records = df.duplicated(
    subset=["name", "email"],
    keep=False
).sum()

print("Records involved in duplicates:", duplicate_records)

Output:

Records involved in duplicates: 4

Records involved in duplicates: 4

The difference is important:

• Duplicate rows: The number of additional repeated records.
• Records involved in duplicates: All records belonging to duplicate groups, including the first occurrence.

Calculate the Duplicate Percentage

You can also calculate the percentage of redundant records in a dataset.

duplicate_percentage = (
df.duplicated(
subset=["name", "email"]
).sum() / len(df)
) * 100
print(f"Duplicate percentage: {duplicate_percentage:.2f}%")

duplicate_percentage = (
    df.duplicated(
        subset=["name", "email"]
    ).sum() / len(df)
) * 100

print(f"Duplicate percentage: {duplicate_percentage:.2f}%")

Output:

Duplicate percentage: 40.00%

Duplicate percentage: 40.00%

In this example, two out of five records are additional duplicate occurrences.

Monitoring this percentage can help organisations evaluate data quality and measure the effectiveness of their deduplication processes.

For an empty DataFrame, check that len(df) > 0 before calculating the percentage to avoid division by zero.

Removing Exact Duplicates Using drop_duplicates()

Once duplicate records have been identified and validated, Pandas provides the drop_duplicates() method to remove redundant rows.

The method works similarly to duplicated() , but instead of returning Boolean values, it returns a DataFrame with duplicate rows removed.

Remove Duplicate Rows

df_cleaned = df.drop_duplicates(
subset=["name", "email"]
)
print(df_cleaned)

df_cleaned = df.drop_duplicates(
    subset=["name", "email"]
)

print(df_cleaned)

Output:

   customer_id           name               email
0          101     John Smith    john@example.com
2          103  Alice Johnson   alice@example.com
3          104   Robert Brown  robert@example.com

By default, drop_duplicates() retains the first occurrence of each duplicate group and removes subsequent occurrences.

Keep the Last Occurrence

Sometimes, you may want to retain the last occurrence instead of the first.

df_cleaned = df.drop_duplicates(
subset=["name", "email"],
keep="last"
)

df_cleaned = df.drop_duplicates(
    subset=["name", "email"],
    keep="last"
)

This approach may be useful when the dataset is already sorted so that the latest or most relevant records appear last.

However, row order alone does not guarantee that the last record is the most accurate or up-to-date.

Remove All Records Belonging to Duplicate Groups

You can also remove every record involved in a duplicate group.

df_unique = df.drop_duplicates(
subset=["name", "email"],
keep=False
)
print(df_unique)

df_unique = df.drop_duplicates(
    subset=["name", "email"],
    keep=False
)

print(df_unique)

Output:

   customer_id          name               email
3          104  Robert Brown  robert@example.com

This keeps only records that have no matching duplicates.

Use this option carefully because it removes all records in duplicate groups, including their first occurrences.

Detecting Duplicates After Data Normalization

Exact matching is sensitive to formatting differences.

For example, consider these customer names:

• John Smith
• john smith
• John Smith
• JOHN SMITH

Although these values represent the same name, Pandas treats them as different strings during a standard exact comparison.

To address this, normalize the data before performing duplicate detection.

data = {
"name": [
"John Smith",
"john smith",
" John Smith ",
"Alice Johnson"
]
}
df = pd.DataFrame(data)
df["name_clean"] = (
df["name"]
.str.strip()
.str.lower()
)
duplicates = df[
df.duplicated(
subset=["name_clean"],
keep=False
)
]
print(duplicates)

data = {
    "name": [
        "John Smith",
        "john smith",
        " John Smith ",
        "Alice Johnson"
    ]
}

df = pd.DataFrame(data)

df["name_clean"] = (
    df["name"]
    .str.strip()
    .str.lower()
)

duplicates = df[
    df.duplicated(
        subset=["name_clean"],
        keep=False
    )
]

print(duplicates)

Output:

           name  name_clean
0    John Smith  john smith
1    john smith  john smith
2   John Smith   john smith

After normalization, Pandas can identify records that previously appeared different because of capitalization or whitespace.

For more complex datasets, preprocessing may also include:

• Standardizing phone number formats.
• Normalizing dates.
• Removing unnecessary punctuation.
• Handling missing values.
• Standardizing common abbreviations.

However, normalization should preserve meaningful information. For example, removing punctuation or leading zeros from certain identifiers could introduce incorrect matches.

Identifying Duplicate Groups

Sometimes, simply flagging duplicate rows is not enough. You may also want to understand which records belong to the same duplicate group.

Pandas provides the groupby() method for this purpose.

duplicate_groups = (
df.groupby(["name_clean"])
.size()
.reset_index(name="count")
)
duplicate_groups = duplicate_groups[
duplicate_groups["count"] > 1
]
print(duplicate_groups)

duplicate_groups = (
    df.groupby(["name_clean"])
    .size()
    .reset_index(name="count")
)

duplicate_groups = duplicate_groups[
    duplicate_groups["count"] > 1
]

print(duplicate_groups)

Output:

   name_clean  count
0  john smith      3

This example groups records by their normalized names and counts how many times each value appears.

Grouping duplicates can be useful for:

• Generating duplicate detection reports.
• Identifying frequently repeated records.
• Reviewing duplicate groups before merging.
• Prioritizing data cleaning activities.

For datasets with missing values in grouping columns, consider using groupby(..., dropna=False) when those records should also be included.

Best Practices for Exact Duplicate Detection

Although Pandas makes exact duplicate detection straightforward, a few best practices can improve accuracy and reliability.

1. Choose meaningful matching columns. Avoid including unique identifiers when your goal is to identify repeated entities rather than completely identical database rows.

2. Normalize data before comparison. Standardize capitalization, whitespace, and relevant formats to reduce unnecessary mismatches.

3. Review duplicates before removing them. Similar or identical records may contain additional information that should be preserved.

4. Preserve the original dataset. Create a cleaned copy instead of modifying source data immediately.

5. Handle missing values carefully. Pandas can treat missing values in corresponding positions as matching during duplicate detection. This may require additional validation.

6. Monitor duplicate rates. Track the number and percentage of duplicate records to identify recurring data quality problems.

7. Use fuzzy matching when necessary. Exact matching cannot reliably identify spelling mistakes, abbreviations, or other variations in text values.

Detecting Near Duplicates Using Fuzzy Matching

While exact duplicate detection works well for identical records, real-world datasets often contain records that are similar but not exactly the same. Differences in spelling, abbreviations, formatting, or data entry can make it difficult to identify records that refer to the same entity.

For example, a customer database might contain the names “Jonathan Smith,” “Jon Smith,” and “Johnathan Smith.” Although these names are not identical, they may refer to the same person.

Traditional exact matching methods, such as Pandas’ duplicated() , cannot reliably identify these variations because they compare values for equality.

This is where fuzzy matching in Python becomes useful.

Fuzzy matching identifies similarities between strings rather than requiring exact matches. By calculating similarity scores, it helps detect potential duplicate records containing spelling errors, missing characters, abbreviations, or other minor differences.

In this section, we’ll explore how fuzzy matching works, how to use the RapidFuzz library, and how to identify near-duplicate records in Python.

What Is Fuzzy Matching?

Fuzzy matching is a technique used to compare strings and determine how similar they are, even when they are not identical.

Unlike exact matching, which checks whether two values are exactly equal, fuzzy matching calculates a similarity score based on the differences between them.

For example, consider the following customer names:

A fuzzy matching algorithm can identify that the first three names share similarities, while “Robert Brown” is considerably different.

Most commonly used fuzzy matching functions return a similarity score between 0 and 100, where:

• 100: The strings are identical under the selected comparison method.
• 0: The strings have no similarity according to that method.
• Between 0 and 100: The strings share some degree of similarity.

A higher score generally indicates greater textual similarity, but it does not necessarily mean that two records represent the same entity.

Fuzzy matching is particularly useful for detecting:

• Typographical errors, such as “Johnson” and “Jonson.”
• Missing characters, such as “Jonathan” and “Jon.”
• Different word orders, such as “Smith John” and “John Smith.”
• Abbreviations, such as “Robert” and “Rob.”
• Formatting inconsistencies in names, addresses, and product descriptions.

However, different fuzzy matching algorithms handle these variations differently, so selecting the appropriate similarity method is important.

Using RapidFuzz for Duplicate Detection

RapidFuzz is a Python library designed for fast string matching and similarity calculations. It provides several algorithms for comparing text values and identifying potential matches.

RapidFuzz is particularly useful for duplicate detection because it offers efficient implementations of common string similarity methods and can be integrated easily with Pandas.

Installing RapidFuzz

Once installed, import the required functions:

from rapidfuzz import fuzz, process

from rapidfuzz import fuzz, process

The fuzz module provides similarity scoring functions, while process provides utilities for finding similar strings within collections.

Comparing Two Strings Using fuzz.ratio()

The fuzz.ratio() function calculates a normalized string similarity score between two values.

from rapidfuzz import fuzz
name1 = "John Smith"
name2 = "Jon Smith"
similarity = fuzz.ratio(name1, name2)
print(f"Similarity Score: {similarity:.2f}")

from rapidfuzz import fuzz

name1 = "John Smith"
name2 = "Jon Smith"

similarity = fuzz.ratio(name1, name2)

print(f"Similarity Score: {similarity:.2f}")

Output:

Similarity Score: 94.74

Similarity Score: 94.74

The high similarity score indicates that the names are textually similar, making them reasonable candidates for further duplicate validation.

However, a similarity score alone is not sufficient to confirm that two records represent the same person.

Comparing Strings with Different Word Orders

Sometimes, duplicate records contain the same words in a different order.

For example:

name1 = "John Smith"
name2 = "Smith John"
similarity = fuzz.token_sort_ratio(name1, name2)
print(similarity)

name1 = "John Smith"
name2 = "Smith John"

similarity = fuzz.token_sort_ratio(name1, name2)

print(similarity)

Output:

100.0

100.0

The token_sort_ratio() function splits strings into tokens, sorts them, and compares the normalized token sequences.

This makes it useful for detecting records where word order differs but the underlying words remain the same.

Comparing Partial String Matches

The fuzz.partial_ratio() function is useful when one string is contained within a longer string.

from rapidfuzz import fuzz
text1 = "Apple iPhone 15"
text2 = "iPhone 15"
similarity = fuzz.partial_ratio(text1, text2)
print(similarity)

from rapidfuzz import fuzz

text1 = "Apple iPhone 15"
text2 = "iPhone 15"

similarity = fuzz.partial_ratio(text1, text2)

print(similarity)

Output:

100.0

100.0

This method can help identify similar product names, organization names, or descriptions where one record contains additional text.

However, partial matching can also produce false positives. For example, two different product variants may share the same base name.

Finding Similar Records in a Dataset

Now that we understand how fuzzy matching works, let’s apply it to a dataset containing near-duplicate customer names.

Consider the following example:

import pandas as pd
data = {
"customer_id": [101, 102, 103, 104, 105, 106],
"name": [
"John Smith",
"Jon Smith",
"Alice Johnson",
"Alice Jonson",
"Robert Brown",
"Emily Davis"
]
}
df = pd.DataFrame(data)
print(df)

import pandas as pd

data = {
    "customer_id": [101, 102, 103, 104, 105, 106],
    "name": [
        "John Smith",
        "Jon Smith",
        "Alice Johnson",
        "Alice Jonson",
        "Robert Brown",
        "Emily Davis"
    ]
}

df = pd.DataFrame(data)

print(df)

Dataset:

Our goal is to identify customer names that are similar enough to be considered potential duplicates.

Step 1: Compare Records Using Fuzzy Matching

We can compare each customer name with every other name using RapidFuzz.

from rapidfuzz import fuzz
matches = []
for i in range(len(df)):
for j in range(i + 1, len(df)):
name1 = df.iloc[i]["name"]
name2 = df.iloc[j]["name"]
score = fuzz.ratio(name1, name2)
if score >= 85:
matches.append({
"customer_id_1": df.iloc[i]["customer_id"],
"customer_id_2": df.iloc[j]["customer_id"],
"name_1": name1,
"name_2": name2,
"similarity": round(score, 2)
})
duplicates_df = pd.DataFrame(matches)
print(duplicates_df)

from rapidfuzz import fuzz

matches = []

for i in range(len(df)):
    for j in range(i + 1, len(df)):
        name1 = df.iloc[i]["name"]
        name2 = df.iloc[j]["name"]

        score = fuzz.ratio(name1, name2)

        if score >= 85:
            matches.append({
                "customer_id_1": df.iloc[i]["customer_id"],
                "customer_id_2": df.iloc[j]["customer_id"],
                "name_1": name1,
                "name_2": name2,
                "similarity": round(score, 2)
            })

duplicates_df = pd.DataFrame(matches)

print(duplicates_df)

Expected matching results:

In this example, the similarity threshold is set to 85.

Records with similarity scores of 85 or higher are flagged as potential duplicates.

Notice that the code compares each pair only once by using range(i + 1, len(df)) . This avoids comparing a record with itself and prevents duplicate pair comparisons.

This approach works well for small datasets. However, comparing every possible pair becomes expensive as the number of records increases.

Finding the Closest Match Using process.extract()

RapidFuzz also provides the process.extract() function, which finds the most similar strings from a collection.

Instead of manually comparing a string with every possible value, we can retrieve the closest matches directly.

from rapidfuzz import process, fuzz
names = [
"John Smith",
"Jon Smith",
"Alice Johnson",
"Alice Jonson",
"Robert Brown"
]
query = "John Smith"
matches = process.extract(
query,
names,
scorer=fuzz.ratio,
limit=3
)
print(matches)

from rapidfuzz import process, fuzz

names = [
    "John Smith",
    "Jon Smith",
    "Alice Johnson",
    "Alice Jonson",
    "Robert Brown"
]

query = "John Smith"

matches = process.extract(
    query,
    names,
    scorer=fuzz.ratio,
    limit=3
)

print(matches)

Output:

[
('John Smith', 100.0, 0),
('Jon Smith', 94.73684210526316, 1),
('Alice Johnson', 33.333333333333336, 2)
]

[
    ('John Smith', 100.0, 0),
    ('Jon Smith', 94.73684210526316, 1),
    ('Alice Johnson', 33.333333333333336, 2)
]

Each result contains the matched string, similarity score, and position in the original collection.

When the query string is already present in the collection, the results may include an exact match to itself.

For duplicate detection, self-matches should be excluded when comparing records from the same dataset.

You can also filter results using a minimum similarity score:

matches = process.extract(
"John Smith",
names,
scorer=fuzz.ratio,
score_cutoff=85
)
print(matches)

matches = process.extract(
    "John Smith",
    names,
    scorer=fuzz.ratio,
    score_cutoff=85
)

print(matches)

This returns only strings that meet the specified similarity threshold.

Choosing the Right Similarity Threshold

One of the most important decisions in fuzzy duplicate detection is selecting an appropriate similarity threshold.

A similarity threshold determines how closely two strings must match before they are flagged as potential duplicates.

For example:

These ranges are illustrative, not universal rules. The appropriate threshold depends on the dataset, comparison algorithm, and consequences of incorrect matches.

High Similarity Threshold

A high threshold, such as 95, identifies records that are almost identical.

Advantages:

• Reduces the number of false positive matches.
• Works well for detecting minor spelling variations.
• Requires less manual review of unrelated records.

Limitations:

• May miss legitimate duplicates with abbreviations or larger text differences.
• Can produce more false negatives.

Lower Similarity Threshold

A lower threshold, such as 75 or 80, identifies a broader range of similar records.

Advantages:

• Detects more potential duplicates.
• Can identify records containing larger spelling variations.
• May improve recall.

Limitations:

• Can produce more false positives.
• May require additional validation using other attributes.

For example, “John Smith” and “Joan Smith” may receive a relatively high similarity score despite potentially referring to different people.

For this reason, duplicate detection systems should not rely solely on name similarity.

Combining Fuzzy Matching with Multiple Attributes

In real-world applications, comparing only customer names may not provide enough information to determine whether records are duplicates.

A more reliable approach combines fuzzy matching with additional attributes such as email addresses, phone numbers, or locations.

Consider the following dataset:

import pandas as pd
data = {
"customer_id": [101, 102, 103, 104],
"name": [
"John Smith",
"Jon Smith",
"Alice Johnson",
"Alice Jonson"
],
"email": [
"john@example.com",
"john@example.com",
"alice@example.com",
"alice@example.com"
]
}
df = pd.DataFrame(data)

import pandas as pd

data = {
    "customer_id": [101, 102, 103, 104],
    "name": [
        "John Smith",
        "Jon Smith",
        "Alice Johnson",
        "Alice Jonson"
    ],
    "email": [
        "john@example.com",
        "john@example.com",
        "alice@example.com",
        "alice@example.com"
    ]
}

df = pd.DataFrame(data)

We can use fuzzy matching for customer names and exact matching for email addresses.

from rapidfuzz import fuzz
matches = []
for i in range(len(df)):
for j in range(i + 1, len(df)):
name_score = fuzz.ratio(
df.iloc[i]["name"],
df.iloc[j]["name"]
)
email_match = (
df.iloc[i]["email"] == df.iloc[j]["email"]
)
if name_score >= 85 and email_match:
matches.append({
"customer_id_1": df.iloc[i]["customer_id"],
"customer_id_2": df.iloc[j]["customer_id"],
"name_similarity": round(name_score, 2),
"email_match": email_match
})
result = pd.DataFrame(matches)
print(result)

from rapidfuzz import fuzz

matches = []

for i in range(len(df)):
    for j in range(i + 1, len(df)):
        name_score = fuzz.ratio(
            df.iloc[i]["name"],
            df.iloc[j]["name"]
        )

        email_match = (
            df.iloc[i]["email"] == df.iloc[j]["email"]
        )

        if name_score >= 85 and email_match:
            matches.append({
                "customer_id_1": df.iloc[i]["customer_id"],
                "customer_id_2": df.iloc[j]["customer_id"],
                "name_similarity": round(name_score, 2),
                "email_match": email_match
            })

result = pd.DataFrame(matches)

print(result)

Expected results:

By combining multiple attributes, we can reduce the likelihood of incorrectly identifying two different customers as duplicates.

However, matching email addresses do not always guarantee identity, particularly when email accounts are shared or reused.

For more complex scenarios, record linkage techniques can combine multiple comparison scores into a more comprehensive matching decision.

Improving Fuzzy Matching Accuracy with Data Preprocessing

Fuzzy matching generally produces more reliable results when data is cleaned and standardized before comparison.

For example, capitalization, extra spaces, and punctuation can affect similarity scores.

Consider these names:

• “John Smith”
• “john smith”
• ” JOHN SMITH “

Although these values are effectively identical, they contain formatting differences.

We can normalize them before comparison:

def normalize_text(text):
if pd.isna(text):
return ""
return " ".join(
str(text).strip().lower().split()
)
df["name_clean"] = df["name"].apply(normalize_text)

def normalize_text(text):
    if pd.isna(text):
        return ""

    return " ".join(
        str(text).strip().lower().split()
    )

df["name_clean"] = df["name"].apply(normalize_text)

This function:

1. Handles missing values.
2. Converts text to lowercase.
3. Removes leading and trailing whitespace.
4. Collapses repeated whitespace into single spaces.

For example:

name1 = normalize_text(" John Smith ")
name2 = normalize_text("john smith")
score = fuzz.ratio(name1, name2)
print(score)

name1 = normalize_text(" John Smith ")
name2 = normalize_text("john smith")

score = fuzz.ratio(name1, name2)

print(score)

Output:

100.0

Normalizing data helps reduce unnecessary differences and improves consistency across comparisons.

Performance Considerations for Large Datasets

The pairwise comparison approach demonstrated earlier works well for small datasets, but it becomes computationally expensive as the number of records grows.

For a dataset containing (n) records, the number of unique record pairs is:

For example:

As the dataset grows, comparing every possible pair becomes impractical.

To improve performance, consider the following techniques:

• Blocking: Compare only records that share certain attributes, such as city or email domain.
• Indexing: Organize records to reduce unnecessary comparisons.
• Candidate generation: Identify likely matches before applying expensive similarity calculations.
• RapidFuzz batch operations: Use functions such as process.extract() or process.extractOne() for efficient string matching.
• Parallel processing: Distribute independent comparison tasks across multiple CPU cores where appropriate.

For large-scale duplicate detection, record linkage frameworks and specialized indexing techniques can significantly reduce the number of candidate pairs.

Detecting Duplicates Using Record Linkage in Python

When working with large datasets or combining information from multiple sources, detecting duplicate records becomes more challenging. Records representing the same customer, product, or organization may contain different names, addresses, email addresses, or other identifying information.

While exact matching and fuzzy matching can identify many duplicate records, they often focus on individual fields or simple comparison rules. In real-world applications, duplicate detection frequently requires comparing multiple attributes simultaneously to determine whether two records refer to the same entity.

This is where record linkage in Python becomes particularly useful.

Record linkage is a technique used to identify records that represent the same real-world entity, either within a single dataset or across multiple datasets. By combining indexing, field comparisons, similarity scoring, and classification rules, record linkage provides a structured approach to identifying potential duplicates.

In this section, we’ll explore how record linkage works, how to use the Python Record Linkage Toolkit, and how to build a practical duplicate detection workflow for matching customer records.

What Is Record Linkage?

Record linkage, also known as entity resolution or data matching, is the process of identifying records that refer to the same entity even when their information is not exactly identical.

For example, consider two customer databases:

Dataset A

Dataset B

Although the records in these datasets are not identical, some may represent the same customers.

For example:

• “John Smith” and “Jon Smith” have similar names and identical email addresses.
• “Alice Johnson” and “Alice Jonson” contain a spelling variation but share the same email address and city.
• “Robert Brown” and “Emily Davis” have no obvious matching attributes.

Record linkage analyzes these similarities to identify potential matches.

Unlike simple exact matching, record linkage can combine multiple comparison methods to evaluate records more comprehensively.

Record Linkage vs. Deduplication

Record linkage and deduplication are closely related, but they are not exactly the same.

• Record linkage identifies records that may refer to the same entity, particularly across different datasets.
• Deduplication focuses on identifying and resolving repeated records, often within a single dataset.

Both processes can use similar algorithms, including exact matching, fuzzy matching, and probabilistic classification.

Installing the Python Record Linkage Toolkit

The Python Record Linkage Toolkit is an open-source library that provides tools for comparing records and identifying potential matches.

It supports:

• Indexing and candidate pair generation.
• Blocking and sorted neighborhood indexing.
• Exact and approximate field comparisons.
• Numerical and string similarity comparisons.
• Rule-based and statistical classification.

After installation, import the required libraries:

import pandas as pd
import recordlinkage

import pandas as pd
import recordlinkage

We’ll use Pandas to manage the datasets and Record Linkage Toolkit to generate and compare candidate record pairs.

Creating Sample Customer Datasets

Let’s create two customer datasets containing exact matches, near duplicates, and unique records.

import pandas as pd
data_a = {
"customer_id": [101, 102, 103, 104],
"name": [
"John Smith",
"Alice Johnson",
"Robert Brown",
"Michael Wilson"
],
"email": [
"john@example.com",
"alice@example.com",
"robert@example.com",
"michael@example.com"
],
"city": [
"New York",
"Chicago",
"Boston",
"Seattle"
]
}
data_b = {
"customer_id": [201, 202, 203, 204],
"name": [
"Jon Smith",
"Alice Jonson",
"Emily Davis",
"Michael Wilson"
],
"email": [
"john@example.com",
"alice@example.com",
"emily@example.com",
"michael@example.com"
],
"city": [
"New York",
"Chicago",
"Seattle",
"Seattle"
]
}
df_a = pd.DataFrame(data_a).set_index("customer_id")
df_b = pd.DataFrame(data_b).set_index("customer_id")

import pandas as pd

data_a = {
    "customer_id": [101, 102, 103, 104],
    "name": [
        "John Smith",
        "Alice Johnson",
        "Robert Brown",
        "Michael Wilson"
    ],
    "email": [
        "john@example.com",
        "alice@example.com",
        "robert@example.com",
        "michael@example.com"
    ],
    "city": [
        "New York",
        "Chicago",
        "Boston",
        "Seattle"
    ]
}

data_b = {
    "customer_id": [201, 202, 203, 204],
    "name": [
        "Jon Smith",
        "Alice Jonson",
        "Emily Davis",
        "Michael Wilson"
    ],
    "email": [
        "john@example.com",
        "alice@example.com",
        "emily@example.com",
        "michael@example.com"
    ],
    "city": [
        "New York",
        "Chicago",
        "Seattle",
        "Seattle"
    ]
}

df_a = pd.DataFrame(data_a).set_index("customer_id")
df_b = pd.DataFrame(data_b).set_index("customer_id")

Setting customer_id as the index makes it easier to identify matching records using their original identifiers.

Our goal is to find records in Dataset A that correspond to records in Dataset B.

Step 1: Generating Candidate Record Pairs

The first step in record linkage is generating candidate pairs.

A candidate pair consists of two records that will be compared to determine whether they represent the same entity.

For small datasets, we can compare every record in one dataset against every record in another dataset.

The Record Linkage Toolkit provides the Index class for generating these pairs.

indexer = recordlinkage.Index()
indexer.full()
candidate_pairs = indexer.index(df_a, df_b)
print(candidate_pairs)

indexer = recordlinkage.Index()

indexer.full()

candidate_pairs = indexer.index(df_a, df_b)

print(candidate_pairs)

Output:

MultiIndex([(101, 201),
(101, 202),
(101, 203),
(101, 204),
(102, 201),
(102, 202),
(102, 203),
(102, 204),
(103, 201),
(103, 202),
(103, 203),
(103, 204),
(104, 201),
(104, 202),
(104, 203),
(104, 204)],
names=['customer_id_1', 'customer_id_2'])

MultiIndex([(101, 201),
            (101, 202),
            (101, 203),
            (101, 204),
            (102, 201),
            (102, 202),
            (102, 203),
            (102, 204),
            (103, 201),
            (103, 202),
            (103, 203),
            (103, 204),
            (104, 201),
            (104, 202),
            (104, 203),
            (104, 204)],
           names=['customer_id_1', 'customer_id_2'])

Because each dataset contains four records, the total number of candidate pairs is: 4 x 4 16

Full indexing is useful for demonstrating how record linkage works, but it becomes expensive when datasets contain thousands or millions of records.

For example, comparing two datasets containing 100,000 records each would generate 10 billion candidate pairs.

To avoid this, record linkage systems commonly use blocking or other indexing strategies.

Step 2: Reducing Comparisons Using Blocking

Blocking is a technique that reduces the number of candidate pairs by comparing only records that share a selected attribute.

For example, if customer records contain city information, we can compare only customers from the same city.

indexer = recordlinkage.Index()
indexer.block("city")
candidate_pairs = indexer.index(df_a, df_b)
print(candidate_pairs)

indexer = recordlinkage.Index()

indexer.block("city")

candidate_pairs = indexer.index(df_a, df_b)

print(candidate_pairs)

Output:

MultiIndex([(101, 201),
(102, 202),
(104, 203),
(104, 204)],
names=['customer_id_1', 'customer_id_2'])

MultiIndex([(101, 201),
            (102, 202),
            (104, 203),
            (104, 204)],
           names=['customer_id_1', 'customer_id_2'])

Instead of generating 16 candidate pairs, blocking reduces the comparisons to four.

This significantly improves efficiency, especially for large datasets.

However, blocking introduces an important trade-off: records with different values in the blocking field will not be compared.

For example, if one dataset stores a customer’s city as “New York” and another stores it as “NYC,” city-based blocking may exclude a genuine match.

Therefore, blocking attributes should be selected carefully. In some cases, multiple blocking strategies can be combined to improve candidate coverage.

Step 3: Comparing Records Using Multiple Attributes

After generating candidate pairs, the next step is to compare their attributes.

The Record Linkage Toolkit provides the Compare class for defining comparison rules.

For example, we can compare customer names using approximate string matching and compare email addresses and cities using exact matching.

compare = recordlinkage.Compare()
# Fuzzy comparison for customer names
compare.string(
"name",
"name",
method="jarowinkler",
threshold=0.85,
label="name_match"
)
# Exact comparison for email addresses
compare.exact(
"email",
"email",
label="email_match"
)
# Exact comparison for cities
compare.exact(
"city",
"city",
label="city_match"
)
features = compare.compute(
candidate_pairs,
df_a,
df_b
)
print(features)

compare = recordlinkage.Compare()

# Fuzzy comparison for customer names
compare.string(
    "name",
    "name",
    method="jarowinkler",
    threshold=0.85,
    label="name_match"
)

# Exact comparison for email addresses
compare.exact(
    "email",
    "email",
    label="email_match"
)

# Exact comparison for cities
compare.exact(
    "city",
    "city",
    label="city_match"
)

features = compare.compute(
    candidate_pairs,
    df_a,
    df_b
)

print(features)

Expected output:

                             name_match  email_match  city_match
customer_id_1 customer_id_2
101           201                   1.0          1.0         1.0
102           202                   1.0          1.0         1.0
104           203                   0.0          0.0         1.0
104           204                   1.0          1.0         1.0

Here, each comparison produces a binary result:

• 1.0 indicates that the comparison meets the matching criteria.
• 0.0 indicates that the comparison does not meet the criteria.

The name comparison uses the Jaro-Winkler similarity algorithm with a threshold of 0.85.

Because a threshold is supplied, the name comparison produces a binary match indicator rather than a continuous similarity score.

For more detailed analysis, we can omit the threshold and retain the actual similarity score.

Step 4: Calculating Similarity Scores

Instead of relying entirely on binary comparisons, we can calculate a similarity score for each candidate pair.

This allows us to evaluate how closely records match across multiple attributes.

First, let’s configure the name comparison to return a continuous similarity score.

compare = recordlinkage.Compare()
compare.string(
"name",
"name",
method="jarowinkler",
label="name_similarity"
)
compare.exact(
"email",
"email",
label="email_match"
)
compare.exact(
"city",
"city",
label="city_match"
)
features = compare.compute(
candidate_pairs,
df_a,
df_b
)
print(features)

compare = recordlinkage.Compare()

compare.string(
    "name",
    "name",
    method="jarowinkler",
    label="name_similarity"
)

compare.exact(
    "email",
    "email",
    label="email_match"
)

compare.exact(
    "city",
    "city",
    label="city_match"
)

features = compare.compute(
    candidate_pairs,
    df_a,
    df_b
)

print(features)

The resulting DataFrame contains comparison scores for each candidate pair.

We can then calculate a weighted score that gives different importance to each attribute.

For example:

features["match_score"] = (
features["name_similarity"] * 0.4
+ features["email_match"] * 0.4
+ features["city_match"] * 0.2
)
print(features)

features["match_score"] = (
    features["name_similarity"] * 0.4
    + features["email_match"] * 0.4
    + features["city_match"] * 0.2
)

print(features)

In this example:

• Name similarity contributes 40% of the total score.
• Email matching contributes 40%.
• City matching contributes 20%.

The resulting score ranges from 0 to 1.

A score closer to 1 indicates stronger agreement between the compared attributes.

These weights are illustrative. In production systems, weights should be selected based on data quality, attribute reliability, and validation results rather than arbitrary assumptions.

Step 5: Identifying Potential Duplicate Records

Once similarity scores have been calculated, we can apply a matching threshold to identify potential duplicates.

For example, let’s classify candidate pairs with a weighted score of at least 0.85 as potential matches.

threshold = 0.85
potential_duplicates = features[
features["match_score"] >= threshold
]
print(potential_duplicates)

threshold = 0.85

potential_duplicates = features[
    features["match_score"] >= threshold
]

print(potential_duplicates)

Expected matching pairs:

These pairs represent records that have strong similarities across the selected attributes.

The threshold should be adjusted based on the dataset and the acceptable balance between false positives and false negatives.

For example, a high threshold may reduce false positives but miss legitimate duplicates with incomplete or inconsistent information.

A lower threshold may identify more potential duplicates but require additional validation.

Step 6: Exporting Duplicate Detection Results

After identifying potential duplicates, we can export the results for review or further processing.

First, convert the MultiIndex into regular columns:

results = potential_duplicates.reset_index()

results = potential_duplicates.reset_index()

Then save the results to a CSV file:

results.to_csv(
"record_linkage_matches.csv",
index=False
)

results.to_csv(
    "record_linkage_matches.csv",
    index=False
)

The resulting CSV file contains the matched record identifiers, comparison scores, and overall matching scores.

This file can be used for:

• Manual review of potential duplicates.
• Data quality reporting.
• Customer record consolidation.
• Downstream deduplication workflows.
• Auditing and validation.

Exporting candidate matches rather than immediately deleting records helps prevent accidental data loss.

Detecting Duplicates Within a Single Dataset

Record linkage is not limited to comparing two different datasets. It can also identify duplicates within a single DataFrame.

For example, consider a customer database containing repeated or slightly different records.

import pandas as pd
import recordlinkage
data = {
"customer_id": [101, 102, 103, 104],
"name": [
"John Smith",
"Jon Smith",
"Alice Johnson",
"Alice Jonson"
],
"email": [
"john@example.com",
"john@example.com",
"alice@example.com",
"alice@example.com"
]
}
df = pd.DataFrame(data).set_index("customer_id")

import pandas as pd
import recordlinkage

data = {
    "customer_id": [101, 102, 103, 104],
    "name": [
        "John Smith",
        "Jon Smith",
        "Alice Johnson",
        "Alice Jonson"
    ],
    "email": [
        "john@example.com",
        "john@example.com",
        "alice@example.com",
        "alice@example.com"
    ]
}

df = pd.DataFrame(data).set_index("customer_id")

We can generate candidate pairs within the same dataset using the Index class.

indexer = recordlinkage.Index()
indexer.full()
candidate_pairs = indexer.index(df)

indexer = recordlinkage.Index()

indexer.full()

candidate_pairs = indexer.index(df)

When indexing a single DataFrame, the toolkit generates unique candidate pairs without comparing each record against itself.

Next, define the comparison rules:

compare = recordlinkage.Compare()
compare.string(
"name",
"name",
method="jarowinkler",
threshold=0.85,
label="name_match"
)
compare.exact(
"email",
"email",
label="email_match"
)
features = compare.compute(
candidate_pairs,
df
)

compare = recordlinkage.Compare()

compare.string(
    "name",
    "name",
    method="jarowinkler",
    threshold=0.85,
    label="name_match"
)

compare.exact(
    "email",
    "email",
    label="email_match"
)

features = compare.compute(
    candidate_pairs,
    df
)

Finally, identify records that match on both attributes:

duplicates = features[
(features["name_match"] == 1)
& (features["email_match"] == 1)
]
print(duplicates)

duplicates = features[
    (features["name_match"] == 1)
    & (features["email_match"] == 1)
]

print(duplicates)

This approach can identify potential duplicate customers even when their names contain spelling variations.

For larger datasets, replace full indexing with blocking or another candidate-generation strategy to reduce the number of comparisons.

Record Linkage vs. Fuzzy Matching

Although record linkage often incorporates fuzzy matching, the two approaches serve different purposes.

Fuzzy matching is useful when comparing individual text values, while record linkage provides a more complete framework for identifying matching records across multiple attributes.

For example, fuzzy matching can determine whether “Jon Smith” resembles “John Smith.” Record linkage can combine that name similarity with email, address, and phone number comparisons to determine whether the records are likely to represent the same customer.

Detecting Semantic Duplicates Using Machine Learning

Traditional duplicate detection techniques, such as exact matching and fuzzy matching, are effective when records contain identical values or minor text variations. However, they may struggle when two records describe the same entity or concept using significantly different words.

For example, consider the following product descriptions:

• “Wireless Bluetooth Headphones”
• “Cordless Bluetooth Audio Headset”
• “Wireless Headphones with Bluetooth Connectivity”

Although these descriptions use different wording, they convey similar meanings and may refer to the same product.

Exact matching would treat them as different strings, while traditional fuzzy matching might assign relatively low similarity scores because the words and their order differ.

This is where semantic duplicate detection using machine learning in Python becomes useful.

Semantic duplicate detection uses natural language processing (NLP) and machine learning techniques to identify similarities based on the meaning of text rather than relying entirely on character-level or word-level matching.

In this section, we’ll explore semantic similarity, use TF-IDF and cosine similarity to detect related text records, and implement more advanced semantic duplicate detection using Sentence Transformers.

Understanding Semantic Similarity

Semantic similarity measures how closely two pieces of text are related in meaning, even when they use different words or sentence structures.

For example, consider these descriptions:

The first three descriptions are semantically related because they describe similar audio products, while the fourth describes a completely different type of product.

Traditional string matching algorithms focus on character differences or token overlap. Semantic similarity models attempt to capture broader relationships between words and phrases.

This makes semantic similarity particularly useful for detecting potential duplicates in:

• Product catalogs containing different product descriptions.
• Document repositories with similar articles or reports.
• Customer support systems containing repeated questions.
• E-commerce listings with inconsistent titles.
• Knowledge bases containing overlapping information.
• Content management systems containing paraphrased text.

However, an important distinction is that semantic similarity does not necessarily imply duplicate identity.

For example, “Apple iPhone 15” and “Apple iPhone 15 Pro” are semantically similar but represent different products.

Therefore, semantic duplicate detection should often be combined with additional attributes, identifiers, or business rules to avoid incorrectly merging distinct records.

Using TF-IDF and Cosine Similarity

One of the simplest approaches to text similarity in Python is combining TF-IDF (Term Frequency–Inverse Document Frequency) with cosine similarity.

TF-IDF converts text into numerical vectors based on the importance of words within a collection of documents.

Words that occur frequently in a particular document but are relatively uncommon across the dataset receive higher weights.

Once text has been converted into vectors, cosine similarity measures how closely those vectors align.

Although TF-IDF does not understand meaning in the same way as modern language models, it provides a useful baseline for identifying records with overlapping vocabulary.

Step 1: Install scikit-learn

Install the required library and then import the necessary modules:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

import pandas as pd

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

Step 2: Create a Sample Dataset

Let’s create a dataset containing product descriptions with overlapping terms and different wording.

data = {
"product_id": [101, 102, 103, 104, 105],
"description": [
"Wireless Bluetooth Headphones",
"Bluetooth Wireless Headset",
"Wireless Headphones with Bluetooth",
"Stainless Steel Water Bottle",
"Insulated Stainless Steel Bottle"
]
}
df = pd.DataFrame(data)
print(df)

data = {
    "product_id": [101, 102, 103, 104, 105],
    "description": [
        "Wireless Bluetooth Headphones",
        "Bluetooth Wireless Headset",
        "Wireless Headphones with Bluetooth",
        "Stainless Steel Water Bottle",
        "Insulated Stainless Steel Bottle"
    ]
}

df = pd.DataFrame(data)

print(df)

Dataset:

Our goal is to identify descriptions that may refer to the same or closely related products.

Step 3: Convert Text into TF-IDF Vectors

The TfidfVectorizer class converts text descriptions into numerical representations.

vectorizer = TfidfVectorizer(
lowercase=True,
stop_words="english"
)
tfidf_matrix = vectorizer.fit_transform(
df["description"]
)
print(tfidf_matrix.shape)

vectorizer = TfidfVectorizer(
    lowercase=True,
    stop_words="english"
)

tfidf_matrix = vectorizer.fit_transform(
    df["description"]
)

print(tfidf_matrix.shape)

Output:

(5, 10)

(5, 10)

The resulting matrix contains five rows, one for each product description, and ten columns representing the distinct terms retained after preprocessing.

Each row is a numerical representation of a product description.

Step 4: Calculate Cosine Similarity

Cosine similarity measures the angle between two vectors.

Its formula is:

Where:

• (A) and (B) are the vector representations of two text records.
• (A \cdot B) is their dot product.
• (|A|) and (|B|) represent their vector magnitudes.

For nonnegative TF-IDF vectors, cosine similarity ranges from 0 to 1.

A score closer to 1 indicates greater similarity in weighted vocabulary, while a score closer to 0 indicates little or no overlap.

Let’s calculate the similarity matrix:

similarity_matrix = cosine_similarity(
tfidf_matrix
)
print(similarity_matrix.round(2))

similarity_matrix = cosine_similarity(
    tfidf_matrix
)

print(similarity_matrix.round(2))

The resulting matrix contains similarity scores for every pair of product descriptions.

The diagonal values are 1 because each description is compared with itself.

Step 5: Identify Potential Duplicates

We can now identify product descriptions with similarity scores above a selected threshold.

threshold = 0.60
matches = []
for i in range(len(df)):
for j in range(i + 1, len(df)):
score = similarity_matrix[i, j]
if score >= threshold:
matches.append({
"product_id_1": df.iloc[i]["product_id"],
"product_id_2": df.iloc[j]["product_id"],
"similarity": round(float(score), 3)
})
duplicates_df = pd.DataFrame(matches)
print(duplicates_df)

threshold = 0.60

matches = []

for i in range(len(df)):
    for j in range(i + 1, len(df)):

        score = similarity_matrix[i, j]

        if score >= threshold:
            matches.append({
                "product_id_1": df.iloc[i]["product_id"],
                "product_id_2": df.iloc[j]["product_id"],
                "similarity": round(float(score), 3)
            })

duplicates_df = pd.DataFrame(matches)

print(duplicates_df)

This code:

1. Compares each product description with every other description.
2. Excludes self-comparisons.
3. Avoids comparing the same pair twice.
4. Flags pairs with cosine similarity scores of at least 0.60.
5. Stores the matching product IDs and similarity scores.

The threshold is illustrative and should be adjusted based on actual validation data.

TF-IDF works particularly well when descriptions share important keywords. However, it may struggle when semantically equivalent descriptions use different vocabulary.

For more advanced meaning-based comparisons, we can use sentence embeddings.

Using Sentence Transformers for Semantic Duplicate Detection

Sentence Transformers is a Python library that provides pretrained language models for generating semantic embeddings.

An embedding is a numerical vector representing the meaning or contextual information of a piece of text.

Unlike TF-IDF, which primarily relies on word frequency, sentence embedding models can capture relationships between different expressions.

For example:

• “Buy a new laptop”
• “Purchase a notebook computer”

These sentences use different words but express similar intentions.

A suitable sentence embedding model can represent them as vectors that are relatively close in semantic space.

This makes Sentence Transformers useful for detecting potential semantic duplicates in product descriptions, documents, customer support tickets, and other unstructured text.

Step 1: Import and Load a Pretrained Model

from sentence_transformers import SentenceTransformer
model = SentenceTransformer(
"sentence-transformers/all-MiniLM-L6-v2"
)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2"
)

The all-MiniLM-L6-v2 model is a lightweight sentence embedding model commonly used for semantic similarity tasks.

It generates 384-dimensional embeddings and offers a useful balance between performance and computational efficiency.

The model is downloaded automatically the first time it is loaded, unless it is already available in the local cache.

Step 2: Create a Sample Dataset

Let’s use product descriptions with different wording.

import pandas as pd
data = {
"product_id": [101, 102, 103, 104, 105],
"description": [
"Wireless Bluetooth Headphones",
"Cordless Bluetooth Audio Headset",
"Bluetooth-enabled Wireless Headphones",
"Stainless Steel Water Bottle",
"Insulated Metal Drinking Bottle"
]
}
df = pd.DataFrame(data)

import pandas as pd

data = {
    "product_id": [101, 102, 103, 104, 105],
    "description": [
        "Wireless Bluetooth Headphones",
        "Cordless Bluetooth Audio Headset",
        "Bluetooth-enabled Wireless Headphones",
        "Stainless Steel Water Bottle",
        "Insulated Metal Drinking Bottle"
    ]
}

df = pd.DataFrame(data)

Some descriptions refer to similar types of products, even though their words differ.

Our goal is to identify these relationships using semantic embeddings.

Step 3: Generate Sentence Embeddings

Use the model’s encode() method to convert the descriptions into numerical vectors.

embeddings = model.encode(
df["description"].tolist(),
normalize_embeddings=True
)
print(embeddings.shape)

embeddings = model.encode(
    df["description"].tolist(),
    normalize_embeddings=True
)

print(embeddings.shape)

Output:

(5, 384)

(5, 384)

Each product description is represented by a 384-dimensional vector.

The normalize_embeddings=True parameter normalizes the vectors to unit length, making them convenient for cosine similarity calculations.

Step 4: Calculate Semantic Similarity

Now calculate cosine similarity between the generated embeddings.

from sklearn.metrics.pairwise import cosine_similarity
similarity_matrix = cosine_similarity(
embeddings
)
print(similarity_matrix.round(3))

from sklearn.metrics.pairwise import cosine_similarity

similarity_matrix = cosine_similarity(
    embeddings
)

print(similarity_matrix.round(3))

Unlike TF-IDF, sentence embeddings can identify similarities between descriptions that use different but related words.

For example, a semantic model may recognize that “cordless” and “wireless” express similar concepts in the context of audio products.

The exact similarity scores depend on the model and input text.

Step 5: Identify Potential Semantic Duplicates

Let’s flag product pairs with semantic similarity scores of at least 0.75.

threshold = 0.75
semantic_matches = []
for i in range(len(df)):
for j in range(i + 1, len(df)):
score = similarity_matrix[i, j]
if score >= threshold:
semantic_matches.append({
"product_id_1": df.iloc[i]["product_id"],
"product_id_2": df.iloc[j]["product_id"],
"description_1": df.iloc[i]["description"],
"description_2": df.iloc[j]["description"],
"similarity": round(float(score), 3)
})
semantic_duplicates = pd.DataFrame(
semantic_matches
)
print(semantic_duplicates)

threshold = 0.75

semantic_matches = []

for i in range(len(df)):
    for j in range(i + 1, len(df)):

        score = similarity_matrix[i, j]

        if score >= threshold:
            semantic_matches.append({
                "product_id_1": df.iloc[i]["product_id"],
                "product_id_2": df.iloc[j]["product_id"],
                "description_1": df.iloc[i]["description"],
                "description_2": df.iloc[j]["description"],
                "similarity": round(float(score), 3)
            })

semantic_duplicates = pd.DataFrame(
    semantic_matches
)

print(semantic_duplicates)

This code returns candidate pairs whose semantic similarity meets the specified threshold.

Depending on the model’s output, the results may include pairs of wireless headphone descriptions or similar water bottle descriptions.

However, these matches should be treated as potential semantic duplicates, not automatically confirmed duplicates.

Two descriptions can be semantically similar while referring to different products.

Building a Reusable Semantic Duplicate Detection Function

Instead of writing the same comparison logic repeatedly, we can create a reusable function that detects semantically similar records in a DataFrame.

import pandas as pd
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
def detect_semantic_duplicates(
df,
text_column,
threshold=0.80,
model_name="sentence-transformers/all-MiniLM-L6-v2"
):
# Load embedding model
model = SentenceTransformer(model_name)
# Prepare text data
texts = (
df[text_column]
.fillna("")
.astype(str)
.tolist()
)
# Generate embeddings
embeddings = model.encode(
texts,
normalize_embeddings=True
)
# Calculate pairwise similarities
similarity_matrix = cosine_similarity(embeddings)
matches = []
# Compare unique record pairs
for i in range(len(df)):
for j in range(i + 1, len(df)):
score = similarity_matrix[i, j]
if score >= threshold:
matches.append({
"record_1": i,
"record_2": j,
"text_1": texts[i],
"text_2": texts[j],
"similarity": round(float(score), 3)
})
return pd.DataFrame(
matches,
columns=[
"record_1",
"record_2",
"text_1",
"text_2",
"similarity"
]
)

import pandas as pd

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

def detect_semantic_duplicates(
    df,
    text_column,
    threshold=0.80,
    model_name="sentence-transformers/all-MiniLM-L6-v2"
):
    # Load embedding model
    model = SentenceTransformer(model_name)

    # Prepare text data
    texts = (
        df[text_column]
        .fillna("")
        .astype(str)
        .tolist()
    )

    # Generate embeddings
    embeddings = model.encode(
        texts,
        normalize_embeddings=True
    )

    # Calculate pairwise similarities
    similarity_matrix = cosine_similarity(embeddings)

    matches = []

    # Compare unique record pairs
    for i in range(len(df)):
        for j in range(i + 1, len(df)):

            score = similarity_matrix[i, j]

            if score >= threshold:
                matches.append({
                    "record_1": i,
                    "record_2": j,
                    "text_1": texts[i],
                    "text_2": texts[j],
                    "similarity": round(float(score), 3)
                })

    return pd.DataFrame(
        matches,
        columns=[
            "record_1",
            "record_2",
            "text_1",
            "text_2",
            "similarity"
        ]
    )

Now we can apply the function to any DataFrame containing text records.

duplicates = detect_semantic_duplicates(
df,
text_column="description",
threshold=0.75
)
print(duplicates)

duplicates = detect_semantic_duplicates(
    df,
    text_column="description",
    threshold=0.75
)

print(duplicates)

The function:

• Accepts a Pandas DataFrame.
• Generates embeddings for the selected text column.
• Calculates semantic similarity between records.
• Filters candidate pairs using a configurable threshold.
• Returns potential duplicates with similarity scores.

This provides a reusable starting point for semantic duplicate detection in Python.

For production applications, the function should be extended with input validation, configurable identifiers, empty-text handling, and more efficient similarity search.

Combining Semantic Similarity with Business Rules

Semantic similarity is powerful, but relying on it alone can lead to incorrect duplicate matches.

Consider these products:

• Apple iPhone 15, 128GB
• Apple iPhone 15, 256GB
• Apple iPhone 15 Pro, 128GB

These descriptions are semantically similar, but they represent different product variants.

A semantic similarity model may assign them high scores because they share many words and concepts.

To improve accuracy, we can combine semantic similarity with structured attributes such as brand, model number, SKU, or product category.

For example:

def is_potential_duplicate(
similarity_score,
brand_match,
model_match,
threshold=0.85
):
return (
similarity_score >= threshold
and brand_match
and model_match
)

def is_potential_duplicate(
    similarity_score,
    brand_match,
    model_match,
    threshold=0.85
):
    return (
        similarity_score >= threshold
        and brand_match
        and model_match
    )

This function requires:

• A sufficiently high semantic similarity score.
• Matching brands.
• Matching model identifiers.

In a real-world product matching system, additional attributes such as storage capacity, size, color, or version may also be required.

Combining semantic similarity with structured matching rules helps distinguish genuinely duplicate records from merely related items.

Optimizing Semantic Duplicate Detection for Large Datasets

The previous examples calculate a complete pairwise similarity matrix.

While this approach is convenient for small datasets, it becomes expensive as the number of records increases.

For (n) records, the number of unique comparisons is:

For example, 100,000 records require nearly five billion unique pair comparisons.

Storing a full similarity matrix also requires memory proportional to (n^2).

For large-scale semantic duplicate detection, more efficient techniques are needed.

Approximate Nearest Neighbor Search

Approximate nearest neighbor (ANN) search identifies vectors that are likely to be close to a query vector without comparing every possible pair.

Popular tools include:

• FAISS
• Annoy
• hnswlib

These tools can help retrieve likely semantic matches from large collections of embeddings.

Blocking and Candidate Generation

Another approach is to group records using structured attributes before performing semantic comparisons.

For example, product descriptions can be grouped by category or brand, reducing unnecessary comparisons between unrelated products.

Batch Processing

Sentence Transformers supports batch processing, allowing multiple text records to be encoded together.

embeddings = model.encode(
df["description"].tolist(),
batch_size=32,
normalize_embeddings=True
)

embeddings = model.encode(
    df["description"].tolist(),
    batch_size=32,
    normalize_embeddings=True
)

Batch size can be adjusted according to available memory and hardware.

Embedding Reuse

For frequently updated datasets, embeddings can be stored and reused instead of recalculating them for every duplicate detection run.

Only new or modified records need to be encoded again, provided the same model and preprocessing rules are used.

Comparing TF-IDF, Fuzzy Matching, and Sentence Transformers

Different similarity techniques offer different advantages depending on the data.

Fuzzy matching is generally a good choice for short strings containing spelling variations.

TF-IDF is useful when duplicate records share important keywords and a lightweight solution is preferred.

Sentence Transformers is better suited to detecting paraphrases or meaning-based similarities, particularly in longer text.

In practice, combining multiple approaches often produces more reliable results than using a single technique.

Building a Complete Duplicate Detection Pipeline in Python

So far, we’ve explored several techniques for duplicate detection in Python, including exact matching with Pandas, fuzzy matching with RapidFuzz, record linkage, and semantic similarity using machine learning.

While each technique is useful on its own, real-world datasets often contain different types of duplicates. Some records may be identical, others may contain spelling variations, and some may describe the same entity using entirely different words.

To handle these scenarios effectively, we can combine multiple techniques into a single duplicate detection pipeline in Python.

A duplicate detection pipeline automates the process of loading data, cleaning and standardizing records, identifying potential matches, calculating similarity scores, and exporting results for review.

In this section, we’ll build an end-to-end duplicate detection workflow using Pandas, RapidFuzz, and Sentence Transformers.

Our pipeline will follow seven steps:

1. Load the dataset.
2. Clean and standardize data.
3. Detect exact duplicates.
4. Detect near duplicates using fuzzy matching.
5. Identify semantic duplicates using machine learning.
6. Classify and review potential matches.
7. Export duplicate detection results.

Step 1: Load the Dataset

Let’s begin by creating a sample dataset containing customer records with exact duplicates, spelling variations, and different company name representations.

import pandas as pd
data = {
"customer_id": [
101, 102, 103, 104, 105,
106, 107, 108, 109, 110
],
"name": [
"John Smith",
"John Smith",
"Jon Smith",
"Alice Johnson",
"Alice Jonson",
"Robert Brown",
"Emily Davis",
"International Business Machines",
"IBM Corporation",
"Michael Wilson"
],
"email": [
"john@example.com",
"john@example.com",
"john@example.com",
"alice@example.com",
"alice@example.com",
"robert@example.com",
"emily@example.com",
None,
None,
"michael@example.com"
]
}
df = pd.DataFrame(data)
print(df)

import pandas as pd

data = {
    "customer_id": [
        101, 102, 103, 104, 105,
        106, 107, 108, 109, 110
    ],
    "name": [
        "John Smith",
        "John Smith",
        "Jon Smith",
        "Alice Johnson",
        "Alice Jonson",
        "Robert Brown",
        "Emily Davis",
        "International Business Machines",
        "IBM Corporation",
        "Michael Wilson"
    ],
    "email": [
        "john@example.com",
        "john@example.com",
        "john@example.com",
        "alice@example.com",
        "alice@example.com",
        "robert@example.com",
        "emily@example.com",
        None,
        None,
        "michael@example.com"
    ]
}

df = pd.DataFrame(data)

print(df)

Our dataset contains several types of potential duplicates:

The final two organization-related records illustrate a potential semantic or alias-based match. They would require additional evidence to confirm that they represent the same organization.

In real-world applications, you would typically load data from a CSV file or database:

df = pd.read_csv("customers.csv")

Before proceeding, verify that the dataset contains the expected columns and unique record identifiers.

Step 2: Clean and Standardize Data

Data preprocessing is essential because inconsistent formatting can prevent matching algorithms from identifying duplicate records.

For example, these values may represent the same name:

• John Smith
• john smith
• John Smith
• JOHN SMITH

We can standardize text values by removing unnecessary whitespace and converting them to lowercase.

def normalize_text(value):
if pd.isna(value):
return ""
return " ".join(
str(value).strip().casefold().split()
)
df["name_clean"] = df["name"].apply(normalize_text)
df["email_clean"] = df["email"].apply(
normalize_text
)
print(df[["name", "name_clean"]])

def normalize_text(value):
    if pd.isna(value):
        return ""

    return " ".join(
        str(value).strip().casefold().split()
    )

df["name_clean"] = df["name"].apply(normalize_text)

df["email_clean"] = df["email"].apply(
    normalize_text
)

print(df[["name", "name_clean"]])

This creates normalized versions of the original fields without overwriting the source data.

For example:

Keeping both original and normalized values is useful for auditing and reviewing detected duplicates.

Important: Missing values should not automatically be considered matching identifiers. Two records with missing email addresses are not duplicates simply because both emails are absent.

Step 3: Detect Exact Duplicates

The first detection stage identifies records containing identical values across selected fields.

Pandas provides the duplicated() method for this purpose.

exact_mask = df.duplicated(
subset=["name_clean", "email_clean"],
keep=False
)
exact_duplicates = df.loc[
exact_mask,
["customer_id", "name", "email"]
]
print(exact_duplicates)

exact_mask = df.duplicated(
    subset=["name_clean", "email_clean"],
    keep=False
)

exact_duplicates = df.loc[
    exact_mask,
    ["customer_id", "name", "email"]
]

print(exact_duplicates)

Expected result:

These records contain identical names and email addresses, even though their customer IDs differ.

We can also count redundant records beyond the first occurrence:

exact_duplicate_count = df.duplicated(
subset=["name_clean", "email_clean"]
).sum()
print("Redundant exact records:", exact_duplicate_count)

exact_duplicate_count = df.duplicated(
    subset=["name_clean", "email_clean"]
).sum()

print("Redundant exact records:", exact_duplicate_count)

Output:

Redundant exact records: 1

Redundant exact records: 1

Exact matching is computationally efficient and should generally be performed before more expensive similarity calculations.

However, it cannot identify spelling variations such as “John Smith” and “Jon Smith.”

For that, we need fuzzy matching.

Step 4: Detect Near Duplicates Using Fuzzy Matching

Next, we’ll use RapidFuzz to identify records with similar names.

Import the similarity function:

from rapidfuzz import fuzz

from rapidfuzz import fuzz

We’ll compare customer names using fuzz.ratio() and combine the results with email matching.

fuzzy_matches = []
for i in range(len(df)):
for j in range(i + 1, len(df)):
record1 = df.iloc[i]
record2 = df.iloc[j]
name_score = fuzz.ratio(
record1["name_clean"],
record2["name_clean"]
)
email_match = (
bool(record1["email_clean"])
and bool(record2["email_clean"])
and record1["email_clean"]
== record2["email_clean"]
)
if name_score >= 85 and email_match:
fuzzy_matches.append({
"record_1": record1["customer_id"],
"record_2": record2["customer_id"],
"name_similarity": round(name_score, 2),
"email_match": True
})
fuzzy_df = pd.DataFrame(fuzzy_matches)
print(fuzzy_df)

fuzzy_matches = []

for i in range(len(df)):
    for j in range(i + 1, len(df)):

        record1 = df.iloc[i]
        record2 = df.iloc[j]

        name_score = fuzz.ratio(
            record1["name_clean"],
            record2["name_clean"]
        )

        email_match = (
            bool(record1["email_clean"])
            and bool(record2["email_clean"])
            and record1["email_clean"]
                == record2["email_clean"]
        )

        if name_score >= 85 and email_match:

            fuzzy_matches.append({
                "record_1": record1["customer_id"],
                "record_2": record2["customer_id"],
                "name_similarity": round(name_score, 2),
                "email_match": True
            })

fuzzy_df = pd.DataFrame(fuzzy_matches)

print(fuzzy_df)

Expected results:

This approach identifies exact and near-duplicate customer names that share an email address.

The results demonstrate how combining fuzzy matching with structured attributes can improve reliability compared with name similarity alone.

However, a shared email address is not always sufficient to prove that two records represent the same person, especially when email accounts are shared.

Step 5: Identify Semantic Duplicates Using Machine Learning

Some potential duplicates cannot be identified reliably through exact or fuzzy matching alone.

For example:

• International Business Machines
• IBM Corporation

These names may refer to the same organization, but they have limited character-level similarity.

To identify semantic relationships, we can use Sentence Transformers.

Load a pretrained sentence embedding model:

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
model = SentenceTransformer(
"sentence-transformers/all-MiniLM-L6-v2"
)

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2"
)

Generate embeddings for the customer names:

embeddings = model.encode(
df["name_clean"].tolist(),
normalize_embeddings=True
)

embeddings = model.encode(
    df["name_clean"].tolist(),
    normalize_embeddings=True
)

Each name is converted into a numerical vector representing information learned by the language model.

Next, calculate pairwise cosine similarity:

semantic_matrix = cosine_similarity(embeddings)

semantic_matrix = cosine_similarity(embeddings)

Now identify records with high semantic similarity:

semantic_matches = []
threshold = 0.80
for i in range(len(df)):
for j in range(i + 1, len(df)):
score = semantic_matrix[i, j]
if score >= threshold:
semantic_matches.append({
"record_1": df.iloc[i]["customer_id"],
"record_2": df.iloc[j]["customer_id"],
"semantic_similarity": round(
float(score), 3
)
})
semantic_df = pd.DataFrame(semantic_matches)
print(semantic_df)

semantic_matches = []

threshold = 0.80

for i in range(len(df)):
    for j in range(i + 1, len(df)):

        score = semantic_matrix[i, j]

        if score >= threshold:
            semantic_matches.append({
                "record_1": df.iloc[i]["customer_id"],
                "record_2": df.iloc[j]["customer_id"],
                "semantic_similarity": round(
                    float(score), 3
                )
            })

semantic_df = pd.DataFrame(semantic_matches)

print(semantic_df)

The results will contain pairs whose semantic similarity meets the selected threshold.

Important limitation: Sentence embedding models do not reliably resolve all company aliases or abbreviations. The model may or may not recognize that “IBM Corporation” and “International Business Machines” refer to the same organization.

For organization matching, combining semantic similarity with a verified alias dictionary, company identifiers, or entity resolution rules is often more reliable.

Additionally, high semantic similarity does not guarantee that two records represent the same customer or organization.

Step 6: Combine and Classify Duplicate Matches

After running exact, fuzzy, and semantic matching, we need to consolidate the results into a single dataset.

A practical duplicate detection system should avoid reporting the same pair multiple times without indicating which detection methods identified it.

We can create a combined list of candidate pairs and their matching evidence.

from collections import defaultdict
match_results = defaultdict(set)
# Exact duplicate pairs
for _, group in df.groupby(
["name_clean", "email_clean"],
sort=False
):
ids = group["customer_id"].tolist()
for i in range(len(ids)):
for j in range(i + 1, len(ids)):
pair = tuple(sorted((ids[i], ids[j])))
match_results[pair].add("exact")
# Fuzzy duplicate pairs
for match in fuzzy_matches:
pair = tuple(sorted((
match["record_1"],
match["record_2"]
)))
match_results[pair].add("fuzzy")
# Semantic duplicate pairs
for match in semantic_matches:
pair = tuple(sorted((
match["record_1"],
match["record_2"]
)))
match_results[pair].add("semantic")

from collections import defaultdict

match_results = defaultdict(set)

# Exact duplicate pairs
for _, group in df.groupby(
    ["name_clean", "email_clean"],
    sort=False
):
    ids = group["customer_id"].tolist()

    for i in range(len(ids)):
        for j in range(i + 1, len(ids)):
            pair = tuple(sorted((ids[i], ids[j])))
            match_results[pair].add("exact")

# Fuzzy duplicate pairs
for match in fuzzy_matches:
    pair = tuple(sorted((
        match["record_1"],
        match["record_2"]
    )))

    match_results[pair].add("fuzzy")

# Semantic duplicate pairs
for match in semantic_matches:
    pair = tuple(sorted((
        match["record_1"],
        match["record_2"]
    )))

    match_results[pair].add("semantic")

Now convert the combined results into a DataFrame:

combined_results = []
for pair, methods in sorted(match_results.items()):
combined_results.append({
"record_1": pair[0],
"record_2": pair[1],
"detection_methods": ", ".join(
sorted(methods)
)
})
results_df = pd.DataFrame(
combined_results,
columns=[
"record_1",
"record_2",
"detection_methods"
]
)
print(results_df)

combined_results = []

for pair, methods in sorted(match_results.items()):

    combined_results.append({
        "record_1": pair[0],
        "record_2": pair[1],
        "detection_methods": ", ".join(
            sorted(methods)
        )
    })

results_df = pd.DataFrame(
    combined_results,
    columns=[
        "record_1",
        "record_2",
        "detection_methods"
    ]
)

print(results_df)

This provides a unified view of which records were flagged and which matching methods identified them.

Classifying Potential Duplicates

We can also classify candidate pairs according to their matching evidence.

For example:

def classify_match(methods):
if "exact" in methods:
return "Exact duplicate"
if "fuzzy" in methods:
return "High-confidence candidate"
if "semantic" in methods:
return "Requires review"
return "Unique or unclassified"
results_df["classification"] = (
results_df["detection_methods"]
.apply(
lambda value: classify_match(
set(value.split(", "))
)
)
)
print(results_df)

def classify_match(methods):

    if "exact" in methods:
        return "Exact duplicate"

    if "fuzzy" in methods:
        return "High-confidence candidate"

    if "semantic" in methods:
        return "Requires review"

    return "Unique or unclassified"


results_df["classification"] = (
    results_df["detection_methods"]
    .apply(
        lambda value: classify_match(
            set(value.split(", "))
        )
    )
)

print(results_df)

These classifications are illustrative and are not calibrated probabilities.

In a production system, even exact field matches may require validation if the selected fields are not reliable unique identifiers.

A robust workflow can use three decision categories:

• Confirmed duplicate: Sufficient evidence exists under validated business rules.
• Possible duplicate: The records show similarities but require additional verification.
• Not a duplicate: The available evidence indicates that the records represent different entities.

For high-impact datasets, uncertain matches should be reviewed before any merging or deletion occurs.

Step 7: Export Duplicate Detection Results

The final step is exporting the detected duplicate pairs for further review, reporting, or downstream processing.

Pandas makes it easy to save the results as a CSV file.

results_df.to_csv(
"duplicate_detection_results.csv",
index=False
)
print("Duplicate detection results exported.")

results_df.to_csv(
    "duplicate_detection_results.csv",
    index=False
)

print("Duplicate detection results exported.")

The exported file contains:

• The identifiers of potential duplicate records.
• The methods used to identify each pair.
• The assigned duplicate classification.

For a more detailed audit report, include similarity scores, source dataset information, timestamps, and review decisions.

You may also want to export the original dataset with an indicator showing whether each record was flagged as a possible duplicate.

flagged_ids = set(
results_df["record_1"]
).union(
results_df["record_2"]
)
df["potential_duplicate"] = (
df["customer_id"].isin(flagged_ids)
)
df.to_csv(
"customers_with_duplicate_flags.csv",
index=False
)

flagged_ids = set(
    results_df["record_1"]
).union(
    results_df["record_2"]
)

df["potential_duplicate"] = (
    df["customer_id"].isin(flagged_ids)
)

df.to_csv(
    "customers_with_duplicate_flags.csv",
    index=False
)

This produces a dataset that preserves every original record while identifying those involved in potential duplicate pairs.

Exporting flagged records rather than deleting them immediately helps prevent accidental data loss and supports manual validation.

Evaluating the Duplicate Detection Pipeline

Building a duplicate detection pipeline is only the first step. To determine whether it performs well, we need to evaluate its accuracy.

The most common evaluation metrics include:

Precision

Precision measures the proportion of predicted duplicate pairs that are actually duplicates.

Recall

Recall measures the proportion of actual duplicate pairs that the system successfully identifies.

F1 Score

The F1 score balances precision and recall.

Where:

• TP represents true positive duplicate matches.
• FP represents incorrectly identified duplicate matches.
• FN represents actual duplicates that were missed.

These metrics require labelled validation data containing known duplicate and non-duplicate pairs.

A system with high precision produces relatively few incorrect matches, while a system with high recall successfully identifies a larger proportion of actual duplicates.

The appropriate balance depends on the use case. For example, incorrectly merging customer records may be more costly than leaving a few duplicates unresolved.

Optimizing the Pipeline for Large Datasets

The pipeline we’ve built is suitable for demonstrating duplicate detection on small datasets. However, the fuzzy and semantic matching stages compare every possible pair of records.

For a dataset containing (n) records, the number of unique comparisons is:

As the number of records grows, this quadratic complexity becomes expensive.

For example, a dataset containing 100,000 records requires nearly five billion unique pair comparisons.

To improve scalability, consider the following approaches:

Blocking: Group records using attributes such as country, postal code, product category, or other suitable fields before performing similarity comparisons.

Candidate generation: Use indexing techniques to identify likely matching pairs instead of comparing every record.

Approximate nearest neighbor search: Use tools such as FAISS or hnswlib to efficiently retrieve similar embeddings.

Batch processing: Process records in batches to manage memory consumption.

Incremental duplicate detection: Compare newly added records against existing records rather than repeatedly processing the entire dataset.

Embedding caching: Store previously generated embeddings and reuse them when source text has not changed.

For larger production systems, record linkage frameworks and vector search infrastructure can help make duplicate detection more efficient.

Best Practices for Building a Production-Ready Pipeline

Before deploying a duplicate detection pipeline in a production environment, consider several important practices.

1. Preserve original data. Never overwrite or delete source records without a validated resolution process.

2. Use stable record identifiers. Every record should have an identifier that allows detected matches to be traced back to the source.

3. Normalize data consistently. Apply the same preprocessing rules during initial detection and subsequent updates.

4. Combine multiple matching methods. Exact matching, fuzzy matching, and semantic similarity can complement one another.

5. Validate thresholds with labeled examples. Avoid relying on arbitrary similarity thresholds when matching accuracy is important.

6. Handle missing values carefully. Missing attributes should not automatically count as evidence of a match.

7. Review ambiguous results. Human validation is valuable for uncertain or high-impact duplicate decisions.

8. Monitor performance. Track processing time, memory usage, candidate pair counts, and detection accuracy.

9. Maintain audit trails. Record which algorithms, thresholds, and rules produced each duplicate match.

10. Separate detection from resolution. Identifying duplicates and deciding whether to merge or delete them should be distinct stages.

Conclusion

Duplicate detection in Python is an essential step in maintaining accurate, consistent, and reliable datasets. Whether you’re working with customer records, product catalogs, transaction data, or large databases, identifying duplicate records helps improve data quality, reduce inconsistencies, and support better decision-making.

Throughout this guide, we’ve explored different approaches to detecting duplicates in Python, from basic exact matching to more advanced machine learning-based techniques.

For straightforward scenarios, Pandas provides efficient methods such as duplicated() and drop_duplicates() to identify and remove exact duplicates. When records contain spelling mistakes, abbreviations, or formatting variations, RapidFuzz offers powerful fuzzy matching capabilities for detecting near duplicates.

For more complex datasets, the Python Record Linkage Toolkit enables multi-attribute comparisons and candidate pair generation, making it useful for matching records within or across multiple data sources. Meanwhile, scikit-learn and Sentence Transformers provide text similarity and semantic matching techniques for identifying potentially duplicate records that use different wording.

However, no single duplicate detection method works perfectly for every dataset. The most effective approach depends on factors such as data structure, dataset size, data quality, and the types of duplicates you need to identify.

In practice, building a reliable duplicate detection workflow often involves combining multiple techniques. A well-designed pipeline should include data preprocessing, exact matching, fuzzy or semantic similarity comparisons, appropriate matching thresholds, and validation of uncertain results.

It’s also important to remember that detecting duplicates is not the same as removing them. Records that appear similar may represent different entities, and automatically deleting or merging them can result in valuable information being lost. Preserving original data and reviewing ambiguous matches are essential for maintaining data integrity.

As datasets grow, performance and scalability become equally important. Techniques such as blocking, indexing, candidate generation, and approximate nearest neighbor search can help reduce unnecessary comparisons and make duplicate detection more efficient.

Ultimately, Python provides a flexible ecosystem for building duplicate detection solutions, from simple scripts that clean CSV files to sophisticated data quality pipelines that process large volumes of structured and unstructured information.

The key is to start with the simplest technique that meets your needs, evaluate its accuracy, and introduce more advanced matching methods as your data becomes more complex. By combining the right Python libraries with thoughtful preprocessing and validation, you can build an effective duplicate detection process that keeps your data clean, trustworthy, and ready for analysis.