Duplicate Titles: Detect and Group Duplicate or Near-Duplicate Content Titles

Question

## Problem

You have a list of content titles (article headlines, video titles, etc.). Some are exact duplicates; some are near-duplicates (minor wording differences). Implement:

1. `find_exact_duplicates(titles)` — return groups of titles that are exactly the same (case-insensitive).
2. `find_near_duplicates(titles, threshold)` — return groups of titles whose normalized edit distance is <= `threshold`. Normalize: lowercase, strip punctuation, collapse whitespace.

```python
def find_exact_duplicates(titles: list[str]) -> list[list[str]]: ...
def find_near_duplicates(titles: list[str], threshold: float) -> list[list[str]]: ...
```

```
titles = [
  "Python Tips for Beginners",
  "python tips for beginners",   # exact dup (case)
  "Python Tips For Beginners!",  # near dup (punct)
  "Java Tips for Beginners"      # different
]

find_exact_duplicates(titles) ->
  [["Python Tips for Beginners", "python tips for beginners"]]

find_near_duplicates(titles, 0.1) ->
  [["Python Tips for Beginners", "python tips for beginners", "Python Tips For Beginners!"]]
```

## Follow-ups

1. `find_near_duplicates` is O(n^2) with pairwise comparisons. For 100,000 titles, how do you scale this using MinHash or locality-sensitive hashing?
2. What normalization steps matter most for your use case (news titles vs. product names vs. code repo names)?
3. How do you choose the right `threshold`? Describe a labeling + precision/recall evaluation approach.
4. Titles keep arriving as a stream. How do you maintain near-duplicate groups incrementally without reprocessing the full corpus?

Duplicate Titles: Detect and Group Duplicate or Near-Duplicate Content Titles

Question Details

Full Details

Topics

More from Netflix