Clean Me If You Can: A Large Collection of Real-World Addresses for Data Cleaning Benchmarking
Source: arXiv:2606.31983 · Published 2026-06-30 · By Fatemeh Ahmadi, Tobias Bernhard, Mohamed Abdelmaksoud, Luca Zecchini, Tilmann Rabl, Ziawasch Abedjan
TL;DR
This paper addresses a critical gap in data cleaning research: the lack of large, realistic benchmark datasets with ground truth. The authors present Clean Me If You Can, a very large real-world dataset containing over 9 million postal address records with naturally occurring errors paired with verified ground truth obtained through an extensive process involving geocoding services (notably Nominatim). This dataset spans 128 countries and captures complex real-world error distributions stemming from extraction faults, formatting inconsistencies, and outdated or missing fields.
The authors thoroughly document their scalable, fully reproducible pipeline for extracting dirty addresses from Web Data Commons dumps of schema.org and microformat annotations, filtering for spatially verifiable records, querying geocoding services, and aligning and validating ground truth addresses. Benchmarking of state-of-the-art data cleaning methods on this dataset reveals that existing techniques perform poorly in this challenging setting, with none reliably cleaning the dataset. The study motivates new research directions to handle diverse error types, leverage external knowledge, and scale across both large data volumes and heterogeneous error characteristics.
Key findings
- The dataset contains 9,317,886 dirty postal address records with corresponding ground truth (full-unnamed), and a 4,338,109 record subset including entity names (full-named).
- Addresses cover 128 countries worldwide, with geographically varied address formats and error distributions.
- About 48.9% of attribute cells in the full-named dataset contain errors compared to ground truth, showing high real-world data noise.
- Existing state-of-the-art error detection and correction methods cannot reliably clean the dataset or its subsets, demonstrating limited effectiveness and scalability.
- Nominatim geocoding service successfully matched 96.5% of a stratified sample of dirty addresses to ground truth, with 99.1% agreement versus Google Address Validation API.
- The dataset preserves provenance by tracking source URLs and includes 14 functional dependencies involving key address attributes useful for rule-based cleaning.
- Country-based and POI-based slices (e.g., US slice with 2,267,895 records, Germany slice with 374,533) are available for focused benchmarking.
- The dataset contains diverse error types: missing values, inconsistent formatting, contradictory data, misspellings, and nonsense entries, reflecting complex real-world error phenomena.
Threat model
n/a - This work focuses on dataset creation and benchmarking for data cleaning methods, rather than adversarial defense or bot detection scenarios.
Methodology — deep read
The authors constructed their benchmark dataset via a multi-step, reproducible pipeline designed to capture realistic, large-scale dirty postal address data with ground truth:
Threat model & assumptions: The work is a benchmarking dataset for general-purpose data cleaning; adversaries or attacks are not explicitly modeled.
Data provenance: Dirty address records were extracted from the Web Data Commons (WDC) December 2024 release, comprising RDF N-Quads from Common Crawl. Two sources were used: 22 class-specific schema.org subsets containing PostalAddress annotations and an html-mf-adr microformats subset using vCard addresses. This yielded initially 247.9M and 11.8M extracted dirty address records respectively.
Data preprocessing & filtering: Records missing key address components (street, locality, or postal code) were discarded, retaining 219.9M schema.org and 8.5M html-mf-adr dirty records. Records were formatted uniformly by mapping schema predicates to a common schema.
Ground truth generation: Each dirty address was queried against several commercial and open geocoding services. After a stratified 1k sample evaluation, the authors selected the Nominatim OSM-based service for best coverage, accuracy, and scalability, deploying a self-hosted instance to handle the 228.4M queries.
Schema alignment: Since address components differ between extracted dirty records and geocoding responses, they performed attribute alignment. For example, PostalAddress's "streetAddress" (combined string) was parsed with libpostal into separate "road" and "house_number" fields matching OSM structure.
Filtering & validation: About 80% of queries yielded no Nominatim match and were discarded. Records with incomplete ground truth attribute values (except missing "name") were removed. Two dataset versions were created: full-unnamed (9.3M records, no "name") and full-named (4.3M records with verified entity names).
Additional quality improvements: Duplicates with identical dirty and ground truth attributes from the same source page were removed. Locality attributes in ground truth were normalized using certified locality lists for consistent granularity.
Dataset statistics & subsets: Detailed cardinality analysis, functional dependency analysis, and creation of country- (US, Germany) and POI-based subsets support fine-grained research.
Evaluation: The authors benchmarked existing error detection and correction methods, combining rule-based, learning-based and LLM-driven approaches, on the dataset. Metrics included precision and recall of error detection, and accuracy of corrected values. Results indicate substantial gaps in current methods’ ability to handle noisy real-world data at scale. The paper also discusses computational scalability limitations.
Reproducibility: All datasets, extraction code, and benchmarking scripts are publicly released at https://github.com/D2IP-TUB/Clean-Me-If-You-Can. The full data collection and ground truth alignment pipeline is fully documented for replication.
Example pipeline end-to-end: One dirty record consisting of a schema.org PostalAddress annotation (e.g., streetAddress='Hauptstraße 17') is extracted from WDC N-Quads. It is filtered if containing minimal info (street or locality or postal code). Then it is queried against the self-hosted Nominatim instance using structured address parameters. The top returned match is parsed and aligned into the benchmark schema. Entries without matches or incomplete fields are filtered out. The final pair (dirty and ground truth) is stored with provenance for benchmarking.
Technical innovations
- A fully reproducible pipeline leveraging large-scale web data extraction and geocoding services to generate a realistic, large-scale, and geographically diverse postal address cleaning benchmark with ground truth.
- Schema alignment methodology combining address parsing (libpostal) and mapping between schema.org PostalAddress and OpenStreetMap address representations.
- Empirical validation and selection of geocoding services (Nominatim vs commercial alternatives) for scalable ground truth generation on hundreds of millions of records.
- Creation of country- and POI-specific dataset slices to enable nuanced benchmarking across geographical and domain-specific contexts.
- Extensive error characterization and real-world distribution analysis demonstrating that existing cleaning methods fail under natural error distributions.
Datasets
- full-unnamed address dataset — 9,317,886 records — derived from WDC and Nominatim geocoding
- full-named address dataset — 4,338,109 records — subset with verified entity names
- us-slice — 2,267,895 records — country-specific subset of full-named
- de-slice — 374,533 records — country-specific subset of full-named
- library POI slice — sized unspecified — POI-specific subset from schema.org classes
- shopping-center POI slice — sized unspecified — POI-specific subset from schema.org classes
Baselines vs proposed
- Rule-based error detection: precision and recall significantly below 0.7 on full-named dataset versus ground truth, failing to scale to large error types
- Learning-based methods: improved detection rates but exhibit strong scalability problems and inconsistent correction accuracy
- LLM-driven cleaning approaches: evaluated but demonstrate mixed effectiveness due to domain-specific issues and heterogeneous error patterns
- Overall, no evaluated method reliably cleans even slices of the dataset, highlighting large room for improvement
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.31983.
Fig 2: Country distribution in the full-named dataset.
Fig 3: Percentages of clean and erroneous cells (grouped
Limitations
- Approximately 80% of initial dirty records could not be matched to ground truth and were discarded, potentially biasing the dataset towards addresses well-covered by the geocoding service.
- Ground truth relies on geocoding service outputs, which may contain their own inaccuracies or incompleteness, especially in less covered countries.
- Dataset represents company and public entity addresses only, excluding individual personal addresses, limiting applicability to some domains.
- The benchmark focuses mainly on postal addresses, so error types and distributions may differ from other tabular data domains.
- Evaluation primarily focuses on effectiveness metrics; runtime and resource scalability evaluation detail is limited.
- Although the dataset is large and diverse, representation remains uneven across countries and languages.
Open questions / follow-ons
- How to design data cleaning methods that scale robustly across highly heterogeneous, geographically diverse error distributions as observed in this dataset?
- How to effectively combine rule-based, learning-based, and LLM-driven techniques to leverage their complementary strengths on complex real-world tabular data cleaning?
- Can external knowledge bases or world knowledge beyond input data improve cleaning accuracy at scale in this postal address domain?
- How to efficiently handle the high dimensionality of functional dependencies and inter-attribute constraints for automated error correction?
Why it matters for bot defense
While this paper does not address CAPTCHAs or bot detection directly, it is highly relevant to practitioners in bot defense who rely on clean large datasets for model training and evaluation. Address data is commonly used in user onboarding, fraud detection, and identity verification pipelines where data quality impacts downstream performance. The dataset and findings highlight the challenges of cleaning real-world, noisy tabular data extracted from crowdsourced or user-generated sources at scale. It underscores the need for robust, scalable cleaning methods that can handle diverse error types and distributions—prerequisites for building trustworthy, automated bot-defense systems that minimize false positives and negatives related to profile data. Furthermore, the authors’ rigorous methodology for obtaining large-scale ground truth at minimal manual cost may inspire similar efforts in constructing datasets for security-related model training and verification.
Cite
@article{arxiv2606_31983,
title={ Clean Me If You Can: A Large Collection of Real-World Addresses for Data Cleaning Benchmarking },
author={ Fatemeh Ahmadi and Tobias Bernhard and Mohamed Abdelmaksoud and Luca Zecchini and Tilmann Rabl and Ziawasch Abedjan },
journal={arXiv preprint arXiv:2606.31983},
year={ 2026 },
url={https://arxiv.org/abs/2606.31983}
}