Representation-Based Data Quality Audits
The representation as its own auditor

A representation is more than a feature vector

Modern deep learning operates on learned representations. The geometry of those representations, the distances between points and the composition of their neighbourhoods, emerges from the data rather than being hand-designed. Whenever a model judges similarity, flags an outlier, or links an example to another, it is reading from this geometry.

A representation is not just a feature vector per sample. It is a structured record of how the model organised everything it has seen during training, and it carries information about the data that the labels alone do not reveal. A sample isolated in the embedding space is often an unrelated or off-topic image. Two samples lying very close to each other are often near-identical copies of the same image. A sample whose neighbourhood is dominated by other labels has likely been given the wrong label itself. Each of these issues leaves a different signature in the representation space, and the geometry is where they become legible.

Despite the recent rise of data-centric machine learning, most of the effort has gone into improving detection methods for individual issue types in isolation. The geometry of the representation itself, and what it already reveals about how these issues are organised in embedding space, has received much less attention. This work pursues that idea directly and asks how far a single representation can carry such an audit on its own.

The setting in which this matters most is medical imaging, where datasets are orders of magnitude smaller than the corpora that train foundation models, and where small contamination can change the conclusion of an experiment. Medical datasets are commonly assumed to be gold-standard because experts produce the labels, yet each entry passes through many manual steps (case selection, lesion cropping, diagnosis assignment, contributor screening, anonymisation, and aggregation), and pipelines like this are well known to introduce label errors and silent leakage across train and test splits.

SelfClean: three signatures in one space

SelfClean (NeurIPS 2024) starts from a dataset-specific self-supervised representation. We train the SSL encoder on the very dataset we want to clean, with no labels, no clean reference, and no external supervision. The encoder's pretext task is unrelated to the downstream task we eventually evaluate on, yet the latent space it produces respects the structure of the data: similar items end up close together, dissimilar items end up far apart. That same space is what our three distance-based indicators are computed from.

Given the encoder $f$ trained on the noisy dataset $\mathcal{D} = \{x_1, \ldots, x_N\}$ with embeddings $\vec{e}_i = f(x_i)$, we compute the pairwise distance matrix $\text{dist}(\vec{e}_i, \vec{e}_j)$ once. Each issue type then reduces to one score on those distances and a ranking on the result.

Off-topic samples sit in sparse isolated regions

An off-topic sample is far from the bulk of the data. We run agglomerative clustering with single linkage on the distances and rank by the order in which each sample is absorbed into a larger cluster: the later a cluster is merged with a larger one, the more it is treated as an outlier. Single linkage is sensitive to outliers, which is desirable here. Samples that remain isolated longest in the dendrogram score lowest on the off-topic ranking $s_{\text{OT}}$.

Near-duplicates collapse to anomalously tight clusters

A pair of near-duplicates has, by construction, a very small distance in latent space. We score each ordered pair $(i, j),\, i < j$ by

and rank ascending. The top of the list is what a human reviewer inspects first. Considering each pair as a candidate requires $N(N-1)/2$ distances.

Label errors disagree with their neighbourhood

A correctly labelled sample is, on average, closer to others in its class than to samples in other classes. For each anchor $\vec{e}_i$ with label $l_i$, define the nearest same-class and nearest different-class distances

and combine them into a single bounded score

A correctly labelled sample has $m_{=}$ small and $m_{\neq}$ large, so $s_{\text{LE}} \to 1$. A mislabelled sample has $m_{=}$ large and $m_{\neq}$ small, so $s_{\text{LE}} \to 0$. Sorting ascending puts label errors at the top of the ranking. The same neighbourhood relation a $k$-NN classifier would use to draw a class boundary is what the ratio measures, so failures of the ratio are exactly the failures the classifier will pay for at evaluation.

From a ranking to verified contamination

The output of SelfClean is a sorted list per issue type. What turns that list into verified data quality issues is a human reviewer working from the top. The companion paper "Towards Reliable Dermatology Evaluation Benchmarks" (ML4H 2023) introduces the annotation protocol we use throughout: a small panel of domain experts independently inspects the ranking, dataset-by-dataset and issue-by-issue, with three components that matter in practice.

Independent multi-expert verification. Each candidate is annotated by several experts in parallel rather than a single annotator. Confirmation requires agreement, which removes the subjectivity that a single rater would carry into the label.

A statistical stopping criterion. Annotation continues only as long as the rate of confirmed issues among the candidates exceeds the rate expected by chance under a two-Bernoulli test. Once the ranking starts surfacing items at chance level, inspection stops. This keeps the budget bounded without committing to a fixed top-$K$ in advance.

Aggregation by majority or unanimity. The final verdict per candidate is the majority (or stricter, unanimous) vote across experts. Reporting both lets readers see how much of the contamination is subjective and how much is unambiguous.

Once a ranking is verified by experts, the dataset gets its first quantitative quality assessment. The same procedure applied across nine dermatology and general-vision benchmarks produces the per-dataset contamination estimates that the rest of this page builds on.

The procedure's output is a ranking. Pick a dataset and an issue type to see the top-15 items SelfClean flagged from a real audit. The same encoder produces all three rankings, no extra training between issue types.

ImageNet-1k CheXpert Fitzpatrick17k PatchCamelyon

Off-topic Duplicates Label errors

ImageNet-1k · off-topic samples. Top-15 inputs ranked highest by the agglomerative-clustering isolation score. These are candidates for inputs that do not belong to the dataset distribution.

The same recipe on audio

The three signatures (sparse isolated regions, tight clusters, neighbourhood-label disagreement) are statements about the geometry of an embedding space, not about pixels. This means the general recipe is modality-agnostic, and the same indicator functions should transfer to any modality where a self-supervised encoder produces a similarity-respecting latent space. The follow-up paper "Representation-Based Data Quality Audits for Audio" (ICASSP 2026) ports SelfClean to audio and confirms exactly that.

The representation needs care, the indicators do not. Training a dataset-specific encoder from scratch on small audio benchmarks (the image-domain recipe) underperforms. Modern pre-trained audio encoders (CLMR, CAV-MAE, EAT, BEATs, M2D) trained on large general-audio corpora generalise well across contamination types, and optional dataset adaptation via LoRA recovers SelfClean's original "intrinsic representation" idea without retraining from scratch. The three indicator functions ($s_{\text{OT}}$, $s_{\text{ND}}$, $s_{\text{LE}}$) integrate without modification.

File-level aggregation for production. Audio inputs have temporal structure: an off-topic segment in an otherwise-on-topic recording is different from an entirely off-topic file. The audio paper introduces a file-level aggregation step that combines per-segment scores into a single ranking entry per file, which is what an industrial review workflow consumes. Benchmarks on ESC-50, GTZAN, and a proprietary industrial dataset confirm the qualitative pattern appears here too: SelfClean either matches or outperforms issue-specific baselines, and the FoE reduction at the top of the ranking translates directly into operator time saved.

Estimated contamination across image benchmarks

The first concrete output of the audit, applied to widely used benchmarks, is a quantitative answer to "how much contamination is in this dataset". The numbers below come from SelfClean's fully-automatic mode (NeurIPS 2024, $\alpha = 0.10$, $q = 0.05$). Highly curated medical sets sit close to or under 1% noise. Less curated collections, including web-crawled ones, carry one to two orders of magnitude more in our table.

The audit changes which model is "best"

A tool is only as valuable as the problems it reveals are consequential. Removing the flagged items from a benchmark changes the score the benchmark reports, and on widely used dermatology evaluations it changes which of two state-of-the-art models the field considers best.

Cleaning the eval set shifts the headline score

On the $k$-NN classifier built on dataset-specific SSL features, cleaning the evaluation split alters F1 in a way that depends on what was being measured (NeurIPS 2024):

Two SOTA dermatology models, re-ranked after cleaning

The "Reliable Benchmarks" paper (ML4H 2023) re-evaluates two open-source binary skin-cancer models, DeepDerm (Esteva et al., Nature 2017) and HAM10000-CNN (Tschandl et al., Scientific Data 2018), on the same dermatology benchmarks before and after the cleaning procedure. The point of the experiment is not to crown one model, but to show that the audit re-orders them on the datasets where contamination is concentrated.

How do you evaluate an auditor?

Rigorous comparison of data-cleaning methods has not been possible until now. Each paper introduces its own ad-hoc evaluation setup, and in their absence the field has had to rely on synthetic injection. Synthetic injection is useful, but it does not test how methods behave on the real distribution they will be deployed against, and there has been no agreed-upon test set on which to compare two approaches.

To close that gap, we built CleanPatrick (DMLR 2026), the first standardised benchmark for image data cleaning. It comprises 496,377 expert-verified annotations across off-topic samples, near-duplicates, and label errors on a real-world dermatology dataset. Methods are evaluated on the same items, on the kind of contamination that real curation pipelines actually produce, with fixed protocols and metrics so that two cleaning methods can be compared head-to-head. The benchmark confirms what the SelfClean experiments suggested: methods that operate in embedding space rank among the strongest on every issue type.

Annotating duplicates in linear, not quadratic, time

Building a near-duplicate ground truth is the part of CleanPatrick that looks easy and is not. A dataset of $N$ images has $\binom{N}{2} = \mathcal{O}(N^2)$ candidate pairs to verify. On Fitzpatrick17k that is roughly $145{,}000{,}000$ judgments, which no annotation budget covers. So we built an algorithm around a weaker, more local assumption that does cover it.

Under the fast-cleaning assumption (a near-duplicate of any sample is closer to it in latent space than any non-duplicate), the duplicate graph can be recovered exactly with a Borůvka-style procedure: find each sample's nearest neighbour, annotate those pairs (at most $N$ annotations per round), merge clusters, then iterate on the merged clusters. The algorithm discovers all duplicate cliques in at most $\lfloor\log_{2} K\rfloor + 1$ rounds (where $K$ is the largest clique size) and requires no more than $2N$ pairwise judgments in total.

# D: dataset of N samples; dist(i, j): pairwise distance in latent space
# is_duplicate(i, j): human annotator's binary verdict on the pair

clusters = [{i} for i in range(N)]   # one singleton cluster per sample

while True:
    # 1. For each cluster, find its nearest neighbouring cluster
    pairs = []
    for c in clusters:
        nearest = argmin(other for other in clusters if other != c,
                         key=lambda o: min_distance(c, o))
        pairs.append((c, nearest))

    # 2. Annotate those candidate pairs (at most N annotations this round)
    confirmed = [(c1, c2) for c1, c2 in pairs if is_duplicate(c1, c2)]
    if not confirmed:
        break

    # 3. Merge the clusters joined by confirmed pairs
    clusters = union_find_merge(clusters, confirmed)

# Output: the final list of clusters is the set of duplicate components.

Using the audit in your own work

If you train or evaluate on an image or audio dataset and have a self-supervised encoder available (or want one trained on your data), the audit is meant to drop into existing pipelines. The Python package is on PyPI:

pip install selfclean

The recipe is a single call against your dataset:

from selfclean import SelfClean

auditor = SelfClean()
report  = auditor.run(dataset=my_dataset, labels=my_labels)

# report.off_topic     -> ranked list of (index, score)
# report.duplicates    -> ranked list of (i, j, distance) pairs
# report.label_errors  -> ranked list of (index, intra/extra ratio)

The notes below cover the choices that matter in practice.