Automatically Fix Data Issues & Label Errors in Most ML Datasets

Gold-standard ML test sets contain significantly more label errors than previously believed (see labelerrors.com). Using confident learning (open-sourced as cleanlab), we found that at least 3.3% of labels in the ten most commonly used benchmark test sets are erroneous (over 10% in some cases). Real-world datasets used to train public-facing models are often less curated than these academic benchmarks, making both benchmarking (errors in test sets) and training reliable models (errors in train sets) difficult.

In this talk, I'll present (1) cleanlab open-source (github.com/cleanlab/cleanlab), a fast-growing python framework for data-centric AI that automatically finds issues in most ML datasets -- for most models, data modalities, and data formats -- often in only a few lines of code and (2) Cleanlab Studio (https://cleanlab.ai/studio), a no-code, automatic web interface used by many universities and fortune 500 companies for finding and fixing issues in ML datasets. Notably, cleanlab algorithms are theoretically proven to work well in some cases. While the proof of exact label error finding is out of scope for this talk, I'll share intuitive theoretical arguments that demonstrate why cleanlab tends to train more accurate models on real-world, messy data than prior approaches.