Extinguishing the Garbage Fire of ML Testing

Testing should be at the heart of Data Science. It is, after all, supposed to be Science: reproducible, reliable, consistent, all that. But testing for ML is a terrible experience. Nobody does it well, and some don't even do it at all. This leaves us all mired in a mess of unreliable models, rickety pipelines, and who-knows-what data in between.
 
As dutiful software engineers, most of us ML Eng or MLOps types have attempted to apply traditional software engineering methods to this domain, resulting in one of two extremes. Either we've achieved a degree of safety through overly-rigid test setups and CI requirements, or, whether through abdication or poor design, we've forced Data Scientists to throw caution to the wind and work around us to get anything done. 
 
Our traditional testing and CI methods are not working, but we can't just give up on providing guardrails. As engineers, how do we solve ML testing?
 
We solve it by doing what we do best: abstracting, decoupling, and separating concerns. We solve it by keeping pytest only where it belongs, which is not everywhere;by substituting testing for observability in appropriate places;by applying data reliability practices and thereby solving some problems at the source. 
 
Most importantly, we solve it by having deep customer empathy for our Data Scientists. By honoring our Data Scientists, their mental models, and their ways of working, we can work together to put out the garbage fire, but there may always be a smolder in the pile.