When Production Machine Learning Fails

Recently, there has been substantial media attention placed on failures in machine learning systems. Here I will present some of the challenges that Predata has faced in building predictive products, as well as giving brief overviews of a few techniques for combatting these. While by no means an exhaustive catalogue of ML failure modes, the nature of our prediction problem and our data has led us to face challenges including but not limited to class imbalance, non-stationarity, seasonality, concept drift, and difficulty establishing good metrics and loss functions.