Data Scientists and ML Engineers invest significant effort in building and productionizing AI/ML solutions. However, in spite of spending so much time and effort on the development process, there was scarcely any system to regularly analyze the model output in production.
This notion is gradually changing as development teams realize that the model performance on training or testing data is usually different from performance on production data due to factors such as drift and poor quality that creep in during the development time.
With post-production analysis, it is possible to:
- Catch disruptive patterns early on without any impact on end-user
- Understand why certain patterns are changing over time
- Correlate with XAI reasoning and narrow down the search for root cause
- Understand early on if a model retraining has to be scheduled
- Explain disruptions or upcoming disruptions to internal stakeholders or even customers
- Prepare the team for downtime planning and delegation