Free preview lesson · Model Evaluation & Testing · Module 1: What good looks like
Defining success for a model
A model is only good or bad relative to a purpose. Before choosing any metric, you need a clear definition of success for the people who will use the output.
Describe the decision
Write down what decision or action the model's output will drive. Flagging a transaction for review, drafting a reply or ranking search results each have different consequences when they go wrong.
Weigh the errors
Most tasks have two kinds of error: false positives and false negatives. Missing a fraudulent transaction and annoying a genuine customer have very different costs. Agree with stakeholders which matters more before looking at any numbers.
Set a threshold in advance
Decide what level of performance is good enough to ship, and what would make you stop. Writing this down before evaluation prevents moving the goalposts once results come in.
Common pitfall
Reporting a single accuracy figure. On imbalanced data a model can be 99% accurate and still miss nearly every case that matters.
Action step
For one model or AI tool you use, write a one-paragraph definition of success, including which error is worse and why.
This is one lesson from Model Evaluation & Testing.
Educational content only. No income, revenue or earnings results are promised or implied. No certification, licence or professional credential is awarded.