As Data Science becomes a more well-known solution to many problems across industries, individuals want to be certain they are getting the best return on investment (ROI). One of the most difficult things is to be sure the comparison between models/algorithms/machine learning is apples-to-apples instead of apples-to-oranges, which is more often the case than not. Things like the number of false positives that a model may throw off, or the underlying prevalence of a particular target variable in the dataset you have before you, play an important role in your desired business outcome.
“Accuracy is the only important measure.”
Unfortunately, many professionals simply do not understand the impact of this statement. Accuracy is not the only important measure; in fact, it may not even be the best measure for the business problem being solved. It is the uninformed professional that does not explore the performance behind a model.
Imagine a scenario where a machine learning model is developed around a particular health condition for patients entering an emergency department. In our example, let us imagine the model is already deployed in a hospital system and has produced results.
Enter the Accuracy Parradox
Notice Model_1 has the highest accuracy of 98.50%, but Model_1 did not predict a single patient that actually had the condition (Positive Cases, Predicted Positive).
Notice Model_2 has the lowest accuracy of 98.00%, but it did predict 150 patients that actually had the condition (Positive Cases, Predicted Positive).
The point is that it is simply not enough to know your model’s accuracy and compare it to another model’s accuracy. In the above example of the accuracy paradox for patient conditions, there are cases when it makes business sense to choose a less accurate model because it yields more predictive power or better captures the underlying causal factors describing the outcome variable your business is interested in – especially in the case of rare events.
Ask Questions: Lots of Questions
My favorite questions are “How?” and “Why?”; they have served me well in Data Science. I talk with my clients in these terms because I understand they need the answers to these questions – even if it does not seem so at the time of our first conversation. If you have a modeling, statistics, or Data Science solution that your company has deployed, ask questions – lots of questions. Remember: while it is nice and neat to have a single number represent everything, such as accuracy, F1 score, Matthew’s coefficient, etc., it is a broad attempt to categorize the algorithm and that may not meet your needs.
Important questions to ask about the accuracy:
- True positive rate (eqv. with hit)
- True negative rate (eqv. with correct rejection)
- False positive rate (eqv. with false alarm)
- False discovery rate (eqv. with miss)
Keep everything in a business context and formulate your questions around the underpinnings of that number. Appropriately mapping what your business cares about to the evaluation metric of a Data Science problem is an important step in determining true ROI.
Getting an investment return from Data Science can seem complex, but it does not have to be. Often, in an effort to satisfy many prospective clients, vendors will simply tell the client what they want to hear – “You have purchased the most accurate model available in the current market.” Understanding the underpinnings of that accuracy should be important to any discriminating professional making use of Data Science. Additionally, knowing the realities of the accuracy paradox and getting comfortable asking questions are key to answering what kind of return you can expect. Alan Alda is quoted as saying “I found I wasn’t asking good enough questions because I assumed I knew something.” Use his insight to your advantage. That is the path to better business and better Data Science.