Explaining using a real-world scenario
One of the key points that data scientists should ask end-users is: How much importance do they give to the accuracy of the prediction? For instance, when I work from home, I am not much worried about the afternoon snowfall prediction. Whereas when I go to work, that is very important to me.
A few years back installed my first doorbell camera, I was very excited to seeing motion alerts on my phone. Gradually the alerts started to annoy me.
All I was interested was alert about human movements near my house. But I got alerts for even leaves swaying in the wind. Then I explored the camera motion settings and lowered the sensitivity. Though it stopped a lot of unnecessary alerts, it also blocked many real alerts from coming in especially after dusk or at times with low visibility.
Then I decided on changing the setting twice a day, both morning and evening. The following images show how my doorbell camera responded to movements during daytime and night.
Image by author.
Image by author.
That gave me the level of alerts I wanted, but it became an hassle changing the settings twice a day. So finally, I settled with one permanent setting in the middle which was neither too good or bad. I now get more alerts than I want during day and miss some during night. The following image show how my doorbell camera responded to the current settings.
Image by author.
What I really wanted with my current setting was a good level of alert accuracy in the morning. I wanted the Doorbell to alert me only the human movements like delivery drivers coming in, postman etc. Whereas at night, I was willing to tolerate a low alert accuracy level to avoid missing alerts.
Analyzing the alert counts
In order to reach this middle setting, I actually had quantified the accuracy of the settings in my mind, even without realizing it. What I really did was adjusting the trade-off between Precision and Recall.
Precision measured how successful I was in receiving human movement alerts for a setting, without receiving any non-human movements falsely as human movement alerts. Any non-human movements which were falsely alerted as human movement (False positive count) penalized the precision.
Mathematically:
Precision = True Positive count / (True positive count + False positive count)
Now take a look at the alerts from my old morning setting.
Based on the above formula for precision:
Precision = True Positive count / (True positive count + False positive count)
= 2 / (2 + 0)
= 2 / 2
= 1.0 or 100%!!!!
This implied that I had a high accuracy, but in reality the Doorbell missed alerting 3 human movements.
That is where Recall comes into picture. It measures the true positive rate, which is way to determine if all the 5 human movements were alerted. Any missed human movement (False negative count) alerts were penalized.
Mathematically:
Recall = True Positive count / (True positive count + False negative count)
= 2 / (2 + 3)
= 2 / 5
= 0.4 or 40%
Now, let's look at the evening setting.
Precision = True Positive count / (True positive count + False positive count)
= 5 / (5 + 2)
= 5 / 7
= 0.71 or 71%
Recall = True Positive count / (True positive count + False negative count)
= 5 / (5 + 0)
= 5 / 5
= 1.0 or 100% !!!
Here, since all human-movements were alerted, Recall became 100%. But the Precision calculation was penalized for alerting non-human movements as human.
Finally, let's look at the current setting.
Precision = True Positive count / (True positive count + False positive count)
= 4 / (4 + 1)
= 4 / 5
= 0.8 or 80%
Recall = True Positive count / (True positive count + False negative count)
= 4 / (4 + 1)
= 4 / 5
= 0.8 or 80%
Here, I didn't even have a vague idea that the Precision and Recall will mathematically balance when I applied the final setting! But it happened.
Analyzing the alert counts
A quick way to look at the counts is using a confusion matrix. Here the Precision and Recall are also shown along with the confusion matrix. As you would notice, when Precision goes up Recall goes down, and vice-versa. Your desired value is based on what problem you are trying to solve, as explained with the morning, evening and current setting.
