At Credit Karma, we want our members to make financial progress — and we believe that the security of our members’ personal, financial and tax information is the cornerstone of that mission. This is why we invest heavily in security measures to help protect our members from fraud, abuse and cyber crime.
The systems that underlie these security measures are often complex and require working with numerous partners, as well as combining data from a number of sources. To further these efforts and move towards a more unified infrastructure, we’ve taken a number of approaches to scale fraud protection for our business now and in the future.
Fraud protection by design
At Credit Karma, we’re firm believers in security by design. This means we build products that implement fraud and abuse measures from the get-go. Anti-fraud protections are incorporated into our product-development life cycle to achieve a layered approach to fraud protection. Below is one example of a way to think about building a layered approach to fraud protection:
Importance of metrics
Each system/layer should have clear, understandable metrics. For many types of fraud systems (especially those based on machine learning), it’s important to keep a close eye on a few key metrics: false positive, true positive, false negative and true negative. For those that are new to these concepts, here’s a quick description of what they mean in the fraud space.
False positive: The system thought something was fraud, but it wasn’t
True positive: The system thought something was fraud, and to the best of our knowledge it was
False negative: The system didn’t think something was fraud, but it was
True negative: The system didn’t think something was fraud, and to the best of our knowledge it wasn’t
One easy way to visualize these is to use a confusion matrix — but you might also want to use other visualization methods to track these metrics over time. The most important metrics for fraud are recall (true positive/(true positive + false negative)) and precision (true positive/(true positive + false positive)) of your system(s).
Recall is particularly important. Consider a fraud system that has 99% precision but recall of 5%. The system may not be performing well in this case — as fraud is usually a problem with heavy class imbalance. If you only looked at the precision, you wouldn’t get the full picture and might incorrectly assume your system is doing well. Why is that? Well, Imagine I’m inspecting a factory line for defective parts and we know 1% of parts are defective. But instead of watching the line, I just decide everything is fine and take a nap. The “system” in this case (me) was right 99% of the time, but I actually did a horrible job of detecting defective parts. These concepts can be hard to grasp at first — Google has a great resource in their machine learning crash course which can help those who are unfamiliar.
One service to rule them all
To better help protect our members, we designed a service to monitor activities on various areas of the site and detect suspected fraudulent sessions. A rule engine that is actively monitored and managed by our Trust & Safety engineers processes the fraud signals and scores from each session to determine the risk level and generate a risk profile for that session.
What happens when something goes wrong?
Even the best engineered solutions cannot be perfect, so we work diligently to mitigate the impact to our members. We try to approach every situation making sure the member has a way forward. As an example, on the Credit Karma app we might ask you to provide a copy of your ID.
Conclusion
Fraud protection by design can help you develop more-secure products from the get-go — helping to create a safer environment for your users. But even with good planning, it’s important to keep a close eye on your metrics. And when things go wrong, it’s important to have a way for the good users to make it through.
