Some time ago, we held a coding event to show proof of concept for Prophecy, our new general prediction platform. Our legacy platform was outdated, running a minimum viable product that we endlessly hacked to scale. We knew machine learning could help our users get more tailored offers, recommendations to help them build their credit, or articles that might be useful. We hoped to build a distributed and fault tolerant scoring application for generating predictions based on a user’s behavior and profile.
We called our project “Prophecy,” a tool to allow machine learning to predict… everything.
So why a 24-hour proof of concept? Organizationally, it’s a much easier pill to swallow when doing things out of the engineering norm. Asking for one day of a team’s engineering time is significantly easier than asking for weeks or months. The coding event gave us a way to reduce risk and constrain investment, but still served as a good vehicle to prove a point.
The day of the coding event went surprisingly well and to be modest, we rocked it. Everyone arrived, knew exactly what to do and how to do it. There weren’t any glaring issues or unforeseen problems. Looking back, the experience opened my eyes to three factors that helped us nail our objectives that day.
1. Measure twice, cut once
We put in significant effort getting ready for the event. To prepare, we wrote API contracts with documentation, we sketched architectural diagrams, and created an execution plan. The team ramped up on Scala and the actor-based programming paradigm, Akka.
With only 24 hours set aside there was little room for error; we wanted to be ready for a demo afterwards with real data.
When we executed the event, the planning we did saved an incredible amount of time because we front loaded the first few hours of prep. Each of our six participant engineers was assigned predefined tasks. That way we could focus more of our time on the actual code without having to worry about what had to be done and who was going to do it. By setting clearly defined API contracts we could code around them and expect everything to follow the contract. Preparation cleared almost every potential blocker and we could just code.
Amazingly, no disasters happened. We even finished early. At the end we had a fully functioning clustered Akka application to score models of any size, type or complexity. We had an admin interface to upload models and we were pulling the results of the models in the Credit Karma web app. Success!
2. Plan for an effective demo afterwards
Our goal was to prove that a different tech stack added significant value compared to our legacy MVP system. We got all the key decision makers to communicate what they wanted to see from this new system. We prepared, planned, and rehearsed to make sure that every stakeholder saw their value in our demo.
We demonstrated how a data scientist would interface with the product, and how the users would benefit by getting offers that could help them take charge of their finances. Most importantly, the demo was real, using actual models and trained with real data. Real data is often a critical buy-in factor and I always recommend you use it. We had it running about 10 times faster, and showed it working on a cluster using all of our laptops instead of production quality systems. We simulated cluster nodes breaking down and spinning up new nodes. Some models were the production equivalent, while others far exceeded the capabilities of our current legacy system. The demo proved that there was a better way. The engineers saw reliability and performance, the data scientists saw infinite possibilities of a general prediction platform and the business analysts saw business value.
The engineering leadership team was hugely impressed and supportive of the work. Our presentation evolved into a dialogue between leadership and our team discussing what needed to be done to take Prophecy into production as soon as possible. We were stoked that Credit Karma leadership had bought into our value proposition.
3. Remember that it’s not about a finished product
Coding events are great for proving that a concept works. But they’re not good for creating scalable, tested production systems. We had to scrap everything when it came time to build production systems. We didn’t want a single line of code we had rushed through to be around in the final state of our system. Roughly six weeks later, Prophecy was rewritten and deployed across Credit Karma datacenters. When we launched, it was running so fast we thought it was broken.
A well-planned proof of concept, as this showed, is a great way to inexpensively facilitate the demonstration of a value proposition, allowing a company to experiment, take risks, and build better products.
Special thanks to Aris Vlaskakis, Craig Giles and Pedro Silva. If you want to contribute to our prediction team, check out our open roles!
