This is a re-posting of a 2010 post.

If you deal with payments in any shape or form, you know you’re going to end up with a “risk management” team. A lot of times it creeps up on you: volume picks up and so you know you need someone to look at orders. If you’re running a small shop it’s most probably going to be you, but a lot of companies just hire one or two folks. These people use whatever tool you have to look at transactions – most times a customer service tool – and make up their technique as they go. With time, and sometimes with chargebacks coming in, you realize that your few analysts can’t review all transactions, so you turn to set up a few rules to make queue and transaction hold decisions. Since your analysts are not technology people you resort to hard coding some logic based on a product manager’s refinement of the analysts’ thoughts, again based on a few (or many) cases they’ve already seen. Not a long while passes, and you realize that the analysts are caught in a cat and mouse game where they try to create a rule to stop the latest attack that found its way to the chargeback report, and put a lot of strain on the engineers who maintain the rule-set. Even after coding some simple rule writing interface the situation isn’t better since the abundance of rules creates unpredictable results, especially if you allowed the rules to actually make automated decisions and place restrictions on transactions and accounts.

It’s at this point that you realize you need a statistician to run regressions, so you bring someone in. Hopefully, you have enough of a data set for them to create a decent regression model, and you can just get anyone off the proverbial street since regression is a very common tool. The statistician comes on board and creates a model that has industry standard false positives, let’s say 80%. Your review volume grows with transaction volume and you have to hire even more analysts and customer service folks to deal with complaints by legitimate customers – makes sense, since placing restrictions with 80% false positives will get you a lot of incoming calls. Then you discover that the regression model’s effectiveness degrades pretty quickly since they’re trying to predict what transaction will get a chargeback, but there are multiple reasons for getting a chargeback, making it harder to predict correctly. You then also discover that to create an updated regression model you need to wait for most of the chargebacks to come in so you’d have a good enough set of problematic transactions, meaning that you have at least 3 months’ lead time before a new process can be kicked off. That’s, of course, given that you have engineers on board to code the model.

Next thing you do is buy fraud prevention tools to add to your modeling power; you start creating black lists of IPs and Emails to mark problematic transactions. This improves things a bit but leads to additional false positives since people share resources. You consider buying a platform to manage rule and model deployment but decide that cost is prohibitive and generally, it looks like risk management is taking over your dev resources. So you decide to hire more analysts to do manual reviews, and a product manager to decide on the rule and model roadmap, and a risk operations manager for the growing group of analysts, and a head of risk. The rules and models you already have in place are blocking so many transactions you start to wonder if they’re not slowing down your growth instead of helping you protect your business. It looks like risk is managing you.

There’s something wrong with this model. Sure, some of what I’m describing makes sense for a company that’s just starting out, but getting caught in the factory approach to risk management is a huge burden in later stages, one that can be prevented by realizing that risk management is just one of a series of classification and inference questions a company needs to deal with, and that those require a different way of upfront investment in building a team.

I had two very interesting conversations about this very subject last week with two very inspiring people, but there’s one thing I remember a colleague telling me a few months back when they took a new job. The person insisted on being responsible not only for the new company’s risk management efforts, but also generally for their data and business intelligence. That was based on the same understanding: with the abundance of data created by organizations’ activity, all attempts to organize that data and make sense of it should be bound together. It doesn’t matter whether you’re qualifying leads, improving conversion or reducing fraud – you’re dealing with users and their actions, and how automated decisions impact them. It is the practice of making sense of data, and it transcends using data to control the experience of bad users. Once you realize that, you start demanding more of your analysts: that they be technical, know how to generalize on trends beyond targeted rules, become sources of truth. You understand that off-the-shelf regression cannot be just carried between domains without adjustment. You build a system that can correct itself. And with that, you create a team that can win risk, and do much more than that:  develop data sources, identify trends in huge data sets, and reach actionable insights that transform the way you work with your users, both fraudulent and legitimate.

What’s missing? I think there first needs to be a critical mass of people dealing with data in a way that sees beyond “intuition” but doesn’t get lost with over complicating inference using huge data sets. It takes time for these people to develop skill and want to continue solving difficult problems; my analysis group had less than 20 people in it and a good part of them have had enough of payments risk and classification for the rest of their career. I’m not even mentioning starting a company. But when you start a risk or data team, make sure you seed it well, or you’ll find that the bad start costs you a lot more money and effort than you have planned.