SuperFreakonomics Book Club: Ian Horsley Answers Your Questions About the Terrorist Algorithm

In the SuperFreakonomics Virtual Book Club, we invite readers to ask questions of some of the researchers and other characters in our book. Last week, we opened up the questioning for “Ian Horsley,” a banker who’s been working with Steve Levitt to develop an algorithm to catch terrorists. His answers are below. Thanks to Ian and to all of you for the questions.

I hope the algorithm is more sophisticated than described here. It sounds like a method for detecting observant Muslims, most of whom are of course not terrorists. – mike c.

The introduction provided in the blog only gives a basic overview of the work done. The algorithm is far more sophisticated than suggested here, with the characteristics mentioned providing some input into the overall evaluation, which actually consists of between 20 and 30 separate elements.

I really enjoyed this part of the book. Although a little creepy, I really enjoy profiling exercises. I like how human nature often betrays itself in ways we don’t typically realize.
If I remember correctly, Ian obtained his job sort of by accident. How would one go about getting a job like his? What would Ian advise I do, to get a job similar to his or helping him? – Kent

The “accidental” element to getting my job was the fact that I wanted to get involved in a particular aspect of investigating fraud but was unsure at the time as to which area of the bank undertook that role and what qualifications I would need to be considered. I wrote to a number of different people holding senior positions in the organization, and one came back suggesting their area was likely to be the one I was after. As it transpired, someone who was in that role was coming up for retirement, and, after considering my CV, they felt I might be a good candidate when the vacancy went out for application.
With fraud losses continuing to rise, many organizations (not just banks) are taking the matter far more seriously and are investing in resources (both human and automated systems) with the specific view of achieving robust fraud detection and prevention processes. If you know of an organization(s) that you’d like to work for (if not the one you’re already working for), and you believe they may have an interest in this area, then why not write to either their CEO or the Head of Security and ask similar sorts of questions to the ones I posed. The worse that can happen is that they don’t write back, but I’m sure that others will, and if they don’t currently have vacancies they will at least advise you of when they may and what qualifications/experience you might need to be considered when they do. Good luck – and let me know how you get on!

I found this to be one of the worst arguments in your book. All it selects for is devout Muslims.
Savings accounts are usury and not permitted.
Friday prayer sessions are not just for jihadis, they’re kind of a pillar of Islam.
Life assurance is a form of gambling and not permitted.
Didn’t the book also claim there was a super-secret fourth element to the algorithm that you couldn’t reveal because of national security? That part really annoyed me when I read it, since I didn’t appreciate being teased. On the strength of the other three examples I just didn’t believe that some mysterious fourth variable would work (unless it’s “are more likely than average to have bought industrial quantities of hair dye and nail varnish remover”).
It sounds about as good as analyzing for IRA terrorists by checking if they have made fewer than average low-value purchases from late-night chemists. – Camp Freddie

When we set out to look at this question, we wanted to ensure that the exercise and outcome was driven by the data; and that the data wasn’t being steered by our own thoughts and assumptions. We spent a great deal of time and consideration looking at different elements and analyzing the intelligence that was available ‘open source’ regarding incidents and suspects before we finally agreed that certain factors just simply could not be ignored. I’m sorry we can’t reveal the secret fourth element to which you refer, but the sensitivity of that element is such that we have concerns it might undermine the work that others are doing in investigating terrorists if it were made public.

Actually I may well find out if he’s being taken seriously, because as I’ve mentioned several times, I pass (or fail?) way too many of the tests you’ve mentioned here from time to time. (One assumes that in order to be any use at all, the “live” data would have to be anonymized, so the bit about Muslim names would seem to apply only to training the algorithm in the first place.)
This algorithm, in my opinion, will throw up far too many false positives to be useful. It sounds like a neural network – and I’m reminded of the (possibly apocryphal) story of the neural network the Pentagon built to identify photos containing camouflaged tanks, only to discover that, because of poorly chosen training data, what they really had was a network which could identify photos taken on cloudy days.
And by the way – you can now get a reasonably wide variety of Sharia-compliant savings accounts and life insurance here in the U.K., so those two parts of the test would appear to have become obsolete since they were first proposed…. – Ian Kemmish

We have had feedback from law enforcement that suggests our work has been taken seriously, and that any methodology that allows their focus to be targeted at the right group of individuals is worthy of further consideration.

Wow, what a waste of time for him to do this when commercial companies abound which could provide it.
I bet that data mining, target market, and demographic research companies could have this done in about a couple of weeks by cross-relating purchases to retail banking to residence locations etc. – econobiker

One of the reasons we started this work was that there were a number of companies that could have undertaken the exercise but had simply decided it wasn’t in their interests (at the time – remember, this started shortly before the London bombings). These companies had assumed that the answer to the question “Can you create a profile of a terrorist” was “No” – but they hadn’t actually tried looking. We wanted to at least attempt to answer the question one way or the other. The fact that our work has led us to this point is both pleasing and, if I’m honest, a little surprising.

Doing the Haj to Mecca in Saudi Arabia, is something all Muslims want to do before they die. Suicidal people do gesture and unique behaviors such as giving away their dear possessions, making a will and making peace with their relations. And if you were a Muslim fanatic and were going on a suicide mission, it would be a nice gesture to Allah, to complete a Haj, just as Westerner would make sure all the insurance and paperwork is in order and the suicide note has been penned.
I wonder how many of the the 21 suicide bombers on 9/11 went on Haj in the months or year prior to the attack?
I would suggest keeping a Haj list of young Muslims, or at least passport control to Saudi Arabia, who may be prone to seek paradise prematurely on a mission to send some infidels to hell. – Drill-Baby-Drill Drill Team

Unfortunately, the data suggested here isn’t available in the line of work I do. That said, I’m sure those people that are reviewing our working methodology and results could give serious consideration to such suggestions should they decide to adopt a similar methodology going forward.

Well, did you ever get any feedback? Does your “secret sauce” work? – Robert G.

As suggested earlier, there has been serious interest in our work. While we haven’t received (or sought to gain) feedback on the specific individuals we highlighted, we are comfortable that our work has been worthy of the amount of time and effort invested in it.

Dismal science if ever there was.
If these people get arrested and tried then we hear about it. They do not have to actually blow anything up.
False positives? Any mass screening program like for prostate or breast cancer throws up so many false positives you have to be careful how you apply them. Similarly the security theater at airports.
So what is the false positive rate of this little fable then?
Has the technique been applied to an independent data set which wasn’t used for the curve fitting and for which we have a good idea of the results? E.g., you could deliberately withhold the data of 2 known terrorists in the initial analysis and then see how they are subsequently classified.
In short is there any actual evidence for the effectiveness of this egregious piece of armchair anti-terrorism? – John Ellis

We have applied our technique to independent data sets, and the findings stand up to scrutiny. We have done all we can to ensure that the false positive ratio is kept to an absolute minimum, and we continue to assess our workings as we gather further intelligence regarding this subject.

Can you get much unique information about people from bank records, which isn’t already present in other data that law enforcement have access to? For instance, suppose that the British police had a long list of potential terrorism suspects, created from other data sources, and already knew that they were all observant Muslims. Would the bank data help to narrow this list down to a much smaller list of suspects who were especially likely to be terrorists, or would it mostly just be redundant with what the police already knew? – Dan

Banks will work as closely as they possibly can with law enforcement, but they are bound by a whole raft of legislation as to the circumstances and processes under which data can be used or provided. There is much information available within the bank’s records that might assist law enforcement and that isn’t available “in the public domain.” If law enforcement should choose to approach us with a list of suspects, then we would be duty-bound to assist in narrowing down the list as best as we could, but at the same time ensuring we acted in a lawful manner while doing so.

Most comments miss the point.
The disclosed parameters are obviously for illustration purposes about how the technique works. By themselves, they probably “single out” about half a million people. It’s ridiculous to reach conclusions about the usefulness of this technique just based on them.
However, if the final “secret ingredient” really narrowed the list down to 30 people out of the 60 million or so in the U.K., it would be equally ridiculous to not make a check on them.
Of course there will be false positives and false negatives, but so what? It is not that these 30 people will be arrested or even contacted. The authorities only need to take a closer look at their activities, for instance see if they are in contact with other suspects. But since the probability of catching at least one terrorist among them is high, not checking on them would be negligent.
To the ones that disagree, please imagine how would you feel if your kid was killed by a terrorist that happened to be on that list. –Alex

Whilst not directly a comment requiring an answer from me, I am grateful for Alex’s feedback as his sentiments are very similar to my own. If our work allows even one terrorist to be brought to the authorities’ attention in a more timely manner, and in doing so prevents them from carrying out their intended actions, then it will have been worth the time and effort we have invested.
Can I just thank everyone for their questions, comments, feedback and ideas – they are very much appreciated.
The feedback provided above are a reflection of my own thoughts and opinions and don’t necessarily reflect (or should be attributed to) those of the organization by whom I am employed.