“Using Metadata to Find Paul Revere” Kieran Healy

"I have been asked by my superiors to give a brief demonstration of the surprising effectiveness of even the simplest techniques of the new-fangled Social Networke Analysis in the pursuit of those who would seek to undermine the liberty enjoyed by His Majesty’s subjects. This is in connection with the discussion of the role of “metadata” in certain recent events and the assurances of various respectable parties that the government was merely “sifting through this so-called metadata” and that the “information acquired does not include the content of any communications”. I will show how we can use this “metadata” to find key persons involved in terrorist groups operating within the Colonies at the present time. I shall also endeavour to show how these methods work in what might be called a relational manner."  Kieran Healy 


This entry was posted in government, Intelligence. Bookmark the permalink.

27 Responses to “Using Metadata to Find Paul Revere” Kieran Healy

  1. Walrus says:

    What a brilliant example. thank you for posting it.

  2. The Twisted Genius says:

    Kieran, so you did this! Brilliant! I read it last week when Jacob Applebaum linked to it. Your page count must be going through the roof. I’ve seen a lot of people linking to it since then. Your amusing story illustrates what the Feds can do with metadata and link analysis… the pathetically suspicious rat bastards.

  3. turcopolier says:

    A different Kieran. pl

  4. The Twisted Genius says:

    Ah, thats right. Ours is Kieran Wanduragala. Still a brilliant piece.

  5. mbrenner says:

    These capabilities are always exaggerated for reasons that earlier posts explain in detail. The glaring discrepancies between what theoretically is possible and what we actually accomplish should give us pause. Don’t we have enough examples of failure re OBL, the Taliban leader cum grocer who fooled Petraeus, Benghazi, etc. to raise doubts about these forecasts of governmental omniscience? From a civil liberties perspective, the best thing we have going for us is incompetence – some rooted in human frailties, some rooted in institutions. Let’s al praise stupid men (and women).
    Moreover, there is the not insignificant question of what you do with information once you have it. I cite in this regard: Operation Barbarossa, Crete, Iran.
    Finally, let’s remind ourselves that the gravest threat to the well-being of the United States over the past decade came from “Wall Street.” Some people at least knew all the shenanigans that were going on and their implications. The Fed was warned of the housing bubble’s potential menace in 2006. The actual tapes of the Open Market Committee at which these concerns were raised is open to the public. Members, led by Ben Bernanke, can be heard laughing at the ‘ridiculous’ idea. So let’s not praise stupid men/women indiscriminately.

  6. confusedponderer says:

    I always wondered to what end Metadata collection of this sort is most suitable, and eventually concluded that with the methods described it probably is best suited to crack down on domestic dissent and activism.
    IMO that is so because it works the better the more data you have. The government always knows most about their own citizen than anybody else. Domestically, this sort of information is fairly easy to get since the government is for one most familiar with their own people and also, at the source, and they have the power of law (and secrecy) to exact it from private service providers.
    Contrast that with intel gathering in far away places like Iraq with the culture and language differences. There all they had was cellphone data, and having called and been called by the wrong number too often qualified you for a drone strike execution. That is fairly close to the ruthless and brutal French approach in Algiers to torture a guy, have him spill out names, add them to your organigram, torture those guys too until they spill out names and so forth.
    The Metadata snooping is in a different league altogether. It is a fishing expedition only when targeted at everyone. It is rather targeted when the government decides to use it against dissenters, analysing and mapping for instance their e-mail, Skype, Facebook, twitter, message board accounts, bank account, credit card information and phone records.
    What about outing a critical journalist’s or political opponent’s affair based on analysis of his Metadata? That’s not at all far fetched – after all Petraeus affair was uncovered in such a way. The intertwined relationship and in part even deputisation of private actors by npolice and national security services only adds another layer concern.
    Didn’t that Stratfor e-mail hack show that Coca-Cola, a private company, asked a private consultant to get PETA’s FBI file (and get it)? There is a real possibility that a company may go after a critic with information so obtained.
    The opportunities for foul play that present itself in the information gathered by Metadata analysis are innumerable.
    Considering the hysterical and heavy handed reaction of US police departments, it is to me a fairly sure bet that folks like occupy or then tea party have been profiled in such a way, with the help of anti-terror fusion centres and with the help of DHS. It is IMO improbable that DHS is not and has not been using these tools at home on US citizens.
    The true menace IMO is that these tools are used in COINTELPRO tactics, aimed at disrupting and discrediting domestic dissent. There is your soft totalitarianism, and there is no guarantee whatsoever that the tools described are not already being used that way. What exactly was NYPD’s “mapping the human terrain” of New York about during occupy?
    All of that ought to be a very serious since real concern in light of today’s overly broad definition of terrorism and WMD, and today’s proactive law enforcement with their pre-cime arrests, when for instance undercover cops or informants talk or trick the slowest kid at the mosque or at occupy into doing something stupid on mike or posting it on facebook, resulting in a thought crime provoked- and a life wrecked and a movement discredited.

  7. confusedponderer says:

    … and yes, it is a brilliant piece indeed.

  8. mbrenner says:

    Additional thought. Network analysis clarifies patterns of social interaction – that’s all. Only direct knowledge of the particulars enables purposeful action. We should have learned that lesson from Iraq, Afghanistan, Yemen. Otherwise, you get the idiotic, counter productive ‘signature strikes,’ mass detentions and a fale presumption of actually accomplishing something other than shooting yourself in the foot.
    By the way, another example of information squandered is ignoring the Chinese moves into North Korea. The record suggests that most ‘intelligence’ failures are actually failures of intelligence.

  9. Duncan Kinder says:

    This is a tale full of sound and fury.
    But what does it signify?
    1) Paul Revere was no secret to the British. He was the fellow who published the famous Boston Massacre prints. They didn’t need any metadata to identify him.
    2) In fact, William Dawes and Dr. Prescott roused the colonists. So even if the British had nabbed Revere, things would have proceeded much as we know anyway. ( Sorry Longfellow ).
    Now what would be interesting would be if we could use historical metadata to identify some Mortimer Snurd living in Connecticut who actually was a prime mover in the American Revolution unbeknownst to history until now.

  10. mbrenner says:

    Some one has asked me what I meant by stupidity since the people involved clearly are not IQ deficient. We know that there are multiple forms of intelligence; there also are many more forms and modalities of “stupidity.” What I have in mind is the degree to which a person has a competent mind and the ability to reason logically in complex circumstances. The Clappers, Alexanders, Holders, etc. have limited minds and a limited capacity for intelligent action. In this context, that is reassuring – on balance.
    By contrast, if there were a Markus Wolff among them, I’d worry.

  11. Castellio says:

    More interesting would be to use metadata to understand who is running, and who has been running, the United States. Lets go back to the pivotal year of 1963, and start to build the memberships of certain interlocking groups, and track them over time through the invasion of Iraq and the financial bail-outs of 2008, 2009.
    Bank directorships, Federal Reserve membership, Supreme Court, Administrative governmental positions, Major defense contractors, Major Corporate owners, Media ownership, Congressional leaders.

  12. Duncan Kinder says:

    Barbara Tuchman provides us with an illustration of how relying upon metadata can be problematic in her A Distant Mirror: The Calamitous 14th Century.
    While researching this book, she noted that an obscure French nobleman, Enguerrand de Coucy, happened to be present at a remarkable number of major events. The metadata all pointed toward him.
    But just because de Coucy wasw there did not mean he had a lot to do with what was happening. Tuchman’s narrative is strained, contrived, and fails to deepen our understanding of those events.
    So perhaps metadata could disclose to us that Snurd, like Flashman, was present at the charge. But that does not necessarily mean his tale would be a good read.
    So she centered

  13. Tyler says:

    I’d say that some of you are missing the point. This metadata isn’t so they can identify people already out in the open, but so that if someone that they don’t like (say, some future populist candidate) starts making inroads, they can drag up any embarassing event and leak it to their mouthpieces.

  14. walrus says:

    The best possible reputation any security service can have is one of bumbling inefficiency. Provided enough care is taken to disguise operations, opponents natural egotism will do the rest when an action succeeds; “They couldn’t possibly have done that! It was sheer bad luck that we lost!”. We are on extremely shaky ground if we make the assumption that the NSA programs concerned have limited capabilities that have no potential impact on civil liberties.
    So lets look at the arguments about why we shouldn’t worry about this allegedly newly discovered eavesdropping, but to properly evaluate them it is first necessary to understand a few things about technology…
    Moores Law more or less predicts a doubling of transistor density, and hence roughly computing power and storage density every Eighteen months to Two years. Historically this has seen the cost of computing hardware and memory become dramatically cheaper. I’ll let others make the comparisons about GM cars and Microsoft, etc.
    The point of this is that the cost of the sort of data collection, storage, mining and analysis allegedly now or about to be undertaken by NSA is only going to get exponentially cheaper with time. Furthermore, although I have not done the analysis, it should be possible using Moores law and the alleged NSA budget to prove that the cost of what is now being attempted was too stratospheric even for the Federal Government to consider it until a few years ago……..and that is without the discovery and application of new technologies, more about that later.
    Having introduced Moores law, let me introduce Arthur C. Clarkes Three Laws:
    1. When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
    2. The only way of discovering the limits of the possible is to venture a little way past them into the impossible.
    3. Any sufficiently advanced technology is indistinguishable from magic.
    It appears that NSA understands this and perhaps agrees with some of our posters here that the data analysis and mining technology available TODAY is not up to the task. However thanks to Moores law the data and/or Metadata can be stored perhaps indefinitely until such time as Clarkes Laws can provide the sufficiently magical solution that can instantly divine terrorist intent in the purchase of a pressure cooker and potassium nitrate.
    Bear in mind that Historical methodology plays an important role in intelligence, for example the Venona project, started in 1943 was only terminated in 1980 – which should give people some idea of the lengths a service will go to to try and pinpoint a target by reference to actions they took Thirty Seven years, and longer, ago. Given this understanding, it now becomes possible to understand the desire of the NSA for long term longitudinal meta data on the population – to be able as Control instructed Smiley in “Tinker Tailor” : “To go forward and backward in time”.
    It is axiomatic in engineering that when Clarkes three laws are proved right once again and a new and unforseen technology matures, we get a step shift, in technological capability, not the usual annual incremental change. This is usually termed “disruptive technology” by its victims since whole technological ecosystems wither and die under the evolutionary pressure of the new.
    The problem for all of us is that our laws are written with reference to the current technological paradigms and we struggle to adapt. It was not possible for the founding fathers to cater for the possibility that a Federal Government might one day have the magical capability to record every citizens transactions in real time and store them for future use.
    It was also not possible for the framers of the current laws regarding mail interception, wire-tapping of analogue phone calls and similar surveillance techniques to forecast the disruptive effect of new technology such that ALL citizens may be subject to scrutiny ALL the time.
    Today it is still not possible to understand how the data being collected may be mined and analysed – although Quantum computing is being developed which has the potential to break strong encryption and perhaps perform other miracles.
    Given the current technologies, we are not in Kansas any more. The potential of the current technology has outstripped the rule of law and unless strong legal controls are applied the people must suffer.

  15. walrus says:

    I agree with Tyler, the data will be misused for political purposes and it will also have a chilling effect on all discourse.
    On that note, the journalist who broke the Snowden story has already been threatened.

  16. optimax says:

    The British did nab Revere on his famous and historically mythologized ride.

  17. Fred says:

    You forgot number 3) John Adams he was no secret to the British. He even defended those soldiers accused of the Boston Massacre and convinced a jury of colonials they were innocent. He must be loyal to King and Country.
    Then number 4) Joseph Warren. He was instrumental in organizing the rebellion and died fighting at Bunker Hill. He’s the one who recruited Dawes and Prescott. One must really watch out for those graduates of Harvard.

  18. Fred says:

    Lets not forget getting the goods on the legislative staff or the campaign staff.

  19. kao_hsien_chih says:

    I think this should make us worried about how pretty much anyone, with even fairly limited technical skills, can deduce all manner of information about a lot of people and things.
    It is one thing that government does it–I don’t suggest one should not be fearful of them snooping around, especially given the resources it has to do harm to those whom it decides to target for whatever reason–but they are hardly the only ones who can use metadata to do harm. As you noted, large corporations can do it. So can political and social activists of all types, terrorists, disgruntled employees, jilted lovers, whatever, basically all sorts of types. All manner of data is available relatively cheaply as is computing power and off-the-shelf packages needed to analyze them. What’s more, it’s getting easier and easier to make use of these even by those with relatively limited skills. Who is going to protect against these threats, provided that we don’t want to go mostly Luddite and shun modern information technology.
    The irony is that it is the government itself that has the most resources to provide that protection–provided that it can be trusted. But if the government is run by self-righteous do-gooders who believe themselves entitled to abuse their resources in pursuit of their “obviously righteous” causes, how can anyone turn to them for this badly needed protection? This is the real dilemma presented by the metadata scandal, I think: not only should government not actively compile and use data and ought to offer credible guarantees against abuse, it should be more actively involved in preventing abuse by others. Instead, we have a government that not only refuses to offer a guarantee of this sort, but is seemingly engaged in promoting abuse by non-government entities, in course of being its agents. This is a dangerous situation indeed.

  20. mbrenner says:

    Our perspectives are divergent because your emphasis is on technology while mine is on people. Neither Nazis nor the Bolsheviks needed more than rudimentary technology to operate a totalitarian system. Those who will be in position to exploit what is taking shape small minded men of petty ambition. Their gratification comes mainly from the sense of power of having all this capability and information at their disposal – not exploiting it systematically to do something diabolical. It’s a boy’s game of secrecy in a high-minded cause Of course, they will persecute and prosecute anyone who either exposes their secret project or highlights that it serves no important purpose. Beyond that?
    The very fact that the systems are being created with no ulterior purpose or even any definable policiesin mind means that success or failure is immaterial. It’s the game that counts – not stopping al-Sabaab friends in Somalia from receiving $8,500 dollars. AND this game of massive domestic surveillance has no bearing on the thwarting of a conjectured 9/11 Redux because a) none is possible; and b) Meta-data mining is not the way to uncover and foil it were it to appear

  21. confusedponderer says:

    “It is axiomatic in engineering that when Clarkes three laws are proved right once again and a new and unforseen technology matures, we get a step shift, in technological capability, not the usual annual incremental change. This is usually termed “disruptive technology” by its victims since whole technological ecosystems wither and die under the evolutionary pressure of the new.
    The problem for all of us is that our laws are written with reference to the current technological paradigms and we struggle to adapt.”
    I observe this in publishing, where a generation of typesetters is fighting for their survival in face of technological revolution. We can now automatically produce excellent typesetting for an entire book, fit for print, at the push of a button in fifteen minutes using properly formatted XML.
    I am currently on the side of the disruptive technology, but as to what will be in 15 years, who knows. I will have to adapt or wither.
    The only factor that is keeping our typesetters employed at the time is the outright refusal or incapability of our editorial office to standardise their books. Naturally this results in friction (and costs).
    The pressure from the top to standardise in order to increase efficiency and make products cross-platform-capable, coupled with the introduction of new and powerful editing tools that will further streamline the production line, will probably be irresistible soon. The new and powerful editing tools are more efficient than outsourcing, and even more cost effective.
    The technology is out there, management has taken notice and it is folly to assume that they will not grasp this opportunity to maximise profits and efficiency with both hands.
    So I tell them to learn XML! Some get it, others don’t. A pity, that. I literally got the reply once ‘I don’t do XML’. True, sadly. The man is doing an excellent job in any other respect. Will that help him? Probably not.

  22. robt willmann says:

    Research into the idea of social networks seems to have picked up steam in the late 1950’s. One of the main papers in the area by Pool and Kochen should be here in pdf format (the 1978 version, 46 pages)–
    It discusses some of the issues and problems involved in generating valid information from network analysis.
    A paper was done in 2003 at Hewlett-Packard Labs using e-mail traffic between persons at the Labs and a social website used by some students at Stanford University. It is much less in-depth and much shorter.
    Here again is the link to a short video I mentioned the other day because it contains a few critically important pieces of information; it is mainly about the long-time NSA employee William Binney who resigned in 2001 and became a whistleblower–
    One thing he says is that the data collected is of separate types of activities, such as telephone, banking, and so forth, with each one being called a domain. Then using identifiers (name, date of birth, phone number, social security number, etc.), as to each person you can pull information on all his activities from each database of each type of activity and create an immediate picture of him going back in time, as well as his social network or community. Most people are in similar types of records — telephone, utilities (electricity, water, and gas), bank, credit card, credit history, medical, prescription drug, vehicle title, vehicle license plate and registration, drivers license, home loan or apartment, vehicle and property insurance, toll road travel, and, since after 2001, every time you get on a domestic airline flight. Your voice print may now also be saved by telephone companies. All the NSA has to do — and apparently does — is to get those records by type in bulk, and the computer software then automatically builds profiles on everybody in the data.
    From this we can see that a second part is the tracking of persons, not only as to past movements, but in as close to real time as possible. This of course depends on direct linkages, which may exist. In this regard, you should also consider the red light cameras and what their sensors and software are capable of. It is clear from the Texas law that authorized the red light camera scam that they have surveillance capability, but the precision of that capability has not yet been revealed. Also be aware of the possibility of Radio Frequency ID chips in credit cards and drivers licenses; they could permit real time tracking.
    The third part is that this type of surveillance, data collection, and tracking system is a huge force multiplier for a central government. No longer do I have to send personnel out to travel and to talk to neighbors to try to find out who your friends are, where you work, what kind of car you have, and where you hang out. I just type in likely identifiers of you, and in seconds I know more than what would take days or weeks of investigation to find out. This is particularly useful in trying to short-circuit movements for political or economic change. Because I can rotate the contents of the databases, I might be able to ferret out your name if I do not know it.
    Michael Brenner expressed above that he has some comfort from the apparently limited thought processes of some of the players in the system, and that they are not like Markus Wolf of the Stasi. But the problem is that the system that has been constructed since September 2001 in the form of the wording of the new laws; the propaganda selling it to the public; the creation of the first-ever secret, national, internal security apparatus (the Dept. of Homeland Security); the enriching of private contractors; and the process of neutralizing and co-opting state and local law enforcement has been the product of a level of sophistication one could attribute to a Markus Wolf.

  23. Fred says:

    “Their gratification comes mainly from the sense of power of having all this capability and information at their disposal …”
    I would say all you have to do is watch an episode of ‘COPS’ and you get a feel for just what kind of intimidation of the general public can and will go one. As you note, it will take better leadership from those who are not “small minded men of petty ambition.”

  24. mbrenner says:

    I do not dispute the technical advances or the nefarious effects of what is being done. My main points are these. One, I see little evidence that MOST of the officials involved have any goal beyond creating the system, i.e. no plans for its full exploitation for an enduring political purpose. Abuse, of course, is possible – and likely in individual cases – but nothing systematic. Two, the technical design may be worthy of a Markus Wolff but not the overall political plan – which, to my mind, simnply does not now exist. Wolff was a dedicated Communist who used all his talent to advance the cause of the DDR/USSR/ & world Communism. He would have been ruthless and unrelenting in exploiting PEISM etc toward these ends. What are our guys aiming at? Supposedly. American security – whatever that means. (See above) But they have neither a defined tangible enemy nor an embracing scheme. Three, the best way to understand these poeple’s psychology is to think of the Masons coming to power and operating such an apparatus. Their goal? Exclusivity, status, secret rituals etc. Frankly, at the end of the day I sense that the thinking (such as it is – or, better, sentiment) of Clapper, Alexander. Holder, Obama, et al is deriving a deep satisfactiomn at being something special and doing something noble even as they can’t define with any precision what tangibly that actually is directed toward. Crazy? Well, America’s political elites manifestly have gone crazy IMHO.

  25. Walrus says:

    I agree with you the possible motives for such a system are seen as relatively benign right now but I have caveats about future use. These are on three grounds.
    The first is that ordinary humans consistently underestimate the depths of evil some people will plumb, so while we cannot discover anyone who regards an Orwellian police state as a desirable outcome doesn’t mean that such folk don’t exist within the corridors of power and their ascent if it came, would be seen as a complete surprise.
    Secondly, I am concerned with the tyranny of the masses. Note how DHS has already expanded its mandate from just terrorism to drug crime. The tools given them to fight jihadis are increasingly being turned on ordinary Americans. For example ordinary aircraft are routinely tracked, stopped and searched by DHS, to the alarm of pilots, looking for “drugs”.
    I imagine pressure will mount to use PRISM for ordinary criminal work, say investigating child pornography, etc. thanks to the existence of “fusion centres” there is no telling how low access to PRISM data may go – to fight crime of course. Do you really want the local Sheriff to know your Internet surfing habits?
    My third concern is leakage to corporate interests. I know there is regular contact between intelligence services and corporations – I remember seeing a CIA profile on a potential foreign distributor on the desk of a friend in the Ford Motor Corporation, circa 1976. However I would have concerns if real time data on my private home emails wound up in my employers hands.

  26. walrus says:

    Sir, I agree with you about the current incumbents. My concern is about leaving the tools and records about for a future “Dear Leader” who may have a less charitable agenda.
    Recent history has shown that such creatures can arrive unexpectedly and that the population consistently underestimates the depths of their depravity.

  27. Fred says:

    Looks like the Brits are keeping tabs on folks and arresting them if they don’t like the police’s offer of where to exercise their, well I guess they don’t have rights to free speech in the UK. I wonder if our royalty worshiping media is paying attention.

Comments are closed.