Big Data, Insider Threats, and International Intelligence Sharing

By: Dr. James Igoe Walsh

How does the "big data" revolution affect intelligence sharing between countries? The sharing of intelligence derived from big data—such as intercepting text and voice communications carried over the internet—could improve intelligence about threats such as transnational terrorist groups by providing intelligence agencies with a much larger pool of data and new software tools that permit faster and more focused analysis. This is why intelligence agencies in the United States and many other countries have moved aggressively over the last decade to develop the capacity to collect and mine online data.

Intelligence agencies now collect and store electronically enormous amounts of information that they want to keep secret; however, the very ease of storing and sharing this collected data also makes the data vulnerable to foreign states, hackers, and others that want access to it without authorization. Some of these are "outsider threats," such as the intelligence agency of a foreign state, and can be thwarted with varying degrees of success by denying access to an intelligence agency's network. Here I focus instead on "insider threats," by which I mean individuals who have legitimate access to some secret intelligence and exploit their position to share the information with outsiders.

Big data magnifies the threats that such insiders pose, because they may be able to access a great deal of sensitive data and reveal it at a low cost to foreign states, criminal or violent organizations, journalists, or human rights groups. Concerns about such insider threats could reduce the willingness of states to share intelligence with each other, thereby reducing the international community's ability to collectively define and respond to security problems.

I suggest a number of ways that states may decide to address this sort of problem. These solutions include

  • sharing only with the most trustworthy partners,
  • establishing hierarchical relations among the intelligence agencies of different countries so that one can closely monitor the other,
  • greater transparency and oversight of the use of big data by intelligence agencies, and
  • foreswearing at least some international intelligence sharing.

Each of these solutions has strengths and weaknesses, and it is likely that we will see a mix of these strategies implemented by different groups of countries. One counterintuitive implication of this discussion is that increased transparency about the process of data collection and analysis is valuable not only because it is consistent with democratic norms, but also because it actually reduces the problem of insider threats and makes promises by one country to share intelligence with another more credible and effective.

Big Data and Intelligence Sharing

Shared intelligence goes beyond the exchange of factual information and includes, in many cases, the sharing of analytical products. Sharing allows countries to spread the costs of intelligence acquisition and analysis across all of the participants, and, as is the case with most exchanges, the benefits from sharing increase when participating states specialize. Thus, sharing should lead to the production of both more and higher-quality intelligence. Before going into how big data alters these calculations, it is important to understand what big data means in the context of this discussion. I suggest that big data is composed of three technological trends. These trends are wellknown and have implications well beyond the issue of intelligence sharing, so I will mention them only briefly here. The first is the ability to collect and store large amounts of electronic communications. Intelligence agencies have been collecting such information for some time, but today the scale is much larger, since individuals rely on electronic forms of communication far more heavily than in the past. A second trend is the ability to analyze this data more effectively and quickly than before, using software tools such as search algorithms, network analysis, geographic information system tools, and other approaches. Thus, while the scale of data collection has grown enormously, the tools for analyzing data have also improved, making it somewhat easier to solve the "needle in a haystack" problem. Third, much lower storage and retrieval costs make it much easier to both collect and analyze electronic data, as well as to share the data and analysis widely within and across organizations.

Big data may create powerful incentives for states to share intelligence more frequently and intensively. There are three reasons for this, as well. The first is that information technology substantially reduces the marginal cost of sharing intelligence. Sharing intelligence in an electronic format at scale requires large investments in information technologies such as storage space and protocols for remote access to data. Once these investments are made and the technologies and practices are in place, it is very inexpensive from a technical standpoint for one country to share an additional piece of information with another. In principle, any intelligence in an electronic format could be shared with any user almost instantaneously at a very low cost. Second, the creation of such sharing technologies could have powerful network effects. Once a critical mass of states develops the capacity to use such technologies and protocols for sharing intelligence, participation should become increasingly valuable as more countries are able to share intelligence with each other. Related to this, the information technology infrastructure required for sharing on a massive scale is expensive and complicated to develop and maintain. It requires investments in hardware, communications equipment, and skilled operators who can design and manage the entire system. The fixed cost is likely higher for intelligence agencies than for comparable activities in the private sector, because the collection and analysis ambitions of intelligence agencies are much higher, and also because intelligence agencies have concerns about information security that might lead them to prefer trusted suppliers inside and outside the government over the ones with the cheapest rates. Some states may lack the financial and technical means to create and maintain such systems. This gives them stronger incentives to collaborate more closely with other states, since collaboration spreads some of the costs across the budgets of multiple countries.

It is important to recognize that participants in intelligence-sharing arrangements differ in their ability to provide useful intelligence to each other. The United States is clearly the dominant actor in all of the sharing networks in which it participates. It spends far more on intelligence than any other state, has global collection and analysis needs, and was the first to develop or implement on a large scale many of the big data applications for intelligence. This dominant position has two implications for intelligence sharing. First, the network effect is really a U.S. network effect. Other states have a great deal to gain from sharing intelligence with the United States, while the marginal benefit to the United States of sharing with any particular state is likely to be much lower. Second, the United States is better positioned to "go it alone" in intelligence sharing. It may be able to collect a considerable amount of electronic intelligence without the cooperation of other states, relying on its own satellites, sensors, and other means of data collection. In some ways, the United States accrues fewer of the benefits that arise from specialization in intelligence collection and analysis. This means that Washington is wellpositioned to exert the greatest influence over the content and interpretation of intelligence-sharing agreements, and that big data may increase the dominance of the United States over its partners and permit it greater leeway to operate without their active support. Big data, then, produces somewhat mixed incentives for the United States to share intelligence. On the one hand, big data allows the United States to distribute the costs of collection and analysis over more partners, and to gather and analyze more data shared by these partners. On the other hand, U.S. dominance of many of the technologies that are applied to intelligence sharing may reduce its need to rely on partners for critical intelligence. How the United States is likely to respond to these incentives, I suggest in the following sections, depends on its ability to effectively manage and oversee its partners' information security practices.

Big Data and Insider Threats

Given the benefits, why might two states not share intelligence with each other? Why would they forgo the potential benefits of sharing? Theories of international cooperation identify an important barrier to sharing in situations where mutual benefit is possible: the enforcement problem, which is very relevant for the issue of insider threats.1 The enforcement problem arises when a country reneges on or "defects" from a commitment to share intelligence. Defection may be either deliberate or involuntary, in the sense that lower-level state officials defect without their leadership's approval. A sender of intelligence can defect by altering intelligence's content, withholding it altogether, or exaggerating the accuracy of its sources. Senders that defect deliberately manipulate shared intelligence with the intent of influencing the recipient's subsequent actions. Alternatively, individuals within a sending state might be operating under the commands of another power or group that controls the intelligence that these individuals then pass to partner states. Corruption or other administrative weaknesses might limit the state's ability to effectively collect intelligence in the first place. Sending states also might not share fully or honestly if some of their personnel who control the relevant intelligence disagree, on political or policy grounds, with the decision to share it. A recipient of intelligence might defect by forwarding shared intelligence to a third country if it concludes that its interests are served by passing along intelligence, even if this might conflict with the interests of the sender. A recipient also might inadvertently forward shared intelligence to third parties, such as hostile states or the media. Individuals who have access to the shared intelligence may be agents of a third state or other outside group and may violate their government's policy by sharing intelligence with their controllers.

The costs of defection can be large for both senders and recipients, which is one reason why the enforcement problem looms particularly large when states consider sharing intelligence. Recipients may be deceived into providing benefits to senders that provide them with low-quality intelligence. More important, though, are the potential indirect costs of cooperating with a sender that defects. A recipient may base important foreign policy decisions involving the use of force on flawed or misleading intelligence shared by other states. Costs for the sending states can be substantial as well, including sharing secrets or sources and methods with third parties. These costs increase when the participating states have developed specialized and complementary intelligence efforts: specialization increases the costs of defection. The most valuable sharing partners have much useful intelligence, but they also can do the most damage when they defect from promises to share. Defection can remove access to the partner's specialized assets and seriously weaken the recipient's ability to gather useful intelligence on a target.

Another reason why the enforcement problem is a powerful barrier to intelligence sharing is because it is difficult to determine whether a partner has reneged, for at least two reasons. First, intelligence almost always includes an analytical element, which may be more easily manipulated by a state than the raw information on which it is based. Second, and perhaps more important, intelligence by definition includes secret information, although most intelligence producers use open sources of information as well as secret or clandestinely obtained information. States and their intelligence communities go to great lengths to secure secret intelligence and to prevent their targets from discovering their sources and methods of intelligence collection and analysis. For this reason, intelligence agencies are reluctant to share all of the intelligence they control, even within their own governments. This concern for security makes it very difficult for one state to determine whether another has defected on a promise to share intelligence. Keeping details of intelligence collection and analysis secret is, on the one hand, recognized as a legitimate security practice, but on the other, it makes it easier for a sending state to alter or fabricate the information it passes to others. The barriers to sharing raised by security requirements also pose difficulties for the sending states. Sending states may want to ensure that recipients do not pass the shared intelligence along to enemies, either deliberately or inadvertently, but at the same time, the recipients do not want to divulge their security arrangements, to prevent others from illicitly gaining access to the intelligence they possess.

Big data can magnify these costs and risks of intelligence sharing, especially those associated with inadvertent defection by insiders in other states. The big data revolution makes it less costly and difficult to share large amounts of intelligence with a greater number of states. States participating in such arrangements, in turn, will need more technical and analytical personnel to process this information and develop it into analytical products for the consumers in their country. This means that there are more "insiders" in the recipient countries who could potentially share this intelligence with third parties. It also means that there is a potentially much larger trove of intelligence for insiders to steal. The fact that the intelligence is in an electronic format rather than on paper makes it much easier to convey to third parties. Furthermore, the highly-skilled technical personnel whom states need to manage their larger and more complex information-sharing infrastructures are in high demand in other sectors of the economy, and governments may lack the salaries to compete effectively to hire the most desirable candidates. This may be a particular difficulty for intelligence agencies, which place a high priority on monitoring the backgrounds and work practices of their employees to minimize insider threats, because information technology experts have a reputation for valuing their autonomy both in and outside of the workplace. Intelligence employees' need for access to a wide range of internal systems and processes in order to do their jobs means that traditional security practices such as compartmentalization and "need to know" may hinder their performance. This requirement gives them potential access to large volumes of intelligence information, as well as the skills to cover their tracks should they choose to copy and share such intelligence with a third party.

Possible Solutions to the Insider Threat Problem

If enforcement problems make sharing intelligence difficult, how can these barriers be overcome? This section looks at four possible solutions: selecting trustworthy partners, establishing hierarchical relations between states that share intelligence, increasing transparency and oversight of the use of big data for intelligence purposes, and scaling back and limiting intelligence sharing.


A straightforward solution to the problems created by big data and insider threats is to select intelligence-sharing partners that are trustworthy. In this context, trust is an expectation on the part of one state that another state will not defect from promises to share intelligence in a secure manner. Trust emerges most robustly when parties do not fear that their partner's interests diverge from their own.2 Divergent interests may give the sender an incentive to deliberately communicate incorrect intelligence, in an effort to convince the recipient to act in a way that is most favorable to the sender. On the other side, a receiving state might pass along shared intelligence to third parties without the sender's knowledge or permission. From this perspective, states should focus on selecting their partners carefully, only sharing intelligence with those that have common interests and a history of living up to their promises.3

Furthermore, there is much evidence that mutual trust facilitates the reliable exchange of information. In the area of intelligence sharing, many of the states with which the United States shares intelligence most intensively—Canada, the United Kingdom and other states in Western Europe, Japan, and South Korea—are long-standing allies with successful histories of cooperation on intelligence sharing and a range of other issues. Big data may erode the utility of such generalized trust, however. The fact that so much more intelligence could be shared today than in the past, and that more personnel may have access to it, increases the risks from insider threats within the intelligence agencies of partner countries. Breaches of security by even one individual could do substantial damage to foreign partners. For example, Edward Snowden's revelations about the collection of cell phone and e-mail communications data by the National Security Agency exposed a remarkably wide range of information, and Bradley Manning's alleged leaking of hundreds of thousands of internal documents to WikiLeaks provided quite detailed information about a huge range of topics. Big data, then, might make reliance on past trustworthiness, as a way to screen desirable from undesirable intelligencesharing partners, a riskier strategy than it previously has been.


Selecting only partners that are trustworthy has another cost: it can exclude partners that may have particularly valuable intelligence. In practice, it appears that states not infrequently share intelligence with partners they consider less than completely trustworthy. The United States is one prominent example. In these circumstances, intelligence-sharing agreements between states can be constructed as a hierarchy. Hierarchy differs from anarchy, often considered the normal state of affairs in international politics, in that a subordinate state voluntarily gives up some autonomy to a dominant state. The dominant state can monitor the subordinate state for defection and punish such defection when it occurs.4

Hierarchy can help to manage the risks of defection. The dominant state in the hierarchy takes the role of making important decisions about the form of the intelligence-sharing partnership. Subordinates give some of their decisionmaking autonomy to the dominant state, in return for which the dominant state provides the subordinate with some combination of its own shared intelligence, diplomatic support, economic assistance, and other valuable goods and services. Hierarchy provides one possible solution to the enforcement problem by allowing the dominant state to take direct and intrusive steps to ensure that the subordinate is not defecting from its promise to securely share intelligence. In the context of big data, hierarchy would involve the setting of commonly agreed standards for preventing insider threats. It would also require the dominant state to play an active role in vetting how well the subordinate partners implement these standards. This role can be carried out in a number of ways. For example, the dominant state can assign liaisons who directly participate in, and can thus oversee, the intelligence activities of subordinate states. Such liaisons can play an important technical role in facilitating cooperation, by, for example, ensuring that each country's information technology can communicate efficiently and securely. But liaisons also serve as a useful way to check on the day-to-day internal security practices of subordinate states. The dominant state can also pay for a disproportionate share of the physical and information technology infrastructure needed to carry out large-scale data sharing, helping its partners develop more sophisticated data collection, storage, and analysis capabilities. Such cost-sharing typically comes with strings: for instance, the dominant state asserts the right to audit how funds are spent and technology is used. This gives the dominant state some direct control over the processes that are used by subordinates.

Hierarchical arrangements also create costs and risks for the participants. Most obviously, they give much greater freedom of maneuver to the dominant state than to subordinate participants. Subordinates surrender some degree of control over their internal standards for preventing insider threats, which will lead some to conclude that these sacrifices are not worth the benefit of sharing intelligence. Hierarchy works most effectively when the participating states' underlying interests are compatible enough to prevent large incentives for defection, since compatible interests allow states to use the institutions of hierarchy to bridge their differences. What do subordinates get in return? As suggested previously, the dominant state typically provides valuable goods and services to the subordinate, such as access to some of its much more extensive collection of intelligence products, technical support for developing the capacity to collect and analyze large amounts of data, and financial inducements. These, in turn, are some of the costs that the dominant state pays to entice subordinates to join it in a hierarchal relationship. A more important cost, though, is that the dominant state must reassure its subordinates that it will not defect on them. There is a risk that the dominant state might use its extraordinary position in the relationship to secretly obtain intelligence from its partner that the latter does not wish to share. The dominant state might also be subject to its own insider threats. This possibility will be of particular concern to subordinates, which often prefer that the details of their intelligence-sharing arrangements, including their subordinate status, not be revealed to outsiders. How the dominant state reassures potential subordinates that it will not defect, deliberately or inadvertently, is a key question that must be addressed if a hierarchy is to function smoothly and serve the interests of participating states.


Another way to address the issue of intelligence sharing is through greater public transparency, especially on the part of the dominant state, about the processes it uses to collect and share intelligence. To this point, I have assumed that all insider threats are identical. But it is likely that the motives and goals of insiders willing to reveal intelligence are at least somewhat varied. Two types of insiders dominate current discussion of the problem. The first, more traditional type are insiders who volunteer, are paid, or are coerced to provide intelligence to a foreign power or organization. The second type are insiders who are disturbed by the scope and scale of contemporary intelligence collection because, in their view, it violates privacy rights or the law. These different types are likely to behave quite differently. Those in the first group have powerful motives not to publicize their theft or unauthorized sharing of intelligence, because their illicit actions will be brought to an end and they probably will be prosecuted. But publicity is the key motive for the second type, who hope that revealing the extent of intelligence collection will spark public outrage and lead to restraints on intelligence activities. Prosecution may even be part of their publicity strategy.

These different types of insiders may be countered with distinct policies. Traditional counterintelligence practices, such as background checks, should have some chance of catching both types. Insiders of the second type, however, may also be dissuaded from releasing information by greater official transparency about how and what data is being collected, and under what circumstances this data is analyzed by intelligence agencies. Such transparency about the general processes of data collection and analysis (but not, of course, about the content of this data) may mollify insiders who want a vigorous public debate about the trade-offs between security and privacy. Subjecting such practices to some sort of systematic legislative and/or judicial oversight would also ensure that new collection techniques, which emerge frequently because of rapid changes in communication and commercial technologies, are regularly included in the oversight process.

In addition to preventing some insider threats, greater transparency could help to reinforce international hierarchies for intelligence sharing. Recall that one difficulty with hierarchical relationships is that the dominant power must figure out a way to reassure subordinates that it will not abuse its extraordinary position. Greater transparency about big data and intelligence collection and analysis within the political institutions of the United States, for example, could also provide some information to foreign partners that could, for example, monitor open hearings in Congress and the courts. Assuming that these institutions have some incentives to exercise vigorous oversight and to push the executive to detail and justify its data collection strategies, they might both reduce the danger from insider threats and also reassure intelligence allies overseas that their dependence on the United States is not being secretly exploited.5 At the same time, transparency is not free of risk. An important dynamic is the trade-off between using intelligence for counterterrorism or other goals, and privacy rights. Greater transparency risks revealing sources and methods to opponents, which can then take active countermeasures to more effectively mask their communications and activities. Striking a balance between enough transparency to mollify the critics of intelligence agencies (and reassure foreign partners) and sufficient secrecy about the details of how intelligence is collected and analyzed is not easy.

Go It Alone

In some cases, technological and political developments arising from big data might render intelligence sharing less attractive compared to other options. If insider threats in partner countries are judged to be severe, an intelligence service might be better served by "going it alone" and refraining from sharing. The United States may be uniquely well-positioned to pursue this sort of approach, because it has more alternatives to foreign intelligence services as sources of intelligence in the big data domain. A large fraction of internet communications traffic transits the territory of the United States, providing the opportunity to intercept foreign communications of interest. Many of the dominant hardware, software, communications technology services, and online communications services (such as e-mail, social media, and cloud storage) are headquartered in the United States. These firms often have terms of service that give them considerable latitude to share the data and communications they facilitate, and their status as American firms gives the U.S. government legal leverage to encourage or require them to collaborate with requests for data from the intelligence community. In addition, the U.S. military and intelligence community have unparalleled technologies for collecting intelligence remotely through networks of satellites, surveillance drones, and other devices.

The key advantage of going it alone is that it makes one country less reliant on another country that may be unwilling or unable to live up to the terms of an intelligence-sharing arrangement. But even if the quantity of intelligence collected by a single state could match that gathered by a group of states sharing intelligence, the quality of the intelligence is likely to suffer. Good intelligence analysis is typically based on both a range of sources and a close knowledge of the context in which the target of analysis is embedded. Both of these elements may be lacking when intelligence collection comes to rely primarily on intercepted communications and related sources. Analysts might misjudge the behavior of targets when they have only one type of information. Targets may begin to communicate strategically, saving their most important messages for offline communications and using electronic media to communicate less important or misleading information. Human intelligence is a useful corrective for these problems, and often a foreign partner is far better positioned to collect it. Another advantage of collaboration among intelligence agencies is that it exposes the agencies to the different points of view, assumptions, and judgments made by their counterparts, pushing them to critically review how they reach their own analytical conclusions. Much of this advantage would be lost if intelligence collection and analysis came to rely more exclusively on information that could be collected only by the domestic intelligence services.

The age of the internet and global social media—big data—has come on us so suddenly that most individuals, businesses, and governments are scrambling to determine the proper boundary between the private and the public, the secret and the open. Wherever this boundary eventually falls, intelligence services will continue to share some of the information they collect. Each of the strategies discussed here has benefits and drawbacks, but some combination of them is likely to provide the best way forward for dealing with intelligence sharing in the age of big data.

About the Author(s): Dr. James Igoe Walsh is a professor of political science at the University of North Carolina at Charlotte. He is the author of two books, including The International Politics of Intelligence Sharing (Columbia University Press, 2010), and a number of articles in academic journals. He is currently writing a book on the political consequences of the use of drones and similar technologies in counterterrorism and counterinsurgency campaigns, and was recently awarded a grant from the Department of Defense's Minerva Research Initiative to analyze how armed groups finance their activities and the consequences for military conflict.


1. There is a large body of literature on enforcement and defection in international politics; some of the fundamental works are James Fearon, "Rationalist Explanations for War," International Organization 49, no. 3 (1995): 379–414; Robert Keohane, After Hegemony (Princeton, N.J.: Princeton University Press, 1984); Kenneth Oye, ed., Cooperation under Anarchy (Princeton, N.J.: Princeton University Press, 1984); and Arthur Stein, Why Nations Cooperate (Ithaca, N.Y.: Cornell University Press, 1990).

2. For social science perspectives on trust, see James S. Coleman, Foundations of Social Theory (Cambridge, Mass.: The Belknap Press of Harvard University Press, 1990); Vincent Crawford and Joel Sobel, "Strategic Information Transmission," Econometrica 50, no. 6 (1982): 1431–51; Russell Hardin, Trust and Trustworthiness (New York: Russell Sage Foundation, 2002); and Carl I. Hovland, Irving L. Janis, and Harold H. Kelley, Persuasion and Communication (New Haven, Conn.: Yale University Press, 1953).

3. Chris Clough, "Quid Pro Quo: The Challenges of International Strategic Intelligence Cooperation," International Journal of Intelligence and Counterintelligence 17, no. 4 (2004): 603; Stéphane Lefebvre, "The Difficulties and Dilemmas of International Intelligence Cooperation," International Journal of Intelligence and Counterintelligence 16, no. 4 (2003): 528; and Derek Reveron, "Old Allies, New Friends: Intelligence-sharing in the War on Terror," Orbis 50, no. 3 (2006): 456.

4. James Igoe Walsh, The International Politics of Intelligence Sharing (New York: Columbia University Press, 2010).

5. On transparency, oversight, and international cooperation, see Lisa Martin, Democratic Commitments: Legislatures and International Cooperation (Princeton, N.J.: Princeton University Press, 2000).

Average (0 Votes)
The average rating is 0.0 stars out of 5.
No comments yet. Be the first.