• The Secret Sentry discloses that the U.S. invasion of Iraq in 2003 was far from the first time when U.S. government officials, including senior military commanders and the White House, “cherry picked” intelligence information to fit preconceived notions or policies and ignored intelligence which ran contrary to their expectations. The Secret Sentry and the documents posted today show that widespread manipulation of intelligence also occurred during the Korean and Vietnam Wars for example, when Washington ignored intelligence on Chinese intervention in Korea, resulting in catastrophic consequences. [See Document 9 and Document 21]

• The Secret Sentry also details how since the end of World War II, constant changes in computer, telecommunications, and communications security technologies have been the most important determinants of NSA’s ability to produce intelligence. NSA has oftentimes found itself behind the curve in terms of its ability or willingness to adapt to technological changes, with delays and bureaucratic inertia causing immense harm to the agency’s ability to perform its mission. [See Document 16, for example]. As a result, during the past four decades NSA has dramatically increased the amount of the raw material that it collects, even while it has produced less and less intelligence information. According to Matthew Aid’s informed sources, during the Reagan administration in the 1980s, NSA processed, analyzed, and reported approximately 20 percent of the communications traffic it intercepted. Today, that number has dropped to less than 1 percent.  For example, during the 2003 invasion of Iraq, NSA was unable to process 60 percent of the Iraqi messages it intercepted, while the U.S. military SIGINT units participating in the invasion processed less than 2 percent of the Iraqi military communications traffic that they intercepted.

Today’s posting of 24 documents consists of a selection of reports and memoranda prepared by NSA officials concerning the role played by Signals Intelligence (SIGINT) in selected military conflicts and crises, a number of classified internal histories written by NSA historians on key events in the agency’s past, and a selection of declassified articles from NSA internal journals.

Archive Visiting Fellow Matthew M. Aid obtained the documents while conducting research for his new book, The Secret Sentry: The Untold History of the National Security Agency (Bloomsbury, 2009). For the National Security Archive, Aid has edited a comprehensive set of declassified documents on the history of the NSA and its predecessor organizations from 1945 to the present, which ProQuest will publish later this year.

The National Security Agency
Edited by Matthew M. Aid

From its headquarters at Fort George G. Meade, Maryland, the National Security Agency (NSA) today controls a massive 60,000+ person intelligence organization with an annual budget estimated at over $10 billion, making it the largest spy agency in America and probably the world. NSA’s power and influence within the U.S. intelligence community has grown without interruption since it was created in November 1952, thanks in large part to the agency’s ability to consistently produce copious quantities of reliable intelligence information on a vast array of subjects from around the world ( Document 5). And despite the agency’s much publicized recent troubles, SIGINT continues to be the single most important intelligence source available to the President of the United States. As of September 2001, NSA was producing 60 percent of all the material that was to be found in President George W. Bush’s daily intelligence summary, the President’s Daily Brief ( Document 4).

But at the time of its birth in November 1952, declassified documents show that NSA was struggling just to survive in a period that a CIA official would later describe as “the Dark Ages of American cryptology." (Note 1) The nascent 7,600-person agency was striving against significant odds to establish itself on a firm footing, build up its intelligence gathering networks aimed at the Soviet Union and the People’s Republic of China, and trying to provide intelligence information for the U.S. military forces fighting in Korea ( Document 2, Document 13).

Nowhere were NSA’s problems more evident than in the Korean War, which eventually degenerated into a bloody stalemate pitting U.S. and United Nations forces against the significantly larger Chinese and North Korean militaries. (Note 2) Despite some significant early successes against North Korean communications during the first year of the war ( Document 7, Document 23), by the fall of 1952 NSA’s ability to provide meaningful intelligence declined markedly because North Korean and Chinese forces had systematically improved their communications security practices and procedures, reducing NSA’s SIGINT effort to whatever could be picked up from monitoring the enemy’s low-level walkie-talkie communications.

The Korean War also revealed one of the inherent problems afflicting all intelligence agencies--what to do when policymakers and military commanders refuse to heed the intelligence you provide because it conflicts with their own preconceived notions. For example, in the summer and fall of 1950, General Douglas MacArthur stubbornly discounted the SIGINT being sent to him showing an alarming increase in the size of Chinese forces based in Manchuria opposite Korea. By October 1950, the information derived from SIGINT and other intelligence sources overwhelmingly indicated that the Chinese intended to intervene militarily in the Korean War. General MacArthur and Washington ignored all of the indicators, with disastrous consequences ( Document 21). In the end, NSA’s experience in the Korea War was not a happy one. A former NSA historian would later write that in Korea “There were successes and there were failures, but the failures tended to overshadow the successes” ( Document 22).

NSA’s survival during the dark days of the 1950s was largely due to the efforts of two men. The first was Lt. General Ralph J. Canine, who served as director of NSA and its antecedent, AFSA, from 1951 to 1956. The combative Canine is today credited with lifting NSA out of the chaos that it found itself in by November 1952, and, in the span of four short years, making NSA a top-flight SIGINT organization and a force to be reckoned with inside the U.S. intelligence community ( Document 8, Document 18). The second person responsible for NSA’s success during the 1950s was President Dwight D. Eisenhower, who proved to be a critically important supporter of the agency. NSA was repeatedly investigated during the 1950s by a series of panels, all of whom issued highly critical reports about the agency’s health and future prospects ( Document 11). But Eisenhower, believing implicitly in the vital importance of the agency and its intelligence product, beat back all attempts by these committees to scale back or alter NSA’s efforts. In retrospect, it is doubtful that NSA would have survived the decade of the 1950s but for Eisenhower’s behind-the-scenes support and encouragement ( Document 12).

The agency’s top target throughout the Cold War was the Soviet Union, which consistently ate up more than half of NSA’s collection and analytic resources between 1945 and 1989 ( Document 6). The public is generally familiar with the significant successes by NSA’s cryptanalysts in breaking the codes used by Soviet intelligence operatives during World War II, an effort that has become known to history as the Venona decrypts ( Document 1).   But once the Moscow realized that the U.S. and Britain were breaking its codes, in the fall of 1948 the Soviets changed all their ciphers. It took three decades before Washington regained a degree of access to high-level Soviet communications. Nevertheless, the agency’s SIGINT efforts proved to be critically important in helping the U.S. intelligence community monitor the rising threat posed by the Soviet Intercontinental Ballistic Missile (ICBM) program ( Document 17), which was the highest priority collection target during the late 1950s and early 1960s. Another high priority operation during the 1960s was NSA’s collection efforts aimed at the Soviet’s manned and unmanned space program ( Document 21). NSA was also involved, albeit indirectly, in the CIA’s famous Berlin Tunnel Operation, which was not discovered by the Soviets until 1956 ( Document 14).

NSA’s performance during the Vietnam War (1961-1975) was marked by a number of notable successes on the battlefield ( Document 15, Document 19), which were credited with being instrumental in helping U.S. forces in Southeast Asia win a series of battles against the North Vietnamese and Viet Cong forces. (Note 3) But NSA was also involved in a number of costly intelligence failures in Vietnam, such as the agency’s mishandling of intelligence during the Tonkin Gulf Crisis of August 1964 ( Document 9), which led to America’s full-fledged entry into the Vietnam War.

By the early 1970s, with U.S. involvement in the Vietnam War winding down to its tragic climax, NSA found that it had to dramatically revamp itself in order to handle a series of new technological advances in the computer and telecommunications fields, which were threatening to severely impair the agency’s ability to perform its mission ( Document 16). NSA also had to restructure its SIGINT collection efforts to encompass a host of new transnational targets, such as economic intelligence and monitoring the rising threat posed by international terrorism ( Document 10).

Throughout its history, a series of catastrophic events have damaged NSA’s codebreaking efforts. One of the worst of these events occurred in January 1968, when North Korean naval forces seized the U.S. Navy spy ship USS Pueblo while it was engaged in an intelligence collection mission off the east coast of North Korea. Within weeks of the loss of the ship, many of NSA’s top intelligence sources in North Korea and elsewhere in the Far East suddenly went off the air ( Document 3). After the seizure of the Pueblo, the North Korean government published an expose on the ship’s mission, with a few pages reproducing some of the captured documents ( Document 24).

Documents
Note: The following documents are in PDF format.
You will need to download and install the free Adobe Acrobat Reader to view.

Document 1: Extracts from Robert Louis Benson and Cecil J. Phillips, History of Venona (Ft. George G. Meade: Center for Cryptologic History, 1995), Top Secret Umbra, NSA FOIA.

Document 2: Thomas L. Burns, The Origins of the National Security Agency U.S. Cryptologic History, Series V, Vol. 1 (Ft. George G. Meade: Center for Cryptologic History, 1990), Top Secret, NSA FOIA.

Document 3: Robert E. Newton, The Capture of the USS Pueblo and Its Effect on SIGINT Operations, U.S. Cryptologic History, Special Series, Crisis Collection, Vol. 7 (Ft. George G. Meade: Center for Cryptologic History, 1992), Top Secret Umbra, NSA FOIA.

Document 4: David A. Hatch, Presidential Transition 2001: NSA Briefs a New Administration (Ft. George G. Meade: Center for Cryptologic History, 2004), Top Secret/COMINT/X1, NSA FOIA.

Document 5: National Cryptologic School, On Watch: Profiles From the National Security Agency’s Past 40 Years (Ft. George G. Meade: National Cryptologic School Press, September 1986), Top Secret Umbra, NSA FOIA.

Document 6: Frank B. Rowlett, Recollections of Work on Russian, February 11, 1965, Top Secret Dinar, NSA FOIA.

Document 7: Unknown Author, SIGINT in the Defense of the Pusan Perimeter: Korea 1950, date unknown, Unclassified with Top Secret Copse attachment, NSA FOIA.

Document 8: Jacob Gurin and deleted, “Ralph J. Canine: Profile of an Unforgettable Personality,” Cryptologic Spectrum, Vol. 1, No. 1, Fall 1969, For Official Use Only, NSA FOIA.

Document 9: Robert J. Hanyok, “Skunks, Bogies, Silent Hounds and the Flying Fish: SIGINT and the Gulf of Tonkin Mystery, 2-4 August 1964, Cryptologic Quarterly, Vol. 19, No. 4/Vol. 20, No. 1, Winter 2000/Spring 2001, Top Secret/COMINT/X1, NSA FOIA.

Document 10: Robert J. Hanyok, “The First Round: NSA’s Effort Against International Terrorism in the 1970s,” Cryptologic Almanac, 50th Anniversary Series, November-December 2002, Top Secret/COMINT/X1, NSA FOIA.

Document 11: David A. Hatch, “Quis Custodiet Ipsos Custodes?,” Cryptologic Almanac, 50th Anniversary Series, February 2003, Top Secret/COMINT/X1, NSA FOIA.

Document 12: David A. Hatch, “DDE & NSA: An Introductory Survey,” Cryptologic Quarterly, Vol. 23, Nos. 1-2, Spring/Summer 2004, Top Secret/COMINT/X1, NSA FOIA.

Document 13: George F. Howe, “The Early History of NSA,” Cryptologic Spectrum, Vol. 4, No. 2, Spring 1974, Secret, NSA FOIA.

Document 14: William T. Kvetkas, “The Last Days of Enigma,” Cryptologic Almanac, 50th Anniversary Series, March-April 2002, Top Secret/COMINT/X1, NSA FOIA.

Document 15: Major James S. Rayburn, USMC, “Direct Support During Operation DEWEY CANYON,” Cryptologic Spectrum, Vol. 11, No. 3, Summer 1981, Secret Spoke, NSA FOIA.

Document 16: Richard C. Raymond, “Challenge to Sigint: Change or Die,” Cryptologic Spectrum, Vol. 1, No. 13, Fall 1969, For Official Use Only, NSA FOIA.

Document 17: Unknown author, “Early History of the Soviet Missile Program (1945-1953), Cryptologic Spectrum, Vol. 5, No. 3, Summer 1975, Secret, NSA FOIA.

Document 18: Unknown Author, “From Chaos Born: General Canine’s First Charge to the NSA Workforce,” Cryptologic Quarterly, Vol. 6, No. 2, Summer 1987.

Document 19: Unknown author, “Operation Starlight: A Sigint Success Story,” Cryptologic Spectrum, Vol. 1, No. 34, Fall 1971, Secret, NSA FOIA.

Document 20: Unknown author, “Stonehouse: First U.S. Collection of deleted [Space] Signals,” Cryptologic Spectrum, Vol. 5, No. 4, Fall 1975, Secret, NSA FOIA.

Document 21: Guy R. Vanderpool, “COMINT and the PRC Intervention in the Korean War,” Cryptologic Quarterly, Vol. 15, No. 2, Summer 1996, Top Secret Umbra, NSA FOIA.

Document 22: Thomas R. Johnson, “American Cryptology During the Korean War: Opening the Door a Crack,” Studies in Intelligence, Vol. 45, No. 3, 2001, Unclassified, CIA FOIA.

Document 23: Richard A. “Dick” Chun, A Bit on the Korean COMINT Effort, 1971, Secret Spoke, NSA FOIA.

Document 24: Les Actes D’Agression Declares de L’Imperialisme U.S. Contre Le Peuple Coreen: Les matériaux concernant les actes d’agression du <Pueblo>, vaisseau-espion de l’armëe d’agression de l’imperialisme americain avant fait profondement intrusion dans les eaux territoriales de la Républicque Populaire Démocratique de Coree [Acts of Aggression Declared by American Imperialism Against the Korean People: Materials concerning the aggressive acts of American imperialism’s armed spy ship, Pueblo, and its deep intrusion in the territorial waters of the Democratic Peoples Republic of Korea ]  (Pyongyang: Editions en Langues Etrangeres, 1968) [Excerpts]

Notes

1. This period in NSA’s history is well covered in Thomas R. Johnson, American Cryptology during the Cold War, 1945-1989: Book I: The Struggle for Centralization, 1945-1960 (National Security Agency: Center for Cryptologic History, 1995), Top Secret Umbra, http://www.gwu.edu/~nsarchiv/NSAEBB/NSAEBB260/index.htm

2. A good general overview of the role played by SIGINT in the Korean War can be found in David A. Hatch and Robert Louis Benson, The Korean War: The SIGINT Background, U.S. Cryptologic History, Series V, Vol. 3 (Ft. George G. Meade: Center for Cryptologic History, 2000), Unclassified, http://www.nsa.gov/public_info/declass/korean_war/sigint_bg.shtml. For an example of a major SIGINT success during the Korean War, see Jill E. Frahm , So Power Can Be Brought Into Play: SIGINT and the Pusan Perimeter, U.S. Cryptologic History, Series V, Vol. 4 (Ft. George G. Meade: Center for Cryptologic History, 2000), Unclassified, http://www.nsa.gov/about/cryptologic_heritage/center_crypt_history/publications/sigint_and_pusan_perimeter.shtml.

3. The most detailed single-volume account of NSA’s successes and failures during the Vietnam War can be found in Robert J. Hanyok, Spartans In Darkness: American SIGINT and the Indochina War, 1945-1975 U.S. Cryptologic History, Series VI, Vol. 7 (Ft. George G. Meade: Center for Cryptologic History, 2002), Top Secret/COMINT/X1, http://www.nsa.gov/public_info/_files/cryptologic_histories/spartans_in_darkness.pdf.