Journalism in the Public Interest


What NSA Transparency Looks Like

Disclosures by the spy agency about violations of privacy rules have been almost entirely redacted.

President Obama spoke about NSA transparency at a news conference on Aug. 9. (Chip Somodevilla/Getty)

Last week, the Washington Post published an internal audit finding the NSA had violated privacy rules thousands of times in recent years.

In response, the spy agency held a rare conference call for the press maintaining that the violations are “not willful” and “not malicious.”

It’s difficult to fully evaluate the NSA’s track record, since the agency has been so tight-lipped on the topic.

What information about rule violations has the agency itself released? Take a look:



That is the publicly released version of a semiannual report from the administration to Congress describing NSA violations of rules surrounding the FISA Amendments Act. The act is one of the key laws governing NSA surveillance, including now-famous programs like Prism.

As an oversight measure, the law requires the attorney general to submit semiannual reports to the congressional intelligence and judiciary committees.

The section with the redactions above is titled “Statistical Data Relating to Compliance Incidents.”

One of the only unredacted portions reads, “The value of statistical information in assessing compliance in situations such as this is unclear. A single incident, for example, may have broad ramifications. Multiple incidents may increase the incident count, but may be deemed of very limited significance.”

The document, dated May 2010, was released after the ACLU filed a freedom of information lawsuit.  

As the Post noted, members of Congress can read the unredacted version of the semiannual reports, but only in a special secure room. They cannot take notes or publicly discuss what they read.

For more on the NSA, see our story on how the agency says it can’t search its own emails, and what we know about the agency’s tapping of Internet cables

UPDATE 8/22/13: The Obama Administration has now declassified the most recent version of the semiannual report to Congress and posted it online. The document includes some information about rates of "compliance incidents" but is also heavily redacted.

Is it possible to get this document as a PDF? It cries out to be used in art.

What Crap!  Why’d they waste the paper?  Our government does exactly as they please and we’d better shut up and like it!  I am very close to a “belly full”!

The huge number of black-outs in these govt docs instantly reminded me of the heavily redacted transcripts of the Watergate tapes when they were finally released, except those were easy to figure out because they were all four-letter words from Nixon’s office.  We were goofy teenagers and gleefully played fill-in-the-black-outs from the NY Times articles.  Kinda scary when we get nostalgic for the simpler scandals of olden days…

a lil levity from Robin Williams in Good Morning Vietnam about Intellegence

I’m always impressed when I see a document that has been redacted into a game of Scrabble.

I suspect that those who worry about the nation’s secrets on behalf of America’s democracy much preferred the old days when documents “generally” faded as a whole through the various levels of classification before becoming “Unclassified”.  I.e., anything that today would require redaction was sufficient to maintain a document’s classified status.

I hate to think of the taxpayer money we spend on document reviews and line-by-line redacting to meet FOI requests - and I further believe that redaction is done “to excess” only because “Rather safe than sorry.” is how you act if you’re tasked with the nation’s security.

And finally, I would remind one and all that politicians - elected officials - drive our nation’s security apparatus.  Recall, for example, that it was the Republicans who created Homeland Security, the TSA, and “the Patriot Act”.

(well, not quite “finally”...America’s peoples don’t seem to grasp the fact that anything a politician/politicians does/do that they feel needs to be associated with “patriotism” to make it palatable is probably going to screw the American people and our democracy at some point down the road.)

My last paragraph above?  Re how dangerous anything a politician does that they feel they should associate with “patriotism” is?

That was how you became an “enemy of the state” in the former Soviet Union…by not conforming to some arbitrary and highly fluid definition of “patriot”.

pathetic !

This is a perfect example of lazy reporting. Naturally, the document released to the public is heavily redacted. it was not meant for public scrutiny. It is a report intended for the DoJ as required by law. It is proof that real oversight of the NSA’s surveillance programs are taking place, not that the agency is trying to hide what they’re doing. That’s why all of these foolish conspiracy theories are popping up like new mushrooms.

Talk with your IT specialists and ask them how they would write an algorithm that could identify an American on the “other end” of a communication with a know terrorist node? If they’re honest, they will tell you that only God could do that before content was captured under warrant. No computer can do this because an IP address, for example, is not an absolute guarantee of location, nor whether or not the American who is inadvertently embroiled in the search is traveling, hence these predictable incidents. This shouldn’t require much thought if you want to be fair and objective.

Your hit piece fails to take into account the this document is proof that the NSA has in place many different layers of “oversight”. For you to characterize this as an attempt to hide something is simply misleading to any rational person, but it will get eyes and sell ads and, maybe, that’s your goal, rather than an honest debate.

On a much larger scale, do you even know whether this report could be used by a terrorist in their jigsaw puzzle as the piece that confirms something far more dangerous about our sources and methods? Only God could, so don’t flatter yourself. If you study the Mosaic Theory, you will come away with, at least, an understanding of why our government agencies err on less rather than more. Using this fact is simply a disingenuous attempt to woo readers, IMHO.

Cheney and his ilk used a similar tactic in the Plame affair. Conservatives and the right wing Echo Chamber continuously stated that the CIA would confirm it, if Valerie was covert. They said this, knowing full well that the CIA would be incapable of responding to such a dumbass idea, but the media ran with it for months. And, guess what we learned during Scooter Libby’s prosecution? She was a covert agent. Duh!

I would really appreciate it, if propublica would think before they publish an ill conceived notion that government is too secret, (because they actually want avoid another 911). The premise that our military intelligence could be opened up to more transparency is truly an idiotic opinion. Al Qaeda would set up shop in DC, if we turn off our electronic surveillance and publish that fact.

I do agree with judges who complain that the Mosaic Theory is so broad as to not even allow reasoned discussion, but to completely disregard what this report does at it’s heart and only focus on the redactions is simply mean-spirited.

clarence swinney

Aug. 20, 2013, 5:14 p.m.

10,000 LIES IN 4000 HOURS
A few years ago, Rush Limbaugh said “that media watch group FAIR(Fairness and accuracy in reporting) edited 4000 hours and found only 45 little errors.”
I called FAIR and got the study. It was over 2 Lies per hour or over 10,000 Lies.
You can label Rush As the 10,000 man.

SeattleGuy ..................

‘conspiracy theory’  ‘right wing echo chamber’

As soon as you hear the the talking point type phrases like this, well :/

Tonight on Goofball with Chris Mathews: Are right wing conspiracy extremist liberatarian gun psychos infiltrating your town!

Steven Sagala

Aug. 20, 2013, 6:21 p.m.

This looks like the Kennedy assassination materials. We should take their redactors away.

@petermcnab- Poor deflection and no substance.

If you witnessed how Cheney planted false stories derived from his parallel intelligence (or was it parallel universe?) with Judith Miller in the lead up to the Iraq war and, then, scheduled himself on all the Sunday talk shows to follow her column and confirm “what the New York Times had written about the aluminum tubes this morning”, as though it wasn’t taken right out of his mouth and published, well you don’t know yellow journalism. The right wing does it best.

I guess that means you agree with all I posted, then, other than your tribe are the good guys and mine are the bad guys, right?

John Henry Bicycle Lucas

Aug. 21, 2013, 7:55 a.m.

Conspiracy theorist? If someone breaks the law in colusion with another, it is a conspiracy. If you think our government has not broken the highest law of the land, please explain it to me.

So much redacted text ... wait a minute, they redacted the rules section of what they’re not supposed to be doing ... no, that can’t be right ~ if the *laws* are a secret, the government is criminal.

Here’s the thing:  Every few years, the CIA takes some time to purge their files.  The way intelligence work operates, everything the operation touches becomes secure by association.  If you found peripheral information in a New York Times article, you classify the entire paper, because otherwise your interest in it becomes knowable.

The upshot is that somewhere in the neighborhood of 99% of all CIA files, at any given time, is usually garbage that nobody cares about, and they need to throw it away.

I explain this to outline a simple case that everybody at the NSA should be fired.

This document?  It speaks volumes.  It releases information that the NSA shouldn’t want released.  It does so not in what little it says, but in what it avoids saying.

If everything is above-board, then nothing but identifying information should be redacted.  That so much needs redaction sends a clear message that they know they’re doing illegal things.  And for a spy agency, specializing in signals intelligence, to make such a primitive blunder tells me that they’re very bad at their jobs.

Add that incompetence to their complete disinterest in both the Constitution and Posse Comitatus, and it’s clear they shouldn’t be working for us.

(Side note:  If you told me about any other country where an intelligence agency had violated the nation’s foundational documents and defied the civilian government—Bush dismissed the Total Information Awareness project, don’t forget—to place the entire country under some sort of surveillance, we’d call the result a military coup and a puppet government.  I don’t think that’s the case, as evidenced by the increasing crapstorm surrounding these programs, but it’s food for thought while officials call Manning and Snowden traitors.)

Plus, while I find lying distasteful and offensive, I expect a certain amount of it from high-ranking officials, because you don’t get ahead in a corrupt organization by being the beacon of truth.  But c’mon, at least do a decent job of it!

This has become strictly amateur hour, the sort of “plans for world domination” I used to kick around with friends at college on slow days in the computer lab.  Wouldn’t surprise me if the entire system is built on “nmap,” at this point.

“Everybody should be fired” because the CIA doesn’t spend the time to declassify everything they can ASAP?

It’s hard to respond to such a foolish notion. As I said the other day. just because A-Rod cheated, we’re not going to shut down all of MLB. That would be just as irresponsible.

With limited resources, every government agency must use their’s wisely. This is no reason to suspect anything if you have some predisposition.

You state that the government has violated the Constitution. How? I don;t see that. Did you read the latest rulings against the ACLU and Amnesty? The could not establish standing because they could not show harm. They were just speculating that maybe their client would come under surveillance.

As someone once, said, “the only exercise some people get is jumping to conclusions”

When you have any evidence of violations of the Constitution or just law breaking like Snowden, call it to the attention of authorities. I am with you that we need to root out any and all law breakers.

The Atlantic article is a case in point. It simply shows that the NSA has oversight in place. You need to read the article again and try to focus on where it came from (the NSA), who it was intended for (the DoJ), and if your paranoid feelings force you to,  why there are redactions (not intended for public consumption). There is nothing here, but Greewaldians are trying desperately to make a case that is simply untrue.

Get back to me with any evidence of wrongdoing. Nothing, huh?

@johnhenry- Actually it is up to you to prove the government broke the law in our country. This is not the UK. Having said that, I will for your benefit.

Since you haven’t even stated what law was supposedly broken, I have to guess. None were, of course.

Many feel their phone calls are being listened to without a warrant-wrong
Many are surprised to know that the metadata is not protected by some privacy right-they’re not. You publish them to the public domain.
Some misunderstand how the NSA’s surveillance work-it’s legal and constitutional.

I’ve tried, but I need to know where you see laws being broken.

SeattleGuy, if you can’t think of a way to respond to me, here’s a thought:  don’t.

I don’t see you contributing anything to the discussion other than ignorance of everything that’s happened in the past two months, so I certainly don’t have much interest in communicating with you.

However, in the naive interest in educating you:

- The First Amendment to the United States Constitution (and legal decisions since) ban government from interfering with free expression.  Lavabit and Groklaw have both shut down rather than expose their users, hence the chilling effects interfere with free expression.  Was it unintentional?  Tough, they’re not allowed to do it, even accidentally.
- The Third Amendment protects the civilian population from housing defense operatives.  Their crap is camped in the back rooms of telecommunications companies.
- The Fourth Amendment protects against warrantless search and seizure.  They seize data and they search it.  They don’t get warrants except after they’ve found what they want.
- The Fifth Amendment protects against self-incrimination, which is difficult to do when the military assumes that your connection to someone (and his connection to someone else) is evidence of potential wrongdoing.
- The Sixth Amendment provides for a jury, defense, and confrontation, none of which is available in a world of secret courts and accusers who are computer programs.
- The Ninth Amendment states that rights are universal even where they’re not listed; most people not working for the NSA seem to agree that privacy is one.

And no, you can’t prove that anybody was harmed because the information is classified.  That was the Bush Administration’s theory on warrantless wiretapping, that you couldn’t have any evidence against them, and if you did have evidence, it was highly classified and it’d be treasonous to present said evidence, so therefore, nobody could sue and everything was clearly legal.

Oh, right.  And there is evidence.

The DEA and IRS are both using the Fruit of the Poisonous Tree, as they say, which they’re getting from these very programs.  And they have directions to cover that up, else the evidence—gathered by the military—gets thrown out of court.

But I assume you’ll dismiss that as civil-libertarian wackiness and paranoia, since that seems to be your only comment.

If there’s nothing wrong with what the NSA is doing, then there’s no reason to do it behind closed doors or obscure it through wordplay and a poor understanding of statistics.  That’s what they all tell me, no?  That if I’m not doing anything wrong, then I have nothing to hide.

I’ll regret encouraging you, but what have you read on the topic other than official statements that everything’s fine, nothing to see, please keep paying the bills, that leads you to believe this is legitimate?

Putting it another way, if Snowden got out of the country with so many secrets, how could their oversight system be remotely useful?

Do you also believe that everything Manning sent to Wikileaks is a lie or grossly misinterpreted?

Steven Sagala

Aug. 21, 2013, 6:20 p.m.

I just would like to say that I am not a part of any “tribe”. I just happen to be an old fogy who tends to side with H.L. Mencken, Dwight Eisenhower, and yes - the arch-criminal Daniel Ellsberg - where the Bill of Rights is involved. I have seen and read a lot more than some might like, and I am suspicious of the course that the government has been taking since the early ‘90s. Too many unanswered questions. Not saying any more.

John Henry Bicycle Lucas

Aug. 21, 2013, 7:11 p.m.

SeattleGuy, they broke the laws by their own admission. Ever hear of the 4th Ammendment?

Secret interpretation of laws? Secret courts that cannot legally release any order for public scrutiny?

Congressmen on intel commities that tell us how disturbed we would be to know how much we are spied on?

I guess you have nothing to worry about because you aren’t doing anything wrong…

@johnhenrybicyclelucas- It is absolutely unbelievable that anyone doesn’t understand the reasoning behind the FAA at this point. It is willful ignorance.

“they broke the laws by their own admission. Ever hear of the 4th Amendment?”

What could you possibly mean by this idiotic statement? Since no laws were broken except by Snowden and the FAA complies with the Constitution, I can’t fathom what you are talking about. Are you of the belief that Congress, the President and the SCOTUS are in a big conspiracy to undermine the Constitution? All 3 branches have stood behind the FAA and reauthorized it recently, so if you think you’re smarter than everybody, why don’t you try to explain yourself. Oh, wait, I know why you don’t. You have nothing to back up what you say. You have never read the FISA law and you have no idea what’s in it, right?

The FISA court is there for those who have security clearance which you don’t. You are, obviously, not qualified to review sensitive government documents. So what?

Our national security is not up for a popularity vote.

John Henry Bicycle Lucas

Aug. 21, 2013, 9:02 p.m.

Many are in agreement with me, including Judge Napalitano. Do you know that you are 8 times more likely to be killed by a law enforcement officer than a terrorist? Do you know in the early 1970’s there were roughly 125 cops being killed in the line of duty then and now about the same number? There are many more cops now than there were then, and they are all wearing bullet resistant gear. Many more people are being executed in the street for petty reasons.

We are entering the phase of an all out police state, in some areas. This includes mass surveilance of the population. The Bill of Rights was put in place to protect the people FROM the government.

The Bill of Rights has been totally discarded so it seems.

No, I don’t have any type of clearance to read any top secret anything. I do have critical thinking skills, however. Don’t piss down my back and tell me it’s rain.

You can drink all the koolaide you want. I will not be drinking it.

Trausti Hraunfjörð

Aug. 21, 2013, 10:15 p.m.

Quote from page 14 of the pdf:

“FBI is authorized to acquire foreign intelligence information”

Wow… that really puts some legitimacy into it… don’t you think?

Let me bring you another quote… this time from a HIGHER authority than anything the US government can ever dish up on their own: The Universal Declaration of Human Rights… article 12:

“No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks.”

This really settles the issue I have with the universal permission the US thugs give their armed wing of FBI to spy on me and my people.


The US has NO right what-so-ever, to violate MY human rights.  I am NOT an American citizen.  Have never been and thankfully never will become either.  This means that the US has absolutely NO authority to stick their nose in my underwear to have a smelling session.  I know they WANT to, but that does not mean they have my permission to do it!  They have already done it several times, and I even had my connection to the internet blocked by a US agency a few years back.  The block was put on the servers of my ISP/phone company, and when THEY discovered it, they were shocked. It was totally and absolutely illegal, but the US doesn’t give a shit about the law, they just write their own permissions, as if they were some universal god who can do whatever it wants.

Criminal gangs do the same thing… they write their OWN “laws” and abide by those.  They write up people they need to kill, and their armed wings do the dirty work.  Same as the US government does. 

US law, made by the abuse enablers, does not in any way or form supersede international law.  That is a fact NO ONE can dispute.  Not even the hard-boiled religious right wing extremists who are blinded by their own fascism and corporatism.  International law is the highest law there is on this planet of ours, and that was made crystal clear during the Nuremberg trials after World War II.  It may be “inconvenient” for America to admit it, but that is how it is.  If America does not want to abide by international law, it is more than welcome to blow it self into space, and away from this planet.

Thank you!

@johnhenry- I should really just let you keep rambling about this paranoid nonsense. You’re making my case.

Judge Napolitano agrees with what, exactly? Cite your source and stop ignoring the fact that at this point, you have only shown fear. You have not given me one example of the US being a police state. I don’t know where you live, but I can go freely everywhere and am not bothered, much less shot at by the local police. So what?

This is all off topic. We were discussing the NSA surveillance programs, remember? Let’s stick to that, otherwise you might start talking about Area 51.

Last time I looked, the Bill of Rights was still in force. Next?

If you’re not using mind altering drugs, I wonder what is causing your delusions? Any ideas?

John Henry Bicycle Lucas

Aug. 22, 2013, 6:49 a.m.

Go ahead, believe the government is going to take care of all of your problems, it is all benevolent, all good.

I have not insulted you…yet. I’m not delusional, I’m awake.

John Henry Bicycle Lucas

Aug. 22, 2013, 6:57 a.m.

How many sources do you want? The truth is not in the NSA.

@Trausti Hraunfjörð- who are “your people”. If you are American citizens, you enjoy all the protections the Constitution provide. If not, you don’t. Is that really too hard to comprehend?

If you have been under FBI surveillance, there must be a reason. Do you have any thoughts on this? The CIA is not the subject of this thread. The NSA is and they’re separate agencies with different charters.

@johnhenry- Couldn’t do any better that Russia Today for a reputable source for the propaganda you read?

I seem to recall they are harboring the fugitive that is at the heart of this issue. You will notice if you read the whole thing, though, that other than recycled spin, there is absolutely no evidence of government wrongdoing and I would imagine if anyone truly had that evidence, it would be Putin and his KGB.

Why don’t I make it even easier for you. Please spell out what you think is illegal or unconstitutional about the NSA or FBI’s surveillance programs. Take your time and give me some meat, not another bit of spin.

@Trausti Hraunfjörð- I think I found you on FB. Are you still an Icelandic citizen? If so, as I said you have no rights in the US. Your homeland is beautiful, though. I have been there twice.

Next mistake. You and your source say that the “FBI’ blah blah blah. The FBI does not do international terrorism. I hate break that to you, so it’s hard to respond to that false notion.

I agree that International laws take precedence over any nation’s, but so what? All nations maintain their sovereignty by force alone. All nations. And all of them use spying for military advantage. Every last one of them. There are no exceptions. In doing so, they routinely violate the laws of other countries. That is not even a major consideration.

You should ask your programmers if the NSA needs content to do traffic analysis on terrorist networks. They will tell you that you’re getting all paranoid about nothing unless you make contact with known terrorists. Then, you will come to the attention of the electronic surveillance of several advanced Western nations. This should not come as a surprise to anyone who is a student of history as you claim.

Now, if you and your friends have come up on the NSA’s radar, I would check who you’ve been collaborating with lately. Do you live in Peru? Does the name Assange ring a bell? Google his picture and take a look. He traffics in state secrets and is, obviously, radioactive for all those who come in contact with him.

By the way, when you figure a way to eliminate the need for spying, let me know. I want to get the patent on that one and I promise to share it with you.

Kind of pointless to get upset about the NSA’s activities when the Supreme Court has already ruled that the 4th Amendment can be circumvented if a law enforcement officer is willing to merely claim to have heard a noise that might be the result of an attempt to destroy evidence.

The burden of proof - being able to substantiate a claimed smell or noise - isn’t even required from law enforcement because reality makes smells and sounds fleeting sensory perceptions.

Remember that there are very many more law enforcement officers than there are NSA analysts…heck, across the Federal, state, county, and municipal levels there are more law enforcement agencies than there are NSA analysts.

While the NSA may gets the media’s focus, it was politicians who selected “conservative” SCOTUS justices that would be willing to erode the rights of the American people and commit bogus acts like making people out of corporations…it was politicians who created and implemented “the Patriot Act”...

Republican politicians, to be specific.

If you want to protect your rights as an American citizen, then stop voting for the Republicans.

@ibsteve2u- Excellent post. And I can verify that this happens frequently. I was at a friends apartment yesterday. I heard a loud noise in the hallway. I went out to look and 3 policemen had a suspect seated on the floor at the door of his apartment. The door had been obliterated and he was taken off for booking after a lengthy interview.

I do like your conclusion. There is nothing that would make me happier and the country prosper more quickly than flushing out the haters and obstructionist crowd, many of whom make up these foolish rants against anything that makes up “government”. Although, from my experience, there seem to be almost an equal split on this particular issue between Rs and Ds. It makes my very uncomfortable to actually see a conservative’s point of view. Normally, it makes no sense at all.

My question is this; if corporations are people why don’t they pay taxes at the individual rate?

Rather than playing party politics, one could look at how individuals actually vote.  There was, after all, a public vote on Amash’s amendment to de-fund the NSA’s surveillance program.

Democrats have an edge, here, but one may wish to actually know if their pseudo-populist representative cares about their rights.  For example, I’m rather shocked to see Rep. Kaptur on the “Nay” list, after her arguments against the bank bailouts.  I’m less surprised to see Rep. Israel, who generally believes that whatever the government does is A-OK.

Incidentally, in the documents released on Tuesday night—which I assume everybody here read, since they have an interest, but I’ll dig up a URL if anybody can’t and thinks the EFF’s blog will eat them or something—showed that FISC believes it was lied to repeatedly by the NSA and that the program is unconstitutional.  Weirdly, though, they don’t believe it violates the Fourth Amendment, probably because judges no longer know what it says.

@john- I actually don’t think this splits along classic party lines. I am progressive and most of the critics I have encountered are Democrats that are preoccupied with what they perceive as an intrusion into their personal privacy. They have none, but they still think they do and are vulnerable to the lazy reporting on the subject.

On the other side of the political spectrum, I find many Republicans who just want to complain because they still can’t accept a black president. When I probe their reasoning, they make little sense. Usually, they make noises about fascism or the Stasi or some other such nonsense.

No, what I find with this issue is that I am standing with many Republicans who understand that terrorism is real, must be contained, and know that even though Bush got ahead of the law in 2006, the basic use of technology is needed. Nothing else can replace it at this time, so they crafted a balance between privacy and national security that’s just about right.

I read one EFF hit piece. It was written by a woman with the WSJ who as often comments on silicone implants. Seriously! As with most of the right-wing criticism, it completely misses that these ‘inadvertent’ incidents are normal. They cannot be avoided. There is absolutely no evidence that any of them was for nefarious purposes. All of which mean, there is no government wrongdoing.

I understand the WSJ’s bias. It is another Murdoch opinion outlet that peddles yellow journalism. It was once a great newspaper. That ship has sailed. Their writer should stick with soft porn or something. She is totally unqualified to report on big data and the intricacies of culling terrorists from publicly available information.

The reason the NSA knows their programs don’t violate the 4th amendment are simple.They are chartered with catching international terrorists and the last time I looked, the Constitution does not convey any rights to foreign terrorists. Did I miss that amendment or is it in your imagination?

Among the Republicans - especially the extremely wealthy Republicans - the split is on wealth possessed.

You have to remember that they use foreign banking systems in other nations to hide their wealth; likewise, in order to manipulate the financial systems of other and their own country

a certain amount of cooperation/coordination with other bankers/brokers/HNWIs of the same ilk offshore is required.

As much of terrorism’s funding is laundered through on/offshore banking systems and/or moved from country to country,

those conversations/transactions are validly of interest to the security infrastructure of the United States of America.

So what I would expect to see is the media of the United States of America being used to flog the American people into demanding rule changes vis–à–vis who the NSA can monitor/record…

Which means Congressional action, which - judging by precedents set over the last 40 years - means the American people will still be monitored/recorded but financial interests and wealthy political donors will have bought themselves a way of circumventing that monitoring/recording from Congress.

Regarding 2,776 “incidents” from 4/11-3/12:

This is an inherently inaccurate number. First of all, we’re dealing with a number of queries at roughly 20,000,000 a month. So we can safely conclude a total number of queries in a year at roughly 240,000,000. Two hundred and forty MILLION.

Second, the list of errors is a tally based upon all the different ways in which an error might be spotted. One or perhaps a few of these different numbers added into 2,776 are automatically detected by the system itself. So whatever the system is designed to detect as a problem, covering the entirety of the system. These numbers are added to audits, self-reported mistakes/errors, reviews, analysis, all the other myriad ways in which a violation might be discovered.

Combining those numbers and presenting them as a finite number of violations is incredibly dishonest. What happens in such a list when the vast majority of errors are detected within the system? Then the vast majority of overall incidents are whatever the system is designed to detect: inadvertent data collection, mistakes made in the tasking process, typos, and so on. Mostly minor unavoidable errors that when misrepresented in such a way create an illusion of an effective auditing and quality control process that finds and corrects the problems people worry about, which appear to be relatively harmless the majority of the time.

In order to have any idea whatsoever the total number of violations you would need a statistically significant random sampling to audit, and then a projection made based upon that. Finding 83 errors through “Auditing” and then lumping that number in with everything else and presenting that total as the true number of violations is blatantly dishonest.

Now what are some commonly used levels of statistical significance to arrive at such a projection that would present a genuine estimate of the true number of violations: 10%, 5%, 1%, 0.5%, 0.1%

If 83 is from an audit of 10% then we have 830 estimated violations.

If 5%, 1660
1%, 8,300
.5%, 16,600
.1%, 83,000

What percentage do you suppose they would audit from 240,000,000, assuming of course they’re even attempting to perform a minimum level of “quality assurance” on surveillance programs of unimaginable scope? There is absolutely no reason to believe in this 2,776 number as anything meaningful, and any broad conclusions drawn from it to the programs overall are so deeply flawed they can be dismissed out of hand.

Given - and I quote -

an internal audit finding the NSA had violated privacy rules thousands of times in recent years.

methinks someone doth protest too much.  (Kudos to Shakespeare.)

Are you saying that it’s somehow an effective measure since it’s entirely internal?

I’m saying that I have reason to believe that NSA personnel aren’t stupid enough to permit an inaccurate audit to reach the media.

(In fact, they probably double- and triple-checked the results of said audit.)

This was an internal “audit” that was never intended to reach the media, and at most selectively used to satisfy minimal oversight demands. If you would care to read what I wrote and illustrate where I am incorrect, great. I know I’m not infallible. If instead you wish to dismiss it outright and place faith in your personal assumptions, then there is no discussion to be had between us.

You’re assuming - or attempting to portray - insider knowledge about the nature of the audit…your argument is dependent upon antique paper-based auditing where the “paperwork” must be sampled rather than analyzed in bulk.

I.e., you’re ruling out database queries of the sort

SELECT COUNT(*) FROM [table of all audits] WHERE [Audit.Justification] IS NOT IN [list of valid justifications] OR (([Audit.IP_Origination] IN (SELECT Country.IP FROM [table of IP address by nations] WHERE [] = ‘US’) AND ([Audit.IP_Destination] IN (SELECT Country.IP FROM [table of IP address by nations] WHERE [] = ‘US’))

which can yield precise results.

Hence, by the way, the justification for the new Utah data center:  Data that you cannot quickly retrieve and analyze is - depending upon which side of the action divider you’re operating on - useless.

I.e., once intel becomes stale, your ability to be preemptive is gone; you can only be reactive…which may enable prosecutions and retaliatory acts, but doesn’t save American lives/protect the United States of America.

No, I’m simply portraying what an effective audit could be, while pointing out the deficiencies present in their audit. How can your database query example be consistent with what we’ve learned? Why would “[Audit.Justification] IS NOT IN [list of valid justifications]”, when they use a pull down tab with a list of justifications? Why would an unjustified option be present? On some of these they type out that sort of information, how do you propose simple database queries to precisely determine that?

Why in those training slides is the idea of self-reporting errors and mistakes so important if those errors and mistakes are so easily determined? For one thing, we’re dealing with a multitude of different programs, with varying modes of access to information.

What about the LOVEINT revelation that’s making the rounds?
“The LOVEINT violations involved overseas communications, officials said, such as spying on a partner or spouse. In each instance, the employee was punished either with an administrative action or termination.

Most of the incidents, officials said, were self-reported. Such admissions can arise, for example, when an employee takes a polygraph tests as part of a renewal of a security clearance.”

This after they’ve claimed intentional abuses never occurred.

Diane Feinstein: She said “in most instances” the violations didn’t involve an American’s personal information.

Or an anonymous source within the intelligence community saying this: More than two months after documents leaked by former contractor Edward Snowden first began appearing in the news media, the National Security Agency still doesn’t know the full extent of what he took, according to intelligence community sources, and is “overwhelmed” trying to assess the damage.

Most evidence overwhelmingly points to them having a dangerously insufficient oversight regime and entirely inadequate auditing capabilities. The evidence to the contrary is severely lacking.

Via an anonymous source:

Sigh…I suppose that is what I get for attempting to provide a simplistic example from which I anticipated that other queries would be logically derived.

While you’re eager to “prove” your point, to do so you must assume the absence of a - for the sake of simplicity - master “work order” that is maintained in a separate table.  That “work order”, one would anticipate, would automatically be provided with a unique identifying (one can assume alphanumeric) “work order number” if and only if the work is “signed off” on…that is authorized.

Then another table would hold the actual work done; from that table you would retrieve the actual originating and/or destination IP address/phone number used for data capture.

Then auditing becomes a simple matter of determining whether or not the originator/destination IP/phone number for the work that was actually done matches the authorized IP address/phone number (or falls within the IP address/phone country code range) using the “work order number” as the foreign key.

Now I can understand why the Wall Street Journal (your apparent source) and such as Rupert Murdoch would be deathly afraid of the NSA: most of the high value money laundering and out-and-out criminality in America is committed by Wall Street and the entangled web that is America’s financial system; with enough data capture, prosecuting the Wall Street and banking barons who so terrorize America’s economy would become a piece of cake.

Lacking sufficient evidence to hypothesize otherwise, I have to assume that no Murdoch property in the U.S. of A. has, as yet, committed the wiretapping and answering machine hacking that has made the Murdoch name so beloved in Britain.  (Or been prosecuted for it, anyway.)

Although the record of such activities would, no doubt, become available to the DOJ if the effort to blind the NSA fails.

My point is that we have no proof that an effective oversight regime or auditing procedure exists that could curtail actual abuse, or even theoretical abuse. If a 3rd party contractor can access troves of information unbeknownst to the NSA and moreover physically remove it, that’s an enormous red flag right there.

You want to argue it’s technically feasible for that capability to exist, and you’re right, it is. You’re making good points on that and I really don’t disagree with you about it. In fact I appreciate your response, you seem fairly knowledgeable. The problem is the NSA is overseeing itself. Ostensibly it is the FISC court, but when they come out and state (in previously classified documents) that they have to rely on the NSA to provide them information in order to do so, then it’s a farce.

Here’s an example: “The Court is troubled that the government’s revelations regarding NSA’s acquisition of Internet transactions mark the third instance in less than three years in which the government has disclosed a substantial misrepresentation regarding the scope of a major collection program.”

The NSA has no reason to perform a thorough and accountable auditing process of themselves. It’s in their best interests not to do so. If they uncover massive problems and report those, it hurts not only the organization but the people in charge of it. I assume you have a reasonable degree of technical expertise, can you tell me with a straight face that the potential for abuse and misuse of these programs and systems is not enormous?

You want to knock the WSJ due it’s ownership, and not uncommonly seen bias. Sometimes it certainly is guilty of that, but it’s not completely a partisan rag. It’s certainly feasible that due to it’s republican/conservative bias that the reporter who acquired those sources was able to do so in the first place. But the theoretical motivation you apply for it to do so is absurd to me. Why on Earth would this system ever target them? The fraud and abuse these institutions have frequently committed are often no big secret at all, and yet no one’s going after them. The true threat, the historically proven threat such capabilities provide is against people such as civil rights activists, government critics, activists in general, protesters, journalists and so on. These tools and powers have always been targeted at the powerless, not the powerful.

I would love nothing more than for the entire Murdoch empire to crumble and be revealed to all(as is already obvious to some) for the sham that it is, and the outright destructive force it has been on society. If you seriously want the NSA to take down Wall Street or the WSJ, and be considered legitimate for such use, you would only be replacing one evil with a potentially greater one. It’s also not as though his organizations didn’t relentlessly defend these programs and agencies during the Bush years. If they weren’t worried then, I don’t see why any of them would be now.

Wasn’t this whole X(classified) Billion/year security state put in place to fight terrorism, anyway? We’ve already killed many, many times over more innocent people fighting terrorism than died here to cause that.

When that bill to defund one aspect of this whole program was about to be voted on, Keith Alexander (longest serving NSA director) called an emergency meeting with representatives.

‘Interview with Congressman Alan Grayson D-FL: “SIROTA:…(is it possible) NSA not only has data and information on all of us but also has a lot of information on a lot of all of you and your colleagues in the Congress?

GRAYSON: It’s possible – one of my colleagues asked the NSA point blank will you give me a copy of my own record and the NSA said no, we won’t. They didn’t say no we don’t have one. They said no we won’t.’

At the very least we should be concerned about ending up with another J. Edgar Hoover.

SeattleGuy, this is progress.  I now understand partly where you’re coming from.  I still disagree, but now I see where the disagreement is.

First, to clarify, I’ve been working on the web since…well, not Day One, because I thought it was a passing fad that’d go the way of gopher (supported forever, but irrelevant), but pretty early, probably in the first year.  I’ve taught courses in programming against TCP/IP for fifteen years, too, so I know exactly what’s possible, and have always assumed the NSA would be listening in.

However, that doesn’t make it right.  I assume that many cops really, really want to torture confessions out of suspects, too, but I’m still displeased when it happens and make a point to argue against it whenever there’s evidence.  That’s what’s important about what Snowden did, after all:  He provided the evidence of what, until now, was dismissed as rumor by the tinfoil-hat crowd.

You can trust the existing government, if you want to, but what about the next administration or the one twenty years from now?  The room for abuse is phenomenal, ranging from terrorists (well, except for the Boston Marathon bomber, who managed to elude tracking even when the Russian government told us he was dangerous; that brain-damaged boxer was clearly a mastermind) to disrupting protests to digging up dirt on political opponents.

Maybe it won’t happen under Obama—though the DEA and IRS’s “parallel construction” training argues that it’s been happening since before Osama bin Laden hit the news and continues under Obama—but there are a lot of candidates in every election, and more of them are Rick Perry, Al Sharpton, or Michelle Bachman, than they are Jimmy Carter or Ronald Reagan.

Don’t want to have too much auto-moderation, so I’ll cut this, and run through links next.

Here’s FISC’s own analysis of the program.  They repeatedly say they think they’re being lied to, and while they’re oddly OK on Fourth Amendment grounds (presumably because judges currently believe it only applies to rifling through underwear drawers), they do call the program unconstitutional.

Think what you want about the EFF itself, but this was released under the Freedom of Information Act because they sued for it.

Here’s a quick summary of highlights, if you don’t have the time to read the whole thing:

Next, we have evidence not entirely in this context, but we can see that the NSA doesn’t care that it’s a military organization operating on American soil and doesn’t care about fallout from any spying it does.

I can only imagine the blowback in the ‘80s if the Russians had done something similar.

Assuming the government can be trusted to not abuse its power, NSA employees may not be prevented from monitoring their ex-girlfriends, because that’s clearly less creepy and more important than stopping terrorists.

You can argue that it’s a limited number, but (a) it’s only what we’ve learned about from a secretive agency and (b) the first one is an atrocity.  It’s the equivalent of cops beating suspects to death, nobody should care that “it’s only a few every year.”

Here’s the DEA/IRS angle, showing that they essentially launder intelligence content to pretend they’re not violating civil rights.  It also shows that the tools aren’t being used exclusively to stop terrorists, since…well, how many terrorists are out there filing tax returns?

I think the Groklaw case also better summarizes the danger of these programs better than anybody whining about their personal privacy.

I don’t worry about my privacy; I’m seriously just as much of a loudmouth when my last name is attached to what I say.  I worry about the privacy of the abuse victim trying to get away from his/her significant other at the NSA.  I worry about the next Glenn Greenwald about to receive a data dump that the government would prefer remain secret.  I worry about the next Dennis Kucinich, willing to actually be a disruptive voice in Washington no matter what powers he offends.  I worry about the political dissident in the Bahrain that our government will sell out to preserve our “special relationship.”  I worry about the next Malcolm X trying to march on Washington.  I worry about the next Occupy Wall Street or TEA Party.

We live in a situation where large companies and our government have control over these situations (by knowing the participants, their power structures, and their locations) and strong motivation to disrupt them, with many means at its disposal and a long history of using such means against enemies, Chelsea (formerly known as Bradley) Manning being the latest, and Aaron Swartz not too distant a memory.

(In fact, note how heavily the government and media try to hammer home the fact that Greenwald is gay, Snowden dated a stripper, and Manning is transsexual.  You’d almost think they want to divert attention from their wrongdoing to a spurious connection between whistle-blowing and some vague idea of “sexual deviance.”  Insert your own crude joke here.)

Speaking as someone who lost more than a few acquaintances in the World Trade Center, I’m willing to let the same thing happen again every day, if it means that reformers can do their thing without worrying what the “military-industrial complex” is going to do about it.

Destroying civil rights over fear of an enemy is what they do in the worst dictatorships in the world, including those in the Arab world producing terrorists, after all.  The solution to “they hate us for our freedoms” is not to eliminate our freedoms.  It also clearly doesn’t work.  The shoe bomber got on his plane.  The underwear bomber got on his plane.  The NYPD didn’t notice the SUV filled with fertilizer parked outside its office until a street vendor mentioned it.  The Boston Marathon bomber succeeded.

The government claims all of those events as victories, however.

@john- Working on the web since day one or not. If you don’t understand datamining and how the NSA is using algorithms with supercomputers to easily find terrorist networks without using personal information, you are a lay person IMHO.

When you cross the line and say something like, “listening in”, you cross into human surveillance involving warrants and usually the FBI, in these instances. That is a very unusual situation and, of course, it is very intrusive, but you have a better chance of attending an MLB game and witnessing an unassisted triple play than being the target of that handoff between the NSA and FBI. Are you still with me at this point?

The loveint issue will always be there no matter what is done with the NSA’s programs. Agreed? Set that aside and all illegal activity by government and commercial enterprise. I don’t think we disagree that those who break the law should be prosecuted. But you have not given me a reason to find and disrupt terrorist attacks as long as it doesn’t penetrate any personal information without a warrant.