Wednesday, October 18, 2006

Interior Dialogue Redux

In case anyone missed it, below is the investigative report I posted at Politics Central on Monday.

Buffalo? I don’t see no buffalo!Last Wednesday an urgent email arrived from a woman I’ll call Penpal, an employee at the Department of the Interior. She alerted me to the selective blocking of conservative blogs on the internal network at DOI, and wanted to get the word out to the right-leaning side of the blogosphere. Since she had no contact information for Charles Johnson at Little Green Footballs, she wrote to me instead.

She soon followed up with a list of blocked and unblocked blogs. Among those blocked were LGF, Gates of Vienna, Michelle Malkin, Belmont Club, and a number of other prominent conservative sites. Left unblocked were Daily Kos, Atrios, Democratic Underground, and other lefty favorites. “In fact,” she said, “every blog linked to off of DailyKos seems to work.”

I quickly put up a post about it and contacted Charles, who posted his own account and updated it with additional information. It quickly spread from there, passing through Instapundit, Atlas Shrugs, Michelle Malkin, and many other blogs, forums, and news sites.

That evening Roger Simon asked me to do an investigation and report on the subject for Pajamas Media. This sounded like a job for the 910 Group. I started a topic there asking for help investigating the internet-filtering software used by the Department of the Interior.

By the next morning, the intrepid 910 sleuths had tips and information about the software used, the way it was likely to have been implemented, and details about the internal organization and contact information in DOI.
- - - - - - - - - -
The Interior Department uses software written and supported by a company called 8e6. Its head office is in California, and I am in Virginia, so there was no point in calling them up for several hours. However, there are two branch offices on the East Coast, and I was able to contact the sales representative at the New York branch of 8e6. He had little technical information on their product, but promised to contact headquarters and have them call me.

I then placed phone calls to the Department of the Interior, beginning with the office of the Chief Information Officer, Hord Tipton. I spoke with his secretary and explained my mission. She took my number and said that Mr. Tipton would return my call.

When California office hours arrived, Eric Lundbohm, 8e6’s Vice President of Marketing, contacted me. He was pleasant and informative about his company’s product, explaining that 8e6 sells software for network use. It classifies websites and provides updates; however, it does not specify which sites will be blocked; the end user performs that function.

This made sense —I had already read the company’s 8e6 white paper. I asked him whether any changes to the software settings made by employees at the sub-group levels were automatically communicated to the upper levels. He said that they were, and as long as the administrators at the top read the logs, they will be aware of all such changes.

“In other words,” I concluded, “they either know what their subordinates are doing, or they are incompetent.”

When asked, he declined to tell me the who his company contact was at DOI. However, he was willing to say that it was “at the network administrator level.”

The next task was to locate the Network Administrator for the Department of the Interior. Web searches failed to turn up a name or phone number, so my only hope was to find someone at DOI who would refer me to the appropriate person.

Mr. Tipton, the CIO, never returned my call, so the next step was to call the Secretary of the Interior, Dirk Kempthorne. I had to go through two gatekeepers for Secretary Kempthorne. The second one, after saying, “Oh, I know what you’re calling about! We’ve had enquiries about it from two other people today.” Evidently the word had already got out to the media.

The second gatekeeper passed me on to the Director of Communications, Mr. Frank Quimby. He was pleasant, and explained that the Department most emphatically did not selectively block conservative blogs. “We block all blogs. We’re still getting the glitches out of the system, so some get through here and there, but we’re blocking all blogs.”

I told him that I would report what he said, but that it strained credulity, because the chance of the system randomly blocking blogs that just happen to have a conservative political content — and not blocking liberal ones — was so small as to be negligible. Someone, somewhere had to have done this intentionally.

Mr. Quimby passed me on to to his deputy, Mr. Ed Meagher, who was more familiar with the technical aspects of their internet filtering system. Mr. Meagher was not as friendly as Mr. Quimby — he bristled at any suggestion that the Interior Department was selectively blocking blogs based on their political orientation. “I can guarantee you 100% that we do not do that,” he told me. “We block all blogs.”

When I explained to him what Penpal had said, Mr. Meagher became quite vehement. “It didn’t happen. If your sources say that, then your sources are lying. Nobody here ever did anything like that.”

That was as unequivocal a denial as one can get

He refused to refer me to his network administrator, nor would he allow me to come into a DOI office and test the system myself, because that would “breach security”. So I returned to my sources and my network security experts for more information.

*   *   *   *   *   *   *   *   *   *   *   *   *   *   *

In addition to Penpal, I had a second source at the Department of the Interior who confirmed the selective blocking, but would not go public or consent to be quoted. A third source turned up at Pagan Vigil, where the blogger NeoWayland said this:

Ordinarily I would put this in the internet rumor file. But as chance would have it, I do know some people who work at the Department of Interior. Sure enough, it looks like the right side of the blogosphere is being blocked while the left side is wide open.

Three separate people at DOI were apparently lying, if we are to accept Mr. Meagher’s version of events. I wrote back to Penpal, and she was quite irate at being called a liar:

As far as the guy insisting categorically that all blogs are blocked, he’s lying himself! I’d guess he’s also technically incompetent. Maybe he even thinks that they are all supposed to be blocked, but they aren’t. Of course, IT Security Managers with DOI are complete idiots. [detailed technical examples followed here] Until recently, the division of DOI that I work for ran all their stuff in the complete free and clear. No firewalls, no filters, nothing. Anything and everything on the network was free for anyone at all to take who was interested.

As of last Thursday evening, October 12, the situation was still the same.

In all fairness, Mr. Meagher is not necessarily lying when he makes his assertions; it’s quite possible that his subordinates are assuring him that what he says is true. My sources have no particular reason, absent sheer malice, to fabricate these tales, whereas Mr. Meagher has a definite interest in maintaining the purity of his position.

Occam’s razor tells me that my sources are truthful, and that Deputy Director Meagher is mistaken.

*   *   *   *   *   *   *   *   *   *   *   *   *   *   *

How does the 8e6 software work?

The general idea is this: the program classifies websites, by their URLs and/or their IP addresses, according to a certain schema that includes news, educational, pornography, gambling, blogs, opinion, and so on. The end user can choose to block any of these categories within the network and its subdomains. The user may also choose, for business reasons, to list some websites specifically as exceptions, and these will be allowed through.

Mr. Quimby had said as much, implying that the Department of the Interior has a business reason to make exceptions of Daily Kos and the Huffington Post.

That doesn’t make any sense.

He also told me that blogs often have obscene or hateful content, which may earn them a “Hate Site” or “Pornography” classification, and hence cause them to be blocked.

Once again, that doesn’t make sense, as anyone who has visited Democratic Underground will attest.

Florence, an investigator at the 910 Group, reported:

It seems that 8e6 creates categories, identifies sites to go in those categories and then daily provides a download of updates them for all clients (who can select from categories to block within a common database). There is a correction functionality, but it is used for everyone. They use human content analysts who will review a proposed correction and recategorize if they agree, then resend the new categorization in the next update. There’s even a function to submit a site (presumably to be blocked but maybe for recategorization as well).

And here are the categories: Note that categories seem to be for things to INCLUDE specifically, not just to EXCLUDE - the list can work either way. Again, only if I’m reading this correctly…

One scenario (I think <50% probability): Could conservative sites have been submitted by pro-Islamist/extreme leftwing activists, for recategorization under any of the following categories that would be likely to be blocked:

Section 15: Dubious/Unsavory
Section 31: Hate & Discrimination
Section 48:  Message Boards
Section 54: Obscene/Tasteless
Section 63: Political Opinion (and note the language used to define this category - whoever wrote this does not suffer from Bush Derangement Syndrome)
Section 74: R-Rated
Section 79: Social Opinion

Why this seems unlikely: it appears that the content analysts at 8e6 actually assess these recategorizations. Wouldn’t they catch a pattern of, say, conservative blogs all being identified as obscene/tasteless?

Good news: they should be able to search their database for specific websites and track when they were submitted, by whom and what category (if any) they are currently filed under.

It doesn’t make sense that 8e6 (and other software vendors) would classify conservative blogs differently from liberal ones. They surely have customers on either side of the political divide, and one or the other group would be bound to object.

I had an email exchange with another internet filtering software expert, who had this to say:

We use a different product; each vendor will define categories differently. It’s a little like an antivirus program; each company has a different programming technique for detecting viruses and the same is true of proxy devices. The filters get updated as well, for example we see adult sites that are listed in the “None” category, which means it is not defined to a particular category. We can submit lists of sites we block and they get updated in the next filter release.

I ran a report on some of the sites you listed to see how our proxy device categorized them, only a couple of the “lefty” sites had any traffic over a 72 hour period:

Gates of Vienna: Political/Activist
Little Green Footballs: Computers/Internet
Michelle Malkin: News/Media
Power Line: Education
DailyKos: Political/Activist
The Huffington Post: News/Media

Our device actually looks at the content of the page, so each page can have sub categories. For example, Michelle Malkin’s site lists the text/html as “news” but certain images on the site show as “web advertisements”. Power Line also had a sub category of “Blogs/Newsgroups”.

The point is that the right-leaning sites are probably classified as blogs and the left sites might be classified news/political. Democratic Underground is probably listed as a forum. So if you block the category on the 8e6 device called “blogs”, it’s going to block everything it considers a blog. The only suggestion I can make would be to change META tags in HTML to not list keywords such as blog, assuming that’s how the 8e6 filter list is generated. However that may work on one type of device but not another.

Is everything clear now?

*   *   *   *   *   *   *   *   *   *   *   *   *   *   *

The Department of the Interior’s Internet Usage Policy does not specifically address political content. Its pincipal concerns are pornography and gambling sites, which — as any employer can tell you — are the main internet activities draining away employee productivity.

There is certainly nothing wrong with any employer, governmental or otherwise, blocking its employees’ access to blogs. But selective blocking of certain blogs based on their political leanings is bound to raise eyebrows.

For several days after my post went up, people in various parts of the federal government emailed me or left comments detailing their own departments’ policies. I heard from Treasury, Justice, the Air Force, and others. They all said more or less the same thing: all of their agencies blocked blogs, but some blogs seemed to be able to break through the filters. However, besides Interior, none of the exceptions lined up within a particular political affiliation.

I consulted a network expert, a friend of mine named Joe who runs his own consulting company. He’s very familiar with internet filtering software, and he also has extensive experience with network administration.

“Here’s what happened”, he said. “The department started blocking certain categories of websites, and then made a list of exceptions that would be allowed through the filter. That’s a long list, and it would be passed down the food chain from the Network Administrator through his subordinates until it reached the poor schmoe at the bottom of the heap who would have to do all the data entry to list the exceptions.

“Now imagine this guy: he just happens to be a left-winger, and likes to hang out at Daily Kos and Atrios during his downtime. He realizes that won’t be able to do that any more, so he adds his favorite sites to the list of exceptions, and then that he can continue with his recreational reading.

“He thinks that nobody will notice, or that his superiors are too stupid to ever figure it out. In any case, it never occurs to him that there are conservatives at DOI who will notice and object.”

I like this explanation. It’s simple, it’s elegant, and it satisfies Occam’s razor. It posits the least amount of conspiracy — at most, a couple of flunkies in the basement of DOI taking action on their own behalf — and it doesn’t require that my sources be liars.

A grand conspiracy at the top of DOI is unlikely. If they wanted to screen out conservative content — say, in order to keep their employees from learning about the Harry Reid caper — they could block all blogs, just as the other federal agencies are doing.

On the other hand, Joe’s theory indicates that the higher-ups in the Department of the Interior are not getting good information from their subordinates. Instead of stonewalling inquiries from the media, Mr. Meagher’s interests would have been better served by looking into the details of the situation. Calling your own employees liars — the very people you consented to hire — is not necessarily the wisest policy.

It’s time for an Interior dialogue.

I’d like to thank the dedicated folks at the 910 Group, particularly Florence, for their invaluable help with this report. Joe and Jolauna at Comperio Technologies provided extensive technical information. I’m also grateful to all the people in different federal agencies who took the trouble to contact me with accounts of what goes on in their departments.


Andrewdb said...

We have blocking software at my law firm. It uses IP addresses. At one point a naughty site and the local opera company used the same hosting service, and thus had the same IP address (at least as far as the national vendor of our blocking software knew). This caused no end of grief for out IT department as one of our partners was on the board of the local opera - eventually the national vendor agreed to update their blocking software and allow the opera company, at least until someone else complained about the naughty site.