ProPublica

Journalism in the Public Interest

Cancel

The ProPublica Nerd Blog

Chapter 3: Turning PDFs to Text

Chapter 2: Reading Data from Flash Sites

Chapter 1. Using Google Refine to Clean Messy Data

Chapter 4: Scraping Data from HTML

Chapter 5: Getting Text Out of an Image-Only PDF

Open Source Project: Thinner

Today we're releasing a new open source project called "Thinner." It's for websites, like ours, that use the open source caching engine Varnish.

Use Our Dollars for Docs Widget on Your Site

As part of ProPublica’s “Dollars for Docs” series and interactive news application, we've created a small widget that you can embed on your web site. It will let your readers look up whether their health care providers are taking money from the drug companies in our database. The widget shows the amount of money paid to each practitioner in our database, which company made the payment, and in some cases, what the companies saidthey were paying for: speaking fees, consulting, etc. The widget also lists what drugs each company sells so readers can check their own prescriptions.

A Tale of Two Documents

.

On Oct. 8, we published an investigation examining how a judicial opinion in a pivotal lawsuit brought by a Guantanamo detainee vanished, only to be replaced weeks later by an entirely different opinion. At the center of our reporting are two documents representing separate versions of that same opinion: the original opinion written by Judge Henry H. Kennedy, and a second opinion quietly put in the original's place more than a month later.

Why are there two opinions? As reporter Dafna Linzer explains, redactions that were supposed to be made in the original opinion never were. Once government security officials, who are responsible for reviewing and redacting classified information from sensitive cases, discovered the error, the decision was quickly removed from the court file. In Judge Kennedy’s courtroom four days later, the Justice Department refused to have the opinion redacted and re-released. With the detainee, Uthman Abdul Rahim Mohammed Uthman, slated for indefinite detention, the stakes were high. Officials did not want to risk that those who had seen the original opinion would know exactly what the government had meant to keep classified.

The Rainbow Connection: How We Made Our CDO Connections Graphic

.

On Wednesday, we launched an interactive news application to help readers understand the cross-owned nature of Collateralized Debt Obligations (CDOs) in 2006-2007. This cross-ownership helped inflate the bubble, and ultimately made the financial crisis worse.

We received a list of cross-owned CDOs as a result of a study ProPublica commissioned from Thetica, a consulting company in New York. It consisted of a list of CDOs, the banks that sponsored them, the CDO managers who managed them, and an enumerated list of other CDOs in which it had both sold and bought a stake. Reporters Jake Bernstein and Jesse Eisinger had already used the data in their story, Banks’ Self-Dealing Super-Charged Financial Crisis.

Pixel Ping: A node.js Stats Tracker

.

Since the day we launched, ProPublica has encouraged people to republish our stories for free. We even license our stories under Creative Commons (CC). However, in the past we've had trouble knowing precisely which stories had been republished where, and we had no way of knowing how many people were reading our stories on sites that republished them under our CC license.

Shortly after the redesign of our site, we started working on a system that would help us solve this problem. When we found out that Jeremy Ashkenas, a developer at DocumentCloud, was working on a similar problem, we joined forces, and finished work on a lightweight stats tracker, which we are open sourcing today.

World, meet Pixel Ping.

Colophon

Get Our Data and Reporting Tools

We frequently publish data, and reporting tools like tipsheets and guides.

Sign Up