Journalism in the Public Interest


The ProPublica Nerd Blog

Scraping for Journalism: A Guide for Collecting Data

A series of technical and programming tutorials on how scraped, parsed, and organized data for “Dollars for Docs.”

Scraping for Journalism: A Guide for Collecting Data

Our Dollars for Docs news application lets readers search pharmaceutical company payments to doctors. We’ve written a series of how-to guides explaining how we collected the data.

The Coder’s Cause in “Dollars for Docs”

Our investigation of the financial ties between drug companies and doctors, Dollars for Docs, was sparked by a computational challenge. Several drug companies had been ordered to disclose who they paid to speak and consult on their behalf. But they made the records hard to analyze, seemingly making the data “impossible to download.”

We wanted to change that.

Chapter 3: Turning PDFs to Text

Chapter 2: Reading Data from Flash Sites

Chapter 1. Using Google Refine to Clean Messy Data

Chapter 4: Scraping Data from HTML

Chapter 5: Getting Text Out of an Image-Only PDF

In the previous guide, we describe several methods for turning PDFs into data usable for spreadsheets. However, those only handle PDFs that have actual text embedded within them. When a PDF contains just images of text, as they do in scanned documents, then the problem isn't just how to convert them into neat tabular data, but how to extract any text, period.

Find Homes With Tainted Drywall

When the Consumer Products Safety Commission provided data in October, the agency said it had received fewer than 3,500 reports of tainted drywall. ProPublica and the Sarasota Herald-Tribune compiled a list of addresses from county property appraiser data and records in consolidated lawsuits filed in New Orleans federal court and found nearly twice that number: around 6,900 homes.

Interactive: Which Banks Got Emergency Loans from the Fed During the Financial Meltdown?

Wednesday the Federal Reserve released data on more than 21,000 loans and other deals it made through a dozen emergency programs created during the financial crisis. We’ve combined the Fed’s three programs that loaned directly to banks and other financial firms with the goal of getting them to start lending again.

Open Source Project: Thinner

Today we're releasing a new open source project called "Thinner." It's for websites, like ours, that use the open source caching engine Varnish.

Use Our Dollars for Docs Widget on Your Site

As part of ProPublica’s “Dollars for Docs” series and interactive news application, we've created a small widget that you can embed on your web site. It will let your readers look up whether their health care providers are taking money from the drug companies in our database. The widget shows the amount of money paid to each practitioner in our database, which company made the payment, and in some cases, what the companies saidthey were paying for: speaking fees, consulting, etc. The widget also lists what drugs each company sells so readers can check their own prescriptions.

A Tale of Two Documents

On Oct. 8, we published an investigation examining how a judicial opinion in a pivotal lawsuit brought by a Guantanamo detainee vanished, only to be replaced weeks later by an entirely different opinion. At the center of our reporting are two documents representing separate versions of that same opinion: the original opinion written by Judge Henry H. Kennedy, and a second opinion quietly put in the original's place more than a month later.

Why are there two opinions? As reporter Dafna Linzer explains, redactions that were supposed to be made in the original opinion never were. Once government security officials, who are responsible for reviewing and redacting classified information from sensitive cases, discovered the error, the decision was quickly removed from the court file. In Judge Kennedy’s courtroom four days later, the Justice Department refused to have the opinion redacted and re-released. With the detainee, Uthman Abdul Rahim Mohammed Uthman, slated for indefinite detention, the stakes were high. Officials did not want to risk that those who had seen the original opinion would know exactly what the government had meant to keep classified.

The Rainbow Connection: How We Made Our CDO Connections Graphic

On Wednesday, we launched an interactive news application to help readers understand the cross-owned nature of Collateralized Debt Obligations (CDOs) in 2006-2007. This cross-ownership helped inflate the bubble, and ultimately made the financial crisis worse.

We received a list of cross-owned CDOs as a result of a study ProPublica commissioned from Thetica, a consulting company in New York. It consisted of a list of CDOs, the banks that sponsored them, the CDO managers who managed them, and an enumerated list of other CDOs in which it had both sold and bought a stake. Reporters Jake Bernstein and Jesse Eisinger had already used the data in their story, Banks’ Self-Dealing Super-Charged Financial Crisis.

CDOs’ Interlocking Ownership

See which CDOs exchanged pieces with other CDOs through our interactive feature that reveals the incestuous nature of Wall Street’s CDO business.

Pixel Ping: A node.js Stats Tracker

Since the day we launched, ProPublica has encouraged people to republish our stories for free. We even license our stories under Creative Commons (CC). However, in the past we've had trouble knowing precisely which stories had been republished where, and we had no way of knowing how many people were reading our stories on sites that republished them under our CC license.

Shortly after the redesign of our site, we started working on a system that would help us solve this problem. When we found out that Jeremy Ashkenas, a developer at DocumentCloud, was working on a similar problem, we joined forces, and finished work on a lightweight stats tracker, which we are open sourcing today.

World, meet Pixel Ping.

Welcome to the Nerd Blog

Today we are introducing our Nerd Blog, a place to talk about what programmer-journalists at ProPublica are working on, announce newly-launched news applications, and to hear from technically-minded readers, as well as our fellow nerdy journalists. We’re going to be writing about each of our projects as we release them, and flagging open source tools we’ve found useful.

So what the heck is a “news application”? It’s an interactive web page that uses software instead of words and pictures to do journalism.

Chart: A Bank’s Best Customers

In the last two years of the boom, CDOs created by one bank commonly purchased slices of other CDOs created by the same bank.

BP Claims Tracker

Follow the damage claims from the Gulf Oil Spill paid by HP.

Bailout Scorecard

Our frequently updated database tracks every dollar and every bailout recipient. Check out our scorecard to see where the spending stands.

The News Apps Team

Download Our Data

Use ProPublica's data — cleaned, categorized and often created from multiple sources — in your reporting and research.

Use Our Code

Explore Our Work