Last week we published a big update to Dollars for Docs, our interactive news application of payments made to U.S. healthcare providers by 15 pharmaceutical companies. Compared to when we launched the project in 2010, the amount of data we’re collecting has grown enormously: The list of payments increased from around 750,000 to almost 2 million, and the grand total of the payments grew from around $750,000 to just under $2 billion.
Compiling the data for it has been an enormous project right from the beginning. After we published the first version, the original developer on the project, Dan Nguyen, compiled all of the things he had to learn into a guide to scraping data. This year’s update took more than eight months of full-time work by me, working with other news-app developers, and at times with our CAR team, a researcher, two editors and two health care reporters. It was a massive effort and presented huge technical and journalistic challenges.
During the 2012 election, we created Free the Files, an interactive news application based on crowdsourced data, built in real time by thousands of volunteers. It was a collaborative effort to track TV ad spending by campaigns, super PACs and so-called “dark money” nonprofit groups in the country’s top swing markets.
Measured by participation rate, Free the Files was an astonishing success. More than 1,000 contributors submitted over 94,000 transcriptions to help turn messy invoices from local TV networks into clean data. One volunteer transcribed over 28,000 filings. Each transcription was “verified” after two or more users agreed on all of its data points. There are currently around 17,000 verified filings, and people are still working.
On Friday, the Spanish chapter of the Society of News Design (SNDE) announced the winners of the 21st Malofiej International Infographics Awards. The jury evaluated over 1,000 entries from 28 countries.
This year, ProPublica was honored with the Miguel Urabayen Award for best map in the online category for StateFace, an open-source font we created. It’s made up of U.S. state shapes and is meant to be used as a design element in interactive web apps and graphics. The judges were impressed by StateFace’s versatility and its availability as a public service. Besides our own work, StateFace has been used by NPR, the Guardian, the Huffington Post and many others, especially in graphics and applications about the 2012 election. ProPublica also received a silver medal in the Innovative Format category for StateFace, and a bronze medal in Online Features/World and Nation for Housing Segregation: The Great Migration and Beyond by Jeff Larson and Nikole Hannah-Jones.
In one of the final sessions of the 21st Malofiej World Infographic Summit on Friday in Pamplona, Spain, Nigel Holmes noted the common theme of “hands” in the week’s presentations.
“We are humans, we are not attached to machines,” Holmes said as he presented a tongue-in-cheek look at “non-informational art” — fine art made by graphic journalists and artists when they weren’t making infographics.
As the conference was coming to a close, Holmes’ drawing attention to hands struck me as particularly insightful. One of the main undercurrents during the week was the tension between illustrative graphics, often borne out of photographs and sketches, and data visualization, whose raw material is often the very machine-based spreadsheet.
Today was the first day of the Malofiej World Infographic Summit in Pamplona, Spain. It’s a very small (the whole attendee list fits on two double sided sheets of paper) two day one-track conference. The Malofiej awards are highly geared towards news graphics (mostly print, but more so on the web), so naturally I thought the conference talks would be, too. But there was a broad spectrum of speakers. The big guys were there (Wilson Andrews of the Washington Post and Graham Roberts of The New York Times), and art directors from magazines in Brazil and Russia, but also a few, for lack of a better term, data artists.
Two talks, in particular, stuck out to me as examples of what I can only describe as “outsider CAR.” Only after I tweeted this, did I realize the coinage.
The fourth P5 Resident started a project in the ProPublica offices today. He’s Casey Thomas from AxisPhilly.
AxisPhilly is a non-profit news startup in Philadelphia. Their mission is to “educate and engage citizens on topics of public interest while empowering them with tools to participate in developing and implementing change.” A big part of their mission is interactive news applications. In a newsroom of nine people, AxisPhilly has two news apps developers and a freelancer who works on data projects. Casey’s projects at AxisPhilly include a map of the effects of delinquent properties on the market value of nearby homes, as well as a map projecting the property tax changes at each address in the city.
See where the over 20,000 SBA rebuilding loans are, half of which fall in FEMA’s new advisory flood zones.
As newsrooms incorporate news application teams, one of the first questions they have to answer for themselves is what technologies to choose, and how to set up developer and web hosting environments that are sane and tuned to serve news apps. As part of her P5 Residency, developer Peggy Bustamante from Digital First Media's Project Thunderdome spent a few days mapping ProPublica's infrastructure. She's written a post that lays out some alternatives that came out of a discussion on the NICAR-L mailing list, and the answers that the Thunderdome team came up with.
What follows is ProPublica's advice on developer and server setups for news apps teams.
Peggy Bustamante is a news app developer with Digital First Media’s Data Team, and was the P5 Resident at ProPublica in January. She spent her time working with ProPublica News App Editor Scott Klein on mapping ProPublica's tech setup. Scott wrote a blog post about ProPublica's setup, and Peggy wrote this post, about alternative scenarios and DFM's own approach.
The first thing you want to do when you join a news apps team is build cool projects.
But if you are a new outfit, as we are at Digital First Media’s Thunderdome data team, there is one big step before you can get to that happy place of creativity and news: you have to set up an environment to build those cool projects.
Today we're publishing a series of guides that we hope will be useful for news app teams everywhere: A News App style guide, a high-level design overview, a coding manifesto, our standard social tags and a data bulletproofing guide. They represent what we've learned and our best advice for designing consistent, social-optimized and impactful apps in a sane dev environment.
With the closing of EveryBlock yesterday, both of the main reasons I started working in News Applications have dissapeared from the internet. The other reason was a New York Times app called Represent, which allowed you to keep an eye on your elected representatives. The world is worse off with these pioneering news apps gone, but retiring old apps is something that our industry has to come to terms with. At ProPublica, we’ve started to retire our old apps, by removing search boxes and dynamic calls to the server. We’re making every effort to make them available on the internet forever.
How big is the natural gas drilling regulatory staff in your state?
Though the Sandy relief bill passed both the Senate and the House, many members of Congress voted no despite their own states receiving millions of dollars in federal disaster assistance in 2012.
Starting today if you connect your Foursquare account to "The Opportunity Gap," we'll send you stats about schools whenever you check into one. If you've checked into a school we've associated with a Foursquare "venue," we'll show you some details and give you a link to that school's profile.
Explore the great migration of African Americans from 1940 to 2000 and segregation in Northern cities.
Nearly 100 backscatter scanners were removed from major airports recently to speed up lines. See if they’re still in use at your airport.
When I did start learning, I was amazed by how much was out there: introductory videos, explanatory blog posts, tips and tricks and step-by-step guides. If you're a journalist who wants to make a news app or a student interested in learning to code, you have plenty of paths to choose from.
ProPublica hosts newsroom developers -- or developers who want to see what it's like to work in news -- for 3-5 day job shadowing residencies called the ProPublica Pair Programming Project, or P5.
Use ProPublica's data -- cleaned, categorized and often created from multiple sources -- in your reporting and research.