Today we're announcing a new open-source project that aims to make web scraping simpler. It's named Upton, after labor journalist Upton Sinclair, because the project started as part of our intern investigation.
Upton is a web-scraping framework packaged as a RubyGem. It abstracts away some of the common parts of web scraping so that developers can concentrate on the unique parts of their project.
The recent disclosure of sweeping surveillance by NSA whistleblower Edward Snowden has prompted a new wave of legal challenges to the U.S. government’s intelligence-gathering programs.
Today we’re releasing a new open source project, which will enable any organization with a DocumentCloud account to do crowdsourcing using documents.
Since we wrapped up our Free the Files project after last year’s U.S. election, many people and organizations have asked us how they could build their own web applications like Free the Files to crowdsource their caches of documents. The full Free the Files codebase is undocumented, a bit messy and isn’t easy to deploy in environments other than our own, so we decided to extract the salient bits into a Rails plugin we’re calling Transcribable.
We as journalists can learn a lot from video games. They can help players explore unfamiliar worlds and experience stories, almost literally, through the eyes of another person. Designed well, video games guide players to feel emotion and conflict, as well as learn the intricacies of complex subjects and systems. They engage users in a highly meaningful, memorable and influential way.
But video games don't require an expensive console system or high-end computer. They can be powered up on our smartphones and on Facebook, and many people who would never call themselves gamers are playing games and getting familiar with how the medium works. And it's not all just escapist fun — there's a community dedicated to exploring how games can be used in education.
As journalists, games can be a great tool for us to use to reach, inform and engage our readers.
ProPublica has created a timeline to appreciate the key moments and often differing aims of the government’s judicial and legislative branches in the ongoing clash over civil rights.
The complaints against Condé Nast, Warner Music and Gawker Media are the latest in a rising tide of lawsuits brought by unpaid interns, many of which are still in progress.
This week’s P5 Resident is Aram Chung.
Aram is currently a dual master’s degree candidate at Columbia University’s joint journalism and computer science program. Aram, who is from Seoul, South Korea, graduated with a degree in mathematical sciences with a minor in industrial design at KAIST in Daejeon, South Korea, where she was also a reporter and editor at the KAIST Herald, the English-language campus monthly newspaper.
Every day I was at the three-day Games For Change conference this week, I learned about new and innovative games built to suit the needs of education and social change. I could have written about almost all of them, but here are four that are especially noteworthy to those of us thinking about building games for news.
Cameras were not present to capture Tiger Woods pulling out of his driving and hitting a fire hydrant, then a tree in 2009, nor to see his wife smash a car window with a golf club and drag Woods out of the vehicle. So to simulate the events of the story, a Taiwanese news service called Next Media Animation created an animated video of the ordeal, complete with shattering glass.
Since then, the company has produced similar videos based on celebrity news and even breaking news. At the time, I thought that these videos might actually catch on, and simulated reality would fill in the gaps when we couldn’t get real video footage. But they didn’t, at least not in the United States.
Yesterday at the Games For Change Festival’s day-two demo session, I saw a game in progress that took animated scenarios to a new level, and it really opened up some ideas for journalism.
If you think about it, positive social impact is a goal that’s similar to ours as journalists. Naturally, as I listened to the diversity of presentations today dedicated to social reach, education, impact and even fighting censorship, I found many lessons that apply to what we’re doing in newsrooms.
FEMA’s released new, preliminary flood insurance maps for New York City, which specify how likely areas are to flood. The new maps, which replace maps that used data from 1983, double the number of structures in flood zones.
The evolution of the National Security Agency’s dragnet under Presidents Bush and Obama.
The sixth ProPublica P5 Resident is Joanna Kao.
While the tools and techniques to present large datasets in graphics and news apps may differ from project to project, the basic design principles stay pretty much the same. Many might pretty familiar – even if you’ve never studied design formally, you probably know some of them instinctively. Let’s name them, explain why they work and see how other designers use them. Once you recognize the concepts, you’ll become more conscious of when and how to use them in your own projects.
We’re looking for the next great News Applications Fellow to work with ProPublica’s world-class news applications desk.
It's a great fellowship for, among others, a coder who's civic-minded and interested in working with journalistic problems and data sets, or a j-school grad who wants to tell stories with code, or a digital humanities student looking to try their skills in journalism. You’ll work side-by-side with the best newsroom developers on the planet, on some of the biggest and most innovative projects in investigative and data journalism. You will leave here a better journalist, a better designer and a better developer.
As part of our ongoing investigation into patient safety, ProPublica reporters Marshall Allen and Olga Pierce produced this interactive story in collaboration with PBS Frontline and Ocupop during a May 11-16 hackathon.
Medicare’s popular prescription-drug program now serves more than 35 million people, but the names of prescribers and the drugs they choose have never previously been public. Use this tool to find and compare doctors and other top prescribers in 2010.
Use our database to find almost 616,000 tax-exempt organizations and see details like their executive compensation, revenue and expenses, as well as download their tax filings going back as far as 2001.
In addition to our Nonprofit Explorer interactive database, here are some resources for researching charities and other tax-exempt organizations.
This week, Justin Elliott wrote about new House Financial Services Committee chairman Rep. Jeb Hensarling (R-Texas) attending a weekend getaway with banking industry officials.
One of the ways he found out who was at the getaway was by using the Instagram photo sharing service, which turned up a snowy snapshot taken by Len Wolfson, a lobbyist for the Mortgage Bankers Association (which had contributed to Hensarling's PAC). Wolfson has since set his account to private.
The Instagram site has no search function, so finding shots like this can take a lot of digging. However, Instagram has an API with a "Media Search" endpoint that returns data both by timeframe and distance from a certain latitude and longitude -- a perfect way to see who's at a certain place at a certain time.
ProPublica hosts newsroom developers -- or developers who want to see what it's like to work in news -- for 3-5 day job shadowing residencies called the ProPublica Pair Programming Project, or P5.
Use ProPublica's data -- cleaned, categorized and often created from multiple sources -- in your reporting and research.