David Herzog

Tripwires in open data policy

David Herzog — Wed, 22 May 2013 19:17:31 +0000

The Obama Administration released its Open Data Policy two weeks ago, getting deserved attention from the open government community.

On the surface, the White House policy gives transparency advocates much to get excited about. When it comes to data, the memo (PDF) directs federal agencies to “adopt a presumption in favor of openness.” Moreover, the memo says that open data should be:

Described through metadata, record layouts and other documentation
Complete and available “at the finest possible level of granularity”
Timely
Managed post-release, which means agencies will need to designate a point of contact to handle questions and complaints about the data

Sweet words for journalists and other data users, for sure.

In addition, the memo directs all agencies to create data inventories and then post public versions of them. So, journalists would be able to scan the lists for ideas, and the public would be able to provide feedback about the data sets they’d like.

But, there are some pretty big shortcomings that could undermine the effectiveness of the policy and disappoint journalists.

For starters, the directive applies only to new federal government information systems, or ones that are getting major overhauls.

Another sticking point for journalists could come if agencies create public inventories that exclude databases that have been deemed “private” or “sensitive” by the agency.

We should be able to know about all of the databases and why an agency considers some private. After all, the federal Freedom of Information Act allows us to get the unrestricted portions of databases. Some agencies, like the U.S. Drug Enforcement Administration, already tell us about their extensive data holdings, including the Confidential Source System.

Perhaps the biggest wild card is the strong caution against agencies releasing personally identifying information. The memo directs agencies to consider how other available public information could help ID someone in data, something called the mosaic effect.

That’s exactly how journalists do their jobs: By getting data and documents, talking to people and figuring out how the world works. Data restrictions would interfere.

The key will be how agencies interpret this. We’ve had a preview of this in the 2011, when the Dept. of Health and Human Services pulled a public database about disciplinary actions against doctors and threatened a Kansas City Star reporter who had used it. The reporter analyzed the public-use file of the National Practitioner Data Bank, which does not identify doctors, to help investigate a surgeon with a history of malpractice payments.

Eventually, HHS restored the data, but said that users could not use it to help identify any doctors.

CAR class story: DWI fatalities persist

David Herzog — Mon, 20 May 2013 16:09:52 +0000

Take a look at the first news report to emerge from the group projects in CAR class. Nice work by Greg Mantell and Xinrui Zhu for this story about how DWI fatalities in mid-Missouri have persisted, even as the number of DWI accidents and injuries has declined.

Building Open States

David Herzog — Thu, 14 Feb 2013 22:06:30 +0000

Here’s how the Sunlight Foundation built Open States, a state legislative information service during the past few years. It’s an interesting look at building a big open government project with a community of web developers.

Assignment: Comment on a CAR story

David Herzog — Thu, 31 Jan 2013 22:52:22 +0000

The wait is over, here is that mystery assignment that’s due by noon Tues., Feb. 5.

Read a CAR/data journalism story that’s been published in the past two years. Then, use the commenting system here to write a few paragraphs that summarize the story and how data was used. Also, note questions that you have about the story.

You can find examples of CAR stories a few different places:

Investigative Reporters and Editors Extra!Extra! investigative story blog, where you can see articles categorized as CAR.
The New York Times and USA Today regularly run data-driven stories, so you might want to poke around on their sites. Search the sites for phrases like “data analysis”, “analysis of computer records” or “computer-assisted.”

Good luck with it. I’m eager to see what you discover!

Spreadsheets for data journalism

David Herzog — Wed, 23 Jan 2013 19:11:59 +0000

For many students and journalists, spreadsheets are the entry-level computer-assisted reporting tool and it’s easy to see why: They come installed on many computers and are relatively simple to use. Most students learn how to use spreadsheets in middle school, if not before. Another attraction for data journalists is that government agencies release a lot of data in formats that can be easily opened in spreadsheet programs. It’s no wonder that some data journalists have joked that spreadsheets are the “gateway drug” of CAR.

Spreadsheets help us run calculations and create pivot table reports. We can use them to sort and sift through huge data tables, too.

We’re going to use Microsoft Excel in class, which is installed in our lab and should be on everyone’s personal laptop. Our lab computers run Excel for Windows 2010 and can open spreadsheet files with more than 1 million rows and more than 16,ooo columns. Older versions of Excel (2003 and earlier) can only handle files with more than 65,000 rows and 256 columns.

More recent (2008 and 2011) versions of Excel for Mac can open the same larger files as their Windows counterparts. Earlier versions share the same limits as in Windows.

However, if you eventually want to use the Microsoft PowerPivot add-on to churn through millions of records, you’ll need Excel for Windows 2010. If you’d like to use the NodeXL tool to create network diagrams, it’s Excel for Windows 2007 or later.

Another great spreadsheet option is the open-source program Calc. You can download Calc as part of OpenOffice or LibreOffice. Open-source software is available free of cost and of licensing restrictions. Calc runs on Mac or Windows and support large files (more than 1 million rows x 256 columns).

You can see that Calc is not as polished as Excel, but it will definitely meet your spreadsheet needs.

Google Drive spreadsheets might be a good option for smaller files — there’s a 20 MB limit for files that you’re uploading and converting. Further, you can only have 400,000 cells. So if you have a spreadsheet with 40 columns, you can only have 10,000 rows.

Pivot tables in Google Drive spreadsheets are primitive, when compared to Excel or Calc. But Google Drive spreadsheets shine at handling live data feeds from the Internet and in allowing you to collaborate on the same file with other journalists or the public. You can even build forms for entering data into your spreadsheet.

Practice safe computing, though: Don’t store any sensitive information on Google Drive or any other cloud-based service, where you lack exclusive control to the data.

CAR class keys to success

David Herzog — Tue, 22 Jan 2013 15:07:08 +0000

Excelling in the Computer-Assisted Reporting class (and at data journalism) takes dedication. It’s easy enough to attend every class, contribute to discussions and complete every homework assignment by deadline to earn a solid grade. But, students who want to excel must go beyond the minimum. In my 10 years of teaching this class at the Missouri School of Journalism, I’ve noticed that the top students are:

Curious. They want to know how the world really works and aren’t satisfied with conventional wisdom or superficial explanations. When they work with data, they keep trying to ask questions and get to the bottom of things.
Detail-oriented. All of journalism requires attention to detail. We strive to spell the names and record the job titles of sources correctly. My top students understand that all databases are dirty and they go to lengths to understand the scope and nature of the problems.
Resourceful. Students run into problems all the time when they’re working with data. Even the most seasoned data journalists encounter new and gnarly challenges. The best know how to solve problems tapping into online resources, such as software documentation, message boards and listservs. No one can know everything, so it’s important to learn how to learn.
Unafraid of failure. We all learn CAR with lots of practice and failure. (I’m speaking from personal experience here.) This is especially true when we learn how to use Structured Query Language (SQL) to ask questions about data, a process that can be highly experimental.
Obsessive about documenting their work. They keep data-negotiation logs that detail their interactions with government officials. Also, they keep audit trails of their data cleaning and analysis, so anyone can see exactly what work they’ve performed.
Practitioners of safe computing. When they work with data, they always keep a safe copy of their original files in another location. If they need to clean their data, they preserve the original columns for reference.
Proactive. The best students stay on top of assignments by meeting homework deadlines and consistently moving ahead with their data negotiation efforts. Even more, they communicate early with me about potential problems and class conflicts.
Engaged. In the classroom, the best students give their undivided attention and presence to whatever is happening that day, whether it’s a lecture, hands-on practice or discussion. When everyone participates, the class is better for all. Outside the classroom, they are aware of what’s going in here in Columbia and around the world.

Data journalism at Mizzou

David Herzog — Fri, 18 Jan 2013 16:21:35 +0000

With classes starting in just four days, it’s a great time to take stock of the data journalism offerings for students at the Missouri School of Journalism. Starting this semester, students can learn anything from analyzing simple spreadsheets to building news apps with programming languages.

Undergrads who want to get a taste of reporting with data should check out Fundamentals of Data Reporting (JOURN 4122) , taught by Assoc. Prof. Scott Swafford. This one-credit weekend boot camp (Feb. 8-10) is for students who want to learn the basics of finding and negotiating for data, then analyzing it using Microsoft Excel spreadsheets.

Students who want a deeper dive should take my Computer-Assisted Reporting class (JOURN 4430/7430), in which we cover data negotiation, cleaning, analysis with spreadsheets and database managers and visualization. All the spring sections are full, unfortunately.

In Mapping for Stories and Graphics (JOURN 4440/7440), we build on CAR by analyzing data using geographic information system (GIS) software and online mapping programs. This two-credit class is available in fall semesters only.

And this semester. for the first time, students can take an Advanced Data Journalism class to learn how to use the python programming language and Django web framework to create news apps. Chase Davis, a Mizzou alum and former director of technology for the California Center for Investigative Journalism, will teach the class remotely, with support from instructors here.

To register, students need to have experience with a programming language or have mastered Structured Query Language (SQL) in the CAR class. This is a section (02) of Emerging Technologies in Journalism (JOURN 4462/7462). I’m the contact for permission numbers.

Assignment: Comment on a CAR story

David Herzog — Thu, 30 Aug 2012 20:32:28 +0000

Due: Noon, Tues., Sept. 4

One of the best ways to dive into the world of computer-assisted reporting is to examine stories that have used data analysis.

So, for this assignment, I would like you to read a CAR story that’s been published in the past two years. Then, I want you to use the commenting system here to write a few paragraphs that summarize the story and how data was used. Also, note questions that you have about the story.

You can find examples of CAR stories a few different places:

Investigative Reporters and Editors Extra!Extra! investigative story blog, where you can see articles categorized as CAR.
The New York Times and USA Today regularly run data-driven stories, so you might want to poke around on their sites. Search the sites for phrases like “data analysis”, “analysis of computer records” or “computer-assisted.”

Good luck and happy digging!

FCC political ads site goes live

David Herzog — Mon, 06 Aug 2012 01:59:43 +0000

Late last week, some of the biggest TV stations in the United States started uploading political ad-buy contracts to a Federal Communication Commission website. A recent FCC rule says that stations in top 50 markets, including some in St. Louis and Kansas City, most post new contracts for political ads soon after the stations execute them.

The FCC’s rule applies only to TV stations that are affiliated with the ABC, CBS, Fox and NBC networks. Stations in smaller TV markets, like our own Columbia-Jefferson City, won’t have to start posting the information until the summer of 2014. The same goes for large-market stations that are not affiliated with one of the big four networks.

The contracts are a rich source of information for journalists, citizens and political operatives who want to keep track of spending by candidates, political actions committees and nonprofit organizations. Here’s one that shows Claire McCaskill’s U.S. Senate campaign bought 33 ads spots to run during the news, game shows and other programs in April on local ABC affiliate KMIZ for a sum of $1,061.

However, you can only search the FCC site by station. In other words, you can’t punch in “McCaskill,” to search and uncover all of the contracts for her campaign. Another shortcoming: you can’t tell by looking at the station’s political file, whether it uploaded any documents of interest. You’ll need to look drill down through folders to see if anything’s there.

We’re gathering the ad contracts as part of a Missouri School of Journalism project for reporting on the 2012 elections. In the coming weeks, we hope to launch the website, where you can see how the candidates and committees are spending millions of dollars on local TV stations to win your vote.

FCC’s political ads rule hits home

David Herzog — Thu, 03 May 2012 02:14:02 +0000

Some TV stations in Missouri will need to make political ad purchase information available on the web in the coming months, thanks to a ruling by the Federal Communications Commission late last week.

Commercial stations that are affiliates of the major networks (ABC, CBS, Fox and NBC) in the top 50 local television markets will need to upload these documents to the FCC’s website after the rules kick in some time this summer. Smaller-market stations will need to start filing on July 1, 2014.

That means we’ll be able to get access to the ad-buy information from the eight affiliates in St. Louis (market 21) and Kansas City (market 31), but not for the 10 or so other TV stations in those markets. Neither will we be able to look up information about ads running in the state’s smaller markets, which include Columbia, Joplin, Springfield, St. Joseph, Hannibal, Cape Girardeau and Kirksville.

The FCC ruling paves the way for a revamp of the way commercial TV stations have long made public the information about political ad purchases. Since 1938, the commission has required stations to keep copies of the ad buys in the stations’ public files. Want to view the documents? You’ll have to travel to the business offices of your local TV stations, just like we’re doing for a website about the 2012 elections that we’re building at the Missouri School of Journalism. We’re visiting the stations, scanning the information and making it available online so you can see how local, state and national campaigns are spending money to win your vote with ads like this.

So when the FCC late last year proposed putting the ad-buy information online, I filed comments with the commission, urging it to do so.

Dive into the details of the FCC’s order and you can see that it’s a mixed bag for citizens, journalists and open-government advocates.

TV stations will not have to load data about past political ad purchases, just those “going forward.” The commission felt that requiring stations to dig back in time would be a burden.
The FCC won’t require stations to report the information in a structured format, which would provide journalists, researchers and political operatives rich data that they could analyze. The FCC says providing the information in a structured data format is the ultimate goal, but one that needs to be set aside to get the ad-buy information online now.
Stations will be able to upload the ad-buy documents in an existing electronic format, such as Word, rich text or PDF.
When stations submit documents that do not include readable text, the commission will use optical character recognition (OCR) software to extract the text.
The commission has no plans to let the public search the contents of the documents.

Taken as a whole, the commission’s order seems like it’s designed to be the least offensive to the broadcasters, who fought the proposal and will need to post information within months. Time will tell whether the rule is strong enough to deliver information that’s easy for the public to navigate and satisfies the thirst for details about who’s paying to influence our elections.