David Herzog » CAR Class

CAR class story: DWI fatalities persist

David Herzog — Mon, 20 May 2013 16:09:52 +0000

Take a look at the first news report to emerge from the group projects in CAR class. Nice work by Greg Mantell and Xinrui Zhu for this story about how DWI fatalities in mid-Missouri have persisted, even as the number of DWI accidents and injuries has declined.

Assignment: Comment on a CAR story

David Herzog — Thu, 31 Jan 2013 22:52:22 +0000

The wait is over, here is that mystery assignment that’s due by noon Tues., Feb. 5.

Read a CAR/data journalism story that’s been published in the past two years. Then, use the commenting system here to write a few paragraphs that summarize the story and how data was used. Also, note questions that you have about the story.

You can find examples of CAR stories a few different places:

Investigative Reporters and Editors Extra!Extra! investigative story blog, where you can see articles categorized as CAR.
The New York Times and USA Today regularly run data-driven stories, so you might want to poke around on their sites. Search the sites for phrases like “data analysis”, “analysis of computer records” or “computer-assisted.”

Good luck with it. I’m eager to see what you discover!

Spreadsheets for data journalism

David Herzog — Wed, 23 Jan 2013 19:11:59 +0000

For many students and journalists, spreadsheets are the entry-level computer-assisted reporting tool and it’s easy to see why: They come installed on many computers and are relatively simple to use. Most students learn how to use spreadsheets in middle school, if not before. Another attraction for data journalists is that government agencies release a lot of data in formats that can be easily opened in spreadsheet programs. It’s no wonder that some data journalists have joked that spreadsheets are the “gateway drug” of CAR.

Spreadsheets help us run calculations and create pivot table reports. We can use them to sort and sift through huge data tables, too.

We’re going to use Microsoft Excel in class, which is installed in our lab and should be on everyone’s personal laptop. Our lab computers run Excel for Windows 2010 and can open spreadsheet files with more than 1 million rows and more than 16,ooo columns. Older versions of Excel (2003 and earlier) can only handle files with more than 65,000 rows and 256 columns.

More recent (2008 and 2011) versions of Excel for Mac can open the same larger files as their Windows counterparts. Earlier versions share the same limits as in Windows.

However, if you eventually want to use the Microsoft PowerPivot add-on to churn through millions of records, you’ll need Excel for Windows 2010. If you’d like to use the NodeXL tool to create network diagrams, it’s Excel for Windows 2007 or later.

Another great spreadsheet option is the open-source program Calc. You can download Calc as part of OpenOffice or LibreOffice. Open-source software is available free of cost and of licensing restrictions. Calc runs on Mac or Windows and support large files (more than 1 million rows x 256 columns).

You can see that Calc is not as polished as Excel, but it will definitely meet your spreadsheet needs.

Google Drive spreadsheets might be a good option for smaller files — there’s a 20 MB limit for files that you’re uploading and converting. Further, you can only have 400,000 cells. So if you have a spreadsheet with 40 columns, you can only have 10,000 rows.

Pivot tables in Google Drive spreadsheets are primitive, when compared to Excel or Calc. But Google Drive spreadsheets shine at handling live data feeds from the Internet and in allowing you to collaborate on the same file with other journalists or the public. You can even build forms for entering data into your spreadsheet.

Practice safe computing, though: Don’t store any sensitive information on Google Drive or any other cloud-based service, where you lack exclusive control to the data.

CAR class keys to success

David Herzog — Tue, 22 Jan 2013 15:07:08 +0000

Excelling in the Computer-Assisted Reporting class (and at data journalism) takes dedication. It’s easy enough to attend every class, contribute to discussions and complete every homework assignment by deadline to earn a solid grade. But, students who want to excel must go beyond the minimum. In my 10 years of teaching this class at the Missouri School of Journalism, I’ve noticed that the top students are:

Curious. They want to know how the world really works and aren’t satisfied with conventional wisdom or superficial explanations. When they work with data, they keep trying to ask questions and get to the bottom of things.
Detail-oriented. All of journalism requires attention to detail. We strive to spell the names and record the job titles of sources correctly. My top students understand that all databases are dirty and they go to lengths to understand the scope and nature of the problems.
Resourceful. Students run into problems all the time when they’re working with data. Even the most seasoned data journalists encounter new and gnarly challenges. The best know how to solve problems tapping into online resources, such as software documentation, message boards and listservs. No one can know everything, so it’s important to learn how to learn.
Unafraid of failure. We all learn CAR with lots of practice and failure. (I’m speaking from personal experience here.) This is especially true when we learn how to use Structured Query Language (SQL) to ask questions about data, a process that can be highly experimental.
Obsessive about documenting their work. They keep data-negotiation logs that detail their interactions with government officials. Also, they keep audit trails of their data cleaning and analysis, so anyone can see exactly what work they’ve performed.
Practitioners of safe computing. When they work with data, they always keep a safe copy of their original files in another location. If they need to clean their data, they preserve the original columns for reference.
Proactive. The best students stay on top of assignments by meeting homework deadlines and consistently moving ahead with their data negotiation efforts. Even more, they communicate early with me about potential problems and class conflicts.
Engaged. In the classroom, the best students give their undivided attention and presence to whatever is happening that day, whether it’s a lecture, hands-on practice or discussion. When everyone participates, the class is better for all. Outside the classroom, they are aware of what’s going in here in Columbia and around the world.

Data journalism at Mizzou

David Herzog — Fri, 18 Jan 2013 16:21:35 +0000

With classes starting in just four days, it’s a great time to take stock of the data journalism offerings for students at the Missouri School of Journalism. Starting this semester, students can learn anything from analyzing simple spreadsheets to building news apps with programming languages.

Undergrads who want to get a taste of reporting with data should check out Fundamentals of Data Reporting (JOURN 4122) , taught by Assoc. Prof. Scott Swafford. This one-credit weekend boot camp (Feb. 8-10) is for students who want to learn the basics of finding and negotiating for data, then analyzing it using Microsoft Excel spreadsheets.

Students who want a deeper dive should take my Computer-Assisted Reporting class (JOURN 4430/7430), in which we cover data negotiation, cleaning, analysis with spreadsheets and database managers and visualization. All the spring sections are full, unfortunately.

In Mapping for Stories and Graphics (JOURN 4440/7440), we build on CAR by analyzing data using geographic information system (GIS) software and online mapping programs. This two-credit class is available in fall semesters only.

And this semester. for the first time, students can take an Advanced Data Journalism class to learn how to use the python programming language and Django web framework to create news apps. Chase Davis, a Mizzou alum and former director of technology for the California Center for Investigative Journalism, will teach the class remotely, with support from instructors here.

To register, students need to have experience with a programming language or have mastered Structured Query Language (SQL) in the CAR class. This is a section (02) of Emerging Technologies in Journalism (JOURN 4462/7462). I’m the contact for permission numbers.

Assignment: Comment on a CAR story

David Herzog — Thu, 30 Aug 2012 20:32:28 +0000

Due: Noon, Tues., Sept. 4

One of the best ways to dive into the world of computer-assisted reporting is to examine stories that have used data analysis.

So, for this assignment, I would like you to read a CAR story that’s been published in the past two years. Then, I want you to use the commenting system here to write a few paragraphs that summarize the story and how data was used. Also, note questions that you have about the story.

You can find examples of CAR stories a few different places:

Investigative Reporters and Editors Extra!Extra! investigative story blog, where you can see articles categorized as CAR.
The New York Times and USA Today regularly run data-driven stories, so you might want to poke around on their sites. Search the sites for phrases like “data analysis”, “analysis of computer records” or “computer-assisted.”

Good luck and happy digging!

Class story: Tax increment financing

David Herzog — Mon, 30 Apr 2012 18:51:21 +0000

Kip Hill got data for class about tax increment financing projects across Missouri and analyzed it for stories in the Missourian last week. I like how Kip gives readers a peek at how he did the story on the Missourian’s Watchword blog. He even provided copies of his Excel spreadsheets and a copy of his query audit trail. I don’t know how much the average Missourian site visitor will get out of the SQL, but it’s great to share it for the people who are deeply interested in the subject.

Open-source tools proving just as robust for learning data journalism

David Herzog — Wed, 14 Mar 2012 23:27:15 +0000

After a few bumps, the great open-source data journalism experiment is rolling along.

My students installed OpenOffice on their Mac or PC laptops weeks ago and have been using the Calc spreadsheet to analyze data. Its interface isn’t as polished as Microsoft Excel, but we’ve been able to make it do almost everything we need.

Calc

Calc lets you run something like Excel’s Pivot Tables to organize and summarize raw data. They’re called Data Pilots in Calc and the exact steps are a little different, but you can end up with the same kinds of results.

We did discover one small shortcoming. Calc lacks a multi-mode function that will tell you whether you have more than one mode. That’s not a deal breaker by any stretch, as you can easily determine that by running a Data Pilot and looking for the values that appear the most.

MySQL for database managers

I expected bigger obstacles with MySQL, the database manager that we’re using as a replacement for Microsoft Access. We initially had some snags when we tried to install the software on students’ laptops. First, it took too long to download the software over our university’s wireless network during class. Lesson learned. So I had my students download all of the files they needed out of class and return ready to go for the next session.

Installing and configuring the free MySQL community server, and the archived GUI tools (Administrator and Query Browser) went smoothly on the PC laptops, mostly because I’m a PC user and have run though this a few times myself. When it came to the Macs, we ran into some configuration errors, which we overcame after some Googling and consulting with DocumentCloud developer Ted Han.

It turns out that we were making things way too complicated. The fix was simple: log into MySQL locally without a password for the root user.

On the whole, I think MySQL with the Query Browser offers a better user experience than Access. You don’t have to go through a bunch of steps to find the SQL window. It’s right there at the top of the screen. Plus, you can see your queries and the results in the same screen.

MySQL also saves a history of your queries. Still, we’re copying and pasting our SQL into a text document, so we can make notes about what questions we were trying to answer.

Some of the students using Macs have opted to use SequelPro as their front-end client for querying.

One big shortcoming of MySQL is that it’s difficult for novices to import data. So instead of wrestling with the command line or using more advanced free tools, we’re going to use Navicat for MySQL. Navicat is a great commercial program that can import a crazy variety of formats, including Access. So, for tomorrow’s class, everyone is supposed to come with with the 30-day trial version installed.

Assignment: Comment on a CAR story

David Herzog — Sat, 28 Jan 2012 22:06:23 +0000

Due: Noon, Tues., Jan. 31

One of the best ways to dive into the world of computer-assisted reporting is to examine stories that have used data analysis.

You can find examples of CAR stories a few different places:

Investigative Reporters and Editors Extra!Extra! investigative story blog, where you can see articles categorized as CAR.
Another option is GreatJournalism.net, where you can find stories by data type used. Derek Willis, a newsroom developer at The New York Times, maintains this site.
The Times and USA Today regularly run data-driven stories, so you might want to poke around on their sites. Search the sites for phrases like “data analysis”, “analysis of computer records” or “computer-assisted.”

I’m eager to see what you all dig up!

The great open source experiment

David Herzog — Sun, 22 Jan 2012 23:39:44 +0000

This semester in my Computer-Assisted Reporting class we’re experimenting with open-source software.

Traditionally, I teach students how to use commercial software — Microsoft Excel spreadsheets and Access database manager — to clean and analyze data. That makes sense: Computers in many professional newsrooms already have Excel installed. Also, most students already have Excel installed on their laptops, which are mandatory at the Mizzou J-School. On the downside, most J-School students have Macs, so they’re unable to run Access. (Yes, they could run Windows on their Macs via Bootcamp or Parallels, but that’s another story and expense).

So we’re banning commercial software from one of the sections, as much as we can. Open-source software is available at no cost and is freely licensed. No need to worry about site licenses, educational licenses and the like. Also, open-source software is created by a community of developers. Open source is mainstream, with many news organizations using tools for data analysis and to power their websites.

Here’s the roster of tools that we’ll use in class:

OpenOffice Calc spreadsheet. Yes, Excel beats Calc in style and substance. But, Calc will do nearly everything journalists need, including create pivot tables (called “data pilots”) and split text into columns.
MySQL Community Server. This database manager is more robust and quicker than Access. But if you want to run it out of the box, you’ll need to learn the command line. However, MySQL has created handy GUI Tools that help you run queries in a design window like Access and administer your databases.
Notepad++ text editor. This Windows-only program lets you open and inspect huge data files that are stored as text. You can also use to write HTML and programming code. Mac users should check out TextWrangler, which is free but not open source.
Google Refine data cleaner. Refine is a new tool released by Google that you can use to understand and clean up your data.

Stay tuned here to learn more about how the experiment goes.