The computer code behind EveryBlock was released this week, and it was — for programming journalists, at least — a much-anticipated event.
Thanks to a $1.1 million Knight News Challenge grant, Adrian Holovaty and a team of five other programmers and designers developed a site that gathers disparate public data (crime stats, real estate information, business licenses and news feeds, among others) and displays it by location.
When EveryBlock launched in January 2008, it demonstrated the value of sorting local news by location.
Although EveryBlock itself now covers 15 cities, the grant conditions included making the code behind the project publicly available.
William Hartnett, online innovations editor at The Palm Beach (Fla.) Post, has been keeping a close eye on EveryBlock, and he sees the chance to peer inside as a huge opportunity. “I don’t know anyone else that is as sophisticated with location-based data. What they do is exciting, and they’ve been working on it for a while now.”
For the journalism programmers I spoke with, much of the excitement stems from the chance to see how Holovaty’s brain works. Holovaty created the Django Web application framework used by a number of news organizations, and getting to see the thinking behind the code is as important as the code itself.
A little detour here to denerdify: Web application frameworks (in this case, Django) are written in a programming language (in this case, Python). These frameworks sit in the middle ground between programming languages and finished software. As Jeremy Bowers, news technologist at the St. Petersburg (Fla.) Times explains it, the programming language is iron and lumber, the web app framework is the hammer and nails. The chance to see how the guy who built the tools used them to build an elegant house is an important one.
Ben Welsh, a database producer at the Los Angeles Times, said that was the big draw for him. “It’s a high profile site that is recognized across the board for its sophistication, and I was curious to see how that sophistication was achieved.”
“All of this code is really simple, clean, really well written,” Bowers said. “We’re working on something similar, but Adrian’s code is much better than mine.”
Because it’s now open source, anyone with the programming skills can take all or part of the code and reuse it. Potentially, new versions of EveryBlock could spring up around the country (although Holovaty and his team retain the rights to the name).
For the near future at least, programmers said that’s not likely. Ryan Pitts, senior editor for digital media at the Spokane (Wash.) Spokesman-Review, said they are more likely in the short term to take modules — essentially, applications inside the larger project — and use them for smaller projects. “We want to take the things that these really, really talented developers have built and plug them into places we can take advantage of.”
Derek Willis, who works for the interactive news technology group at The New York Times, and who worked with Holovaty at The Washington Post, says this is the beauty of Holovaty’s theory of programming. “One of his philosophies is to have things that are useful in and of themselves, not just tied to a particular program. The code to extract addresses from blocks of text isn’t buried in a pile of code.”
That utility of pulling parts out of the whole is what’s most important about this release for Willis. “I think it will help a lot of people. It’s a great gift.”
It hasn’t taken long for this regifting to begin. At the St. Petersburg Times, Bowers already has the wiki module up and running, and is getting it ready to use as an internal development tool.
This is exactly what the Knight Foundation was hoping would happen, said Gary Kebbel, journalism program director of the Knight Foundation. “If we fund something and it’s open source, it can be the gift that keeps on giving. We want to make it easier for more people to join the digital conversation and the easiest way to do that is to make it free.”
Less than 24 hours after the code was released, someone has taken the code that pulls out the address information from text (like a newspaper story), created a little utility, and placed it on a Google application server.
This “natural language” tool is one that news developers are particularly interested in, along with the ability to handle massive amounts of data.
Over and over again, programmers mentioned the growing importance of location awareness in journalism.At The Indianapolis Star, digital development director Chris Vannoy is working on what he jokingly referred to as “EveryBlock-Like.” While large aspects of the EveryBlock code will not apply directly to their project, which is written in a different programming language and contains static information on schools and restaurants, he’s excited about the ability to tease addresses out of text, then geocode it.
“We built something similar, but we never could quite get it right. It’s about 80-90 percent accurate,” Vannoy said. “If we can get something more accurate, and it looks like it will be, that will be something we can use.”
And the potential for more is there. Vannoy says, “There’s nothing stopping a news org from taking it directly or tailoring it to what they want to do. Local news is more than the city they live in.”
Over and over again, the growing importance of location awareness in journalism — the ability to connect the events of the day to a particular spot on the planet — was mentioned, along with EveryBlock’s role in paving the way for more news organizations to deliver news by location.
Bowers believes that “every new geocoding app that springs up will inevitably start with this.”
For Pitts, looking at how the data is handled is going to help their own best practices. “They’re really, really good at ingesting data from all sorts of sources. How are they doing that, what can we learn from that?”
Still, implementing a project like EveryBlock is not for the faint of heart. Welsh warns that it’s not something that someone new to Django will be able to just pick up and use. “The learning curve is pretty high. It’s advanced stuff.”
“If you think you just turn it on and put some ads on it and it will make money for you, you’re high,” said Matt Waite, news technologist at the St. Pete Times. “By the same token, it’s a real gift for the people like us who do this to see how the guy who built the framework eats his own dog food.” (With a silver spoon, Bowers added.)
Welsh has just gotten started digging into the code, and says he wants to spend a weekend sinking in to it and asking, “What can I think of that might be a spin off these tools?”
That’s just fine with Kebbel. “My goals are that news organizations, other organizations, anybody, uses it in ways or applications or processes that we haven’t even thought of yet. If it spreads, that is a measure of success. The numbers of times it’s used and the number and different types of people that use it, those are all measures of success.”
Even though that’s the intent of the project, Welsh said he feels a little guilty getting to use someone else’s work when “it’s so clear how much hard work went into it.”
And that’s what Holovaty said he is hoping people recognize: “A LOT of love and hard work has gone into it over the past two years. I hope that whoever uses it appreciates it and uses it for good, not evil.”
Disclosure: Adrian Holovaty is a member of Poynter’s National Advisory Board. The St. Petersburg Times is owned by The Poynter Institute.
