Web Semantics: Toiling in the Data Mines

*Listen to the way this infoviz guy describes his daily labors. It truly is a strange and foreign territory.

http://berglondon.com/blog/2009/10/23/toiling-in-the-data-mines-what-data-exploration-feels-like/

(…)

“Programmers very rarely talk about what their work feels like to do, and that’s a shame. Material explorations are something I’ve really only done since I’ve joined BERG, and both times have felt very similar – in that they were very, very different ...

to writing production code for an understood product. They demand code to be used as a sculpting tool, rather than as an engineering material, and I wanted to explain the knock-on effects of that: not just in terms of what I do, and the kind of code that’s appropriate for that, but also in terms of how I feel as I work on these explorations. Even if the section on the code itself feels foreign, I hope that the explanation of what it feels like is understandable.

“Material explorations

“BERG has done material explorations before – they were a big part of our Nokia Personalisation project, for instance – and the value of them is fairly immediate when the materials involved are things you can touch.

“But Ashdown is a software project for the web – its substrate is data. What’s the value of a material exploration with an immaterial substrate? What does it look like to perform such explorations? And isn’t a software project usually defined before you start work on it?

“Not always. Invention comes from design, and until the data’s been exposed to designers in a way that they can explore it, and manipulate it, and come to an understanding of what design is made possible by the data, there essentially is no product. To invent a product, we need to design, and to design, we need to explore the material. It’s as simple as that….”

(((It’s simple as that, okay, but about five paragraphs into the datamines my canary died

and I started hallucinating:)))

“because we’re working on very large chunks of data, it’s important that any long running processes – scrapers, parsers, processors – need to be really granular

“it’s not just enough to have big lists, and vast, RESTful interfaces on top of blobs of data

“Because now isn’t the time to be shaving too many yaks, when it comes to visualisation libraries and tools

“able to mouse around a pie chart

“It can also be overwhelming…. That dataset – its meaning, its structure – gets stuck in your head, and it’s easy to lose yourself to it. That often makes it harder to explain to others – you start talking in a different language – so it becomes critical to get it out of your head and onto screens.

“It also feels lonely in the data-mines at times. Not because you’re the only person working on it, but because no-one else can speak the language you do; the deeper you get into the data, the harder you have to work to communicate it, and the quicker you forget how little anyone else on the project knows…”

Hide