The One True Language

In Seven Nights, Borges has an essay where he describes the process by which he first read the Divine Comedy:

They were very handy books, published by Dent. They fit into my pocket. On the left was the Italian text, and on the right a literal translation. I devised this modus operandi: I first read a verse, a tercet, in the English prose; then I read the verse in Italian; and so on through to the end of the canto. Then I read the whole canto in English, and finally in Italian. With that first reading I realized that the translations were no substitute for the original text. The translation could be, at best, a means and a stimulus for the reader to approach the original. ... Poetry is, among so many other things, an intonation, an accentuation that is often untranslatable.

I was recently reminded of this because I decided that my digital history grad class should use the Processing programming language for their group project. Since I haven't programmed in the language before, I bought a couple of textbooks and sat down to read them, slowing when I needed to mentally translate unfamiliar commands into more familiar idioms.

Beginning programmers often worry about which language to learn first. Which one is the most powerful? The most useful? The easiest to learn? Which one will help me to get a high paying job? The investment of a semester or a year seems like a long time to study something when it might turn out to be the wrong choice. At a theoretical level, programming languages are deeply equivalent, but that is more a matter of theory than practice... because every programming language makes some things easy and some things hard. Or in the slogan of one language, "makes easy things easy and hard things possible." These language differences become the stuff of holy wars, but they shouldn't. The best language for the job depends largely on the job.

Processing, for instance, has built-in commands that make it easy to map numbers from one range of values to another. Now this isn't something that is too difficult to program from more primitive commands; it comes up frequently enough that you learn how to do it in whatever language you're using. But when I read the description of the Processing commands, I realized that I have implemented similar functions in almost every language that I've ever programmed in. By choosing to make this a language primitive, the designers of Processing made it easier for beginners to do a number of different tasks, including scaling the ranges of values returned by different analog sensors (which is something my students will need to do).

There's no one true language for programming any more than there is one true language for humanism, or one true wood for carpenters. As Borges says, the intonations of poetry are often untranslatable, and it's true for code, too. In a sense, you don't really know how to program until you're familiar with more than one language, because the essence of programming consists in knowing how to translate the idioms of one language into a more or less familiar one. And this is something that humanists have long known: if there is a oneness and truth to language, it is to be found in the multiple practices of translation.

Tags: programming

Navigating Digital History

This year, one of the first slides that I put up for the new Science, Technology and Global History class that Rob MacDougall and I are teaching was a quote from Patrick Manning's Navigating World History:

Navigating world history is an ambitious but limited goal, one quite distinct from the unattainable aim of "mastering" the topic. No one can learn all of world history. Anyone who pursues such a goal is sure to become lost. To strike an analogy, all those who have attempted to conquer the world have failed, but many of those who have traveled the globe have gained pleasure and expanded their understanding. (x)

I originally intended this forewarning as a way of managing expectations. I figured the students wouldn't be so disappointed in me when they found out that one consequence of taking on the history of everything from the Big Bang to human extinction is that the sum of the prof's knowledge asymptotically approaches zero. The students, however, seem to be taking my relative ignorance in stride, and the quote has mostly served to console me when I have to leave some stuff out of my lectures.

I was reminded of navigation the other day when I met with a PhD student who is close to finishing her doctorate and thinking about her second project. She wants to do something with digital sources but is having a hard time getting her bearings. Our conversation made me realize that I didn't have a single-page "getting started" guide for people who have never seriously worked with online sources. So here it is.

1. You won't be able to read everything. In fact, new material on your topic will appear online faster than you can read it. The longer you work on a topic, the more behind you will get. It's OK, because everyone faces this problem whether they realize it or not.

2. The first tool you should master is the search engine. Most people think that typing a word or two into the Google or Yahoo! search box is all that you need to know. Not so! First of all, search engines have an advanced search page that lets you focus in on your topic, exclude search terms, weight some terms more than others, limit your results to particular kinds of document, to particular sites, to date ranges, and so on. Second, different search engines introduce different kinds of bias by ranking results differently. You get a better view when you routinely use more than one.

3. You should have a strategy for information trapping. An explicit search is something that you do once, but the web is constantly changing. By using RSS feeds it is possible to set up a number of searches that run automatically and provide you with a constantly updated view of your subject. You can learn more about the technique in Tara Calishain's Information Trapping.

4. You can organize citations right in your browser. Until you start doing advanced work in digital history, you will access almost all of your online sources through your web browser. If you use Zotero, you can keep track of those sources in your browser, too. It really speeds up the research process.

5. It is possible to automate the process of downloading sources. There are a number of tools that make it easy to grab large batches of online sources without having to download them one at a time. In the Firefox browser, for example, you can use something like DownThemAll. Another option is GNU Wget.

6. The web is not structured like a ball of spaghetti. A lot of the most interesting information to be gleaned from digital sources lies in the hyperlinks leading into and out of various nodes, whether personal pages, documents, archives, institutions, or what have you. Search engines provide some rudimentary tools for mapping these connections, but much more can be learned with more specialized tools.

7. Assume that what you want to know is out there, and go looking for it.

Tags: digital history | history education | pedagogy

Hello World!

It's traditional when learning a new programming language to have your first program simply say "Hello world!" and terminate. It not only boosts your confidence, it signals that you've got all of the basics in place: an editor to create programs, an interpreter or compiler that can follow the instructions that you've programmed, and a way for the information to get out of the program and the computer into a form where you can make use of it. What you do at that point is up to you... hopefully more programming.

Over the last few years, I've been wrapping up my first book project: a study of how people reconstruct the past from various kinds of physical traces. My interest in the ways that material evidence and places can inform historical consciousness, and a growing interest in the potential of digital and public history, have led me to a related set of research questions. How can we use new technologies like ubiquitous / pervasive computing, ambient and tangible interfaces, and desktop fabrication to build historical interpretations into physical devices and environments? What happens when all of the bits that we've been creating through various kinds of digitization can become material atoms again? And how can this help us to better understand various pasts and make them usable in the present?

For a couple of years I've been doing projects with my students and research assistants that use technology to augment everyday places and objects, to put historical interpretations back into stuff. These projects have made use of GPS-enabled handheld and tablet computers; microcontrollers, analog sensors and actuators; and other electronic technologies. Up until now, however, we've had to buy physical components or fashion them by hand.

Last week, Adam, Devon and I had a chance to set up our new Roland Modela MDX-20 and try making something with it, a kind of physical "hello world."



The MDX-20 is a (relatively simple) computer-controlled milling machine. It is able to move a rapidly spinning, sharp tool in three dimensions, gradually removing material from a solid block, so that it comes to precisely resemble a three-dimensional model in the computer. What this means is that something that is almost purely virtual can be materialized in foam, plastic, wood, and other soft physical media. (Although our first efforts look pretty chunky, the machine is capable of much more precise contours--we have a lot to learn.) The MDX-20 also has a scanning probe, which we haven't had a chance to test yet. When used in scanning mode, the MDX-20 automates the creation of 3D models from physical objects. This allows you to start with one or more objects in the real world, scan them to create 3D models, edit or remix as desired, then replicate them in material form. At this point, the possibilities seem nearly endless.

In Thing Knowledge, the philosopher Davis Baird argues that "Things and theory can both constitute our knowledge of the world." Things can serve as models, physical representations that act in a similar way to theories. They can create phenomena, separating action "from human agency and buil[ding it] into the reliable behavior of an artifact." Or they can serve as measuring instruments, combining both representation and work (11-12). There's a long tradition of ignoring things to focus on ideas, cyberspace being one of many guises for idealism. It's time for digital humanists to say, "Hello, world!"

Tags: digitization | fabrication | history appliances | stepwise refinement

Practices, Not Products

The first week of school is a good time to expect to see Murphy's law in action. This year my server suddenly decided to start falling over at diminishing intervals. A couple of weeks of sporadic debugging have left me where I started: with an unreliable server. All of the code and images for this blog are hosted there, so they are temporarily unavailable, and I had to scramble a bit to find a new online home for the group project that my students will be doing in their grad class in digital history. This summer, too, the keepers of my old standby the online Dictionary of Canadian Biography suddenly decided to overhaul their website. While I applaud the fact that they moved from Active Server Pages to PHP, I'm not so happy that so many of the code examples in Digital History Hacks and the Programming Historian have to be revised.

I've solved my server problem for the time being, or more accurately, sidestepped it, by moving a lot of my online stuff to a new home: digitalhistory.wikispot.org. In the process I was reminded again that wikis really are the fastest and most awesome way to get your stuff online in a form that is durable but plastic enough to be continually reshaped. I can thank Raymond Yee for the inspiration. Although I've used a number of online tools, it didn't occur to me that a wiki can replace most of them until I saw Raymond give a talk at THATCamp. Rather than bust out an Open Office presentation or something like that, Raymond pointed his browser to his own wiki, a "working space / public knowledge repository". He had already entered some of the material that he wanted to talk about, and as he gave his presentation he continued to edit. When his presentation was over, he clicked 'save' and everything was already available online.

The beauty of a wiki, as many people have noted, is that it allows online material to grow quickly and organically. Rather than try to build my new online presence in one pass, I was able to sketch the outlines of what I wanted to add. Now, every time I look at the site, I see a whole bunch of work that still needs to be done. I can chip away at it, rethink, reorganize, and everything remains available to other people. On some of the pages I've roughed out sections for my students or research assistants to fill in; I expect them to chip away, rethink and reorganize, too. In effect, wiki software can provide scaffolding for practices. There's no real final product, just the most recent edit. (And, of course, access to the entire history of edits).

This year, Rob MacDougall and I are teaching a new course on science, technology and global history, and I find myself in the (exciting? unenviable?) position of writing my lectures the week before I give them. A lot of my projects feel like they may be on hold until November, when I can hand the lecturing off to Rob and start to deal with some of the changes that have broken things that used to work. I can't feel too bothered, however. All is flux, especially on the internet. The trick is to find the techniques and tools that help you deal gracefully with change, to think in clay and not in stone.

Tags: Dictionary of Canadian Biography | digital history | entropy | wikis

Traces of Use

When he figured that April was the cruelest month, I think TS Eliot was off by four. I find that the early summer stretches into an endless vista of exciting possibilities for new research and teaching. I make far too many commitments, all of which come back to haunt me in late August. Other than dropping in to do light maintenance, for example, I haven't had time recently to write much new material for the Programming Historian. The last time that I did, however, I noticed that visitor logs tell an interesting story.

To date, the front page has received around 12 thousand hits, as people arrive at the site and decide what to do next. At that point, most of them leave. They may have ended up there by accident; they may bookmark the site to look at later. The next two sections are prefatory. The first (around 4 thousand visits) suggests why you may want to learn how to program. The second (almost 5 thousand visits) tells you how to install the software that you need to get started. My interpretation is that about a fifth of our visitors are already convinced they want to learn how to program, which I think is a good sign. The actual programming starts in the next section (2 thousand visits) and goes from there (while the number of visitors for subsequent sections slowly drops to about a thousand each). These numbers could be interpreted in various ways, but to me they suggest that (1) historians and other humanists want to learn how to program, (2) good intentions only get you so far, and (3) if you do stick with it, it gets harder gradually.

These are pretty crude metrics, although more informative ones than I'm getting from, say, the sales figures for my award- winning- but- otherwise- neglected- monograph (buy a copy today!) My friends who work in psycholinguistics have much more sophisticated ways of determining how people read and understand text, with devices that track the subject's gaze and estimate the moment-by-moment contents of their short term memory. I want people to get something out of the Programming Historian, but I don't need that level of detail about what they're getting.

In The Social Life of Information, Brown and Duguid have an anecdote about a historian who goes through batches of eighteenth-century letters rapidly by sniffing bundles of them. When asked what he is doing, he explains that letters written during a cholera outbreak were disinfected with vinegar. "By sniffing for the faint traces of vinegar that survived 250 years and noting the date and source of the letters, he was able to chart the progress of cholera outbreaks." Brown and Duguid go on to note that "Digitization could have distilled out the text of those letters. It would, though, have left behind that other interesting distillate, vinegar."

Probably, but not necessarily. Digitization simply refers to the explicit digital representation of something that can be measured. We are content at the moment with devices that take pictures of documents, and those devices have been steadily improving. We wouldn't be as content with the scanning quality of 2002, when The Social Life of Information was published, and we'd, like, totally hate the scanning quality of 1982 or 1962 ... just ask my students when they have to work with microfilm. That said, high resolution infrared spectroscopy makes it possible to build chemical sniffers that outperform human noses. They also make it possible to go through an archive and digitize the smells of every document.

Saying that we can digitize any trace that we can discover and measure isn't the same thing as saying we can discover and measure any trace that we might need at the moment, episodes of CSI notwithstanding. The material world is almost infinitely informative about the past, but the traces that are preserved have nothing to do with our interests and intents. And one shouldn't draw too fine a line between the analog and the digital, because digital representations are always stored on real-world analog devices, something Matt Kirschenbaum explores in his new book Mechanisms.

Tags: analog | clues | digitization | representation

Arms Races

[Cross-posted to Cliopatria and Digital History Hacks]

Like many people who blog at Blogger, I was recently notified by e-mail that my blog had been identified by their automated classifiers "as a potential spam blog." In order to prove that this was not the case, I had to log in to one of their servers and request that my blog be reviewed by a human being. The e-mail went on to say "Automatic spam detection is inherently fuzzy, and occasionally a blog like yours is flagged incorrectly. We sincerely apologize for this error." The author of the e-mail knew, of course, that if my blog were sending spam then his or her e-mail would fall on deaf ears (as it were)... you don't have to worry about bots' feelings. The politeness was intended for me, a hapless human caught in the crossfire in the war of intelligent machines.

That same week, a lot of my e-mails were also getting bounced. Since I have my blog address in my .sig file, I'm guessing that may have something to do with it. Alternately, my e-mail address may have been temporarily blocked as the result of a surge in spam being sent from GMail servers. This to-and-fro, attack against counter-attack, Spy vs. Spy kind of thing can be irritating for the collaterally damaged but it is good news for digital historians, as paradoxical as that may seem.

One of the side effects of the war on spam has been a lot of sophisticated research on automated classifiers that use Bayesian or other techniques to categorize natural language documents. Historians can use these algorithms to make their own online archival research much more productive, as I argued in a series of posts this summer.

In fact, a closely related arms race is being fought at another level, one that also has important implications for the digital humanities. The optical character recognition (OCR) software that is used to digitize paper books and documents is also being used by spammers to try and circumvent software intended to block them. This, in turn, is having a positive effect on the development of OCR algorithms, and leading to higher quality digital repositories as a collateral benefit. Here's how.

• Computer scientists create the CAPTCHA, a "Completely Automated Public Turing test to tell Computers and Humans Apart." In essence, it shows a wonky image of a short text on the screen, and the (presumably human) user has to read it and type in the characters. If they match, the system assumes a real person is interacting with it.
• Google releases the Tesseract OCR engine that they use for Google Books as open source. On the plus side, a whole community of programmers can now improve Tesseract OCR. On the minus side, a whole community of spammers can put it to work cracking CAPTCHAs.
• In the meantime, a group of computer scientists comes up with a brilliant idea, the reCAPTCHA. Every day, tens of millions of people are reading wonky images of short character strings and retyping them. Why not use all of these infinitesimal units of labor to do something useful? The reCAPTCHA system uses OCR errors for its CAPTCHAs. When you respond to a reCAPTCHA challenge, you're helping to improve the quality of digitized books.
• The guys with white hats are also using OCR to crack CAPTCHAs, with the aim of creating stronger challenges. One side effect is that the OCR gets better at recognizing wonky text, and thus better for creating digital books.

Tags: machine learning | optical character recognition (OCR) | Turing test

Towards a Computational History

[Cross-posted to Cliopatria & Digital History Hacks]

Given that relatively few of our colleagues are familiar with digital history yet--and that those of us who practice some form of it aren't sure what to call it: digital history? history and computing? digital humanities?--it may seem a bit perverse to start talking about computational history. Nevertheless, it's an idea that we need, and the sooner we start talking and thinking about it, the better.

From my perspective, digital history simply refers to the idea that many of our potential sources are now online and available on the internet. It is possible, of course, to expand this definition and tease out many of its implications. (For more on that, see the forthcoming interchange on "The Promise of Digital History" in the September 2008 issue of The Journal of American History). To some extent we're all digital historians already, as it is quickly becoming impossible to imagine doing historical research without making use of e-mail, discussion lists, word processors, search engines, bibliographical databases and electronic publishing. Some day pretty soon, the "digital" in "digital history" is going to sound redundant, and we can drop it and get back to doing what we all love.

Or maybe not. By that time, I think, it will have become apparent that having networked access to an effectively infinite archive of digital sources, and to one another, has completely changed the nature of the game. Here are a few examples of what's in store.

Collective intelligence. Social software allows large numbers of people to interact efficiently and focus on solving problems that may be too difficult for any individual or small group. Does this sound utopian? Present-day examples are easy to find in massive online games, open source software, and even the much-maligned Wikipedia. These efforts all involve unthinkably complex assemblages of people, machines, computational processes and archives of representations. We have no idea what these collective intelligences will be capable of. Is it possible for an ad hoc, international, multi-lingual group of people to engage in a parallel and distributed process of historical research? Is it possible for a group to transcend the historical consciousness of the individuals that make it up? How does the historical reasoning of a collective intelligence differ from the historical reasoning of more familiar kinds of historian?

Machines as colleagues. Most of us are aware that law enforcement and security agencies routinely use biometric software to search through databases of images and video and identify people by facial characteristics, gait, and so on. Nothing precludes the use of similar software with historical archives. But here's the key point. Suppose you have a photograph of known provenance, depicting someone in whom you have an interest. Your biometric software skims through a database of historical images and matches your person to someone in a photo of a crowd at an important event. If the program is 95% sure that the match is valid, are you justified in arguing that your person was in the crowd that day?

Archives with APIs. Take it a step further. Most online archives today are designed to allow human users to find sources and read and cite them in traditional ways. It is straightforward, however, for the creators of these archives to add an application programming interface (API), a way for computer programs to request and make use of archival sources. You could train a machine learner to recognize pictures of people, artifacts or places and turn it loose on every historical photo archive with an API. Trained learners can be shared amongst groups of colleagues, or subject as populations to a process of artificial selection. At present, APIs are most familiar in the form of mashups, websites that integrate data from different sources on-the-fly. The race is on now to provide APIs for some of the world's most important online archival collections.

Models. Agent-based and other approaches from complex adaptive systems research are beginning to infiltrate the edges of the discipline, particularly amongst researchers more inclined toward the social sciences. Serious games appeal to a generation of researchers that grew up with not-so-serious ones. People who might once have found quantitative history appealing are now building geographic information systems. In every case, computational processes become tools to think with. I was recently at the Metropolis on Trial conference, loosely organized around the 120 million word online archive of the Old Bailey proceedings. At the conference, historians talked and argued about sources and interpretations, of course, but also about optical character recognition and statistical tables and graphs and search results generated with tools on the website. We're not yet at a point where these discussions involve much nuanced analysis of layers of computational mediation... but it is definitely beginning.

Tags: computational history | digital history