An Experimental Interface

This whole week I'm at the Doing Digital History workshop sponsored by the Center for History and New Media at George Mason University. There's a great bunch of people, a lot of interesting sessions and good ideas flying around. In short, it's nerdvana for digital historians. One of the workshop activities is regular blogging so I will try to post something here at the end of each day.

The first activity today was to self-organize into small groups and study a number of different websites, talking them over together while surfing through them. In itself, that was an interesting activity; as Josh Greenberg noted, we rarely surf as a communal or conversational practice. My favorite site was HistoryWired at the Smithsonian, which has an admittedly experimental interface that uses something like a heatmap (aka treemap) to cross-classify 450 interesting objects from the 3 million the institution has. It takes a while to get used to the HistoryWired interface, and a number of the workshop participants found it to be too visually busy and resource-intensive for their taste. What I liked about it was that the more that I played with it, the more features I discovered. It is possible to zoom into a particular region of the collection, to explore classes of artifacts with a timeline, and to get an immediate visual sense of the overlap of particular categories. Given the roots of this kind of representation in data mining and visualization, I imagine that the interface would scale up quite nicely if it were used as the front-end for a very large collection of sources. Judging from the criticism of the workshop group, I suspect that this is not a successful way to present history to the general public, but it could be a very useful exploratory tool for some kinds of research.

Tags: data mining | Doing Digital History workshop | visualization

Experimenting with the TAPoR Tools

This summer I'm in the process of developing a new graduate course on digital history. One of the things that we will study is the creation of online historical materials, and for this, I plan to assign Cohen and Rosenzweig's Digital History. I would also like to emphasize the new computational techniques that historians will increasingly need to use with digital sources. This raises some interesting challenges. I can't assume that my students will know how to program or that they will be familiar with markup languages like HTML or XML. We don't even really have time for the systematic exploration of a particular language, like Perl. (Although we will have time for some fun stuff.) I've decided to focus on specific problems faced by historians working in the digital realm, and show how computation makes them tractable. I'll say more about the course in future posts; for now, suffice it to say that it will teach stepwise refinement, be very hands-on and, no doubt, a bit hackish.

The beta release of the TAPoR Text Analysis Portal gives students the chance to experiment with text processing without having to code everything from scratch. It allows the user to enter the URL of a digital source and then explore the text with an interactive concordance.

For example, suppose you want to get (or convey) a sense of how the historian's job of interpretation can be augmented with computational tools. Go to the online Dictionary of Canadian Biography and choose an entry at random. I picked Robert McLaughlin, someone with whom I wasn't already familiar. Using the TAPoR tool it is possible to find the most frequently occurring distinctive words and phrases in McLaughlin's biography:

mclaughlin carriage

in oshawa

company

toronto

motor

automobiles

business

It is also possible to get information about keywords in context. For example, clicking on "carriage" returns the following:

Enniskillen, where he built a	carriage	works, which, in at least
him to build the Oshawa	Carriage	Works, a three-storey brick
which became known as McLaughlin	Carriage	about , was facilitated by careful
new designs (some influenced by	Carriage	Monthly, a Philadelphia journal), and
patents (and buying others), refining	carriage	mechanisms, tabulating the credit ratings
mostly wholesale business of McLaughlin	Carriage	is all the more impressive
transportation. Boosted as the largest	carriage	maker in the British empire

Without reading the biography yet, I can now guess that Robert McLaughlin lived in Oshawa and founded a carriage works which became very successful. At this point, it is reasonable to object that I could have learned the same thing by reading his biography. The point, however, is that a computer can't learn by reading, but it can make use of text processing to produce more useful output. For example, suppose you wanted to create a "smarter" search engine. If you type "Robert McLaughlin" into Google, you get the following results.

1. An art gallery in Oshawa
2. (ditto)
3. Bible Ministries
4. (ditto)
5. A photographer in Glasgow
6. An art gallery in Oshawa
7. A book about the battle of Okinawa in WWII
8. A role-playing game called "Cthulhu Live"
9. Realtors in New Jersey
10. The blog of a Californian graphic artist

Now these results have less to do with one another than the animals in Borges' "Chinese Encyclopedia". But what if your search engine was to recognize "Robert McLaughlin" as a proper name, first submit the search to the Dictionary of Canadian Biography, process the text for keywords and then submit the query "Robert McLaughlin"+oshawa+carriage to Google? Then the first ten results would look like this:

1. A Wikipedia entry on Oshawa with information about the McLaughlin Carriage Company
2. The Answers.com entry on Oshawa with information about the McLaughlin Carriage Company
3. A popular history website (Mysteries of Canada) with an article about the McLaughlin Carriage Company and General Motors
4. The history page of the City of Oshawa website with information about McLaughlin and his carriage company
5. An art gallery in Oshawa
6. The Canadian Encyclopedia entry on Oshawa with information about the McLaughlin Carriage Company
7. The Oshawa Community Museums and Archives page about the McLaughlin Carriage Company
8. An art gallery in Oshawa
9. An article about McLaughlin from the Financial Post, reproduced by the Business Library at the University of Western Ontario
10. A history page on the GM Canada website which talks about McLaughlin and his company

Tags: concordance | dictionary of canadian biography | digital history | history education | pedagogy | search | stepwise refinement | text mining

Blogging and/as Stepwise Refinement

In the foreward to the Wizard Book, Alan J. Perlis writes that "The programmer must seek both perfection of part and adequacy of collection." He goes on to say that

Every computer program is a model, hatched in the mind, of a real or mental process. These processes, arising from human experience or thought, are huge in number, intricate in detail, and at any time only partially understood. They are modeled to our permanent satisfaction rarely by our computer programs. Thus even though our programs are carefully handcrafted discrete collections of symbols, mosaics of interlocking functions, they continually evolve: we change them as our perception of the model deepens, enlarges, generalizes until the model ultimately attains a metastable place within still another model with which we struggle.

I think that this is a very nice description of what it's like to think like a programmer. Furthermore, it suggests the kind of role that blogging can play in the research process. Every time we post, we struggle to find a balance between getting it right and getting it written. If you put too little effort into a post, it comes across as lightweight, disposable. Too much effort and you eventually have something that you might as well send to a journal. The optimal blog post is timely enough to enter the flow of communication while the topic is still of interest, and substantive enough to travel. Most blog posts aren't optimal, of course, but that shouldn't stop us from trying.

So much for perfection of part. What about adequacy of collection? The advantage of having a research blog is that it serves as an archive of steps taken. Sometimes they seemed promising but went nowhere; sometimes an initial mis-step turned out to be very productive. Over time, the blog as a whole becomes more focused, more refined, a better model for processes "arising from human experience or thought." That is to say that the process of blogging, much like programming, can be one of stepwise refinement.

Somewhere in Discovering, Root-Bernstein has an anecdote about a scientist who wrote the most significant research questions on a blackboard in the lab, so they would always be in front of people and could be constantly modified to reflect new understandings. Blogs can serve the same purpose, placing an evolving set of questions and models before the members of a virtual lab.

Tags: digital history | information costs | programming | stepwise refinement | writing

Developments at Kodak (and Elsewhere)

Two interesting news items have been posted at Engadget recently about research being done at Kodak that may help to automate the digitization of historical photographs. First, new scanners have the ability to estimate the decade that a print was made based on the paper, and may someday be able to recognize watermarks or handwriting on the back. Second, a 2004 patent makes use of the red-eye effect from flash photography to determine the subject's age.

Historians, curators and archivists already have a number of techniques for dating photographs (see Joe Nickell's Camera Clues for an accessible introduction to some of them.) Digitization of any historical source, however, brings it into the realm of computation. I don't know exactly how the Kodak scanning software works, but unless it is doing some kind of physical analysis of the photographic paper itself (rather than the image of the paper) it should, in principle, work on a high resolution TIFF scanned somewhere else. In other words, it might be possible to build a spider that sifts through online archives looking for photographic prints from the 1920s.

There is a lot of interest in biometrics right now, much of it geared toward present-day concerns with security and identity. The Kodak age-detection patent suggests, however, that we may see some spinoffs for historical research. To take another example, a research group at Georgia Tech is working on programs to recognize people from their gait. It doesn't seem farfetched to imagine a system that uses biometric techniques to search through, say, old newsreel footage.

Tags: biometrics | data mining | digital history | historical photographs | new information

SIP Mapping

In an earlier post, I mentioned the fact that Amazon keeps track of phrases that are distinctive to a small set of books. These SIPs (statistically improbable phrases) can be used to get some idea of the conceptual landscape in and around particular works, and thus can be used to generate bibliographies. Ideally, of course, the process could be automated. If machine-readable versions of the books were available, it could also be used as part of a text mining project.

I haven't had a chance to do much programming recently so I thought I would put together a rudimentary hack to scrape SIPs and create a map. I also wanted to learn how to use the open source Graphviz visualization toolkit, so I used a Perl module to link to it. If you look at the code for the hack, you can see how simple it is to create pretty neat graphs. The figure below (1Mb) shows what happens when you start with Diamond's Guns, Germs, and Steel and follow the SIPs to adjacent books. The figure is more than 8,000 pixels wide, so you have to zoom in to see the detail ... and at that level it is pretty complicated. I will leave the implementation of a better graph browser for a future hack.

 

 

Tags: digital history | hacking | open source | perl | SIPs | text mining | visualization

My Folders at the McCord

I've just returned from a history education workshop in Vancouver, where I met Marie-Claude Larouche, co-ordinator of the online education program at the Musée McCord in Montreal. The McCord has digitized a large number of sources already, including more than 120,000 images, and has created a number of innovative online displays. One of my favorites is "Urban Life through Two Lenses" which uses some clever Flash programming to allow the user to superpose contemporary and historical photographs in interesting ways. The McCord has also contracted with historians to write a number of short tours of the collection, highlighting major events and aspects of everyday life.

So much is to be expected from a savvy museum in the 21st century. One of the innovative things the McCord has done, however, is to allow users to create their own tours through a My Folders mechanism. Anyone can create an account, select digital images from the McCord's collection, and use them to support a historical narrative. As an example of the potential of this, see "A Vile Style," a narrative created by Christy Yau, a student in Tom Morton's grade 10 history class at David Thompson High School in Vancouver. Using corseting as an example, Yau asks the question "Is fashion worth dying for?" She shows that nineteenth-century history can be relevant to present concerns. Her argument is well-supported by her pictorial sources, and charmingly written, e.g., "As one should know, the human body was never meant to be compressed to the point of deformity for the sake of fashion, or anything else, for that matter."

Tours written by students and members of the public are stored on the McCord's server and made available, with a disclaimer, on the web. This kind of mechanism has a number of important implications for the practice of history. For one thing, it greatly reduces the information costs associated with using a distant archive. It is easier for students at a high school in Vancouver to use the digital resources in Montreal than it would be for them to use the material resources in their own city. The McCord is also building a resource which can be used by scholars of history education and/or public history. How do people construct historical narratives from visual sources? What kinds of inferences do they think are supported or warranted by what kinds of sources? How do their understandings of particular sources differ from the interpretations of professional historians?

Tags: digital history | folksonomy | history education | information costs | pedagogy | public history

Information Costs

The basic idea of an information cost is pretty simple: it costs something to learn something. We all know that books cost money, reading takes time, universities charge tuition, archival work and fieldwork are expensive, file folders need to be stored, computers need to be replaced (frequently) and people are forgetful. Once you start to take information costs into account, however, there are surprising consequences for economic history, property rights, law, and many other fields (see, for example, the work of Douglass North, Yoram Barzel and Ronald Coase.)

We are at a point where it is possible to imagine that nearly all historical sources could become digital and readily accessible over the next few decades. This means that the relative cost of accessing any particular source will be near zero, and the practice of history will be completely transformed as a result.

Past historical projects were largely shaped by information costs, although not explicitly framed in those terms. It was easier to read through the contents of one archival box than to go through a number of different boxes: typically, each box had to be requested, retrieved from storage, stored in the reading room while someone was looking at it, and then returned to storage. For practical reasons, archives limited the number of boxes that could be requested at a time, and often took a substantial amount of time to process each request. It was easier to use the resources of a single archive than a number of archives. The costs of access were multiplied by travel between archives and by the need to learn the ropes at each one. Furthermore, much of the material in archives was not indexed in finding aids, and it was even more difficult to search effectively across archives.

As archives digitize their holdings, historians can no longer expect to face these costs. At the moment, it is much easier for me to examine the 80,000 historical photographs online at the BC Archives in Victoria, BC (3,285 kilometres away) than it is to study historical photographs in the regional collection of my own university library. Eventually, these kinds of discrepancies will vanish. In the meantime, however, historians are confronted with an unfamiliar and counterintuitive set of information costs as they approach new projects, or advise students beginning research.

In the long run, the complete digitization of our archival base may be accompanied by the emergence of a separate field of historical informatics. The current situation in biology is instructive. At first glance, the stuff of biology—genes, cells, organisms, ecosystems, and so on—would seem to have little to do with information processing. The past few decades, however, have seen the emergence of bioinformatics, an explicitly computational form of biology. Students in many areas of the life sciences now find that they need a basic understanding of statistics, applied math, and programming. Precisely the kind of things that young historians need to start learning now.

In a sense, the information-processing revolution in history is one part of a much larger and longer-term trend that J. R. and William H. McNeill have traced in The Human Web. Patterns of interaction and exchange have become ever denser and faster over the course of the Holocene, with a consequent reduction in information costs.

Tags: digital history | historiography | information costs