One of my projects this summer is to put together a new graduate course on digital history (Hist513F), primarily for our public history students, but open to others. My goal is to provide a course that reflects the kind of work that I've been doing in this blog, while recognizing the fact that many of the students in the course may have never programmed before. Following the O'Reilly hacks model, I've come up with three categories of lab exercises: easy (no programming), medium (a bit of programming) and hard (a lot of programming). My hope is that students will be able to find something fun and challenging to do, whatever their level of expertise.
About a third of the seminar readings are drawn from multiple versions of GMU's "Clio Wired" course. The rest come from the blogosphere, online journals, and other sources. So far, I haven't made use of any gated resources, so the course should be accessible to anyone who wants to make use of it. If you have any suggestions or ideas, I'd be happy to hear from you at wturkel@uwo.ca.
Tags: digital history | history education | pedagogy | public history
Sunday, July 16, 2006
Wednesday, June 28, 2006
Retconning? Or Revision?
In a series of thought-provoking posts, Steven Shaviro has been putting up "fragments, excerpts, and outtakes" from a book that he is writing called Age of Aesthetics. In the most recent, "Retconning History," Shaviro takes on the implications of history as "a vast database of 'information'." He argues that we now tend to understand history as an algorithmic search through a constraint space, much like the computer game Civilization. Furthermore, this is not due to information technology as much as to commodity culture. "The market mechanism defines our possibilities in the present, and colonizes our hopes and dreams for the future," he writes, "so it’s scarcely surprising that it remakes the past in its own image as well."
Shaviro goes on to argue that the past, the object of history, is retroactively changed in the present, a process that he likens to the retcons that are familiar from some forms of literature. As examples, he gives Ursula K. LeGuin's novel The Lathe of Heaven, where the protagonist's dreams change the past for other characters in the story (although he remembers the earlier reality) and season 5 of the TV show Buffy the Vampire Slayer, when a new character is suddenly introduced as if she had been there all along. "But is this not the way that History actually works?" he asks. "It’s a commonplace that history is written by the victors. The past is never secure from the future." Shaviro's wider point is that "capitalism is, among other things, a vast machine for imposing its own retrospective continuity upon everything it encounters." We imagine that it is possible to write history in all-encompassing terms, and project rational, economic man back to the origin of the species.
It's a pretty bleak picture. The idea of retconning, however, suggests a way out of the matrix. Like irony, retconning only works if the reader/interpreter is able to maintain a kind of double vision. Something is ironic if we are simultaneously aware of its expected reading and our deflection from that reading. Something has been retconned if we are simultaneously aware of the past that previously was, and the past we are now presented with. Readers of The Lathe of Heaven, or viewers of Buffy, understand that the past of the story has been changed precisely because they remember how it used to be. If they didn't remember, it would be as if the change had never occurred.
Historians, too, are constantly revising their interpretation of the past. (Here I would disagree with Shaviro by saying that it is not the past that is retroactively changed, but history. The past is gone, inaccessible, whereas history is what we make of it.) That constant revision is understood to be revisionist precisely because we remember what earlier interpretations were. Unlike Civilization, each new interpretation becomes part of the archive, and changes the rules of the game.
Tags: computer games | digital history | feature space | historiography | simulation
Shaviro goes on to argue that the past, the object of history, is retroactively changed in the present, a process that he likens to the retcons that are familiar from some forms of literature. As examples, he gives Ursula K. LeGuin's novel The Lathe of Heaven, where the protagonist's dreams change the past for other characters in the story (although he remembers the earlier reality) and season 5 of the TV show Buffy the Vampire Slayer, when a new character is suddenly introduced as if she had been there all along. "But is this not the way that History actually works?" he asks. "It’s a commonplace that history is written by the victors. The past is never secure from the future." Shaviro's wider point is that "capitalism is, among other things, a vast machine for imposing its own retrospective continuity upon everything it encounters." We imagine that it is possible to write history in all-encompassing terms, and project rational, economic man back to the origin of the species.
It's a pretty bleak picture. The idea of retconning, however, suggests a way out of the matrix. Like irony, retconning only works if the reader/interpreter is able to maintain a kind of double vision. Something is ironic if we are simultaneously aware of its expected reading and our deflection from that reading. Something has been retconned if we are simultaneously aware of the past that previously was, and the past we are now presented with. Readers of The Lathe of Heaven, or viewers of Buffy, understand that the past of the story has been changed precisely because they remember how it used to be. If they didn't remember, it would be as if the change had never occurred.
Historians, too, are constantly revising their interpretation of the past. (Here I would disagree with Shaviro by saying that it is not the past that is retroactively changed, but history. The past is gone, inaccessible, whereas history is what we make of it.) That constant revision is understood to be revisionist precisely because we remember what earlier interpretations were. Unlike Civilization, each new interpretation becomes part of the archive, and changes the rules of the game.
Tags: computer games | digital history | feature space | historiography | simulation
Monday, June 26, 2006
Setting up a Server
Up until now, all of the hacks that I've posted here have been client-side and somewhat labor-intensive. If you want to try it yourself (and that, after all, is the point of hacking) then you have to set up Perl on your own computer, download the appropriate files and start modifying them.
For some time, I've been planning to buy and set up a server, and the recent Doing Digital History workshop at the Center for History and New Media at GMU gave me the impetus to do so. So I ordered a server, and, in the meantime, have been doing test installations of server software on my home machine, so I have some idea of what to expect. The new server will be a not-very-expensive Windows XP Pro machine with 2Gb of RAM and two 320Gb hard drives. I will be using open source software whenever possible. After three days of installing, testing, making mistakes, uninstalling, and trying again, I've come up with the following sequence.
1. Install XAMPP. In order to run a minimal server, you're going to need a webserver (Apache), a database server (MySQL), FTP client and server (FileZilla), a way to make dynamically generated web pages (PHP) and an all-purpose programming language (Perl). In my first few attempts, I tried installing these packages separately and then trying to get them to work together. This turned out to be less straightforward than I had hoped, and I was in the process of trying to debug my installation when I discovered that it is possible to install everything you need in one pass. This is not only much faster, but everything works together as advertised. Once the basic XAMPP package is installed, you will want to install the Perl and mod_perl addons, and fix potential security holes. Before the following step, you will also need to enable CURL for PHP.
2. Install WordPress. At the workshop, I was very impressed with the way that Josh Greenberg was able to quickly set up a workshop website, blog, syllabus, links to readings, individual blogs for class members, group blog feed and wiki. The magic behind this was WordPress, a popular open source package. If CURL is enabled for PHP (see step 1) it is very easy to import your existing blogs from Blogger and other popular sites. There are also a number of different themes available to change the look and feel of your site or of various parts of it. After playing around with Alex King's Theme Browser, I downloaded about a dozen themes for further experimentation.
3. Install Streetprint Engine and/or Greenstone Digital Library Software. I want our public and digital history students to be able to create online repositories with a minimum of effort. These two packages both worked very well in my test installations; the former is easier to use and more oriented toward page-at-a-time display of images or artifacts, while the latter is oriented more toward libraries. I will probably install both and use whichever is more appropriate to a particular project.
4. (Optional) Install content management system. I experimented with Joomla! and Drupal and decided not to install a CMS, at least not yet. Both systems allow users to create database-driven websites without programming, but I figure that WordPress will give me most of the functionality that I need. Besides, I want my students to program!
More once the server is up and running...
Tags: blogs | digital history | hacking | open source | perl | relational databases | server
For some time, I've been planning to buy and set up a server, and the recent Doing Digital History workshop at the Center for History and New Media at GMU gave me the impetus to do so. So I ordered a server, and, in the meantime, have been doing test installations of server software on my home machine, so I have some idea of what to expect. The new server will be a not-very-expensive Windows XP Pro machine with 2Gb of RAM and two 320Gb hard drives. I will be using open source software whenever possible. After three days of installing, testing, making mistakes, uninstalling, and trying again, I've come up with the following sequence.
1. Install XAMPP. In order to run a minimal server, you're going to need a webserver (Apache), a database server (MySQL), FTP client and server (FileZilla), a way to make dynamically generated web pages (PHP) and an all-purpose programming language (Perl). In my first few attempts, I tried installing these packages separately and then trying to get them to work together. This turned out to be less straightforward than I had hoped, and I was in the process of trying to debug my installation when I discovered that it is possible to install everything you need in one pass. This is not only much faster, but everything works together as advertised. Once the basic XAMPP package is installed, you will want to install the Perl and mod_perl addons, and fix potential security holes. Before the following step, you will also need to enable CURL for PHP.
2. Install WordPress. At the workshop, I was very impressed with the way that Josh Greenberg was able to quickly set up a workshop website, blog, syllabus, links to readings, individual blogs for class members, group blog feed and wiki. The magic behind this was WordPress, a popular open source package. If CURL is enabled for PHP (see step 1) it is very easy to import your existing blogs from Blogger and other popular sites. There are also a number of different themes available to change the look and feel of your site or of various parts of it. After playing around with Alex King's Theme Browser, I downloaded about a dozen themes for further experimentation.
3. Install Streetprint Engine and/or Greenstone Digital Library Software. I want our public and digital history students to be able to create online repositories with a minimum of effort. These two packages both worked very well in my test installations; the former is easier to use and more oriented toward page-at-a-time display of images or artifacts, while the latter is oriented more toward libraries. I will probably install both and use whichever is more appropriate to a particular project.
4. (Optional) Install content management system. I experimented with Joomla! and Drupal and decided not to install a CMS, at least not yet. Both systems allow users to create database-driven websites without programming, but I figure that WordPress will give me most of the functionality that I need. Besides, I want my students to program!
More once the server is up and running...
Tags: blogs | digital history | hacking | open source | perl | relational databases | server
Wednesday, June 21, 2006
A Roundup of Digital History Blogs
There aren't quite enough digital history blogs to justify a carnival yet, but there is certainly enough interesting work going on to try and gather a collection of links about digital history and of use to digital historians. If I've missed you or someone you know, please e-mail me at wturkel@uwo.ca and I will include a link in the next roundup. Without further ado...
Jeff Barry, "Endless Hybrids." Jeff blogs about libraries and the remix culture.
John Battelle, "John Battelle's Searchblog." Everything Google, and more.
Jeremy Boggs, ClioWeb. Jeremy is a web designer at the Center for History and New Media, and a PhD candidate at GMU. He blogs about the intersection of history, new media and design. You can also catch him at the new group blog Revise and Dissent.
Tara Calishain, "ResearchBuzz." A great place to find up-to-the-minute information about internet research.
Dan Cohen. Co-author of Digital History (U Penn, 2005), Dan writes an eponymous blog about digital humanities, Google and searching, and "programming for poets." He is also the author of GMU's clever H-Bot.
Tom Daccord, "THWT Edublogger." Tom blogs about teaching humanities with technology.
Lorcan Dempsey, "Lorcan Dempsey's Weblog." Lorcan is the vice-president and chief strategist at OCLC. He blogs about libraries, services and networks.
Brian Downey, "Behind AotW." Brian is the webmaster of the site Antietam on the Web. Here in the so-called "backwash of a digital history project," he blogs on the nature of digital history.
Josh Greenberg, Epistemographer. Josh, Associate Director of Research Projects at CHNM, brings an STS perspective to digital humanities and Web 2.0. He hasn't posted for an embarassing 62 days, but I'm hoping to shame him into it...
Mills Kelly, edwired. Mills blogs about teaching and learning history online. In recent posts he's taken on PowerPoint, edited Wikipedia articles (and made his students do the same), and critiqued collaborative learning and best practices in education.
Rob Macdougall, Old is the New New. Strictly speaking, Rob doesn't usually blog about digital history per se. But he loves robots, and he's been writing quality posts for years longer than most of us.
David Mattison, The Ten Thousand Year Blog. David, a BC archivist, blogs about digital preservation and libraries. Among many other categories, his "history findings" tracks an interesting and growing collection of online repositories.
Musematic. A group blog about museum informatics.
ProgrammableWeb. Because the world really is your programmable oyster.
Geoffrey Rockwell, "grockwel." Geoffrey is an Associate Professor of Multimedia at McMaster and project leader for the TAPoR project. He blogs about digital humanities in general.
Tom Scheinfeldt, Found History. Tom writes about "unintentional, unconventional, and amateur history all around us." It makes for fascinating reading, and since "all around us" includes the internet, this is often digital history at the margins.
Andrew Vande Moere, "Information Aesthetics." Andrew blogs about data visualization and visual culture. This is a great place to get ideas for presenting history in a new way.
Rebecca Woods, Past Matters. This fall, Rebecca will be starting her PhD at MIT. In the meantime, she's got a brand new blog where she can discuss learning to program and trying to text mine the archives of the Old Bailey.
Michael Yunkin, Digitize Everything. Michael is a librarian at UNLV, who is "helping dig the grave of all things analog." He blogs about digitization, search and oral history.
Updates, June 24, 2006 - July 10, 2006
Manan Ahmed, Chapati Mystery. Manan, who is working on a PhD at U Chicago, blogs about a variety of topics. His recent Polyglot Manifesto is essential reading (I, II).
Sheila Brennan, Relaxing on the Bayou. Sheila blogs about both physical and online museums.
Google has a new group blog to watch, Inside Google Book Search.
Tom Goskar, Past Thinking. Tom blogs about archaeological computing from the UK.
Paula Petrik, HistoryTalk. Among other things, Paula blogs about history, markup and web/tech. She's got a great new discussion of the problems and possibilities of XHTML and CSS for online scholarship.
Rachel "I'm Too Sexy for My Master's Thesis," blogs about new publishing models and the use of technology to aid research and writing.
Richard Urban, Inherent Vice. Richard is a PhD student in library and information science who blogs about digital cultural heritage.
Scott Weingart, a research assistant on a digital humanities project at the University of Florida has just started Ludus Historiae, to bring "the absurdities of the past to the computers of the future."
Tags: blogs | digital history
Jeff Barry, "Endless Hybrids." Jeff blogs about libraries and the remix culture.
John Battelle, "John Battelle's Searchblog." Everything Google, and more.
Jeremy Boggs, ClioWeb. Jeremy is a web designer at the Center for History and New Media, and a PhD candidate at GMU. He blogs about the intersection of history, new media and design. You can also catch him at the new group blog Revise and Dissent.
Tara Calishain, "ResearchBuzz." A great place to find up-to-the-minute information about internet research.
Dan Cohen. Co-author of Digital History (U Penn, 2005), Dan writes an eponymous blog about digital humanities, Google and searching, and "programming for poets." He is also the author of GMU's clever H-Bot.
Tom Daccord, "THWT Edublogger." Tom blogs about teaching humanities with technology.
Lorcan Dempsey, "Lorcan Dempsey's Weblog." Lorcan is the vice-president and chief strategist at OCLC. He blogs about libraries, services and networks.
Brian Downey, "Behind AotW." Brian is the webmaster of the site Antietam on the Web. Here in the so-called "backwash of a digital history project," he blogs on the nature of digital history.
Josh Greenberg, Epistemographer. Josh, Associate Director of Research Projects at CHNM, brings an STS perspective to digital humanities and Web 2.0. He hasn't posted for an embarassing 62 days, but I'm hoping to shame him into it...
Mills Kelly, edwired. Mills blogs about teaching and learning history online. In recent posts he's taken on PowerPoint, edited Wikipedia articles (and made his students do the same), and critiqued collaborative learning and best practices in education.
Rob Macdougall, Old is the New New. Strictly speaking, Rob doesn't usually blog about digital history per se. But he loves robots, and he's been writing quality posts for years longer than most of us.
David Mattison, The Ten Thousand Year Blog. David, a BC archivist, blogs about digital preservation and libraries. Among many other categories, his "history findings" tracks an interesting and growing collection of online repositories.
Musematic. A group blog about museum informatics.
ProgrammableWeb. Because the world really is your programmable oyster.
Geoffrey Rockwell, "grockwel." Geoffrey is an Associate Professor of Multimedia at McMaster and project leader for the TAPoR project. He blogs about digital humanities in general.
Tom Scheinfeldt, Found History. Tom writes about "unintentional, unconventional, and amateur history all around us." It makes for fascinating reading, and since "all around us" includes the internet, this is often digital history at the margins.
Andrew Vande Moere, "Information Aesthetics." Andrew blogs about data visualization and visual culture. This is a great place to get ideas for presenting history in a new way.
Rebecca Woods, Past Matters. This fall, Rebecca will be starting her PhD at MIT. In the meantime, she's got a brand new blog where she can discuss learning to program and trying to text mine the archives of the Old Bailey.
Michael Yunkin, Digitize Everything. Michael is a librarian at UNLV, who is "helping dig the grave of all things analog." He blogs about digitization, search and oral history.
Updates, June 24, 2006 - July 10, 2006
Manan Ahmed, Chapati Mystery. Manan, who is working on a PhD at U Chicago, blogs about a variety of topics. His recent Polyglot Manifesto is essential reading (I, II).
Sheila Brennan, Relaxing on the Bayou. Sheila blogs about both physical and online museums.
Google has a new group blog to watch, Inside Google Book Search.
Tom Goskar, Past Thinking. Tom blogs about archaeological computing from the UK.
Paula Petrik, HistoryTalk. Among other things, Paula blogs about history, markup and web/tech. She's got a great new discussion of the problems and possibilities of XHTML and CSS for online scholarship.
Rachel "I'm Too Sexy for My Master's Thesis," blogs about new publishing models and the use of technology to aid research and writing.
Richard Urban, Inherent Vice. Richard is a PhD student in library and information science who blogs about digital cultural heritage.
Scott Weingart, a research assistant on a digital humanities project at the University of Florida has just started Ludus Historiae, to bring "the absurdities of the past to the computers of the future."
Tags: blogs | digital history
Sunday, June 18, 2006
Refracted Footnotes
In my last post, I suggested that broken links, although typically perceived to be a problem, might also serve as an interesting historical source if properly mined. Today I'd like to address a related topic: the fact that the content of websites changes even while the URL remains the same. In this blog, for example, I've made a lot of references to articles in the constantly changing, user-contributed Wikipedia. This means that the article on concordances that I cited on 29 Jan 2006 has changed since I cited it ... 16 times, in fact.
The reason that I know how many times the article has changed is that Wikipedia has a 'history' page for each article that keeps track of revisions and allows the user to compare selected versions of the article. The most recent change to the article on concordances (as of this post) was to fix a misspelling of the word 'frequently'.
A sophisticated Wikipedia user consults the history page before reading the article, much the way that EH Carr recommended that we study the historian before reading his or her work. It soon becomes clear that some articles offer a relatively stable interpretation, while others are subject to ceaseless revision. Although similar processes can be observed in more traditional historical discourse, the rapidity with which Wikipedia changes, and its extensive and automatic philological apparatus make it a natural laboratory for experiments in digital history. Mills Kelly, for example, has blogged about his experiences trying to change the article on the fate of the Donner Party, widely believed to have resorted to cannibalism when snow bound in the Sierra Nevada in 1846. Mills revised Wikipedia to reflect new work in historical archaeology which calls this interpretation into question, and then watched as other people revised his changes toward a temporary consensus. Mills has also assigned the task of writing Wikipedia entries and watching them be revised to his grad students. [For more on his experiments, see What's for Dinner? (Cont'd 1) (Cont'd 2) and Whither Wiki?]
As with other forms of new media, we need to be teaching young historians how to read wikis critically and how to write them effectively. We should also be aware of the new resources that they provide for computational historiography. To take a single example, look at the "history flow visualization" created by Martin Wattenberg and Fernanda Viégas, which maps the revisions of the Wikipedia article on evolution.
So is all of this completely new? Not really. As any work is reinterpreted over time, citations to that work will change in meaning, too. Traditional historical works age along with the literature that they cite. As a physical analog, we might think of refraction: a wave bending when it enters a new medium. There's a good description of the phenomenon at Wikipedia ... at least there is right now.
Tags: bibliography | citation | data mining | digital history | historiography | public history | Wikipedia
The reason that I know how many times the article has changed is that Wikipedia has a 'history' page for each article that keeps track of revisions and allows the user to compare selected versions of the article. The most recent change to the article on concordances (as of this post) was to fix a misspelling of the word 'frequently'.
A sophisticated Wikipedia user consults the history page before reading the article, much the way that EH Carr recommended that we study the historian before reading his or her work. It soon becomes clear that some articles offer a relatively stable interpretation, while others are subject to ceaseless revision. Although similar processes can be observed in more traditional historical discourse, the rapidity with which Wikipedia changes, and its extensive and automatic philological apparatus make it a natural laboratory for experiments in digital history. Mills Kelly, for example, has blogged about his experiences trying to change the article on the fate of the Donner Party, widely believed to have resorted to cannibalism when snow bound in the Sierra Nevada in 1846. Mills revised Wikipedia to reflect new work in historical archaeology which calls this interpretation into question, and then watched as other people revised his changes toward a temporary consensus. Mills has also assigned the task of writing Wikipedia entries and watching them be revised to his grad students. [For more on his experiments, see What's for Dinner? (Cont'd 1) (Cont'd 2) and Whither Wiki?]
As with other forms of new media, we need to be teaching young historians how to read wikis critically and how to write them effectively. We should also be aware of the new resources that they provide for computational historiography. To take a single example, look at the "history flow visualization" created by Martin Wattenberg and Fernanda Viégas, which maps the revisions of the Wikipedia article on evolution.
So is all of this completely new? Not really. As any work is reinterpreted over time, citations to that work will change in meaning, too. Traditional historical works age along with the literature that they cite. As a physical analog, we might think of refraction: a wave bending when it enters a new medium. There's a good description of the phenomenon at Wikipedia ... at least there is right now.
Tags: bibliography | citation | data mining | digital history | historiography | public history | Wikipedia
Tuesday, June 13, 2006
Broken Links
Day two of CHNM's Doing Digital History workshop kicked off with a topic close to my heart: data and text mining. One of the things that we discussed was link analysis (also known as graph mining or relational data analysis), the ability to exploit connections between entities as a way of making inferences or refining searches. Google makes a different, but related, use of links in its page rank algorithm.
The discussion got me to thinking about broken links. We've all had the experience of clicking on a hypertext link and getting an HTTP 404 error, the message that the requested file cannot be found. It seems to be generally accepted that broken links are a bad thing. It is quite easy to write scripts that check each of the links on a web page and report on broken ones. If you don't want to write your own script, the W3C has an online link checker. In fact, a 2004 article in the BBC Technology News offered the hope that broken links might one day be eliminated altogether.
The article describes research done by student interns in conjunction with IBM. Their system follows working links to create a "fingerprint" of each page that is linked to. It can then determine when content of the target page changes and notify system administrators or even change the link automatically. Such a system would reduce lost productivity and prevent large corporations from getting into the embarassing position of linking to an innocuous site only to have it change into something disreputable.
So far, so good. But what if we consider broken links to be a kind of historical evidence? Existing link checkers can already spider through whole sites looking for broken links. Rather than fixing them (or before fixing them) why not compile an archive of them to study? We could ask why links get broken in the first place. Sure, some are bound to be typos. But in many cases, the target site has moved to a different address, and this continual process of renaming reflects other processes: of rebranding, search engine positioning, system or business process reorganization, and so on. We should take a page from the archaeologists' book, and pay more attention to what our middens have to tell us.
Tags: broken links | document fingerprinting | Doing Digital History workshop | link analysis
The discussion got me to thinking about broken links. We've all had the experience of clicking on a hypertext link and getting an HTTP 404 error, the message that the requested file cannot be found. It seems to be generally accepted that broken links are a bad thing. It is quite easy to write scripts that check each of the links on a web page and report on broken ones. If you don't want to write your own script, the W3C has an online link checker. In fact, a 2004 article in the BBC Technology News offered the hope that broken links might one day be eliminated altogether.
The article describes research done by student interns in conjunction with IBM. Their system follows working links to create a "fingerprint" of each page that is linked to. It can then determine when content of the target page changes and notify system administrators or even change the link automatically. Such a system would reduce lost productivity and prevent large corporations from getting into the embarassing position of linking to an innocuous site only to have it change into something disreputable.
So far, so good. But what if we consider broken links to be a kind of historical evidence? Existing link checkers can already spider through whole sites looking for broken links. Rather than fixing them (or before fixing them) why not compile an archive of them to study? We could ask why links get broken in the first place. Sure, some are bound to be typos. But in many cases, the target site has moved to a different address, and this continual process of renaming reflects other processes: of rebranding, search engine positioning, system or business process reorganization, and so on. We should take a page from the archaeologists' book, and pay more attention to what our middens have to tell us.
Tags: broken links | document fingerprinting | Doing Digital History workshop | link analysis
Monday, June 12, 2006
A Search Engine for 17th-Century Documents
One of the questions raised in the discussions today at the Doing Digital History workshop was how to make a search engine that returned documents written in a particular era. I'm assuming that we don't have access to metadata (that would be too easy). Here's the plan that I came up with. I haven't had a chance to try it yet, but it might make for some interesting hacking later on. Take the pages returned by a standard search engine, filter out any HTML or other markup tags and pass the plain text through a part-of-speech tagger to identify how each word is being used in context. After tagging, remove all of the stop words, things like 'and', 'the', 'is', etc. Now check each of the remaining words against an online etymological dictionary (like the Oxford English Dictionary) to determine the earliest attested date for each, and note whether the word has since fallen into disuse. You should end up with a vector of dates, the latest of which will put a bound on the earliest that the document could have been written. Less-common words will tend to be better indicators of date than more common ones, so it might help to take overall word frequency into account in the algorithm. The earliest date that this blog post could have been written, for example, would be bounded by the earliest attested dates of 'metadata' (1969), 'search engine' (1984), 'HTML' (1993) and 'blog post' (1999).
Tags: dictionaries | Doing Digital History workshop | etymology | search
Tags: dictionaries | Doing Digital History workshop | etymology | search
Subscribe to:
Posts (Atom)
