One of the great benefits of having a blog has been that people who are interested in digital history find me and let me know what they are doing in the field. For a couple of years now, I've enjoyed an intermittent but invariably thought-provoking correspondence with Tim Hitchcock, one of the creators of the wonderful digital archive of the Old Bailey proceedings. The OB team has recently added records for the period from 1834 to 1913, resulting in a total of almost 200,000 trial records, all tagged with XML. When Tim offered me access to the XML files for a data mining project a few months ago, I jumped at the chance. This is still very much work in progress, but I've decided to blog about the process for others who are interested in doing similar things, whether with the Old Bailey archive or some other.
I started by downloading local copies of all of the files. This is usually a good idea both because it makes the processing faster and because you aren't hammering the archive's servers every time you need to access a record. There are a number of different ways to do something like this, and it is very handy for historians to be familiar with at least some of them. One possibility is to use a Firefox extension like DownThemAll. This allows you to download all of the links or images in a webpage. It also allows you to pause and resume the download process, which can be useful when you're working with a large number of files. For those who are more comfortable with scripting and prefer command line tools, it is hard to beat GNU Wget. Both programs are free. The third alternative is to write your own script in a language like Python or Perl. This option is most difficult, but gives you more control over various kinds of preprocessing, like dealing with accented characters. (For more, see the section on this in The Programming Historian.) It takes a while to download a large batch of files, but once you have them you're ready to move on to the next step.
Tags: archive | data mining | digital history | feature space | machine learning | text mining
Saturday, May 24, 2008
Saturday, May 17, 2008
Geo-DJ, Part 3: The Simplest Working Version
Some people may have the ability to come up with something awesome on their first pass--say, Athena springing from the forehead of Zeus fully formed--but I've learned that I have to make some mistakes along the way. So I try to come up with the simplest working version of a project, then complexify it gradually. Of course, things being what they are, you can usually improve something by simplifying it, so the first, apparently simplest, version is actually somewhere in the middle of the scale from perfect to perfectly foobar.
With the geo-DJ, I imagine the simplest working version to be something like a metal detector for historical landscape features. Suppose you know that there used to be an electric streetcar running through the middle of downtown, but most material traces of it have since been torn up. If you have a map of the streetcar route, you can use existing landmarks to georeference it, and determine the latitude and longitude of the endpoints (and any additional inflection points, but let's ignore those and work with a purely linear feature). The locations of the endpoints need to be stored in memory.
As the user walks around, the geo-DJ loops through the following algorithm. First, determine the user's current position. Then, determine the line through the endpoints (the former rail), determine the length of the perpendicular line from the user to the rail line (i.e., the magnitude of a normal vector), and scale the pitch of a tone that is playing in the headphones. Repeat, ad infinitum... or until the batteries drain, whichever comes first. If the user steps toward the rail, the pitch of the sound increases. If he or she steps away from it, the pitch decreases. Using this version of the system, a person can explore the lineaments of landscape features which may no longer exist. See Michal Migurski's great air photo of San Francisco "healing" around a former railroad.
Tags: ambience | digital history | electronica | hacking | historical consciousness | history appliances | place | place-based computing
With the geo-DJ, I imagine the simplest working version to be something like a metal detector for historical landscape features. Suppose you know that there used to be an electric streetcar running through the middle of downtown, but most material traces of it have since been torn up. If you have a map of the streetcar route, you can use existing landmarks to georeference it, and determine the latitude and longitude of the endpoints (and any additional inflection points, but let's ignore those and work with a purely linear feature). The locations of the endpoints need to be stored in memory.
As the user walks around, the geo-DJ loops through the following algorithm. First, determine the user's current position. Then, determine the line through the endpoints (the former rail), determine the length of the perpendicular line from the user to the rail line (i.e., the magnitude of a normal vector), and scale the pitch of a tone that is playing in the headphones. Repeat, ad infinitum... or until the batteries drain, whichever comes first. If the user steps toward the rail, the pitch of the sound increases. If he or she steps away from it, the pitch decreases. Using this version of the system, a person can explore the lineaments of landscape features which may no longer exist. See Michal Migurski's great air photo of San Francisco "healing" around a former railroad.
Tags: ambience | digital history | electronica | hacking | historical consciousness | history appliances | place | place-based computing
Friday, May 16, 2008
Geo-DJ, Part 2: Storage vs. Computation
In my last post, I mentioned that I'm working with a couple of talented students this summer on digital history projects, and talked a bit about Adam Crymble's Zotero translators. The other person who is working with me is Devon Elliott. Last year Devon came up with a plan to use wikis in archives and built a model of Sputnik that contained a microcontroller, a thermistor to sense temperature changes and an accelerometer to respond to motion. The information about the model's state was conveyed by modulating the frequency and duration of a beeping signal. Devon did the programming and electronics without any help from me, so I knew he would be the perfect collaborator for the geo-DJ project.
The geo-DJ is a wearable iPod-like device. As you wander around a present-day environment, it uses GPS to determine your position and synthesizes an electronic soundtrack that reflects former land-use patterns. Creating something like this wouldn't be too difficult using a lightweight laptop or a powerful handheld computer running GIS software. But we're interested in doing the project at as low a level as possible, preferably using an open source microcontroller board like Arduino.
In the history of computing, people often faced the limits of both memory capacity and processing speeds. Consider the problem of determining trigonometric functions for particular values. There are algorithms for computing the sine of an angle, but they're complicated. Before the widespread adoption of digital calculators it was common for people to use trig tables, a clear case of using more storage space to simplify or speed up calculation. With digital calculators or general-purpose computers, it is simpler and faster to punch in the calculation than to look it up in a trig table. But here is the tricky part: it may not be simpler for the computer to do the computation. The software may involve looking up the value of various trig functions in tables, even though that is not apparent to the user.
Doing the geo-DJ project on a small computer like Arduino approaches these limits in (at least) two places: GIS and music synthesis. In the case of the GIS, we want to know the person's distance from the various points, lines and polygons that are used to represent historical features of interest. There are algorithms for computing these measures, but our processor is slow and our application requires real-time feedback. It might make more sense to pre-compute the measures and store the information about distances in a multi-dimensional array. Of course, the basic amount of memory on an Arduino is also very limited, so we have to find the optimal balance. In the case of music synthesis, a similar problem arises. Sounds have complicated waveforms which can be computed or looked up in a wave table. Once again, we will have to find the right balance between storage and computation.
It may be that the platform that we're trying to use is too simple. We may have to add more memory, or dedicated signal processing hardware, or both. But that is one of the things that makes a project like this fun. By working close to computational limits we not only have more of a challenge, but more of a sense what computing used to be like, long ago, when we were kids.
Tags: ambience | digital history | electronica | hacking | historical consciousness | history appliances | place | place-based computing
The geo-DJ is a wearable iPod-like device. As you wander around a present-day environment, it uses GPS to determine your position and synthesizes an electronic soundtrack that reflects former land-use patterns. Creating something like this wouldn't be too difficult using a lightweight laptop or a powerful handheld computer running GIS software. But we're interested in doing the project at as low a level as possible, preferably using an open source microcontroller board like Arduino.
In the history of computing, people often faced the limits of both memory capacity and processing speeds. Consider the problem of determining trigonometric functions for particular values. There are algorithms for computing the sine of an angle, but they're complicated. Before the widespread adoption of digital calculators it was common for people to use trig tables, a clear case of using more storage space to simplify or speed up calculation. With digital calculators or general-purpose computers, it is simpler and faster to punch in the calculation than to look it up in a trig table. But here is the tricky part: it may not be simpler for the computer to do the computation. The software may involve looking up the value of various trig functions in tables, even though that is not apparent to the user.
Doing the geo-DJ project on a small computer like Arduino approaches these limits in (at least) two places: GIS and music synthesis. In the case of the GIS, we want to know the person's distance from the various points, lines and polygons that are used to represent historical features of interest. There are algorithms for computing these measures, but our processor is slow and our application requires real-time feedback. It might make more sense to pre-compute the measures and store the information about distances in a multi-dimensional array. Of course, the basic amount of memory on an Arduino is also very limited, so we have to find the optimal balance. In the case of music synthesis, a similar problem arises. Sounds have complicated waveforms which can be computed or looked up in a wave table. Once again, we will have to find the right balance between storage and computation.
It may be that the platform that we're trying to use is too simple. We may have to add more memory, or dedicated signal processing hardware, or both. But that is one of the things that makes a project like this fun. By working close to computational limits we not only have more of a challenge, but more of a sense what computing used to be like, long ago, when we were kids.
Tags: ambience | digital history | electronica | hacking | historical consciousness | history appliances | place | place-based computing
Saturday, May 10, 2008
Beginning in the Middle
For the past few summers, I've been taking on talented students to work on digital stuff. Rather than giving them a canned project or expecting anything in particular to happen, I usually give them a difficult problem and then step back. The results have been very encouraging, especially since I tend to choose independent students who are OK with my laissez faire approach.
One of the people who is working with me this summer is Adam Crymble. Last year he managed to come up with a low-tech public history hack, make some 3D animations, and teach himself enough HTML and CSS to hand code a web page. So for a summer project I suggested he try and write some translators for Zotero. He doesn't have any training for this, and I am of limited assistance since I don't really know JavaScript. Sink or swim, buddy!
Adam intuitively started where I would. He printed out all the code and documentation that he could get his hands on, then started using colored highlighters to focus his attention on the parts that he could understand. He also used Wikipedia, the W3 Schools, and our library's Safari subscription to O'Reilly books online. In the space of a couple of weeks, he's made great progress and learned enough so that I'm still of no use to him.
Reading other people's code is always hard, but it is one of the best ways to learn how to program. As Abelson and Sussman write in Structure and Interpretation of Computer Programs, "a computer language is not just a way of getting a computer to perform operations but rather ... a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute." The beginning programmer starts out much like a child who is acquiring a natural language: immersed in a medium produced by people who are already fluent.
Historians have a secret advantage when it comes to learning technical material like programming: we are already used to doing close readings of documents that are confusing, ambiguous, incomplete or inconsistent. We all sit down to our primary sources with the sense that we will understand them, even if we're going to be confused for a while. This approach allows us to eventually produce learned books about subjects far from our own experience or training.
I believe in eating my own dogfood, and wouldn't subject my students to anything I wouldn't take on myself. As my own research and teaching moves more toward desktop fabrication, I've been reading a lot about materials science, structural engineering, machining, CNC and other subjects for which I have absolutely no preparation. It's pretty confusing, of course, but each day it all seems a little more clear. I've also been making a lot of mistakes as I try to make things. As humanists, I don't think we can do better than to follow Terence's adage that nothing human should be alien to us. It is possible to learn anything, if you're willing to begin in the middle.
Tags: digital history | historiography | interdisciplinarity | learning
One of the people who is working with me this summer is Adam Crymble. Last year he managed to come up with a low-tech public history hack, make some 3D animations, and teach himself enough HTML and CSS to hand code a web page. So for a summer project I suggested he try and write some translators for Zotero. He doesn't have any training for this, and I am of limited assistance since I don't really know JavaScript. Sink or swim, buddy!
Adam intuitively started where I would. He printed out all the code and documentation that he could get his hands on, then started using colored highlighters to focus his attention on the parts that he could understand. He also used Wikipedia, the W3 Schools, and our library's Safari subscription to O'Reilly books online. In the space of a couple of weeks, he's made great progress and learned enough so that I'm still of no use to him.
Reading other people's code is always hard, but it is one of the best ways to learn how to program. As Abelson and Sussman write in Structure and Interpretation of Computer Programs, "a computer language is not just a way of getting a computer to perform operations but rather ... a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute." The beginning programmer starts out much like a child who is acquiring a natural language: immersed in a medium produced by people who are already fluent.
Historians have a secret advantage when it comes to learning technical material like programming: we are already used to doing close readings of documents that are confusing, ambiguous, incomplete or inconsistent. We all sit down to our primary sources with the sense that we will understand them, even if we're going to be confused for a while. This approach allows us to eventually produce learned books about subjects far from our own experience or training.
I believe in eating my own dogfood, and wouldn't subject my students to anything I wouldn't take on myself. As my own research and teaching moves more toward desktop fabrication, I've been reading a lot about materials science, structural engineering, machining, CNC and other subjects for which I have absolutely no preparation. It's pretty confusing, of course, but each day it all seems a little more clear. I've also been making a lot of mistakes as I try to make things. As humanists, I don't think we can do better than to follow Terence's adage that nothing human should be alien to us. It is possible to learn anything, if you're willing to begin in the middle.
Tags: digital history | historiography | interdisciplinarity | learning
Sunday, May 04, 2008
The Programming Historian is Now Available
The Programming Historian is now available on the NiCHE: Network in Canadian History & Environment website. This work is an open-access introduction to programming in Python, aimed at working historians (and other humanists) with little previous experience. Introductory lessons teach you how to
Tags: browser | digital history | open access | open source | programming | Python | Zotero
- install Zotero, the Python programming language and other useful tools
- read and write data files
- save web pages and automatically extract information from them
- count word frequencies
- remove stop words
- automatically refine searches
- make n-gram dictionaries
- create keyword-in-context (KWIC) displays
- make tag clouds, and
- harvest sets of hyperlinks
Tags: browser | digital history | open access | open source | programming | Python | Zotero
Thursday, April 10, 2008
Fitness Functions
[Cross-posted to Cliopatria & Digital History Hacks]
One of the distinctions that applied mathematicians make is between linear and nonlinear problems. In a linear problem, you have a set of variables that you can tweak, and as you adjust each variable you can get ever closer to an optimal configuration. Using techniques such as linear programming, it is straightforward to determine precisely how many scoops of raisins to put in your box of bran, or how many Cherries will make a Garcia. Many problems, alas, don't admit of this kind of solution. In the days before digital everything, it was all too common to futz around with the brightness knob, color balance, rabbit ears, and position of pets and small children to try and get a TV signal that didn't look like it was being relayed from the dark side of the moon. The slightest change could make things drastically better or worse, with no apparent logic.
The problem with nonlinear problems is that you pretty much have to get every variable right at the same time. Think of the space of all possible states of your problem as a kind of dark landscape, and the optimal solution as the highest point in that space. Linear problems have smooth landscapes. If you start groping your way up a hill, you end up at the top and that's the best you can do overall. Nonlinear problems have jagged landscapes. It is easy to feel your way up a low peak and get stuck there, unaware of higher peaks elsewhere.
There are different methods for solving nonlinear optimization problems; one of the more popular makes use of genetic algorithms. First you find a way of representing all of the possible solutions to your problem. In the TV example, you might want to represent the angle of each of the two antennas, the xy coordinates of the napping cat, the rotational angle of the brightness knob, and so on. A list of each of these variables is known as a genome, and a list of particular values as a genotype. Generate a small random population of genotypes, and test each one to see how good it is. This test is called the fitness function. In our example, it is the person sitting on the couch shouting "not bad," "pretty good" or "awful" each time an adjustment is made. Once you know how well each of your solutions performed, you make a new generation of solutions by mutating and recombining the genomes of your old ones. Over time, the fitness of the population increases, and the artificial selection mechanism eventually finds solutions that are near optimal. (If you want to start programming your own GAs, I recommend Mitchell's Introduction and Goldberg's Genetic Algorithms as good places to start).
One of the perennial tragedies of academia is that we constantly pretend that our careers or those of our students are linear optimization problems. Grades are the most obvious way that we do this. Students learn that their mark on one test is independent of their mark on another, that it is better to have a high GPA than to risk taking hard courses that interest them, that exploration and failure will usually be punished. Teachers justify marks by appealing to rubrics, bemoaning grade inflation and students "who look good on paper." Too many of us think of a good career in terms of lines on a CV, a list of so many independent accomplishments, each of which can be attained and then forgotten.
On a rainy day in 1992, I wandered into a Vancouver technical bookstore on my way home from school. I think I was probably avoiding a problem set or some other homework, as I've never been very good at doing what I should be doing rather than what I want to be doing. Anyway, I remember finding a copy of John Holland's Adaptation in Natural and Artificial Systems on the shelf of new releases and really wanting to buy it. I stood in the store holding the book for the longest time. It was more than I could afford, it was a distraction from my school work, I had a bad habit of buying books and losing interest in them. I had been doing a lot of exploring and a fair bit of failing. I finally made the decision that was, in context at least, sub-optimal. I bought the book and went home to read it rather than doing my schoolwork.
I often tell my students that they should follow their curiosity, take chances and not be afraid to fail. You never really know what whim, what chance encounter or distraction is going to change your life. In my case, I read a lot of science fiction and graphic novels and ate a lot of guacamole. I played role playing games and got married early and happily. I watched TV. I got bad grades in linear algebra and analysis, but I liked math enough to keep trying until I got better at it. And my first published work was on a subject that was novel and trendy enough that my reputation as an up-and-coming researcher outweighed my uneven transcript: genetic algorithms. It's tempting to look back at that moment in the bookstore as a crucial inflection point in my life, but that would be too linear. The choices that we make affect our fitness, but never in a way that makes it easy to assign credit or blame.
Tags: feedback | genetic algorithms | nonlinear optimization | pedagogy
One of the distinctions that applied mathematicians make is between linear and nonlinear problems. In a linear problem, you have a set of variables that you can tweak, and as you adjust each variable you can get ever closer to an optimal configuration. Using techniques such as linear programming, it is straightforward to determine precisely how many scoops of raisins to put in your box of bran, or how many Cherries will make a Garcia. Many problems, alas, don't admit of this kind of solution. In the days before digital everything, it was all too common to futz around with the brightness knob, color balance, rabbit ears, and position of pets and small children to try and get a TV signal that didn't look like it was being relayed from the dark side of the moon. The slightest change could make things drastically better or worse, with no apparent logic.
The problem with nonlinear problems is that you pretty much have to get every variable right at the same time. Think of the space of all possible states of your problem as a kind of dark landscape, and the optimal solution as the highest point in that space. Linear problems have smooth landscapes. If you start groping your way up a hill, you end up at the top and that's the best you can do overall. Nonlinear problems have jagged landscapes. It is easy to feel your way up a low peak and get stuck there, unaware of higher peaks elsewhere.
There are different methods for solving nonlinear optimization problems; one of the more popular makes use of genetic algorithms. First you find a way of representing all of the possible solutions to your problem. In the TV example, you might want to represent the angle of each of the two antennas, the xy coordinates of the napping cat, the rotational angle of the brightness knob, and so on. A list of each of these variables is known as a genome, and a list of particular values as a genotype. Generate a small random population of genotypes, and test each one to see how good it is. This test is called the fitness function. In our example, it is the person sitting on the couch shouting "not bad," "pretty good" or "awful" each time an adjustment is made. Once you know how well each of your solutions performed, you make a new generation of solutions by mutating and recombining the genomes of your old ones. Over time, the fitness of the population increases, and the artificial selection mechanism eventually finds solutions that are near optimal. (If you want to start programming your own GAs, I recommend Mitchell's Introduction and Goldberg's Genetic Algorithms as good places to start).
One of the perennial tragedies of academia is that we constantly pretend that our careers or those of our students are linear optimization problems. Grades are the most obvious way that we do this. Students learn that their mark on one test is independent of their mark on another, that it is better to have a high GPA than to risk taking hard courses that interest them, that exploration and failure will usually be punished. Teachers justify marks by appealing to rubrics, bemoaning grade inflation and students "who look good on paper." Too many of us think of a good career in terms of lines on a CV, a list of so many independent accomplishments, each of which can be attained and then forgotten.
On a rainy day in 1992, I wandered into a Vancouver technical bookstore on my way home from school. I think I was probably avoiding a problem set or some other homework, as I've never been very good at doing what I should be doing rather than what I want to be doing. Anyway, I remember finding a copy of John Holland's Adaptation in Natural and Artificial Systems on the shelf of new releases and really wanting to buy it. I stood in the store holding the book for the longest time. It was more than I could afford, it was a distraction from my school work, I had a bad habit of buying books and losing interest in them. I had been doing a lot of exploring and a fair bit of failing. I finally made the decision that was, in context at least, sub-optimal. I bought the book and went home to read it rather than doing my schoolwork.
I often tell my students that they should follow their curiosity, take chances and not be afraid to fail. You never really know what whim, what chance encounter or distraction is going to change your life. In my case, I read a lot of science fiction and graphic novels and ate a lot of guacamole. I played role playing games and got married early and happily. I watched TV. I got bad grades in linear algebra and analysis, but I liked math enough to keep trying until I got better at it. And my first published work was on a subject that was novel and trendy enough that my reputation as an up-and-coming researcher outweighed my uneven transcript: genetic algorithms. It's tempting to look back at that moment in the bookstore as a crucial inflection point in my life, but that would be too linear. The choices that we make affect our fitness, but never in a way that makes it easy to assign credit or blame.
Tags: feedback | genetic algorithms | nonlinear optimization | pedagogy
Saturday, April 05, 2008
Visualizing the Emergence of a Strategic Knowledge Cluster
In the summer of 2004, when I had just arrived at the University of Western Ontario, my new colleague Alan MacEachern invited me to join a small group that was putting together a grant application. The federal agency SSHRC had just announced funding for the design of something called 'research clusters'. At the time none of us was particularly clear what these clusters were supposed to be, and like many of the best kinds of opportunity, I don't think that SSHRC was really clear either. We eventually settled on the idea that the main task of clusters was 'knowledge mobilization', which left the matter nicely open.
Our initial grant application was successful, and five of us set to work to develop NiCHE, the Network in Canadian History & Environment / Nouvelle initiative canadienne en histoire de l'environnement. As we tried various things we kept track of activities and participants, allowing us to visualize the emergence of our research network. I should say up front that NiCHE doesn't cause research and is prohibited from directly funding research per se. Instead we find ways to facilitate research and training in environmental history broadly construed, and to mobilize the knowledge that researchers create.
One of the tools that we use for visualization is an open source package called Graphviz. We create a file that specifies entities (people, publications, field trips, etc.) and the relationships between them, then we hand off that file to Graphviz, which uses sophisticated algorithms to figure out a neat way to plot the network. We've found such visualization to be very useful, even though it can only ever show the tip of a much larger social iceberg. In our graphs, two people may be linked because they attended the same meeting or each published a chapter in a book. Our data doesn't show whether they knew each other in grad school, have a longstanding rivalry, or both secretly like Buffy the Vampire Slayer.
The original NiCHE executive group worked quite closely together. One of the interesting facts about networks is that the number of possible pairwise relations between entities grows much faster than the number of entities as the network gets larger. Two people have at most one relationship, three people can have three (AB, BC, AC), four people can have six (AB, AC, AD, BC, BD, CD). The ten possible pairwise relationships between the five of us looked like this:
One of the first things that we tried to do was provide licenses for Groove collaborative software to all of the people who were interested in joining NiCHE. For people with Windows machines the software worked very well. Unfortunately, it never really worked for people with Macs. We had to supplement Groove with other software, find suboptimal workarounds, and eventually abandon it. For a while, however, it gave us a way to interact relatively closely with NiCHE members who also happened to be tech-savvy Windows users. Our network took on a hub-and-spoke form.
To reach out to more potential participants, we formed an advisory group and held a meeting in Toronto. Instead of one hub, we now had two, with some bridging members who participated in both online and face-to-face activities.
The executive group split up to host regional meetings in other cities across Canada.
We put together an online directory so members could add information about themselves. The directory allowed us to contact people and tell them about upcoming activities. Since it was publicly accessible, the directory also allowed NiCHE members to learn more about one another.
Although adding one's name to a directory is a relatively weak form of participation, we found that many people became more active in NiCHE over time. The network seemed to extend to new participants, many of whom would then get involved in a number of subsequent projects. There is a saying in free / open source software, "contribute nothing, expect nothing." Conversely we could say that the people who contributed something to NiCHE could expect something from us. Some of them contributed articles to a special issue of the journal Environmental History. Some contributed chapters to a new textbook, Method and Meaning in Canadian Environmental History.
Subsequent activities like a summer school and a graduate student workshop brought in some new participants, and brought back many more:
When SSHRC announced a much larger grant for strategic knowledge clusters, we were able to include a version of the last figure as part of our application. (The Graphviz script that generated it is here.)
A year and half later, we're in the process of scaling up NiCHE activities by a couple of orders of magnitude. Network visualization gives us some insight into the work of a few hundred people who are loosely affiliated with NiCHE and collaborating in many different ways. We can identify people who have energy and initiative to share, and try to help them. Some provide 'bonding capital', tying tightly-linked groups closer together. Some provide 'bridging capital', mobilizing knowledge from one region or disciplinary specialization to another. We can also be more strategic about developing the connections that still need to be made, to make our network stronger and more effective. (For more about social networks, see Clay Shirky's new Here Comes Everybody.)
What is more exciting is that we are getting closer to the point where we can make these kind of tools available to everyone in NiCHE. People will be able to enter their own information about research collaborations and interests, and explore social connections within the network. It will become much easier to find joint acquaintances to make introductions or to find people with particular skills or expertise.
Tags: Graphviz | social network analysis | SSHRC (Social Sciences and Humanities Research Council of Canada) | visualization
Our initial grant application was successful, and five of us set to work to develop NiCHE, the Network in Canadian History & Environment / Nouvelle initiative canadienne en histoire de l'environnement. As we tried various things we kept track of activities and participants, allowing us to visualize the emergence of our research network. I should say up front that NiCHE doesn't cause research and is prohibited from directly funding research per se. Instead we find ways to facilitate research and training in environmental history broadly construed, and to mobilize the knowledge that researchers create.
One of the tools that we use for visualization is an open source package called Graphviz. We create a file that specifies entities (people, publications, field trips, etc.) and the relationships between them, then we hand off that file to Graphviz, which uses sophisticated algorithms to figure out a neat way to plot the network. We've found such visualization to be very useful, even though it can only ever show the tip of a much larger social iceberg. In our graphs, two people may be linked because they attended the same meeting or each published a chapter in a book. Our data doesn't show whether they knew each other in grad school, have a longstanding rivalry, or both secretly like Buffy the Vampire Slayer.
The original NiCHE executive group worked quite closely together. One of the interesting facts about networks is that the number of possible pairwise relations between entities grows much faster than the number of entities as the network gets larger. Two people have at most one relationship, three people can have three (AB, BC, AC), four people can have six (AB, AC, AD, BC, BD, CD). The ten possible pairwise relationships between the five of us looked like this:
One of the first things that we tried to do was provide licenses for Groove collaborative software to all of the people who were interested in joining NiCHE. For people with Windows machines the software worked very well. Unfortunately, it never really worked for people with Macs. We had to supplement Groove with other software, find suboptimal workarounds, and eventually abandon it. For a while, however, it gave us a way to interact relatively closely with NiCHE members who also happened to be tech-savvy Windows users. Our network took on a hub-and-spoke form.
To reach out to more potential participants, we formed an advisory group and held a meeting in Toronto. Instead of one hub, we now had two, with some bridging members who participated in both online and face-to-face activities.
The executive group split up to host regional meetings in other cities across Canada.
We put together an online directory so members could add information about themselves. The directory allowed us to contact people and tell them about upcoming activities. Since it was publicly accessible, the directory also allowed NiCHE members to learn more about one another.
Although adding one's name to a directory is a relatively weak form of participation, we found that many people became more active in NiCHE over time. The network seemed to extend to new participants, many of whom would then get involved in a number of subsequent projects. There is a saying in free / open source software, "contribute nothing, expect nothing." Conversely we could say that the people who contributed something to NiCHE could expect something from us. Some of them contributed articles to a special issue of the journal Environmental History. Some contributed chapters to a new textbook, Method and Meaning in Canadian Environmental History.
Subsequent activities like a summer school and a graduate student workshop brought in some new participants, and brought back many more:
When SSHRC announced a much larger grant for strategic knowledge clusters, we were able to include a version of the last figure as part of our application. (The Graphviz script that generated it is here.)
A year and half later, we're in the process of scaling up NiCHE activities by a couple of orders of magnitude. Network visualization gives us some insight into the work of a few hundred people who are loosely affiliated with NiCHE and collaborating in many different ways. We can identify people who have energy and initiative to share, and try to help them. Some provide 'bonding capital', tying tightly-linked groups closer together. Some provide 'bridging capital', mobilizing knowledge from one region or disciplinary specialization to another. We can also be more strategic about developing the connections that still need to be made, to make our network stronger and more effective. (For more about social networks, see Clay Shirky's new Here Comes Everybody.)
What is more exciting is that we are getting closer to the point where we can make these kind of tools available to everyone in NiCHE. People will be able to enter their own information about research collaborations and interests, and explore social connections within the network. It will become much easier to find joint acquaintances to make introductions or to find people with particular skills or expertise.
Tags: Graphviz | social network analysis | SSHRC (Social Sciences and Humanities Research Council of Canada) | visualization
Subscribe to:
Posts (Atom)
