Cloud Street

Strange clothes of sand

2007-03-28T23:31:00.000+01:00

There hasn't been a lot here lately; there hasn't been a huge amount on my home weblog Actually Existing either, and quite a lot of what I have posted there has been tagged as work-related. So this will be the last post here; I'm merging my weblogs and taking the opportunity to leave Google and go to Wordpress. I'll see you at The Gaping Silence. (True to the name, there's nothing much there now, but I'll put some new stuff up one of these days.)

Great big bodies

2007-02-07T13:20:00.000Z

I think the thing that really irritates me about the Long Tail is just how basic the statistical techniques underlying it are. If you've got all that data, why on earth wouldn't you do something more interesting and more informative with it? It's really not hard. (In fact it's so easy that I can't help feeling the Long Tail image must have some other appeal - but more on that later.)

As you may have noticed, this weblog hasn't been updated for a while. In fact, when I compared it with the rest of my RSS feed I found it was a bit of an outlier:

The Y axis is 'number of blogs': two updated today (zero days ago), 11 in the previous 10 days, 1 in the 10-day period before that, and so on until you get to the 71-80 column. Note that each column is a range of values, and that the columns are touching; technically this is a histogram rather than a bar chart.

You can do something similar with 'posts in last 100 days':

This shows that the really heavy posters are in the minority in this sample; twelve out of the eighteen have 30 or fewer posts in the last 100 days.

So it looks as if I'm reading a lot of reasonably regular but fairly light bloggers, and a few frequent fliers. If you put the two series together you can see the two groups reflected in the way the sample smears out along the X and Y axes without much in the middle:

My question is this. If you can produce readable and informative charts like this quickly and easily (and I assure you that you can - we're talking an hour from start to finish, and most of that went on counting the posts), what on earth would make you prefer this:

or this:

I can only think of two reasons. One is that it looks kind of like a power law distribution, and that's a cool idea. Except that it isn't a power law distribution, or any kind of distribution - it's a list ranked in descending order, and, er, that's it. The same criticism applies, obviously, to the classic 'power law' graphic ranking weblogs in descending order of inbound links.

DIGRESSION
You can compute a distribution of inbound links across weblogs using very much the techniques I've used here - so many weblogs with one link, so many with two and so forth. Oddly enough, what you end up with then is a curve which falls sharply then tapers off - there are far fewer weblogs with two links than with only one, but not so much of a difference between the '20 links' and '21 links' categories. However, even that isn't a power law distribution, for reasons explained here and here (reasons which, for the non-mathematician, can be summed up as 'a power law distribution means something specific, and this isn't it').
END DIGRESSION

The other reason - and, I suspect, the main reason - is that the Long Tail privileges ranking: the question it suggests isn't how many of which are doing what? but who's first?. A histogram might give more information, but it wouldn't tell me who's up there in the big head, or how far down the tail I am.

People want to be on top; failing that, they want to fantasise about being on top and identify with whoever's up there now. Not everyone, but a lot of people. The popularity of the Long Tail image has a lot in common with the popularity of celebrity gossip magazines.

They don't know about us

2006-11-22T10:30:00.000Z

Some dystopian thoughts on data harvesting, usage tracking, recommendation engines and consumer self-expression. First, here's Tom, then me:

"This is going to be one of the great benefits of ambient/pervasive computing or everyware - not the tracking of objects but the tracking and collating of you yourself through objects."

This sentence works just as well with the word 'benefits' replaced by 'threats'. It all depends who gets to do the tracking and collating, I suppose.

Now here's Max Levchin, formerly of Paypal, and his new toy Slide (via Thomas):

If Slide is at all familiar, it's as a knockoff of Flickr, the photo-sharing site. Users upload photos, which are displayed on a running ticker or Slide Show, and subscribe to one another's feeds. But photos are just a way to get Slide users communicating, establishing relationships, Levchin explains.

The site is beginning to introduce new content into Slide Shows. It culls news feeds from around the Web and gathers real-time information from, say, eBay auctions or Match.com profiles. It drops all of this information onto user desktops and then watches to see how they react.

Suppose, for example, there's a user named YankeeDave who sees a Treo 750 scroll by in his Slide Show. He gives it a thumbs-up and forwards it to his buddy" we'll call him Smooth-P. Slide learns from this that both YankeeDave and Smooth-P have an interest in a smartphone and begins delivering competing prices. If YankeeDave buys the item, Slide displays headlines on Treo tips or photos of a leather case. If Smooth-P gives a thumbs-down, Slide gains another valuable piece of data. (Maybe Smooth-P is a BlackBerry guy.) Slide has also established a relationship between YankeeDave and Smooth-P and can begin comparing their ratings, traffic patterns, clicks and networks.

Based on all that information, Slide gains an understanding of people who share a taste for Treos, TAG Heuer watches and BMWs. Next, those users might see a Dyson vacuum, a pair of Forzieri wingtips or a single woman with a six-figure income living within a ten-mile radius. In fact, that's where Levchin thinks the first real opportunity lies - hooking up users with like-minded people. "I started out with this idea of finding shoes for my girlfriend and hotties on HotOrNot for me," Levchin says with a wry smile. "It's easy to shift from recommending shoes to humans."

If this all sounds vaguely creepy, Levchin is careful to say he's rolling out features slowly and will only go as far as his users will allow. But he sees what many others claim to see: Most consumers seem perfectly willing to trade preference data for insight. "What's fueling this is the desire for self-expression," he says.

Nick:

I'm not sure that I see, in today's self-portraits on MySpace or YouTube or Flickr, or in the fetishistic collecting of virtual tokens of attention, the desire to mark one's place in a professional or social stratum. What they seem to express, more than anything, is a desire to turn oneself into a product, a commodity to be consumed. And since, as I wrote earlier, "self-commoditization is in the end indistinguishable from self-consumption," the new portraiture seems at its core narcissistic. The portraits are advertisements for a commoditized self

Granny Weatherwax:

"And sin, young man, is when you treat people as things. Including yourself. That's what sin is. ... People as things, that's where it starts."

More precisely, that's where some extraordinarily unequal and dishonest social relationships can start.

Got a web between his toes

2006-11-13T14:34:00.000Z

Now that Nick has read the last rites for Web 2.0, perhaps it's safe to return to a question that's never quite been resolved.

To wit: what is Web 2.0? (We've established that it's not a snail.) Over at What I wrote, I've just put up a March 2003 article called "In Godzilla's footprint". In it, I asked similar questions about e-business, taking issue with the standard rhetoric of 'efficiency' and 'empowerment'. I suggested that e-business wasn't - or rather isn't - a phenomenon in its own right, but the product of three much larger trends: standardisation, automation and externalisation of costs. (Read the whole thing.)

Assuming for the moment that I called this one correctly - and I find my arguments pretty persuasive - what of Web 2.0? More of the same, only featuring the automation of income generation (AdSense) and the externalisation of payroll costs ('citizen journalism')? Or is there more going on - and if so, what?

Update 16/11

It would be remiss of me not to give any pointers to my own thinking on Web 2.0. So I'm republishing another column at What I wrote, this time from February of this year. Most of you will probably have seen it the first time round, when it appeared in iSeries NEWS UK, but I think it's worth giving it another airing. Have a gander.

Simplify, reduce, oversimplify

2006-11-06T11:24:00.000Z

An interesting post on 'folksonomies' at Collin Brooke's blog prompted this comment, which I thought deserved a post of its own.

I think Peter Merholz's coinage 'ethnoclassification' could be useful here. As I've argued elsewhere, I think we can see all taxonomies (and ultimately all knowledge) as the product of an extended conversation within a given community: in this respect a taxonomy is simply an accredited 'folksonomy'.

However, I think there's a dangerous (but interesting) slippage here between what folksonomies could be and what folksonomies are: between the promise of the project of 'folksonomy' (F1) and what's delivered by any identifiable folksonomy (F2). (You can get into very similar arguments about Wikipedia 1 and Wikipedia 2 - sometimes with the same people.) Compared to the complexity and exhaustiveness of any functioning taxonomic scheme, I don't believe that any actually-existing 'folksonomy' is any more than an extremely sketchy work in progress.

For this reason (among others), I believe we need different words for the activity and the endpoint. So we could contrast classification with Peterme's 'ethnoclassification', on one hand, and note that the only real difference between the two is that the former takes place within structured and credentialled communities. On the other hand, we could contrast actual taxonomies with 'folksonomies'. The latter could have very much the same relationship with officially-credentialled taxonomies as classification does with ethnoclassification - but they aren't there yet.

The shift from 'folksonomy' to 'ethnoclassification' has two interesting side-effects, which I suspect are both fairly unwelcome to folksonomy boosters (a group in which I don't include Thomas Vander Wal, ironically enough). On one hand, divorcing process and product reminds us that improvements to one don't necessarily translate as improvements in the other. The activity that goes into producing a 'folksonomy', as distinct from a taxonomy, may give more participants a better experience (more egalitarian, more widely distributed, more chatty, more fun) but you wouldn't necessarily expect the end product to show improvements as a result. (You'd expect it to be a bit scrappy, by and large.) On the other hand, divorcing process from technology reminds us that ethnoclassification didn't start with del.icio.us; the aggregation of informal knowledge clouds is something we've been doing for a long time, perhaps as long as we've been human.

A taxonomy of terror

2006-10-31T10:42:00.000Z

I attended part of a very interesting conference on terrorism last week. The organisers intend to launch a network and a journal devoted to 'critical terrorism studies', a project which I strongly support. As the previous blog entry suggests, I've studied a bit of terrorism in my time - and I'm very much in favour of people being encouraged to approach the phenomenon critically, which is to say without necessarily endorsing the definitions and interpretive frameworks offered by official sources.

However, it seems to me that the nature of the object of study still needs to be defined - and defined at once more precisely and more loosely. In other words, I don't believe there's much common ground between someone who thinks of terrorism in terms of gathering intelligence on the IRA, and someone who maintains that George W. Bush is a bigger terrorist than Osama bin Laden; I don't think it's particularly productive to try to find common ground between those two images of terrorism, or to simply allow them to coexist without defining the differences between them. On the other hand, I don't see much mileage in a 'purist' Terrorism Studies which would focus solely on groups akin to the IRA - or in an alternative purism which would concentrate on terror attacks by Western governments.

A third approach offers to resolve the gap between these two - although I should say straight away that I don't believe it does so. This approach is that of terrorism as an object of discourse: what is under analysis is not so much an identifiable set of actions, or types of action, as the texts and utterances which purport to analyse and describe terrorism. The effect is to turn the analytical gaze back on the governmental discourse of terrorism, which in turn makes it possible to contrast the official image of the terrorist threat with data from other sources; an interesting example of this approach in practice is Richard Jackson's paper Religion, Politics and Terrorism: A Critical Analysis of Narratives of “Islamic Terrorism” (DOC file available from here).

I think this is a powerful and constructive approach - my own thesis (as yet unpublished) includes some quite similar work on Italian left-wing armed groups of the 1970s, whose presentation in both the mainstream and the Communist press was heavily shaped by differing ideological assumptions. But I think it should be recognised that it's an approach of a different order from the other two. To combine them would be to mix ontological and epistemological arguments - to say, in other words, That's what is officially labelled terrorism, but this is real terrorism. (Or: That's what they call terrorism, but this is what we know to be the reality of terrorism.) The problem with this is that it implies a commitment to a particular idea of real terrorism, without actually suggesting a candidate. At best, this formulation frees the analyst to retain his or her prior commitments, bolstered with added ontological certitude. At worst, it suggests that real terrorism is the inverse of officially labelled terrorism - or at least that there is no possible overlap between officially labelled terrorism and real terrorism. This is surely inadequate: a critical approach should be able to do more with the official version than simply reverse it.

I believe that the study of terrorism must include all of these elements, and recognise that they may overlap but don't coincide. In other words, it must include the following:

Organised political violence by non-state actors: 'terrorism' as a political intervention (call it T1)
Indiscriminate large-scale attacks on civilians: terror as a tactic, in warfare or otherwise (T2)
The constructed antagonist of the War on Terror: 'Terrorism' as object of discourse (T3)

We can think of it as a three-circle Venn diagram, with areas of intersection between each pair of circles and a triple intersection in the middle.

What is immediately apparent about this list is how little of the field of terrorism falls into all three categories. The (white) triple intersect - mass killing of civilians by a non-state political actor, officially labelled (and denounced) as terrorism - is represented by a relatively small number of horrific events, chief among them September 11th. By contrast, much of what students of terrorism - myself included - would like to be able to look at under that name falls into only two categories, or even one. The (red) intersect of T1 and T3, most obviously, is represented by those acts by armed groups which are officially denounced but don't involve mass killing of civilians: the 'execution' of Aldo Moro and the IRA's Brighton bomb, for example. The use of terror tactics by non-governmental death squads, such as the Nicaraguan Contras and the Salvadorean ORDEN militia, falls into the blue intersect of T1 and T2. The use of state terror by official enemies and 'rogue states' - such as the Syrian Hama massacre or Saddam Hussein's gassing of the people of Halabja - falls into the green intersect of T2 and T3. And this is without considering all those activities which fall into only one category: T1 (magenta) alone, activities by armed groups which fall below the radar of the discourse of 'terrorism' (a large and interesting category); T2 (cyan) alone, terror tactics used by states and not denounced as terrorism; and T3 (yellow) alone, officially-denounced 'terrorism' which involves neither an organised armed group nor a mass attack on civilians.

I don't, myself, see any problem with studying all three of these categories - or rather, all seven. I hope the remit of the new Critical Terrorism Studies is broad enough to encompass all of these without imposing an artificial unity on them. Paramilitary fundraising in Northern Ireland cannot be studied in the same way as the attack on Fallujah or press reporting of the 'ricin plot'; each of these deserves to be studied, however, and the different approaches appropriate to studying them can only strengthen the field.

The people with the answers

2006-09-18T11:40:00.000+01:00

Nick:

Larry Sanger, the controversial online encyclopedia's cofounder and leading apostate, announced yesterday, at a conference in Berlin, that he is spearheading the launch of a competitor to Wikipedia called The Citizendium. Sanger describes it as "an experimental new wiki project that combines public participation with gentle expert guidance."

The Citizendium will begin as a "fork" of Wikipedia, taking all of Wikipedia's current articles and then editing them under a new model that differs substantially from the model used by what Sanger calls the "arguably dysfunctional" Wikipedia community. "First," says Sanger, in explaining the primary differences, "the project will invite experts to serve as editors, who will be able to make content decisions in their areas of specialization, but otherwise working shoulder-to-shoulder with ordinary authors. Second, the project will require that contributors be logged in under their own real names, and work according to a community charter. Third, the project will halt and actually reverse some of the 'feature creep' that has developed in Wikipedia."

I've been thinking about Wikipedia, and about what makes a bad Wikipedia article so bad, for some time - this March 2005 post took off from some earlier remarks by Larry Sanger. I'm not attempting to pass judgment on Wikipedia as a whole - there are plenty of good Wikipedia articles out there, and some of them are very good indeed. But some of them are bad. Picking on an old favourite of mine, here's the first paragraph of the Wikipedia article on the Red Brigades, with my comments.

The Red Brigades (Brigate Rosse in Italian, often abbreviated as BR) are

The word is 'were'. The BR dissolved in 1981; its last successor group gave up the ghost in 1988. There's a small and highly violent group out there somewhere which calls itself "Nuove Brigate Rosse" - the New Red Brigades - but its continuity with the original BR is zero. This is a significant disagreement, to put it mildly.

a militant leftist group located in Italy. Formed in 1970, the Marxist Red Brigades

'Marxist' is a bizarre choice of epithet. Most of the Italian radical left was Marxist, and almost all of it declined to follow the BR's lead. Come to that, the Italian Communist Party (one of the BR's staunchest enemies) was Marxist. Terry Eagleton's a Marxist; Jeremy Hardy's a Marxist; I'm a Marxist myself, pretty much. The BR had a highly unusual set of political beliefs, somewhere between Maoism, old-school Stalinism and pro-Tupamaro insurrectionism. 'Maoist' would do for a one-word summary. 'Marxist' is both over-broad and misleading.

sought to create a revolutionary state through armed struggle

Well, yes. And no. I mean, I don't think it's possible to make any sense of the BR without acknowledging that, while they did have a famous slogan about portare l'attacco al cuore dello stato ('attacking at the heart of the state'), their anti-state actions were only a fairly small element of what they did. To begin with they were a factory-based group, who took action against foremen and personnel managers; in their later years - which were also their peak years - the BR, like other armed groups, got drawn into what was effectively a vendetta with the police, prioritising revenge attacks over any kind of 'revolutionary' programme. You could say that the BR were a revolutionary organisation & consequently had a revolutionary programme throughout, even if their actions didn't always match it - but how useful would this be?

and to separate Italy from the Western Alliance

Whoa. I don't think the BR were particularly in favour of Italy's NATO membership, but the idea that this was one of their key goals is absurd. If the BR had been a catspaw for the KGB, intent on fomenting subversion so as to destabilise Italy, then this probably would have been high on their list. But they weren't, and it wasn't.

In 1978, they kidnapped and killed former Prime Minister Aldo Moro under obscure circumstances.

Remarkably well-documented circumstances, I'd have said.

After 1984's scission

This is just wrong - following growing and unresolvable factionalism, the BR formally dissolved in October 1981.

Red Brigades managed with difficulty to survive the official end of the Cold War in 1989

This is both confused and wrong. Given that there was a split, how would the BR have survived beyond 1981 (or 1984), let alone 1989? As for the BR's successor groups, the last one to pack it in was last heard from in 1988.

even though it is now a fragile group with no original members.

Or rather, even though the name is now used by a small group about which very little is know, but which is not believed to have any connection to the original group (whose members are after all knocking on a bit by now).

Throughout the 1970’s the Red Brigades were credited with 14,000 acts of violence.

Good grief. Credited by whom? According to the sources I've seen, between 1970 and 1981 Italian armed struggle groups were responsible for a total of 3,258 actions, including 110 killings; the BR's share of the total came to 472 actions, including 58 killings. (Most 'actions' consisted of criminal damage and did not involve personal violence.) I'd be the first to admit that the precision of these figures is almost certainly spurious, but even if we doubled that figure of 472 we'd be an awful long way short of 14,000.

I'm not even going to look at the body of the article.

I think there are two main problems here; the good news is that Larry's proposals for the neo-Wikipedia (Nupedia? maybe not) would address both of them.

Firstly, first mover advantage. The structure of Wikipedia creates an odd imbalance between writers and editors. Writing a new article is easy: the writer can use whatever framework he or she chooses, in terms both of categories used to structure the entry and of the overall argument of the piece. Making minor edits to an article is easy: mutter 1984? no way, it was 1981!, log on, a bit of typing and it's done. But making major edits is hard - you can see from the comments above just how much work would be needed to make that BR article acceptable, starting from what's there now. It would literally be easier to write a new article. What's more, making edits stick is hard; I deleted one particularly ignorant falsehood from the BR article myself a few months ago, only to find my edit reverted the next day. (Of course, I re-reverted it. So there!)

Larry's suggestion of getting experts on board is very much to the point here. Slap my face and call me a credentialled academic, but I don't believe that everyone is equally qualified to write an encyclopedia article about their favourite topic - and I do think it matters who gets the first go.

Secondly, gaming the system. Wikipedia is a community as well as an encyclopedia. I'll pass over Larry's suggestion that Wikipedia is dysfunctional as a community, but I do think it's arguable that some behaviours which work well for Wikipedia-the-community are dysfunctional for Wikipedia-the-resource. It's been suggested, for instance, that what really makes Wikipedia special is the 'history' pages, which take the lid off the debate behind the encyclopedia and let us see knowledge in the process of formation. It follows from this that to show the world a single, 'definitive' version of an article on a subject would actually be a step backwards: The discussion tab on Wikipedia is a great place to point to your favorite version ... Does the world need a Wikipedia for stick-in-the-muds? W. A. Gerrard objects:

Of what value is publicly documenting the change history of an encyclopedia entry? How can something that purports to be authoritative allow the creation of alternative versions which readers can adopt as favorites?

If an attempt to craft a wiki that strives for accuracy, even via a flawed model, is considered something for “stick-in-the-muds”, then it’s apparent that many of Wikipedia’s supporters value the dynamics of its community more than the credibility of the product they deliver.

I think this is exactly right: the history pages are worth much more to members of the Wikipedia community than to Wikipedia users. People like to form communities and communities like to chat - and edits and votes are the currency of Wikipedia chat. And gaming the system is fun (hence the word 'game'). Aaron Swartz quotes comments about Wikipedia regulars who delete your newly[-]create[d] article without hesitation, or revert your changes and accuse you of vandalis[m] without even checking the changes you made, or who "edited" thousands of articles ... [mostly] to remove material that they found unsuitable. This clearly suggest the emergence of behaviours which are driven more by social expectations than by a concern for Wikipedia. The second writer quoted above continues: Indeed, some of the people-history pages contained little "awards" that people gave each other -- for removing content from Wikipedia.

Now, all systems can be gamed, and all communities chat. The question is whether the chatting and the gaming can be harnessed for the good of the encyclopedia - or, failing that, minimised. I'm not optimistic about the first possibility, and I suspect Larry Sanger isn't either. Larry does, however, suggest a very simple hack which would help with the second: get everyone to use their real name. This would, among other things, make it obvious when a writer had authority in a given area. I don't entirely agree with Aaron's conclusion:

Larry Sanger famously suggested that Wikipedia must jettison its anti-elitism so that experts could feel more comfortable contributing. I think the real solution is the opposite: Wikipedians must jettison their elitism and welcome the newbie masses as genuine contributors to the project, as people to respect, not filter out.

This is half right: Wikipedia-the-community has produced an elite of 'regulars', whose influence over Wikipedia-the-resource derives from their standing in the community rather than from any kind of claim to expertise. I agree with Aaron that this is an unhealthy situation, but I think Larry was right as well. The artificial elitism of the Wikipedia community doesn't only marginalise the 'masses' who contribute most of the original content; it also sidelines the subject-area experts who, within certain limited domains, have a genuine claim to be regarded as an elite.

I don't know if the Citizendium is going to address these problems in practice; I don't know if the Citizendium is going anywhere full stop. But I think Larry Sanger is asking the right questions. It's increasingly clear that Wikipedia isn't just facing in two directions at once, it's actually two different things - and what's good for Wikipedia-the-community isn't necessarily good for Wikipedia-the-resource.

Back in the garage

2006-09-05T11:23:00.000+01:00

I have begun to see what I think is a promising trend in the publishing world that may just transform the industry for good.

Paul Hartzog's Many-to-Many post on publishing draws some interesting conclusions from the success of Charlie Stross's Accelerando (nice one, Charlie). but makes me a bit nervous, partly because of the liberal use of excitable bolding.

What I am suggesting is happening is the reversal of traditional publishing, i.e. the transformation of the system in which authors create and distribute their work. In the old system, it is assumed that the publishing process acts as a quality control filter ... but it ends up merely being a profit-capturing filter.
[...]
Conversely, in the new system, the works are made available, and it is up to the community-at-large to pass judgement on their quality. In the emerging system, authors create and distribute their work, and readers, individually and collectively, including fans as well as editors and peers, review, comment, rank, and tag, everything.

Setting aside the formatting - and the evangelistic tone, something which never fails to set my teeth on edge - this is all interesting stuff. My problem is that I'm not sure about the economics of it. It's not so much that writers won't write if they don't get paid - writers will write, full stop - as that writers won't eat if they don't get paid: some money has to change hands some time. If the kind of development Paul is talking about takes hold, I can imagine a range of more-or-less unintended consequences, all with different overtones but few of them, to this jaundiced eye, particularly desirable:

Mass amateurisation means that nobody pays for anything, which in turn means that nobody makes a living from writing; this is essentially the RIAA/BPI anti-filesharing nightmare scenario, transposed to literature
Mass amateurisation doesn't touch the Dan Brown/Katie Price market, but gains traction in specialist areas of literature to the point where nobody can make a living from writing unless they're writing for the mass market; this is Charlie Gillett's argument for keeping CDs expensive (and the line the BPI would use against filesharing if they had any sense)
Downloads like Accelerando function essentially as tasters and people end up buying just as many actual books, if not more; this scenario will also be familiar from filesharing arguments, as it's the line generally used to counter the previous two
Mass amateur production becomes a new sphere of economic activity, linked in with and subordinate to the major mainstream operators: this is the MySpace scenario (at least, the MySpace makes money for Murdoch scenario)
Mass amateur production becomes a new sphere of non-economic activity, with a few star authors subsidised by publishing companies for the sake of the cachet they bring: the open source scenario
Mass amateur production becomes a new sphere of economic activity, existing on the margins and in the shadows, out of the reach of the major mainstream operators: the punk scenario (or, for older readers, the hippie scenario)

We can dismiss the first, RIAA-nightmare scenario. The third ('tasters') would be bearable, although it wouldn't go halfway to justifying Paul's argument. Most of the rest look pretty ghastly to me. Perhaps Paul is thinking in terms of the last scenario or something like it - but in that case I'd have to say that his optimism is just as misplaced, for different but related reasons, as the pessimism of the first scenario (although a new wave of garage literature would be a fine thing to see).

The trouble with making your own history is that you don't do it in circumstances of your own choosing. The participatory buzz of Web 2.0 tends to eat away at the structural and procedural walls that stop people getting their hands on stuff - but that can just mean that only the strongest and highest walls are left standing. Besides, walls can be useful, particularly if you want to keep a roof over your head.

We're all together now, dancing in time

2006-08-31T09:49:00.000+01:00

Ryan Carson:

I’d love to add friends to my Flickr account, add my links to del.icio.us, browse digg for the latest big stories, customise the content of my Netvibes home page and build a MySpace page. But you know what? I don’t have time and you don’t either...

Read the whole thing. What's particularly interesting is a small straw poll at the end of the article, where Ryan asks people who actually work on this stuff what social software apps they use on a day-to-day basis. Six people made 30 nominations in all; Ryan had five of his own for a total of 35.

Here are the apps which got more than one vote:

Flickr (four votes)
Upcoming (two)
Wikipedia (two)

And, er, that's it.

Social software looks like very big news indeed from some perspectives, but when it's held to the standard of actually helping people get stuff done, it fades into insignificance. I think there are three reasons for this apparent contradiction. First, there's the crowd effect - and, since you need a certain number of users before network effects start taking off, any halfway-successful social software application has a crowd behind it. It can easily look as if everyone's doing it, even if the relevant definition of 'everyone' looks like a pretty small group to you and me.

Then there's the domain effect: tagging and user-rating are genuinely useful and constructive, in some not very surprising ways, within pre-defined domains. (Think of a corporate intranet app, where there is no need for anyone to specify that 'Dunstable' means one of the company's offices, 'Barrett' means the company's main competitor and 'Monkey' means the payroll system.) For anyone who is getting work done with tagging, in other words, tagging is going to look pretty good - and, thanks to the crowd effect, it's going to look like a good thing that everyone's using.

Thirdly, social software is new, different, interesting and fun, as something to play with. It's a natural for geeks with time to play with stuff and for commentators who like writing about new and interesting stuff - let alone geek commentators. The hype generates itself; it's the kind of development that's guaranteed to look bigger than it is.

Put it all together - and introduce feedback effects, as the community of geek commentators starts to find social software apps genuinely useful within its specialised domain - and social software begins to look like a Tardis in reverse: much, much bigger on the outside than it is on the inside.

That's not to say that social software isn't interesting, or that it isn't useful. But I think that in the longer term those two facets will move apart: useful and productive applications of tagging will be happening under the commentator radar, often behind organisational firewalls, while the stuff that's interesting and fun to play with will remain... interesting and fun to play with.

So much that hides

2006-08-03T12:45:00.000+01:00

Alex points to this piece by Rashmi Sinha on 'Findability with tags': the vexed question of using tags to find the material that you've tagged, rather than as an elaborate way of building a mind-map.

I should stress, parenthetically, that that last bit wasn't meant as a putdown - it actually describes my own use of Simpy. I regularly tag pages, but almost never use tags to actually retrieve them. Sometimes - quite rarely - I do pull up all the pages I've tagged with a generic "write something about this" tag. Apart from that, I only ever ask Simpy two questions: one is "what was that page I tagged the other day?" (for which, obviously, meaningful tags aren't required); the other is "what does my tag cloud look like?".

Now, you could say that the answer to the second question isn't strictly speaking information; it's certainly not information I use, unless you count the time I spend grooming the cloud by splitting, merging and deleting stray tags. I like tag clouds and don't agree with Jeffrey Zeldman's anathema, but I do agree with Alex that they're not the last word in retrieving information from tags. Which is where Rashmi's article comes in.

Rashmi identifies three ways of layering additional information on top of the basic item/tag pairing, all of which hinge on partitioning the tag universe in different ways. This is most obvious in the case of faceted tagging: here, the field of information is partitioned before any tags are applied. Rashmi cites the familiar example of wine, where a 'region' tag would carry a different kind of information from 'grape variety', 'price' or for that matter 'taste'. Similar distinctions can be made in other areas: a news story tagged 'New Labour', 'racism' and 'to blog about' is implicitly carrying information in the domains 'subject (political philosophy)', 'subject (social issue)' and 'action to take'.

There are two related problems here. A unique tag, in this model, can only exist within one dimension: if I want separate tags for New Labour (the people) and New Labour (the philosophy), I'll either have to make an artificial distinction between the two (New_Labour vs New_Labour_philosophy) or add a dimension layer to my tags (political_party.New_Labour vs political_philosophy.New_Labour). Both solutions are pretty horrible. More broadly, you can't invoke a taxonomist's standby like the wine example without setting folksonomic backs up, and with some reason: part of the appeal of tagging is precisely that you start with a blank sheet and let the domains of knowledge emerge as they may.

Clustered tagging (a new one on me) addresses both of these problems, as well as answering the much-evaded question of how those domains are supposed to emerge. A tag cluster - as seen on Flickr - consists of a group of tags which consistently appear together, suggesting an implicit 'domain'. Crucially, a single tag can occur in multiple clusters. The clusters for the Flickr 'election' tag, for example, are easy to interpret:

vote, politics, kerry, bush, voting, ballot, poster, cameraphone, democrat, president

wahl, germany, deutschland, berlin, cdu, spd, bundestagswahl

canada, ndp, liberal, toronto, jacklayton, federalelection

and, rather anticlimactically,

england, uk

Clustering, I'd argue, represents a pretty good stab at building emergent domains. The downside is that it only becomes possible when there are huge numbers of tagging operations.

The third enhancement to tagging Rashmi describes is the use of tags as pivots:

When everything (tag, username, number of people who have bookmarked an item) is a link, you can use any of those links to look around you. You can change direction at any moment.

Lurking behind this, I think, is Thomas's original tripartite definition of 'folksonomy':

the three needed data points in a folksonomy tool [are]: 1) the person tagging; 2) the object being tagged as its own entity; and 3) the tag being used on that object. Flattening the three layers in a tool in any way makes that tool far less valuable for finding information. But keeping the three data elements you can use two of the elements to find a third element, which has value. If you know the object (in del.icio.us it is the web page being tagged) and the tag you can find other individuals who use the same tag on that object, which may lead (if a little more investigation) to somebody who has the same interest and vocabulary as you do. That person can become a filter for items on which they use that tag.

This, I think, is pivoting in action: from the object and its tags, to the person tagging and the tags they use, to the person using particular tags and the objects they tag. (There's a more concrete description here.)

Alex suggests that using tags as pivots could also be considered a subset of faceted browsing. I'd go further, and suggest that facets, clusters and pivots are all subsets of a larger set of solutions, which we can call domain-based tagging. If you use facets, the domains are imposed: this approach is a good fit to relatively closed domains of knowledge and finite groups of taggers. If you've got an epistemological blank sheet and a limitless supply of taggers, you can allow the domains to emerge: this is where clusters come into their own. And if what you're primarily interested in is people - and, specifically, who's saying what about what - then you don't want multiple content-based domains but only the information which derives directly from human activity: the objects and their taggers. Or rather, you want the objects and the taggers, plus the ability to pivot into a kind of multi-dimensional space: instead of tags existing within domains, each tag is a domain in its own right, and what you can find within each tag-domain is the objects and their taggers.

What all of this suggests is that, unsurprisingly, there is no 'one size fits all' solution. I suggested some time ago that

If 'cloudiness' is a universal condition, del.icio.us and Flickr and tag clouds and so forth don't enable us to do anything new; what they are giving us is a live demonstration of how the social mind works.

All knowledge is cloudy; all knowledge is constructed through conversation; conversation is a way of dealing with cloudiness and building usable clouds; social software lets us see knowledge clouds form in real time. I think that's fine as far as it goes; what it doesn't say is that, as well as having conversations about different things, we're having different kinds of conversations and dealing with the cloud of knowing in different ways. Ontology is not, necessarily, overrated; neither is folksonomy.

The users geeks don't see

2006-07-05T11:04:00.000+01:00

Nick writes, provocatively as ever, about the recent 'community-oriented' redesign of the netscape.com portal:

A few days ago, Netscape turned its traditional portal home page into a knockoff of the popular geek news site Digg. Like Digg, Netscape is now a "news aggregator" that allows users to vote on which stories they think are interesting or important. The votes determine the stories' placement on the home page. Netscape's hope, it seems, is to bring Digg's hip Web 2.0 model of social media into the mainstream. There's just one problem. Normal people seem to think the entire concept is ludicrous.

Nick cites a post titled Netscape Community Backlash, from which this line leapt out at me:

while a lot of us geeks and 2.0 types are addicted to our own technology (and our own voices, to be honest), it's pretty darn obvious that A LOT of people want to stick with the status quo

This reminded me of a minor revelation I had the other day, when I was looking for the Java-based OWL reasoner 'pellet'. I googled for
pellet owl
- just like that, no quotes - expecting to find a 'pellet' link at the bottom of forty or fifty hits related to, well, owls and their pellets. In fact, the top hit was "Pellet OWL Reasoner". (To be fair, if you google
owl pellet
you do get the fifty pages of owl pellets first.)

I think it's fair to say that the pellet OWL reasoner isn't big news even in the Web-using software development community; I'd be surprised if everyone reading this post even knows what an OWL reasoner is (or has any reason to care). But there's enough activity on the Web around pellet to push it, in certain circumstances, to the top of the Google rankings (see for yourself).

Hence the revelation: it's still a geek Web. Or rather, there's still a geek Web, and it's still making a lot of the running. When I first started using the Internet, about ten years ago, there was a geek Web, a hobbyist Web, an academic Web (small), a corporate Web (very small) and a commercial Web (minute) - and the geek Web was by far the most active. Since then the first four sectors have grown incrementally, but the commercial Web has exploded, along with a new sixth sector - the Web-for-everyone of AOL and MSN and MySpace and LiveJournal (and blogs), whose users vastly outnumber those of the other five. But the geek Web is still where a lot of the new interesting stuff is being created, posted, discussed and judged to be interesting and new.

Add social software to the mix - starting, naturally, within the geek Web, as that's where it came from - and what do you get? You get a myth which diverges radically from the reality. The myth is that this is where the Web-for-everyone comes into its own, where millions of users of what was built as a broadcast Web with walled-garden interactive features start talking back to the broadcasters and breaking out of their walled gardens. The reality is that the voices of the geeks are heard even more loudly - and even more disproportionately - than before. Have a look at the 'popular' tags on del.icio.us: as I write, six of the top ten (including all of the top five) relate directly to programmers, and only to programmers. (Number eight reads: "LinuxBIOS - aims to replace the normal BIOS found on PCs, Alphas, and other machines with a Linux kernel". The unglossed reference to Alphas says it all.) Of the other four, one's a political video, two are photosets and one is a full-screen animation of a cartoon cat dancing, rendered entirely in ASCII art. (Make that seven of the top ten.)

I'm not a sceptic about social software: ranking, tagging, search-term-aggregation and the other tools of what I persist in calling ethnoclassification are both new and powerful. But they're most powerful within a delimited domain: a user coming to del.icio.us for the first time should be looking for the 'faceted search' option straight away ("OK, so that's the geek cloud, how do I get it to show me the cloud for European history/ceramics/Big Brother?") The fact that there is no 'faceted search' option is closely related, I'd argue, to the fact that there is no discernible tag cloud for European history or ceramics or Big Brother: we're all in the geek Web. (Even Nick Carr.) (Photography is an interesting exception - although even there the only tags popular enough to make the del.icio.us tag cloud are 'photography', 'photo' and 'photos'. There are 40 programming-related tags, from ajax to xml.)

Social software wasn't built for the users of the Web-for-everyone. Reaction to the Netscape redesign tells us (or reminds us) that there's no reason to assume they'll embrace it.

Update Have a look at Eszter Hargittai's survey of Web usage among 1,300 American college students, conducted in February and March 2006. MySpace is huge, and Facebook's even huger, but Web 2.0 as we know it? It's not there. 1.9% use Flickr; 1.6% use Digg; 0.7% use del.icio.us. Answering a slightly different question, 1.5% have ever visited Boingboing, and 1% Technorati. By contrast, 62% have visited CNN.com and 21% bbc.co.uk. It's still, very largely, a broadcast Web with walled-garden interactivity. Comparing results like these with the prophecies of tagging replacing hierarchy, Long Tail production and mashups all round, I feel like invoking the story of the blind men and the elephant - except that I'm not even sure we've all got the same elephant.

We hear the sound of machines

2006-06-12T11:46:00.000+01:00

Sooner or later, the Internet will need to be saved from Google. Because Google - which appears to be an integral part of the information-wants-to-be-free Net dream, the search engine which gives life to the hyperlinked digital nervous system of a kind of massively-distributed Xanadu project - is nothing of the sort. Google is a private company; Google's business isn't even search. Google's business is advertising - and, whatever we think about how well search goes together with tagging and folksonomic stumbling-upon, search absolutely doesn't go with advertising. (Update 15th June: this is a timely reminder that Google is a business, and its business is advertising. Mass personalisation, online communities, interactive rating and ranking, it's all there - and it's all about the advertising.)

I had thought that, in the context of plain vanilla Web search, Google actually had this cracked - that the prominence of 'sponsored links', displayed separately from search results, allowed them to deliver an unpolluted service and still make money. I hadn't reckoned with AdSense. AdSense doesn't in itself pollute Google's search results. What it does is far worse: it encourages other people to pollute the Net. Which will mean, ultimately, that Google will paint (or choke) itself into a corner - but that, if we're not careful, an awful lot of users will be stuck in that corner with them.

For a much fuller and more cogent version of this argument, read Seth Jayson (via Scott). One point in particular stood out: Google (Nasdaq: GOOG) insiders are continuing to drop shares on the public at a rate that boggles the mind. It's true. Over the last year, as far as published records show, Sun insiders have sold $50,000 worth of shares, net. In the same period, IBM insiders have sold $6,500,000; Microsoft insiders have sold $1,500,000,000; and Google insiders have sold $5,000,000,000. See for yourself. That's a lot of shares.

I couldn't make it any simpler

2006-06-05T11:28:00.000+01:00

I hate to say this - I've always loathed VR boosters and been highly sceptical about the people they boost - but Jaron Lanier's a bright bloke. His essay Digital Maoism doesn't quite live up to the title, but it's well worth reading (thanks, Thomas).

I don't think he quite gets to the heart of the current 'wisdom of the crowds' myth, though. It's not Maoism so much as Revivalism: there's a tight feedback loop between membership of the collective, collective activity and (crucially) celebration of the activity of the collective. Or: celebration of process rather than end-result - because the process incarnates the collective.

Put it this way. Say that (for example) the Wikipedia page on the Red Brigades is wildly wrong or wildly inadequate (which is just as bad); say that the tag cloud for an authoritative Red Brigades resource is dominated by misleading tags ('kgb', 'ussr', 'mitrokhin'...). Would a wikipedian or a 'folksonomy' advocate see this situation as a major problem? Not being either I can't give an authoritative answer, but I strongly suspect the answer would be No: it's all part of the process, it's all part of the collective self-expression of wikipedians and the growth of the folksonomy, and if the subject experts don't like it they should just get their feet wet and start tagging and editing themselves. And if, in practice, the experts don't join in - perhaps, in the case of Wikipedia, because they don't have the stomach for the kind of 'editing' process which saw Jaron Lanier's own corrections get reverted? Again, I don't know for sure, but I suspect the answer would be another shrug: the wiki's open to all - and tagspace couldn't be more open - so who's to blame, if you can't make your voice heard, but you? There's nothing inherently wrong with the process, except that you're not helping to improve it. There's nothing inherently wrong with the collective, except that you haven't joined it yet.

Two quotes to clarify (hopefully) the connection between collective and process. Michael Wexler:

our understanding of things changes and so do the terms we use to describe them. How do I solve that in this open system? Do I have to go back and change all my tags? What about other people’s tags? Do I have to keep in mind all the variations on tags that reflect people’s different understanding of the topics?

The social connected model implies that the connections are the important part, so that all you need is one tag, one key, to flow from place to place and discover all you need to know. But the only people who appear to have time to do that are folks like Clay Shirky. The rest of us need to have information sorted and organized since we actually have better things to do than re-digest it.
<...>
What tagging does is attempt to recreate the flow of discovery. That’s fine… but what taxonomy does is recreate the structure of knowledge that you’ve already discovered. Sometimes, I like flowing around and stumbling on things. And sometimes, that’s a real pita. More often than not, the tag approach involves lots of stumbling around and sidetracks.
<...>
It's like Family Feud [a.k.a. Family Fortunes - PJE]. You have to think not of what you might say to a question, you have to guess what the survey of US citizens might say in answer to a question. And that’s really a distraction if you are trying to just answer the damn question.

And our man Lanier:

there's a demonstrative ritual often presented to incoming students at business schools. In one version of the ritual, a large jar of jellybeans is placed in the front of a classroom. Each student guesses how many beans there are. While the guesses vary widely, the average is usually accurate to an uncanny degree.

This is an example of the special kind of intelligence offered by a collective. It is that peculiar trait that has been celebrated as the "Wisdom of Crowds,"
<...>
The phenomenon is real, and immensely useful. But it is not infinitely useful. The collective can be stupid, too. Witness tulip crazes and stock bubbles. Hysteria over fictitious satanic cult child abductions. Y2K mania. The reason the collective can be valuable is precisely that its peaks of intelligence and stupidity are not the same as the ones usually displayed by individuals. Both kinds of intelligence are essential.

What makes a market work, for instance, is the marriage of collective and individual intelligence. A marketplace can't exist only on the basis of having prices determined by competition. It also needs entrepreneurs to come up with the products that are competing in the first place. In other words, clever individuals, the heroes of the marketplace, ask the questions which are answered by collective behavior. They put the jellybeans in the jar.

To illustrate this, once more (just the once) with the Italian terrorists. There are tens of thousands of people, at a conservative estimate, who have read enough about the Red Brigades to write that Wikipedia entry: there are a lot of ill-informed or partially-informed or tendentious books about terrorism out there, and some of them sell by the bucketload. There are probably only a few hundred people who have read Gian Carlo Caselli and Donatella della Porta's long article "The History of the Red Brigades: Organizational structures and Strategies of Action (1970-82)" - and I doubt there are twenty who know the source materials as well as the authors do. (I'm one of the first group, obviously, but certainly not the second.) Once the work's been done anyone can discover it, but discovery isn't knowledge: the knowledge is in the words on the pages, and ultimately in the individuals who wrote them. They put the jellybeans in the jar.

This is why (an academic writes) the academy matters, and why academic elitism is - or at least can be - both valid and useful. Jaron:

The balancing of influence between people and collectives is the heart of the design of democracies, scientific communities, and many other long-standing projects. There's a lot of experience out there to work with. A few of these old ideas provide interesting new ways to approach the question of how to best use the hive mind.
<...>
Scientific communities ... achieve quality through a cooperative process that includes checks and balances, and ultimately rests on a foundation of goodwill and "blind" elitism — blind in the sense that ideally anyone can gain entry, but only on the basis of a meritocracy. The tenure system and many other aspects of the academy are designed to support the idea that individual scholars matter, not just the process or the collective.

I'd go further, if anything. Academic conversations may present the appearance of a collective, but it's a collective where individual contributions are preserved and celebrated ("Building on Smith's celebrated critique of Jones, I would suggest that Smith's own analysis is vulnerable to the criticisms advanced by Evans in another context..."). That is, academic discourse looks like a conversation - which wikis certainly can do, although Wikipedia emphatically doesn't.

The problem isn't the technology, in other words: both wikis and tagging could be ways of making conversation visible, which inevitably means visualising debate and disagreement. The problem is the drive to efface any possibility of conflict, effectively repressing the appearance of debate in the interest of presenting an evolving consensus. (Or, I could say, the problem is the tendency of people to bow and pray to the neon god they've made, but that would be a bit over the top - and besides, Simon and Garfunkel quotes are far too obvious.)

Update 13th June

I wrote (above): It's not Maoism so much as Revivalism: there's a tight feedback loop between membership of the collective, collective activity and (crucially) celebration of the activity of the collective. Or: celebration of process rather than end-result - because the process incarnates the collective.

Here's Cory Doctorow, responding to Lanier:

Wikipedia isn't great because it's like the Britannica. The Britannica is great at being authoritative, edited, expensive, and monolithic. Wikipedia is great at being free, brawling, universal, and instantaneous.
<...>
If you suffice yourself with the actual Wikipedia entries, they can be a little papery, sure. But that's like reading a mailing-list by examining nothing but the headers. Wikipedia entries are nothing but the emergent effect of all the angry thrashing going on below the surface. No, if you want to really navigate the truth via Wikipedia, you have to dig into those "history" and "discuss" pages hanging off of every entry. That's where the real action is, the tidily organized palimpsest of the flamewar that lurks beneath any definition of "truth." The Britannica tells you what dead white men agreed upon, Wikipedia tells you what live Internet users are fighting over.

The Britannica truth is an illusion, anyway. There's more than one approach to any issue, and being able to see multiple versions of them, organized with argument and counter-argument, will do a better job of equipping you to figure out which truth suits you best.

Quoting myself again, There's nothing inherently wrong with the process, except that you're not helping to improve it. There's nothing inherently wrong with the collective, except that you haven't joined it yet.

When there is no outside

2006-05-25T10:36:00.000+01:00

Nick Carr's hyperbolically-titled The Death of Wikipedia has received a couple of endorsements and some fairly vigorous disagreement, unsurprisingly. I think it's as much a question of tone as anything else. When Nick reads the line

certain pages with a history of vandalism and other problems may be semi-protected on a pre-emptive, continuous basis.

it clearly sets alarm bells ringing for him, as indeed it does for me ("Ideals always expire in clotted, bureaucratic prose", Nick comments). Several of his commenters, on the other hand, sincerely fail to see what the big deal might be: it's only a handful of pages, it's only semi-protection, it's not that onerous, it's part of the continuing development of Wikipedia editing policies, Wikipedia never claimed to be a totally open wiki, there's no such thing as a totally open wiki anyway...

I think the reactions are as instructive as the original post. No, what Nick's pointing to isn't really a qualitative change, let alone the death of anything. But yes, it's a genuine problem, and a genuine embarrassment to anyone who takes the Wikipedian rhetoric seriously. Wikipedia ("the free encyclopedia that anyone can edit") routinely gets hailed for its openness and its authority, only not both at the same time - indeed, maximising one can always be used to justify limits on the other. As here. But there's another level to this discussion, which is to do with Wikipedia's resolution of the openness/authority balancing-act. What happens in practice is that the contributions of active Wikipedians take precedence over both random vandals and passing experts. In effect, both openness and authority are vested in the group.

In some areas this works well enough, but in others it's a huge problem. I use Wikipedia myself, and occasionally drop in an edit if I see something that's crying out for correction. Sometimes, though, I see a Wikipedia article that's just wrong from top to bottom - or rather, an article where verifiable facts and sustainable assertions alternate with errors and misconceptions, or are set in an overall argument which is based on bad assumptions. In short, sometimes I see a Wikipedia article which doesn't need the odd correction, it needs to be pulled and rewritten. I'm not alone in having this experience: here's Tom Coates on 'penis envy' and Thomas Vander Wal (!) on 'folksonomy', as well as me on 'anomie'.

It's not just a problem with philosophical concepts, either - I had a similar reaction more recently to the Wikipedia page on the Red Brigades. On the basis of the reading I did for my doctorate, I could rewrite that page from start to finish, leaving in place only a few proper names and one or two of the dates. But writing this kind of thing is hard and time-consuming work - and I've got quite enough of that to do already. So it doesn't get done.

I don't think this is an insurmountable problem. A while ago I floated a cunning plan for fixing pages like this, using PledgeBank to mobilise external reserves of peer-pressure; it might work, and if only somebody else would actually get it rolling I might even sign up. But I do think it's a problem, and one that's inherent to the Wikipedia model.

To reiterate, both openness and authority are vested in the group. Openness: sure, Wikipedia is as open to me as any other registered editor d00d, but in practice the openness of Wikipedia is graduated according to the amount of time you can afford to spend on it. As for authority, I'm not one, but (like Debord) I have read several good books - better books, to be blunt, than those relied on by the author[s] of the current Red Brigades article. But what would that matter unless I was prepared to defend what I wrote against bulk edits by people who disagreed - such as, for example, the author[s] of the current article? On the other hand, if I was prepared to stick it out through the edit wars, what would it matter whether I knew my stuff or not? This isn't just random bleating. When I first saw that Red Brigades article I couldn't resist one edit, deleting the completely spurious assertion that the group Prima Linea was a Red Brigades offshoot. When I looked at the page again the next day, my edit had been reverted.

Ultimately Wikipedia isn't about either openness or authority: it's about the collective activity of editing Wikipedia and being a Wikipedian. From that, all else follows.

Update 2/6/06 (in response to David, in comments)

There are two obvious problems with the Wikipedia page on the Brigate Rosse, and one that's larger but more diffuse. The first problem is that it's written in the present tense; it's extremely dubious that there's any continuity between the historic Brigate Rosse and the gang who shot Biagi, let alone that they're simply, unproblematically the same group. This alone calls for a major rewrite. Secondly, the article is written very much from a police/security-service/conspiracist stance, with a focus on question like whether the BR was assisted by the Czech security services or penetrated by NATO. But this tends to reinforce an image of the BR as a weird alien force which popped up out of nowhere, rather than an extreme but consistent expression of broader social movements (all of which has been documented).

The broader problem - which relates to both of the specific points - goes back to a problem with the amateur-encyclopedia format itself: Wikipedia implicitly asks what a given topic is, which prompts contributors to think of their topic as having a core, essential meaning (I wrote about this last year). The same problem can arise in a 'proper' encyclopedia, but there it's generally mitigated by expertise: somebody who's spent several years studying the broad Italian armed struggle scene is going to be motivated to relate the BR back to that scene, rather than presenting it as an utterly separate thing. The motivation will be still greater if the expert on the BR has also been asked to contribute articles on Prima Linea, the NAP, etc. This, again, is something that happens (and works, for all concerned) in the kind of restricted conversations that characterise academia, but isn't incentivised by the Wikipedia conversation - because the Wikipedia conversation doesn't go anywhere else. Doing Wikipedia is all about doing Wikipedia.

Who's there?

2006-05-15T10:58:00.000+01:00

At Many-to-Many, Ross Mayfield reports that Clay Shirky and danah boyd have been thinking about "the lingering questions in our field", viz. the field of social software. I was a bit surprised to see that

How can communities support veterans going off topic together and newcomers seeking topical information and connections?

still qualifies as a 'lingering question'; I distinctly remember being involved in thrashing this one out, together with Clay, the best part of nine years ago. But this was the one that really caught my eye, if you'll pardon the expression:

What level of visual representation of the body is necessary to trigger mirror neurons?

Uh-oh. Sherry Turkle (subscription-only link):

a woman in a nursing home outside Boston is sad. Her son has broken off his relationship with her. Her nursing home is taking part in a study I am conducting on robotics for the elderly. I am recording the woman’s reactions as she sits with the robot Paro, a seal-like creature advertised as the first ‘therapeutic robot’ for its ostensibly positive effects on the ill, the elderly and the emotionally troubled. Paro is able to make eye contact by sensing the direction a human voice is coming from; it is sensitive to touch, and has ‘states of mind’ that are affected by how it is treated – for example, it can sense whether it is being stroked gently or more aggressively. In this session with Paro, the woman, depressed because of her son’s abandonment, comes to believe that the robot is depressed as well. She turns to Paro, strokes him and says: ‘Yes, you’re sad, aren’t you. It’s tough out there. Yes, it’s hard.’ And then she pets the robot once again, attempting to provide it with comfort. And in so doing, she tries to comfort herself.

What are we to make of this transaction? When I talk to others about it, their first associations are usually with their pets and the comfort they provide. I don’t know whether a pet could feel or smell or intuit some understanding of what it might mean to be with an old woman whose son has chosen not to see her anymore. But I do know that Paro understood nothing. The woman’s sense of being understood was based on the ability of computational objects like Paro – ‘relational artefacts’, I call them – to convince their users that they are in a relationship by pushing certain ‘Darwinian’ buttons (making eye contact, for example) that cause people to respond as though they were in relationship.

Further reading: see Kathy Sierra on mirror neurons and the contagion of negativity. See also Shelley's critique of Kathy's argument, and of attempts to enforce 'positive' feelings by manipulating mood. And see the sidebar at Many-to-Many, which currently reads as follows:

Recent Comments

viagra on Sanger on Seigenthaler’s criticism of Wikipedia

hydrocodone cheap on Sanger on Seigenthaler’s criticism of Wikipedia

viagra on Sanger on Seigenthaler’s criticism of Wikipedia

alprazolam online on Sanger on Seigenthaler’s criticism of Wikipedia

Timur on Sanger on Seigenthaler’s criticism of Wikipedia

Timur on Sanger on Seigenthaler’s criticism of Wikipedia

Recent Trackbacks

roulette: roulette

jouer casino: jouer casino

casinos on line: casinos on line

roulette en ligne: roulette en ligne

jeux casino: jeux casino

casinos on line: casinos on line

Some day this will all be yours

2006-05-09T10:46:00.000+01:00

Scott Karp:

What if dollars have no place in the new economics of content?
...
In media 1.0, brands paid for the attention that media companies gathered by offering people news and entertainment (e.g. TV) in exchange for their attention. In media 2.0, people are more likely to give their attention in exchange for OTHER PEOPLE’S ATTENTION. This is why MySpace can’t effectively monetize its 70 million users through advertising — people use MySpace not to GIVE their attention to something that is entertaining or informative (which could thus be sold to advertisers) but rather to GET attention from other users.
...
MySpace can’t sell attention to advertisers because the site itself HAS NONE. Nobody pays attention to MySpace — users pay attention to each other, and compete for each other’s attention — it’s as if the site itself doesn’t exist.

You see the same phenomenon in blogging — blogging is not a business in the traditional sense because most people do it for the attention, not because they believe there’s any financial reward. What if the economics of media in the 21st century begin to look like the economics of poetry in the 20th century? — Lots of people do it for their own personal gratification, but nobody makes any money from it.

Pedantry first: it's inconceivable that we'll reach a point where nobody makes any money from the media, at least this side of the classless society. Even the hard case of blogging doesn't really stand up - I could name half a dozen bloggers who have made money or are making money from their blogs, without pausing to think.

It's a small point, but it's symptomatic of the enthusiastic looseness of Karp's argument. So I welcomed Nicholas Carr's counterblast, which puts Karp together with some recent comments by Esther Dyson:

"Most users are not trying to turn attention into anything else. They are seeking it for itself. For sure, the attention economy will not replace the financial economy. But it is more than just a subset of the financial economy we know and love."

Here's Carr:

I fear that to view the attention economy as "more than just a subset of the financial economy" is to misread it, to project on it a yearning for an escape (if only a temporary one) from the consumer culture. There's no such escape online. When we communicate to promote ourselves, to gain attention, all we are doing is turning ourselves into goods and our communications into advertising. We become salesmen of ourselves, hucksters of the "I." In peddling our interests, moreover, we also peddle the commodities that give those interests form: songs, videos, and other saleable products. And in tying our interests to our identities, we give marketers the information they need to control those interests and, in the end, those identities. Karp's wrong to say that MySpace is resistant to advertising. MySpace is nothing but advertising.

Now, this is good, bracing stuff, but I think Carr bends the stick a bit too far the other way. I know from my own experience that there's a part of my life labelled Online Stuff, and that most of my reward for doing Online Stuff is attention from other people doing Online Stuff. Real-world payoffs - money, work or just making new real-world friends - are nice to get, but they're not what it's all about.

The real trouble is that Karp has it backwards. Usenet - where I started doing Online Stuff, ten years ago - is a model of open-ended mutual whuffie exchange. (A very imperfect model, given the tendency of social groups to develop boundaries and hierarchies, but at least an unmonetised one.) Systematised whuffie trading came along later. The model case here is eBay, where there's a weird disconnect between meaning and value. Positive feedback doesn't really mean that you think the other person is a "great ebayer" - it doesn't really mean anything, any more than "A+++++" means something distinct from "A++++" or "A++++++". What it does convey is value: it makes it that much easier for the other person to make money. It also has attention-value, making the other person feel good for no particular real-world reason, but even this is quantifiable ("48! I'm up to 48!").

Ultimately Dyson and Carr are both right. The 'attention economy' of Online Stuff is new, absorbing and unlike anything that went before - not least because the way in which it gratifies fantasies of being truly appreciated, understood, attended to. But, to the extent that the operative model is eBay rather than Usenet, it is nothing other than a subset of the financial economy. Karp may be right about the specific case of MySpace, but I can't help distrusting his exuberance - not least because, in my experience, the suffix '2.0' is strongly associated with a search for new ways to cash in.

Not a fish at all

2006-04-27T16:17:00.000+01:00

On the subject of broadcast vs broadband, Tom writes:

There's nothing rapid about this transition at all. It's been happening in the background for fifteen years. So let me rephrase it in ways that I understand. Shock revelation! A new set of technologies has started to displace older technologies and will continue to do so at a fairly slow rate over the next ten to thirty years!
...
My sense of these media organisations that use this argument of incredibly rapid technology change is that they're screaming that they're being pursued by a snail and yet they cannot get away! 'The snail! The snail!', they cry. 'How can we possibly escape!?'. The problem being that the snail's been moving closer for the last twenty years one way or another and they just weren't paying attention.

In comments, Will writes:

If one person is claiming that the world is moving fairly slowly, and has some sound advice on what this might look like (as you are doing here), and another person is claiming that the world is moving extraordinarily quickly, but offers some quickfire measures through which to cope with this, the sense of emergency will win purely because it is present. From here, it almost becomes *risky* not to then adopt the quickfire measures suggested by the second person. Panic becomes a safer strategy than calmness. Which explains management consultancy...

and John asks:

does web2.0 count as a snail too?

But Web 2.0 is not a snail.

Web 2.0 is the people pointing and shouting 'The snail! The snail!'

Web 2.0 is also the people who overhear the first group and join in, shouting 'The whale! The whale!' and pointing vaguely upwards and towards the nearest ocean.

Web 2.0 is also the people who hear the second group and panic about the approaching whale, or is it a land-whale? what is a land-whale anyway? whatever it is, there's one coming and we'd all better... well, we'd better tell someone about it, anyway - I mean, there's a land-whale coming, how often does something like that happen?

Web 2.0 is also the people who hear the third group and improvise a land-whale parade, with floats and dancers and drummers and at its centre a giant paper land-whale held aloft by fifteen people, because, I don't know, but everyone was talking about land-whales and it just seemed like a good idea, you know?

And Web 2.0 is the people who come along halfway through the parade and sell the roadside spectators standing-room tickets.

Cloudbuilding (3)

2006-04-27T11:25:00.000+01:00

By way of background to this post - and because I think it's quite interesting in itself - here's a short paper I gave last year at this conference (great company, shame about the catering). It was co-written with my colleagues Judith Aldridge and Karen Clarke. I don't stand by everything in it - as I've got deeper into the project I've moved further away from Clay's scepticism and closer towards people like Carole Goble and Keith Cole - but I think it still sets out an argument worth having.

Mind the gap: Metadata in e-social science

1. Towards the final turtle

It’s said that Bertrand Russell once gave a public lecture on astronomy. He described how the earth orbits around the sun and how the sun, in turn, orbits around the centre of our galaxy. At the end of the lecture, a little old lady at the back of the room got up and said: “What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise.”

Russell smiled and replied, “What is the tortoise standing on?”

“You’re very clever, young man, very clever,” said the old lady. “But it’s turtles all the way down.”

The Russell story is emblematic of the logical fallacy of infinite regress: proposing an explanation which is just as much in need of explanation as the original fact being explained. The solution, for philosophers (and astronomers), is to find a foundation on which the entire argument can be built: a body of known facts, or a set of acceptable assumptions, from which the argument can follow.

But what if infinite regress is a problem for people who want to build systems as well as arguments? What if we find we’re dealing with a tower of turtles, not when we’re working backwards to a foundation, but when we’re working forwards to a solution?

WSDL [Web Services Description Language] lets a provider describe a service in XML [Extensible Markup Language]. [...] to get a particular provider’s WSDL document, you must know where to find them. Enter another layer in the stack, Universal Description, Discovery, and Integration (UDDI), which is meant to aggregate WSDL documents. But UDDI does nothing more than register existing capabilities [...] there is no guarantee that an entity looking for a Web Service will be able to specify its needs clearly enough that its inquiry will match the descriptions in the UDDI database. Even the UDDI layer does not ensure that the two parties are in sync. Shared context has to come from somewhere, it can’t simply be defined into existence. [...] This attempt to define the problem at successively higher layers is doomed to fail because it’s turtles all the way up: there will always be another layer above whatever can be described, a layer which contains the ambiguity of two-party communication that can never be entirely defined away. No matter how carefully a language is described, the range of askable questions and offerable answers make it impossible to create an ontology that’s at once rich enough to express even a large subset of possible interests while also being restricted enough to ensure interoperability between any two arbitrary parties.
(Clay Shirky)

Clay Shirky is a longstanding critic of the Semantic Web project, an initiative which aims to extend Web technology to encompass machine-readable semantic content. The ultimate goal is the codification of meaning, to the point where understanding can be automated. In commercial terms, this suggests software agents capable of conducting a transaction with all the flexibility of a human being. In terms of research, it offers the prospect of a search engine which understands the searches it is asked to run and is capable of pulling in further relevant material unprompted.

This type of development is fundamental to e-social science: a set of initiatives aiming to enable social scientists to access large and widely-distributed databases using ‘grid computing’ techniques.

A Computational Grid performs the illusion of a single virtual computer, created and maintained dynamically in the absence of predetermined service agreements or centralised control. A Data Grid performs the illusion of a single virtual database. Hence, a Knowledge Grid should perform the illusion of a single virtual knowledge base to better enable computers and people to work in cooperation.
(Keith Cole et al)

Is Shirky’s final turtle a valid critique of the visions of the Semantic Web and the Knowledge Grid? Alternatively, is the final turtle really a Babel fish — an instantaneous universal translator — and hence (excuse the mixed metaphors) a straw person: is Shirky setting the bar impossibly high, posing goals which no ‘semantic’ project could ever achieve? To answer these questions, it’s worth reviewing the promise of automated semantic processing, and setting this in the broader context of programming and rule-governed behaviour.

2. Words and rules

We can identify five levels of rule-governed behaviour. In rule-driven behaviour, firstly, ‘everything that is not compulsory is forbidden’: the only actions which can be taken are those dictated by a rule. In practice, this means that instructions must be framed in precise and non-contradictory terms, with thresholds and limits explicitly laid down to cover all situations which can be anticipated. This is the type of behaviour represented by conventional task-oriented computer programming.

A higher level of autonomy is given by rule-bound behaviour: rules must be followed, but there is some latitude in how they are applied. A set of discrete and potentially contradictory rules is applied to whatever situation is encountered. Higher-order rules or instructions are used to determine the relative priority of different rules and resolve any contradiction.

Rule-modifying behaviour builds on this level of autonomy, by making it possible to ‘learn’ how and when different rules should be applied. In practice, this means that priority between different rules is decided using relative weightings rather than absolute definitions, and that these weightings can be modified over time, depending on the quality of the results obtained. Neither rule-bound nor rule-modifying behaviour poses any fundamental problems in terms of automation.

Rule-discovering behaviour, in addition, allows the existing body of rules to be extended in the light of previously unknown regularities which are encountered in practice (“it turns out that many Xs are also Y; when looking for Xs, it is appropriate to extend the search to include Ys”). This level of autonomy — combining rule observance with reflexive feedback — is fairly difficult to envisage in the context of artificial intelligence, but not impossible.

The level of autonomy assumed by human agents, however, is still higher, consisting of rule-interpreting behaviour. Rule-discovery allows us to develop an internalised body of rules which corresponds ever more closely to the shape of the data surrounding us. Rule-interpreting behaviour, however, enables us to continually and provisionally reshape that body of rules, highlighting or downgrading particular rules according to the demands of different situations. This is the type of behaviour which tells us whether a ban is worth challenging, whether a sales pitch is to be taken literally, whether a supplier is worth doing business with, whether a survey’s results are likely to be useful to us. This, in short, is the level of Shirky’s situational “shared context” — and of the final turtle.

We believe that there is a genuine semantic gap between the visions of Semantic Web advocates and the most basic applications of rule-interpreting human intelligence. Situational information is always local, experiential and contingent; consequently, the data of the social sciences require interpretation as well as measurement. Any purely technical solution to the problem of matching one body of social data to another is liable to suppress or exclude much of the information which makes it valuable.

We cannot endorse comments from e-social science advocates such as this:

variable A and variable B might both be tagged as indicating the sex of the respondent where sex of the respondent is a well defined concept in a separate classification. If Grid-hosted datasets were to be tagged according to an agreed classification of social science concepts this would make the identification of comparable resources extremely easy.
(Keith Cole et al)

Or this:

work has been undertaken to assert the meaning of Web resources in a common data model (RDF) using consensually agreed ontologies expressed in a common language [...] Efforts have concentrated on the languages and software infrastructure needed for the metadata and ontologies, and these technologies are ready to be adopted.
(Carole Goble and David de Roure; emphasis added)

Statements like these suggest that semantics are being treated as a technical or administrative matter, rather than a problem in its own right; in short, that meaning is being treated as an add-on.

3. Google with Craig

To clarify these reservations, let’s look at a ‘semantic’ success story.

The service, called “Craigslist-GoogleMaps combo site” by its creator, Paul Rademacher, marries the innovative Google Maps interface with the classifieds of Craigslist to produce what is an amazing look into the properties available for rent or purchase in a given area. [...] This is the future….this is exactly the type of thing that the Semantic Web promised
(Joshua Porter)

‘This’ is is an application which calculates the location of properties advertised on the ‘Craigslist’ site and then displays them on a map generated from Google Maps. In other words, it takes two sources of public-domain information and matches them up, automatically and reliably.

That’s certainly intelligent. But it’s also highly specialised, and there are reasons to be sceptical about how far this approach can be generalised. On one hand, the geographical base of the application obviates the issue of granularity. Granularity is the question of the ‘level’ at which an observation is taken: a town, an age cohort, a household, a family, an individual? a longitudinal study, a series of observations, a single survey? These issues are less problematic in a geographical context: in geography, nobody asks what the meaning of ‘is’ is. A parliamentary constituency; a census enumeration district; a health authority area; the distribution area of a free newspaper; a parliamentary constituency (1832 boundaries) — these are different ways of defining space, but they are all reducible to a collection of identifiable physical locations. Matching one to another, as in the CONVERTGRID application (Keith Cole et al) — or mapping any one onto a uniform geographical representation — is a finite and rule-bound task. At this level, geography is a physical rather than a social science.

The issue of trust is also potentially problematic. The Craigslist element of the Rademacher application brings the social element to bear, but does so in a way which minimises the risks of error (unintentional or intentional). There is a twofold verification mechanism at work. On one hand, advertisers — particularly content-heavy advertisers, like those who use the ‘classifieds’ and Craigslist — are motivated to provide a (reasonably) accurate description of what they are offering, and to use terms which match the terms used by would be buyers. On the other hand, offering living space over Craigslist is not like offering video games over eBay: Craigslist users are not likely to rely on the accuracy of listings, but will subject them to in-person verification. In many disciplines, there is no possibility of this kind of ‘real-world’ verification; nor is there necessarily any motivation for a writer to use researchers’ vocabularies, or conform to their standards of accuracy.

In practice, the issues of granularity and trust both pose problems for social science researchers using multiple data sources, as concepts, classifications and units differ between datasets. This is not just an accident that could have been prevented with more careful planning; it is inherent in the nature of social science concepts, which are often inextricably contingent on social practice and cannot unproblematically be recorded as ‘facts’. The broad range covered by a concept like ‘anti-social behaviour’ means that coming up with a single definition would be highly problematic — and would ultimately be counter-productive, as in practice the concept would continue to be used to cover a broad range. On the other hand, concepts such as ‘anti-social behaviour’ cannot simply be discarded, as they are clearly produced within real — and continuing — social practices.

The meaning of a concept like this — and consequently the meaning of a fact such as the recorded incidence of anti-social behaviour — cannot be established by rule-bound or even rule-discovering behaviour. The challenge is to record both social ‘facts’ and the circumstances of their production, tracing recorded data back to its underlying topic area; to the claims and interactions which produced the data; and to the associations and exclusions which were effectively written into it.

4. Even better than the real thing

As an approach to this problem, we propose a repository of content-oriented metadata on social science datasets. The repository will encompass two distinct types of classification. Firstly, those used within the sources themselves; following Barney Glaser, we refer to these as ‘In Vivo Concepts’. Secondly, those brought to the data by researchers (including ourselves); we refer to these as ‘Organising Concepts’. The repository will include:

• relationships between Organising Concepts
‘theft from the person’ is a type of ‘theft’

• associations between In-Vivo Concepts and data sources
the classification of ‘Mugging’ appears in ‘British Crime Survey 2003’

• relationships between In-Vivo Concepts
‘Snatch theft’ is a subtype of the classification of ‘Mugging’

• relationships between Organising Concepts and In-Vivo Concepts
the classification of ‘Snatch theft’ corresponds to the concept of ‘theft from the person’

The combination of these relationships will make it possible to represent, within a database structure, a statement such as

Sources of information on Theft from the person include editions of the British Crime Survey between 1996 and the present; headings under which it is recorded in this source include Snatch theft, which is a subtype of Mugging

The structure of the proposed repository has three significant features. Firstly, while the relationships between concepts are hierarchical, they are also multiple. In English law, the crime of Robbery implies assault (if there is no physical contact, the crime is recorded as Theft). The In-Vivo Concept of Robbery would therefore correspond both to the Organising Concept of Theft from the person and that of Personal violence. Since different sources may share categories but classify them differently, multiple relationships between In-Vivo Concepts will also be supported. Secondly, relationships between concepts will be meaningful: it will be possible to record that two concepts are associated as synonyms or antonyms, for example, as well as recording one as a sub-type of the other. Thirdly, the repository will not be delivered as an immutable finished product, but as an open and extensible framework. We shall investigate ways to enable qualified users to modify both the developed hierarchy of Organising Concepts and the relationships between these and In-Vivo Concepts.

In the context of the earlier discussion of semantic processing and rule-governed behaviour, this repository will demonstrate the ubiquity of rule-interpreting behaviour in the social world by exposing and ‘freezing’ the data which it produces. In other words, the repository will encode shifting patterns of correspondence, equivalence, negation and exclusion, demonstrating how the apparently rule-bound process of constructing meaning is continually determined by ‘shared context’.

The repository will thus expose and map the ways in which social data is structured by patterns of situational information. The extensible and modifiable structure of the repository will facilitate further work along these lines: the further development of the repository will itself be an example of rule-interpreting behaviour. The repository will not — and cannot — provide a seamless technological bridge over the semantic gap; it can and will facilitate the work of bridging the gap, but without substituting for the role of applied human intelligence.

Seldom a dread

2006-04-26T15:48:00.000+01:00

It's been quiet around here for a while, and probably will be for a while yet. For now, a small question. Is anyone reading this? More specifically, is anyone reading this in Britain? Even more specifically, is anyone reading this who is in Britain and knows about academic funding, in particular how to obtain and where from? (I've got a few ideas, but more is generally better.) Drop me a comment if so.

Searching low and high

2006-04-04T15:09:00.000+01:00

Update 14th June: it's fixed. The search I describe below now returns 91 results on both Google and Yahoo!. (And one on MJ12 (thanks Paulie), but it's early days.)

Help - Google's broken.

Google's 'exact phrase' search, to be precise. Earlier today I was looking for an English counterpart to the French phrase 'basse police' (elsewhere I've rendered it as 'low policing', following J-P. Brodeur, but the idiomatic content of the phrase gets lost that way). If in doubt, Google - so I googled

"basse police" definition

secure in the knowledge that Google would find the French 'définition' as well as the unaccented English word. And it's true, I didn't need to worry about that; the word 'definition' was present and correct, with and without accent. The only trouble was, only 19 of the pages Google brought back (1-87 of about 67,000) also included the phrase 'basse police'; in particular, none of the first 66 results displayed included the phrase, although some included the word 'basse' and others the word 'police'.

It gets worse (for Google). I tried the same query on Yahoo and got results 1-64 of about 114 (about 114?). Here are the first few, minus a couple of duplicates:

qu'elle participe de la définition de ses fins et qu'elle n'est pas dénuée ... l'ordre semble d'abord relever de la basse police

entre " haute " police et " basse " police, entre surveillance d'un territoire et surveillance ... des services secrets sont, par définition, opaques

Ces méthodes de basse police ont déjà eu lieu à Genève avec les persécutions du Parti Communiste ... Ta définition de "stalinien" est fausse

utiliser (pertinemment) les expressions "haute police " et " basse police ... je veux voir le mot et sa définition

And so it goes on. You see what they've done there? Yahoo has brought back pages containing both the word 'definition' and the phrase 'basse police', and only those pages. Fiendish.

To be fair to Google, this is a problem I've only noticed in the last couple of days. To revert to being hard on Google, it's a major, major, service-vitiating-if-not-actually-disabling problem, and I would like to know what on earth they were thinking of to allow it to happen. (And I'd like it fixed, obviously.)

We are bored in the city

2006-03-27T11:46:00.000+01:00

Et la piscine de la rue des Fillettes. Et le commissariat de police de la rue du Rendez-Vous. La clinique médico-chirurgicale et le bureau de placement gratuit du quai des Orfèvres. Les fleurs artificielles de la rue du Soleil. L'hôtel des Caves du Château, le bar de l'Océan et le café du Va et Vient. L'hôtel de l'Epoque.

Et l'étrange statue du Docteur Philippe Pinel, bienfaiteur des aliénés, dans les derniers soirs de l'été. Explorer Paris.

The early situationists, following Chtcheglov's lead, turned urban wandering into a form of political/psychological exploration, a group encounter with the city mediated only by alcohol. At a less exalted level, I've long been fascinated by the kind of odd urban poetry evoked here, in Manchester as much as Paris, and by the changing articulation of city space: established cities are a slow-motion example of Marx's dictum about how we make our lives within conditions we have inherited. So it's easy to see how well this could work:

Socialight lets you put virtual "sticky" notes called StickyShadows anywhere in the real world. Share pictures, notes and more using your cell phone.

But - for all that the site says about restricting access to Groups and Contacts - it's also easy to see how very badly it could work.

* I leave a note for all my friends at the mall to let them know where I'm hanging out. All my friends in the area see it.
* A woman shows all her close friends the tree under which she had her first kiss.
* An entire neighborhood gets together and documents all the unwanted litter they find in an effort to share ownership of a community problem.
* A food-lover uses Socialight to share her thoughts on the amazing vanilla milkshakes at a new shop.
* The neighborhood historian creates her own walking tour for others to follow.
* A group of friends create their own scavenger hunt.
* A tourist takes place-based notes about stores in a shopping district, only for himself, for a time when he returns to the same city.
* A small business places StickyShadows that its customers would be interested in finding.
* A band promotes an upcoming show by leaving a StickyShadow outside the venue.

It was all going so well (although I did wonder why that entire neighbourhood couldn't just pick up the litter) right up to the last two. Advertising - yep, that's just what we all want more of in our urban lives. Lots of nice intrusive advertising.

Anne:

The worst thing about taking-for-granted that our experiences with the city and each other will be "enriched" by more data, by more information, by making the invisible visible, etc., is that we never have to account for or be accountable to how.

More specifically, there's a huge difference between enabling conversation and enabling people to be informed - in other words, between talking-with and being-talked-at. Social software is all about conversation - about enabling people to talk together. Moreover, any conversation is defined as much by what it shuts out as what it includes; it's hard to listen to the people you want to talk with when you're being talked at. Even setting aside the information-overload potential of all those overlapping groups (do I need to know where so-and-so had her first kiss? do I need to know now?), it's clear that Socialight is trying to serve two ends which are not only incompatible but opposed - and only one of which pays money. Which is probably why, even though the technology is still in beta, I already feel that using it constructively would be going against the grain.

Cloudbuilding (2)

2006-03-03T16:30:00.000Z

Here's a problem I ran into, halfway through building my first ontology, and some thoughts on what the solution might be.

Question 47 of the Mixmag survey reads:

Have you ever had an instance[sic] where your drug use caused you to:
Get arrested?
Lose a job?
Fail an exam?
Crash a car/bike?
Be kicked out of a club?

What this tells us is that one of the things the Mixmag questionnaire is 'about' - one of the in vivo concepts (or groups of in vivo concepts) that we need to record - is misadventures consequent on drug use. The question is how we define this concept logically - and this isn't just an abstract question, as the way that we define it will affect how people can access the information. There are three main possibilities.

1. Model the world
We could say that to have a job is to be a party to a contract of employment, which is a type of agreement between two parties, which is agreed on a set occasion and covers a set timespan. Hence to lose a job is to cease to be a party to a previously-agreed contract of employment; this may occur as a consequence of drug use (defined, in the Mixmag context, as the use of a psychoactive substance other than alcohol and tobacco).

This is all highly logical and would make it explicit that the Mixmag data contains some information on terminations of contracts of employment (as well as on drug-related stuff). However, the Mixmag survey isn't actually about contracts of employment, and doesn't mandate the definitional assumptions I made above. So this isn't really legitimate. (It would also be incredibly laborious, particularly when we turn our attention away from the relatively succinct Mixmag survey and look at more typical social survey data: surveys of physical capacity, for example, routinely ask people whether they can (a) walk to the shops (b) walk to the Post Office (c) walk to the nearest bus stop, and so on down to (j) or (k). All, in theory, capable of being modelled logically - but perhaps only in theory.)

2. Stick to the theme
Alternatively, we could begin by taking a view as to the key concepts which a data source is about - in this case, psychoactive consumption, feelings about psychoactive consumption, consequences of psychoactive consumption, and sexual behaviour - and draw the line at anything beyond those concepts. On this assumption the fact that the survey covers misadventures consequent on drug use would be within scope, but the list of misadventures given above wouldn't be: that's part of the data that researchers will find when they look at the data source itself, not part of the conceptual 'catalogue' that we're building. The advantage of this is that it's conceptually very 'clean' and makes it that much clearer what a source is about; the disadvantage is obviously that it cuts off some ways in to the data and hides some information.

3. Include black boxes
What I've got at the moment - following the principle of using the definitions supplied by the source - is an ontology in which some concepts are defined and others are undefined (black boxes). For instance, I've got a concept of Job loss, but all that OWL 'knows' about it is that it's a type of Misadventure (which may be consequent on drug use) - which is in turn a type of Life event, (which is a type of event that happens to one person). This would allow anyone searching for events consequent on drug use to get to job loss as a type of misadventure, but wouldn't let them get to drug-related misadventure from job loss - unless they happened to enter the exact name of the 'job loss' concept. I'm coming to believe that this is unsatisfactory: we should define the model in terms of what a data source is about. This means that we've got to either take a narrow, domain-specific view or take the view that each source gives us one piece of a much larger picture - in which case we're inevitably committed to modelling the world. But the 'black box' option isn't really sustainable.

Cloudbuilding (1)

2006-03-03T16:09:00.000Z

This one's about work.

I'm currently documenting the concepts underlying the 2005 Mixmag Drug Survey using Protege. Here's why:

The documentation of social science datasets on a conceptual level, so as to make multiple datasets comprehensible within a shared conceptual framework, is inherently problematic: the concepts on which the data of the social sciences are constructed are imprecise, contested and mutable, with key concepts defined differently by different sources. When a major survey release is published, for example, the accompanying metadata often includes not only a definition of key terms, but discussion of how and why the definitions have changed since the previous release. This information is of crucial importance to the social scientist, both as a framework for understanding statistical data and as a body of social data in its own right.

It follows that we cannot think in terms of ironing out inconsistencies between social science datasets and resolving ambiguities. Rather, documenting the datasets must include documenting the definitions of the conceptual framework on which the datasets are built, however imprecise or inappropriate these concepts might appear in retrospect. This will also involve preserving - and exposing - the variations between different sources, or successive releases from a single source.

There are currently two main approaches to conceptually-oriented data documentation. A ‘top down’ approach is exemplified by the European Language Social Sciences Thesaurus (ELSST). The Madiera portal allows researchers to explore ELSST and access European survey data which has been linked to ELSST keywords. The limitations of the top-down approach can be gauged from ELSST’s concepts relating to drug use. Drug Abuse, Drug Addiction, Illegal Drugs and Drug Effects are all 'leaf' concepts - headings which have no subheadings under them. However, they are in different parts of the overall ELSST tree: for example, Drug Abuse is under Social Problems->Abuse, while Drug Effects is under Biology->Pharmacology. Although the hierarchy is augmented by a list of 'related' concepts, to some extent facilitating horizontal as well as vertical navigation, the hierarchy inevitably makes some types of enquiry easier than others. Anyone using the ELSST 'tree' will be visually reminded of the affinities identified by ELSST’s authors between Pharmacology and Physiology, or between Drug Abuse and Child Abuse. These problems follow from the initial design choice of a single conceptual hierarchy.

This approach to classification has recently come under criticism. Advocates of 'bottom-up' approaches argue that top-down taxonomies like the Dewey Decimal System or ELSST are an artificial imposition on the world of knowledge, which is better represented as a set of individual acts of labelling or ‘tagging’. It is argued that the 'trees' of hierarchical taxonomies can be replaced with a pile of 'leaves'.

One successful 'bottom-up' approach is the framework for documenting survey data developed by the Data Documentation Initiative (DDI). The DDI standard makes it possible to search on keywords associated with surveys, sections of surveys and individual questions; the short text of individual questions is also searchable. Searches of DDI metadata can also be run from the Madiera portal: a search on ‘marijuana’, for instance, brings back short text items including the following:

CONSUMED HASHISH,MARIJUANA
- Health Behaviour in School-Aged Children (Switzerland, 1990)

Smoking cannabis should be legal? Q2.31
- Scottish Social Attitudes Survey (Scotland, 2001)

Q92C DRUGS EV B OFFERED - MARIJUANA
- Eurobarometer 37.0 (EU-wide, 1992)

Clearly, this way in to the data makes it easy for a well-prepared researcher to track the use of particular concepts 'in the wild' (in vivo concepts). However, this gain comes at the cost of some information. There is wide variation both in the terminology used in the surveys and in the concepts to which they refer. In one survey smoking cannabis might be a type of petty crime; in others it might figure as a type of leisure activity or a potential health risk. These conceptual differences are reflected in the vocabulary used by data sources - and by researchers. Depending on context, three researchers using 'marijuana', 'hashish' and 'cannabis' as search terms may be asking for the same data or for three different sets of data.

Neither the 'top-down' nor the 'bottom-up' approach articulates the conceptual assumptions which underlie the construction of a dataset - assumptions expressed both in the definition of in vivo concepts and in relationships between them. Rather than leaving much of this conceptual information undocumented (the DDI approach) or encoding one 'correct' set of assumptions while excluding or sidelining others (the ELSST approach), we propose to offer a coherent hierarchy of in vivo concepts for each individual source, based on the definitions (explicit and implicit) used in each source. Comparing the in vivo conceptual hierarchies used in multiple datasets will enable researchers both to see where concepts are directly comparable and to see where - and how - their definitions diverge and overlap.

To document hierarchies of in vivo concepts, we shall use description logic and the Semantic Web language OWL-DL (Web Ontology Language - Description Logic). OWL-DL makes it possible to formulate a precise logical specification of concepts such as

- use of cannabis (either marijuana or hashish) in the month prior to the survey
- use of either Valium or temazepam, at any time
- seizures of Class A drugs by HM Customs in the financial year 2004/5

At least, that's the idea. Now wait for part 2...

Nor mine, now

2006-03-02T13:07:00.000Z

I nearly installed Hyperwords this morning; the only reason I didn't is that I haven't moved to Firefox 1.5 yet (and don't intend to until I'm confident it won't break any of the extensions I'm already using). And, in principle, it looks great:

With the Hyperwords Firefox Extension installed just select any text and a menu appears. You can search major search engines, look things up in reference sites, check dictionary definitions, translate, email quickly and much more.

So why does the thought of actually using it give me the creeps? Alex is similarly ambivalent:

In principle, it's a handy tool. But I would have to overcome a few personal adoption barriers before I started using it on a regular basis. As a consumer, I can see the appeal of opening up texts to interact with the rest of the Web; but as a writer, I instinctively bristle at the idea of giving up that kind of control. I suspect that disposition colors the way I read things on the Web; I like my documents to feel fixed, not fluid. And the Web feels squishy enough as it is. That, and somehow the premise of cracking open someone else's document with a toolbox of Web services feels like a kind of violation. This is undoubtedly my own personal neurotic hangup.

Well, if it is, it's mine too. Mark Bernstein gets some of it:

In the very early days of hypertext research, people worried a lot about hand-crafted links. "How will we ever afford to put in all those links?" We also worried about how we'd ever manage to afford to digitize stuff for the Web, not to mention paying people to create original Web pages. Overnight, we discovered that we'd got the sign wrong: people would pay for the privilege of making Web sites. The problem isn't the 'tyranny' of the links, and replacing it with the tyranny of the link server might not be a great solution.

and

Authors don't offer navigation options to be "useful"; thoughtful writers use links to express ideas. Argumentation seeks understanding, not merely access.

Let's put some of that together: cracking open someone else's document with a toolbox of Web services; the tyranny of the link server; thoughtful writers use links to express ideas. In other words, Hyperwords doesn't extend existing hyperlink practice but undermines it. In the Hyperwords world you'll no longer read a document, you'll mine it for information - or rather, mine it for jumping-off points for retrieving information from authoritative sources. (Or retrieving whatever other stuff you may want to retrieve.)

Alex mentioned Xanadu, but I don't think Hyperwords is a step in that direction. If anything, it's a step backwards. (One of Xanadu's key words is "author-based".) Hyperlinks and the Web of dialogic, socially-produced content go together just fine; as Mark says, mass amateurism is already providing an answer to the question of where all those links are going to come from. It's messy and incomplete, but it's here - and it's, well, ours (as a writer, I instinctively bristle at the idea of giving up that kind of control). You can see two visions of the Web here: the mass amateurisation of writing as against the 'consumer'-oriented, authority-led, broadcast Web. Hyperwords ostensibly enhances horizontal, transverse linkage, but its effect would be to pull the Web further towards broadcast mode - albeit an 'empowered', roll-your-own broadcast mode.

Can't keep quiet for long - I'm a human being!
Can't help singing this song - I'm a human being!
You won't listen to me,
I'm not an authority...
- Steve Mason, "Eclipse"

All the things I could do

2006-02-28T12:30:00.000Z

But (for new readers, this is point 2; point 1 is here, and you should go and read it immediately), it's becoming clear that Web 2.0 is all about the walled gardens. As I wrote in that post, In the context of social software, when I use a word like 'enclose' - or a word like 'monetise' - it means something quite specific and entirely negative: it's a red-flag word. Which means that, oddly, when I started reading Russell Beattie's WTF 2.0 I found a lot to agree with.

The worst thing about all the Web 2.0 hype is the complete loss of business perspective. There’s a few companies out there that seem to get it but just about every other new website I’ve seen lately is nothing but features parading as businesses. Sure, these guys get to be entered in the “Flip It Quick Acquisition Lottery”, but beyond that, none seem to be creating anything of any real value.

"Features masquerading as businesses", the "Flip It Quick Acquisition Lottery" - all good stuff. Except that Russell's objections aren't quite the same as mine.

You can create a new website, fill it with all the goodness in the world, be good to your users, and be a good netizen and use every open standard there is while you’re at it, if at the end of the day your users didn’t put money into your bank account, it’s a useless waste of time for everyone involved. I mean, hey, if you want to create the next non-profit service like Wikipedia, all the more power too you. But if you want to get VC cash, an office in downtown Palo Alto, do a bunch of development, attract lots of users and pretend you’re a business? Then act like one, create something of real value and make some real money from it.

"Real value", "real money". You don't have to be a Marxist to suspect that those aren't necessarily the same thing (although, to be honest, it does help). In the next paragraph Russell draws a hazy distinction between the two himself:

look at the Weblog federations for example. They’re making money like people have done for a hundred years or so: hire writers, sell some ads, publish using standard technologies. Nothing too innovative, but they’re making money and I totally dig that. Then again, those writers are generating real value, IMHO, so there’s something there to make money from.

Russell commends the Weblog federations, whoever they are (didn't they have trouble with the spice routes a while back?), for making money. He then stresses that they're also creating real value, which means there’s something there to make money from - but 'real value' is qualified rather worryingly with 'IMHO', suggesting that it may or may not be real. At the end of the day the money's real, though, and Russell digs that.

Russell then reminds us that things are different in the 'mobile world'. (If your immediate reaction to this sentence was "Damn right, things are obscenely expensive in the mobile world", or words to that effect, you're ahead of me already.)

I deal with companies every day who have no qualms about charging 25 cents to send 160 characters of data from one person to another, or who have no problems charging $3.00 for a 10kb .gif image or a bad .midi version of a popular song, or even up to $10.00 for a small Java clone of Tetris - a 20 year old game. Unlike the web world, the mobile world is accustomed to charging for every thing that has the slightest bit of value. The difference between the markets couldn’t be more drastic. I know of a mobile chat site that’s on many carrier decks that’s a great example of this. To use it, you need to sign up to a subscription for $3.00 a month, and in return you get a URL which links to a very basic WAP based chat. This would be okay in my mind if there was some sort of extra special functionality, but there’s not.

Follow this reasoning. Money is being charged; in Russell's mind this would be okay if there was 'extra special functionality' involved; but there isn't. So, by implication, it's not okay. The money is real, the value isn't. An equally poor service which was free would be better. A better service which was free would be better still. Right? Well...

But don’t get me wrong, it’s not that this is a bad service or a rip off - they are providing a chat app as promised and it works. It’s just the fact that this particular app could be written by any developer in the Valley in less than an hour, and yet they easily have thousands if not millions of paying subscribers world wide.

The part about how the value isn't real and it's not okay? Forget that. The value is real, obviously, because they are providing a chat app as promised and it works. In other words, the measure of the value of a service is the fact that people are willing to pay for it. And if people aren't paying for a service that has value to them (because it does stuff that they want it to do), then that's just wrong and we shouldn't encourage them.

Why will people gladly pay $3.00 for a basic mobile chat site and not pay anything for a decent web service? I think it’s mostly because of expectations, and honestly, the naivete of many of the people trying to start “businesses” on the web today.

Really, the hype around Web 2.0 has got to stop until all concerned stop acting like a bunch of hippies and start concentrating on what really matters, which is of course money:

I really do think there should be a litmus test for new web apps launched from now on - something very basic and if they don’t pass, they don’t qualify for any buzz or linkage. It’s a simple test: Will they take my credit card? That’s it. I don’t care if they have advertisers or sponsors or god knows what else, all I want to see is a place where I can type in my credit card for some service.

Money: that's what Russell wants. Or rather, that's what he wants to be charged. After all, if you're giving it away, it can't have very much value.

Ultimately, for Russell, there are two very simple questions which software developers need to be able to answer if they're going to have any hope of jumping the Web 2.0 train. Do you want to get VC cash and an office in downtown Palo Alto, or not? And if not, WTF is wrong with you?

Talk to my machine

2006-02-28T11:52:00.000Z

Today, two loosely-related points about social software. Here's the first. When I heard about coComment, it seemed like a really good idea; I signed up not once but twice (once for each of my main blogs). (Yes, I've got more than two blogs. Sort of. It's a long story.) But I've been increasingly dissatisfied with it since then, and Ben Metcalfe has explained why.

What I hadn't realised was that the coComment bookmarklet submit[s] the comment to both the coComment server and the original blog server. Consequently,

at the point of submission your comment is essentially semantically forked - with a version going into coComment and an identical version going into the blog server.

Ugh. As Ben says, if the blog administrator - or, in the case of sites which allow comment editing, the commenter hirself - chooses to edit the content of the comment, it isn’t reflected in the coComment representation of the post conversation. The possibilities for abuse are obvious - look at the comments thread below this post.

What's worse is that the coComment representation of discussion is only of those who have also used coComment to submit their comment. Ugh^2. This is yet another attempt at snowball-effect marketing, in other words: coComment becomes useful when it gains momentum, which it gets from adopters (like me) who started using it before it was useful, in the hope that it would gain sufficient momentum to become useful.

To which I can only say, sod that for a game of soldiers. Flickr would still be useful for me if I were the only Flickr user in the world; Simpy would still be useful if I were the only user. And so on - it's part of the definition of social software that it's useful for everyone, even for a single user with no interest in its 'socialness'. First you build functionality that works, then you extract value from the use of that functionality, then you expose that value back to the users. (Or user.) And, er, that's it. At no point in the process do you say "hold on, we need to get more people in here before we go on".

It could have been so different - although I confess that a profound ignorance of the underlying technology is lurking behind that 'could'. When I comment on a blog that I don't follow, what I want is to grab a comment feed from that specific post and look at it along with other comment feeds from blog posts I've commented on (excluding blogs whose post feeds I read - but in the first instance those exclusions could be managed manually). And, er, that's it - I don't want or need to bring a third party into the equation.

Apps like coComment are street performers - without a big crowd looking the same way there's no event. Apps like Simpy are Katamari meetings: the crowd is the event. It's obvious to me which of the two looks more like 'social software'. Unfortunately it's also obvious which of the two is easier to monetise.

Which brings us to the second point...

A mean idea to call my own

2006-02-15T16:24:00.000Z

Technorati's new "Filter by Authority" feature depresses me intensely - not least because I thought they'd abandoned the word 'authority' some time after my last rant on the subject. There are three problems here. Firstly, as I wrote last year:

Technorati is all about in-groups and out-groups. ... authority directly tracks popularity - although this is 'popularity' in that odd American high-school sense of the word: 'popular' sites aren't the ones with the most friends (most out-bound links, most distinct participants in Comments threads or even most traffic) but the ones with the most people envying them (hence: most in-bound links).

In other words, 'authority' is a really lousy synonym for 'high inbound link count', raising completely groundless expectations of quality and reliability. McDonald's is a popular provider of hot food; it's not an authority on cooking. The relative popularity (or enviability) of a site may signify many things, but it doesn't signify that the site possesses absolute qualities like veracity, completeness, beauty - or authority.

But hold on - is it absurd to call McDonald's authoritative? You've got to admit, they're good at what they do... There's a sense in which this is a tautology - because what they do is maximise the numbers who come through the doors - but never mind. Let's say that we can identify the McDonald's branch with the highest number of burgers sold (or repeat customers, or stars on uniforms - the precise metric doesn't matter). There's a good argument for using the word 'best': it looks like this is the best McDonald's branch in the world. And the best fast food joint in the world? Well, maybe. The best restaurant in the world? Um, no. Quality tracks popularity, to some extent, but only within a given domain - otherwise USA Today would be the best newspaper in the USA . (To say it's the best national mass-market tabloid would be less controversial.) [Edited with thanks to commenters who know about this stuff.]

This is the second problem with authority-as-link-count, and one which Technorati shows no sign of recognising, much less addressing. I can live with the idea that the Huffington Post is more popular than Beppe Grillo's blog - but more authoritative? I really don't think so. (Any right-wingers reading this may substitute Huffington for Grillo and Kos for Huffington, and re-read. And rest.) At bottom, Technorati's 'authority' ranking is based on the laughably outdated idea that there is a single Blogosphere, within which we're all talking to pretty much the same people about pretty much the same things. Abandon that assumption and the problems with an 'authority' metric are staringly apparent: who am I authoritative for? who am I more authoritative than?

But if this is an error it's not an error of Dave Sifry's invention. As I've said, within any given domain of ideas, it's not entirely meaningless to say that authority tracks popularity: among academic authors, the author who sells books and fills halls is likely to be the author who is cited, even if he or she hasn't written anything particularly inspired since Thatcher was in power. The question is whether this is a feature or a bug: if we're going to read one writer rather than another, should we choose the popular dullard or the unknown genius? Put it another way: if we're choosing who to read in the context of a new publication medium with massively lowered entry costs - and with an accompanying ideology rich in levelled playing-fields, smashed barriers and dismantled hierarchies - who should we be trying to seek out: Dullard (Popular) or Genius (Unknown)?

The third and most fundamental problem with ranking by 'authority' is that it brings to the Web one of the very features of offline life which Web evangelists told us we were leaving behind. This kind of 'feature' - and the buzz-chasing worldview that promotes it - is part of the problem, not part of the solution.

I find that it often helps me to also answer the question, "Who is the most influential blogger talking about XXX this week, and what did she say?" - Dave Sifry

We climbed and we climbed

2006-02-13T11:21:00.000Z

I don't trust Yahoo!, for reasons which have nothing to do with my dislike of misused punctuation marks (although the bang certainly doesn't help); I don't trust Google either. Maybe it's because I'm old enough to remember when MicroSoft [sic] were new and exciting and a major attractor of geek goodwill; maybe it's just because I'm an incurable pinko and don't trust anyone who's making a profit out of me. Anyway, I don't trust Yahoo!, or like them particularly; I switched to Simpy when Yahoo! bought del.icio.us, and I've felt a bit differently about Tom - hitherto one of my favourite bloggers anywhere - since he joined Yahoo!.

Still. This (PDF) is Tom's presentation to the Future of Web Apps conference, and it's good stuff - both useful and beautiful, to use William Morris's criteria. The fourth rule (precept? guideline? maxim?) spoke to me particularly clearly:

Identify your first order objects and make them addressable

Start with the data, in other words; then work out what the data is; then make sure that people (and programs) can get at it. (Rule 5: "Use readable, reliable and hackable URLs".) It's a simple idea, but surprisingly radical when you consider its implications - and it's already meeting resistance, as radical ideas do (see Guy Carberry's comments here).

More or less in passing, Tom's presentation also shows why the Shirkyan attempt to counterpose taxonomy to folksonomy is wrongheaded. If you're going to let people play with your data (including conceptual data), then it needs to be exposed - but if you're going to expose data in ways that people can get at, you need structure. And it doesn't matter if it's not the right structure, not least because there is no right structure (librarians have always known this); what matters is that it's consistent and logical enough to give people a way in to what they want to find. To put it another way, what matters is that the structure is consistent and logical enough to represent a set of propositions about the data (or concepts). Once you've climbed that scaffolding, you can start slinging your own links. But ethnoclassification builds on classification: on its own, it won't get you the stuff you're looking for - unless what you're looking for isn't so much the stuff as what people are saying about stuff. (Which is why new-media journalists and researchers like tagging, of course.)

Anyway - very nice presentation by the man Coates. Check it out.

It's just work

2006-02-08T16:44:00.000Z

Suw Charman types too fast. She's produced what looks like a fascinating record of the Future of Web Apps conference, but I can't see myself ever reading the whole thing. But this jumped out at me (slight edits):

Joshua Schachter - The things we've learned
Tagging is not really about classification or organisation, it's user interface. It's a way to store your working state or context. Useful for recall. OK for discovery because someone might tag similarly to you. Bad for distribution.

Not all metadata is tags. People ask for automatic metadata, but that's not the value - the value is attention, that you saw it and decided that it was important enough to tag. Auto-tagging doesn't help you do what you're trying to do. ... because there's a small transaction cost that adds value. But don't make them do too much work.

the value is attention ... because there's a small transaction cost, that adds value The value of tagging is in the meaning it encodes, and the meaning is created by people doing a bit of work. If you make things easy by automating the process of getting meaning out of data, that creativity is not called upon and what you get doesn't have the same value.

This parallels my thoughts about the impoverishment of technology through the collapse of alternative ways of using it, often in the name of ease of use - not to mention the thoughts I put down on my other blog about how the best communication (and the best narrative) is gappy and open to multiple interpretations. One way of understanding why gappiness and plurivalence might be a positive virtue, finally, is suggested by Anne, who counterposes predictability and foretelling to potentiality and hope.

I think what all these arguments have in common is a sense of meaning as not-yet-(finally)-constructed. In this perspective the point of social software, in particular, is not to connect data but to enable people to talk about data - while preventing that talk from being entirely weightless by imposing a certain level of friction, a certain opportunity cost. (A cost which can always be raised or lowered. Thought experiment: Wikipedia makes it impossible to revert an article to a version less than a week old. What happens?) In the case of tagging systems, there has to be a reason why you would want to tag a resource, and want to tag it in ways that have meaning for you. Meaning is created through conversations that require a bit of effort, within the shared context of an open horizon: it's work, but it's work without a known outcome. A journey of hope, as someone wrote.

(My blogs are crossing over - I hate it when that happens...)

And the high plains too

2006-02-06T12:48:00.000Z

Tom comments on this post from last year:

Thoughts: (1) Pledgebank is about increasing the perceived effect of ones actions by connecting it to a larger purpose (2) Wikipedia already seems to have that mechanism but (3) I like the idea of building social processes alongside wikipedia a lot...

Yes and No to point 2. Wikipedia already has social reinforcement/reputation feedback effects built in, but they only really work once you're on the inside. If you're on the outside, the fabled dedication and energy of the Wikipedia community is actually a barrier - not least because, if you're unlucky, all that dedication and energy will be applied to reversing your edits. (Think of Thomas Vander Wal's discovery that he disagreed radically with Wikipedia's definition of 'folksonomy', and his subsequent struggle to get the definition changed - the point here being that Thomas actually coined the term, and not that long ago.)

This isn't a new discovery: reputation-based regulation inevitably creates a barrier to entry, as anyone who's tried to get noticed on Usenet can confirm. Reputation adds a bit of friction to the weightless process of making your mark online, and adds a bit of glue to the shapeless aggregate of people who do it; the fact that you have to build up a bit of reputation before your words gain traction is, mostly, feature rather than bug.

So is the Pledgebank idea reinventing the wheel, simply trying to use reputation-based peer pressure to mobilise a group who could have been subjecting themselves to Wikipedia peer pressure all along? I don't think so. Compared with a Usenet newsgroup or a Web board community, Wikipedia has a couple of curious and atypical aspects. Firstly, the currency of Wikipedia reputation-building is work, and plenty of it. I've known people make a reputation on Usenet with a single post. The size and complexity of Wikipedia makes that highly unlikely. Secondly, Wikipedia is unusual in parallelling areas where people already have reputations, built up through domain-specific conversations. As always, issues of authority and reliability come into sharpest focus when the area's one that you know personally. I can say that, if you're interested in processes of consensus-formation in an area of hotly contested political debate, the Wikipedia page on the Lega Nord makes fascinating reading. If you're interested in getting some reasonably authoritative views on the Lega Nord, it's no substitute for reading the literature. This isn't to say that Wikipedia is wrong - but it's less right than it could be. And this is partly because Wikipedia's informal reputation management mechanisms are orthogonal to the mechanisms which produce subject area experts, and partly because Wikipedia's mechanisms operate to repel anyone who isn't committed to building a Wikipedia reputation - perhaps because they're more interested in building one within their subject area.

Hence the proposed Wikipedant posse. If - like me and Tom and Thomas - you've seen something on Wikipedia & thought That's just wrong, but it would take a long time to fix it; and if you not only (a) know stuff, but (ii) know when you don't know something and (3) know how to find stuff out; then this could be your kind of thing. The idea is simple: we compile a list of wrong-but-timeconsuming Wikipedia pages (usually involving simplistic or tendentious renderings of a subject); we dish them out, presumably at random; and, when we get assigned a page, we take ownership of it and try to put it right. This wouldn't be a lifetime commitment, but it would almost certainly involve a couple of months of checking back and reverting unhelpful edits, on top of the researching and writing time.

I'll be appealing to pedants, autodidacts and (OK, I admit it) academics rather than Wikipedia enthusiasts, and I'll be appealing on a strictly time-limited basis rather than trying to create new Wikipedians. It will, unavoidably, involve quite a lot of work, which is why I'll be calling in aid an external source of peer pressure in the form of Pledgebank.

And I'll be doing this... some time soon. This year, definitely. (Terrors of the earth, I'm telling you.)

Update I wrote:

I'll be appealing to pedants, autodidacts and (OK, I admit it) academics

and

Wikipedia's mechanisms operate to repel anyone who isn't committed to building a Wikipedia reputation - perhaps because they're more interested in building one within their subject area.

Which perhaps isn't precisely the impression I gave last September, when I wrote:

I'll just reiterate that I'm not talking about people with expert knowledge, so much as perfectionists with inquiring minds.

What a difference a few months' full-time employment makes. (I was a freelance journalist from 1999 to 2004, and kept it up on a part-time basis until last summer.) Let's split the difference: subject experts will be welcome, just as long as they're also perfectionists with inquiring minds. (Which of course they will be, what with being subject experts and everything.)

You may look like we do

2006-02-02T15:40:00.000Z

David cites an empirical analysis of social network evolution in a large university community, based on a registry of e-mail interactions between more than 43,000 students, faculty, and staff. ("Hey, gang, let's do the research right here!")

The results show that at least in this particular environment, people were more likely to form ties with others when they had a shared "focus" such as a class that brought them together or a mutual acquaintance, but were less likely to interact solely on the basis of shared characteristics such as age or gender.

David headlines his post "Interests, not demographics", but I don't think the study is quite saying that. It's true that demographics do not a network make - but then, I've known that ever since my mother first enjoined me to play with a complete stranger of my own age and sex while she talked to the kid's mother, who wasn't a complete stranger (to her).

But I don't think the data's there to conclude that 'interests' are key either, as much as I might like to. The reference to a shared "focus" such as a class that brought them together or a mutual acquaintance sounds more like history than interests. It may be a reasonable generalisation to say that enduring communities are interest-based - particularly if we include the granfalloonish limit case of communities which perpetuate themselves by making a shared interest of their own perpetuation. Conversations, though, just happen. A conversation starts for any number of reasons - not least because two people find each other simpatico/a - and once it's started the participants generally want to carry it on. History, not interests.

From this it also follows that there are times when conversations just don't happen, and all the shared interests in the world won't make them happen. And, given that people who are having a conversation generally want it to continue, there are sometimes very few gaps in which a new conversation can get a foothold. Which brings us back to the granfalloons. Perhaps we can see some communities as large-scale conversations which have outlived any connection with interest, for many or most of the participants, but still persist - and, by persisting, prevent new and potentially interest-based conversations from arising.

(I can be a phenomenologist and a Marxist, can't I?)

Home again

2006-01-30T11:25:00.000Z

So, I'm a researcher. (At least until the money runs out next year; hopefully I'll have something similar lined up by then.) Before I was a researcher I was a freelance journalist for about six years, while I did my doctorate; before that I was a full-time journalist for three years; and before that I worked in IT. Which is a whole other dark and backward abysm of time - I was a Unix sysadmin, and before that I was an Oracle DBA, and before that... database design, data analysis, Codasyl[1] database admin, a ghastly period running a PC support team, and before that systems analysis and if you go back far enough you get to programming, and frankly I still don't trust any IT person who didn't start in programming. (I'm getting better - at one time I didn't trust anyone who didn't start in programming.)

Now, there's an odd kind of intellectual revelation which you sometimes get, when you're a little way into a new field. It's not so much a Eureka moment as a homecoming moment: you get it, but it feels as if you're getting it because you knew it already. You feel that you understand what you've learnt so fully that you don't need to think about it, and that everything that's left to learn is going to follow on just as easily. Which usually turns out to be the case. The way it feels is that the structures you're exploring are how your mind worked all along - or, perhaps, how your mind would have been working all along if you'd had these tools to play with. (Or: "It's Unix! I know this!")

I had that feeling a few times in my geek days - once back at the start, when I was loading BASIC programs off a cassette onto an Acorn Atom (why else would I have carried on?); once when I was introduced to Codasyl databases; and once (of course) when I met Unix, or rather when I understood piping and redirection. But the strongest homecoming moment was when, after being trained in data analysis, I saw a corporate information architecture chart (developed by my employer's then parent company, with a bit of help from IBM). Data analysis hadn't come naturally, but once I'd got it it was there - and, now that I had got it, just look what you could do with it! It was a sheet of A3 covered with lines and boxes, expressing propositions such as "a commercial transaction takes place between two parties, one of which is an organisational unit while the other may be an individual or an organisational unit"; propositions like that, but mostly rather more complex. I thought it was wonderful.

Fast forward again: database design, DBA, sysadmin, journalism, freelancing, PhD, research. Research which, for the last month or so, has involved using OWL (the ontology language formerly known as DAML+OIL) and the Protege logical modelling tool - which has enabled me to produce stuff like this.

It's not finished - boy, is it not finished. But it is rather lovely. (Perhaps I just like lines and boxes...)

[1] If you don't know what this means, don't worry about it. (And if you do, Hi!)

The hippies were evil

2006-01-26T12:04:00.000Z

Like a lot of people, I've been playing around with the Chinese version of Google. If my searches are anything to go by, they don't seem bothered about whether you know about the Dalai Lama, but they do seem to be concerned that you get the right sources on Falun Gong (which is a very bad thing) and the Taishi Village incident (which probably never happened). The nastiest piece of censorship I've seen so far concerns the BBC news site, which seems to be blocked in its entirety. But I've only scratched the surface, and obviously I can't speak for the results of Chinese-language searches.

Here in Britain, Google redirects 'google.com' invocations to google.co.uk, from where you can choose to go to google.com if you really want to. Danny explains: "Google routinely redirects those outside the US to a country-specific version of Google. Those who want to reach Google.com can do so by selecting the "Google.com in English" link on the home page of these versions." (The link on the .co.uk page doesn't specify 'in English'.) So in China it's still possible to use google.com as well as google.cn. Similarly in France and Germany, where google.fr and google.de search results are silently censored to comply with legislation banning neo-Nazism. But it gets worse: thanks to the Digital Millennium Copyright Act there is (visible) censorship on google.com, which appears to be replicated on all the country-specific Googles. (Try searching for 'kazaalite'.) And, of course, we don't know whether there is any silent censorship. Considering that we know of effective public campaigns to bring pressure to bear on Google, it seems unlikely that no pressure has been exerted behind the scenes. (More on google.fr and google.de here; these are some of the sites that are hidden, and here's how it's done.)

David's argument is typical of the bloggers who are calling for Google to be shown some leniency with regard to this one. I don't share his conclusions, primarily because I don't agree with his premise. His argument seems to start from identification with the people at Google who have had a hard choice to make and have done, in their judgment, the best they could: "It's a tough world. Most of what we do is morally mixed.", and so forth. But for me the operative metric is not the relative quality of service Google can provide the people of China - which is certainly higher under the google.cn regime - but Google's relative complicity with restrictions on the free flow of information. This is important because of Google's extraordinarily unusual position. The company does one thing; the one thing it does - provide information - is an unqualified ethical good; and it does it well. Any complicity with censorship tarnishes the company's ethical reputation; it also threatens its reputation for delivering its service well, since it suggests that this can be compromised by external considerations. The Google.cn story threatens Google on both these grounds. Previously, Google was guilty of tolerating censorship; now, it's guilty of assisting censorship.

David concedes that this story "shows once and for all that Google's motto is just silly in a world as complex as this one". I'd go further. Unless you take the (Lutheran?) view that obedience to the government - any government - is a pious duty, Google's co-operation with the Chinese government has made nonsense of their proclaimed commitment to avoid evil. But I'd also add that Google crossed that line some time ago, when they doctored search results to comply with French, German and US law - and, in the case of France and Germany, did so without any indication that results were incomplete.

Ultimately this is primarily a lesson about what Will (in a completely different context) calls digital exuberance - and about the enthusiasm for big business - as long as it's a cool big business which I identified as a growing element of "Web 2.0". (Hang on to those double-quotes, you'll be glad of them later.) Owen sums it up:

I now notice that the corporate philosophy illustrates "don’t be evil" with the example that advertisements should be unobtrusive; and [Eric Schmidt and Hal Varian, writing in Newsweek] interpreted it to mean that management should not throw chairs. Google never actually said they would not cut a deal with an undemocratic regime to deny information and access to news to hundreds of millions of repressed people. But that was the kind of thing that "don’t be evil" implied to me.

I have some sympathy with Google’s dilemma - they are, after all, a shareholder-owned company, not a branch of Reporters sans frontières. But companies that say one thing and do another eventually get themselves into trouble.

Google was once the underdog; a quirky startup, doing one thing (search) really well: and quickly without all those annoying ads. We got cool free gizmos, like Google Earth and webmail with big storage. And it seemed to have a corporate philosophy that hackers and the internet generation could relate to. Today Google seems a lot more like Microsoft, AOL or any other large corporation. It buys companies to get their technology (what exactly has Google invented, since PageRank?). It introduces Digital Rights Management systems for video. And now it cuts deals with the Chinese government to expand its market, instead of standing up for uncensored access to the internet.

Could do better

2006-01-09T10:32:00.000Z

"Yes, I know, I know. 2005's over already, we're more than halfway through the decade... you're not the only one who's embarrassed, how do you think I feel? Yes, I know it was supposed to be all m-computing and always-on GPS and ubiquitous wifi jetpacks by now, but it's been difficult, I mean, look at the state of the economy... OK, OK, I shouldn't have said 'jetpacks', forget I said that, no, I am taking this seriously, really I am... but come on, apart from anything else there's been this war going on, that hasn't helped... yes, I know military spending is supposed to be a major driver of high-tech R&D... maybe it just isn't driving R&D the way we'd like it to, have you thought of that? look at Segways, they were going to be the next big thing at one stage... OK, OK, I'll stop trying to change the subject... Look, what can I say? We'll do better this year. I'll do better this year. Trust me. OK? OK."

- Web promises to become more pervasive in 2006

Soft enough for you

2006-01-06T11:03:00.000Z

Anne:

Is it reasonable to have to learn to ride a bike but expect a computer to be as simple to figure out as a toaster? (Not the perfect analogy I know, but you know what I'm getting at...) Some days I think that user-friendliness was/is a really bad idea, not least because it's obdurate, so hard to change.

If you have to work at using a technology, in other words, you necessarily end up working with it and through it. You work to adapt it to your needs - and you adapt it. Technologies which offer ease of use, by contrast, make it easy to work in certain pre-defined ways - and resist adaptation by the individual user. (There are, of course, technologies which are both easy to use and flexible - ask any Flickr user. But I think the 'user-friendliness' Anne is talking about here is more like the comment a tutor of mine once made on the BBC and 'open access' broadcasting: "They say they'll come and help you, show you how to do it. They don't, of course - what they do is show you how to do what you do because that's how you do it." User-friendliness is very often a matter of HTDWYDBTHYDI.)

But there's more to it than that. What is this thing called obduracy? Anne again:

[Anique Hommels] argues that one way to emphasise the material aspects [of technologies in society] is to focus on their obduracy or resistance to change. (Imagine what it would *actually* take to replace the infrastructure that currently provides our electricity with something more sustainable.) The notion of obduracy is inextricably connected to embeddedness - a matter of interest to any kind of computing that seeks to become part of something else, be it an event, a habit, a skirt, a chair, a building, a street, a city. As Hommels reminds us, obduracy (or embeddedness) is a relational concept:

"Because the elements of a network are closely interrelated, the changing of one element requires the adaptation of other elements. The extent to which an artifact has become embedded determines its resistance to efforts aimed at changing it."

An embedded technology, then, would be one which has behind it a community of people who do a certain thing in a certain way. Becoming a user entails enrolment in that community. In short, the technology adapts you.

Where does this leave user-friendliness? Perhaps we could think of the embedding of a new technology as a process, which can continue to the point of the collapse of the possible ends and uses inherent in the technology and its reduction to the status of tool: a toaster, not a bicycle. And perhaps a 'user-friendly' technology - at least in the HTDWYD sense - is one designed to enlist a tool-using community and collapse its own potential into instrumentality.

(Relatedly, from Dan Hills' essential critique of digital music: "there is a powerful necessity to think long term; to not take such short cuts which may inadvertently delete possible outcomes; to enable the flexibility and endless modifications seen in previous generations of music devices". Dan has a lovely quote from William Gibson: "That which is overdesigned, too highly specific, anticipates outcome; the anticipation of outcome guarantees, if not failure, the absence of grace.")

More broadly, what all this highlights is the value of difficulty, incompatibility, misunderstanding. Dan also led me (indirectly) to this quote from the late Derek Bailey:

There has to be some degree, not just of unfamiliarity, but incompatibility [with a partner]. Otherwise, what are you improvising for? What are you improvising with or around? You've got to find somewhere where you can work. If there are no difficulties, it seems to me that there's pretty much no point in playing. I find that the things that excite me are trying to make something work. And when it does work, it's the most fantastic thing.

One of the great frustrations in my work with ontologies and e-social science is the recurrent assumption that the concepts used in social science data can be documented cleanly and consistently - or, conversely, that if they can't be documented cleanly and consistently they're not worth documenting. The point, surely, is to find ways of recording both the logic of individual classifications and the incompatibilities between them - and the (qualified, partial) correspondences between them. And, of course, to make this documentation changeable over time, without effacing the historical traces which contribute to its meaning. Parenthetically, it's worth noting here that preservation of historical data has nothing to do with obduracy. History is not obdurate, having no power to resist and (by and large) no enrolled community; the erasure of history can facilitate embeddedness and instrumentality, while the preservation of an artifact's history may actually preserve resources of flexibility. (That's enough abstractions - Ed.)

We're never together

2006-01-05T12:09:00.000Z

Back here, I wrote:

Social software may start with connecting data, but what it's really about is connecting people - and connecting them in dialogue, on a basis of equality. If this goal gets lost, joining the dots may do more harm than good.

It's not about connecting machines, either - and the same caveat applies. Via Thomas, I recently read this item about location-based services (which, I remember, were going to be quite the thing a couple of years ago, although they seem to have faded since people started actually getting their hands on 3G technology). Anyway, here are the quotes:

This project focuses on [location-based technology's] collaborative uses: how group of people benefits from knowing others' whereabouts when working together on a joint activity ... we set up a collaborative mobile environment called CatchBob! in which we will test how a location awareness tool modifies the group interactions and communications, the way they perform a joint task as well as how they rely on this spatial information to coordinate.

And how did that work out?

"We found that players who were automatically aware of their partners’ location did not perform the task better than other participants. In addition, they communicated less and had troubles reminding their partners' whereabouts (which was surprising). These results can be explained by the messages exchanged. First the amount of messages is more important in the group without the location-awareness tool: players had then more traces to rely on in order to recall the others’ trails. And when we look at the content, we see that players without the location-awareness tool sent more messages about position, direction or strategy. They also wrote more questions."

Really, we're back with 'push' technology - which was going to be quite the thing round about 1998, as I remember. Give people device for talking to each other: works. Give people device which gives them a constant stream of information: doesn't work.

The trouble is, we've got the technology. The problems with social software are social; see this deeply depressing Register story.

Alongside video on demand TV services from Homechoice, the SDB [Shoreditch Digital Bridge] will offer a "Community Safety Channel" which will allow residents "to monitor estate CCTV cameras from their own living rooms, view a 'Usual Suspects' ASBO line up, and receive live community safety alerts."
...
Other aspects of the Shoreditch Digital Bridge are less controversial, but likely to be considerably harder to execute. The SDB proposes an education channel, "allowing children and adults to take classes, complete on-line homework assignments and log-on to 'virtual tutors'", a "Health Channel" allowing patients to book GP appointments, and providing "virtual Dr/Nurse consultations and on-line health and diagnosis information", a "Consumer Channel, allowing on-line group buying of common services such as gas, electricity and mobile phone tariffs", and an "Employment Channel, providing on-line NVQ courses, local jobs website and virtual interview mentoring."

So within that little lot, the educational aspects will require substantial input from, and involvement of, existing schools and colleges, the Health Channel will need a whole new interface to NHS systems that are already struggling to implement their own new electronic booking systems, and the Consumer Channel will merely have to reinvent the co-operative movement electronically.

But CCTV - ah, now, we've got CCTV...

Will:

Yet again, the technology arrives promising us a vibrant civic and economic future ... then beds down as a means of protecting us from each other.

Or rather, as a means of protecting us from Them (caution - sweary link).

If we're talking about social software or social networks, let's be clear that we're talking about connecting people rather than dividing them. Connecting machines doesn't necessarily help connect people.

When the sweet turns sour

2005-12-14T12:43:00.000Z

I’ve! just! exported! my! bookmarks! and! deleted! my! account!.
- Sophie, in comments at Burningbird

If this keeps up all the "Web 2.0" blog nerds will be working at Yahoo! by next month.
- Jake
Yup! I think that's the plan!
- Tom
(comments at plasticbag.org)

delicious was not only a community. It was also an experiment. A place for us geeks to meet and discuss. A place where we were changing the Web. Yes, WE were changing the Web through our ideas. And Joshua was good in picking the best ideas. Inviting us to give more. Now do you really think this will continue under Yahoo!’s reign?
- Pietro

Some lessons to learn here:
1. Never trust a startup service to store your important data no matter how the owner seems honest to you.
2. Never trust a corporate entity to continue storing your important data.
3. Never act like a fanboy on services you don’t trust.
- Ronald Johnson, in comments at the del.icio.us blog

Companies offer web services to get free ideas, exploit free R&D, and discover promising talent. They offer the APIs so people can build clever toys, the best of which the company will grab -- thank you very much -- and develop further on their own. There is no business model for mashups. If Web 2.0 really is just mashups, this is going to be one short revolution.
- Greg

This enthusiasm for big business - as long as it's a cool big business - strikes me as both dangerous and weird, not to mention being the antithesis of what's made the Net fun to work with all these years. But it is a logical development of one branch of the 'Web 2.0' hype - an increasingly dominant branch, unfortunately.
- me (on Google)

I promise not to be successful if you all give me money.
- Shelley

Update: I've switched to Simpy. It's great.

In any English town

2005-12-12T11:20:00.000Z

Mark Honigsbaum, the Guardian:

Curators, librarians and archivists across Britain are being asked to scour their collections in search of documents and items relating to the lives of gay people, with a view to establishing a "virtual museum" of lesbian and gay history.

Backed by the museums documentation watchdog, MDA, the group Proud Heritage this week began sending out a two-page survey requesting that institutions throughout the country list the gay and lesbian documents and artefacts in their collections. "For the first time ever, we are asking museums, libraries and archives throughout Britain to revisit their holdings and reveal what they have that is queer," said Proud Heritage's director Jack Gilbert. "At the moment these are not classified correctly, or held completely out of context and never see the light of day."
[...]
According to Mr Gilbert the aim is to establish a national database first, featuring a few key virtual exhibits. Once the database was up and running, he said, Proud Heritage would look for a site for a permanent museum, possibly in the King's Cross area of London.

At the moment these are not classified correctly... You don't have to be Dave Weinberger to have mixed feelings about that statement. Clearly there's a case for saying that many of these artifacts aren't classified adequately, inasmuch as historians of gay experience don't have an obvious point of access to them - and this could be provided by the proposed database. And, clearly, tagging an artifact with 'gay' doesn't preclude tagging it with 'Wales' and 'early nineteenth-century society' (the Ladies of Llangollen) or with 'literature' and 'penal reform' (Wilde's cell at Reading Gaol).

But classifying an artifact only as 'gay' would, in almost all imaginable cases, be no more 'correct' than classifying it under any other single term. The project of a physical museum of gay history is welcome in terms of visibility, but in taxonomic terms it's a step back from the purely 'virtual' database project. Like any other thematically-organised museum, it would consist - almost by definition - of exhibits which were 'not classified correctly' and 'held completely out of context'.

Museums promote the illusion that the map is the territory: the structure and layout of the galleries, and the arrangement of the exhibits they contain, are designed to reproduce a certain way of structuring knowledge. (The perfect museum would be its own memory palace.) But an illusion is what it is. Objects can only reside in one place, but knowledge can be fluid and multi-dimensional; pressures to collapse those dimensions - whether in the name of group identity or commerce - should be resisted.

Not that group identity and commerce are necessarily that far apart.

Denis Campbell, the Guardian:

Six per cent of the population, or about 3.6 million Britons, are either gay or lesbian, the government's first attempt to quantify the homosexual population has concluded.
[...]
Publication of the figure comes as big name companies such as Barclays bank, Hilton hotels and cosmetics giant L'Oreal join the growing rush to cash in on a gay economy which is worth tens of billions of pounds. Barclays has just received research which showed that gays and lesbians enjoy a combined annual income of £60 billion.
[...]
Barclays spokesman Michael O'Toole admitted the bank is very keen to woo Britain's gays and lesbians by portraying itself as sympathetic to gays' desire for equality. 'We want to position ourselves as the bank of choice for Britain's gay and lesbian community,' he said. 'There's more of a push going on now to enter this market of about 2.5 million adults.'

The key word here is 'market'. If Barclay's is planning to make it easier for gay couples to take out mortgages and insurance policies, this is all to the good, but O'Toole's ambitions clearly go further. The gay 'market' is not like, say, the 'market' represented by devout Muslims: Barclay's plan for those 2.5 million adults is not to introduce them to personal banking, but to encourage them, firstly, to identify as gay; secondly, to perceive Barclay's as a gay-friendly bank; and thirdly, to switch to Barclay's on that basis. Identifying as English or middle-aged, a Frascati-drinker or a Manchester City supporter, a dog-owner or a Labour voter would just get in the way: if you're gay, Barclay's is the bank for you. The benefits for the bank are obvious; the benefits for their prospective customers, less so.

Knowledge - including our knowledge about ourselves - can be fluid and multi-dimensional; pressures to collapse those dimensions should be resisted.

PS I usually let my titular quotes stand with their cover unblown, but in this case I'll make a partial exception. For me these are probably the best - certainly the most moving - four lines ever written on the subject of exploring knowledge and where it leaves you. What I'm still not sure about is whether it's a despairing renunciation, a challenge or a celebration. Cue music:

About the university the pages let you down
It helps you find your way around in any English town
About the university the pages are in French
It helps you find your way around in any English town

We are your friends

2005-11-23T14:28:00.000Z

I recently attended an "e-Government Question Time" session, organised in connection with this conference. There were some good points made: one speaker stressed the importance of engaging with the narratives which people build rather than assuming that the important facts can be read off from an accumulation of data; one questioner called the whole concept of 'e-government' into question, pointing out that the stress seemed to be entirely on using the Web/email/digital TV/texting/etc as a mechanism for delivering services rather than as a medium for democratic exchanges. Much more typical, though, was the spin which the mediator put on this question as he passed it on to the panel:

That's a very good question - what about democracy? And conversely, if it's all democracy where does that leave leadership?

The evening was much less about democracy than it was about leadership - or rather, management. This starting-point produced some strikingly fallacious arguments, particularly in the field of privacy. The following statements were all made by one panellist; I won't single him out, as they were all endorsed by other panellists - and, in some cases, members of the audience. (And no, identifying him as male doesn't narrow it down a great deal. The people in the hall were 3:1 male to female (approximately), the people on stage 6:1 (precisely).)

I like to protect my own privacy, but I'm in the privileged position of having assets to protect. When you're looking at people who have got nothing, and in many cases aren't claiming benefits to which they're entitled, I don't think safeguarding their privacy should be our main concern.

At first blush this argument echoes the classic Marxist critique of the bourgeois definition of human rights - if we have the right to privacy, what about the right to a living wage? But instead of going from universalism to a broader (and hence more genuine) universalism, we've ended up with the opposite of universalism: you and I can worry about privacy, but it doesn't apply to them. Superficially radical, or at least populist - you can just hear David Blunkett coming out with something similar - this is actually a deeply reactionary argument: it treats the managed as a different breed from the people who manage them (I like to protect my own privacy, but...). Management Fallacy 1: 'they're not like us'.

We're talking about improving people's life chances. We need to make personal information more accessible - to put more access to personal information in the hands of the people who can change people's lives for the better.

Management Fallacy 2: 'we mean well'. If every intervention by a public servant were motivated by the best interests of the citizens, safeguards against improper intervention would not be required. And if police officers never stepped out of line, there'd be no need for a Police Complaints Commission. In reality, good intentions cannot be assumed: partly because the possibility of a corrupt or malicious individual getting at your data cannot be ruled out; partly because government agencies have other functions as well as safeguarding the citizen's interests, and those priorities may occasionally come into conflict; and partly because a government agency's idea of your best interests may not be the same as yours (see Fallacy 3). All of which means that the problem needs to be addressed at the other end, by protecting your data from people who don't have a specific reason to use it - however well-intentioned those people may be. One questioner spoke wistfully of the Data Protection Act getting in the way of creative, innovative uses of data. It's true that data mining technology now makes it possible to join the dots in some very creative and innovative ways. But if it's data about me, I don't think prior consent is too much to ask - and I don't think other people are all that different (see Fallacy 1).

I've got no objection to surrendering some of my civil liberties, so-called

Have to stop you there. Management Fallacy 3: 'it looks all right to me'. The speaker was a local government employee: a private individual. His policy for handling his own private data doesn't concern me. But I would hope that, before he came to apply that policy more generally, he would reflect on how the people who would be affected might feel about surrendering their civil liberties, so-called. (Perhaps he could consult them, even.)

Carry on:

I've got no objection to surrendering some of my civil liberties, so-called, if it's going to prevent another Victoria Climbie case.

Management Fallacy 4: 'numbers don't lie'. (Or: 'Everything is measurable and what can be measured can be managed'.) This specific example is a common error with statistics, which can be illustrated with the example of a hypothetical test for the AIDS virus. Let's say that you've got an HIV test which is 95% accurate - that is, out of every 100 people with HIV it will correctly identify 95 and mis-identify 5, and similarly for people who do not carry the virus. And let's say that you believe, from other sources, that 1,000 people in a town of 100,000 carry the virus. You administer the test to the entire town. If your initial assumption is correct, how many positive results will you get? And how confident can you be, in percentage terms, that someone who tests positive is actually HIV-positive?

The answers are 5900 and 16.1%. The test would identify 950 of the 1000 people with the virus, but it would also misidentify 4950 people who did not have it: consequently, anyone receiving a positive test result would have a five in six chance of actually being HIV-negative. What this points to is a fundamental problem with any attempt to identify rare phenomena in large volumes of data. If the frequency of the phenomenon you're looking for is, in effect, lower than the predictable rate of error, any positive result is more likely to be an error than not.

Contra McKinsey, I would argue that not everything can or should be measured, let alone managed on the basis of measurement. (If the data-driven approach to preventing another Climbie case sounds bad, imagine it with the addition of performance targets.) Some phenomena - particularly social phenomena - are not amenable to being captured through the collection of quantitative data, and shouldn't be treated as if they were.

What all these fallacies have in common is a self-enclosed, almost solipsistic conception of the task of management. With few exceptions, the speakers (and the questioners) talked in terms of meeting people's needs by delivering a pre-defined service with pre-defined goals, pre-defined techniques, pre-defined identities (me service provider, you service recipient). There were only occasional references to the exploratory, dialogic approach of asking people what their needs were and how they would like them to be met - despite the possibilities for work in this area which new technologies have created. But then, management is not dialogue.

Social software may start with connecting data, but what it's really about is connecting people - and connecting them in dialogue, on a basis of equality. If this goal gets lost, joining the dots may do more harm than good.

Which side of the table?

2005-11-17T14:55:00.000Z

[Pardon the long silence - life called.]

Shelley:

I think Google Base is a fun experiment, and I’m willing to play a little. It will be interesting to see the directory, especially if the company provides web services that aren’t limited to so many queries a day. But I never forget that Google is in the business to make a profit. If we give it the power, it will become the Wal-Mart of the waves–by default if not by design. Is that what you all want? If it is, just continue getting all misty eyed, because you’ll need blurred vision not to see what should be right in front of you.

I think this is precisely right. I find a lot of the comment on Google Base strange and slightly depressing, in the same way I find a lot of Web 2.0 talk strange and depressing. In the context of social software, when I use a word like 'enclose' - or a word like 'monetise' - it means something quite specific and entirely negative: it's a red-flag word. So it's weird, to say the least, to see the same words used positively. It's only a little less strange to see these concerns acknowledged, then batted away as trivial or meaningless (Tom: "Making data available for everyone to use is keeping it in the public sphere." (I'm Phil #2 in those comments, by the way)).

It seems to me that there's a fundamental tension between the demands of commerce and the nature of social software, as defined by Tom some time ago:

We believe that for a piece of Social Software to be useful:
Every individual should derive value from their contributions
Every contribution should provide value to their peers as well
The site or organisation that hosts the service should be able to derive value from the aggregate of the data and should be able to expose that value back to individuals

You add content, which has value for you and to other users; the host derives further value from the aggregate of content; the host exposes that added value to all users. Beautiful.

What this suggests is that social software - unlike, say, the e-business of the late 1990s - is all about the content. Specifically, it's all about freely and collectively contributed content, either held in common or held in trust for the commons. So monetising social software is qualitatively different from making money out of a new piece of stand-alone software, because it's the contributed content which has the original value - and makes the added value possible. Tangentially, I'm not sure whether Doc Searls gets this or not, and not only because his article's had a well-deserved slashdotting. On a first reading I got the impression he was saying that both the Net and the Web are valuable common resources which should not be fenced off for the sake of making money, but that part of what makes them valuable is that they're great environments for fencing things off and making money. (Oh, and we should stop saying 'common' because if you put 'ist' on the end it sounds kind of like 'communist', and when Eric Raymond hears the word 'Communist' he reaches for well, you know.) It's a great article, though, and I look forward to taking a more considered look at it when the tide subsides.

One of Doc's points is that the Web is, figuratively, all about publishing - a profession which, like many bloggers, I've seen close up. Just under ten years ago, I went from Unix sysadmin to magazine editor, and rapidly discovered that commercial publishing looks very different from the inside. Perhaps the biggest single shock was the realisation that content doesn't matter. Obviously I tried to make it the best magazine I could (and it got better still under my successor), but at a fundamental level editorial content wasn't what it was about. If the advertising department sold enough space, we made a profit; if they didn't, we didn't. (Show me a magazine that relies on the cover price and I'll show you a magazine with money worries. Show me a publication that gets by on the cover price and I'll show you an academic journal.) The purpose of the magazine was to put advertisements in front of readers - and the purpose of the editorial was to make readers turn all the pages.

So there's nothing very new about Google's business model: Google Base is to the Web what a commercial magazine is to a fanzine - or rather, a whole mass of different fanzines. The only novelty is that we, the fanzine writers, are providing the content: the content whose sole function, from the point of view of Google as a commercial entity, is to attract an audience which will look at ads.

But that's quite a big novelty.

Bill Burnham:

In my next post I will talk about Google Base's impact on the "walled garden" listings sites. I'll give you a hint: it won't be pretty.

Unless, of course, you like really big gardens with really high walls.

Update: I wrote, I find a lot of the comment on Google Base strange and slightly depressing, in the same way I find a lot of Web 2.0 talk strange and depressing. Cue Cringely:

Google has the reach and the resources ... And you know whose strategy this is? Wal-Mart's. And unless Google comes up with an ecosystem to allow their survival, that means all the other web services companies will be marginalized. ... the final result is that Web 2.0 IS Google. Microsoft can't compete. Yahoo probably can't compete. Sun and IBM are like remora, along for the ride. And what does it all cost, maybe $1 billion? That's less than Microsoft spends on legal settlements each year. Game over.

As an aside, I love the idea of International Business Machines as a parasite on the behemoth that is Google; I don't think we're quite there yet. But the accuracy or not of Cringely's prediction concerns me less than his tone, which I think can reasonably be called lip-smacking: "Google's going to 0wn the Web! Wow!" (Quick test: try reading that sentence out loud with a straight face. Now try substituting 'Microsoft' - or, for older readers, 'IBM'.) This enthusiasm for big business - as long as it's a cool big business - strikes me as both dangerous and weird, not to mention being the antithesis of what's made the Net fun to work with all these years. But it is a logical development of one branch of the 'Web 2.0' hype - an increasingly dominant branch, unfortunately.

This is the new stuff

2005-11-03T10:05:00.000Z

Thomas criticises Wikipedia's entry on folksonomy - a term which was coined just over a year ago by, er, Thomas. As of today's date, the many hands of Wikipedia say:

Folksonomy is a neologism for a practice of collaborative categorization using freely chosen keywords. More colloquially, this refers to a group of people cooperating spontaneously to organize information into categories, typically using categories or tags on pages, or semantic links with types that evolve without much central control. ... In contrast to formal classification methods, this phenomenon typically only arises in non-hierarchical communities, such as public websites, as opposed to multi-level teams and hierarchical organization. An example is the way in which wikis organize information into lists, which tend to evolve in their inclusion and exclusion criteria informally over time.

Thomas:

Today, having seen an new academic endeavor related to folksonomy quoting the Wikipedia entry on folksonomy, I realize the definition of Folksonomy has become completely unglued from anything I recognize (yes, I did create the word to define something that was undefined prior). It is not collaborative, it is not putting things into categories, it is not related to taxonomy (more like the antithesis of a taxonomy), etc. The Wikipedia definition seems to have morphed into something that the people with Web 2.0 tagging tools can claim as something that can describe their tool

I'm resisting the temptation to send Thomas the All-Purpose Wikipedia Snark Letter ("Yeah? Well, if you don't like the wisdom of the crowds, Mr So-Called Authority..."). In fact, I'm resisting the temptation to say anything about Wikipedia; that's another discussion. But I do want to say something about the original conception of 'folksonomy', and about how it's drifted.

Firstly, another quote from Thomas's post from today:

Folksonomy is the result of personal free tagging of information and objects (anything with a URL) for one's own retrival. The tagging is done in a social environment (shared and open to others). The act of tagging is done by the person consuming the information.
...
There is tremendous value that can be derived from this personal tagging when viewing it as a collective, when you have the three needed data points in a folksonomy tool: 1) the person tagging; 2) the object being tagged as its own entity; and 3) the tag being used on that object. ... [by] keeping the three data elements you can use two of the elements to find a third element, which has value. If you know the object (in del.icio.us it is the web page being tagged) and the tag you can find other individuals who use the same tag on that object, which may lead (if a little more investigation) to somebody who has the same interest and vocabulary as you do. That person can become a filter for items on which they use that tag. You then know an individual and a tag combination to follow.

This is admirably clear and specific; it also fits rather well with the arguments I was making in two posts earlier this year:

[perhaps] the natural state of knowledge is to be 'cloudy', because it's produced within continuing interactions within groups: knowledge is an emergent property of conversation, you could say ... [This suggests that] every community has its own knowledge-cloud - that the production and maintenance of a knowledge-cloud is one way that a community defines itself.

If 'cloudiness' is a universal condition, del.icio.us and flickr and tag clouds and so forth don't enable us to do anything new; what they are giving us is a live demonstration of how the social mind works. Which could be interesting, to put it mildly.

Thomas's original conception of 'folksonomy' is quite close to my conception of a 'knowledge cloud': they're both about the emergence of knowledge within a social interaction (a conversation).

The current Wikipedia version of 'folksonomy' is both fuzzier and more closely tied to existing technology. What's happened seems to be a kind of vicious circle of hype and expectations management. It's not a new phenomenon - anyone who's been watching IT for any length of time has seen it happen at least once. (Not to worry anyone, but it happened quite a lot around 1999, as I remember...)

There's Vision: someone sees genuinely exciting new possibilities in some new technology and writes a paper on - oh, I don't know, noetic telepresence or virtual speleology or network prosody...
Then there's Development: someone builds something that does, well, a bit of it. Quite significant steps towards supporting network prosody. More coming in the next release.
Phase three is Hype. Hype, hype, hype. Mm-hmm. I just can't get enough hype, can you?
The penultimate phase is Dissemination: in which everyone's trying to support network prosody. Or, at least, some of it. That stuff that those other people did with their tool. There we go, fully network prosody enabled - must get someone to do a writeup.
Finally we're into Hype II, also known as Marketing: 'network prosody' is defined less by the original vision than by the tools which have been built to support it. The twist is that it's still being hyped in exactly the same way - tools which don't actually do that much are being marketed as if they realised the original Vision. It's a bit of a pain, this stage. Fortunately it doesn't last forever. (Stage 6 is the Hangover.)

What's to be done? As I said back here, personally I don't use the term 'folksonomy'; I prefer Peter Merholz's term 'ethnoclassification'. Two of my objections to 'folksonomy' were that it appears to denote an end result as well as a process, and that it's become a term of (anti-librarian) advocacy as well as description; Thomas's criticisms of Wikipedia seem to point in a similar direction. Where I do differ from Thomas is in the emphasis to be placed on online technologies. Ethnoclassification is - at least, as I see it - something that happens everywhere all the time: it's an aspect of living in a human community, not an aspect of using the Web. If I'm right about where we are in the Great Cycle of Hype, this may soon be another point in its favour.

None of you stand so tall

2005-10-31T16:34:00.000Z

In the previous post, I showed that the canonical 'power law' chart which underlies the Long Tail image does not, in fact, represent a power law. What it represents is a ranked list, which happens to have a similar shape to a power law series: as it stands, the 'power law' is an artifact of the way the list has been sorted. In particular, the contrast which is often drawn, in this context, between a power law distribution and a normal distribution is inappropriate and misleading. If you sort a list high to low, it can only ever have the shape of a descending curve.

There are counter-arguments, which I'll go through in strength order (weakest first).

Counter-argument 1: the Argument from Inconsequentiality.

In the post which started it all, Clay wrote:
the shape of Figure #1, several hundred blogs ranked by number of inbound links, is roughly a power law distribution.

Note weasel wordage: it would be possible to argue that what Clay (and Jason Kottke) identified wasn't really a power law distribution, it was just some data which could be plotted in a way which looked oddly like a power law curve. Thankfully, Clay cut off this line of retreat, referring explicitly to power law distributions:

power law distributions are ubiquitous. Yahoo Groups mailing lists ranked by subscribers is a power law distribution. LiveJournal users ranked by friends is a power law ... we know that power law distributions tend to arise in social systems where many people express their preferences among many options.

And so on. When we say 'power law', we mean 'power law distribution': we're all agreed on that.

Except, of course, that what we're talking about isn't a power law distribution. Which brings us to...

Counter-argument 2: the Argument from Intuition.

The pages I excerpted in the previous post specifically contrast the power law distribution with the 'normal' bell curve.

many web statistics don’t follow a normal distribution (the infamous bell curve), but a power law distribution. A few items have a significant percentage of the total resource (e.g., inbound links, unique visitors, etc.), and many items with a modest percentage of the resources form a long “tail” in a plot of the distribution.

we find a very few highly connected sites, and very many nearly unconnected sites, a power law distribution whose curve is very high to the left of the graph with the highly connected sites, with a long "tail" to the right of the unconnected sites. This is completely different than the bell curve that folks normally assume

The Web, like most networks, has a peculiar behavior: it doesn't follow standard bell curve distributions ... [it] follows a power law distribution where you get one or two sites with a ton of traffic (like MSN or Yahoo!), and then 10 or 20 sites each with one tenth the traffic of those two, and 100 or 200 sites each with 100th of the traffic, etc.

One of my Latin teachers at school had an infuriating habit, for which (in the best school-story tradition) I'm now very grateful. If you read him a translation which didn't make sense (grammatically, syntactically or literally) he'd give you an anguished look and say, "But how can that be?" It was a rhetorical question, but it was also - infuriatingly - an open question: he genuinely wanted you to look again at what you'd written and realise that, no, actually that noun in the ablative couldn't be the object of the verb... Good training, and not only for reading Latin.

If you've got this far, do me a favour and re-read the excerpts above. Then ask yourself: how can that be?

As long as we're talking about interval/ratio variables - the only type for which a normal distribution can be plotted - it's hard to make sense of this stuff. What, to put it bluntly, is being plotted on the X axis? The best I can do is to suppose that the X axis plots number of sites: A few items have a significant percentage of the total resource; a very few highly connected sites; one or two sites with a ton of traffic. There's your spike on the left: a low X value (a few items) and a high Y (a significant percentage of the total resource).

But this doesn't really work either. Or rather, it could work, but only if every group of sites with the same number of links had a uniquely different number of members - and if the number of members in each group were in inverse proportion to the number of links (1 site with n links, 2 sites with n/2 links, 3 sites with n/3 links, 4 sites with n/4 links...). This isn't impossible, in very much the same way that the spontaneous development of a vacuum in this room isn't impossible; a pattern like that wouldn't be a power law so much as evidence of Intelligent Design.

This is an elaborate and implausible model; it's also something of a red herring, as we'll see in a minute. It's worth going into in detail, though; as far as I can see, it's the only way of getting these data into a power law distribution, with high numbers of links on the left, without using ranking. And cue...

Counter-argument 3: the Argument from Ranking.

Over to Clay:

The basic shape is simple - in any system sorted by rank, the value for the Nth position will be 1/N. For whatever is being ranked -- income, links, traffic -- the value of second place will be half that of first place, and tenth place will be one-tenth of first place. (There are other, more complex formulae that make the slope more or less extreme, but they all relate to this curve.)

"The value for the Nth position will be 1/N" (or proportionate to 1/N, to be more precise); alternatively, you could say that N items have a value of 1/N or greater. (Have a think about this one - we'll be coming back to it later.) Either way, it's a power law, right? Well, yes - and no. It's certainly true to say that a ranked list with these properties confirms to a version of the power law - specifically, Zipf's law. It's also true to say that Zipfian rankings are associated with Pareto-like power law distributions: we may yet be able to find a power law in this data. But we're not there yet - and Clay's presentation of the data doesn't help us to get there. (Jason's has some of the same problems, but Clay's piece is a worse offender; it's also much more widely known.)

The first problem is with the recurrent comparison of ranked graphs with bell curves. Adam: "a ranked graph ... by definition is *always* decreasing, and can *never* be a bell curve". If anyone tells you that such and such a phenomenon follows a power law rather than a normal distribution, take a good look at their X axis. If they've got ranks there, the statement is meaningless.

Secondly, the graph Clay presented - a classic of the 'big head, long tail' genre - isn't actually a Zipfian series, for the simple reason that it includes tied ranks: it's not a list of ranks but a list of nominals sorted into rank order.

I'll clarify. Suppose that we've got a series which only loosely conforms to Zipf's Law, perhaps owing to errors in the real world:

Rank	Value
1	1000
2	490
3	340
4	220
5	220
6	180
7	140

Now, what happens on the graph around values 4 and 5? If the X axis represents ranking, it makes no sense to say that the value of 220 corresponds to a rank of 4 and a rank of 5: it's a rank of 4, followed by no ranking for 5 and a rank of 6 for the value of 180. We can see the point even more clearly if we take the alternative interpretation of a Zipfian list and say that the X axis tracks 'number of items with value greater than or equal to Y'. Clearly there are 6 items greater than or equal to 180 and 5 greater than or equal to 220 - but it would be nonsensical to say that there are also 4 items greater than or equal to 220. Either way, if you have a ranked list with tied rankings this should be represented by gaps in the graph.

This may seem like a minor nitpick, but it's actually very important. Back to Adam:

One nice thing about a ranked graph is that the “area” under the curve is equal to the total value associated with the items spanned on the ranked axis

Or, in the words of one of the pieces I quoted in the previous post:

In such a curve the distribution tapers off slowly into the sunset, and is called a tail. What is most intriguing about this long tail is that if you add up all the traffic at the end of it, you get a lot of traffic

What we're talking about, clearly, is the Long Tail. Looking at some actual figures for inbound linkage (collected from NZ Bear earlier this year), there are few tied ranks in the higher rankings and more as we go further out: 95 unique values in the first 100 ranks and 79 in the next 100. Further down, the curve grows flatter, as we'd expect. The first ten rankings (ranging from 5,389 down to 2,142 links) correspond to ten sites; the last ten (ranging, predictably, from 9 down to zero) correspond to a total of 14,445. As Adam says, if you were to graph these data as a list of nominals ranked in descending order, the 'area' covered by the curve would give you a good visual impression of the total number of links accounted for by low-linked sites: the Long Tail, no other. But this graphic does not conform to a power law - not even Zipf's Law. A list conforming to Zipf's Law would drop tied ranks - it would exclude duplicates, if that's any clearer. Instead of a long tail, it would trail off to the right with a series of widely-spaced fenceposts. ("In equal 9126th place, blogs with 9 links; in equal 9593rd place, 8-linkers...")

Long Tail, power law: choose one.

You can have a Long Tail, but only by graphing a list of nominals ranked in descending order.

You can have a power law series with rankings, but only by replacing the long tail with scattered fenceposts.

Even more importantly, neither of these is a power law distribution. Given the appropriate data values, you can derive a power law distribution from a ranked list - but it doesn't look like the 'long tail' graphic we know so well. I'll talk about what it does look like in the next post.

Put your head back in the clouds

2005-10-21T10:41:00.000+01:00

OK, let's talk about the Long Tail.

I've been promising a series of posts on the Long Tail myth for, um, quite a while. (What's a month in blog time? A few of those.) The Long Tail posts begin here.

Here's what we're talking about, courtesy of our man Shirky:

We are all so used to bell curve distributions that power law distributions can seem odd. The shape of Figure #1, several hundred blogs ranked by number of inbound links, is roughly a power law distribution. Of the 433 listed blogs, the top two sites accounted for fully 5% of the inbound links between them. (They were InstaPundit and Andrew Sullivan, unsurprisingly.) The top dozen (less than 3% of the total) accounted for 20% of the inbound links, and the top 50 blogs (not quite 12%) accounted for 50% of such links.

Figure #1: 433 weblogs arranged in rank order by number of inbound links.

It's a popular meme, or it would be if there were any such thing as a meme (maybe I'll tackle that one another time). Here's one echo:

many web statistics don’t follow a normal distribution (the infamous bell curve), but a power law distribution. A few items have a significant percentage of the total resource (e.g., inbound links, unique visitors, etc.), and many items with a modest percentage of the resources form a long “tail” in a plot of the distribution. For example, a few websites have millions of links, more have hundreds of thousands, even more have hundreds or thousands, and a huge number of sites have just one, two, or a few.

Another:

if we measure the connectivity of a sample of 1000 web sites, (i.e. the number of other web sites that point to them), we might find a bell curve distribution, with an "average" of X and a standard deviation of Y. If, however, that sample happened to contain google.com, then things would be off the chart for the "outlier" and normal for every other one.

If we back off to see the whole web's connectivity, we find a very few highly connected sites, and very many nearly unconnected sites, a power law distribution whose curve is very high to the left of the graph with the highly connected sites, with a long "tail" to the right of the unconnected sites. This is completely different than the bell curve that folks normally assume

And another:

The Web, like most networks, has a peculiar behavior: it doesn't follow standard bell curve distributions where most people's activities are very similar (for example if you plot out people's heights you get a bell curve with lots of five- and six-foot people and no 20-foot giants). The Web, on the other hand, follows a power law distribution where you get one or two sites with a ton of traffic (like MSN or Yahoo!), and then 10 or 20 sites each with one tenth the traffic of those two, and 100 or 200 sites each with 100th of the traffic, etc. In such a curve the distribution tapers off slowly into the sunset, and is called a tail. What is most intriguing about this long tail is that if you add up all the traffic at the end of it, you get a lot of traffic

All familiar, intuitive stuff. It's entered the language, after all - we all know what the 'long tail' is. And when, for example, Ross writes about somebody who started blogging about cooking at the end of the tail and is now part of the fat head and has become a pro, we all know what the 'fat head' is, too - and we know what (and who) is and isn't part of it.

Unfortunately, the Long Tail doesn't exist.

To back up that assertion, I'm going to have to go into basic statistics - and trust me, I do mean 'basic'. In statistics there are three levels of measurement, which is to say that there are three types of variable. You can measure by dividing the field of measurement into discrete partitions, none of which is inherently ranked higher than any other. This car is blue (could have been red or green); this conference speaker is male (could have been female); this browser is running under OS X (could have been Win XP). These are nominal variables. You can code up nominals like this as numbers - 01=blue, 02=red; 1=male, 2=female - but it won't help you with the analysis. The numbers can't be used as numbers: there's no sense in which red is greater than blue, female is greater than male or OS X is - OK, bad example. Since nominals don't have numerical value, you can't calculate a mean or a median with them; the most you can derive is a mode (the most frequent value).

Then there are ordinal variables. You derive ordinal variables by dividing the field of measurement into discrete and ordered partitions: 1st, 2nd, 3rd; very probable, quite probable, not very probable, improbable; large, extra-large, XXL, SuperSize. As this last example suggests, the range covered by values of an ordinal variable doesn't have to exhaust all the possibilities; all that matters is that the different values are distinct and can be ranked in order. Numeric coding starts to come into its own with ordinals. Give 'large' (etc) codes 1, 2, 3 and 4, and a statement that (say) '50% of size observations are less than 3' actually makes sense, in a way that it wouldn't have made sense if we were talking about car colour observations. In slightly more technical language, you can calculate a mode with ordinal variables, but you can also calculate a median: the value which is at the numerical mid-point of the sample, when the entire sample is ordered low to high.

Finally, we have interval/ratio or I/R variables. You derive an I/R variable by measuring against a standard scale, with a zero point and equal units. As the name implies, an I/R variable can be an interval (ten hours, five metres) or a ratio (30 decibels, 30% probability). All that matters is that different values are arithmetically consistent: 3 units minus 2 units is the same as 5 minus 4; there's a 6:5 ratio between 6 units and 5 units. Statistics starts to take off when you introduce I/R variables. We can still calculate a mode (the most common value) and a median (the midpoint of the distribution), but now we can also calculate a mean: the arithmetic average of all values. (You could calculate a mean for ordinals or even nominals, but the resulting number wouldn't tell you anything: you can't take an average of 'first', 'second' and 'third'.)

You can visualise the difference between nominals, ordinals and I/R variables by imagining you're laying out a simple bar chart. It's very simple: you've got two columns, a long one and a short one. We'll also assume that you're doing this by hand, with two rectangular pieces of paper that you've cut out - perhaps you're designing a poster, or decorating a float for the Statistical Parade. Now: where are you going to place those two columns? If they're nominals ('red cars' vs 'blue cars'), it's entirely up to you: you can put the short one on the left or the right, you can space them out or push them together, you can do what you like. If they're ordinals ('second class degree awards' vs 'third class') you don't have such a free rein: spacing is still up to you, but you will be expected to put the 'third' column to the right of the 'second'. If they're I/R variables, finally - '180 cm', '190 cm' - you'll have no discretion at all: the 180 column needs to go at the 180 point on the X axis, and similarly for the 190.

Almost finished. Now let's talk curves. The 'normal distribution' - the 'bell curve' - is a very common distribution of I/R variables: not very many low values on the left, lots of values in the middle, not very many high values on the right. The breadth and steepness of the 'hump' varies, but all bell curves are characterised by relatively steep rising and falling curves, contrasting with the relative flatness of the two tails and the central plateau. The 'power law distribution' is a less common family of distributions, in which the number of values is inversely proportionate to the value itself or a power of the value. For example, deriving Y values from the inverse of the cube of X:

X value	Y formula	Y value
1	1000 / (1^3)	1000
2	1000 / (2^3)	125
3	1000 / (3^3)	37.037
4	1000 / (4^3)	15.625
5	1000 / (5^3)	8
6	1000 / (6^3)	4.63

As you can see, a power law curve begins high, declines steeply then 'levels out' and declines ever more shallowly (it tends towards zero without ever reaching it, in fact).

Got all that? Right. Quick question: how do you tell a normal distribution from a power-law distribution? It's simple, really. In one case both low and high values have low numbers of occurrences, while most occurrences are in the central plateau of values around the mean. In the other, the lowest values have the highest numbers of occurrences; most values have low occurrence counts, and high values have the lowest counts of all. In both cases, though, what you're looking at is the distribution of interval/ratio variables. The peaks and tails of those distribution curves can be located precisely, because they're determined by the relative counts (Y axis) of different values (X axis) - just as in the case of our imaginary bar chart.

Back to a real bar chart.

Figure #1: 433 weblogs arranged in rank order by number of inbound links.

The shape of Figure #1, several hundred blogs ranked by number of inbound links, is roughly a power law distribution.

As you can see, this actually isn't a power law distribution - roughly or otherwise. It's just a list. These aren't I/R variables; they aren't even ordinals. What we've got here is a graphical representation of a list of nominal variables (look along the X axis), ranked in descending order of occurrences. We can do a lot better than that - but it will mean forgetting all about the idea that low-link-count sites are in a 'long tail', while the sites with heavy traffic are in the 'head'.

[Next post: how we could save the Long Tail, and why we shouldn't try.]

Good neighbors

2005-10-19T16:59:00.000+01:00

[Updated 20/10 - tidying-up, response to Adam, Malik quote ect ect]

Shelley:

Through the various link services, last week I found that my RSS entries were being published to a GreatestJournal site. I’d never heard of GreatestJournal, and when I went to contact the site to ask them to remove the feed, there is no contact information. I did find, though, a trouble ticket area and submitted a ticket asking the site to remove the account.

In reply, "GreatestJournal" (whoever they are) told Shelley that her RSS feed was in the public domain, so they could do whatever they liked with it. ("You might wish to take your feed down if you don’t want people to use it." That's helpful.)

One other thing: the email in which they conveyed this information had a copyright notice at the bottom. (Shelley reprinted it anyway.)

Coincidentally, I'd recently been reading this post on EconoMeta, in which Adam talks about our changing relationship with our personal data:

one important part of Web 2.0 is the separation of user data from the applications that use it, and the idea that users should own and control this data.
...
the switching costs imposed by Web 1.0 companies to get a competitive advantage are being replaced by different switching costs created by the *users* of Web 2.0 companies ... [e.g.] the switching costs created by the value of a social network at MySpace or a reputation on eBay, as opposed to the switching cost created by the email address and “walled garden” at AOL.

Separation of user data from applications? Check. User ownership and control? Um, not so much.

It seems to me that this is (depending on how charitable you're feeling) a naive oversight, a lurking contradiction or a dirty little secret at the heart of the "Web 2.0" vision: it's not about the users. Here's Tim O'Reilly, no less:

Let's close, therefore, by summarizing what we believe to be the core competencies of Web 2.0 companies:
Services, not packaged software, with cost-effective scalability
Control over unique, hard-to-recreate data sources that get richer as more people use them
Trusting users as co-developers
Harnessing collective intelligence
Leveraging the long tail through customer self-service
Software above the level of a single device
Lightweight user interfaces, development models, AND business models

So we've got software companies harnessing collective intelligence, leveraging the Snaggly Fence* - and, of course, exercising control over unique data. Unique and hard-to-recreate data. Unique data that's continually enriched by its users. We're talking social software, aren't we?

It seems increasingly clear that there are two sides to Web 2.0. The sunny side - the 'social software' side - is where we ask questions like:

Q: How will the data sources become unique and impossible to recreate?
A: By being enriched!
Q: How will the data be enriched?
A: Through being used by people!
Q: How will people use the data?
A: Quickly, easily, intuitively and in their thousands!

That's also the easy side of Web 2.0 - there aren't too many posers there, as you can see.

But there's another side, where we ask questions like "Who will own those data sources?" - and, increasingly, "How will they get hold of them to begin with?" Which, I think, is where GreatestJournal comes in. In comments at Shelley's post, Roger Benningfield made the Web 2.0 connection:

I came across a whole swarm of Web 2.0 stuff in my aggregator. “Microformats, XHTML, death to walled gardens!” they cried.

And I thought, “Oh, you guys are *fucked*.” Because ultimately, the business models they’re envisioning are going to make GreatestJournal’s response look friendly in comparison. If they ever manage to build any momentum (questionable), they’re going to hit a brick wall of posts like this one… a *big* wall.

Case in point: a thoroughly odd development called Sxore. Adina: "The idea is that if a user signs up to comment on one blog, they'll be able to comment on other blogs. ... Sxore creates an RSS feed for each user. Presumably you can follow comments made by that user across different blogs. So, if you think someone has good ideas about blog visualizations, you get to read what they also think about President Bush." Hmmm. What was that about users owning and controlling their data again?

Om Malik has been having similar thoughts:

if we tag, bookmark or share, and help del.icio.us or Technorati or Yahoo become better commercial entities, aren’t we seemingly commoditizing our most valuable asset - time. We become the outsourced workforce, the collective, though it is still unclear what is the pay-off. While we may (or may not) gain something from the collective efforts, the odds are whatever “the collective efforts” are, they are going to boost the economic value of those entities. Will they share in their upside? Not likely!

Take Skype as an example - it rides on our broadband pipes, for which we a hefty monthly charge. It uses our computers and pipes to replace a network that cost phone companies billions to build. In exchange we can make free phone calls to other Skype users. I have no problems with that. I had no problems with Skype charging me for SkypeIN and SkypeOUT calls as well, for this was only a premium service only to be used if and when needed.

However, now that it is part of eBay, I do cringe a little.

It seems to me that the Web 2.0 hype is about social software, but only in the sense that it's about monetising social software: in Marxist terms it's a form of primitive accumulation. In non-Marxist terms, it's enclosure: appropriating something that exists outside the circuit of trading and ownership and managing the supply so that it can only be obtained within that circuit. Or: stealing it and selling it back. I don't know what the GreatestJournal business model is, or how Sxore are planning on making their money; probably something perfectly obvious and straightforward. But it seems to involve turning our work into their assets. I'm not too keen.

In response to Adam (in comments), my concern isn't that it's impossible to draw a line where the benefits of social software can coexist with monetisation (I myself use and endorse the fine products of Blogger.com, after all). What worries me, firstly, is that the drive for monetisation is producing pressures for closure (and enclosure). Secondly, that half the people who advocate Web 2.0 seem to share the company perspective to the point of positively welcoming these developments (see the O'Reilly sermon linked above) - while a lot of the rest are so committed to the vision as to be spectacularly ill-prepared to put up any resistance.

My immediate reaction to Shelley's GreatestJournal post was to leap to the defence of walled gardens - "Walled gardens are full of people!". It's a nice line, but on reflection I don't think it's quite right. What we're hearing is a sublime (although far from unprecedented) example of chutzpah - a critique of barriers by advocates of enclosure. The blogosphere isn't a walled garden, it's a wide-open common where nobody has ownership rights. An enclave which can't be strip-mined isn't walled in; all that's happened is that the predators - who would put their own fences around it if they could - have been walled out. Long may they remain so.

(The Americanism in the title is deliberate, incidentally.)

*There Is No Long Tail

Everything playing at once

2005-10-05T15:11:00.000+01:00

Dave:

I no longer look at the front page of the NY Times to tell me what's important. I look at it to see what people like the editors of the NY Times think is important. I'm finding the news that matters through the Internet recommendation engine: Blogs, emails, mailing lists, my aggregator, websites that aggregate and comment on news, etc.

Brief thoughts (also appearing in comments at Dave's): we're back with finding out what people say about stuff. Which is, ultimately, all there is to find out. Knowledge - and, for that matter, news - has always been produced in cloud form, as an emergent property of conversations. When we counterpose knowledge to conversation, we're really saying that certain conversations have ended - or been brought to an end - and left unchallenged conclusions behind them. What's changed is that, until recently, the conversations which produce knowledge (and news) have taken place within small and closed groups, so that most of us have only seen the crystallised end-product of the conversation. What Wikipedia, blogging, RSS and del.icio.us give us is the rudiments of a distributed conversation platform, enabled by pervasive broadband. (Which is why the ownership of the authority to stop the conversation - and crystallise the cloud - is such a big issue.)

Know what I mean

2005-09-29T11:05:00.000+01:00

Back here, I wrote:

Tagging, I'm suggesting, isn't there to tell us about stuff: it's there to tell us about what people say about stuff. As such, it performs rather poorly when you're asking "where is X?" or "what is X?", and it comes into its own when you're asking "what are people saying about X?"

This relates back to my earlier argument that all knowledge is cloud-shaped, and that tagging is simply giving us a live demonstration of how the social mind works. In other words, all there is is "what people are saying about X" - but some conversations have been going on longer than others. Some conversations, in fact, have developed assumptions, artefacts, structures and systems within and around which the conversation has to take place. The conversation carried on in the medium of tagging isn't at that stage yet, perhaps, but it will be - the interesting question is about the nature of those artefacts and structures.

Now (with thanks to Anne Galloway) over to Dan Sperber.

When say, vervet monkeys communicate among themselves, one vervet monkey might spot a leopard and emit an alarm cry that indicates to the other monkeys in his group that there's a leopard around. The other vervet monkeys are informed by this alarm cry of the presence of a leopard, but they're not particularly informed of the mental state of the communicator, and they don't give a damn about it. The signal puts them in a cognitive state of knowledge about the presence of a leopard, similar to that of the communicating monkey — here you really have a smooth coding-decoding system.

In the case of humans, when we speak we're not interested per se in the meaning of the words, we register what the word means as a way to find out what the speaker means. Speaker’s meaning is what's involved. Speaker’s meaning is a mental state of the speaker, an intention he or she has to share with us some content. Human communication is based on the ability we have to attribute mental state to others, to want to change the mental states of others, and to accept that others change ours.

When I communicate with you I am trying to change your mind. I am trying to act on your mental state. I'm not just putting out a kind of signal for you to decode. And I do that by providing you with evidence of a mental state in which I want to put you in and evidence of my intention to do so. The role of what is often known in cognitive science as "theory of mind," that is the uniquely human ability to attribute complex mental states to others, is as much a basis of human communication as is language itself.

I am full of admiration for the mathematical theory of information and communication, the work of Shannon, Weaver, and others, and it does give a kind of very general conceptual framework which we might take advantage of. But if you apply it directly to human communication, what you get is a mistaken picture, because the general model of communication you find is a coding-decoding model of communication, as opposed to this more constructive and inferential form of communication which involves inferring the mental state of others, and that's really characteristic of humans.
[...]
For Dawkins, you can take the Darwinian model of selection and apply it almost as is to culture. Why? Because the basic idea is that, just as genes are replicators, bits of culture that Dawkins called “memes” are replicators too. If you take the case of population genetics, the causal mechanisms involved split into two subsets. You have the genes, which are extremely reliable mechanisms of replication. On the other hand, you have a great variety of environmental factors — including organisms which are both expression of genes and part of their environment — environmental factors that affect the relative reproductive success of the genes. You have then on one side this extremely robust replication mechanism, and on the other side a huge variety of other factors that make these competing replication devices more or less successful. Translate this into the cultural domain, and you'll view memes, bits of culture, as again very strong replication devices, and all the other factors, historical, ecological, and so on, as contributing to the relative success of the memes.

What I'm denying, and I've mentioned this before, is that there is a basis for a strong replication mechanism either in cognition or in communication. It's much weaker than that. As I said, preservative processes are always partly constructive processes. When they don’t replicate, this does not mean that they make an error of copying. Their goal is not to copy. There are transformation in the process of transmission all the time, and also in the process of remembering and retrieving past, stored information, and these transformations are part of the efficient working of these mechanisms. In the case of cultural evolution, this yields a kind of paradox. On the one hand, of course, we have macro cultural stability — we do see the same dish being cooked, the same ideologies being adopted, the same words being used, the same song being sung. Without some relatively high degree of cultural stability — which was even exaggerated in classical anthropology — the very notion of culture wouldn't make sense.

How then do we reconcile this relative macro stability at the cultural level, with a lack of fidelity at the micro level? ... The answer, I believe, is linked precisely to the fact that in human, transmission is achieved not just by replication, but also by construction. ... Although indeed when things get transmitted they tend to vary with each episode of transmission, these variations tend to gravitate around what I call "cultural attractors", which are, if you look at the dynamics of cultural transmission, points or regions in the space of possibilities, towards which transformations tend to go. The stability of cultural phenomena is not provided by a robust mechanism of replication. It's given in part, yes, by a mechanism of preservation which is not very robust, not very faithful (and it's not its goal to be so). And it’s given in part by a strong tendency for the construction — in every mind at every moment — of new ideas, new uses of words, new artifacts, new behaviors, to go not in a random direction, but towards attractors. And, by the way, these cultural attractors themselves have a history.

There's more - much more - but what I've quoted brings out two key points. Firstly, communication is not replication: in conversation, there is no smooth transmission of information from speaker to listener, but a continuing collaborative effort to present, construct, re-present and reconstruct shared mental models. The overlap between this and the 'knowledge cloud' model is evident. Secondly, construction has a context: the process of model-building (or 'thinking' as we scientists sometimes call it) is always creative, always innovative, and always framed by pre-existing cultural 'attractors'. And these cultural attractors themselves have a history - you could say that people make their own mental history, but they do not do so in circumstances of their own choosing...

This is tremendously powerful stuff - from my (admittedly idiosyncratic) philosophical standpoint it suggests a bridge between Schutz, Merleau-Ponty and Bourdieu (and I've been looking for one of those for ages). My only reservation relates to Sperber's stress on speaker's meaning ... a mental state of the speaker. I think it would enhance Sperber's model, rather than marring it, to focus on mental models as they are constructed within communication rather than as they exist within the speaker's skull - in other words, to bracket the existence of mental states external to communicative social experience. On this point Schutz converges, oddly, with Wittgenstein.

Sperber's argument tends to underpin my intuition on tagging and knowledge clouds: if all communication is constructive - if there is no simple transmission or replication of information - then conversation really is where knowledge develops, or more precisely where knowledge resides. Sperber also helps explain the process by which some conversations become better-established than others; we can see this as a feedback process, involving the development of a domain-specific set of 'attractors'. These would perhaps serve as a version of Rorty's 'final vocabulary': a shared and unquestionable set of assumptions, a domain-specific backdrop without which the conversation would make no sense.

One final thought from Sperber:

The idea of God isn't a supernatural idea. If the idea of God were supernatural, then religion would be true.

Well, I liked it.

If I drew a detailed map

2005-09-26T13:03:00.000+01:00

Several months ago, I wrote (regarding the Wikipedia page on 'anomie'):

For what I'd want to know about a concept like that, that page is pretty dreadful. It veers wildly between essentialism (there is a thing called 'anomie' and we know what it is, across time and space) and nominalism (different people have used this combination of letters to mean different things, who knew?). What's not there is any sense of the history of the concept

I was reminded of this argument by Tom's recent comments on the 'penis envy' page ("I know this article on penis envy is bullshit, and it's been on my 'to do' list of things to fix for weeks, and I've got nowhere"). The problem here is that making things more complicated is a lot harder than keeping them simple. What's worse, the kind of people who are critical of other people's simplifications tend also to be critical of their own work, which means that getting the complicated version written and getting it right is a long and painstaking job. Which, in turn, means that in the absence of serious incentives it's quite likely not to get done. Wikipedia's native system of informal incentives breaks down, in other words, where the workload gets too large - and, when it comes to making things more complicated (and getting it right), the workload starts at 'large' and goes up.

I was talking about this stuff with a friend the other day (hi Chris!) when he came up with a proposal for filling the incentive gap. The idea is to mobilise peer pressure among the population of disgruntled complexifiers. What we want isn't so much an army of subject experts as a group of people who mistrust simple explanations and are good at digging out and writing down the underlying complications, in any of a number of fields. Hacks rather than professors, essentially - but good hacks. A list of apparently oversimplified Wikipedia articles could then be drawn up, and each one could be offered to names picked from the pool. I'll just reiterate that I'm not talking about people with expert knowledge, so much as perfectionists with inquiring minds. The Wikipedia articles I've mentioned left me with a stack of unanswered questions, which I'd happily devote a few evenings to answering if I was being paid to do so - or if I had any incentive to do so. A virtual tap on the shoulder from an online group of pedantic curmudgeons might just do the job.

That just leaves the task of assembling the group. Here, Chris made the brilliant suggestion of using PledgeBank. Something like this:

I will take part in a group of volunteers who will improve Wikipedia by correcting and extending inaccurate and simplistic entries on social science concepts, but only if another 99 people do so too.

I think it could work. What do you think?

A place for everything

2005-09-22T13:42:00.000+01:00

Or: what ethnoclassification is, and what folksonomy isn't.

When it comes to tagging, I'm facing both ways. I think it's fascinating and powerful and new - qualitatively new, that is: it's worth writing about not just because it's shiny, but because there's still work to be done on understanding it. At the same time, I think it's been massively oversold, often on the back of rhetorical framings which only have a glancing relationship with evidence or logic. Tagging is fascinating and powerful and new, but a lot of the talk about tagging has me tearing my hair.

I'll pick on a recent post by Dave Weinberger. (Personal to DW: sorry, Dave. I'm emphatically not (is that emphatic enough?) suggesting that you're the worst offender in this area.)

Let's say you type in "africa," "agriculture" and "grains" because that's what you're researching. You'll get lots of results, but you may miss pages about "couscous" because Google is searching for the word "grain" and doesn't know that that's what couscous is made of. Google knows the words on the pages, but doesn't know what the pages are about. That's much harder for computers because what something is about really depends on what you're looking for. That same page on couscous that to you is about economics could be about healthy eating to me or about words that repeat syllables to someone else. And that's the problem with all attempts by experts and authorities to come up with neat organizations of knowledge: What something is about depends on who's looking.
...
Let's say you come across the Moroccan couscous web page and you want to remember it. So you upload its Web address to your free page at del.icio.us that lists all the pages you've saved. Then del.icio.us asks you to enter a word or two as tags so you can find the Moroccan page later. You might tag it with Morocco, recipe, couscous, and main course, and then later you can see all the pages you've tagged with any of those words.

That's a handy way to organize a large list of pages, but tagging at del.icio.us really took off because it's a social activity: Everyone can see all the pages anyone has tagged with say, Morocco or main course or agriculture. This is a great research tool because just by checking the tag "agriculture" now and then, you'll see every page everyone else at delicious has tagged that way. Some of those pages will be irrelevant to you, of course, but many won't be. It's like having the world of people who care about a topic tell you everything they've found of interest. And unlike at Google, you'll find the pages that other humans have decided are ABOUT your topic.

What strikes me about this passage is that Dave changes scenarios in mid-stream: Let's say you come across the Moroccan couscous web page... How? Google couldn't find it. Let's compare like with like, and say that you're still looking for your couscous page: what do you do then, if not go to del.icio.us and type in "africa," "agriculture" and "grains"? Once again, assuming that whole-site searches aren't timing out, you'll get lots of results (particularly since del.icio.us doesn't seem to allow ANDing of search terms) but you may miss pages about "couscous" - and checking the tag "agriculture" now and then won't necessarily help. Google will miss the page if the term 'couscous' doesn't appear in the source (which doesn't necessarily mean 'appear on screen', of course); del.icio.us will miss it if the term hasn't been used to tag it (even if it is in the source).

Google vs del.icio.us is an odd comparison, in other words, and it's not at all clear to me that the comparison favours del.icio.us. It's great to get classificatory(?) input from the users of a document, of course - as I said above, tagging is fascinating and powerful and new - but in terms of information retrieval it can only score over a full-text search if

1. the page has been purposefully tagged by a user
2. the page has been tagged with a term which doesn't appear in the page source
3. a second user is searching for information which is contained in the page, using the term with which the first user tagged it

I don't think tagging advocates think enough about what those conditions imply. For example, at present I'm the only del.icio.us user to have tagged Mr Chichimichi's Tags are not a panacea; I tagged it with 'tagging', 'search' and 'ethnoclassification'. Until I did so, anyone looking for it would have been out of luck. Even Google wouldn't be much help - the word 'ethnoclassification' doesn't appear anywhere in the text. No, until a couple of days ago your only way of stumbling on that post would have been to run a clumsy, counter-intuitive Google search on terms like 'tagging', 'tags', 'folksonomies' and 'social software'. (Google even knows that 'folksonomies' is the plural of 'folksonomy', so searching on the singular form would work just as well. That's just not fair.)

Dave also contrasts the world of collective knowledge through distributed tagging with attempts by experts and authorities to come up with neat organizations of knowledge. Further along in the same piece, he writes:

This takes classification and about-ness out of the hands of authors and experts. Now it's up to us readers to decide what something is about.

Not only does this let us organize stuff in ways that make more sense to us, but we no longer have to act as if there's only one right way of understanding everything, or that authors and other authorities are the best judges of what things are about.

One question: who ever said that there was only one right way of understanding everything? OK, too easy. I'll rephrase that: before tagging came along, who was saying there was one right way, etc? Who are the tagging advocates actually arguing against? (It certainly isn't librarians (context here).)

There's a difference between classifications which have a single pre-determined set of definitions and classifications which are user-defined and user-extensible. But that's not the same as the difference between having an underlying ontology and not having one, or the difference between hierarchical and flat organisations of knowledge, or the difference between single and multiple sets of classifications. A closed, expert-defined, locked-down controlled vocabulary may contain multiple sets of overlapping terms; it may be a flat list of categories rather than a 'tree'; it may even be innocent of ontology. (Thanks to Jay for pointing this out, in comments here.) If tagging is better than top-down classification, it's better because it's user-defined and user-extensible - not because it's free of the vices of ontology, hierarchy and uniformity. The idea that tagging - and only tagging - stands in opposition to a classifying universe built on hierarchical uniformity is a straw man. (But the librarians get it both ways - if a top-down classifying system is shown to be flat and plural, this can be put forward as a sign of the weakness of top-down systems; the fact that bottom-up systems are more, not less, vulnerable to Chinese Encyclopedia Syndrome is passed over.)

So, tagging systems make lousy search engines, and they don't mark a qualitative leap in the organisation of human knowledge. What they're really good for - and what makes them fascinating and powerful - is conversation. Tagging, I'm suggesting, isn't there to tell us about stuff: it's there to tell us about what people say about stuff. As such, it performs rather poorly when you're asking "where is X?" or "what is X?", and it comes into its own when you're asking "what are people saying about X?" (Of course, much tag-advocacy is driven by the tacit belief that there's no fundamental difference between what people say about X and expert knowledge of X - and that an aggregate of what people say would be equivalent, if not superior, to expert knowledge. But that's an argument for another post.)

Tagging is good for telling us what people say about stuff, anyway - and when it's good, it's very good. To see what I'm talking about, have a look at Reader2 (via Thomas). It's a book recommendation site, implemented on the basis of a del.icio.us-like user/tag system. It's powerful stuff already, and it's still being developed. Does it tell me what books are really like? No - but it tells me what people are saying about them, which is precisely what I want to know. And it couldn't do this nearly as well, it seems to me, without tags - and tag clouds in particular. This, for me, is what tagging's all about. Ethnoclassification: classification as a open-ended collective activity, as one element of the continual construction of social reality.

Who took the money?

2005-09-20T14:05:00.000+01:00

This is a fascinating post (in Italian) by Pietro Speroni on the relationship between authority, communities and markets. This is an interesting and controversial area; the fact that Pietro also invokes the Long Tail (which, as you'll recall, is not what it seems) makes it all the more compelling (to me at least).

I'll translate as I go along; hopefully Pietro will correct me if I go wrong.

I don't believe that the ruling class has vanished. I believe that it has simply been transformed - just as the world itself is being continually transformed from day to day. Decades ago, our world was simpler - more homogeneous, less diverse. If you followed a martial art, it would be judo or karate. A game? Chess. A religion? Christian, Jewish, perhaps Muslim at the outside.
...
on the Net, via Google (and wikipedia), you can find the specific branch of the specific religious tradition which best meets your needs. ... And this is not true only of religions, but of everything: interests, political groups, passions, games, ways of life.
...
Now, every one of these groups has its own implicit hierarchy. ... And everyone is a member of more than one group. And in every group you listen to some people, and what you say influences other people.
...
[In every area of my life] I have leaders: people I trust; people who I admire and learn from. But they're not the same people as your leaders. Not only that, but there are other people who come to me to learn (worse luck for them!), in some fields more than in others. The process of diversification tends towards having as many groups as people - and every one of us, of necessity, becomes the small-scale leader of a small-scale group, scattered around the world.
...
This whole process mirrors what's happening in the economy, where a market consisting of niches is growing explosively ... The key phrase is Long Tail.
...
So I don't believe that the ruling class is vanishing, but that we're seeing a gradual diversification of interests, which leads to the diversification of the ruling class - accompanied by the redefinition and contraction [ridimensionamento] of the role of traditional leaders.

There's a lot that I like about this - I think Pietro's right to say that there's a new kind of process of diversification under way, and to trace it back to the Internet's basic sociality, its nature as a medium for conversation.

But... a transformation of the ruling class? Non tanto. Pietro's larger argument is undermined by a couple of strange elisions. Firstly, it's true that we all have multiple 'authorities' - the topics of folk music, statistics, Belgian beer and operaismo are all important to me, for instance, and in each case I could name an authority I'd willingly defer to. But those people aren't the people who enforce the laws I obey, or set the level of tax I pay, or price the goods I buy, or write the newspapers I read, or appear on the news programmes I watch. The ruling class, it seems to me, is still very much in place, and whether I'm a tequila-crazed Quaker or a tea-drinking Tantric Buddhist is a matter of sublime indifference to it. Roy Bhaskar has written that historical materialists, by virtue of starting from the material facts of social existence, cannot propose absolute freedom, "a realm free of determination"; what we can envisage is moving "from unneeded, unwanted and oppressive to needed, wanted and empowering sources of determination". The world Pietro describes is a world which is governed only by those needed, wanted and empowering sources of determination. It sounds good, but I don't think we're there yet.

Secondly, on the matter of niche marketing. Pietro assumes that a proliferation of niche markets will lead to a proliferation of niche suppliers, and hence the dilution of the authority of the big suppliers. I don't see any reason to believe that this is the case. Indeed, one of Chris Anderson's own preferred examples is based on Amazon sales rank - and there's nothing very diffuse about Amazon, or the authority wielded by Amazon. Much of the buzz around the 'Long Tail' seems to derive, ultimately, from this confusion of the two meanings of 'niche'. Clearly, mining niche markets can be profitable, if you're a monopolistic behemoth like Amazon; but, equally clearly, it doesn't follow that niche suppliers can make a living in the same way. Indeed, making niches visible to companies like Amazon actually threatens existing niche suppliers. (Ask your local bookshop, if you've still got one.)

Of course, Long Tail proponents tell a different story. Back in July, Scott Kirsner quoted George Gilder thus:

His central thesis is that Internet-connected screens in the home – whether it’s the PC in your den or the plasma screen on your living room wall – are going to change the way we consume video by offering us infinite choice.
...
“The film business will increasingly resemble the book business,” he says, with a few best-sellers that achieve widespread popularity, and lots of publishers making a profit selling titles that no one’s ever heard of.

Lots of who doing what? Run that past us again, could you? While you're at it, send the good news to the novelist A.L. Kennedy, whose wonderful FAQ includes this:

SO, WHAT’S HAPPENING WITH THE WONDERFUL WORLD OF PUBLISHING?

Fewer publishing houses concentrated in conglomerate hands, trying to produce more books of less quality. No full time readers, no full time copy editors and therefore missed newcomers and pisspoor final presentation of texts on the shelves, silly covers, greedy and simple-minded bookshop chains, lunatic bidding wars designed to crush the spirit of unknown newcomers, celebrity “tighten your buns and nurture your inner pot plant” hard backs and much related insanity.

Mass markets are where the units get shifted; niche markets - like literary fiction - are where survivors linger on (until they're bought out) and upstart competitors emerge (and hang on until they're bought out). It's the logic of the monopoly, which is to say that it's the logic of the market. Some years ago a McDonald's spokesman, asked if the fast food market had reached saturation point, responded that, as far as his company was concerned, the market would only be saturated if there were no cooked food outlets anywhere on the planet apart from McDonald's. I don't think Amazon, or the publishing conglomerates, or the media companies who would source Gilder's 'infinite choice', think any differently.

But Pietro's half right: there is something interesting going on, even if it doesn't mirror what's going on in the economy; there is a process of diffusion and diversification, even if it doesn't affect the main sources of authority over our lives. In fact, what's significant about the Net is that it can host conversations which escape the marketplace and evade pre-existing ('unneeded and unwanted') forms of authority. That said, it can also reproduce the marketplace and reinvent old forms of authority - just like other conversational media.

In short, what's good about the Web is - or can be - very good; what's bad about is - or should be - very familiar.

So say I

2005-08-05T12:00:00.000+01:00

Why I use 'ethnoclassification' rather than 'folksonomy'.

'Ethnoclassification' recalls 'ethnomethodology', Harold Garfinkel's coinage for the study of the collective construction of everyday life. Garfinkel took a great deal from Alfred Schutz; I think some of his work develops Schutz's social phenomenology in the wrong direction, but to have Schutz's work developed at all is a good thing. In this context, the term 'ethnoclassification' suggests a process that's continual, provisional and embedded in practical activity: the place where it happens (to borrow a phrase from Russell Hoban) is Everywhere All The Time. I think this is a good emphasis.
'Folksonomy', by contrast, suggests both a process and the end result (a viable folk-taxonomy); as such it's confusing and promotes fuzzy argument.
It's also a term with a strong positive value: forward the taxonomy of the folk! a bas les bibliothecaires! It's a marketing term as well as a term of analysis, and lends itself to slippage between description and advocacy.
(Last and least) It's etymologically ghastly and obtrusively American (I don't say 'candy', I don't say 'diaper' and I don't say 'folks').

Henceforth - starting in the previous post, to be more precise - I'll be using 'ethnoclassification' to refer to the (real, universal, continuing) process and 'folksonomy' to refer to the (hyped, unrealised, arguably unrealisable) end result.

Not available before

2005-08-05T11:56:00.000+01:00

Thanks to a couple of links posted by Thomas, I've just read Bryan Boyer's Correspondance Romano (Corriere Romano, surely? never mind) closely followed by this post from February by Tom Evslin. Tom:

People don’t think hierarchically – at least most people don’t. We think in terms of associations. Our dreams give this away as they hyperlink through experiences of the day and memories of the distant past. A conversation meanders horizontally from one topic to the next.
...
Hierarchies like Lotus Notes or the Dewey Decimal System were necessary when computing power was non-existent or very expensive. As computing power has become relentlessly cheaper thanks to Moore’s law, hierarchies of information have become unnecessary. ... So long as Google or its competitors can index almost everything I might ever want to find, why should any arbitrary order be imposed on information?
...
Once we didn’t need hierarchies to organize our approach to information, they became an impediment. It is very hard for one person to figure out which node in which folder tree another person would have put a particular piece of information. A document may be relevant to one researcher for entirely different reasons than it is relevant to another researcher.
...
The relationship between documents is actually dynamic depending on the needs of the reader. Not incidentally, open tagging and hyperlinking are both ways to impose particular relationships on documents to meet the need of some subset of readers.

In passing, this suggests that the contribution of tagging to the grunt work of actually finding stuff may not be all that significant. After all, "a document may be relevant to one researcher for entirely different reasons than it is relevant to another researcher": in this respect the same strictures apply to tags as to folders, with the proviso that tagging does at least give you multiple chances to get it right. I've found useful and interesting stuff by browsing del.icio.us, but I've also found useful and interesting stuff by browsing library catalogues, running partial name searches on booksellers' sites, googling common phrases and going to the eighth page of results, and so forth. But then, I'm a catalogue-hound and I like being surprised. If you're looking for something specific, Tom's argument (inadvertently?) suggests, you're probably better off with Google.

Bryan's post doesn't discuss taxonomies, ontologies or search engines, largely because it's a series of emails from 2002. But it does contain this beautiful piece of ethnoclassification:

Italy is about all of these things: cured meats, standing up to drink your coffee, stiffling heat, mid-day naps, skulls in churches, hot men in suits on scooters, Ananas, and cheap groceries.

This is very much the kind of freewheeling associational approach to knowledge that Tom describes - and very much the kind of ground-up, non-exclusive, plural, open-ended classifying process which has become known as 'folksonomy'.

But what happens if we take that sentence and map it onto the current 'folksonomic' toolset? Is there an 'Italy' resource somewhere - a really really authoritative Web page, say - that we can tag with 'curedmeat', 'coffeestandingup', 'stifflingheat' and so on? (Never mind the problem of cross-matching with the tags 'meat.cured', 'coffee.standing' and 'heat.stiffling' - let alone 'heat.stifling'.) Or are we going to use an 'italy' tag and apply it to single identifiable resources on 'cured meat', 'hot men in suits on scooters', etc? If so, did all those resources exist before we tried to tag them - and if not, are we going to have to create them?

The kind of association described by Tom - and exemplified by Bryan's old mails - is actually a very bad fit for the Technorati/del.icio.us style of document tagging, for two reasons. One is that it's two-way: if 'Italy' is associated with 'skulls in churches' then 'skulls in churches' is necessarily associated with 'Italy'. (In the case of document-based tagging, the relationship is asymmetrical and the inverse relationship is weaker: Document 1 'is about' T1, T2, T3; Topic 1 'has some relevant information in' D1, D2, D3.) The other is that it's descriptive rather than annotative: we're not tagging stuff-about-stuff, we're tagging... well, stuff, and tagging it with other stuff. These bi-directional relationships between concepts can be approximated by the associations between tags which emerge out of the cumulative process of document tagging, but this seems like going a very long way round. "We think in terms of associations": should we have to say

this has been applied to resources which have also been classified as that

when what we want to say is

this is like that ?

There's one glaring exception to this argument: Flickr. It's easy to imagine an 'italy' photoset including images which were also tagged with 'curedmeat', 'churchskull' and so forth. Descriptive tagging, bi-directional associations, it's all there - job done. This is deceptive, however. Flickr runs on discrete objects - individual images - and the relationships between Flickr tags really describe the images themselves, or at most the universe of Flickr images. If we didn't have any images of stifling heat in Italy, that association wouldn't exist; if we had three salami pictures and only one of a skull in a church, the 'curedmeat'/'italy' association would automatically be three times as strong as 'churchskull'/italy'. Once again, we'd have to go to considerable lengths in order to represent the associations which Bryan effortlessly set out in 32 hastily-composed words.

Ethnoclassification: do we have the technology?

Tag tag tag

2005-07-21T11:08:00.000+01:00

Tom Coates' interesting post Two cultures of fauxonomies collide has been getting a lot of attention lately, mainly thanks to Dave. There's a particularly interesting discussion running at Many-to-Many. The discussion has progressed quite rapidly, with several bright and articulate people pitching in to illustrate how Tom's original insight can be developed. My problem is that I'm not sure what the discussion's based on. For example, Emil Sotirov writes:

Seemingly, given the freedom of folksonomy, people tend to move from hierarchical "folder" modes of tag interpretation (one-to-many) towards more open "keyword" modes (many-to-many).

Keywords are flat, many-to-many, open; folders are hierarchical, one-to-many, closed. (In short, folders are bad, m'kay?) But what does this really mean? If I think that tags are 'like' keywords or that tags are 'like' folders, what difference does it actually make?

From Tom's original piece:
Matt's concept was quite close to the way tagging is used in del.icio.us - with an individual the only person who could tag their stuff and with an understanding that the act of tagging was kind of an act of filing. My understanding was heavily influenced by Flickr's approach - which I think is radically different - you can tag other people's photos for a start, and you're clearly challenged to tag up a photo with any words that make sense to you. It's less of a filing model than an annotative one.

Incidentally, "an individual the only person who could tag their stuff"? That's Technorati rather than del.icio.us, surely?

But anyway - the main question is, what are you actually doing differently if you use a tag as an 'annotative' keyword rather than a 'classifying' folder? In either case, it seems to me, you're pulling out a couple of characteristics of an object and using them to lay a trail back to it. The only real difference I can see is that you'd expect to have more 'keywords' than objects and fewer 'folders' than objects, but I can't see how this changes the way you actually interact with the tags or the tag-holder services - or the objects, for that matter.

Perhaps I'm just not getting something - all enlightenment is welcome. But I suspect that, in practice, Flickr and del.icio.us and... er, all those other social tagging services... are converging on a model somewhere between 'keyword' and 'folder'. The tag cloud is crucial here. Flickr may start by enabling you to "tag up a photo with any words that make sense to you", but the tag cloud display "conceals the less popular [tags] and lets recurrence form emergent patterns" (as Tom notes here); it also prompts users to select from previously-used tags if possible. Conversely, the (more rudimentary) tag-cloud display in del.icio.us gives less-used tags more prominence than they had when they were left to scroll off the screen, prompting users to select more widely from previously-used tags. In effect, the tag cloud draws del.icio.us users away from big-tree-of-folders thinking, while also drawing flickr users away from the keyword-pebbledash approach.

[No, that wasn't my promised post about the Long Tail. (It doesn't exist, you know.) Yes, I will get round to it, some time.]

Cloud A-Z

2005-07-06T10:57:00.000+01:00

David Gratton (found via Thomas) argues that all communities are communities of interest. He argues - I think correctly - that what appear to be, for example, professional, demographic or geographic 'communities' are created and maintained through shared interests and shared activity around those interests. Where those interests and that activity are lacking, what's left isn't a community but a statistical abstraction masquerading as reality. What's particularly interesting about this is that it destroys the notion of a 'virtual community' as something new and interesting; insofar as it's a community, a 'virtual community' is a community of interest, like all the others. Technology may facilitate the creation of communities which wouldn't have been created before, but the community itself is nothing new.

Here are a few further thoughts (initially written as a comment on David's site).

You could take it a bit further by saying that a community (of whatever flavour) is a process rather than an object or an achieved state - community is something that people produce and reproduce by doing stuff together (including talking). Once you've said that community only exists as the continuing aggregate of its members' interactions, you can start asking questions about those interactions - how frequent are they? how are they structured: does everyone talk to a single central 'hub', does everyone talk to everyone, are there 'daisy-chains'? do they produce or redistribute anything identifiable - the physical necessities of life, or money, or information, or social status - or is it all about sociality and shooting the breeze? The answers to questions like those would say a lot about the shape of the community, which in turn would enable us to ask some interesting questions about the impact of 'virtuality', and the conditions under which 'virtual communities' are more or less viable than their face-to-face counterparts.

An interesting aspect of this model of community is the significance of talk: conversation is the bedrock, the basis on which everything else happens. (I'm reluctant to call it a medium: partly because of the concerns Dave highlighted back here; partly, relatedly, because that suggests that conversation is always a carrier for a signal, that there is always something else going on. This, I think, is profoundly misleading. We're social beings: a large part of what we do, how we live, is social interaction, more or less as an end in itself.) What the technologies we associate with 'virtual community' do is, essentially, to make it easier to spend more of the time talking to more people, albeit in some oddly formalised ways. Some of the interesting questions about 'virtuality' are questions about this strange pairing of talk overload and talk formalisation.

What does this have to do with clouds? One idea I've been playing with is that the natural state of knowledge is to be 'cloudy', because it's produced within continuing interactions within groups: knowledge is an emergent property of conversation, you could say. What the argument about 'community' suggests is that every community has its own knowledge-cloud - that the production and maintenance of a knowledge-cloud is one way that a community defines itself. The question then is whether existing technologies enable communities to do anything useful with their 'clouds', or if the services are still too attenuated - or too overloaded.

(I'll get back to the Long Tail soon, hopefully. It doesn't exist, you know.)

A trick of the eye

2005-06-24T17:20:00.000+01:00

A long time ago on a Web site far, far away, Clay Shirky wrote:

"We are all so used to bell curve distributions that power law distributions can seem odd."

He then traced Pareto-like 'power law' curves operating in a number of domains where large numbers of people make unconstrained choices - most memorably, inbound link counts for blogs. The inverse 'power law' curve dives steeply, then levels out, glides downwards almost to zero and peters out slowly. And thus was born the 'Long Tail'.

As I wrote here, there's a problem with this article, and hence with the 'Long Tail' image itself. Despite repeated references to 'power law distributions', none of the curves Clay presented were distributions. They were histograms representing ranked lists: in other words series of numbers ordered from high to low.

What's the difference? A short answer is that the data Clay presents makes his own comparison with 'bell curve' (normal) distributions unsustainable: order from high to low and you will only ever get a downward curve.

For a longer answer, you'll have to look at some numbers. Here are some x,y values which would give you a normal distribution. (For anyone in danger of glazing over, that's 'x' as in horizontal axis, low to high values running left to right; 'y' values are on the vertical axis, low to high running bottom to top).

1	1
2	30
3	100
4	240
5	400
6	600
7	750
8	900
9	960
10	1000
11	1000
12	960
13	900
14	750
15	600
16	400
17	240
18	100
19	30
20	1

OK? And here are some co-ordinates which would give you an inverse power-law distribution:

1	1000
2	444
3	250
4	160
5	111
6	82
7	63
8	49
9	40
10	33
11	28
12	24
13	20
14	18
15	16
16	14
17	12
18	11
19	10
20	9

Just for the hell of it, here are some numbers that would give you a direct (ascending) power law distribution:

1	9
2	10
3	11
4	12
5	14
6	16
7	18
8	20
9	24
10	28
11	33
12	40
13	49
14	63
15	82
16	111
17	160
18	250
19	444
20	1000

Finally, by way of contrast, here's a series of numbers.

1000
444
250
160
111
82
63
49
40
33
28
24
20
18
16
14
12
11
10
9

I've sorted these numbers high to low, but - unlike the other three examples - there's nothing in the data that told me to do that. You could arrange them that way; you could sort them low to high instead; you could even hack them about manually to produce a rather lumpy and uneven bell curve. It's up to you.

I'm not saying that a ranked listing - arranging numbers like these high to low - is meaningless. The ranked histogram is quite a good graphic - it's informative (within limits) and easy to grasp. What I am saying is that it's an arbitrary ordering rather than a distribution. Which is to say, it's not the best way of representing this data - let alone the only way. It's a relatively information-poor representation, and one which tends to promote perverse and unproductive ways of thinking about the data.

More about this - and a couple of constructive suggestions - next time I post.

Authority you can respect

2005-06-23T00:58:00.000+01:00

Or: on popularity, deference, knowledge domains and knowledge clouds.

Or: "if Pietro was right, and Dave was right (and I was right about how they fit together), does that mean Shelley was wrong to say that Technorati was wrong?"

I'm not sure I get Technorati. As far as I can understand, it does three things.

Tagging. Using some standard HTML, bloggers can tag their own articles with keywords; Technorati then tracks and aggregates these tags, allowing users to find similarly-tagged entries in other blogs. I'm not sure I see the point of this. Compared with del.icio.us - which builds a public archive of tagged material by enabling users to tag other people's articles (and their own, if they so wish) - this seems underpowered at best, ego-driven at worst.
Linking. Technorati tracks blog-to-blog links, enabling users to find out who's been linking to their articles. I've used this a few times, but I'm not convinced it's that great a feature. Firstly, Google purports to do the same thing with its 'link:' search option; it's only the fact that 'link:' is broken that makes me use Technorati. Secondly, after tracking them for a while, it's dawned on me that I don't really care about links: I care about people reading my articles (which my hit-counter can tell me about), and I care about getting into conversations, either through an exchange of posts or in Comments threads. If people aren't interested in talking to me, I'd just as soon they didn't advertise my blog. (What would it gain me, after all?) Which brings me to
Popularity and Authority. This is the big one. From the name on down, Technorati is all about in-groups and out-groups. 'Authority' is one of the two sort orders which appear when you search for links to your blog (the other being 'date'). 'Authority' is measured by the number of in-bound links the sites linking to yours have in their own right. To put it another way, authority directly tracks popularity (although this is 'popularity' in that odd American high-school sense of the word: 'popular' sites aren't the ones with the most friends (most out-bound links, most distinct participants in Comments threads or even most traffic) but the ones with the most people envying them (hence: most in-bound links)).

The equation of authority with 'popularity' is, in one sense, neither inappropriate nor avoidable. In another sense it's both reprehensible and wrong. First, the argument in favour. As I wrote here, the distinction between the knowledge produced in academic discourse and the knowledge produced in conversation is ultimately artificial: in both cases, there's a cloud of competing and overlapping arguments and definitions; in both cases, each speaker - or each intervention - draws a line around a preferred constellation of concepts. At some level, all knowledge is 'cloudy'. Moreover, in both cases, the outcome of interactions depends in large part on the connections which speakers can make between their own arguments and those of other speakers, particularly those who speak with greater authority. (Hence controversy: your demonstration that an established writer is wrong about A, B and C will interest a lot more people - and do more for your reputation - than your utterly original exposition of X, Y and Z.) You may not like the internationally-renowned scholar who's agreed to look in on your workshop - you may resent his refusal to attend the whole thing and disapprove of his attitude to questioners; you may not even think his work's that great - but you still invite him: he's popular, which means he's authoritative, which means he reflects well on you. Domain by domain, authority does indeed track popularity.

But there's the rub - and here begins the argument against Technorati. Domain by domain, authority tracks popularity, but not globally: it makes a certain kind of sense to say that the Sun is more authoritative than the Star, but to say that it's more authoritative than the Guardian would be absurd. (Perverse rankings like this are precisely an indicator of when two distinct domains are being merged.) Similarly, it's easy to imagine somebody describing either the Daily Kos or Instapundit as the most 'authoritative' site on the Web; what's impossible to imagine is the mindset which would say that Kos was almost the most authoritative source, second only to Glenn Reynolds. But this is what drops out if we use Technorati's (global) equation of popularity with authority.

Some counter-propositions. Firstly, more is not (necessarily) better. The intrinsic appeal of different domains of knowledge varies enormously: in most academic specialities, if you've got a regular audience in three figures you're doing extraordinarily well. Conversely, if you want a mass audience, you'll need to write the kind of stuff that will get you a mass audience.

Secondly, broadcasting is not conversation; linking is not conversation. My only concern about readership is that I'm reaching enough people with similar interests to have a decent conversation. I'm particularly concerned that the people I'm responding to in this blog are reading it - but I've got no way of knowing that they are, unless they carry the conversation on, either in comments or on their own blogs (hi Adam, hi Dave). A blogroll link, while it would please my vanity, would tell me nothing at all about whether the words I write are actually being read.

Thirdly, domain by domain, popularity records itself: if you keep your eyes and ears open, you very rapidly discover the sources being cited, the authors you need to line up with (or against), the major arguments and their proponents. In this perspective, Technorati is of dubious merit at best, positively misleading at worst. A domain-by-domain popularity meter - like the information you can glean from del.icio.us link-shading, and to some extent from Technorati's tagging - could give you a condensed who's who, although the effort you could save by this kind of shortcut has to be set against the information you'd lose by not taking part in the arguments yourself. A global popularity meter - like Technorati's link-count - will tell you nothing you need to know and a lot that you don't. (This effect has been masked up to now by the prevalence of a single domain among Technorati tags (and, indeed, Technorati users): it's a design flaw which has been compensated by an implementation flaw.)

So I tend to agree with Shelley: the globally 'popular' blogs are quite popular enough already without their readers directing yet more traffic their way - and, for most of us, global 'popularity' is an irrelevant distraction. From which it follows that blogs don't need blogrolls. If we blogroll everyone whose posts we respond to, the blogroll's unnecessary. If, on the other hand, we blogroll everyone whose blogs we read - or, from the look of some blogrolls, every ~~blog~~Web site we've ever ~~read~~heard of - the power law will kick in: links will inevitably tend to cluster around the 'top' five or ten or fifty blogs, the blogs Everybody Knows, the A List (ugh).

Some final brief thoughts. Blogging tends towards conversation. Conversation routes around gatekeepers (Technorati is, precisely, a gatekeeper - but an avoidable gatekeeper). Conversations happen within domains. People cross domains, but domains don't overlap. Every domain thinks it's the only one.

And there is no long tail. (That's not connected, it's just a trailer for my next post...)

The cloud of knowing

2005-06-13T10:22:00.000+01:00

Dave Weinberger has got me thinking again (cheers, Dave).

1. I had been planning on beginning by talking briefly about Aristotle's discovery of the shape of knowledge: To know this robin is to see its place in a hierarchy of similarities (it's like other birds) and differences (it's different from other birds), an incredibly efficient way to organize complex systems.

2. I had been planning on ending by talking about knowledge as a property of conversations.

3. Last year, when writing about why blogs are not (generally) echo chambers, I had talked about conversation as the iterating of differences on a shared ground.

So, in the middle of last night it occurred to me that conversations, as the iteration of differences on the basis of similarity, are formally like Aristotle's description of knowledge as the placing of the known in a system of differences and similarities.

(The 'echo chamber' piece is well worth a look, incidentally, even if you aren't interested in the Dean campaign.)

The key point here, I think, is that the Aristotelian hierarchy is an achieved system of differences-within-similarity. If we characterise a conversation as 'the iteration of differences on the basis of similarity', the stress should be on iteration, on process. In other words, it's not a collaborative attempt to chip away the accumulated crud of ambiguity and tautology and reveal the true hierarchy of knowledge in all its crystalline precision. The knowledge produced by a conversation exists within the conversation, and grows within it; there's always another difference to be iterated (or collapsed). (Compare wikis - although not, oddly, (the public face of) Wikipedia.)

Conversations don't produce a tidy set of definitions which can be picked up and applied elsewhere. What they produce - in one light, what they are - is a tangle of more-or-less definitive associations and exclusions, all resting on a set of prior assumptions whose own definitions are fairly hazy. The sense you make of any argument depends on what you think of its reference points, the argument it's responding to, the person advancing it, the person being responded to... The knowledge produced within a conversation is the (continuing) accumulation of this kind of 'sense'. Structurally, it's not a tree; it's more like a swarm. Conversations are knowledge clouds.

Now pull back. I recently wrote a paper on the 2001-5 Italian government led by Silvio Berlusconi. I quoted several news sources, but also cited sources with titles like "Interpretive approaches and the study of Italian politics" and "System crisis and the origins of a new Right". In other words, I situated my argument within the context of arguments already advanced by other authors. I'm a newcomer to the field of Italian studies; as such, I have little or no standing in the field, and what I have is enhanced if I can underpin what I write with assertions from established writers. The credibility of my arguments is also enhanced, at least among readers who agree with the writers I've cited; to turn it round, the credibility of my arguments, advanced without supporting quotations, is minimal. (Referees made this point, without commenting on the merits of the arguments themselves.) Academic publication, I would suggest, is a continuing conversation - and academic discourse is a knowledge cloud.

Two conclusions. If 'cloudiness' is a universal condition, del.icio.us and flickr and tag clouds and so forth don't enable us to do anything new; what they are giving us is a live demonstration of how the social mind works. Which could be interesting, to put it mildly. On the other hand, those of us who are into tagging need to give some thought to what we've been doing in all these other areas to mitigate the adverse effects of clouds - ranging from group pathologies to the undue influence exerted by anti-social young guys.

The Web as Umwelt

2005-06-03T11:29:00.000+01:00

Alfred Schutz: "since human beings are born of mothers and not concocted in retorts, the experience of the exixstence of other human beings and of the meaning of their actions is certainly the first and most original empirical observation man[sic] makes"

Dave Weinberger: "some things become clearer if you do not start with the premise that people are fundamentally isolated and battle against noise in order to connect with others. Instead, we find ourselves in a world shared by others. Connection comes first. Isolation and alienation are withdrawals from the pre-existence of what is shared."

Connection comes first. I think this insight is significant and underrated, even though in some ways it's staringly obvious. If we take it seriously, it gives us (among other things) a new way of looking at the geek-pathologies of online life: in this view, it wouldn't be a question of isolated (in Real Life) individuals kidding themselves that they're connected (online), but of connected individuals distorting some of their connections - overloading some and neglecting others. But then, in obvious but significant ways, we all are connected individuals: if there's an element of self-deception in these cases, perhaps it starts with the denial of connection.

Semiological, or almost entirely?

2005-06-02T10:23:00.000+01:00

Mike Harper:

Semiotics, which is clearly older than the semantic web, tells us you can’t always map signs to real world objects. You can do it for things like, say, the Taj Mahal, but not for things like democracy, justice etc. So they map to concepts. Trouble is, you’re talking really about what’s inside someone else’s head. And you can’t really be sure what that is. So, the argument goes, stuff like RDF is just “syntactic sugar". It’s neatly structured but can’t escape the fact that the tags, urns etc have to have an agreed meaning ... I can’t bring myself to agree with this completely. In practice people seem to get by. I think there must be a feedback loop involved. If you interpret a statement about X and act on it, and your interpretation is wrong, and the interpretation matters in this case, something bad will probably happen. You will then revise your understanding of what is meant by X.

This is all good phenomenological stuff - see the Schutz quote above. One of Schutz's great arguments was that there is no definitional God's eye view - there is only human social experience, including the experience of making and using signs.

So surely the semantic web can work in small ways where all parties are agreed on the meaning of the vocabulary.

The trouble is - as Clay pointed out back here - that if you've got that level of agreement among all participants you don't need the semantics. If you're all using the same schema anyway, your respective schemas don't need to describe themselves - and if they do need to describe themselves, there needs to be a common language they can do it in, and hence a higher level of shared context.

What you can do is say "I'm using [x] to mean $FOO, which is a subtype of $BAR but does not overlap with $BAZ; how about you?" Or rather, "On 2005-06-03, writing in Manchester (England/UK/EU), I used [x] to mean $FOO..." and so on. That, to me, is (or rather will be) where it gets interesting - the point is not to encode semiotics but to encode semantics in such a way that the semiotics can be inferred.

Or rather, in such a way that the semiotics can't not be inferred. Which they need to be. Once you get away from the physical sciences and their geek spinoffs, it's very, very hard to reach a final level of granularity. You can map the physical contours of France in exactly the same way that you can map Britain - and with enough data you could map Britain 100 years ago and map France 100 years ago in exactly the same way. What you can't do is chart the number of suicides or street thefts or families in poverty or users of illegal drugs or asylum applications or hospital admissions in Britain and compare them with the figures for Britain 100 years ago, let alone with French figures. This is not because the data isn't there, but (in all those cases) because it's the product of a complex set of social interactions - and, as such, it doesn't have a stable meaning, in time or in space.

This is what I mean about inferring semiotics: figures on 'drug use', to take the most obvious example, are produced in particular ways and classified using particular criteria, which correspond to patterns of public health and law enforcement activity as well as to broader social attitudes. The data doesn't contain or express those attitudes and patterns of activity - but if you don't know about them it's effectively meaningless. ("Hey, look, there are twice as many people using drugs! Oh, wait, there are twice as many substances classified as drugs. Never mind.") The only way forward, it seems to me, is to (as it were) factory-stamp data with the conditions of its production, as far as they can be established: "this source on 'drugs' covers this period in this jurisdiction, and consequently uses definitions derived from this legislation, including this but excluding this and this".

That's what I'd like to do, anyway.

When is a spike not a spike?

2005-05-25T10:42:00.000+01:00

When it's a long tail. Maybe.

David Weinberger writes:

In a conversation with Erica George at the Berkman she pointed out that the demographics of Live Journal don't always represent one's experience of Live Journal — the demographics say that teenage girls are the largest users, but if you're a 25 year old, your social group there may not look that way at all.

Which raises an issue about the way the "long tail" is pictured. Clay's charts are accurate depictions of his data, but they have a mythic power that's misleading: The long tail looks like, well, a long tail when in fact it's a fractal curlicue of relationships.

This is an interesting point in itself - perhaps the blogosphere would be better viewed as a series (archipelago? galaxy?) of more or less closed, more or less interlinked 'spheres'. I'm not sure how you'd visualise that, though - perhaps something like the Jefferson High School network diagram?.

But there's a broader point about the accuracy of those 'long tail' graphics. Adam Marsh made an interesting point here about a recently-discovered 'long tail':

Clay refers to “the characteristic long tail of people who use many fewer tags than the power taggers.” While this chart does exhibit a “long tail,” this is simply a result of the fact that the users were ordered by decreasing tag usage (also true of the following three charts) — the X axis here doesn’t represent a value, it is just a sequence of users.

The phrase “long tail” usually refers to the observation that for many distributions, the number of elements with outlying values (the “tail”) may be cumulatively significant compared to the number of elements clustered near the average.

On inspection, it turns out that this is also true of the celebrated 'Power law and Weblogs' graphic: there are no values on the X axis, just a list of blogs arranged in descending order of number of links. This matters, because in a graphical representation of a statistical distribution both axes carry information. Typically, values of the variable being measured run low to high on the X axis, left to right, while the count of occurrences of each value runs high to low on the Y axis, top to bottom. Clay wrote, "We are all so used to bell curve distributions that power law distributions can seem odd." But Clay's own graphics aren't so much odd as misleading, and not only because he's put high values on the left of the graph rather than the right. In effect, he's got two axes conveying one piece of information. Andrew Sullivan's blog and Instapundit get a high Y value (lots of links) and a high X value (because all the sites with lots of links have been sorted to the left).

If you took the same numbers and plotted them on an X axis with values - if you produced a graph showing how many blogs had how many links, with zero at the origin on both scales... Well, I don't know what would happen - but five minutes' experimentation ~~tells~~reminds me that, if you wanted to produce a nice clear series of vertical bars rather than a line that wanders all over the place, you'd need to put 'number of blogs' on the Y axis and 'number of inbound links' on the X axis, rather than vice versa. (There's a simple reason for this: some values are unique by definition, others aren't.) Which in turn means that any vertical spike would represent large numbers of blogs (say, for example, blogs with small numbers of inbound links) while any long tail would represent small numbers (say, for example, the few blogs with lots of links).

Caveat: I haven't crunched any actual numbers, or even mumbled them gently. But maybe we've been looking at this the wrong way round, statistically speaking. Perhaps the long tail is the spike; perhaps the spike is really the long tail.

Greetings and salutations (and anomie)

2005-03-16T19:29:00.000Z

I've started this blog as a place to collect my thoughts on user-centred ontologies, ethnoclassification, folksonomies, emergent semantics and so on. I'm looking at this area as part of a project for a repository of social science data sources at Manchester University. In my spare time I run another blog, Actually Existing; Chris at qwghlm has the rare distinction of being on the blogroll in both places. (Hi Chris!)

I wasn't going to post anything else, but what do you know. I've just spent half an hour composing a comment on this entry on Many-to-Many - or rather, a reply to this comment - only to find that comments were closed. Not that it says so anywhere on the page. H'mph. Oh well, their loss is our gain.

Larry Sanger wrote: "There is nothing magical about how Wikipedia does things; it is just one system. I have every confidence that another system will arise, probably quite soon, that will blow Wikipedia out of the water, in terms of quality, while being equally productive and nearly as open".

Does Wikipedia have a big (potentially insuperable) deficit? If so, what is it?

It seems to me that Wikipedia (or any collective, open repository) has two problems. One is enabling contributions to be challenged, debated and refined; the other is getting the articles to be written by, and the debates to be conducted among, people who know stuff ('actual philosophers' in Larry's example). As I understand it, Wikipedia does the first of these very well, but can't guarantee - and, more to the point, doesn't necessarily promote - the second. The ideal is that, thanks to the process of open debate, good stuff will go into the repository and stay there, while errors are weeded out, weak entries are improved and gaps are filled. It may take a while for some of the more obscure areas to get populated adequately, but we trust that more topic area experts will come on board over time. I wonder, firstly, if that's enough - whether the quality of Wikipedia is ever likely to be uniform, or (ware straw-man) near enough to uniform for the value of a Wikipedia citation to be fairly consistent.

Secondly, I wonder if there's a risk of mistaking the goal (a repository of near-enough uniform quality) for what exists (a highly uneven repository with a few local areas of uniformly high quality). In other words, even for those people who believe that the goal is realisable and the Wikipedia framework will ultimately allow it to be realised, it's important to bear in mind - and to make it known - that we're not there yet.

Thirdly, I wonder (having read Danah's comments and looked at the 'anomie' page discussed there) if the quality problem, in some areas, is more fundamental - if the problem isn't that Wikipedia's got ground to make up but that it's facing the wrong way. For what I'd want to know about a concept like that, that page is pretty dreadful. It veers wildly between essentialism (there is a thing called 'anomie' and we know what it is, across time and space) and nominalism (different people have used this combination of letters to mean different things, who knew?). What's not there is any sense of the history of the concept: it derives from a Greek word meaning [x] (if there was such a word - the wording is unclear); it was coined by Durkheim (or it already existed and was given more prominence - again, it's unclear); he used it to mean [y]; Hayek later got hold of it (from Durkheim? from the Greeks?) and used it to mean [z]; that's different from Durkheim's conceptualisation in this way and this way; and it's since entered common parlance, probably because of [well, how? I'm not sure].

I'm not saying this to slag off some Wikipedia contributors I don't even know. My point - or rather, my tentative suspicion - is that Wikipedia may actually lend itself to problems like these, by starting from the question "What does $FOO mean?" For any value of $FOO there are two easy ways to answer that question - essentialist ('here's what it really means') and nominalist ('here are all the different ways people use it') - and if you're asking about the OSI 7-layer model, say, that's precisely what you want. (Essentialist answer: "Level 1 is defined as [a]..." Nominalist answer: "'OSI' also stands for 'Open Source Initiative'...") Philosophy, and many of the social sciences, need a very different approach - which I'll try to describe another day, maybe.