Evaluating Tag Clouds

All tags have some meaning. Even the most obscure tag has meaning to the user who submitted it. Clay Skirky argues that in a folksonomy there is no such thing as a synonym because each user generates a particular tag for a reason (Folksonomies). In fact, this is one of the greatest strengths of a folksonomy. It allows for the existence of terms that would normally be considered synonyms in a thesaurus-based taxonomy. While a taxonomy may state that “movies” and “cinema” are equivalent terms, many film critics would disagree. This strength can turn into a weakness when users submit misspelled tags or otherwise avoid tagging best practices (outlined to the right). Poor quality tags, though embedded with meaning, are what lead to folksonomies being labeled as chaotic systems.

If we are to incorporate a folksonomy into our online library catalogs in a meaningful way that minimizes chaos, then we must find a way to evaluate how well a tag cloud describes a work.
This raises two questions:

• How are we to evaluate tags when all tags have meaning?
• How are we to evaluate a folksonomy without turning it into a taxonomy?
  

Any editorial decisions made about tags immediately are problematic. First, if all tags have meaning than you cannot delete or edit them. Second, editorial decisions imply a type of rule making that would destroy a folksonomy by turning it into a taxonomy. Therefore the only way to evaluate a tag cloud is to do so quantitatively. A quantitative metric removes the judgment and rule making of what tags are acceptable and what are not. Further such a metric could be done non-invasively so that the folksonomy can continue to grow undisturbed. Such a metric would be iterative so as to adjust as tag clouds change. The table below summarizes the advantages and disadvantages of various metrics we considered to evaluate the descriptive quality of a tag cloud.

We decided that the best metric to use is one based on the h index created by J. E. Hirsch, which evaluates an academic’s citation rates (An index to quantify an individual’s scientific research output). For our purposes the h index is defined as follows:

A tag cloud has an index of h if h of its N unique tags have been repeated at least h times, and the other (N-h) tags have no more than h citations each.

The h index is directly proportional to number of unique tags and frequency of tags, both of which lead to better description.