I have been searching for network systems in which users are able to add
content for other users, and which enhance the
quality of that content through judgements of other users. This article endeavors to describe the most effective of the several factors that contribute to the
quality enhancement effect of such
network systems.
Membership
I distinguish between a
member and a
visitor by using the first term only in reference to someone who has identified himself or herself to the system. The methods a website or other networking system can use to perform
authorization and
authentication are outside the scope of this article. Enabling visitors to become members is an obvious first step in using member judgements to enhance the quality of contributions.
When members'
identities are publicly displayed with the content they provide, members will be more likely to think before they submit. However,
slashdot creators recognized that valuable content might be available from a member or visitor who would submit it only if it could be done
anonymously. Slashdot therefore allows anonymous submissions. Although most anonymous submissions have very
poor quality, there are some that are recognized through other members' judgements as having high quality and so rise in
stature.
I know of three kinds of judgement procedures but no single site or system that uses them all. Since they each have a
special kind of of positive effect on content,
they could be mixed very effectively.
One procedure is "select the best judgement(s)" wherein the member making the judgement
chooses one or more descriptions of the content. For example, slashdot allows
moderators to tag comments as "
Funny", "
Insightful", "
Off-topic", etc.
A related procedure for judging contributions is to lump all possible
positive descriptions together and all possible
negative descriptions together, and ask a member to pick
positive or negative. This method is used in
voting on
E2 (+ or -) and
e-thepeople (encourage or discourage). Most democracies use this procedure to
maximize the quality of their leaders. Unfortunately,
it doesn't
work very well.
Another procedure is to rank two or more similar contributions against each other. I have not seen any network system that uses this method, but Australia has been using it since 1918 to elect its leaders. One of the problems with using ranks is in the
interpretation of the collected data. In fact, Australia uses
preferential voting, which often causes the
Condorcet Winner to lose. The most effective interpretation I have seen is as follows: Any member who ranks a number of contributions provides one vote in a competition between each pair of contributions. For example, suppose contributions A, B, and C were ranked B, C, A by me. I thus provide three votes in three competitions: 1 vote for B over C, 1 vote for B over A, and 1 vote for C over A. An aggregation of this information might show how many votes are in favor, how many are against, how many other contributions it has been compared to, and the result against each them.
The results of judging are useful in two ways. First, they
encourage authors to submit their
best work, and to improve their writing skills over time. Second, judgements can be used by
readers to get at the highest quality content. In both cases, the simplest use ends up being an ordered list of contributions returned from some kind of search. This allows an
author to quickly identify his or her best and worst work, and it allows readers to skip over poor quality content.
For judgements that allow members to choose from a list of descriptions, the reader or author can choose the
value of each description and the system can order the search results according to those personalized description-value profiles. This capability can be found and used at slashdot.
For lumped "positive and negative" judgements, either a
percentage ("66% of the votes were positive") or a simple sum (four plus votes, and two minus votes = "+2") can be used to calculate the
score of a contribution. E2 uses the simple summing method which causes good writeups to gain in
score as they collect more votes, and therefore the reputation of a decent E2 writeup will slowly increase. Using the percentage method causes a contribution's score to be
attracted to its true value as more votes are cast so that over time, it will stabilize and
additional votes will have less effect.
Turning the ranked judgements into a single aggregate ranking can be tricky. The simplest way I can think of to handle this is the same as the lumped positive and negative method. Since a ranking like "A, B, C" provides two votes against C, two votes for A, and one vote for and one against B, this one ranking effort by a user provides 6 votes. This method asks the user to judge using the qualities of other contributions instead of his or her own internal standards. One can argue which is the better
metric, a judge's internal quality standards or the quality of related contributions, but I feel that each has its own merits and therefore that both should be used. This is indeed how
Condorcet with Approval Voting works.
The results of judging can be used in other ways as well. For example, e-thepeople calculates a relevance score which
decays over 17.5 days. This relevance score starts at 100 * the simple sum of encourage(+1) and discourgage(-1) votes, and
logarithmically decays to 1 * that sum over 17.5 days from the day it was submitted. Articles with the highest relevance score are featured on the front page of the site.
Another use of judging is to
reward members for receiving positive judgements. E2 does this in two
subtle ways. The first is by requiring a certain number of
XP to advance a level, and rewarding contributors with an XP for some percentage of the positive votes received by their contributions. The second is by also requiring a certain number of contributions to
survive editorial review which partly depends on the judgements. At some level, only one of these requirements has to be filled, but since both requirements are affected by judgements, the judgement process continues to be used in level-advancement.
Examples
E2*
Slashdot
E-ThePeople*
Wikipedia
We-Rank
*My interest in these websites is not trivial. I believe they are creating the kinds of communication that are necessary to solve the world's greatest problems. If you know of other sites that are as good or better than these two, please write them up and let me know so I can add them to the list.
Slashdot provides a long list of sites running the SLASH engine so this writeup will not mirror that list. However, if you find a slash site that warrants a writeup, node it and let me know and I'll add it to the list. But it better have a positive rep :-)