I have been searching for network systems in which users are able to add content for other users, and which enhance the quality of that content through judgements of other users. This article endeavors to describe the most effective of the several factors that contribute to the quality enhancement effect of such network systems.


I distinguish between a member and a visitor by using the first term only in reference to someone who has identified himself or herself to the system. The methods a website or other networking system can use to perform authorization and authentication are outside the scope of this article. Enabling visitors to become members is an obvious first step in using member judgements to enhance the quality of contributions.

When members' identities are publicly displayed with the content they provide, members will be more likely to think before they submit. However, slashdot creators recognized that valuable content might be available from a member or visitor who would submit it only if it could be done anonymously. Slashdot therefore allows anonymous submissions. Although most anonymous submissions have very poor quality, there are some that are recognized through other members' judgements as having high quality and so rise in stature.


I know of three kinds of judgement procedures but no single site or system that uses them all. Since they each have a special kind of of positive effect on content, they could be mixed very effectively.

One procedure is "select the best judgement(s)" wherein the member making the judgement chooses one or more descriptions of the content. For example, slashdot allows moderators to tag comments as "Funny", "Insightful", "Off-topic", etc.

A related procedure for judging contributions is to lump all possible positive descriptions together and all possible negative descriptions together, and ask a member to pick positive or negative. This method is used in voting on E2 (+ or -) and e-thepeople (encourage or discourage). Most democracies use this procedure to maximize the quality of their leaders. Unfortunately, it doesn't work very well.

Another procedure is to rank two or more similar contributions against each other. I have not seen any network system that uses this method, but Australia has been using it since 1918 to elect its leaders. One of the problems with using ranks is in the interpretation of the collected data. In fact, Australia uses preferential voting, which often causes the Condorcet Winner to lose. The most effective interpretation I have seen is as follows: Any member who ranks a number of contributions provides one vote in a competition between each pair of contributions. For example, suppose contributions A, B, and C were ranked B, C, A by me. I thus provide three votes in three competitions: 1 vote for B over C, 1 vote for B over A, and 1 vote for C over A. An aggregation of this information might show how many votes are in favor, how many are against, how many other contributions it has been compared to, and the result against each them.
How judgement results can be used
The results of judging are useful in two ways. First, they encourage authors to submit their best work, and to improve their writing skills over time. Second, judgements can be used by readers to get at the highest quality content. In both cases, the simplest use ends up being an ordered list of contributions returned from some kind of search. This allows an author to quickly identify his or her best and worst work, and it allows readers to skip over poor quality content.

For judgements that allow members to choose from a list of descriptions, the reader or author can choose the value of each description and the system can order the search results according to those personalized description-value profiles. This capability can be found and used at slashdot.

For lumped "positive and negative" judgements, either a percentage ("66% of the votes were positive") or a simple sum (four plus votes, and two minus votes = "+2") can be used to calculate the score of a contribution. E2 uses the simple summing method which causes good writeups to gain in score as they collect more votes, and therefore the reputation of a decent E2 writeup will slowly increase. Using the percentage method causes a contribution's score to be attracted to its true value as more votes are cast so that over time, it will stabilize and additional votes will have less effect.

Turning the ranked judgements into a single aggregate ranking can be tricky. The simplest way I can think of to handle this is the same as the lumped positive and negative method. Since a ranking like "A, B, C" provides two votes against C, two votes for A, and one vote for and one against B, this one ranking effort by a user provides 6 votes. This method asks the user to judge using the qualities of other contributions instead of his or her own internal standards. One can argue which is the better metric, a judge's internal quality standards or the quality of related contributions, but I feel that each has its own merits and therefore that both should be used. This is indeed how Condorcet with Approval Voting works.

The results of judging can be used in other ways as well. For example, e-thepeople calculates a relevance score which decays over 17.5 days. This relevance score starts at 100 * the simple sum of encourage(+1) and discourgage(-1) votes, and logarithmically decays to 1 * that sum over 17.5 days from the day it was submitted. Articles with the highest relevance score are featured on the front page of the site.

Another use of judging is to reward members for receiving positive judgements. E2 does this in two subtle ways. The first is by requiring a certain number of XP to advance a level, and rewarding contributors with an XP for some percentage of the positive votes received by their contributions. The second is by also requiring a certain number of contributions to survive editorial review which partly depends on the judgements. At some level, only one of these requirements has to be filled, but since both requirements are affected by judgements, the judgement process continues to be used in level-advancement.



*My interest in these websites is not trivial. I believe they are creating the kinds of communication that are necessary to solve the world's greatest problems. If you know of other sites that are as good or better than these two, please write them up and let me know so I can add them to the list.
Slashdot provides a long list of sites running the SLASH engine so this writeup will not mirror that list. However, if you find a slash site that warrants a writeup, node it and let me know and I'll add it to the list. But it better have a positive rep :-)

Log in or register to write something here or to contact authors.