display | more...

On Fixing the Voting, Experience, and Level Systems

Back In November, when kthejoker posted his plan for revising the level system, I had certain objections to it, which I will go into later. In response, and because I did not want to be all talk, I began to draft an RFC on the level system and possible alternatives. After about a week it was getting longer and longer and still failed to account for several elements that had to be taken into account. Levels, voting, writeup counts, and by extension voting privileges, C!s, reputation, assorted metrics, and by association all sort of rewards and baggage, tangible and less tangible, that came with levels. In December dann stopped by my place and we got to talk shop. As a result, I abandoned the effort to write a document by myself about nothing but level changes, and began assembling a group that will examine the entire voting/level/xp/reward system in detail, with the goal of revising it as the single, complex entity that it is.

Those who read the daylogs will have seen sam512's post, which mentions a new project group tasked to reform the incentive system. As its instigator, I suppose that I ought to start by explaining my role in it. I am responsible for setting this thing in motion and am responsible for guiding it to a successful conclusion. I've asked golFUR to preside over the first and second phases. I guess at the moment that makes me kind of like a vice-admiral on board. I can say what the ship's mission is and where it has to go but the captain gets it there and I don't tell him how to get it there. I myself report to dann.

Project groups have a sad history here. We've had more petered out, fizzled out, and inactive project groups than I care to admit. all with the best intentions to reform one thing or the other. More often than not they end up like a committee of sorts and are about as functional. Of course we, too, have the best of intentions but can make no ironclad promise that there will be a result. Depending on what we come up with, more or less time may be required to implement it code-wise, and that's always been one of E2's weakest points since we have an all-volunteer coding staff and are at the mercy of how much time Real Life will let them contribute. We hope to avoid that by having a couple of good coders on board who can shoot down what is simply not feasible before we incorporate the unworkable into the proposal, and can help us choose between equally good options based on how easy or hard they are to code. That ought to save everyone a lot of time and pain, and expedite the implementation.

So, what's wrong with the incentive system that needs fixing?

Well, once upon a time someone thought that it would be a cool idea to have incentives for contributions. First we had votes, levels, and XP. This led to grumbling as some people rose meteorically, far faster than anyone ever intended them to, and others did not. Suddenly there were people who had earned their levels by virtue of Noding For Numbers or vote dumping or whatever other less savoury habits there are for gaining XP. A writeup count requirement was brought into play. Which didn't do anything about noding for numbers but did put the brakes on the rapid risers. Still, it led to more grumbling, much of it from the aforementioned rapid risers. There used to be a setting with fixed minimum levels for certain noders of old after they complained. This node was unkindly (but accurately as far as I'm concerned) named "level weenies." Very few of the people on it are still here. This grandfathering of levels was discontinued quite some time ago, in 2002 if I remember correctly.

Then we Raised the Bar and decided that there should be more rewards for people whose writeups were perceived as being of high quality. After a lot of soul-searching and statistical analysis of voting habits, trends, user metrics, and writeup reputations, the Honor Roll (HR) was born. Professor Pi did most of the hard work. This led to yet more grumbling. Worse, it also led to people deleting low-rep content. On the other hand it encouraged people to aim high. On the other hand of the other hand, it discouraged many people from writing content that was not destined to be high-rep. The Raised Bar in combination with the HR slowly developed into a monster. It evolved into a culture of denigrating writeups that were never meant to be anything more than humble and informative, or amusing or whimsical. This has deprived us of content that, in bulk, would be substantial and useful, or at least fun. It has also deprived us of users who are or were unwilling to contribute on those terms.

Then the view was (inevitably, perhaps) voiced that the Honor Roll Must Go. I can see where that view is coming from. However, what we are missing is a replacement for it. We need a complete solution that will take into account the way users vote, the way people perceive their role on E2, the character of incentives as social currency, and so many other things. We have tinkered with voting, with XP, and with level requirements on a piecemeal basis and without regard for the fact that the three are inextricably linked and a vital part of E2's fabric and character. There has never been a reevaluation of the entire complex of chings, votes, XP, levels, and rewards. If the Honor Roll goes, what replaces it? Probably something that will lead to more grumbling. I do believe that the HR was necessary and valuable at the time of its creation. I also agree with the majority that it's run its course and can be retired. I disagree with the notion that it can be retired without a proper replacement.

Nothing in the system accounts for the diversity of the content on E2 and the people who contribute it. The level system has always been a one-size-fits-all affair. The HR, well, we saw what the problem was when there were two sizes but only one was fashionable. Virtually everyone tried to squeeze into it kinda like we squeezed into drainpipe jeans in the 80s. Like drainpipes were easier to put on than they were to take off, level changes were easier made than discarded as each change lowered the requirements and increased users' expectations. The newer generations of noders do not remember E2 without the HR. In fact very few of the folks now on the staff were on it during the time the HR was implemented. It's easier to see the flaws in both conditions when you've experienced them both.

I'm already on the record as being unconvinced by Kyle's new Writeup Bonus system because I think it doesn't manage to be more than a variant of the original, pre-HR system. This is not new. Kyle says as much himself. Two things in its favour are that it does not have a level-up factor that depends on everyone else's reputation, and that it rewards high-rep content without demeaning low-rep content. However, I think that it reinstates and perpetuates the weaknesses that brought about the HR. My biggest issue with it is that it effectively tries to render the HR irrelevant by being even more generous with rewards than the HR itself. This is not a solution. It simply subverts the HR and this is not fair to those noders who have striven to excel based on the principles behind the HR.

Many active noders, if they turn off the HR, will rise a level, some even two levels. This may lead to more equity but also leads to massive inflation, devaluing the social currency that is XP and levels. I'll take my own figures as an example. Being somewhere in the middle to lower range of the HR scale, my 310 writeups (not including this one) are equivalent to 488 under the Honor Roll, making me a level 7 user. With Writeup Bonus they're worth 644 and level 8. That's more than twice as many as I've written and, you know what, I'm really not that good a writer. Frankly, it feels like being given a licence to cheat, a pass to game the system. Neither the HR nor Writeup Bonus do anything to reduce the inequity between established noders who have had seven or eight years of accumulated votes to drive them upwards. Rather, one would expect old, even fled, users to continue levelling up on the basis of newly cast votes and C!s. In this aspect WB is probably worse than the HR.

The thing about the Voting/XP system is that it's hard to ignore and so are any changes that are made to it. As we saw with the HR, not only were most of those who were not interested in it unable to ignore it, but it also weaseled its way all the up to being the dominant paradigm for levelling up, despite the fact that it was still perfectly possible to level up under the old system. The documentation may say that the administration does not take the voting/XP system too seriously (and woe to those who do) but it is far from insignificant. You may dismiss its "game-y" aspects as fluff but the truth is that we believe in rewards and incentives, and so do most users. Many users love the game but wish for better gameplay.

What we're doing about it and who is "we" anyway?

Nate describes what he calls the "I like it" relationship between users and content as being something very valuable; it is something that will have to form the foundation of any level/XP system. There is no easy, simple solution. The situation calls for something that will have to be more elaborate than any of the existing proposals for reform yet it still needs to be elegant and transparent. It will not be complex for the sake of being complex--we've had enough protests about the Honor Roll being arcane to anyone without a background in statistics despite our efforts to explain it. Rather, it will have to be somewhat involved and detailed because it will address all aspects of the incentive and voting system.

With the help of Oolong I have recruited a bunch of noders, some of who you probably know, based on several criteria: we know them; we know that they have an opinion and are not shy to share it; their opinions, outlook, skills, and personalities are diverse; they're willing to devote the time to the task. If you've read the public proposals of folks like sam512 and in10se, as well as those of others like ushdfgakjasgh and kthejoker, you've seen just how diverse they are and you can bet your bottom dollar that some of their suggestions will fly out the window and onto the same refuse pile as some of mine are sure to end up on. That's inevitable since the existing proposals clash on many levels. We have quite a task ahead of us and not the slightest chance of merrily agreeing to what any one of us comes up with. That's how we want it to be. On the other hand, we have set our sights high based on the fact that we share the good will and determination to come up with a complete proposal and not a bunch of half-arsed, conflicting compromises.

Our mission is to:

  1. Examine the rationale behind the voting/xp system, its past, and its present
  2. Examine the benefits and drawbacks of the existing and historical options
  3. Debate proposals point by point
  4. Integrate the successful proposals into a complete solution
  5. Determine which aspects are vital and which may be modular or optional
  6. Present a master solution to top management

Here are some of the proposals that will be on the table. Not one of them is guaranteed to pass, at least not without a healthy argument over its merits and drawbacks. Some are shady to begin with and will likely be dropped early but not before they've had their time on the table. Your mileage will certainly vary, as will ours. Many folks, both staffers and non-staffers, have debated these in private or in groups. Likewise, none of the existing institutions are sacred and immune to revision.

  • Fewer levels
  • Different level requirements. This is the focus of most existing proposals
  • More levels
  • Changing XP rewards for certain activities
  • A greater variety of level-based powers
  • A choice of paths to levelling up
  • Expiring old votes and C!s
  • Unlimited votes
  • Earning C!s by getting C!s
  • Votes for level 1 users
  • Level system opt-outs or voluntary caps
  • Changing the reward and penalties for posted/deleted writeups
  • Permitting or abolishing the exchange of XP for other "valuables"
  • Integrating Achievements into the incentive system
  • Canning downvotes

Our mandate is broad. Our thoughts will also have to be broad and our minds open. We may come up with something that looks conventional and similar to what we have now. We may come up with something completely radical. We may even end up suggesting that we recalculate everyone's level and XP from scratch. All related activity has been recorded in the database. It's possible. Perhaps the best thing that could come out of this project is a shift in mentality--a clear statement that E2 is still about "everything" and simple content is not just acceptable but valuable. I believe that the public followed the management and the staff in adopting the Raised Bar. After we put it on a pedestal, too many people started worshipping it as the Only Way To Node. Perhaps we need to place more emphasis on the users, not the content. Perhaps we need to do the opposite.

The group conducts business in a usergroup here on the site and will have one or more collaboration nodes but the bulk of the business is best off on the forum, where the interface, unlike the E2 message system, is designed for such debates. The forum discussion is closed, though we may consider making it read-only or opening it later. I'm one of the biggest advocates of transparency but letting everyone jump into the discussion or critique everything in real time is inefficient and distracting. The former is the main reason why we've kept the group small. We expect to get better and faster results with a "commando" size group than with a large, unwieldy committee. You may speak your mind freely elsewhere on the forum and will probably get replies from one or more of us if you do. Any suggestions, however odd, will more likely than not find their way into our discussion.

We expect the process to take at least a month, since most of us do have real lives to attend to, and have given ourselves a deadline--and I am committing the group and myself to it by stating it here--of the end of next month to leave a complete proposal in dann's mailbox. Should the project fail, the responsibility will be mine and you all get to pelt me with sour grapes or something. Dann and I both understand that there are few constant downers bigger than the inability of the administration to deliver on promised changes to those who expect them, and that we've delivered way more of those disappointments than is permissible. We will do our best to recruit the staff to swiftly implement the proposal, which the group hopes will be good enough to be accepted in toto. Should the project succeed, it shall do so as a collective effort and everyone will get credit, praise, and glory (but not a single XP). More importantly, there will be a well thought-out incentive system that everyone can live with--or be given the option of living without, but that's exactly the kind of thing that needs to be discussed.

Wish us luck and keep talking to us.

Log in or register to write something here or to contact authors.