On Rating Systems and User Motivations

I've been a neglectful blogger and now, I'm going to spurn writing about tagging in favor of discussing ratings systems. Primarily because I recently encountered this really interesting blog post by the fine people at YouTube. The sum of what you'll learn in that post is this: the vast majority of videos at YouTube are rated 5 stars.

Relevant quote: "Great videos prompt action; anything less prompts indifference. Thus, the ratings system is primarily being used as a seal of approval, not as an editorial indicator of what the community thinks about a video."

The general intention of designers when they put together a rating style system such as the one at YouTube is with the idea that ratings will be used as a way users can judge content. Unfortunately, if all your content is either not rated at all or rated 5 stars, users won't get much out of those ratings.

This isn't a universal issue though. Some sites use star ratings with great success. Take Amazon and Netflix for example. Users of those sites are diligent about rating the content and sometimes will spend time on the site for the sole purpose of rating. Why are users willing to spend time rating movies on Netflix but won't take a couple of seconds to rate videos on YouTube? The answer has to do with user motivation and the inherent selfishness of users. Now, do not take umbrage, there's nothing wrong with being a selfish user. It's the job of a site designer to encourage you to act otherwise, and if they fail, it's no skin off your teeth. Carry on.

What's important for us, the designers, is to recognize and be aware that users are inherently selfish and will be highly unlikely to engage in activities that do not offer them a benefit of some kind. No one rates videos on YouTube, because the act of rating a YouTube video does not provide you with any benefit. The only people it could, in theory, assist are the submitter of the video and some small percentage of the user community when they browse for the video. But you the rater? You don't get anything.

In contrast, if I go to Netflix and spend some time rating the movies I've watched I can actually see the recommendations improve. The more movies I rate, the better Netflix's recommendations turn out to be. Thus, I have a good motivator for continuing to rate the content. Luckily for Netflix's user base, my ratings also benefit the rest of the community. Amazon works the same way. Ratings work out exactly the way we want because there's a clear benefit to everyone when it comes to contributing.

So what if YouTube took a page from that book? What would be interesting to see is if they instituted a recommendation system that actually used those video ratings to pull up similar videos relevant to you, the selfish user, would people start using the rating system differently?

Not easy to test without some significant work, but my inclination is that yea, you'd see some changes over time.

We could probably extrapolate the general theory at work here to a lot of other features and applications. The key take away is that if you want a user to take an action, you have to provide reasonable motivation no matter how small and quick the interaction appears. For example, most discussion forums are populated mostly by lurkers, only a small percentage contributes. Designing better motivators into your design could increase participation and lead to a better overall experience.

It's food for thought, in any case.

Comments

[Christy]

Interesting. Where do you think a site like Goodreads fits into this? You don't get recommendations for books to read based on the ratings you give, but users of the site rate and review regularly. Does this site just draw the kind of users who like to rate/review? Or do you think there's a more fundamental distinction between Goodreads and either YouTube or Netflix?

[Shane McDaniel]

I had two thoughts around YouTube staring. I kind of don't like the heart you have to use to favorite something, I wonder if people 5 star just as a favoriting system.

While providing a design that isn't overly unnecessary for a user is good I think they could recover a bit from the usage pattern. Basically rank a video by number of cumulative stars divided by unique user views. While this isn't actually useful for a video itself, it does provide a relative measure still I believe.

So 1 5 star rating across 10 unique users is a .5 star overall. Not really useful on its own, but since it normalizes across all videos it makes it easier to compare heavily and lightly viewed videos.

[Rachel Knickmeyer]

Shane: You might see a slight shift in how the ratings play out by making that change, but I don't think that would result in a change in user behavior. I'm relatively confident that ratings (and I've seen this in projects I've worked on as well) are only worthwhile when the user is going to actually get some return on the investment.

Christy: Except on literary sites right? That is an interesting use case. LibraryThing is similar. I was wondering if perhaps it has to do with the time investment and effort involved in the initial consuming of the thing being rated. YouTube videos take a couple of minutes to view. Movies, about 2 hours on average. Books, depending on the novel and the speed of reading, it's still well over what you'd spend with a movie. Maybe adding a rating on top of that is minor, and is also a way to share the experience in a meaningful way, thus users of a site like Good Reads are more likely to make use of the rating system.