A few weeks ago, we posted a blog post describing some of our thinking about user reviews, some tools we added, and some of the next areas we were planning on addressing. If you haven't yet, it's worth a read to understand our thinking about reviews and the goals we think user reviews should be addressing.
Today we're making a few more improvements to user reviews to address some concerns and feedback that we've been hearing, and to ensure that the system remains a useful tool for you to identify games that you will enjoy. In this update, we're focusing on how we determine the helpful reviews you see on each game's page and the order in which they are shown.
What is a helpful review
Reviews should help paint a picture of what it's like to play the game and how well the game has been enjoyed by the people who have already played it. A good review typically describes some of the factors that directly impact the experience of playing the game, which can include a wide variety of things like how well the matchmaking works, how buggy the game is, or whether the game represents a good value for the price.
While everyone that has played a game probably has opinions on how much they enjoyed the game or not, some people are better than others at writing thoughtful descriptions that are useful for other potential players. We think it's fine for players to be able to indicate whether they liked a game or not, but not every review is useful to show in greater detail. A helpful review then is one that includes enough information to aid in your understanding of whether you are likely to find the game fun, or whether you should avoid the game and explore more alternatives.
As of this writing, there are over 36,579,839 reviews posted by players across all of Steam. Some games have a handful of user reviews while others have hundreds of thousands. Regardless of the number of reviews, we want to make sure that the most helpful and relevant reviews are the ones that you see first when you are looking at a game's page in the Steam store.
Sorting helpful reviews
Below each review you've probably seen the question of "did you find this review helpful?" along with a set of buttons to indicate yes or no. You can use these controls whether you have already purchased the game or not, because it is usually prior to purchase that you are looking at user reviews and considering which of them are helpful in making your purchase decision.
These inputs are then used to determine which reviews people generally find most helpful. Up to now, our system simply looked at how many people had rated each review as 'helpful' and how many people had rated it as 'not helpful' and then highlighted the ten reviews that the most percentage of people found helpful. Since games can change (sometimes dramatically) over time, we introduced a change a while ago that prioritizes showing recently-posted helpful reviews, as they are more likely to reflect the current state of the game. Of course these are just the defaults that are shown on a store page, and you can still easily browse all of the reviews for a particular game if you want to investigate further.
In a perfect world, people would truthfully mark a few reviews that were helpful for deciding to purchase or not purchase the game and we could use that data to directly determine the ten most helpful reviews. Alas, it turns out that not everyone is as helpful as we would like. Instead, we are seeing more and more feedback from players that the helpful reviews shown on store pages aren't representative of how well people are actually enjoying the game.
Taking a closer look
So we took a closer look at the patterns and behaviors of people that are rating reviews. Of the 11 million people that have used the helpful buttons, most follow a reasonable pattern of usage: Typical players rate a few reviews as helpful or unhelpful while deciding whether to make a purchase. However, we found a small set of users on the far extreme that are clearly trying to accomplish something quite different from normal players, and are rating more than 10,000 reviews as helpful or unhelpful on a single game. This behavior is not only humanly impossible, but definitely not a thoughtful indication of how 'helpful' each of those reviews were. These users also tend to rate up just the negative reviews while rating down the positive reviews (or vice-versa) in an attempt to distort which reviews are shown by default.
Because of how many reviews these users are rating, they each have a disproportionate amount of influence over the display of helpful reviews and cause certain reviews to appear more prominently than they should be. This can result in a confusing appearance where the default set of reviews shown are negative, even when most players have posted positive reviews and clearly enjoy the game.
What we are changing
Up to this point, we've tried to maintain just showing the simple math behind how we calculate whether a review is helpful or not--the percentage and number of people who indicated a review as helpful. We like systems that are transparent and easy to understand, as they are also easier to believe and trust. Unfortunately, this has resulted in a system that allows a small group to manipulate reviews to a degree that is clearly decreasing the value of Steam for many other players. So we're making two main changes.
Firstly, our system will use a new method of calculating the helpfulness of each review, taking into account the users that are trying to manipulate the system. One way we're doing that is by counting the helpful ratings on reviews differently for users that are far outside the norm. Ratings from users that follow normal patterns of rating will continue to be counted the same way that they have, whereas accounts that rate an excessive number of reviews on an individual game will see the weight of each individual rating count for less and less.
Secondly, store pages will now show the default helpful positive and negative reviews in a similar proportion to that of the overall review score for the game. For example, if the game is reviewed positively by 80% of reviewers, then the ten reviews shown by default on the store page will be 80% positive, showing eight positive and two negative. This should keep the reviews shown on a game's page from being so easily manipulated by a few determined players and should more accurately represent the overall sentiment of the people playing the game.
We're rolling out both of these changes as a beta today. You can turn the new method on and off to see how it impacts the default display of reviews on any given store page. Note that these changes only impact the default listing of reviews (Called "Summary") and the "Most Helpful" display option.
Still much more to do
There are still a number of further changes we're considering for the user review system. One thing we're looking at, is how review scores on games change over time as games develop or languish, and ways to better indicate how players are enjoying the game right now. We also want to better indicate when players are reviewing issues in a game that only pertain to a particular region (such as server locations), or particular language (such as poor translations). We're also exploring ways to calculate a personalized review score for each player and thinking about how that would look.
We know the review system is important for players and developers and we're going to continue making improvements to ensure that user reviews remain useful and trustworthy. Please let us know your thoughts on these latest changes.
7.07c: == * Bounty Runes: Base XP reduced from 50 to 25 * Bounty Runes: XP Growth increased from 5/min to 7/Min
* TP Scroll: Initial cooldown increased to 10 seconds after creeps spawn
* Bloodstone: Mana Regeneration per charge reduced from 0.4 to 0.35
* Meteor Hammer: Impact stun duration increased from 1.75 to 2
* Aeon Disk: The damage that triggers the item will no longer drop your health below 80% (So if you are 82% and take damage that would have made you 70% health, your health drops to 80% instead) * Aeon Disk: cost reduced from 1675 to 1600
* Neutral Troll Priest: Mana Regeneration aura reduced from 3 to 1.5
* Alchemist: Level 10 Talent changed from +5 Armor to -5s Unstable Concoction Cooldown * Alchemist: Level 15 Talent increased from +250 Health to +350 Health * Alchemist: Level 20 Talent increased from +200 Unstable Concoction Damage to +360 * Alchemist: Level 25 Talent changed from -8s Unstable Concoction Cooldown to +30 Chemical Rage Regeneration
* Anti-Mage: Mana Break damage reduced from 60% to 50% * Anti-Mage: Blink cooldown increased from 12/9/7/5 to 15/12/9/6 * Anti-Mage: Level 15 Talent increased from -1s Blink Cooldown to -2s * Anti-Mage: Level 20 Talent illusion incoming damage increased from 300% to 350%
* Bane: Nightmare cast range reduced from 500/550/600/650 to 425/500/575/650
* Beastmaster: Wild Axes damage reduced from 40/70/100/130 to 30/60/90/120 * Beastmaster: Hawk movement speed reduced from 450 to 430
* Brewmaster: Earth Brewling armor reduced from 5 to 3
* Broodmother: Level 10 Talent increased from +60 Spawn Spiderling Damage to +75 * Broodmother: Level 20 Talent reduced from +60 Attack Speed to +50 * Broodmother: Level 25 Talent reduced from +75 Insatiable Hunger Damage/Lifesteal to +60
* Centaur: Agility gain reduced from 1.6 to 1.0
* Chaos Knight: Phantasm cooldown increased from 130 to 145
* Chen: Holy Persuasion manacost reduced from 100/110/120/130 to 70/90/110/130 * Chen: Level 10 Talent increased from +20% XP Gain to +35% * Chen: Level 10 Talent increased from +125 Cast Range to +200
* Clockwerk: Power Cogs attacks to kill rescaled from 2/2/2/3 to 2
* Drow Ranger: Precision Aura reduced damage from 16/22/28/34% to 10/18/26/34% * Drow Ranger: Level 15 Talent increased from +300 Gust Distance/Knockback to +400
* Earth Spirit: Geomagnetic Grip silence duration reduced from 2.5/3/3.5/4 to 2/2.5/3/3.5
* Ember Spirit: Level 10 Talent increased from +150 Flame Guard Absorption to +200 * Ember Spirit: Level 10 Talent increased from +20 Damage to +25 * Ember Spirit: Level 15 Talent increased from +30 Flame Guard DPS to +50
* Huskar: Base movement speed reduced by 5
* Lone Druid: Spirit Bear Health increased from 1400/1800/2300/2700 to 1500/2000/2500/3000
* Luna: Strength gain reduced from 2.5 to 2.2 * Luna: Lucent Beam cast point reduced from 0.5 to 0.4 * Luna: Level 10 Talent increased from +150 Cast Range to +200 * Luna: Level 15 Talent increased from -2s Lucent Beam cooldown to -3s
* Meepo: Base magic resistance reduced from 35% to 25% * Meepo: Level 10 Talent changed from +10% Lifesteal to +7 Strength * Meepo: Level 10 Talent changed from +30 Movement Speed to +20 Damage * Meepo: Level 15 Talent changed from +10 Strength to +10% Lifesteal * Meepo: Level 20 Talent changed from +40 Damage to -4s Earthbind Cooldown * Meepo: Level 25 Talent increased from -4s Poof Cooldown to -5s * Meepo: Level 25 Talent reduced from +700 Health to +600
* Morphling: Morphing back to your hero sets you back to the same percentage Health and Mana you originally had before you morphed (so if you are low health after taking damage in a Morph, morphing back effectively heals you) * Morphling: Morph now changes your primary attribute to the target's one * Morphling: Morph duration reduced from 30 to 20 seconds * Morphling: Level 20 Talent changed from -25s Morph Cooldown to +15s Morph Duration
* Naga Siren: Agility gain increased from 2.75 to 3.2 * Naga Siren: Level 10 Talent changed from +175 Health to +10% Mirror Image Damage
* Nightstalker: Darkness vision limit against buildings and wards changed from 675 to 800 * Nightstalker: Hunter in the Night Flight Vision reduced from 1200 to 1000
* Omniknight: Base intelligence reduced by 2 * Omniknight: Degen Aura AoE reduced from 350 to 300 * Omniknight: Level 20 Talent reduced from +6 Mana Regen to +4
* Shadow Fiend: Shadow Raze bonus stacking damage increased from 60 to 80
* Sniper: Strength gain reduced from 2.0 to 1.7 * Sniper: Level 10 Talent increased from +12% Cooldown Reduction to +15% * Sniper: Level 15 Talent increased from +20 Shrapnel DPS to +25
* Spectre: Desolate damage reduced from 20/35/50/65 to 20/30/40/50 * Spectre: Desolate now pierces Spell Immunity
* Spirit Breaker: Level 15 Talent increased from +30 to +40 Damage * Spirit Breaker: Empowering Haste now has seperate values for your hero and your team * Spirit Breaker: Empowering Haste on self increased from 6/10/14/18% to 8/12/16/20% * Spirit Breaker: Empowering Haste on allies reduced from 6/10/14/18% to 4/6/8/10%
* Storm Spirit: Agility gain reduced from 1.8 to 1.5 * Storm Spirit: Level 20 Talent reduced from +40 Attack Speed to +30
* Templar Assassin: Level 10 Talent changed from +250 Health to +3 Psionic Traps * Templar Assassin: Psionic Trap damage increased from 175/250/325 to 250/350/450 * Templar Assassin: Psionic Trap max charge time reduced from 6s to 4s
* Terrorblade: Intelligence gain from 1.75 to 1.6 * Terrorblade: Metamorphosis cooldown increased from 140 to 155 * Terrorblade: Metamorphosis movement speed reduction from -30 to -25 * Terrorblade: Level 20 Talent increased from -8s Reflection Cooldown to -10s
* Tiny: Grow now grants +5/10/15 Armor * Tiny: Grow no longer increases movement speed * Tiny: Grow bonus base damage reduced from 40/80/120 to 30/45/60 * Tiny: Tree Grab bonus building damage rescaled from 75% to 60/80/100/120% * Tiny: Tree Grab unit damage rescaled from 30% to 10/20/30/40% * Tiny: Tree Grab attack count rescaled from 2/3/4/6 to 5 * Tiny: Tree Grab cooldown rescaled from 15 to 18/16/14/12 * Tiny: Level 20 Talent reduced from -10s Tree Grab Cooldown to -7s * Tiny: Toss no longer throws Spell Immune enemies
* Tusk: Strength gain increased from 2.6 to 3.0
* Underlord: Level 20 Talent reduced from +30 Health Regen to +20
* Vengeful Spirit: Level 20 Talent increased from -4s Magic Missile Cooldown to -5s * Vengeful Spirit: Vengeance Aura attack damage rescaled from 10/18/25/34% to 12/18/24/30%
* Windranger: Turn rate improved from 0.6 to 0.8 * Windranger: Windrun movement speed bonus increased from 50% to 60% * Windranger: Level 15 increased from +75 Attack Range to +100 * Windranger: Level 15 increased from +90 Powershot Damage to +100 * Windranger: Level 20 changed from +25% Cooldown Reduction to +1 Shackleshot Target * Windranger: Level 25 changed from +1 Shackleshot Target to +30% Cooldown Reduction * Windranger: Level 25 increased from +20% Ministun Focus Fire to +30%
* Winter Wyvern: Arctic Burn damage reduced from 9% to 7.5/8/8.5/9% * Winter Wyvern: Base damage reduced by 3 * Winter Wyvern: Level 15 Talent reduced from +600 Night Vision to +500