May 21, 2019

Steam News - TomB

In our recent post on User Reviews Revisited we covered our process for identifying off-topic review bombs on games, and how you can decide for yourself whether or not you'd like to see them included in Review Scores throughout the store. Back in 2017, we defined a review bomb as an event where players post a large number of reviews in a very compressed time frame, aimed at lowering the Review Score of a game. At the time, we wondered if there'd be "positive review bombs", but there were no examples of one. We decided to wait and see.

A few weeks ago, in response to the Notre Dame tragedy, Ubisoft did something great for their fans by making Assassin's Creed: Unity available for free on Uplay, and committing funds towards rebuilding the monument. This led to a significant spike in players of AC: Unity on Steam, and a large number of positive reviews for the game. This led us, and members of the community, to wonder if this was finally a positive review bomb, and whether it should be considered off-topic.

Data-wise, it doesn't quite fit the pattern of negative review bombs: in the case of AC:Unity there was a significant increase in actual players alongside the increase in reviews. That isn't necessarily the case with a typical off-topic review bomb (but, to be clear, we have seen some negative review bombs with that characteristic).

Without reading the actual reviews, the data here all looks very much like a game that's gone on sale, or received an update. It's seen a spike in players, and as many people have come to realize, there's a fairly good correlation between player count and user reviews - if you get more players, you're going to get more reviews.

But we also went and read a large chunk of the reviews. Some reference Notre Dame or the giveaway. But most just look like standard reviews of a new player, or a player that's returning to a product they bought a while ago. Ubisoft has released significant updates to AC:Unity since launch, and it appears that some players who bounced off it at launch have returned, and found themselves enjoying the game more.

So it's not clear it's a review bomb. It certainly doesn't fit our original definition in the "aimed at lowering the Review Score" section, but back in 2017 the community's terminology around "review bombs" was also focused only on concerted negative efforts. It'd be nice to change that terminology to something that doesn't imply positive or negative, but that's really up to the community.

But moving on. We thought we should still spend some time discussing whether it's off-topic. As a refresher - if we mark it off-topic, the only result will be that the Review Score won't include the reviews over this time period. In this case, the game clearly didn't change (although it does appear to have changed since some of the reviewers last played it). But the context around it has. It's not uncommon for us to see context changes around a game that then result in changes to the game's popularity. But are those games actually better, or worse, after those context changes? Should that be reflected in the Review Score?

When thinking about whether it's off-topic, we often ask ourselves if the "general" Steam customer browsing the store would be better served if the Review Scores included the reviews. We don't want players buying games they don't have fun with, because that's not good for any of us in the industry, so we want that Review Score to be as useful as possible. But with that question in mind, it still doesn't seem like there's a single answer to whether context changes should be reflected in the Score.

Some context changes are largely divorced from the game itself, such as news about the political convictions of the developer. Important to some customers, for sure, but unlikely to be something we would feel confident should be included by default for all users. But other context changes can be significant predictors of whether or not you'll be happy with your purchase. For instance, news that the live team of a multiplayer online game has been laid off is a context change that seems useful to have reflected in the Review Score for prospective buyers.

In this case, the Notre Dame tragedy has made it so that AC:Unity happens to now include the world's best virtual recreation of the undamaged monument. That's a context change that could be increasing the value players are getting from the game, so perhaps the game really is better than it was before? Or maybe that's unrelated, and it's actually players feeling good about Ubisoft's significant donation to rebuilding the monument? Irrespective of the reason, perhaps this is a short-term temporal effect? Should temporal effects even be included in Review Scores? If a game was heavily focused on a time of the year, like Christmas, we suspect we'd see it have Review Score fluctuations around Christmas-time, as more people bought it and thought it was better on average than people who bought it at other times of the year.

If visiting the virtual Notre Dame is a reason players have reviewing the game more positively, we'd expect the Review Score to continue to reflect it in the future, albeit at a lower volume. But that's still the case even if it's not the reason - the future Review Score would revert to where it was prior to this event.

So, we're not really sure what to do here. It doesn't actually seem to be a review bomb in the way we've previously defined them, but maybe that's just our definition being wrong. But even if we define it as one, we're not sure whether it should be off-topic or not. The overall Review Score would decrease by 1.3% if we marked it, which wouldn't have any significant effect on its visibility in the store (see the FAQ below for more on that). So either way, the game itself wouldn't be affected by our decision.

As a result, we've decided we're just going to leave it alone. But hopefully, this post has helped you understand that thinking behind why we've ended up there. If you have any thoughts on how we should approach this case, or similar cases in the future, we'd love to hear from you in the comments below.

FAQ

Q: Why wouldn't AC:Unity's visibility in the Steam Store be affected by the Review Score change if you marked it off-topic?

A: There are a number of places in the Store where we factor in Review Scores when sorting games into lists. Games receive a "boost" based upon which user review bucket they're in (Mixed, Mostly Positive, etc). The actual boost amount is quite small relative to other factors in the Store, and it's essentially the same for all the Mixed or above review buckets. But there's a big boost drop-off as soon as a game drops below Mixed into the negative buckets, which occurs at the point where less than 40% of the user reviews are positive. A Mixed game receives over 500% more boost than a game in Mostly Negative. That might seem scary, but we're still talking about a boost that's small relative to many other store factors, and it's the minority case - 71.7% of titles on the Store are Mixed or above.

In the case of AC:Unity, the positive review spike looks significant in the Recent Reviews view, but in the overall Review Score it only shifted from 59.7% up to 61%, both of which are squarely in the Mixed reviews bucket.

Q: Why were people buying AC:Unity if it was free on Uplay?

A: Any time there's an increase in visibility of a game, we generally see an increase in players and sales. Many of Steam's systems are designed to multiply player interest and activity around a game. Obviously, some players who saw the news might decide to go and buy AC:Unity to explore virtual Paris. But even players who already own AC:Unity may drive further sales, because they may decide to fire up AC:Unity to look at the Notre Dame. Steam will broadcast that activity in the form of toasts, achievements earned, trading cards, and so on, and that increases the visibility of AC:Unity to other players. For example, in the time period since the events in Paris, there's been more than a 500% increase in the number of toasts shown to players telling them that a friend has launched AC:Unity.

You might expect that increase in visibility to have resulted in more players going over and picking it up for free on Uplay. It's possible that these are players who simply didn't know they could get it for free, in spite of the most prominent user review on the store page doing its best to let them know that.