May 21, 2019

Steam News - TomB

In our recent post on User Reviews Revisited we covered our process for identifying off-topic review bombs on games, and how you can decide for yourself whether or not you'd like to see them included in Review Scores throughout the store. Back in 2017, we defined a review bomb as an event where players post a large number of reviews in a very compressed time frame, aimed at lowering the Review Score of a game. At the time, we wondered if there'd be "positive review bombs", but there were no examples of one. We decided to wait and see.

A few weeks ago, in response to the Notre Dame tragedy, Ubisoft did something great for their fans by making Assassin's Creed: Unity available for free on Uplay, and committing funds towards rebuilding the monument. This led to a significant spike in players of AC: Unity on Steam, and a large number of positive reviews for the game. This led us, and members of the community, to wonder if this was finally a positive review bomb, and whether it should be considered off-topic.

Data-wise, it doesn't quite fit the pattern of negative review bombs: in the case of AC:Unity there was a significant increase in actual players alongside the increase in reviews. That isn't necessarily the case with a typical off-topic review bomb (but, to be clear, we have seen some negative review bombs with that characteristic).

Without reading the actual reviews, the data here all looks very much like a game that's gone on sale, or received an update. It's seen a spike in players, and as many people have come to realize, there's a fairly good correlation between player count and user reviews - if you get more players, you're going to get more reviews.

But we also went and read a large chunk of the reviews. Some reference Notre Dame or the giveaway. But most just look like standard reviews of a new player, or a player that's returning to a product they bought a while ago. Ubisoft has released significant updates to AC:Unity since launch, and it appears that some players who bounced off it at launch have returned, and found themselves enjoying the game more.

So it's not clear it's a review bomb. It certainly doesn't fit our original definition in the "aimed at lowering the Review Score" section, but back in 2017 the community's terminology around "review bombs" was also focused only on concerted negative efforts. It'd be nice to change that terminology to something that doesn't imply positive or negative, but that's really up to the community.

But moving on. We thought we should still spend some time discussing whether it's off-topic. As a refresher - if we mark it off-topic, the only result will be that the Review Score won't include the reviews over this time period. In this case, the game clearly didn't change (although it does appear to have changed since some of the reviewers last played it). But the context around it has. It's not uncommon for us to see context changes around a game that then result in changes to the game's popularity. But are those games actually better, or worse, after those context changes? Should that be reflected in the Review Score?

When thinking about whether it's off-topic, we often ask ourselves if the "general" Steam customer browsing the store would be better served if the Review Scores included the reviews. We don't want players buying games they don't have fun with, because that's not good for any of us in the industry, so we want that Review Score to be as useful as possible. But with that question in mind, it still doesn't seem like there's a single answer to whether context changes should be reflected in the Score.

Some context changes are largely divorced from the game itself, such as news about the political convictions of the developer. Important to some customers, for sure, but unlikely to be something we would feel confident should be included by default for all users. But other context changes can be significant predictors of whether or not you'll be happy with your purchase. For instance, news that the live team of a multiplayer online game has been laid off is a context change that seems useful to have reflected in the Review Score for prospective buyers.

In this case, the Notre Dame tragedy has made it so that AC:Unity happens to now include the world's best virtual recreation of the undamaged monument. That's a context change that could be increasing the value players are getting from the game, so perhaps the game really is better than it was before? Or maybe that's unrelated, and it's actually players feeling good about Ubisoft's significant donation to rebuilding the monument? Irrespective of the reason, perhaps this is a short-term temporal effect? Should temporal effects even be included in Review Scores? If a game was heavily focused on a time of the year, like Christmas, we suspect we'd see it have Review Score fluctuations around Christmas-time, as more people bought it and thought it was better on average than people who bought it at other times of the year.

If visiting the virtual Notre Dame is a reason players have reviewing the game more positively, we'd expect the Review Score to continue to reflect it in the future, albeit at a lower volume. But that's still the case even if it's not the reason - the future Review Score would revert to where it was prior to this event.

So, we're not really sure what to do here. It doesn't actually seem to be a review bomb in the way we've previously defined them, but maybe that's just our definition being wrong. But even if we define it as one, we're not sure whether it should be off-topic or not. The overall Review Score would decrease by 1.3% if we marked it, which wouldn't have any significant effect on its visibility in the store (see the FAQ below for more on that). So either way, the game itself wouldn't be affected by our decision.

As a result, we've decided we're just going to leave it alone. But hopefully, this post has helped you understand that thinking behind why we've ended up there. If you have any thoughts on how we should approach this case, or similar cases in the future, we'd love to hear from you in the comments below.

FAQ

Q: Why wouldn't AC:Unity's visibility in the Steam Store be affected by the Review Score change if you marked it off-topic?

A: There are a number of places in the Store where we factor in Review Scores when sorting games into lists. Games receive a "boost" based upon which user review bucket they're in (Mixed, Mostly Positive, etc). The actual boost amount is quite small relative to other factors in the Store, and it's essentially the same for all the Mixed or above review buckets. But there's a big boost drop-off as soon as a game drops below Mixed into the negative buckets, which occurs at the point where less than 40% of the user reviews are positive. A Mixed game receives over 500% more boost than a game in Mostly Negative. That might seem scary, but we're still talking about a boost that's small relative to many other store factors, and it's the minority case - 71.7% of titles on the Store are Mixed or above.

In the case of AC:Unity, the positive review spike looks significant in the Recent Reviews view, but in the overall Review Score it only shifted from 59.7% up to 61%, both of which are squarely in the Mixed reviews bucket.

Q: Why were people buying AC:Unity if it was free on Uplay?

A: Any time there's an increase in visibility of a game, we generally see an increase in players and sales. Many of Steam's systems are designed to multiply player interest and activity around a game. Obviously, some players who saw the news might decide to go and buy AC:Unity to explore virtual Paris. But even players who already own AC:Unity may drive further sales, because they may decide to fire up AC:Unity to look at the Notre Dame. Steam will broadcast that activity in the form of toasts, achievements earned, trading cards, and so on, and that increases the visibility of AC:Unity to other players. For example, in the time period since the events in Paris, there's been more than a 500% increase in the number of toasts shown to players telling them that a friend has launched AC:Unity.

You might expect that increase in visibility to have resulted in more players going over and picking it up for free on Uplay. It's possible that these are players who simply didn't know they could get it for free, in spite of the most prominent user review on the store page doing its best to let them know that.

Mar 15, 2019

User Reviews Revisited

Steam News - TomB

Some time ago we made some changes to how we presented the User Reviews for games, and their resulting Review Score. We talked about those changes in this blog post. As we describe in that post, we want to ensure that players who've played a game can voice their opinions about why other people should or shouldn't buy the game, and that our summary of those opinions into a single Review Score should represent the likelihood that a future purchaser will be happy with their purchase.

Since that post, we've continued to listen to feedback from both players and developers. It's clear to us that players value reviews highly, and want us to ensure they're accurate and trustworthy. Developers understand that they're valuable to players, but want to feel like they're being treated fairly. We've also spent a bunch of time building analysis tools to help us better understand what's happening in the reviews across all titles on Steam. With that feedback and data in hand, we think we're ready to make another change.

That change can be described easily: we're going to identify off-topic review bombs, and remove them from the Review Score.

But while easy to say, it raises a bunch of questions, so let's dig into the details. First, what do we mean by an off-topic review bomb? As we defined back in our original post, a review bomb is where players post a large number of reviews in a short period of time, aimed at lowering the Review Score of a game. We define an off-topic review bomb as one where the focus of those reviews is on a topic that we consider unrelated to the likelihood that future purchasers will be happy if they buy the game, and hence not something that should be added to the Review Score.

Obviously, there's a grey area here, because there's a wide range of things that players care about. So how will we identify these off-topic review bombs? The first step is a tool we've built that identifies any anomalous review activity on all games on Steam in as close to real-time as possible. It doesn't know why a given game is receiving anomalous review activity, and it doesn't even try to figure that out. Instead, it notifies a team of people at Valve, who'll then go and investigate. We've already run our tool across the entire history of reviews on Steam, identifying many reasons why games have seen periods of anomalous review activity, and off-topic review bombs appear to only be a small number of them.

Once our team has identified that the anomalous activity is an off-topic review bomb, we'll mark the time period it encompasses and notify the developer. The reviews within that time period will then be removed from the Review Score calculation. As before, the reviews themselves are left untouched - if you want to dig into them to see if they're relevant to you, you'll still be able to do so. To help you do that, we've made it clear when you're looking at a store page where we've removed some reviews by default, and we've further improved the UI around anomalous review periods.

Finally, we've also enabled you to opt out of this entirely, if that's your preference - there's now a checkbox in your Steam Store options where you can choose to have off-topic review bombs still included in all the Review Scores you see.

While we're working on some other features around User Reviews, we thought this one was worth shipping by itself. As always, if you have thoughts or concerns, feel free to voice them in the comments below.

Q&A

Q: I care about some things that I worry other players don't, like DRM or EULA changes. Review bombs have been about them in the past. Do you consider them unrelated or off-topic?

A: We had long debates about these two, and others like them. They're technically not a part of the game, but they are an issue for some players. In the end, we've decided to define them as off-topic review bombs. Our reasoning is that the "general" Steam player doesn't care as much about them, so the Review Score is more accurate if it doesn't contain them. In addition, we believe that players who do care about topics like DRM are often willing to dig a little deeper into games before purchasing - which is why we still keep all the reviews within the review bombs. It only takes a minute to dig into those reviews to see if the issue is something you care about.

Q: So if I post a review inside in the period of an off-topic review bomb, my review won't be included in the Review Score?

A: Unfortunately, this is correct. We've tested our process of identifying off-topic review bombs on the entire history of reviews on Steam, and in doing so, we've found that while we can look through reviews and community discussions to determine what's behind the review bomb, it isn't feasible for us to read every single review. But as we mentioned back in our first User Review post, our data shows us that review bombs tend to be temporary distortions, so we believe the Review Score will still be accurate, and other players will still be able to find and read your review within the period.

Q: Are you deleting reviews?

A: No, we are only changing the way we calculate a game's Review Score. All reviews are left untouched, and if you still want to see the raw Review Score, you're welcome to make that change in your Steam Store Preferences.

Q: Does this mean you'll be going back to identify and remove old off-topic review bombs?

A: Based on internal conversations and consultation with some of the partners that have experienced off-topic review bombs, we decided not to unilaterally grandfather in what's happened in the past.

Q: What about when I don't agree with you about what's an "off-topic review bomb?"

A: We'd suggest setting your Steam Reviews default to "Include reviews from all Steam purchases in Review Scores" in under the "Review Score Settings" in your Steam Store Preferences

Q: Reviews are extremely important to me: they're one of the few tools customers have to react to deception or discovering something they've been sold is sub-par. I guess that really wasn't a question, but I just wanted to say that.

A: We agree. We remain in active conversation with you, the community, about what you want from reviews along with the various partners who sell their games on Steam. Reviews are an important part of Steam, which is why we continue to do the work to make sure that they are not being manipulated by anyone. It's the same reason that we decisively ban partners who engage in review manipulation -- customers need to be able to trust the system for it be valuable.

Dec 28, 2018

Top Controller-Friendly Games of 2018

Steam News - Alden

Yesterday we revealed the top-selling and top-played games of 2018 across all of Steam. In building those year-end lists, we also became curious as to which games were most popular with different kinds of controllers. We know from past analysis, that a lot of players on Steam have used a controller of some kind to play a game (36.7 Million players at last count). And we know that not every game works equally as well with each model of controller, so we wanted to put together some guides to help find the games that you might enjoy most with the particular kind of controller that you have.

With that in mind, we've put together a set of pages to highlight the most popular games being played with each of the most common controller models: Top Games played with Xbox controllers (including a huge range of 3rd party controllers that are very similar), Top games played with PlayStation controllers, Top games played with Steam Controllers, and finally Top Games Played With Nintendo Switch Pro controllers.

Each page is broken down into three sections: