Study finds people are more likely to leave bad reviews on Steam | PCGamesN

Study finds people are more likely to leave bad reviews on Steam

steam header

An empirical study of Steam reviews has been published that suggests that people are more likely to leave quick negative reviews on Steam than positive ones, and are more likely to criticise game design than bugs.

A brief history of how Steam review bombing hurts games developers.

The study, which you can download in full here, is by Queen’s University analysts Dayi Lin, Cor-Paul Bezemer, Ying Zou and Ahmed Hassan, with the intent of comparing Steam reviews to those for mobile apps and games. 

They took a snapshot of the 8,025 games on Steam as of March 2016, cut that down to 6,224 games by removing titles that had less than 25 reviews, which left them with 10,954,956 reviews across all supported natural languages. Using a mix of filters and analytical crawlers they cut this down to 28,159 reviews with an accurate number of playing hours. They also used Steam Spy, so it’s lucky the data was collected before it was shut down

Their study shows that the average Steam review is around 30 words long. Early access and indie games receive longer reviews, and free-to-play titles have shorter reviews than paid ones. 96% of the games tested received less than ten reviews per day. Indie games receive more suggestions in reviews, which the team proposes is because “the player community of indie games is more engaged than the community of non-indie games”. 

Only 42% of reviews studied contain “valuable” feedback, meaning specific pros, cons, or bug reports. Both positive and negative reviews can contain such feedback, and while negative comments contain useful criticism the team stress that developers “should also not ignore the potentially useful information in positive reviews” too. 

The study highlights - quite literally, it’s in bold on page 23 - that “players complain more about game design than bugs”. Only 8% of reviews surveyed mention bugs, but 34% talk about problems related to game design - rising to 57% when just considering negative feedback. The analysts therefore suggest that “players value well-designed gameplay over software quality”. Even more interestingly, it’s more likely to be positive reviews that reference bugs, at 42% compared to just 17% of negative reviews. Negative reviews will be more likely to focus on game-breaking bugs or crashing, whereas positive reviews will comment on lag or audio glitches. 

It also shows that on average a gamer will play 13.5 hours before posting a review, but free-to-play games may only get an hour of play - with “the majority of negative reviews” posted within seven hours of gameplay. The study notes that negative reviewers with longer play times are generally satisfied with the game itself, but a toxic community, poor quality of updates, or DLC pricing may sour their experience. 

Returning to the original focus, the study suggests that “game reviews are different from mobile app reviews” in a number of aspects. While Steam titles receive on average two reviews per day compared to 22 for apps, the former receive longer reviews. They also note that a “higher percentage of [Steam] reviews contain useful information for a developer than mobile apps”. They conclude that “prior work on mobile app reviews needs to be revisited”.

Sign in to Commentlogin to comment
The DJ avatarWhiteCrow avatar
The DJ Avatar
2 Months ago

I feel like they can really have a way to actually have reviews that could work. First, have a certain amount of hours played in a game instead of 0.1 hours played. That means that they can't say the game was bad because they didn't find in fun. Second, they could also have something where instead of saying it's bad, they could instead have something where the pros and cons sections where people can actually give feedback instead of dissing it because it might be bad. And no more "negative" reviews, but instead some improvements from the community since they are giving negative feedback to people. I would like to see comments that were helpful play apart of the game development because I'm pretty sure if one person is complaining about a bug, many other people have experienced it.

WhiteCrow Avatar
2 Months ago

I agree. The current system fosters this behavior, though not outright. With a few more options available to the reviewer and different wording I think it could have a powerful effect.