Study finds people are more likely to leave bad reviews on Steam

steam header

An empirical study of Steam reviews has been published that suggests that people are more likely to leave quick negative reviews on Steam than positive ones, and are more likely to criticise game design than bugs.

A brief history of how Steam review bombing hurts games developers.

The study, which you can download in full here, is by Queen’s University analysts Dayi Lin, Cor-Paul Bezemer, Ying Zou and Ahmed Hassan, with the intent of comparing Steam reviews to those for mobile apps and games.

They took a snapshot of the 8,025 games on Steam as of March 2016, cut that down to 6,224 games by removing titles that had less than 25 reviews, which left them with 10,954,956 reviews across all supported natural languages. Using a mix of filters and analytical crawlers they cut this down to 28,159 reviews with an accurate number of playing hours. They also used Steam Spy, so it’s lucky the data was collected before it was shut down.

Their study shows that the average Steam review is around 30 words long. Early access and indie games receive longer reviews, and free-to-play titles have shorter reviews than paid ones. 96% of the games tested received less than ten reviews per day. Indie games receive more suggestions in reviews, which the team proposes is because “the player community of indie games is more engaged than the community of non-indie games”.

Only 42% of reviews studied contain “valuable” feedback, meaning specific pros, cons, or bug reports. Both positive and negative reviews can contain such feedback, and while negative comments contain useful criticism the team stress that developers “should also not ignore the potentially useful information in positive reviews” too.

The study highlights – quite literally, it’s in bold on page 23 – that “players complain more about game design than bugs”. Only 8% of reviews surveyed mention bugs, but 34% talk about problems related to game design – rising to 57% when just considering negative feedback. The analysts therefore suggest that “players value well-designed gameplay over software quality”. Even more interestingly, it’s more likely to be positive reviews that reference bugs, at 42% compared to just 17% of negative reviews. Negative reviews will be more likely to focus on game-breaking bugs or crashing, whereas positive reviews will comment on lag or audio glitches.

It also shows that on average a gamer will play 13.5 hours before posting a review, but free-to-play games may only get an hour of play – with “the majority of negative reviews” posted within seven hours of gameplay. The study notes that negative reviewers with longer play times are generally satisfied with the game itself, but a toxic community, poor quality of updates, or DLC pricing may sour their experience.

Returning to the original focus, the study suggests that “game reviews are different from mobile app reviews” in a number of aspects. While Steam titles receive on average two reviews per day compared to 22 for apps, the former receive longer reviews. They also note that a “higher percentage of [Steam] reviews contain useful information for a developer than mobile apps”. They conclude that “prior work on mobile app reviews needs to be revisited”.