Whenever Valve make a change to their platform, it’s big news. No matter how small it may seem, the size of Steam means it’s had a drastic effect on someone. When they restructure their whole review system, that’s going to be a lot more people. So we reached out to those in the know, developers and publishers who regularly put their games on Steam and whose ratings have been affected by the change, to find out what they thought about it.
For a better user review experience, have a look at BetterCritic.
We got a huge number of responses from a variety of folks, and you can see them laid out below, along with links to their games. Of note is this graph by Eugene Krylov that shows off how much games changed, and his Twitter feed of related information.
Matt Donatelli, Graphite Lab – Hive Jump
“I do agree that Valve’s recent changes to the review system will reduce artificially inflated scores and cut down on abuse, however I am troubled by our backers’ loss of a voice on the platform. Our backers are being treated as non-paying customers, when they have in fact paid in two currencies: money and time.
“Backers are in a unique position to highlight certain aspects of a brand new title in their reviews, such as how communicative the developers have been, as well as the overall strength of the game’s community.
“This information is especially valuable to new players whose purchasing decisions may be influenced by the strength of the community surrounding the game. For these reviews to be hidden and un-scored is doing Steam players, developers, and crowdfunding backers a major disservice.”
Simon Roth, Machine Studios – Maia
“Firstly it’s worth mentioning that I think Steam reviews, if implemented well, are a good idea. Steam really lacks a decent customer feedback and review process, but I don’t think today’s changes really fixed the core issue with the system which is low quality reviews.
“If users were given more direction and guidance in the review process, they would produce better quality reviews that would help players, and also cut out the low effort spam. It would also increase the cost of scamming the system as an in-depth review would take a little time, effort and an actual understanding of the game.
“Reviews should be able to focus on individual aspects of the game too, with their scores not adding to the aggregate. For instance a user might want to talk about performance on Linux, which doesn’t speak for the game as a whole, but might still help others. Giving the customers the ability to search those tags and add their own tags would be very useful too.
“Reviewers should have a small profile that gives their review more context. Have it post their rough system specification for instance, or what genre of games they usually review. Reviews are subjective after all, so knowing about the reviewer may shed more light on why they held an opinion. [Plus], users should be asked if they want to message the dev before writing a review. A lot of reviews read like tech support issues.
“Steam also needs to incentivise players to leave a review. Perhaps once a week/month in the pop-up news window have a generated page allowing them to review their recently played games.
“From a business sense, once you go under 50% your sales drop off hard. Most people will give a middling game a look, but when you get lower they start to ignore you entirely. So you are stuck as you are unable to sell any more. Which if your reviews are thoughtful breakdowns of the failings of a product is fine, but if they are all hilarious jokes or single sentence reviews it’s pretty harmful.
“As for my own case, our reviews have gone down as we spent a lot of time in Early Access. However we were starting to crawl back upwards and this change has disappointingly kicked us down a peg or two. Many of our Kickstarter and direct buyers got in early to support us and are waiting for a beta or more final version to play and review. I was hoping on that bump to aid us for our big upcoming releases, but can no longer count on it. Having low scores will likely hit our sales pretty hard and that puts me in a bit of a corner.”
Andy Hodgetts, The Indie Stone – Project Zomboid
“That said, that’s rather secondary to my personal opinion on the subject. To me, it strikes me as odd that it was ever possible to leave a review on a store which you did not use to purchase the game. The purpose of Steam reviews is for Steam *customers* to leave reviews for other Steam customers. If you bought the game elsewhere and redeemed the key on Steam, then leave your review on the store you purchased it from.
“I do sympathise with developers who, perhaps, have a Kickstarter project and were hoping to benefit from the reviews of those backers to help push the visibility of their game – your visibility on Steam is tied, in part, to the number and rating of user-reviews – purely based on my own observations, it affects the order and visibility in the real-time results in the search box [for example].
“At the same time, we ourselves did benefit from this very thing when we first launched Zomboid on Early Access – our Desura customers who redeemed a Steam Key were able to leave reviews. But looking at the numbers as they are now, of 12,000 or so reviews only 200 or so were from key activations (i.e. Desura or direct sale customers) so, ultimately, losing those from the overall score makes almost no difference to the resulting value. The same appears to be true of almost all comparable games, including much, much larger triple-A games.
“It’s the much smaller games which will be affected more heavily, of course. But while I do sympathise with developers in that position, I do believe it makes sense that the Steam Store uses Steam purchases for its metric of popularity. And the same goes for all other storefronts. If you buy from GOG, review on GOG. Buy from Amazon, review on Amazon. If you buy a shop copy… well, you’re probably out of luck finding an equivalent place to leave a review. But that a game store doesn’t have a user-review section doesn’t mean that Steam should become the place to leave it just because you’ve redeemed your copy on that service. For customers, by customers. Valve, to me, are simply trying to do the best for their customers and have compromised in allowing non-Steam bought copies to still leave a review, just not one which contributes to the score.”
Trent Oster, Beamdog – Baldur’s Gate: Siege of Dragonspear
“I’m still digging into what was changed, but from the ten thousand foot experience it seems to make sense. If someone is gaming the system, make the system harder to game. The entire Steam key system has spawned an industry of bundle deals and other businesses, so it will be interesting to see how this change affects the key market as from a review standpoint, keys were just downgraded in importance.
“From our perspective this doesn’t make much difference. We’ve sold some keys for our games through various partners and it has usually been a positive experience. I’m not sure the value people assign to having the ability to create a review on Steam will really influence any buying decisions.”
Tom Francis, Suspicious Developments – Gunpoint
“I do feel bad for devs negatively affected by this change without warning. But I also think the new rule is not unreasonable. It’s pretty extraordinary that Valve give us limitless free keys we can sell for pure profit and give them nothing – if they need to tweak how their site views those sales to prevent it being exploited, I still feel like I’m getting a good deal.”
Alexis Kennedy, Failbetter & Freelance – Sunless Sea
“The key thing to my mind is that this won’t be the final situation. Valve like to make big sweeping simple changes based on data, not people – it’s a technocratic, programmer’s mindset.
“But they always iterate on those changes. An effective review system is essential for the store’s health, so that’s always their priority. But they know perfectly well that Steam has competitors, and they always have an eye on the long term. So I imagine they’re throwing the bathwater out, fully intending to pick up the baby later when they’ve heard how loudly it squeaks. What we as devs need to do is squeak loudly, firmly and not too apocalyptically. They do care about indies, but only in aggregate.
“It’s hurting community-driven indies right now, and my heart goes out to the people who’ve seen their scores lurch downwards, but it’s what happens when you have a market hegemon, even a relatively benign one. And in the long term building a community will still pay off.”
Paul Kilduff-Taylor, Mode 7 – Frozen Synapse 2
“Valve give developers a massive amount of flexibility in terms of registering and using keys on Steam: it’s really important to us that this doesn’t change. So, a measure like this which is intended to prevent abuse of that system is definitely fundamentally welcome.
“The type of fraud which this is intended to counteract is really awful: it’s misleading to customers and it makes the situation worse for everyone; I’ll be glad to see it gone and I can see why Valve felt the need to make a big move against it.
“I do think this new system could cause an issue for devs who want to release an early version of their game on their own site, then move to Steam later on. Early adopters tend to be quite vocal fans; you don’t want them to lose influence. It’s a big deal that their reviews will still be posted but it’s a shame nonetheless. I’d like to see some way for those customers to have an equal say in a game’s overall user review score in future, but equally I can understand why that may just not be technically possible.
“As I’ve said elsewhere, I think the most important thing is that developers focus on making games which can appeal to players outside that initial rush of alpha enthusiasm. If other devs feel this change has had a negative impact on them, then I’m certainly not diminishing that: everyone’s experience varies massively. However, if a game has a lasting impact and it can continue to draw in new players after launch, then I very much doubt this change will affect its prospects.”
Jay Tholen – Dropsy
“I’m not sure how much I like it in a broader sense, but it has definitely helped Dropsy out, percentage wise. We’d always get a couple of negative reviews after bundles from folks who tried it on a whim who had evidently never played a Point and Click. I don’t mind limiting the weight of reviews from people who would never have, by choice, purchased my game. Conversely, I’d not want to reduce the prominence of our Kickstarter Backers’ reviews, as they’re likely the ones with the most intimate knowledge of the game.”
Rami Ismail, Vlambeer – Nuclear Throne
“Ultimately, I think the decision is sound. The review system is one of the most powerful ways to promote a game within the ecosystem, and that makes it a popular target for abuse.
“There are drawbacks that are rather serious. Giveaways and bundles are a good way to increase buzz, and while cheap copies of the game tend to generate more negative reviews, they’re still a way to increase the buzz for the game. Similarly, there’s been a movement towards custom Early Access programs controlled by the developer rather than Steam – those movements are now less useful as they can’t generate reviews.
“In the end, the quality of reviews is important to ensure consumers know what they buy, and if this system helps curtailing low quality or even fraudulent reviews, then it’s a good move as far as I’m concerned.
Andrew Smith, Spilt Milk Studio – Tango Fiesta
“Valve’s changes are very heavy handed and – by their own admission – only address a small subset of the problems facing the user review system. I don’t expect to see our sales affected all that much, and while it does undercut a part of the value nurturing a community outside of Steam, I’ve yet to see much hard data showing a correlation between user reviews and sales (at least at a scale whereby this change will have any wide reaching effects).
“More concerning to me is the speed with which Valve usually react in these (constantly shifting) situations – they take so much time to address any issues that they’ve inevitably allowed room for new problems to grow. Piecemeal firefighting is not the way they will make Steam a greater place for games, gamers & devs. I’m hopeful they’ll introduce something more meaningful with their next update, whenever that may be.
Gary Chambers, Introversion – Prison Architect
“To me it seems like a solution to a problem that no one was having. They said it was about 160 games that had “suspicious” reviews on them, out of what must be over 6000 at this point. And they are clearly able to detect when this is happening, so if they were so concerned about it, with such low numbers they could easily just review it manually every now and then.
“And the impact this has on a number of developers is pretty severe – I’ve seen quite a few people today saying that their average reviews have dropped significantly as a result of this, primarily those who had kickstarters or did well in things like the Humble Bundle. The reality is that Steam reviews do have a noticeable impact on a game’s sales, and having reviews cut like that is a pretty big blow.
“We’re pretty lucky with Prison Architect in that the vast majority of our reviews are from people who bought through Steam, so this hasn’t had much noticeable impact on its average, but there are plenty of games not in that position. Hopefully they change their minds on this and restore all of these lost reviews.”
Brian Hicks, Bohemia – DayZ
“The changes to Steam’s user reviews are an example of a repeat issue with Steam as a platform. Frequently Valve fails to properly discuss or inform developers of large scale changes that are made to both store pages, and the store as a whole. Going back to the implementation of user reviews, and culminating in issues such as refunds, and workshop mod monetization. However, I honestly don’t see this behaviour pattern changing any time in the future. This leaves us as developers forced to rapidly adapt to changes that directly impact our success on the platform.
“The new changes to reviews specifically hurt developers who sell through more than one portal than Steam. For example, DayZ can be purchased on the Bohemia Interactive Store in addition to Steam. This will provide you with a Steam key that you can activate, gift, do whatever you like. Sometimes, in order to drive attention to our own store we might run a sale over there – while I’m not at liberty to disclose the exact key sales difference between the two store fronts, I can say the amount of keys sold through this portal is most definitely not insignificant.
“However, that said – our concern is not that greatly focused on Steam reviews while we are in Early Access. We fully recognize that many users can and probably will be frustrated with the full length development process that comes with our engine, and game development being part of that Early Access period. Those that are, and should be concerned are indie titles that do not benefit from the large amount of sales, and user reviews that we on DayZ benefit from. Titles such as Maia, The Long Dark, Miscreated, or even Wasteland 2 – or really anyone who has participated in programs such as the Humble Bundle or similar options can and probably will end up seeing a massive hit to their reviews – something smaller titles without the benefit of large brand awareness like DayZ has will suffer because of.
“So, we find ourselves here again at Valve’s mercy as indie developers. Having to adapt with little, or almost no notice to systems that can greatly impact our marketplace presence and consumer perception. Don’t get me wrong, Steam offers a vast amount of major benefits that allow indie developers to toss aside the traditional evil publisher model – but at what point does this start to become enough of an issue that having that safety blanket of an established, and powerful publisher starts becoming more appealing?”
Debbie Bestwick, Team 17 – Worms WMD
We are pleased to see Steam taking measures to improve their review system and address some of the validity issues surrounding user reviews being obtained disingenuously. However, excluding genuine Steam customers via crowdfunding and other stores from being able to provide a score is something we’d like to see being considered.
There are many indie teams we support via our label who have run successful crowdfunding campaigns and earned a core fan base eager to give their opinion on the game in a meaningful way. In an ideal scenario there would be a good balance struck between the current system and allowing those customer reviews to have an impact on the games percentage score.
So, what do you think? Let us know in the comments below.