A few days ago, Valve changed how user reviews work on Steam. Now, user reviews on the search page and at the top of the product page will only reflect customers who purchased the game from Steam. Other user reviews will still be collected, but only from the product's reviews panel with a more broad filter applied, which must be done manually.

This change was made because Valve detected some titles where review scores varied greatly between Steam user and outside keys. If the vast majority of reviewers who purchased the content on Steam and the vast majority of reviewers who acquired the game outside of Steam are the same, then random error converges quickly. An average of 1000 reviews should be within 3% of the average opinion of 1,000,000 random customers, for 95% of titles. 99% of titles would be within 4% of the average opinion, given 1000 reviews for a million customers.

Of course, the differences are not always truly random. Keys which were given to crowd-funding backers could be abnormally good, if it well-served the niche audience that helped it get made, or abnormally bad, if it slighted that audience.

In the worst case, developers could be giving away keys to services that flood fraudulent reviews.

As such, Valve took the position that it will (Update: Yeah, I kind-of messed up the grammar on this sentence when I restructured it in editing… read it without the strikethrough, and this update of course) only reviews from their direct customers would be promoted. This upset many developers, although some games received a bump in score, if you trust Steam Spy. Again, if their title was a hit on Kickstarter, Patreon, or other services, then it subtracts their most evangelical users.

On the other hand, from Valve's perspective, they want to promote the opinion that best applies to someone browsing on Steam. This makes sense, since a review should be intended to guide someone who doesn't already have an opinion of the title one way or the other. Again, reviews are designed to be the general consensus of a random group of people — the expected value of an average user — but constrainted to a certain set of properties.

Of course, it would be beneficial to Valve to run further experiments to make sure that an average Steam reviewer reflects an average Steam customer for each, specific title. Basically, it's a good hypothesis, but testing isn't done. It could change greatly as it evolves through the Scientific Method.