VR-Zone tossed the bees nest in a paint shaker and received a fairly sedate outcome.
A little background information is required. AnTuTu, a mobile benchmark developed by AnTuTu Labs, has been accused of inaccurate scores and bias towards specific hardware. Leaked BayTrail-T benchmarks, surpassing our expectations of Intel's capabilities, were harshly refuted based on AnTuTu's credibility. More recently, certain Samsung GPUs have been allegedly recorded self-overclocking during that benchmark but not elsewhere.
Scene from Cloud Gate, latest Windows 3DMark.
Oliver Baltuch, president of Futuremark, accepted an interview with VR-Zone to discuss business and ethics in their marketplace. Futuremark is a direct competitor to AnTuTu and a household name in the benchmarking community. Being modest Fins, self-proclaimed, they did not wish to discuss whether AnTuTu was less honest than they are. Futuremark does disagree with AnTuTu's process, however, and has some suggestions for better results.
The design process for 3D Mark Android begun with 25 pages of specification proposal. Each vendor is given a chance to reply to that proposal and these responses are compared. Changes to the specification must be reviewed by a committee sitting between the financial department and the engineering department.
Baltuch made the point that all of their finances for the last five years, according to Finnish law, can be reviewed for about $7 USD. Despite being a private company, the law mandates no deals can be made in secret.
On the engineering side of things, drivers are approved only if they follow specific guidelines. Unapproved results will be removed from their website and leaderboards followed by a polite conversation with the manufacturer. Drivers are not allowed to identify their benchmarks intent on modifying settings due to that information.
Almost every benchmark they release gets negative responses from some upset vendor or vendors.
The relatively short interview is wrapped up with commentary on iOS benchmarks. Futuremark is nearing completion of their first benchmarking app. Apple disallows apps to exceed 60 frames per second, through vsync, which unnecessarily hinders benchmark scores. Working around this, Futuremark developed a method to render frames which are not displayed on screen to keep the processors from idling once at frame rate cap.
Ryan must love that idea…
This concept has, according to the interview, reached internal QA review and is expected to be released in a few weeks.
Futuremark develops benchmarks for x86 Windows, Windows RT, Android, and iOS. Scores are intended to scale linearly to their metrics and are designed to allow cross-platform performance comparisons.