A change is coming in 2013
In this short article and video we start the discussion about a new graphics performance metric coming in 2013.
If the new year will bring us anything, it looks like it might be the end of using "FPS" as the primary measuring tool for graphics performance on PCs. A long, long time ago we started with simple "time demos" that recorded rendered frames in a game like Quake and then played them back as quickly as possible on a test system. The lone result was given as time, in seconds, and was then converted to an average frame rate having known the total number of frames recorded to start with.
More recently we saw a transition to frame rates over time and the advent frame time graphs like the ones we have been using in our graphics reviews on PC Perspective. This expanded the amount of data required to get an accurate picture of graphics and gaming performance but it was indeed more accurate, giving us a more clear image of how GPUs (and CPUs and systems for that matter) performed in games.
And even though the idea of frame times have been around just a long, not many people were interested in getting into that detail level until this past year. A frame time is the amount of time each frame takes to render, usually listed in milliseconds, and could range from 5ms to 50ms depending on performance. For a reference, 120 FPS equates to an average of 8.3ms, 60 FPS is 16.6ms and 30 FPS is 33.3ms. But rather than average those out by each second of time, what if you looked at each frame individually?
Scott over at Tech Report started doing that this past year and found some interesting results. I encourage all of our readers to follow up on what he has been doing as I think you'll find it incredibly educational and interesting.
Through emails and tweets many PC Perspective readers have been asking for our take on it, why we weren't testing graphics cards in the same fashion yet, etc. I've stayed quiet about it simply because we were working on quite a few different angles on our side and I wasn't ready to share results. I am still not ready to share the glut of our information yet but I am ready to start the discussion and I hope our community find its compelling and offers some feedback.
At the heart of our unique GPU testing method is this card, a high-end dual-link DVI capture card capable of handling 2560×1600 resolutions at 60 Hz. Essentially this card will act as a monitor to our GPU test bed and allow us to capture the actual display output that reaches the gamer's eyes. This method is the best possible way to measure frame rates, frame times, stutter, runts, smoothness, and any other graphics-related metrics.
Using that recorded footage, sometimes reaching 400 MB/s of consistent writes at high resolutions, we can then analyze the frames one by one, though with the help of some additional software. There are a lot of details that I am glossing over including the need for perfectly synced frame rates, having absolutely zero dropped frames in the recording, analyzing, etc, but trust me when I say we have been spending a lot of time on this.
The result are multi-GB files (60 seconds of game play produces a 16GB file) that have each frame presented to the monitor of the gamer in perfect fashion.
The second image here was made in Photoshop to show you the three different "frames" of Unigine that make up the single frame that the gamer would actually see. With a 60 Hz display, this equates to three frames being shown in 16ms, though each frame has a variable amount of real estate on the screen.
We are still finalizing what we can do with all of this data, but it definitely allows us to find some unique cases:
This shot from Sleeping Dogs shows three frames on our single 16ms instance to the monitor, but notice how little screen spaces the green frame takes up – is measuring this frame even useful for gamers? Should it be dropped from the performance metrics all together?
I put together this short video below with a bit more on my thoughts on this topic but I can assure you that January and February are going to bring some major change to the way graphics cards are tested. Please, we want your thoughts! This is an important dialogue for not only us but for all PC gamers going forward. Scott has done some great work "Inside the Second" and we hope that we can offer additional information and clarification on this problem. Rest assured we have not been sitting on our hands here!
Is this kind of testing methodology going to be useful? Is it overkill? Leave us your thoughts below!