Testing Setup and Frame Rating Explanation
Testing Configuration
While a change to the X99 + Haswell-E platform is coming soon, for this launch review, we are keeping with the standard SNB-E setup, still treating us well.
Test System Setup | |
CPU | Intel Core i7-3960X Sandy Bridge-E |
Motherboard | ASUS P9X79 Deluxe |
Memory | Corsair Dominator DDR3-1600 16GB |
Hard Drive | OCZ Agility 4 256GB SSD |
Sound Card | On-board |
Graphics Card | NVIDIA GeForce GTX 980 4GB EVGA GeForce GTX 970 4GB NVIDIA GeForce GTX 780 Ti 3GB NVIDIA GeForce GTX 780 3GB AMD Radeon R9 290X 4GB AMD Radeon R9 290 4GB |
Graphics Drivers | AMD: 14.7 NVIDIA: 344.07 |
Power Supply | Corsair AX1200i |
Operating System | Windows 8 Pro x64 |
GeForce GTX 980 Reference
EVGA GeForce GTX 970 Reference
What you should be watching for
- GTX 980 vs R9 290X – These flagship monsters will go head to head to see if the GTX 980 can overpower the Hawaii GPU that has been dominant for a while. Does the higher price of the GTX 980 warrant itself?
- GTX 980 vs GTX 780 Ti – Even though the GTX 980 is a Gxx04 part, can it handle besting the last generation GK110 GPU? And by how much?
- GTX 970 vs R9 290 – For $329 the GTX 970 has a lot to offer but can it put the screws to AMD as the price was intended to?
- GTX 970 vs GTX 780 – Finally, can the GTX 970 stand up to the GTX 780 that currently sells for quite a bit more cash?
If you don't need the example graphs and explanations below, you can jump straight to the benchmark results now!!
Frame Rating: Our Testing Process
If you aren't familiar with it, you should probably do a little research into our testing methodology as it is quite different than others you may see online. Rather than using FRAPS to measure frame rates or frame times, we are using an secondary PC to capture the output from the tested graphics card directly and then use post processing on the resulting video to determine frame rates, frame times, frame variance and much more.
This amount of data can be pretty confusing if you attempting to read it without proper background, but I strongly believe that the results we present paint a much more thorough picture of performance than other options. So please, read up on the full discussion about our Frame Rating methods before moving forward!!
While there are literally dozens of file created for each “run” of benchmarks, there are several resulting graphs that FCAT produces, as well as several more that we are generating with additional code of our own.
If you don't need the example graphs and explanations below, you can jump straight to the benchmark results now!!
The PCPER FRAPS File
While the graphs above are produced by the default version of the scripts from NVIDIA, I have modified and added to them in a few ways to produce additional data for our readers. The first file shows a sub-set of the data from the RUN file above, the average frame rate over time as defined by FRAPS, though we are combining all of the GPUs we are comparing into a single graph. This will basically emulate the data we have been showing you for the past several years.
The PCPER Observed FPS File
This graph takes a different subset of data points and plots them similarly to the FRAPS file above, but this time we are look at the “observed” average frame rates, shown previously as the blue bars in the RUN file above. This takes out the dropped and runts frames, giving you the performance metrics that actually matter – how many frames are being shown to the gamer to improve the animation sequences.
As you’ll see in our full results on the coming pages, seeing a big difference between the FRAPS FPS graphic and the Observed FPS will indicate cases where it is likely the gamer is not getting the full benefit of the hardware investment in their PC.
The PLOT File
The primary file that is generated from the extracted data is a plot of calculated frame times including runts. The numbers here represent the amount of time that frames appear on the screen for the user, a “thinner” line across the time span represents frame times that are consistent and thus should produce the smoothest animation to the gamer. A “wider” line or one with a lot of peaks and valleys indicates a lot more variance and is likely caused by a lot of runts being displayed.
The RUN File
While the two graphs above show combined results for a set of cards being compared, the RUN file will show you the results from a single card on that particular result. It is in this graph that you can see interesting data about runts, drops, average frame rate and the actual frame rate of your gaming experience.
For tests that show no runts or drops, the data is pretty clean. This is the standard frame rate per second over a span of time graph that has become the standard for performance evaluation on graphics cards.
A test that does have runts and drops will look much different. The black bar labeled FRAPS indicates the average frame rate over time that traditional testing would show if you counted the drops and runts in the equation – as FRAPS FPS measurement does. Any area in red is a dropped frame – the wider the amount of red you see, the more colored bars from our overlay were missing in the captured video file, indicating the gamer never saw those frames in any form.
The wide yellow area is the representation of runts, the thin bands of color in our captured video, that we have determined do not add to the animation of the image on the screen. The larger the area of yellow the more often those runts are appearing.
Finally, the blue line is the measured FPS over each second after removing the runts and drops. We are going to be calling this metric the “observed frame rate” as it measures the actual speed of the animation that the gamer experiences.
The PERcentile File
Scott introduced the idea of frame time percentiles months ago but now that we have some different data using direct capture as opposed to FRAPS, the results might be even more telling. In this case, FCAT is showing percentiles not by frame time but instead by instantaneous FPS. This will tell you the minimum frame rate that will appear on the screen at any given percent of time during our benchmark run. The 50th percentile should be very close to the average total frame rate of the benchmark but as we creep closer to the 100% we see how the frame rate will be affected.
The closer this line is to being perfectly flat the better as that would mean we are running at a constant frame rate the entire time. A steep decline on the right hand side tells us that frame times are varying more and more frequently and might indicate potential stutter in the animation.
The PCPER Frame Time Variance File
Of all the data we are presenting, this is probably the one that needs the most discussion. In an attempt to create a new metric for gaming and graphics performance, I wanted to try to find a way to define stutter based on the data sets we had collected. As I mentioned earlier, we can define a single stutter as a variance level between t_game and t_display. This variance can be introduced in t_game, t_display, or on both levels. Since we can currently only reliably test the t_display rate, how can we create a definition of stutter that makes sense and that can be applied across multiple games and platforms?
We define a single frame variance as the difference between the current frame time and the previous frame time – how consistent the two frames presented to the gamer. However, as I found in my testing plotting the value of this frame variance is nearly a perfect match to the data presented by the minimum FPS (PER) file created by FCAT. To be more specific, stutter is only perceived when there is a break from the previous animation frame rates.
Our current running theory for a stutter evaluation is this: find the current frame time variance by comparing the current frame time to the running average of the frame times of the previous 20 frames. Then, by sorting these frame times and plotting them in a percentile form we can get an interesting look at potential stutter. Comparing the frame times to a running average rather than just to the previous frame should prevent potential problems from legitimate performance peaks or valleys found when moving from a highly compute intensive scene to a lower one.
While we are still trying to figure out if this is the best way to visualize stutter in a game, we have seen enough evidence in our game play testing and by comparing the above graphic to other data generated through our Frame rating system to be reasonably confident in our assertions. So much in fact that I am going to going this data the PCPER ISU, which beer fans will appreciate the acronym of International Stutter Units.
To compare these results you want to see a line that is as close the 0ms mark as possible indicating very little frame rate variance when compared to a running average of previous frames. There will be some inevitable incline as we reach the 90+ percentile but that is expected with any game play sequence that varies from scene to scene. What we do not want to see is a sharper line up that would indicate higher frame variance (ISU) and could be an indication that the game sees microstuttering and hitching problems.
From the benchmarks it seems
From the benchmarks it seems obvious to me that the game is CPU bottlenecked. The highest FPS is outside of combat. During combat the FPS all drops to 100fps no matter 1440p or 4k. This is my experience as well, from looking at CPU/GPU usage while playing Skyrim. Since it’s limited to 2 cores, basically a highly overclocked 4 core is the way to go. However, with something like ENB, I wonder if that’ll shift enough of the burden towards the GPU to the point where a better GPU solution actually does something.
I am talking about Skyrim,
I am talking about Skyrim, BTW, in my above comment.
Ryan are you planning on
Ryan are you planning on testing multimonitor in the near future?
Below is my question.
For example: I have my 7950 sapphire flex in crossfire. I play eyefinity games at 5920×1440. My 3 screens are 1680x1050x2 (16×10) for the outiside screens. And my u2711 at 2560×1440 (16×9) in the middle. All are dp monitors.
I do this because for the games that do not support multi monitor. I like the bigger 2560×1440 screen.
It would be great to know if Nvidia has updated surround capabilities to match AMD’s!
I’m happy to see nVidia
I’m happy to see nVidia endorse downsampling in the form of a supported feature. I’m curious about the downsampling filter they use though – a 13-tap Gaussian filter should produce a decently sharp image without ringing, but is there any word on whether or not it is gamma-aware? That last detail is important when downsampling and particularly for high-contrast details.
Hi,
I have a request to
Hi,
I have a request to benchmark skyrim with enb. full quality.
real vision option a full quality is a good videocard destroyer!
my system is i7 4820k @ 4.5ghz and a 290X
skyrim at 4k without enb is 45-50fps
enb on full quality is 17-19FPS..
can you setup a skyrim enb benchmark for reference from now on?
im very interested in your benchmarks for skyrim enb with 290x, 780ti, 970gtx and 980gtx
I know its alot of work but please please please! hehehe
ohhh, if you do, please add
ohhh, if you do, please add unoffical hd textures, flora overhaul and that hurts performance even more! makes the game so beautiful to play…
Most of those supplements
Most of those supplements work by stopping the cause
of baldness. Excess consumption of zinc may cause bleeding stomach and severe abdominal pain. There are only 2 St Johns wort products that I know of,
that have had been properly researched and the Flordis Remotiv is one of those.
My site … ev44.pl (Priscilla)
I have been watching your
I have been watching your channel for a long time now. I would like to say that i enjoy the thorough way in which you benchmanrk every card. That being said, i would guess to say that 98% of pc gamers play in 1080p. Im wondering why you test such hi resolutions? Im sure you have 1080p benchmarks on another page. I just feel raked over the coals with GSync and 4K. Im tired of forking over thousands for small increases in performance. This bleeding edge is making my wallet bleed!
I agree with Shaun, the
I agree with Shaun, the realvision ENB would be a great benchmark tester as with the realvision ENB on skyrim. I had average of 15fps in open area outside of white run and the rest. Was average of 20-35fps on a gigabyte r9 290x OC 4gb and I’m not sure if that did but my card eventually broke Or overheating issues but wouldn’t load into windows just a black screen with fans spinning fullspeed after windows load screen after POST. So I RMA’d that card. And got credit refund and bought the MSI GTX 970 4Gb and waiting for it to arrived with also with a new motherboard.
So I think that it would be a great Benchmark as it really pushes the GPU not so much the CPU and Skyrim with mods normally uses upto 4gb of VRAM
Anyway Thanks
Awesome Review, SLI power
Awesome Review, SLI power consumption for dual 980’s is hard to come by and you sir have slayed my doubts about overdrawing a 850W power supply. THANKYOU!!! 😀
Im going to build this
Im going to build this system
I7 4790K
SLI gtx 970s
16gb ram 1600mhz
Is a 630 watt PSU sufficient to run in sli? If yes can i also overclock?
630W? are you using a brand
630W? are you using a brand name PSU?
Dont trust PSU’s that come preinstalled with a case..
I would think 630 would be buggy for SLI..
you want at least 25% to spare.. I’d say at least a 750W..
Tho my Coolermaster ran 2x R9 280’s fine.. but that was me slightly underclocking my cpu so allow that..
just make sure the psu is a quality one.
Where AMD will have problems,
Where AMD will have problems, not so much in pricing, but in the thermals that are required for the mini/micro sized systems for HTPC/Etc. that may not be able to take the AMD SKUs even if the prices are lower, getting as much GPU power into as small a form factor as possible is going to be a much more important market segment, as more of these products are being introduced.
Small portable form factor portable mini desktop systems, linked wirelessly to tablets, and relying on the mini desktop for most of the processing needs, are going to appear, systems that can be easily carried around in a laptop bag, along with a tablet, the tablet acting as the desktop host for direct(Via ad hoc WiFi) remote into the mini desktop PC. these type of systems will be more powerful than a laptop(the Mini PC part of the pair), but just as portable, and plugged in at the coffee houses/ETC. and wirelessly serving games, and compute to one, or more tablets. Fitting More powerful discrete GPUs into these systems that will not overburden the limited cooling available in the Mini/Micro form factor will be a big business, especially for gaming/game streaming on the go, and taking these devices along while traveling, and having a device that can be configured to be more like a laptop when on battery power, but ramp up the power beyond what a laptop is capable of while plugged in.
Can i run gtx970 on my intel
Can i run gtx970 on my intel DH61HO Motherboard??
It’s obvious Ryan you have
It’s obvious Ryan you have taken heaps of time doing this (well done mate), but as someone wanting to build a rig to use on a big TV, I’m holding back until I can get my head around the 4k TV vs PC gaming output thing.
HDMI and 4k is my worry. I’ll be buying a big (thinking 65″) TV, only 4k for the gaming. It’ll do service as a normal TV too, but in Australia it’ll be obsolete before we see 4k content on the air! So that leaves gaming.
Is a big 4k TV a good option for high resolution gaming? Or are there land mines hidden in HDMI 1.x/2.x specs that’ll catch out the unaware? Certainly look better than 3 monitors.
It looks like the 980 will push BF4 to 4k @ ~30fps, but is that enough, or is SLI to get 45-60fps needed to be a pleasure to play?
A pair of 290x SLI watercooled would have to be an option, quiet yet in the running on fps. OK, uses more power but the purchase price difference buys a lot of electricity, unless water cooling costs a bomb!
Why are the specific settings
Why are the specific settings not disclosed?
That’s pretty much benchmarking 101, and things like AA & AO can make a massive difference.
While this is the only place
While this is the only place I have seen that has benchmarked Skyrim in 4k with a 970, and Thank you very much for that! But what settings did you use? You post what settings you used at the top? But did the 970 really pull ~50 fps at 4k with 8x A and ultra? Find that hard to believe. I have a new 4k Samsung and really just want to play Skyrim in vanilla 4k, no need for AA and trying to decide if 970 is enough.