What to look for and our Test Setup
We take another trip down the road of Frame Rating with the GeForce GTX 660 and the Radeon HD 7870 GHz Edition.
Because of the complexity and sheer amount of data we have gathered using our Frame Rating performance methodology, we are breaking it up into several articles that each feature different GPU comparisons. Here is the schedule:
- 3/27: Frame Rating Dissected: Full Details on Capture-based Graphics Performance Testing
- 3/27: Radeon HD 7970 GHz Edition vs GeForce GTX 680 (Single and Dual GPU)
- 3/30: AMD Radeon HD 7990 vs GeForce GTX 690 vs GeForce GTX Titan
- 4/2: Radeon HD 7950 vs GeForce GTX 660 Ti (Single and Dual GPU)
- 4/5: Radeon HD 7870 GHz Edition vs GeForce GTX 660 (Single and Dual GPU)
- 4/16: Frame Rating: Visual Effects of Vsync on Gaming Animation
Today marks the conclusion of our first complete round up of Frame Rating results, the culmination of testing that was started 18 months ago. Hopefully you have caught our other articles on the subject at hand, and you really will need to read up on the Frame Rating Dissected story above to truly understand the testing methods and results shown in this article. Use the links above to find the previous articles!
To round out our Frame Rating testing in this interation, we are looking at more cards further down the product stack in two different sets. The first comparison will look at the AMD Radeon HD 7870 GHz Edition and the NVIDIA GeForce GTX 660 graphics cards in both single and dual-card configurations. Just like we saw with our HD 7970 vs GTX 680 and our HD 7950 vs GTX 660 Ti testing, evaluating how the GPUs compare in our new and improved testing methodology in single GPU configurations is just as important as testing in SLI and CrossFire. The GTX 660 ($199 at Newegg.com) and the HD 7870 ($229 at Newegg.com) are the closest matches in terms of pricing though both card have some interesting game bundle options as well.
AMD's Radeon HD 7870 GHz Edition
Our second set of results will only be looking at single GPU performance numbers for lower cost graphics cards like the AMD Radeon HD 7850 and Radeon HD 7790 and from NVIDIA the GeForce GTX 650 Ti and GTX 650 Ti BOOST. We didn't include multi-GPU results on these cards simply due to time constraints internally and because we are eager to move onto further Frame Rating testing and input testing.
NVIDIA's GeForce GTX 660
If you are just joining this article series today, you have missed a lot! If nothing else you should read our initial full release article that details everything about the Frame Rating methodology and why we are making this change to begin with. In short,
Because FRAPS measures frame times at a different point in the game pipeline (closer to the game engine) its results can vary dramatically from what is presented to the end user on their display. Frame Rating solves that problem by recording video through a dual-link DVI capture card that emulates a monitor to the testing system and by simply applying a unique overlay color on each produced frame from the game, we can gather a new kind of information that tells a very unique story.
The capture card that makes all of this work possible.
I don't want to spend too much time on this part of the story here as I already wrote a solid 16,000 words on the topic in our first article and I think you'll really find the results fascinating. So, please check out my first article on the topic if you have any questions before diving into these results today!
| Test System Setup | |
| CPU | Intel Core i7-3960X Sandy Bridge-E |
| Motherboard | ASUS P9X79 Deluxe |
| Memory | Corsair Dominator DDR3-1600 16GB |
| Hard Drive | OCZ Agility 4 256GB SSD |
| Sound Card | On-board |
| Graphics Card |
NVIDIA GeForce GTX 660 2GB AMD Radeon HD 7870 2GB NVIDIA GeForce GTX 650 Ti 1GB NVIDIA GeForce GTX 650 Ti BOOST 2GB AMD Radeon HD 7850 2GB AMD Radeon HD 7790 1GB |
| Graphics Drivers |
AMD: 13.2 beta 7 NVIDIA: 314.07 beta |
| Power Supply | Corsair AX1200i |
| Operating System | Windows 8 Pro x64 |
On to the results!
Great work! I guess you’re as
Great work! I guess you’re as happy as your readers to get to the end of this generation cards.
It would be nice, in order to facilitate readers choice, to have a summary page with all cards presents (maybe with dynamic selection).
What I image would be best is the graphs, where the reader selects a Game, a Screen Resolution, and gets all the cards on the same graphs (with tick-boxes to removes the cards he isn’t interested in).
Graph= ƒ(Game, Screen resolution, GPU selection)
That way a reader looking for the best performance for his screen and favourite game would have all the information needed on a single graph. (you could be motivated to create another graph à-la TechReport taking into account the price)
Anyway Thanks again for the good work!
thumbs up for this
thumbs up for this suggestion!
thats what here in brazil they call “info-graphics” (redundant i know by you get the idea).
Great work Ryan!
Great work Ryan!
Thanks for a another great
Thanks for a another great article. As a few others have pointed out it would be great to see older cards tested such as the 8800GTX, HD 4870 and up. I realize that this would be a huge amount of work but not everyone is a graphics whore like me that upgrades almost every cycle.
Ah, the good ole’ 8800GTX. My
Ah, the good ole' 8800GTX. My little brother is running one of those still and when I finally upgrade he'll get my psuedo-6970 (unlocked xfx 6950). Unfortunately he missed out on my 3850 upgrade when it died on me and would no longer output anything to the display. It plays Kerbal Space Program, Minecraft, and most other games (with lots of stuff turned out of course) so yup, I would like to see Frame Rating through the several generations of graphics cards–even if it's only one card from each NV and AMD's generation and not their whole lineups (heh, we'd never see or hear from Ryan or Ken for 6 months if they had to Frame Rate all those cards :P).
I still have a HD 3850, 2x
I still have a HD 3850, 2x 8800 GTX, HD 4870×2, 2x HD 6870s and 2x GTX 570s SLI in various PCs of my wife and kids . My current gaming PC is 2x GTX 680s SLI. I told you I have a sickness. 🙂
At one point I ran a 4870×2+4870 in Trifire and it didn’t seem as bad as the 6870s in Crossfire if I remember correctly, so it would be interesting to see what impact Trifire has on Observed Frame Rates.
Your sickness gives you the
Your sickness gives you the power of intuition:
http://www.tomshardware.com/reviews/radeon-geforce-stutter-crossfire,2995-6.html
AMD fans: “see? you just need to buy a third card! im sure is cheaper than 2 nvidias”
Unfortunately only 1 card was
Unfortunately only 1 card was tested, so conclusions are hard to be definite. What might have fixed it may be what fixes 2-way Crossfire, a CPU bottleneck. When you add more cards, the GPU puts more stress on the CPU, and if it couldn’t keep up, it will bottleneck, which also has shown to fix stuttering.
More testing is needed. Same with V-sync. Only 2 games were used on the v-sync test. It may be that v-sync only helps due to its FPS limiting component, though I have a feeling it would help anyways, but more testing is needed to be sure.
I meant that only 1 game was
I meant that only 1 game was tested, not one card.
Maybe its time for you elite
Maybe its time for you elite of reviews (pcper, techreport, toms, anand) get together to work on the data generation in gpu reviews.
The value of each reviewer is what he points for us to see more closely and his conclusions.
Toms for instance is saying “just turn on vsync and everything would be ok”.
PCPER refuted that.
Thats why god gave us multi-tab browsing.
Unfortunately different
Unfortunately different people have different preferences. You typically find competitive gamers refuse to use v-sync, because it induces latency and you lose the advantage of high FPS.
And curiously enough, v-sync doesn’t work with Eyefinity.
@Ryan:
I’m having a hard time
@Ryan:
I’m having a hard time finding what you are defining a runt frame. THG has it set to under 21 rows, which they said was the default, are you using the same definition, or something else?
Our versions of the scripts
Our versions of the scripts define it based on percentages. If the frame is 20% or less of the preceding 10 frames average, then we consider it a "runt".
Cool, that is actually a cool
Cool, that is actually a cool method. The higher the FPS, the smaller the frames will be shown, so having it scale based on the average is good.
With 1080 horizontal “scan
With 1080 horizontal “scan lines” on the average screen, calling 21 scan lines a frame is a joke really.
25% of one full screen would be 270 scan lines. Calling a quarter of a screen a “frame” is stretching it quite a lot.
Maybe one fifth the screen is a reasonable cutoff point, or one fourth. Anything less is a joke really, as once 5 “frames” are “sharing” one screen, we see slivered runts in all the examples given.
So I expect some reason for the answer, but I suspect we’ll get ridiculous things like 7 or 8 scan lines out of over 1,000 on the screen “counts” as “one frame” or some tiny portion of the whole screen, far less than 20%, less than 10%, less than 5%, as 7 or 8 scan lines is less than 1% !
I can do the math, and do the visuals. Good luck arguing me down to 10% of a 1920×1080, or “108 scan lines” counting as one FPS.
I can’t tell if you are
I can’t tell if you are arguing or agreeing with my thought.
I will add that as your FPS go higher, the smaller the slice of a frame will be on average. At 300 FPS, the average frame size will be about 20% of the screen. This is why I like what they did, as it scales based on the average frame size.
Amazing timing! Needed
Amazing timing! Needed exactly this.
Good work!
Good work!
I might have felt this
I might have felt this dragged on a bit too long but I was wrong. It was important to do the tests across the whole line up of current cards. I was mainly concerned that you would end the investigation here, after doing so much work on this, but now that you have stated it is on going I am very impressed. Anyway just thought I’d say good work. Really good work. Very glad to hear that you will be looking at user submitted solutions.
It will be interesting to see if peoples claims about radeonpro etc are true. I have seen people post pretty impressive results in terms of frame latency using these solutions. I have also been promoting this solution going off other peoples reports. Hopefully there aren’t too many costs associated with using it (other than setting up individual game profiles, and making sure you do it right..)
I agree with other users in that it would also be great to check say 6950s or 6970s in crossfire. AMD has suggested things like the GCNs memory manager needs work. AMD haven’t specifically suggested that this is where crossfire is falling down and I don’t think it’s related. However, that isn’t stopping people around the internet claiming it is the cause though. So it would be interesting to have a single non-GCN card thrown in, even if its only tested in two or three games, just to confirm it is AMD’s whole crossfire solution rather than their GCN products and its memory management.
Good work nVidia in the last couple of years for making an SLI solution that works out of the box so well.
Good work to AMD for producing cards that are performing so well in the single card tests. Feeling pretty happy about my 7870 tahiti LE purchase.
Believe me when I say I
Believe me when I say I understand that asking for more after all this feels terrible, and am greatful for the work on these articles but I do wish Ryan would test Tahiti LE…as well as show some ‘typical’ overclocking now that stock comparisons have been made.
It’s very simple to argue LE is one of the most relevant cards for the largest market because of it’s price/performance and absolute (overclocked) performance at 1080p, even if it’s ‘limited edition’/highly salvaged (which are usually awkward niche parts) and certainly soon to be replaced by Hainan…which will undoubtedly have a slightly better stock placement relative to 660ti. This is similar to the way ‘7870’ became 7950 in performance through Tahiti LE, 7950B became the old 7970, and 7970GHZ ED was to have a stock config to beat the nvidia competition which often equaled or beat those original configs). Am I the only one telegraphing that coming stock above 660ti (which will drop to 250) with partner overclocks (like 7790 has with 1075/6400mhz) very similar to a stock 670 (but probably priced cheaper? Think 250, 270, 300, 330 (or something like that).
I would say I hope we get a 670/Tahiti LE update article, but I feel 670 is best left for when Hainan launches…and wonder if that is Ryan’s thinking as well considering the deliberate choice to leave it out. I do wish both would’ve been thrown in the 660ti/7950B article, but understand his reasoning. 670 is in it’s own price niche, just like 7870XT. I know the later may be a ‘tweener, but it really would be interesting to see the results as I think it will show a nice point of reference/cutoff where most people should be looking to get smooth 1080p HQ game-play.
Personally I think there is
Personally I think there is enough data to extrapolate how the gtx670 and tahiti LE will perform.
More epic data !
More epic data !
I am interested to see how
I am interested to see how crossfire/sli are effected by slower processors. I know that there is no end to the types of configurations out there but I wonder if causing a bottleneck elsewhere in the system will cause either adverse or positive results.
I also know that originally the setup used was to show the performance of the cards unrestricted, to which I say is a good thing when it comes to comparison from card to card, but now I find that adding in the cpu (crossfire/sli) may generate different results, unless you feel that the cpu still doesn’t cause any “real” problems as far as perceived framerate. ex. the bulldozer review.
Shocking results from AMD.
Shocking results from AMD. These results almost completely invalidate any price savings that might be had by going crossfire (eg CF 7970’s vs SLI 680’s). I’m definitely going green for my next multi card rig and won’t be giving AMD a look until they pull their thumb out and fix crossfire.
Very comprehensive look at
Very comprehensive look at frame rating, excellent write up.
Really interested to see what some of software solutions previously suggested about will effect this problem.
A great work!!
I would like
A great work!!
I would like you to test older hardware (such as HD5870) to see when this crossfire problem occurred.
I found one link which might be interesting for you (http://www.tomshardware.com/reviews/radeon-geforce-stutter-crossfire,2995-5.html). Although the test in this link is not sufficient for withdrawing the final conclusions, it suggested that microstuttering was more in HD6870 crossfire than in GTX560 SLI setup and also that triple crossfire setup with HD6870 alleviated microstuttering greatly. Since their methods were not as finesse as yours, they probably were not able to detect some issues (like runts) like you were. But still their data is worth noting, I guess. I think that their data hinted that:
(1) Crossfire issues might be older than HD7000 series.
(2) Triple crossfire setup might be able to alleviate the problem which occurs with dual crossfire setup.
I hope that you have a chance to test triple crossfire setup with your apparatus as well as older gpu cards.
Thank you for your interesting data.
I think we are going to test
I think we are going to test one or two of the previous generations of cards to see how things scaled throughout. Right now I am considering the HD 5870 and the GTX 480.
Great work. This is exactly
Great work. This is exactly the type of investigation that will make ALL companies better and go to a higher level.
Great test as always, but I
Great test as always, but I think you should test GTX 580 SLI vs 660 SLI ASAP!
both have around the same performance, but Kepler supposedly deals with frame rate metering with dedicated hardware for that, and the 580 doesn’t, both will use the same driver, from the same vendor, so it would be a great test and maybe help to understand how close AMD could get just with software tweaks!?
That is a huge amount of data
That is a huge amount of data presented Ryan. On the whole a killer article, one of the best I’ve ever read. The idea of testing the older 480 and 5870 should prove very interesting.
Thank you for such an informative, cohesive and thorough and downright geeky article. I loved reading through it. I think I went through the first article about five times just to ensure I had all the concepts fully understood. Awesome. 🙂
I just saw a video from valve
I just saw a video from valve discussing a tool called Telemetry from RAD games, it seems like it could be useful for analysis of game performance.
SUGGESTION FOR FRAME VARIANCE
SUGGESTION FOR FRAME VARIANCE GRAPH:
You have already noted elsewhere that Percentile (1-99) is insufficient, for tracking large SPIKES in Frame Times, spikes which may occur only once every few seconds (i.e., once every few hundred frames, thus well beyond the 99th percentile threshold).
So you need a different dependent variable (x-axis), with ‘milliseconds’ remaining the independent variable (y-axis).
In a 10 minute gampeplay section (36000 Frames at 60 FPS), 36 Frames is equivalent to the 99.9th percentile (36000 divided by 1000).
If you put ’36’ on the x-axis (or perhaps multiples of 10), you could easily show people the effects of LARGE SPIKES IN FRAME TIMES. In many cases/circumstances, more illuminating than Frame Variance Graphs which show nothing beyond the 99th percentile.
Are there any redeeming
Are there any redeeming features to AMDs Crossfire solution?
Yes, they drop every other
Yes, they drop every other frame in crossfire and count it anyway and sell it to their fans as a superior frame rate than SLI.
AMS CF Frame numbers sold 80!(actual screen rate 40+stutter ! )
Wow it sure beats 76 from SLI !
Yes, a very “redeeming” quality.
Thank you for not answering
Thank you for not answering my question 🙂
I just wanted to know if there might be an upside with regards to perhaps input lag. Kind of hard to measure though.
Ok, here’s a few thoughts.
Ok, here’s a few thoughts. With the AMD runts and drops, and the fans now trying to claim “less input lag”, my question is of course, “What are you aiming at?” !
With every other frame dropped or runted, you’re getting ~50% of the game screen opponent positions from the frames displayed (first person shooter as an example of course since most games are that ) – so your opponents position on screen is “jumping” from one frame over to two frames ahead, changing position in a lurch – with that type of inaccuracy of on screen opponents, how is “less lag” going to help when one cannot see half the positions of the opposition, and aiming at those in between spots just is not directly occurring? you have no visual data for those positions, half the positions of your opponent !
You can see how that may present some real issues for the gamer, as far as firing accuracy.
Yes, one could develop a sort of compensation with practice, just like a similar amount of consistent input lag is gotten used to as the mind and muscles acclimate and compensate accordingly.
So, it appears to me missing frames and runts are anything but helpful, and the lag thenno lag, or the lag not running in sync with what appears on screen could be a real problem.
With equalized frame time outputs to screen, the “smoothness” factor certainly helps aiming at opponents and learning to time things correctly.
Thank you for not answering
Thank you for not answering my question 🙂
I just wanted to know if there might be an upside with regards to perhaps input lag. Kind of hard to measure though.
Are we seeing that crossfire
Are we seeing that crossfire is inherently borked? Or are we seeing that the current drivers are not leveraging the potential of the hardware?
I’m thinking that drivers can be updated.
A note on the fanboy thingy, surely we need both companies to succeed to push one another to create ever faster and more powerful cards. No?
The problem has persisted at
The problem has persisted at least two series of cards, but based on AMD’s response, it likely has always existed. AMD had said they never considered frame metering. Instead, they had been focusing on delivering frames as fast as possible.
That to me would say the drivers were not borked per say, they just weren’t trying to do anything about it. They weren’t even aware of it.
Nvidia has their metering done with special hardware, but AMD believes they can do the same with software, which they probably can, though it may not be as ideal.
Not sure why the 660 is being
Not sure why the 660 is being compared to a 7870. MSRP is over $100 difference. 7850 makes more sense to me but there must be a good reason the 7870 was chosen.
The GTX 660 is the cheaper of
The GTX 660 is the cheaper of the two cards. You must have the GTX 660ti in mind.
http://www.newegg.com/Product/Product.aspx?Item=N82E16814130826
http://www.newegg.com/Product/Product.aspx?Item=N82E16814127699
The reason is a big fat
The reason is a big fat smooch on the behind of amd and their fanboys.
Keep the raging natives a bit less restless.
You see, only 2 pages of comments, they didn’t come in screaming bias while their radeon I don’t believe any of this and “if it’s true” it doesn’t matter heads exploded.
They have all been made absolute idiots, as the years of their stupidity is apparent.
The years of them screeching no one can see over 30 fps anyway is exposed. It doesn’t matter what they knew or did not know, they made every excuse and lie and fanboy promo they could over the course of years and now it is absolutely clear they were seeing only half the frames gobbled with screen wretching runts, but still claimed it was beauty incarnate.
They are fools, and we all know it.