Competition is a Great Thing
We spent some time testing the Star Swarm benchmark with mainstream graphics cards on both a high end and budget platform. The results are surprising!
While doing some testing with the AMD Athlon 5350 Kabini APU to determine it's flexibility as a low cost gaming platform, we decided to run a handful of tests to measure something else that is getting a lot of attention right now: AMD Mantle and NVIDIA's 337.50 driver.
Earlier this week I posted a story that looked at performance scaling of NVIDIA's new 337.50 beta driver compared to the previous 335.23 WHQL. The goal was to assess the DX11 efficiency improvements that the company stated it had been working on and implemented into this latest beta driver offering. In the end, we found some instances where games scaled by as much as 35% and 26% but other cases where there was little to no gain with the new driver. We looked at both single GPU and multi-GPU scenarios on mostly high end CPU hardware though.
Earlier in April I posted an article looking at Mantle, AMD's answer to a lower level API that is unique to its ecosystem, and how it scaled on various pieces of hardware on Battlefield 4. This was the first major game to implement Mantle and it remains the biggest name in the field. While we definitely saw some improvements in gaming experiences with Mantle there was work to be done when it comes to multi-GPU scaling and frame pacing.
Both parties in this debate were showing promise but obviously both were far from perfect.
While we were benchmarking the new AMD Athlon 5350 Kabini based APU, an incredibly low cost processor that Josh reviewed in April, it made sense to test out both Mantle and NVIDIA's 337.50 driver in an interesting side by side.
Here is the setup. Using a GeForce GTX 750 Ti and a Radeon R7 260X graphics card, somewhat equivalent in terms of pricing and performance, we ran the Star Swarm stress test benchmark. This application, built originally to demonstrate the performance abilities of AMD's Mantle API, was also used in some of NVIDIA's slides to demonstrate the performance improvement in its latest beta driver stack. To add some interest to the test, we ran these on both the AMD Athlon 5350 APU (considered one of the lowest performing platforms you'll find a x16 PCIe slot) as well as the high-end Core i7-3960X 6-core Sandy Bridge-E platform we use for our normal GPU test bed.
| Test System Setup | |
| CPU | Intel Core i7-3960X Sandy Bridge-E AMD Athlon 5350 Kabini APU |
| Motherboard | ASUS P9X79 Deluxe Gigabyte AM1M-S2H |
| Memory | Corsair Dominator DDR3-1600 16GB |
| Hard Drive | OCZ Agility 4 256GB SSD |
| Sound Card | On-board |
| Graphics Card | AMD Radeon R7 260X 2GB NVIDIA GeForce GTX 750 Ti 2GB |
| Graphics Drivers | NVIDIA: 335.23 WHQL, 337.50 Beta AMD: Catalyst 14.3 Beta |
| Power Supply | Corsair AX1200i |
| Operating System | Windows 8 Pro x64 |
I think you'll find the results are quite interesting. Let's take a look.
Our first set of results looks at a pre-optimized software stack. That means we are looking at the GeForce GTX 750 Ti with the 335.23 driver and the Radeon R7 260X with DirectX on the 14.3 driver.
If you look at the results from the Core i7-3960X platform, the GeForce GTX 750 Ti has the advantage in average frame rate by 52% – a strong margin. On the Athlon 5350, the much slower processor in our set of tests, that performance lead for NVIDIA's GTX 750 Ti shrinks to 18%. Clearly the hardware and driver for the GTX 750 Ti is more able to take advantage of the available CPU headroom with DirectX compared to what AMD is doing with the 260X and its DX11 implementation.
Moving from the slower CPU to the much faster CPU, NVIDIA's hardware sees improvement by 116%. AMD on the other hand only improves by 68% indicating that either the R7 260X itself, or the driver, isn't able to take advantage over the extra CPU headroom.
Now, let's look at these same sets of results but using a post-optimized software stack that enabled Mantle for the R7 260X with the Catalyst 14.3 driver and moved to the NVIDIA 337.50 driver for the GTX 750 Ti.
Things look a little bit different this time! At the top end of the graph on the results run on the Core i7-3960X processor, NVIDIA's GeForce GTX 750 Ti maintains a lead, though slightly smaller, at 45%. Keep in mind that this is WITH Mantle enabled on the 260X and with the new DX11 changes made in NVIDIA's 337.50 driver.
On the Athlon 5350 platform, AMD's R7 260X is able to take the lead away from NVIDIA's GTX 750 Ti by 8%. This indicates that the performance advantage of Mantle on the lower end platform is larger for AMD than the DX11 changes in NVIDIA's 337.50 driver are for the GTX 750 Ti.
If we compare the 260X performance on both platforms though, clearly something is holding it back. Moving from the Athlon 5350 to the Core i7-3960X only sees a 12% performance improvement while NVIDIA's GeForce GTX 750 Ti average frame rate increases by more than 75%.
Finally, let's see all these results on the same graphic for a different view.
On this image you can see how each platform with each graphics card is able to scale with the software changes made with Mantle and with the 337.50 driver. In the red bar we have results from the 335.23 NVIDIA driver and the 14.3 Catalyst driver running in DirectX mode while the blue represents the Mantle and 337.50 scores.
Impressively, the Radeon R7 260X sees a performance improvement of 91% by enabling the Mantle version of Star Swarm on the lower end Athlon 5350 APU. On that same system, NVIDIA's GeForce GTX 750 Ti is able to achieve a 49% frame rate increase while continuing to use DirectX 11 and improving driver efficiency. On the high end platform with the Core i7-3960X, the R7 260X GPU improves by just 27% from enabling Mantle; the GTX 750 Ti scales by 21%.
Both GPUs and both software changes (Mantle and 337.50) see more performance advantages when running on the Athlon 5350 APU system than on the Core i7-3960X system, which obviously makes a lot of sense. Of these two different solutions we are showing, only on two processors and only with one game, AMD Mantle appears to have the bigger potential performance advantage for games that are CPU bound, or in CPU-bound sections of a game.
But what NVIDIA has done with the 337.50 driver changes are impressive considering they are staying within the bounds of the existing, well entrenched and well known API of DirectX 11. A 49% gain is nothing to sneeze at even though Mantle saw a 91% advantage under the same conditions.
Many users will argue (and have in our comment sections) that what NVIDIA has done with 337.50 is really just game-specific optimizations and isn't anything different than what these two GPU companies have been doing for the past decade. Even though NVIDIA has clearly stated that is not the case, you are make up your own mind if you choose to believe them, but I would posit that it doesn't really matter. Is not a game-specific Mantle integration the same thing and possibly even more troublesome as it requires the developer, rather than the hardware vendor, to take the brunt of the work?
Clearly we would like to have more data on these graphs with more graphics cards and we might be able to do some of that testing starting next week. I think seeing how a GeForce GTX 780 or Radeon R9 290X scales on these two platforms would be equally as compelling.
These tests above aren't meant to be conclusive evidence for either vendor's current direction but are simply there to add more data points to our discussion moving forward. AMD Mantle has huge potential upside but requires a lot of commitment from game developers and AMD's own software team to keep up. NVIDIA's continued DirectX improvement seems have a lower peak but can be implemented on a much wider array of games and without the need for developers to commit to something long term.
I'm almost afraid to ask but…please leave me your thoughts on this debate and the future of both Mantle and DX11 in the comments below!
And BooM!, Ryan bitch slaps
And BooM!, Ryan bitch slaps the hatters! You want fair and biased test and reviews, y’all get it, bitches! great read, interesting results. And it answers the question, “Can I throw a gpu at a AM1?” Yes, yes you can.
Hopefully you meant UNbiased,
Hopefully you meant UNbiased, right?
Yea, good catch
Yea, good catch
No temp or clock results yet
No temp or clock results yet for 335.23 vs 337.50 ?
http://www.hardocp.com/article/2014/04/08/amd_radeon_r9_295x2_video_card_review/12#.U1GKWmdOWUk
hello ryan,
it’s your 3rd
hello ryan,
it’s your 3rd article and benchs, and you are still avoiding cpu overhead situation, either by putting gpu limited situation or 3960x for no bottleneck.
after all these benchs it just seem really unlikely that your knowledge about how thing works just vanished, or the most likely sponsored article.
as i commented on the last article this became weird for me to even comment on, you seemed very persistant to avoid giving relevant bench to the cpu overhead issue repeatedly, by hiding behind gpu limited situations, tweeking settings to show special effect management, buged games, SLI, very high setting, anyone with half a brain knows that multi-gpu isnt cpu overhead, or gpu bound isn’t cpu overhead, or effects resource management optimisation isn’t cpu overhead.
but there you go, you seem convinced that most readers are too stupid to notice, and most definitly there are some, just by- reading some of the comments here.
1st article i thought you didnt know and made a mistake.
2nd article i found it weird, to see how persistant you are at avoiding cpu overhead.
3rd article just removed any doubt i might had.
You mean Ryan didn’t forget
You mean Ryan didn’t forget to read the 337.50 release notes this time ?
335.23 & prior
NO Star Swarm optimization
337.50
◦Up to 21% in Star Swarm
Star Swarm optimization
Well maybe he did
Some how he is refusing to read release notes and take what ever Nvidia tell him as truth. That would be fine if it were true and not contradicted by the driver release notes.
He goes on to pose that driver optimization is better then game engine optimization?
I want what he is smoking so I can believe the Bengals will have a shot this year too.
The patch notes include a
The patch notes include a list of measured gains that Nvidia observed using 337.50… the fact that they noticed performance gains in Star Swarm is not indicative of game-specific optimization.
If Nvidia released a driver that forced your GPU to run 100Mhz faster, and in the patch notes Nvidia says it will give you a 10% increase in Star Swarm, would you call that a game-specific optimization?
Also, he didn’t say that Nvidia’s solution was BETTER, nor that Mantle is WORSE. He’s saying they both have their pros and cons.
Nvidia’s driver optimizations give benefits to the consumer without requiring anything extra from the developers, but (as demonstrated) it doesn’t seem to have as great an improvement on performance.
Mantle (as demonstrated) can give a hell of a boost in performance, but it will require a lot more work from the developers if they want to take advantage of its strength.
Did you even bother to read the article you’re commenting on?
Until Dx12 shows up I think
Until Dx12 shows up I think MAntle will gain adoption and why not. It’s fast and seems reliable enough, many of us have upgraded the drivers for far less. For one I can’t wait the Mantle-enabled Civ5 that uses the CPU better so I’d finally be able to play a decent sized map on the laptop
By the time DX12 get released
By the time DX12 get released there should be some improvments to mantle, as AMD does have a jump on DX, so maybe there will be a Mantle 1.1, or whatever before DX12 get out the door! It’s good to see all the intrested parties turn their attention to the Graphics APIs/Drivers. It makes one wonder why it took so long, on the Driver and Graphics API side, or was there an intentional intent to meter these graphics driver/API improvments out slowly to milk the incremental improvments for profits over these years. Yes let’s slow the hardware improvments down a bit, and rebrand some SKUs, and only slightly improve the graphics API/Drivers, just enough to make the customer think that thay are getting some improvment for their money. But the rise of the Mobile Market, and AMD’s lack of funds for rapid hardware improvments has forced AMD to make up for the slower hardware advances in its APU roadmap with a more rapid improvment in graphics API/driver development, and Nvidia is now forced to do the same. The rise of the mobile market as a money maker, and the more competition in the mobile market, with players like imagination technologies powerVR GPUs, and others, has pushed Nvidia to merge its mobile GPU microarchitecture with its desktop microarchitecture, that and AMD’s HSA gamble beginning to pay off! The mobile market has a GPU(PowerVR wizard) with hardware raytracing, and in the future with the Mobile and Desktop GPU microarchitectures merged(Nvidia,others) between Mobile and desktop SKUs, it will be just a matter of scaling up the GPU, and that will make the only difference between desktop and mobile GPUs, just a matter of adding more execution units. I would not be suprised if one of the GPU IP players, in the mobile market, takes advantage of the modern GPUs ability to be scaled up, and introduces a descrete GPU to the market, to compete with AMD and Nvidia. A third player would help, Intel’s intrest in the descrete GPU market appears to be nil, at the current time, but some of the mobile GPU technology innovations in GPUs and SOCs, just may begin to work their way into the descrete market, and descrete GPUs may just all become APU type complete SOCs unto themselves in not too many years. (See AMD’s console APUs, and Nvidias talk of putting custom ARM Denver CPUs on some desktop Maxwell or later descrete GPUs)
MS won’t release half
MS won’t release half finished beta like AMD did. They will make sure DX12 is ready hence why it won’t be out til late next year.
That’s a bad point
Min
That’s a bad point
Min me?
Vista?
Win 8?
M$ is known to only pay
M$ is known to only pay attention to gaming when it serves their needs( Need to maintain OS domination, or as a carrot to entice gamers into their closed ecosystem, 30% cut of the action from game developers, middleman milking TIFKAM 8 OS crAPP store). Mantel and AMD’s version of HSA, is starting to show results, as it will for all the other members of the HSA foundation with their versions of HSA, Arm Holdings, Samsung and Qualcomm, to name a few. The Khronos group will also be improving the OpenGL, OpenCL, etc graphics APIs! Steam OS will begin the transition to an open OS, designed and supported by the gaming industry, and other Open OS groups (Debian/Debian bsed). There is no dirver/API system that is not in a state of constent revision, and getting to the market first, and getting the gaming engines makers to support it will provide AMD and gaming with a definite advantage. M$ by not allowing its latest graphics API on windows 7, is just shooting itself in the foot, beacuse Mantle will not have that restriction for any AMD based GPUs, Nvidia is going to be at a disadvantage in going with a M$ DX based only solution, as the majority of computers at this time are running windows 7, and Nvidia is just attatching itself to an anchor known as 8. Mantle can and does provide an improved gamimg experience, not restricted to M$’s OS whims, or M$’s need to force usere into ots closed ecosystem. Gaming OS competition is good for gaming, and AMD with Mantle, and Khronos with its OpenGL, OpenCL will also benifit, hopefully AMD will make the Mantle API completely open, but Mantle’s influnce has resulted in much more innovation, as healthy competition always does. M$ is and always will be a detriment to gaming and OS innovation, as long as it is allowed to maintain an OS monopoly.
Are you talking about the
Are you talking about the same MS that broke win XP with MS Security essentials?
Or the same MS that release a broken patch that drains the battery life out of Surface and didn’t really care after a whole month(December) before releasing a fix?
The only thing i get from
The only thing i get from this is what AMD has been saying that Mantle works great with low-end CPUs.
With these test being done with beta drivers and a beta game demo, i don’t see much of a difference between the two at all.
whatever it takes to drive
whatever it takes to drive the industry forward to give better performance for no extra money is a win-win for us consumers 🙂
Huh ?
Many were saying these
Huh ?
Many were saying these were normal game specific changes, but if so you would not get better % on lower card and lower CPU’s .
What Nvidia is doing “works” on all Dx11 app, sure you may not see massive gains but it should help an smooth it out as normally min frame rate is caused by CPU .
Performance improved 2x times on low end HW here .
yea AMD seems to dropped
yea AMD seems to dropped doing anything dx wise in games with mantle, very bad for end users if mantle ends up having a serious bug at some point.
No just bad for M$’s wallet,
No just bad for M$’s wallet, and if Nvidia desires to sink along with the M$ rudderless Tub, then so be it, but I suspect, that Nvidia has some API of their own in development, windows 7 is not going away anytime soon, but it will not get the latest DX, anyway! Steam and OpenGL, OpenCL, etc. will also be bad for Redmond’s balence sheet, but good for gaming, and gamers! So spin away you M$ Spinion, but the M$ dog has had its day! Time for that little trip out back, behind the shed, for a permanent cure for the OS/Graphics API mange and foaming at the mouth! No one will cry, at this M$ TIFKAM dog’s demise.
I would be one of those that
I would be one of those that insist that these are specific game optimizations. And as mentioned we have seen them from about forever.
That said thanks for the results using the 5350.
Using the 260x in the tests
Using the 260x in the tests was a bad idea. In the Mantle test with the Athlon CPU, the 260x is the bottleneck, not the CPU. Using a faster GPU would’ve showed the real difference Mantle makes with such a low-end CPU.
Well its for people with
Well its for people with little money to show them what boosts happen. Game yes is pretty cpu bound but not everyone has 2grand to drop on a decent rig, most people seem to be around 1000$ or less. Probably could tossed in say around 100-130$ i3 part just to compare.
Well, the test configuration
Well, the test configuration isn’t really meant then to show how effective Mantle is in reducing CPU usage. Using faster GPUs with the low-end Athlon would’ve made for much more reliable benchmark.
To be fair one has to say
To be fair one has to say that the 337.50 driver from Nvidia is pretty much optimized for Starswarm only, with the gains in other application cases being pretty minuscule (if one does not include the SLI changes to RTW2).
Things look similar with Mantle atm, but there are about two dozen games planed to support Mantle in the near future.
I could be wrong but is this
I could be wrong but is this the first nvidia driver to support star swarm? Yes or no, I must admit that it is a nice boost in performance over the 335 drivers. That said you can really see how well mantle scales. The 260x only drops 3fps going from a 6c/12t monster to a 25w 4c CPU. The 750ti’s performance dropped almost 50% by changing the CPU.
How about you see how well matle really scales? Take a 290x and run that test again. Will we still see a 3fps drop in rates?
Shows how cpu intensive the
Shows how cpu intensive the game is if it dropped that much. But AMD 5350 is super budget cpu, most people in say US won’t buy it less building an HTPC or something want to keep low power.
Many users are reporting the
Many users are reporting the new driver gave them 80% more performance in Star Swarm with cards like 660Ti, 670, 680 and 780,and CPUs like i7 3770 and 4770.
Please do a 290X vs 780Ti too.
Using a GeForce GTX 750 Ti
Using a GeForce GTX 750 Ti and a Radeon R7 260X graphics card, somewhat equivalent in terms of pricing and performance, we ran the Star Swarm stress test benchmark. This application, built originally to demonstrate the performance abilities of AMD’s Mantle API.
The equivalent GTX 750 Ti, is Radeon 265.
All we know that radeon 265 , beats de Gtx 750 Ti . I would like to see the test.
Sweet baby Jesus, I wish
Sweet baby Jesus, I wish these AMD trolls would leave…or at least READ THE F* ARTICLE!
“To be fair one has to say that the 337.50 driver from Nvidia is pretty much optimized for Starswarm only, with the gains in other application cases being pretty minuscule (if one does not include the SLI changes to RTW2).”
DId you read Ryan’s previous article? Clearly not as you would have seen 36% gains in Hitman , 31% gains in Thief, 23% gains for Sleeping Dogs, 22% in CoD: Ghosts, and more for SLI. Single GPU gains were also high, with 26% gains in Hitman and 17% in Skyrim.
We get it. You like AMD and get stickers or some other garbage from AMD’s “special forum” program. You don’t need to keep trolling with this garbage if you aren’t even reading PCPer’s articles! Go troll at Anandtech. They have plenty of you trolls there already.
LOL!!!
Some of those titles
LOL!!!
Some of those titles you mention aren’t even DirectX 11.
DX11 shares functions with
DX11 shares functions with DX10 and DX9. It’s one of the benefits of having a unified and backwards compatible pipeline instead of taking your ball home and making Mantle.
Different code paths
Different code paths
anger management, I read it.
anger management, I read it. it speaks to driver efficiency changes, not reduction in CPU overhead for NV. mantle clearly does reduce CPU overhead…
as I said below, CPU graph would be handy.
now, relax before you stroke out.
As the person who wrote the
As the person who wrote the passage quoted by you: Yes, I did read all recent articles from Ryan/PC Perspective about Mantle & the new Nvidia drivers.
But just because I read them, I don’t necessarily agree with them, because I have read many other articles claiming different things. Just because you have a different opinion I am not a troll.
I specifically mentioned that in other application cases the gains are minuscle, because they are! Most of these are normal driver improvments you get with the most high-end hardware available might be a way to get you such increases.
But other test done in more realistic conditions show that Nvidia claims about reducing the CPU load are nonsense. The German hardware magazine PC Games Hardware tested a Core 2 Quad (QX9650) and a i7-3770K with a GTX 780Ti and the new 337.50 drivers. The performance increase was minuscle for all games except for Star Swarm.
HWL did test a GTX 750 Ti and the greatest increase for was below 8%. Not a single two digits increase into frame rates.
The same can be said for the tests from Tom’s Hardware with a GTX 780 Ti.
I visit multiple PC hardware website regularily and PC Perspective is the only one ending up with such insane increases in performance.
Based on the results from the other sites however the new 337.50 driver is not anywhere compared to Mantle and most of the increases are only available at larger resolution with a lot of post-processing.
So, Sir, I don’t see how I am a troll and why my comment was “garbage”. But someone who clearly put that much effort into writing how much I am a troll and who clearly has the right attitude to make internet discussions is the right person to put my trust in.
can we get a CPU usage
can we get a CPU usage graph?
you had one in previous mantle testing.
so, with that we could see if the Nv driver
is actually doing what mantle is known to do,
decrease CPU overhead.
hope I hit the lotto this weekend, I’ll come pay y’all to let me hang out. 😛
Please, test Nvidia
Please, test Nvidia 3dVision.
Big improvements in BF4, Far Cry 3, etc, with the new driver.
Gtx 750 vs R7 260 ? why not
Gtx 750 vs R7 260 ? why not R7 265 ?
That picture looks like
That picture looks like something that should go in a car and not a computer case. At the rate these GPUs are sucking down power, we’ll be measuring PSUs in horsepower anyhow.
Obviously, Mantle is a
Obviously, Mantle is a “divide and conquer” strategy. If it works by accumulating developers that use Mantle to optimize performance on AMD before MS gets DirectX 12 out, good for them. However, one can’t help but wonder if AMD’s drivers could also benefit from optimizations similar to what Nvidia has found, and implemented, in theirs. If AMD has the resources to do so, the combined efficiencies would be really impressive.
well that would be stupif,
well that would be stupif, one of the major points that pushed AMD to bring Mantle is that they dont see the need to spend more resources on driver dev teams, while they can have all the optimisations done by the game developers on a lower level API, thats why AMD and game Devs were nagging at mecrosoft for years to make DX lower, devs dont want to be limited by cpu overhead, and they dont want optimisation that comes months after the game is released, in the case of this driver Years.
one of the reasons Nvidia was refusing lower level API directx for years because it had a competitive tool against AMD, by puting more money into drivers to bring out the extra fps needed to beat on perf, and ofc the only answer from nvidia to AMD’s Mantle is wonder driver, hiding behind massive sli scalling and profile addition, optimisation of AMD gaming evolved games, and fixing some bugs along with effect resource management, branded as cpu overhead reduction on a LOL 1000$ cpu, on specific settings for specific games, ppl would believe anything, sad though that pcper went along, because without sites and news outlets like these companies like nvidia would have to work hard to give truthfull and straightforward innovations, instead of lying and disguising it for what it’s not.
the results of AMD quarter are out, shows how well they are doing on GPU end, and guess what’s that market share is taken from ? nvidia.
Frankly Star Swarm should
Frankly Star Swarm should have never been used ever by anybody. It is either written by incompetent programmers who don’t understand a thing and thus is crappy thing or it was sabotaged to make look Mantle great.
Sole explanations for what I have seen when I analyzed Star Swarm using their own logs and using VTune. (Every draw call has all or nearly all parameters reset even when nothing changed – extremely bad idea as it forces each draw call to recalculate and synchronize threads to protect data structures) And they are using way too much Command buffers, so overhead of creation of it and its execution would overshadow massively any gain. And similar goes for simulation itself. (Only few seconds of higher parallelism for entire run of timed benchmark)
This craptastic “benchmark” should be ignored until devs explain every single idiocy they committed and until they fix them all. But then we discover that not even this engine needs Mantle, because DX11 already provides all they need…
This is becoming
This is becoming pityful.
Stop trying to defend your love for nVidia with more and more articles.
This is a standard driver, optimizing standard stuff, period.
Yes, in specific scenarios it will show as good an improvement as changing the API.
And yes the point of mantle is to avoid having developpers limited by an API hoping the driver team will understand what they really wanted to do and have them write what they want in the first place.
That doesn’t change anything.
And seriously, starswarm … a alpha tech demo … Good for nVidia they have money to spend on optimizing a specific code path for that to try to steal a bit of mantle thunder. They still rip ppl off.
450€ 680 10% slower than 350€ 7970GHz for the lulz all over again plz.
Only hardcore amd fanboy buys
Only hardcore amd fanboy buys loud, driver bugs & flickering shitty AMD cards ;( damn u dummie!
http://www.newegg.com/Product/Product.aspx?Item=N82E16814125463