A quick look at performance results
We spent a bit of time with the new BF4 Mantle patch and the AMD Catalyst 14.1 beta driver.
Late last week, EA and Dice released the long awaited patch for Battlefield 4 that enables support for the Mantle renderer. This new API technology was introduced by AMD back in September. Unfortunately, AMD wasn't quite ready for its release with their Catalyst 14.1 beta driver. I wrote a short article that previewed the new driver's features, its expected performance with the Mantle version of BF4, and commentary about the current state of Mantle. You should definite read that as a primer before continuing if you haven't yet.
Today, after really just a few short hours with a useable driver, I have only limited results. Still, I know that you, our readers, clamor for ANY information on the topic. I thought I would share what we have thus far.
As I mentioned in the previous story, the Mantle version of Battlefield 4 has the biggest potential to show advantages in times where the game is more CPU limited. AMD calls this the "low hanging fruit" for this early release of Mantle and claim that further optimizations will come, especially for GPU-bound scenarios. Because of that dependency on CPU limitations, that puts some non-standard requirements on our ability to showcase Mantle's performance capabilities.
For example, the level of the game and even the section of that level, in the BF4 single player campaign, can show drastic swings in Mantle's capabilities. Multiplayer matches will also show more consistent CPU utilization (and thus could be improved by Mantle) though testing those levels in a repeatable, semi-scientific method is much more difficult. And, as you'll see in our early results, I even found a couple instances in which the Mantle API version of BF4 ran a smidge slower than the DX11 instance.
For our testing, we compiled two systems that differed in CPU performance in order to simulate the range of processors installed within consumers' PCs. Our standard GPU test bed includes a Core i7-3960X Sandy Bridge-E processor specifically to remove the CPU as a bottleneck and that has been included here today. We added in a system based on the AMD A10-7850K Kaveri APU which presents a more processor-limited (especially per-thread) system, overall, and should help showcase Mantle benefits more easily.
We have several other platforms setup for testing, with various graphics cards and processors (including a Core i3-4330), but testing will take quite a bit more time to get all of the results we want to present.
Test System 1
- Intel Core i7-3960X
- 16GB DDR3-1866 memory
- ASUS R9 290X DirectCU II
Test System 2
- AMD A10-7850K
- 16GB DDR3-2133 memory
- ASUS R9 290X DirectCU II
As you might be aware, we use a graphics performance testing methodology called Frame Rating for nearly all of our reviews. This process uses capture hardware to record the output from the test PC that gets post-processed to evaluate real-world end user experience. Unfortunately, for Mantle, our tools don't quite work yet. Dice helped us out though by integrating a new feature with the latest BF4 patch to enable frame time logging directly through the in-game console.
PerfOverlay.FrameFileLogEnable 1 starts recording frame times to a CSV file. Setting that variable to 0 will stop the recording. All log files will find their way into the My Documents/Battlefield 4 folder. Even better, because this is done at the engine level, we can get some interesting information from it. Not only is the frame time for each display frame recorded, we also get the GPU and CPU time for each frame as well. This is interesting as it allows us to see how much time each individual component spent on each frame to check for bottlenecks on both ends.
THIS. IS. BETA!!
I'm sure you have all used beta Catalyst drivers before. These are much different. I might go as far as to consider these drivers, and the Mantle libraries themselves, in an "alpha" state. There is a pretty sizeable list of known issues which AMD is being very upfront about. Its includes the following:
- Intermittent stuttering or stability issues may occur when utilizing Mantle with AMD CrossFire technology in BattleField 4
- Mantle performance for the AMD Radeon HD 7000/HD 8000 Series GPUs and AMD Radeon R9 280X and R9 270X GPUs will be optimized for BattleField 4 in future AMD Catalyst releases. These products will see limited gains in BattleField 4 and AMD is currently investigating optimizations for them.
- Notebooks based on AMD Enduro or PowerXpress technologies are currently not supported by the Mantle codepath in Battlefield 4.
- AMD Eyefinity configurations utilizing portrait display orientations are currently not supported by the Mantle codepath in Battlefield 4.
- Graphics hardware in the AMD A10-7850K and A10-7700K may override the presence of a discrete GPU under the Mantle codepath in Battlefield 4.
- AMD testing for the AMD Catalyst 14.1 Beta has been concentrated on the following products: AMD Radeon R9 290X, R9 290, R9 280, R9 270, R7 260X, R7 260, HD 7000 Series, HD 8000 Series, A10-7850K and A10-7700K. Future AMD Catalyst releases will include full test coverage for all AMD products supported by Mantle.
The CrossFire stability issue, at least in my experience, is not a minor thing as I was never able to get it working properly on either an X79 platform or the A88X + Kaveri system.
In several talks with AMD this week I did express concern over this very early beta status of Mantle. My fear is that with AMD having difficulty getting Mantle working with a single gaming title (Battlefield 4 today) after having a full month of extra time (remember the initial promised December release) does not bode well for Mantle in the long term. AMD tells me that Mantle has been in the works for more than two years which should be enough to perfect the platform for these limited variables. It is possible that these growing pains of stability and performance are a one time occurrence to go along with a new graphics API. But, once several or a dozen games are available using Mantle, how will AMD's support and development teams be able to handle it?
A Quick Note on Image Quality
I know that some of you are curious about the image quality comparisons between DX11 and Mantle codepaths. Though I didn't do any testing in this story I did go to the man himself to ask the question. Johan Andersson, Technical Director on Frostbite at Electronic Arts, told me quite clearly that "no, there shouldn't be any difference" in image quality between the two options in Battlefield 4. I don't usually just take people at their word, but he is an exception!
Worst Case Scenario
Our standard level and section used for testing Battlefield 4 is the opening portions of the single player SP_Dam stage. As it turns out, this section is quite GPU limited (as we would normally want for our GPU testing…) but this doesn't fair well for AMD's Mantle out of the gate. Here are results with the Core i7-3960X and R9 290X at 2560×1600 with the Ultra preset.
Click to Enlarge
Click to Enlarge – Note the incorrect labeling here, it is NOT 1920×1080 but in fact 2560×1600
Results from both the Mantle version and the DX11 version are essentially the same here, averaging about 43 FPS. That's a bit disappointing but, after the preparation from AMD about setting expectations, it more or less makes sense. This is a high-end processor in a section of the game that is very GPU limited and thus we didn't expect to see a dramatic advantage for the Mantle API. We did however expect to see SOME KIND of advantage…
Click to Enlarge
Click to Enlarge
If we look at the CPU time graph though from these two runs, it is clear that BF4's Mantle implementation is having an effect. Notice how much more consistent the CPU time is on the Mantle test run without any kind of spikes of CPU usage. Our Core i7-3960X is definitely being utilized more efficiently although, in this particular combination of hardware and level, it just doesn't result in any performance advantages.
With all due respect, Ryan. I
With all due respect, Ryan. I like the reviews but please just put a little more time in your plots. They have always been very fuzzy and faded.
And when you make a typo (like 1920×1080) couldn’t you go back to Excel and change the title?
I am not trying to tell you how to run your site. But, I can’t help but wonder, don’t you have a little “perfectionist” in you? Don’t you take pride in presenting your work?
Glad I read this…DX 11 is a
Glad I read this…DX 11 is a standard way of doing things , and this Mantle sounds very limited.
It`s not going to change the world.
Engadget said…AnandTech and HotHardware have used almost-final Mantle drivers to achieve frame-rate gains of at least 7-10 percent in BF4, rising to 30 percent with some configurations, by doing away with the need for Microsoft’s relatively inefficient DirectX drivers.
Ryan tells us what it really does.
My experience so far
My experience so far (i5-2500K @ 4.3GHz, MSI HD7950, 8G DDR3-1866,1080P, mostly Ultra settings, Win7pro x64): I ran a 10 minute run on Siege of Shanghai under both Mantle and DX11, logging the runs with the built in performance monitor. DX11 was ~66fps and Mantle gave me ~73fps. An increase of ~10.6%. HOWEVER, in the latter few minutes of the run, Mantle starts to hiccup and stutter annoyingly bad.
Checking the log file shows that the Max CPU Time keeps spiking, as high as 776ms (~1.3fps minimum), frequently (40+ instances over 10min) saw spikes above 100ms (10fps) whereas the Max GPU Time only got as high as 47ms (~21fps minimum). At first i thought it may be another background program using the cpu to do misc work, so i closed programs and minimized the number of necessary startup programs and retried. Still get the awful stutter. I set BF4 to High Priority in the task manager to prioritize it over other cpu usage and still get the stutter. The Render.DrawScreenInfo option shows 3 job threads on my 4core processor. Not sure what the issue is.
My DX11 runs, while having a lower average fps, gave me much more consistent frame rates. Max CPU Time only spiked up to 54ms (18.5fps min), Max Frame Time up to 44ms (22.7fps min), and Max GPU Time up to 27.9ms(35.8fps min)
Perhaps this is part of the 7000 series optimizations they mention, but it seems like it isn’t the GPU that is having issues because GPU Time under each API is low without spiking. I think if i can fix the CPU Time spikes, i’ll see a bigger performance increase under Mantle. I’ll see if i can force it to use a 4th cpu thread and retry also. As it stands, i think i’m going to wait for another patch or two before i really switch over to Mantle.
Ryan, if you read this, can you run your tests out further than a minute on future testing? My first major spike doesnt come until about 4min in. Maybe its an issue that manifests itself over time? Others are speculating that its a PunkBuster issue, where its trying to take a system screenshot. Some also suspecting GPU clocks changing due to power requirements or heat.
Can you guys do 64 player
Can you guys do 64 player server comparison between DX 11 and Mantle.
In 64 players in DX 11 there are major dips in frames under heavy action.
Am curious to see how Mantle min fps in high load scenarios does.
Watch that video ^
Someone should have told that
Someone should have told that guy 290(X)s scale 20% better then 780s in BF4. Then you add mantle boost. Even if he plugs that 4th card in he is going to be slower then a 290(X) tri-fire setup.