Tech-Hounds.com

Benchmarking on Trial - Part 2

Gamers usually want high enough frame rates for comfortable gameplay and that means less low frame rates and stutter (sharp drops in frame rates). Low frame rates can be caused by a variety of factors, including but not limited to graphics cards. Running benchmarks does provide an idea what kind of performance we can expect. After all, benchmarks was originally used to pinpoint system performance bottlenecks. However, benchmark results alone is not enough to make a judgment of performance.

What do we actually want to achieve with a benchmark? Since we are talking about graphics cards, we're talking about graphics benchmarks. A graphics benchmark is a benchmark that focuses on graphics card's performance. So we want to measure the graphics card's performance. In this case, it would be best to remove any system limitations that may hinder the card from using all its potential. However, that's not entirely possible with game benchmarks. Even timedemo runs in game benchmarks are not immune to system limitations, but it is as close as we can get without resorting to a pure synthetic graphics benchmark. Gameplay testing sessions will more likely be heavily affected by system limitations than timedemo runs.

Experience from our last article shows us that test results from replays / timedemo runs can be significantly different at times. That's actually pretty natural because the game may be graphically intensive in some areas and more system intensive in others. The benchmark may also not be 'heavy' enough to represent the most graphically intensive situations and scenes you will encounter throughout the entire game. That's one thing we want to explore more deeply in this article.

There's also the issue of gameplay relevance. Results that show significant difference between two cards on less graphically situations is not going to accurately portray what these two cards will do in a much graphically intensive scenes. Results from a graphics benchmark that 'capture' parts of the game with the lowest frame rates (that's graphically related) is more informative in examining the differences (if any) between two cards.

The purpose? To find out what kind of performance difference we can expect between two graphics cards that have varying degrees of pixel processing power. We've narrowed down the game and the game levels we're going to use in the previous article. This time we're going to focus on finding situations and scenes where we most likely have lower / lowest frame rates in the game. However, the results must indicate that particular scene scale with more pixel processing power - a common focus point determining graphics cards performance.

For this test, we're still using the same test setup we used in the first part of the article. To vary the degree of pixel processing power, we disable one and two quads of the GeForce 7900GT we used. So, effectively we have a GeForce 7900GT with 4, 5 and 6 quads. The 4 and 5 quads setup can be considered a proxy for a graphics card that' has slightly lower pixel processing power - 33 and 16 percent slower, respectively.

F.E.A.R

First of, we have F.E.A.R. This game prove to be very scalable, graphically speaking. We ran test three times, computing an average of three runs. We also compute the standard deviation of those three runs - this is very likely the normal variations between runs. To get a better idea of how these variations may effect result, we compare the normal variations with the average as a percentage. Here are the results..

FEAR Performance Test 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	104.82	105.75	104.51	105.02	0.48	0.46%
Minimum fps	41	40	42	41	0.67	1.63%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	101.33	99.95	101	100.76	0.54	0.54%
Minimum fps	45	40	41	42	2	4.76%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	92.6	94.24	93.47	93.44	0.56	0.60%
Minimum fps	33	39	42	38	3.33	8.77%

FEAR Performance Test 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	64.27	65.33	64.87	64.82	0.37	0.57%
Minimum fps	39	40	41	40	0.67	1.67%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	60	60.36	59.6	59.99	0.26	0.43%
Minimum fps	36	36	35	35.67	0.44	1.25%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	54.55	53.51	54.05	54.04	0.35	0.65%
Minimum fps	32	31	31	31.33	0.44	1.42%

all numbers are in frame rates (except for percentages)

F.E.A.R Performance Test is unique as benchmarks go. While it's still is basically a replay,. there's always some differences on each run because of the game's physics engine. Let's talk about variations between runs. At 1024 x 768, there's still some areas in the performance test where we are slightly more system bound than others. At 1600 x 1200, we're practically graphics bound all the time, that's why variations are smaller - particularly for minimum fps.


FEAR Performance Test 1024

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	105.02	100.76	93.44		11.59	7.32	4.27		12.40%	7.84%	4.23%
Minimum fps	41	42	38		3	4	-1		7.89%	10.53%	-2.38%

FEAR Performance Test 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	64.82	59.99	54.04		10.79	5.95	4.84		19.96%	11.01%	8.06%
Minimum fps	40	35.67	31.33		8.67	4.33	4.33		27.66%	13.83%	12.15%

all numbers are in frame rates (except for percentages)

At 1024 x 768, we're not seeing that much of difference, only about 12 percent from 4 to 6 quads (or 16 to 24 pixel pipelines, if you will). Any difference seem to occur on areas where we are less system bound - corridors and rooms where there are no firefights or explosions. So, is a more powerful graphics card useless to get higher minimum frame rates in F.E.A.R? No, because, as we said earlier, we are more most likely system limited here and that occurs mostly on parts where we're seeing much lower frame rates than the average fps.

Once we become more graphically bound, we're seeing a much different picture. We experience a jump of around 20 percent from 4 to 6 quads, but more importantly, minimum frame rates are also higher with 6 quads (27 percent). Nominally, 40 fps is actually the lowest you can go in F.E.A.R without noticing too much lag. Your mileage may vary though.

The statistics and graph above quite effectively shows us that to benchmark graphics cards, you have to use settings and resolutions that's more graphically bound than system bound. However, it's not good enough to show where the difference between 4, 5 and 6 quads lie - we have to find situations which are more graphically intensive. So, we've decided to dig a little deeper and split the F.E.A.R Performance Test into two separate parts. The first part runs from the beginning to when the firefight ended - the picture below.



The second part begins after that picture (zooming through the corridor) all the way to the end. With no AI and a lot less physics, we expect this part to scale better graphically. However, we are more interested in seeing results from the first part, because this is the part where we most likely encounter lower frame rates. The previous graph is enough evidence to support this. Since we want to be more graphically bound, we're only going to look at 1600 x 1200 results.


FEAR Performance Test 1 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	51.09	48.85	45.34		5.74	3.51	2.24		12.66%	7.73%	4.58%
Minimum fps	28.79	27.53	25.56		3.24	1.98	1.26		12.66%	7.73%	4.58%

FEAR Performance Test 2 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	82.57	74.38	65.26		17.31	9.11	8.19		26.52%	13.96%	11.02%
Minimum fps	36.03	32.45	28.48		7.55	3.98	3.58		26.52%	13.96%	11.02%

all numbers are in frame rates (except for percentages)

Wow. F.E.A.R Performance Test practically behaves much like a truly synthetic benchmark. We see a similar increase in percentage on both minimum and average fps. Obviously, the graphics card is the determining factor at these settings. It's also obvious that the second part of the test is 'easier' on the graphics card than the first part.

Looking at the first part of the benchmark, we can see the performance difference between 4, 5 and 6 quads is not as big as in the second part. Remember, we 'removed' system limitations by running the benchmark at 1600 x 1200. If we were still system limited, we should be seeing the exact same frame rates as we saw at 1024 x 768. Important to note though - that minimum fps seem to occur on the very beginning in the benchmark - quite probably at the point where you see light shining through the ventilation duct (the one with the fan). This is where you most likely notice whether or not Volumetric Lights is set to 'on' or 'off'. Frame rate in the firefight scene generally stays above 35 fps, which is a good thing.

OK. We saw that in the F.E.A.R Performance Test, more pixel processing power isn't going to help a lot, particularly with resolutions as high as 1600 x 1200 (and above). Let's test that hypothesis with a gameplay testing session. Below you can results taken on F.E.A.R's Afterimage levels, using the same settings.

Afterimage 1 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	111.58	110.17	108.22	109.99	1.18	1.07%
Minimum fps	47	41	39	42.33	3.11	7.35%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	114.16	104.88	104.82	107.95	4.14	3.83%
Minimum fps	52	45	44	47	3.33	7.09%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	96.66	96.23	96.89	96.59	0.24	0.25%
Minimum fps	40	41	38	39.67	1.11	2.80%

Afterimage 1 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	63.87	67.12	67.57	66.18	1.54	2.33%
Minimum fps	28	33	33	31.33	2.22	7.09%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	62.04	59.72	58.63	60.13	1.27	2.12%
Minimum fps	31	27	27	28.33	1.78	6.27%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	53.4	55.6	52.36	53.79	1.21	2.25%
Minimum fps	22	26	26	24.67	1.78	7.21%

all numbers are in frame rates (except for percentages)

Although the length of the level helped in keeping variation levels low, average fps wise, we can see minimum frame rates have higher variations than average fps. This is expected with gameplay testing session results - it lacks the precise repeatability of a timedemo / replay in F.E.A.R Performance Test. Next, we have results from the second part of the Afterimage level.

Afterimage 2 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	98.99	102.75	98.61	100.12	1.76	1.75%
Minimum fps	38	51	41	43.33	5.11	11.79%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	98.24	96.35	98.6	97.73	0.92	0.94%
Minimum fps	46	43	43	44	1.33	3.03%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	90.53	95.13	90.21	91.96	2.11	2.30%
Minimum fps	40	43	35	39.33	2.89	7.34%

Afterimage 2 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	64.39	63.43	63.43	63.75	0.43	0.67%
Minimum fps	28	28	26	27.33	0.89	3.25%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	58.74	56.2	58.07	57.67	0.98	1.69%
Minimum fps	21	24	25	23.33	1.56	6.67%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	47.61	46.94	47.62	47.39	0.3	0.63%
Minimum fps	18	19	21	19.33	1.11	5.75%

all numbers are in frame rates (except for percentages)

Variation wise, we're seeing a similar trend with the second part of Afterimage. In fact, variations in minimum fps seems to be higher than in the first part. Running at 1600 x 1200 is enough to keep it under 10 percent. One thing for sure - minimum fps are nowhere near the levels we're seeing in F.E.A.R Performance Test. Instead of a near perfect 40 fps minimum with a higher clocked, 6 quad GeForce 7900GT, we're seeing minimum fps of 26 to 33 fps.

Afterimage 1024

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	105.05	102.84	94.28		10.78	8.57	2.21		11.43%	9.09%	2.15%
Minimum fps	42.83	45.5	39.5		3.33	6	-2.67		8.44%	15.19%	-5.86%

Afterimage 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	64.97	58.9	50.59		14.38	8.31	6.07		28.42%	16.43%	10.30%
Minimum fps	29.33	25.83	22		7.33	3.83	3.5		33.33%	17.42%	13.55%

all numbers are in frame rates (except for percentages)

The numbers in the above tables is an estimate for the entire Afterimage level, computed by averaging the results from the first and second part. Look back at the results we got from F.E.A.R Performance Test - the major difference between the two tables are minimum fps at 1600 x 1200. This confirms our earlier conclusion in our last article - F.E.A.R Performance Test doesn't always represent frame rates you see in actual gameplay.

That raises another question - we've seen that minimum fps in F.E.A.R Performance Test does not occur in firefights. So, where are we seeing / experiencing lower and minimum fps in Afterimage with gameplay testing sessions? Let's take a look at both parts of Afterimage to answer that question.