Tech-Hounds.com

Benchmarking on Trial - Part 2

Gamers usually want high enough frame rates for comfortable gameplay and that means less low frame rates and stutter (sharp drops in frame rates). Low frame rates can be caused by a variety of factors, including but not limited to graphics cards. Running benchmarks does provide an idea what kind of performance we can expect. After all, benchmarks was originally used to pinpoint system performance bottlenecks. However, benchmark results alone is not enough to make a judgment of performance.

What do we actually want to achieve with a benchmark? Since we are talking about graphics cards, we're talking about graphics benchmarks. A graphics benchmark is a benchmark that focuses on graphics card's performance. So we want to measure the graphics card's performance. In this case, it would be best to remove any system limitations that may hinder the card from using all its potential. However, that's not entirely possible with game benchmarks. Even timedemo runs in game benchmarks are not immune to system limitations, but it is as close as we can get without resorting to a pure synthetic graphics benchmark. Gameplay testing sessions will more likely be heavily affected by system limitations than timedemo runs.

Experience from our last article shows us that test results from replays / timedemo runs can be significantly different at times. That's actually pretty natural because the game may be graphically intensive in some areas and more system intensive in others. The benchmark may also not be 'heavy' enough to represent the most graphically intensive situations and scenes you will encounter throughout the entire game. That's one thing we want to explore more deeply in this article.

There's also the issue of gameplay relevance. Results that show significant difference between two cards on less graphically situations is not going to accurately portray what these two cards will do in a much graphically intensive scenes. Results from a graphics benchmark that 'capture' parts of the game with the lowest frame rates (that's graphically related) is more informative in examining the differences (if any) between two cards.

The purpose? To find out what kind of performance difference we can expect between two graphics cards that have varying degrees of pixel processing power. We've narrowed down the game and the game levels we're going to use in the previous article. This time we're going to focus on finding situations and scenes where we most likely have lower / lowest frame rates in the game. However, the results must indicate that particular scene scale with more pixel processing power - a common focus point determining graphics cards performance.

For this test, we're still using the same test setup we used in the first part of the article. To vary the degree of pixel processing power, we disable one and two quads of the GeForce 7900GT we used. So, effectively we have a GeForce 7900GT with 4, 5 and 6 quads. The 4 and 5 quads setup can be considered a proxy for a graphics card that' has slightly lower pixel processing power - 33 and 16 percent slower, respectively.

F.E.A.R

First of, we have F.E.A.R. This game prove to be very scalable, graphically speaking. We ran test three times, computing an average of three runs. We also compute the standard deviation of those three runs - this is very likely the normal variations between runs. To get a better idea of how these variations may effect result, we compare the normal variations with the average as a percentage. Here are the results..

FEAR Performance Test 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	104.82	105.75	104.51	105.02	0.48	0.46%
Minimum fps	41	40	42	41	0.67	1.63%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	101.33	99.95	101	100.76	0.54	0.54%
Minimum fps	45	40	41	42	2	4.76%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	92.6	94.24	93.47	93.44	0.56	0.60%
Minimum fps	33	39	42	38	3.33	8.77%

FEAR Performance Test 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	64.27	65.33	64.87	64.82	0.37	0.57%
Minimum fps	39	40	41	40	0.67	1.67%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	60	60.36	59.6	59.99	0.26	0.43%
Minimum fps	36	36	35	35.67	0.44	1.25%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	54.55	53.51	54.05	54.04	0.35	0.65%
Minimum fps	32	31	31	31.33	0.44	1.42%

all numbers are in frame rates (except for percentages)

F.E.A.R Performance Test is unique as benchmarks go. While it's still is basically a replay,. there's always some differences on each run because of the game's physics engine. Let's talk about variations between runs. At 1024 x 768, there's still some areas in the performance test where we are slightly more system bound than others. At 1600 x 1200, we're practically graphics bound all the time, that's why variations are smaller - particularly for minimum fps.


FEAR Performance Test 1024

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	105.02	100.76	93.44		11.59	7.32	4.27		12.40%	7.84%	4.23%
Minimum fps	41	42	38		3	4	-1		7.89%	10.53%	-2.38%

FEAR Performance Test 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	64.82	59.99	54.04		10.79	5.95	4.84		19.96%	11.01%	8.06%
Minimum fps	40	35.67	31.33		8.67	4.33	4.33		27.66%	13.83%	12.15%

all numbers are in frame rates (except for percentages)

At 1024 x 768, we're not seeing that much of difference, only about 12 percent from 4 to 6 quads (or 16 to 24 pixel pipelines, if you will). Any difference seem to occur on areas where we are less system bound - corridors and rooms where there are no firefights or explosions. So, is a more powerful graphics card useless to get higher minimum frame rates in F.E.A.R? No, because, as we said earlier, we are more most likely system limited here and that occurs mostly on parts where we're seeing much lower frame rates than the average fps.

Once we become more graphically bound, we're seeing a much different picture. We experience a jump of around 20 percent from 4 to 6 quads, but more importantly, minimum frame rates are also higher with 6 quads (27 percent). Nominally, 40 fps is actually the lowest you can go in F.E.A.R without noticing too much lag. Your mileage may vary though.

The statistics and graph above quite effectively shows us that to benchmark graphics cards, you have to use settings and resolutions that's more graphically bound than system bound. However, it's not good enough to show where the difference between 4, 5 and 6 quads lie - we have to find situations which are more graphically intensive. So, we've decided to dig a little deeper and split the F.E.A.R Performance Test into two separate parts. The first part runs from the beginning to when the firefight ended - the picture below.



The second part begins after that picture (zooming through the corridor) all the way to the end. With no AI and a lot less physics, we expect this part to scale better graphically. However, we are more interested in seeing results from the first part, because this is the part where we most likely encounter lower frame rates. The previous graph is enough evidence to support this. Since we want to be more graphically bound, we're only going to look at 1600 x 1200 results.


FEAR Performance Test 1 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	51.09	48.85	45.34		5.74	3.51	2.24		12.66%	7.73%	4.58%
Minimum fps	28.79	27.53	25.56		3.24	1.98	1.26		12.66%	7.73%	4.58%

FEAR Performance Test 2 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	82.57	74.38	65.26		17.31	9.11	8.19		26.52%	13.96%	11.02%
Minimum fps	36.03	32.45	28.48		7.55	3.98	3.58		26.52%	13.96%	11.02%

all numbers are in frame rates (except for percentages)

Wow. F.E.A.R Performance Test practically behaves much like a truly synthetic benchmark. We see a similar increase in percentage on both minimum and average fps. Obviously, the graphics card is the determining factor at these settings. It's also obvious that the second part of the test is 'easier' on the graphics card than the first part.

Looking at the first part of the benchmark, we can see the performance difference between 4, 5 and 6 quads is not as big as in the second part. Remember, we 'removed' system limitations by running the benchmark at 1600 x 1200. If we were still system limited, we should be seeing the exact same frame rates as we saw at 1024 x 768. Important to note though - that minimum fps seem to occur on the very beginning in the benchmark - quite probably at the point where you see light shining through the ventilation duct (the one with the fan). This is where you most likely notice whether or not Volumetric Lights is set to 'on' or 'off'. Frame rate in the firefight scene generally stays above 35 fps, which is a good thing.

OK. We saw that in the F.E.A.R Performance Test, more pixel processing power isn't going to help a lot, particularly with resolutions as high as 1600 x 1200 (and above). Let's test that hypothesis with a gameplay testing session. Below you can results taken on F.E.A.R's Afterimage levels, using the same settings.

Afterimage 1 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	111.58	110.17	108.22	109.99	1.18	1.07%
Minimum fps	47	41	39	42.33	3.11	7.35%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	114.16	104.88	104.82	107.95	4.14	3.83%
Minimum fps	52	45	44	47	3.33	7.09%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	96.66	96.23	96.89	96.59	0.24	0.25%
Minimum fps	40	41	38	39.67	1.11	2.80%

Afterimage 1 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	63.87	67.12	67.57	66.18	1.54	2.33%
Minimum fps	28	33	33	31.33	2.22	7.09%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	62.04	59.72	58.63	60.13	1.27	2.12%
Minimum fps	31	27	27	28.33	1.78	6.27%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	53.4	55.6	52.36	53.79	1.21	2.25%
Minimum fps	22	26	26	24.67	1.78	7.21%

all numbers are in frame rates (except for percentages)

Although the length of the level helped in keeping variation levels low, average fps wise, we can see minimum frame rates have higher variations than average fps. This is expected with gameplay testing session results - it lacks the precise repeatability of a timedemo / replay in F.E.A.R Performance Test. Next, we have results from the second part of the Afterimage level.

Afterimage 2 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	98.99	102.75	98.61	100.12	1.76	1.75%
Minimum fps	38	51	41	43.33	5.11	11.79%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	98.24	96.35	98.6	97.73	0.92	0.94%
Minimum fps	46	43	43	44	1.33	3.03%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	90.53	95.13	90.21	91.96	2.11	2.30%
Minimum fps	40	43	35	39.33	2.89	7.34%

Afterimage 2 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	64.39	63.43	63.43	63.75	0.43	0.67%
Minimum fps	28	28	26	27.33	0.89	3.25%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	58.74	56.2	58.07	57.67	0.98	1.69%
Minimum fps	21	24	25	23.33	1.56	6.67%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	47.61	46.94	47.62	47.39	0.3	0.63%
Minimum fps	18	19	21	19.33	1.11	5.75%

all numbers are in frame rates (except for percentages)

Variation wise, we're seeing a similar trend with the second part of Afterimage. In fact, variations in minimum fps seems to be higher than in the first part. Running at 1600 x 1200 is enough to keep it under 10 percent. One thing for sure - minimum fps are nowhere near the levels we're seeing in F.E.A.R Performance Test. Instead of a near perfect 40 fps minimum with a higher clocked, 6 quad GeForce 7900GT, we're seeing minimum fps of 26 to 33 fps.

Afterimage 1024

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	105.05	102.84	94.28		10.78	8.57	2.21		11.43%	9.09%	2.15%
Minimum fps	42.83	45.5	39.5		3.33	6	-2.67		8.44%	15.19%	-5.86%

Afterimage 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	64.97	58.9	50.59		14.38	8.31	6.07		28.42%	16.43%	10.30%
Minimum fps	29.33	25.83	22		7.33	3.83	3.5		33.33%	17.42%	13.55%

all numbers are in frame rates (except for percentages)

The numbers in the above tables is an estimate for the entire Afterimage level, computed by averaging the results from the first and second part. Look back at the results we got from F.E.A.R Performance Test - the major difference between the two tables are minimum fps at 1600 x 1200. This confirms our earlier conclusion in our last article - F.E.A.R Performance Test doesn't always represent frame rates you see in actual gameplay.

That raises another question - we've seen that minimum fps in F.E.A.R Performance Test does not occur in firefights. So, where are we seeing / experiencing lower and minimum fps in Afterimage with gameplay testing sessions? Let's take a look at both parts of Afterimage to answer that question.


Afterimage 1 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	66.18	60.13	53.79		12.4	6.34	6.05		23.04%	11.79%	10.07%
Minimum fps	31.33	28.33	24.67		6.67	3.67	3		27.03%	14.86%	10.59%

Afterimage 2 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	63.75	57.67	47.39		16.36	10.28	6.08		34.52%	21.69%	10.54%
Minimum fps	27.33	23.33	19.33		8	4	4		41.38%	20.69%	17.14%

all numbers are in frame rates (except for percentages)

On the first part of Afterimage, the graph shows a drop on the last part of the graph. That really did occur in a firefight - probably because we used one grenade and two anti personnel mine in succession to deal with 3 baddies. Well, the result is obvious - more explosions, debris and smoke than you see on F.E.A.R Performance Test. Since there's no other drop in the graph, we can safely assume that's where we hit those minimum fps. Checking the nominal frame rates, we can see going with 6 quads (more pixel processing power) will help maintain a slightly higher fps (31 fps as opposed to 28 fps and 24 fps).

Obviously, the second part of Afterimage is more graphically demanding than the first. There are three firefights here - the second part occurs in an atrium area with at least 5 baddies, including 1 mobile armor. Pardon the graph, it's crappy in resolution since we have to average frame rate from several seconds. Despite that, the graph does show a graphics card with a more powerful pixel processing power can maintain a higher fps in that firefight - 27 fps as opposed to 23 fps and 19 fps.

F.E.A.R Performance Test is not the best model of firefights for the entire game - it's obviously too 'light'. Now, we know minimum fps occur in firefights. However the graph we're using is not capable of showing that as much as we would like. So, we narrow down testing to include only firefights - separating the entire level into three parts - Afterimage 1-1 (in the first part), Afterimage 2-1 and 2-2 (in the second part). Let's take a look at Afterimage 1-1 first.

Afterimage 1-1 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	93.31	90.71	87.45	90.49	2.03	2.24%
Minimum fps	49	40	52	47	4.67	9.93%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	81.87	85.03	84.66	83.85	1.32	1.58%
Minimum fps	46	45	47	46	0.67	1.45%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	84.82	82.51	80.7	82.68	1.43	1.73%
Minimum fps	49	47	45	47	1.33	2.84%

Afterimage 1-1 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	58.62	56.55	55.88	57.02	1.07	1.88%
Minimum fps	30	29	28	29	0.67	2.30%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	49.29	53.47	53.51	52.09	1.87	3.59%
Minimum fps	27	26	25	26	0.67	2.56%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	47.44	48.69	48.35	48.16	0.48	1.00%
Minimum fps	26	25	26	25.67	0.44	1.73%

all numbers are in frame rates (except for percentages)

There are actually three firefights in this scene, two with ATC forces and one with ATC plus Replica soldiers (the one we mentioned earlier). There are some brief interludes of walking / running between firefights, so that may have an effect of 'inflating' the average fps. As we expected, at 1024 x 768 variations can be quite high for minimum fps but otherwise the levels are close enough to frame rates we're seeing for the first part of Afterimage and F.E.A.R Performance Test. However, things change quite a bit at 1600 x 1200. Comparing minimum fps, it's now clear lower / minimum frame rates did occur on this part of the test. Let's look at the graph to see things a little bit better - we will only be looking at 1600 x 1200 results where we are much more graphically limited than at 1024 x 768.

Afterimage 1-1 1024

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	90.49	83.85	82.68		7.81	1.18	6.64		9.45%	1.42%	7.91%
Minimum fps	47	46	47		0	-1	1		0.00%	-2.13%	2.17%

Afterimage 1-1 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	57.02	52.09	48.16		8.86	3.92	4.93		18.39%	8.15%	9.47%
Minimum fps	29	26	25.67		3.33	0.33	3		12.99%	1.30%	11.54%

all numbers are in frame rates (except for percentages)

for a more clear graph, click here.

Well, it looks like having more pixel processing power isn't going to help that much, but it may be enough depending on how you cope with lower than ideal minimum fps. In a slightly different note, it looks like loading a save game may have some effect on frame rate - we saw quite a drop in the first part of this test. That does seem to interfere with the results, since here we're seeing 26 fps on the 5 quads results. Minimum fps on the firefight is actually 29 fps or close to 6 quads. Of course, you should take into account variations, meaning the minimum fps is more of a range than an exact number (28 - 30 fps for 6 quads).

Afterimage 2-1 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	96.45	93.49	89.94	93.29	2.24	2.40%
Minimum fps	61	52	45	52.67	5.56	10.55%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	82.76	86.52	88.51	85.93	2.11	2.46%
Minimum fps	46	40	45	43.67	2.44	5.60%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	84.91	85.21	83.79	84.64	0.56	0.67%
Minimum fps	41	43	43	42.33	0.89	2.10%

Afterimage 2-1 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	64.14	65.1	63.19	64.14	0.64	1.00%
Minimum fps	34	32	25	30.33	3.56	11.72%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	58.98	62.11	56.89	59.33	1.85	3.12%
Minimum fps	28	28	25	27	1.33	4.94%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	49.44	51.96	50.11	50.5	0.97	1.92%
Minimum fps	25	28	27	26.67	1.11	4.17%

all numbers are in frame rates (except for percentages)

The second firefight seems to be 'lighter' on the graphical side at 1024 x 768 with 6 quads. We're getting consistently higher minimum fps here. However, we also have much higher variations here as well. Oddly enough, this is also present at 1600 x 1200, though it's generally lower with 4 and 5 quads. Let's take a closer look at this test..

Afterimage 2-1 1024

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	93.29	85.93	84.64		8.65	1.29	7.36		10.22%	1.53%	8.57%
Minimum fps	52.67	43.67	42.33		10.33	1.33	9		24.41%	3.15%	20.61%

Afterimage 2-1 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	64.14	59.33	50.5		13.64	8.82	4.82		27.00%	17.46%	8.12%
Minimum fps	30.33	27	26.67		3.67	0.33	3.33		13.75%	1.25%	12.35%

all numbers are in frame rates (except for percentages)

for a more clear graph, click here.

Well, this is one of the caveat of gameplay testing sessions - the results you get don't always match second by second. This may have some effect on what kind of conclusion we can make. The test seem to indicate that this particular firefight scales well with pixel processing power. So, having more of it may help you get higher frame rates. The 6 quads setup is able to maintain around 39 fps - which is very significant to what we're seeing on the 4 and 5 quads results (around 30 fps). 

Afterimage 2-2 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	72.43	72.27	79.88	74.86	3.35	4.47%
Minimum fps	49	54	48	50.33	2.44	4.86%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	61.93	68.96	60.79	63.89	3.37	5.28%
Minimum fps	40	43	38	40.33	1.78	4.41%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	59.88	61.48		60.68	0.8	1.32%
Minimum fps	41	44		42.5	1.5	3.53%

Afterimage 2-2 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	41.9	40.9	38.94	40.58	1.09	2.70%
Minimum fps	27	30	27	28	1.33	4.76%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	37.15	38.77	33.8	36.57	1.85	5.06%
Minimum fps	25	26	20	23.67	2.44	10.33%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	32.44	34.54	32.88	33.29	0.84	2.51%
Minimum fps	23	24	19	22	2	9.09%

all numbers are in frame rates (except for percentages)

Variations are notably higher than both Afterimage 1-1 and 2-1 and that applies to both average and minimum fps. However, most of the differing values are lower, so it's pretty likely the two, closer values are more accurate. What complicates things is that variations tend to be higher with higher resolutions (1600 x 1200). It will be harder to conclude anything from these numbers alone. 

Afterimage 2-2 1024

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	74.86	63.89	60.68		14.18	3.22	10.96		23.37%	5.30%	17.16%
Minimum fps	50.33	40.33	42.5		7.83	-2.17	10		18.43%	-5.10%	24.79%

Afterimage 2-2 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	40.58	36.57	33.29		7.29	3.28	4.01		21.89%	9.86%	10.96%
Minimum fps	28	23.67	22		6	1.67	4.33		27.27%	7.58%	18.31%

all numbers are in frame rates (except for percentages)

for a more clear graph, click here.

You can see just how messy gameplay testing session results can be. There's just so much variations between runs in this particular test that it's hard to make sense of anything without seeing the graph. From the graph, we can see that having more pixel processing can provide you with higher minimum fps. Look how even the lowest points of the graph for 6 quads is noticeably higher than minimum points on 5 and 4 quads. The 6 quads frame rate stays above 30 fps most of the time, while 4 and 5 quads dips below 30 fps on several occasions. However, keep in mind that because variations are higher than ideal in this test, it's not exact. 6 quads minimum fps can range from 27 to 30 fps. Of course, 4 and 5 quads tend to be lower than that.

Quake 4

Just like what we did with F.E.A.R, we played through the level three times at different resolutions, recording frame rates for each run. You can see the results in the table below - we're focusing on variations between runs here.

Data Processing Terminal (Gameplay) - 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	119.78	119.53	118.56	119.29	0.49	0.41%
Minimum fps	69	68	67	68	0.67	0.98%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	119.44	119.53	119.7	119.56	0.09	0.08%
Minimum fps	69	68	67	68	0.67	0.98%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	119.56	119.41	119.16	119.38	0.15	0.12%
Minimum fps	69	69	68	68.67	0.44	0.65%

Data Processing Terminal (Gameplay) - 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	106.06	105.53	105.68	105.75	0.2	0.19%
Minimum fps	66	64	65	65	0.67	1.03%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	103.19	101.46	101.95	102.2	0.66	0.65%
Minimum fps	67	67	66	66.67	0.44	0.67%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	98.36	97.46	97.67	97.83	0.35	0.36%
Minimum fps	65	65	65	65	0	0.00%

all numbers are in frame rates (except for percentages)

This is one of Quake 4 levels that is more responsive to resolution changes, the results seems to show that at 1600 x 1200, having more pixel processing power isn't going to give you much. The sweet spot with this graphics card seems to 1280 x 1024, which unfortunately we didn't use for this article.

Variations between runs are minimal, despite the fact that these are gameplay testing sessions. At 1600 x 1200, the differences between 6, 5 and 4 quads is minimal. Minimum frame rates are generally the same, which leads to the conclusion that even at this setting Quake 4 is system limited (though less so than 1024 x 768). Below is the graph for this test.


Data Processing Terminal (Gameplay) - 1024

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	119.29	119.56	119.38		-0.08	0.18	-0.26		-0.07%	0.15%	-0.22%
Minimum fps	68	68	68.67		-0.67	-0.67	0		-0.97%	-0.97%	0.00%

Data Processing Terminal (Gameplay) - 1600

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	105.75	102.2	97.83		7.89	4.37	3.52		8.07%	4.47%	3.45%
Minimum fps	65	67.67	65.67		-0.67	2	-2.67		-1.02%	3.05%	-3.94%

all numbers are in frame rates (except for percentages)

By removing the frame cap in Quake 4, we allow the system to render the scene as fast as it can - much like a timedemo. Because of that, it's actually pretty easy to spot which setup is the fastest with Quake 4 - you pick the setup with the shortest run. At 1600 x 1200, where we are not as system limited, the 6 quads setup is obviously faster. Although it has basically the same minimum fps, it 'bounces back' to higher frame rates quicker - thus the shorter run. A timedemo run at these settings would likely give higher results, because the system doesn't have to do AI, physics and sound.

Now, we're going to take a look at a level that's more representative of frame rates you'll likely see through the entire game (though not necessarily represent the gameplay). The Nexus level is the ending boss level, with an ending cutscene. While it doesn't have a checkpoint save, we can generally separate it into two parts - the cutscene and gameplay. First of, we'll be looking at the entire level.

The Nexus (Gameplay) - 1024

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	84.79	86.58	87.1	86.16	0.91	1.06%
Minimum fps	42	46	48	45.33	2.22	4.90%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	88.93	84.98	86.96	86.96	1.97	2.27%
Minimum fps	53	45	49	49	4	8.16%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	83.58	82.31	85.03	83.64	0.93	1.11%
Minimum fps	47	50	45	47.33	1.78	3.76%

The Nexus (Gameplay) - 1600

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	71.81	71.65	72.5	71.99	0.34	0.47%
Minimum fps	41	42	40	41	0.67	1.63%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	68.16	72.29	69.66	70.04	1.5	2.15%
Minimum fps	41	43	38	40.67	1.78	4.37%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	68.29	67.87	68.08	68.08	0.21	0.31%
Minimum fps	41	41	41	41	0	0.0%

all numbers are in frame rates (except for percentages)

As you can see, even at 1600 x 1200, there's little difference between 5 and 6 quads. The graphics card must be substantially slower at these settings for us to spot a difference. It's not until we drop pixel processing power by 1/3 (to 4 quads) do we see any change in frame rate. Variations are higher than Data Processing Terminal, but it is lower at 1600 x 1200. Since we're system limited, there will be higher variations with a more powerful graphics card - simply because it's just waiting for the rest of the system to catch up. We could just turn on AA and AF to make this test more graphically intensive, but that would mean we include the penalty of AA and AF into the mix. Going higher in resolution is also not possible, we don't have a monitor capable of higher resolutions than 1600 x 1200.


The Nexus (Gameplay) - 1024

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	86.16	86.96	83.64		2.51	3.31	-0.8		3.00%	3.96%	-0.92%
Minimum fps	45.33	49	47.33		-2	1.67	-3.67		-4.23%	3.52%	-7.48%

The Nexus (Gameplay) - 1600 AA AF

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	71.99	70.04	68.08		3.91	1.96	1.95		5.74%	2.87%	2.79%
Minimum fps	41	40.67	41		0	-033	0.33		0.00%	-0.82%	0.82%

all numbers are in frame rates (except for percentages)

The results in the previous table don't show it, but we're seeing a similar situation at 1024 x 768 here like we saw with Data Processing Terminal at 1600 x 1200. The more powerful graphics card (6 quads) is quicker in pushing higher frame rates after lower / minimum frame rates in this test. However, since we're not even putting that much of a burden on the graphics card, we need to use a higher resolution (1600 x 1200). At this setting, it's slightly easier to understand what's happening though normal variations is causing quite a mess. It's easy to see the more powerful setup can maintain a significantly higher frame rate. Just look at the drop in the end of the graph.

However, there is a problem with that assessment - that drop occurred in the later parts of the test - the cutscene parts. Generally, we don't really care about cutscenes, we want frame rate in gameplay situations. So, we narrowed down the scope of the test - from the very beginning to just before the cutscene started (when The Nexus brain exploded). We also opted to use a timedemo instead of doing a gameplay testing session - we want to see what the graphics card can do in less system limited situations.

The Nexus (Focused Timedemo) - 1024 AA AF

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	74.52	72.88	74.28	73.9	0.68	0.91%
Minimum fps	51	51	51	51	0	0.00%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	74.58	74.45	74.07	74.36	0.2	0.27%
Minimum fps	52	52	52	52	0	0.00%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	71.84	71.96	71.94	71.91	0.05	0.06%
Minimum fps	51	52	51	51.33	0.44	0.87%

The Nexus (Focused Timedemo) - 1600 AA AF

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	50.18	49.64	49.63	49.82	0.24	0.49%
Minimum fps	33	32	32	32.33	0.44	1.37%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	45.67	45.64	45.68	45.66	0.02	0.03%
Minimum fps	29	29	29	29	0	0.00%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	43.79	43.81		43.8	0.01	0.03%
Minimum fps	28	28		28	0	0.00%

all numbers are in frame rates (except for percentages)

Variations can't get much lower than that. Since they are so low, we've decided to skip an additional run for 4 quads - we won't need a third run to confirm the results of the second. Let's look at how frame rate progress throughout these tests.


The Nexus (Focused Timedemo) - 1024 AA AF

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	73.9	74.36	71.91		1.98	2.45	-0.47		2.76%	3.41%	-0.63%
Minimum fps	51	52	51.33		-0.33	0.67	-1		-0.65%	1.30%	-1.92%

The Nexus (Focused Timedemo) - 1600 AA AF

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	49.82	45.66	43.8		6.02	1.86	4.15		13.73%	4.25%	9.09%
Minimum fps	32.33	29	28		4.33	1	3.33		15.48%	3.57%	11.49%

all numbers are in frame rates (except for percentages)

Despite the relatively same level of performance, we can see there's quite a difference in the graphs above. Even at a much more system limited 1024 x 768, we can already see the more powerful setup doesn't drop frames as much as the 4 quads setup. The 1600 x 1200 graph confirm this trend. If we were to use results from gameplay testing sessions, the graphs would not be so clear, but we will see a similar trend.

So, will more pixel processing power help maintain a sustained higher minimum fps in Quake 4? The results seems to indicate that it does and this applies to firefight scenes as well. There's basically only one area in this test that scale most with resolution - where you are viewing or shooting at The Nexus brain. Although the first battle have lower frame rates at 1024 x 768, it didn't drop as much when we ran the test at 1600 x 1200. Even at that point, the 5 and 6 quads results are noticeably higher than 4 quads, indicating that it also scale graphically.

Conclusion:

Replay / timedemo runs generally have far lower variations between runs, making conclusion easier. You can trust the results are accurate, with a standard deviation / variations between runs lower than 5 percent. Though gameplay testing sessions results are more representative of actual gameplay, they can be misleading in the sense you must take into account higher normal variations between runs. There are several things you could do to minimize that effect. You may take more samples (increase the number of runs) and compute an average of the results of those runs. You can also narrow down the scope of the test, minimizing the effects of unrelated, system limited scenes from the results. As an additional measure, you can also try to visualize the frame rate in an fps progress graph so you can make intuitive judgment and 'spot' a trend.

However, this is still not an ideal solution, particularly if you want a more precise results. Evening out the graph may help 'absorb' some variations, but that means you also absorb variations into the final average frame rate. There's also repeatability - there's no guarantee the runs will take the same amount of time.

In regards to F.E.A.R, it's obvious that minimum frame rate in actual gameplay and F.E.A.R Performance Test can be significantly different in some situations. F.E.A.R Performance Test shows that you can still play at 1600 x 1200 with our test setup, but gameplay testing sessions prove that is not always the case. In F.E.A.R Performance Test, a 1/6 drop (or a 15 percent difference) of pixel processing power nets you a minimum fps that's 5 frames lower, but the nominal value is still 35 fps. Gameplay testing sessions indicate that nominal values are much lower than that (25 to 27 fps). You'll definitely experience a more noticeable lag with those frame rate than with 35 fps. What makes this more important is that in these situations (like we saw with Afterimage 2-1 and 2-2) having a more powerful graphics card can help. Even when variations between runs are taken into account, the difference is still noticeable.

With that in mind, it would be more 'accurate' to compare graphics cards with gameplay testing sessions in scenes or situations like Afterimage 2-1 and 2-2 than just using F.E.A.R Performance Test. In addition to getting minimum frame rates that are actual gameplay frame rates, we will get an idea of what kind of performance these graphics card offer in more graphically intensive situations than F.E.A.R Performance Test - where frame rates are usually at their lowest. It simply reinforces the argument to go with a faster card - because that additional pixel processing power will be handy when you most needed it (as long as the influencing factor is the graphics card). The trade off is of course precision. We can not pinpoint  the exact minimum fps for those particular gameplay testing sessions. Using the lowest or even an average is usually used to represent minimum fps but we think it's wiser to include a range rather than an exact number.

In regards to Quake 4, results from timedemo runs, either with a third party app like FRAPS or the game's internal benchmark tool will always be different from actual frame rate in gameplay. There are several different reasons - first, Quake 4 is capped at 60 fps by default. Second, even if we remove the frame rate cap, gameplay testing sessions are, well, gameplay. Timedemo runs is not burdened with AI, physics and audio routines so performance influencing factors in a timedemo will either be the graphics card, processor and memory (bandwidth). Because of this, it is easier to isolate the graphics card and measure its performance - just run the timedemo at different resolutions. Any difference will likely be graphics related, not system related. Vice versa, a lack of difference (or just a small one) would likely mean the graphics card is being held up by other system components.

Gameplay testing sessions results at 1024 x 768 and 1600 x 1200 is generally higher than timedemo runs, This indicates that we're most likely system limited. Because of this, we opted to use a timedemo run to focus on graphics card performance. It is a trade off, but this way it is more easier to spot areas where pixel processing power comes into play. In one of the most graphically intensive levels, we found graphics cards with more pixel processing power is able to offer a much more 'sustained' frame rate. In addition to helping achieve higher minimum fps at some instances, more pixel processing power also helps the graphics card 'bounce' back to higher frame rates.  We wouldn't know this if we only look at the nominal results, where even a drop of 15 percent in pixel processing power is shown only as a 2 to 4 fps difference. The question is whether or not we will be able to see the difference with gameplay testing sessions. Though we didn't include the results in this article, we can tell you the results are very system limited - thus we were not able to spot a trend with those results. The reason: there are higher variations between each gameplay testing session results.

Go to top