Tech-Hounds.com

The Nexus (Focused Timedemo) - 1024 AA AF

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	74.52	72.88	74.28	73.9	0.68	0.91%
Minimum fps	51	51	51	51	0	0.00%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	74.58	74.45	74.07	74.36	0.2	0.27%
Minimum fps	52	52	52	52	0	0.00%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	71.84	71.96	71.94	71.91	0.05	0.06%
Minimum fps	51	52	51	51.33	0.44	0.87%

The Nexus (Focused Timedemo) - 1600 AA AF

6 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	50.18	49.64	49.63	49.82	0.24	0.49%
Minimum fps	33	32	32	32.33	0.44	1.37%
5 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	45.67	45.64	45.68	45.66	0.02	0.03%
Minimum fps	29	29	29	29	0	0.00%
4 Quads
	1st	2nd	3rd	Average	Variations	Variations (Percent)
Average fps	43.79	43.81		43.8	0.01	0.03%
Minimum fps	28	28		28	0	0.00%

all numbers are in frame rates (except for percentages)

Variations can't get much lower than that. Since they are so low, we've decided to skip an additional run for 4 quads - we won't need a third run to confirm the results of the second. Let's look at how frame rate progress throughout these tests.


The Nexus (Focused Timedemo) - 1024 AA AF

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	73.9	74.36	71.91		1.98	2.45	-0.47		2.76%	3.41%	-0.63%
Minimum fps	51	52	51.33		-0.33	0.67	-1		-0.65%	1.30%	-1.92%

The Nexus (Focused Timedemo) - 1600 AA AF

	6 Quads	5 Quads	4 Quads		4 to 6	4 to 5	5 to 6		4 to 6 (Percent)	4 to 5 (Percent)	5 to 6 (Percent)
Average fps	49.82	45.66	43.8		6.02	1.86	4.15		13.73%	4.25%	9.09%
Minimum fps	32.33	29	28		4.33	1	3.33		15.48%	3.57%	11.49%

all numbers are in frame rates (except for percentages)

Despite the relatively same level of performance, we can see there's quite a difference in the graphs above. Even at a much more system limited 1024 x 768, we can already see the more powerful setup doesn't drop frames as much as the 4 quads setup. The 1600 x 1200 graph confirm this trend. If we were to use results from gameplay testing sessions, the graphs would not be so clear, but we will see a similar trend.

So, will more pixel processing power help maintain a sustained higher minimum fps in Quake 4? The results seems to indicate that it does and this applies to firefight scenes as well. There's basically only one area in this test that scale most with resolution - where you are viewing or shooting at The Nexus brain. Although the first battle have lower frame rates at 1024 x 768, it didn't drop as much when we ran the test at 1600 x 1200. Even at that point, the 5 and 6 quads results are noticeably higher than 4 quads, indicating that it also scale graphically.

Conclusion:

Replay / timedemo runs generally have far lower variations between runs, making conclusion easier. You can trust the results are accurate, with a standard deviation / variations between runs lower than 5 percent. Though gameplay testing sessions results are more representative of actual gameplay, they can be misleading in the sense you must take into account higher normal variations between runs. There are several things you could do to minimize that effect. You may take more samples (increase the number of runs) and compute an average of the results of those runs. You can also narrow down the scope of the test, minimizing the effects of unrelated, system limited scenes from the results. As an additional measure, you can also try to visualize the frame rate in an fps progress graph so you can make intuitive judgment and 'spot' a trend.

However, this is still not an ideal solution, particularly if you want a more precise results. Evening out the graph may help 'absorb' some variations, but that means you also absorb variations into the final average frame rate. There's also repeatability - there's no guarantee the runs will take the same amount of time.

In regards to F.E.A.R, it's obvious that minimum frame rate in actual gameplay and F.E.A.R Performance Test can be significantly different in some situations. F.E.A.R Performance Test shows that you can still play at 1600 x 1200 with our test setup, but gameplay testing sessions prove that is not always the case. In F.E.A.R Performance Test, a 1/6 drop (or a 15 percent difference) of pixel processing power nets you a minimum fps that's 5 frames lower, but the nominal value is still 35 fps. Gameplay testing sessions indicate that nominal values are much lower than that (25 to 27 fps). You'll definitely experience a more noticeable lag with those frame rate than with 35 fps. What makes this more important is that in these situations (like we saw with Afterimage 2-1 and 2-2) having a more powerful graphics card can help. Even when variations between runs are taken into account, the difference is still noticeable.

With that in mind, it would be more 'accurate' to compare graphics cards with gameplay testing sessions in scenes or situations like Afterimage 2-1 and 2-2 than just using F.E.A.R Performance Test. In addition to getting minimum frame rates that are actual gameplay frame rates, we will get an idea of what kind of performance these graphics card offer in more graphically intensive situations than F.E.A.R Performance Test - where frame rates are usually at their lowest. It simply reinforces the argument to go with a faster card - because that additional pixel processing power will be handy when you most needed it (as long as the influencing factor is the graphics card). The trade off is of course precision. We can not pinpoint  the exact minimum fps for those particular gameplay testing sessions. Using the lowest or even an average is usually used to represent minimum fps but we think it's wiser to include a range rather than an exact number.

In regards to Quake 4, results from timedemo runs, either with a third party app like FRAPS or the game's internal benchmark tool will always be different from actual frame rate in gameplay. There are several different reasons - first, Quake 4 is capped at 60 fps by default. Second, even if we remove the frame rate cap, gameplay testing sessions are, well, gameplay. Timedemo runs is not burdened with AI, physics and audio routines so performance influencing factors in a timedemo will either be the graphics card, processor and memory (bandwidth). Because of this, it is easier to isolate the graphics card and measure its performance - just run the timedemo at different resolutions. Any difference will likely be graphics related, not system related. Vice versa, a lack of difference (or just a small one) would likely mean the graphics card is being held up by other system components.

Gameplay testing sessions results at 1024 x 768 and 1600 x 1200 is generally higher than timedemo runs, This indicates that we're most likely system limited. Because of this, we opted to use a timedemo run to focus on graphics card performance. It is a trade off, but this way it is more easier to spot areas where pixel processing power comes into play. In one of the most graphically intensive levels, we found graphics cards with more pixel processing power is able to offer a much more 'sustained' frame rate. In addition to helping achieve higher minimum fps at some instances, more pixel processing power also helps the graphics card 'bounce' back to higher frame rates.  We wouldn't know this if we only look at the nominal results, where even a drop of 15 percent in pixel processing power is shown only as a 2 to 4 fps difference. The question is whether or not we will be able to see the difference with gameplay testing sessions. Though we didn't include the results in this article, we can tell you the results are very system limited - thus we were not able to spot a trend with those results. The reason: there are higher variations between each gameplay testing session results.

[Previous Page]
[Go to top]
[Home]