What is the most stable operation?

Ok, guys, this time we are testing four various
test methods temporal stability to find THE best
operation for CPU performance test.

makc3d

forked:2
favorite:0
lines:86
license : see code comments
modified : 2010-11-11 20:26:20

Embed: Tweet

package {
	import flash.display.Bitmap;
	import flash.display.BitmapData;
	import flash.display.Sprite;
	import flash.events.Event;
	import flash.filters.BlurFilter;
	import flash.geom.Rectangle;
	import flash.text.TextField;
	import flash.utils.getTimer;
	/**
	 * Ok, guys, this time we are testing four various
	 * test methods temporal stability to find THE best
	 * operation for CPU performance test.
	 */
	public class Foo extends Sprite {
		public var bd:BitmapData;
		public var bf:BlurFilter = new BlurFilter;
		public var tf:TextField;
		public var times:Vector.<Vector.<int>>;
		public var count:int = 0;
		public function Foo () {
			addChild (tf = new TextField);
			tf.autoSize = "left";

times = new Vector.<Vector.<int>> (4);
			for (var i:int = 0; i < times.length; i++) times [i] = new Vector.<int> (400);
			addEventListener (Event.ENTER_FRAME, loop);

// okay, okay, smaller size now to run faster :)
			bd = new BitmapData (1023, 1023);
		}
		public function loop (e:Event):void {
			tf.text = "Progress: " + Math.round (count * 0.25) + "%";

// get raw data 1st, 400 frames
			var t:int = getTimer ();
			bd.getColorBoundsRect (2, 2);
			bd.getColorBoundsRect (2, 2);
			bd.getColorBoundsRect (2, 2);
			bd.getColorBoundsRect (2, 2);
			bd.getColorBoundsRect (2, 2);
			times [0] [count] = getTimer () - t;

t += times [0] [count];
			bd.histogram ();
			times [1] [count] = getTimer () - t;

t += times [1] [count];
			bd.applyFilter (bd, bd.rect, bd.rect.topLeft, bf);
			times [2] [count] = getTimer () - t;

t += times [2] [count];
			bd.dispose (); bd = new BitmapData (1023, 1023);
			bd.dispose (); bd = new BitmapData (1023, 1023);
			bd.dispose (); bd = new BitmapData (1023, 1023);
			bd.dispose (); bd = new BitmapData (1023, 1023);
			bd.dispose (); bd = new BitmapData (1023, 1023);
			times [3] [count] = getTimer () - t;

count++;
			if (count == 400) {
				removeEventListener (Event.ENTER_FRAME, loop);
				processResults ();
			}
		}

public function processResults ():void {
			// min values for every 10 measurements (would-be test results)
			var mins:Vector.<Vector.<int>> = new Vector.<Vector.<int>> (times.length);
			for (var i:int = 0; i < times.length; i++) {
				mins [i] = new Vector.<int> (40);
				for (var j:int = 0; j < 400; j++) {
					var k:int = j / 10;
					if ((mins [i] [k] == 0) || (mins [i] [k] > times [i] [j])) {
						mins [i] [k] = times [i] [j];
					}
				}
			}
			// averages of min values
			var avgs:Vector.<int> = new Vector.<int> (mins.length);
			for (i = 0; i < mins.length; i++) {
				for (j = 0; j < mins [i].length; j++) {
					avgs [i] += mins [i] [j];
				}
				avgs [i] /= 40;
				// replace mins with squared deviations
				for (j = 0; j < mins [i].length; j++) {
					mins [i] [j] -= avgs [i];
					mins [i] [j] *= mins [i] [j];
				}
			}
			// coefficients of variation
			tf.text = "Coefficients of variation:";
			for (i = 0; i < mins.length; i++) {
				var sd:Number = 0;
				for (j = 0; j < mins [i].length; j++) {
					sd += mins [i] [j];
				}
				var cv:Number = Math.sqrt (sd) / 40 / avgs [i];
				tf.appendText ("\nMethod " + (i + 1) + ": " + cv.toFixed (6));
			}
			tf.appendText ("\n\nThanks!");
		}
	}
}

Code Fullscreen

Preview Fullscreen

Please copy-paste 4 numbers you get (smaller numbers mean better stability, if you care).

by makc3d
at 2010/11/11 12:27:44

Method 1: 0.000000
Method 2: 0.000893
Method 3: 0.002500
Method 4: 0.078062

by djankey
at 2010/11/11 12:35:00

0.000000 wtf?? o_O

by makc3d
at 2010/11/11 12:36:59

For FireFox running FP10.0 non-debug version:
Method 1: 0.027003
Method 2: 0.003499
Method 3: 0.004592
Method 4: 0.118732

by IQAndreas
at 2010/11/11 12:37:34

my results in ff on same machine
Method 1: 0.007454
Method 2: 0.002842
Method 3: 0.004588
Method 4: 0.025505
ha. well, this can only be sorted out with lots of results :/

by makc3d
at 2010/11/11 12:41:01

Win7, Intel 2.4GHz (Core2Duo), FP 10.1.85.3 debug, Chrome 9 dev.
Method 1: 0.000000
Method 2: 0.000000
Method 3: 0.000000
Method 4: 0.000000

by djankey
at 2010/11/11 12:43:00

Method 1: 0.022048
Method 2: 0.003598
Method 3: 0.001761
Method 4: 0.000000

by timknip2
at 2010/11/11 12:48:17

  Model Name: MacBook Pro
  Model Identifier: MacBookPro5,5
  Processor Name: Intel Core 2 Duo
  Processor Speed: 2.53 GHz
  Number Of Processors: 1
  Total Number Of Cores: 2
  L2 Cache: 3 MB
  Memory: 4 GB
  Bus Speed: 1.07 GHz

by timknip2
at 2010/11/11 12:49:45

Browser               Method 1     Method 2     Method 3    Method 4
---------------------------------------------------------------------------------------
Chrome 9 dev.   0.000000 -  0.001263 -  0.002795 -  0.000000
IE 9 beta              0.008333 -  0.000000 -  0.001768 -  0.077055
Safari 5                0.000000 -  0.001263 -  0.006495 -  0.076035
Opera 10.6         0.000000 -  0.000000 -  0.001768 -  0.075000
FF 4 beta6          0.000000 -  0.000000 -  0.002500 -  0.077055
--------------------------------------------------------------------------------------
Win7, Intel 2.4GHz (Core2Duo), FP 10.1.85.3 debug

by djankey
at 2010/11/11 13:02:43

Coefficients of variation:
Method 1: 0.077055
Method 2: 0.000000
Method 3: 0.004744
Method 4: 0.075000

Thanks!

Win7, Intel Quad Core 2.84ghz FP 10,1,103,19 installed

by alex.petrescu
at 2010/11/11 13:46:54

Coefficients of variation:
Method 1: 0.048412
Method 2: 0.006052
Method 3: 0.008448
Method 4: 0.030619

Thanks!

Chrome, Dell Latitude E6500

by Daganev
at 2010/11/11 16:20:01

second run... same settings
Coefficients of variation:
Method 1: 0.053033
Method 2: 0.005954
Method 3: 0.006898
Method 4: 0.030619

Thanks!

by Daganev
at 2010/11/11 16:21:46

Win7 x64, FireFox 3.6.12, AMD Phemon x6 @ 2.4Ghz, 8gb DDR3 @ 1600Mhz, FP 10.1.85.3
Method 1: 0.064952
Method 2: 0.003675
Method 3: 0.003248
Method 4: 0.000000

by matanube
at 2010/11/11 16:21:48

http://gyazo.com/8ef343c0841a1d460efb49b62cd97fe4.png

Win7 x64 i5 430m 4gb ram

by smakhtin
at 2010/11/11 16:34:08

Coefficients of variation:
Method 1: 0.025000
Method 2: 0.008839
Method 3: 0.000000
Method 4: 0.000000

Thanks!

by MilanOrszagh
at 2010/11/11 17:02:42

Method 1: 0.000000
Method 2: 0.013824
Method 3: 0.009065
Method 4: Infinity

by marcel.klammer
at 2010/11/11 17:15:01

WIN7 version10,1,82,76

Coefficients of variation:
Method 1: 0.025345
Method 2: 0.003289
Method 3: 0.003918
Method 4: 0.078062

Thanks!

by 9re
at 2010/11/11 17:15:46

Test has been fixed to avoid infinities on fast machines, but no worries, I'll use all the data I can

by makc3d
at 2010/11/11 20:53:47

Updated test on same netbook
Method 1: 0.013893
Method 2: 0.012346
Method 3: 0.009757
Method 4: 0.015091
Or see code thumbnail... methods 1,4 are down to 2,3 numbers. This, and results collected so far (both here and others) suggest that this test will not find any difference at all. E.g., any method works equally well on average.

by makc3d
at 2010/11/11 22:49:26

Method 1: 0.000000
Method 2: 0.010412
Method 3: 0.000000
Method 4: 0.000000

by PESakaTFM
at 2010/11/12 00:26:39

Method 1: 0.000000
Method 2: 0.010258
Method 3: 0.000000
Method 4: 0.000000

by PESakaTFM
at 2010/11/12 00:27:25

Method 1: 0.000000
Method 2: 0.002083
Method 3: 0.000000
Method 4: 0.000000

by devon_o
at 2010/11/12 02:03:10

With 64-bit beta plugin - LNX 10,2,161,23
(this plugin has some BitmapData bugs, so I don't know how reliable these numbers are...)
Method 1: 0.006239
Method 2: 0.004167
Method 3: 0.003675
Method 4: 0.000000

With standalone 32-bit (non-debug) player - LNX 10,1,53,64
Method 1: 0.007160
Method 2: 0.007265
Method 3: 0.001316
Method 4: 0.013674

Intel(R) Core(TM)2 Quad CPU Q9300 @ 2.50GHz

by yonatan
at 2010/11/12 03:34:16

With Win XP SP2 - FlashPlayer 10,1,85,3 - Intel(R) Core(TM)2 CPU 6700 @ 2.66GHz 1.98GB RAM
Method 1: 0.003093
Method 2: 0.000000
Method 3: 0.001964
Method 4: 0.053033

by bkzen
at 2010/11/12 08:28:45

Method 1: 0.012500
Method 2: 0.001000
Method 3: 0.007806
Method 4: 0.005096

by taiwancasa
at 2011/02/23 18:10:13

sort

pv175

forked from: What is the most ..

Daganev

forked:0
favorite:0
lines:78 (diff:88)

pv215

forked from: What is the most ..

DaniilTutubalin

forked:0
favorite:0
lines:100 (diff:161)