Junk Byte Code for Kicks

10Mar/126

Flash performance ranking ‘framework’

I am writing a simple framework which lets you score the performance of a given piece of code.

The motivation
A lot of the time when I am writing a piece of code that needs to perform really fast (due to it being called so many times per frame, etc) I struggle to figure out which approach is faster.
I am talking something very basic, such as do I use "for each (var num:uint in list)" or "for (var i:uint = 0; i < len; i++)" to loop through a list.

The framework

It is just called "PerformanceTest" at the moment.

  • You register a group of TestCases to be tested
  • Each test case should only do very simple task synchronously
  • It makes sense to group similar test cases in a 'TestGroup'
  • The amount of times it managed to perform each test within a set time period (500 ms) is recorded .
  • A 'standard test' is also ran for the same time period and recorded.
  • The score for each test is produced by comparing against the perform count of 'standard test'.

The score hopefully makes it relatively comparable between different machines - given the same SDK and player version.
Haven't tested against different machines yet. This does not matter much as we are more interested in how pieces of code perform compared to another. on the same machine.

The test scores are about ±5% accurate (according to tests so far).
i.e, if you run the tests multiple times the scores given will be ±5% different from previous result.

Current state

Currently it just prints the results in trace, but I hope to find time to display graphs using Flex framework.
Will probably share it as open source once its at that stage.
Unless I find someone else have already done this :)

Example results:

Looping a list and Object

loop count: 10
ForEachVector --- score: 4349.30 performs: 512453
ForVarILenVector --- score: 3249.75 performs: 382900
ForVarIVector --- score: 2454.79 performs: 289234
ForVarXVector --- score: 1500.88 performs: 176841
ForEachObject --- score: 4134.51 performs: 487146
ForVarXObject --- score: 1450.78 performs: 170937
ForEachDictionary --- score: 3966.92 performs: 473648
ForVarXDictionary --- score: 1363.43 performs: 162793

loop count: 2000
ForEachVector --- score: 76.24 performs: 9000
ForVarILenVector --- score: 26.66 performs: 3148
ForVarIVector --- score: 16.858 performs: 1990
ForVarXVector --- score: 4.11 performs: 486
ForEachObject --- score: 41.81 performs: 4936
ForVarXObject --- score: 4.02 performs: 475
ForEachDictionary --- score: 40.97 performs: 4822
ForVarXDictionary --- score: 3.95 performs: 465

Observations:
Using 'for each' on vector is the fastest.
I used to think 'for (var i:uint = 0; i < len"... was faster.

Looping through objects is not very slow (compared to list) when the count is low, but does get slower when it has many items.
Looping through plain Object is slightly faster than through Dictionary.

Test code:

ForEachVector
for each (var num:uint in vector)

ForVarILenVector
var len:uint = vector.length;
for (var i:uint = 0; i < len; i++){
var num:uint = vector[i];

ForVarIVector
for (var i:uint = 0; i < vector.length; i++){
var num:uint = vector[i];

ForVarXVector
for (var X:String in vector){
var num:uint = vector[X];

ForEachObject 
for each (var num:uint in object)

ForVarXObject
for (var X:String in object){
var num:uint = object[X];

ForEachDictionary
for each (var num:uint in dictionary)

ForVarXDictionary
for (var X:String in dictionary){
var num:uint = dictionary[X];

 

Sorting a String numerically: Array vs Vector

Length of list: 50
Array --- score: 941.73 performs: 111685
Vector --- score: 70.86 performs: 8404

Length of list: 5000
Array --- score: 7.98532 performs: 940
Vector --- score: 0.30582 performs: 36

Test code:

Array
array.sort(Array.NUMERIC);

Vector
vector.sort(numSort);
function numSort (numx:Number, numy:Number):int
{
if (numx < numy){ return 1; }
if (numx > numy){ return -1; }
return 0;
}

 

 

Filed under: Flash Leave a comment
Comments (6) Trackbacks (0)
  1. Great! Did you ever tried Grant Skinner’s PerformanceTest (http://gskinner.com/blog/archives/2010/02/performancetest.html)? I often use it for the same target.

  2. i don’t know how you figured out that for each is fast. that’s one of the damn slowest things i know.
    i re-tested this several times with different kinds of loops. what i get was:
    [Sun Apr 29 09:46:51 GMT+0200 2012] while: 37ms
    [Sun Apr 29 09:46:52 GMT+0200 2012] for each: 629ms
    [Sun Apr 29 09:46:52 GMT+0200 2012] for: 42ms

    code goes like this:
    [code]this.vec = new[];
    var i :int = -1,
    intLen :int = 10000000,
    num :int;
    while(++i !== intLen)
    {
    this.vec[this.vec.length] = Math.random() * int.MAX_VALUE;
    }

    i = -1;
    var begin:int = getTimer();
    while(++i !== intLen)
    {
    num = this.vec[i];
    }
    DebugTracer.println(‘while: ‘ + (getTimer() – begin) +’ms’);

    begin = getTimer();
    for each(i in this.vec)
    {
    num = i;
    }
    DebugTracer.println(‘for each: ‘+ (getTimer() – begin) +’ms’);

    begin = getTimer();
    for (i = 0; i !== intLen; i++)
    {
    num = this.vec[i];
    }
    DebugTracer.println(‘for: ‘+ (getTimer() – begin) +’ms’);[/code]

    so for me, “while is king” :) this hasn’t changed since 10.1

    • The difference with my test is my for-each loop didn’t need “num = i;” assignment inside.

      Your example is already valid if you replace with this code:

      for each(num in vec)
      {

      }
      We do not need the assignment inside the loop as the loop operator already provide us with the num value. The assignment in your example is causing the extra stress in scoring.

      Results on my computer:
      while: 999ms
      for each: 270ms
      for: 958ms

      So I guess we should also remove the assignments in other loops and just have “vec[i]” inside for more valid test.

    • Humm, now i added “vec[i] * 100;” and “num * 100;” to the tests and for each loop become slower. Perhaps if the loop is empty it is optimizing it somehow and therefore reporting wrong :(

  3. I think that part of the work is in the lookup of the object you want to work with. All these methods use different lookups, resulting in different amounts of extra lost performance. Not doing any lookups results in better performance, but does not resemble any real world application, which was the setting you were testing for in the first place.

    Check out this blog for a nice overview of loop speed tests:
    http://jacksondunstan.com/articles/358


Leave a comment

No trackbacks yet.