523863 - Misc improvements to performance runtests to improve reporting.

Status: VERIFIED FIXED

(Tamarin Graveyard :: Tools, enhancement)

Description

[Chris Peyer]

From Edwin Smith:

Some suggestions to make #s more relevant:

1. calculate %stdev instead of stdev, that is, %stdev = stdev/avg.  this will
normalize all the stdev results, currently they are in the measurement units
and its easier to understand if they are unitless %.

2. calculates "significance" as %diff / (%stdev1 + %stdev2).  Test with large
standard devitions (lots of noise) thus require a larger %diff before the
change can be considered significant.

Not to touch on religious issues here... some small changes are still
desireable (or not desireable) even if they are "insignificant" calculated this
way.  but more importantly this will highlight the truly significant changes. 
(with signifiance outside the noise band, say > 1 or < -1).

Comment 1

[Chris Peyer]

Attachment: Improve performance reporting

Add %diff and %stddev to report as well as significance indicators.

Comment 2

[Chris Peyer]

Attachment: Updated patch - also includes improved cygwin support

Patch also adds the ability to specify files in cygwin path-format even when using windows python.

Comment 3

[Brent Baker]

Comment on attachment 408155 [details] [diff] [review]
Updated patch - also includes improved cygwin support

- the /bold/ does not work on windows and is actually messes up the formatting a little because there are additional characters being inserted. I am guessing you could see the same issue if you piped the output to a file on mac

- could you comment a little around the "--", "-", "+" "++" code, exactly what is this trying to tell the user?

Comment 4

[Brent Baker]

Comment on attachment 408155 [details] [diff] [review]
Updated patch - also includes improved cygwin support

Have you checked to see what the performance difference is, if any, with compiling with -AS3? Should new baseline data be generated so that we can compare the performance of milestone builds properly?

Comment 5

[Chris Peyer]

Attachment: Add -p --prettyprint option

-p option will bold tests with significant performance change

+/- also indicates significant performance change:
- : -2 < sig < -1
-- : sig < -2
+ : 1 < sig < 2
++ : sig > 2

Comment 6

[Chris Peyer]

pushed redux 2923:5f7b60c15f76

Comment 7

[Chris Peyer]

Attachment: Update to not fix cygwin paths on winmobile.

Was previously trying to fix winmobile paths which was causing failures.

Comment 8

[Dan Schaffer]

Comment on attachment 409086 [details] [diff] [review]
Update to not fix cygwin paths on winmobile.

looks ok.  on mac I did no see the results appear in bold on my shell but the ++,+,--,- is useful to point out signficant changes in pretty-print mode.

Comment 9

[Chris Peyer]

pushed (fixed patch) changeset 2940	e962fef8d694