Calibrating Tbox run perf tests with manual run tests
When we run manual tests, there is basically nothing else running on the machine. But when Tbox script is executing the perf tests the Tbox script is obviously another process that is running, and may be causing slowdown. We need to measure the runs on the same machine with the same revision using both manual and Tbox script and see what the difference is.
Windows, revision 7503 (Tuesday, Sep 27, 2005)
| | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | switch | stamp |
|---|
| Manual | 9.34 | 1.76 | 1.09 | 1.00 | 78.8 | 57.6 | 12.6 | 6.26 | 1.26 | 0.578 | 1.70 |
| Tbox | | 1.50 | 1.02 | 0.700 | 80.6 | | | | | 0.578 | 1.64 |
| Tbox | | -0.26 | -0.07 | -0.3 | 1.8 | | | | | 0 | -0.06 |
The negative values look somewhat strange, but it could be because of python compiling py files in the manual case (assuming already done for tbox), or it could be random noise. Largest difference is 30.0%. A conservative approach is to say tbox overhead is 30%.
OSX, revision 7503
| | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | switch | stamp |
|---|
| Manual | 18.6 | 2.06 | 1.76 | 0.850 | 178 | 73.4 | 24.0 | 12.0 | 2.84 | 1.30 | 2.10 |
| Tbox | | 1.76 | 2.16 | 1.30 | 200 | | | | | 1.14 | 2.18 |
| Tbox adj | | -0.3 | 0.4 | 0.45 | 22 | | | | | -0.16 | 0.08 |
The negative values look somewhat strange, but it could be because of python compiling py files in the manual case (assuming already done for tbox), or it could be random noise. Largest difference is 52.9%, although most are less than 23% difference. A conservative approach is to say tbox overhead is 63%.
Linux, revision 7503
| | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | switch | stamp |
|---|
| Manual | 7.14 | 1.48 | 1.46 | 0.750 | 97.4 | 45.8 | 11.0 | 13.0 | 1.10 | 1.20 | 1.32 |
| Tbox | | 1.34 | 1.42 | 0.672 | 97.8 | | | | | 1.20 | 1.46 |
| Tbox adj | | -0.14 | -0.04 | -0.078 | 0.4 | | | | | 0 | 0.14 |
The negative values look somewhat strange, but it could be because of python compiling py files in the manual case (assuming already done for tbox), or it could be random noise. Largest difference is 10.6%, so a conservative approach is to say tbox overhead is 11%.
Performance trends on Windows
Measured at three different revisions:
Performance trends on OSX
Measured at three different revisions, some incomplete measurements:
Windows, revision 8059 (Tuesday, Oct 25, 2005)
| | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | switch | stamp |
|---|
| Manual | 8.12 | 1.06 | 1.10 | 0.26 | 102 | 19 | 5.86 | 5.60 | 1.14 | 0.78 | 0.438 |
| Tbox | | 1.11 | 1.05 | 0.23 | 100.59 | | 5.67 | 5.49 | 1.12 | 0.77 | 0.41 |
| diff | | 4% | -5% | -12% | -1% | | -3% | -2% | -2% | -1% | -6% |
Windows, revision 8145 (Tuesday, Nov 1, 2005)
| | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | switch | stamp |
|---|
| Manual | 8.82 | 0.86 | 0.658 | 0.25 | 104 | 19 | 4.54 | 5.00 | 0.016 | 0.703 | 0.406 | 4.06 | 1.72 | 0.03 | 0.25 |
| Tbox | 8.1 | 0.84 | 0.62 | 0.23 | | 19.85 | 4.36 | 4.86 | 0.03 | 0.69 | 0.36 | 4.09 | 1.69 | 0.03 | 0.25 |
| diff | -8% | -1% | -6% | -8% | | 5% | -4% | -3% | 88% | -2% | -6% | 1% | 2% | 0% | 0% |
+5/-8% (88% anomaly should smooth itself out on average)
OSX, revision 8145 (Tuesday, Nov 1, 2005)
| | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | switch | stamp |
|---|
| Manual | 17.6 | 1.54 | 1.40 | 0.26 | 218 | 49.4 | 9.88 | 9.84 | 0.444 | 1.08 | 1.13 | 7.0 | 2.94 | 0.1 | 0.35 |
| Tbox | 15.2 | 1.58 | 1.38 | 0.21 | | 55.7 | 14.0 | 14.31 | 0.56 | 1.02 | 0.94 | 8.17 | 3.78 | 0.1 | 0.34 |
| diff | -14% | 3% | -2% | -19% | | 3% | 42% | 46% | 26% | -5% | -6% | 16% | 29% | 0% | -3% |
+16/-19% (more than 20% changes are anomalies (there was a jump in values for no reason for this revision))
Linux, revision 8145 (Tuesday, Nov 1, 2005)
| | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | switch | stamp |
|---|
| Manual | 6.14 | 0.96 | 0.51 | 0.594 | 114 | 19.8 | 5.98 | 6.49 | 0.064 | 1.1 | 0.48 | 6.6 | 3.17 | 0.028 | 0.202 |
| Tbox | 6.09 | 0.97 | 0.51 | 0.51 | | 20.0 | 5.86 | 6.61 | 0.07 | 1.01 | 0.48 | 6.6 | 3.65 | 0.03 | 0.14 |
| diff | -1% | 1% | -0% | -14% | | 1% | -2% | 2% | 9% | -8% | 0% | 0% | 15% | 7% | -31% |
+9/-8 (larger than 10% anomalies that average out)
Log In