Links

• via Andi in IRC Skip List based Berkeley DB clone - claims to be 4x faster than BerkeleyDB^?

Notes

More performance ceiling

See yesterdays' page TedLeung20041129 for context

Andi objected to a few things in the initial version of my script.

1. I was using Berkeley DB's auto commit mode -- I now explicity bracket the operations in the transaction -- but in the inital version every item write was followed by a commit
2. I was committing one item at a time, which Andi thought was unrealistic (but I think we're going to be doing lots of small commits) -- I added a test which put's all the "items" into Berkeley DB and then does a single commit (Average Transactional batch commit rate).

So here are some new numbers using a revised version of the script. Things to observe. The commit after every item rate has improved somewhat, to the 1800 commits/sec range. The "batch" commit rate is 10x the individual commit rate (but this is not that surprising, since now you only commit once). The "batch" commit rate is about half the non-transactional commit rate.

If you think that we are going to do lots of small commits, then the ceiling got a little higher. If you think that we are going to be able to reduce the number of commits dramatically by batching (which I doubt because the app UI will not reflect changes), then the ceiling has lifted by one order of magnitude.

A new version of the script is attached.

Generating data... done.
Cache Size = 2097152
Average Non transactional commit rate = 42867.2856863
Average Non transactional sequential read rate = 42883.1730643
Average Non transactional random read rate = 40147.8439506

Average Transactional commit rate = 1826.84404417
Average Transactional batch commit rate = 21849.7904862
Average Transactional sequential read rate = 32208.9726274
Average Transactional random read rate = 30306.810744

Cache Size = 4194304
Average Non transactional commit rate = 42402.312798
Average Non transactional sequential read rate = 42526.4744871
Average Non transactional random read rate = 40223.0419162

Average Transactional commit rate = 1839.82637067
Average Transactional batch commit rate = 21920.4222654
Average Transactional sequential read rate = 32094.6588583
Average Transactional random read rate = 30715.2917284

Cache Size = 8388608
Average Non transactional commit rate = 43096.935911
Average Non transactional sequential read rate = 44003.853649
Average Non transactional random read rate = 41620.5928869

Average Transactional commit rate = 1866.39079434
Average Transactional batch commit rate = 22129.4487017
Average Transactional sequential read rate = 33546.3952612
Average Transactional random read rate = 31398.669283

Cache Size = 16777216
Average Non transactional commit rate = 42401.3466943
Average Non transactional sequential read rate = 42583.9380461
Average Non transactional random read rate = 41593.6176544

Average Transactional commit rate = 1858.26440091
Average Transactional batch commit rate = 22007.8002072
Average Transactional sequential read rate = 31812.1726999
Average Transactional random read rate = 30229.0820306

Cache Size = 33554432
Average Non transactional commit rate = 41585.5358601
Average Non transactional sequential read rate = 43359.3976266
Average Non transactional random read rate = 41882.729666

Average Transactional commit rate = 1815.84225843
Average Transactional batch commit rate = 21982.5434605
Average Transactional sequential read rate = 31961.4990775
Average Transactional random read rate = 29639.0330628

Cache Size = 67108864
Average Non transactional commit rate = 42649.6568258
Average Non transactional sequential read rate = 40991.9770915
Average Non transactional random read rate = 41248.213117

Average Transactional commit rate = 1798.67498679
Average Transactional batch commit rate = 21541.895478
Average Transactional sequential read rate = 31763.4833364
Average Transactional random read rate = 30132.894743

Cache Size = 134217728
Average Non transactional commit rate = 42238.1479371
Average Non transactional sequential read rate = 42316.0078112
Average Non transactional random read rate = 41088.3410098

Average Transactional commit rate = 1844.56404272
Average Transactional batch commit rate = 21453.068402
Average Transactional sequential read rate = 31419.2011513
Average Transactional random read rate = 30245.8936534

RunPython bdbtest.py  339.70s user 193.56s system 45% cpu 19:22.53 total

-- TedLeung - 01 Dec 2004

• bdbtest.py.txt: