Summary of results from Sep 19-20 benchmark run:
I’d say it’s all more or less statistical variance. As for SP2Bench, the slowest query was a little bit slower than usual in one of the runs, which might be due to background processes (in the worst case, GC). So as for these three benchmarks, nothing to worry.
Regarding BSBM, the results also don’t look that bad in general:
- No issues for BSBM EXPLORE, all in the range of +/- 0.5% compared to the previous release
- BSBM EXPLORE+UPDATE is a bit different:
a. Results for the different #threads vary a bit more (<=~1% though), which is nothing I would be concerned about
b. We observed (only one) GCOverheadExceeded exception, in the MT16 run (in the previous run with 1.5.3, I also observed one such exception for MT64)
c. For the MT64 version (~1% slower), we observe a significant increase in the cumulative collection time (summing up the collection times: from 513240 in 1.5.2 to 929706 in 1.5.3); not sure if this is critical
I consider b. the only critical blocker for the release.