Here’s a summary of the benchmark results for the 1.5.2_RC1 branch (rev. 3b3d81885c0d7e8c98b39382b37694d62ea265e3), compared with the latest results available for the 1.5.1 release:
- SP2B: https://docs.google.com/spreadsheets/d/1fMLuawcsqey0kzT5wfS8BMxrVHDG04IsTRJC8vll2Zk/edit#gid=1796656657
In overall, there’s a drop in performance of 0.98%. Looking at the individual queries, we observe significant changes only for q10, q3a, q3c, and q6. As for q10, q3a, and q3c, these are short running queries — running these queries only 9 times, we simply do not get a representative result here and I would account the changes to statistics variance (also looked at some of the QEPs and there were no changes). Regarding q6: performance drops from 75-76s (in 1.5.1) to 77-78s (in 1.5.2 RC1), which is due to a correctness fix (additional distinct projection before projecting variables into subgroup). So also here, no concerns from my side.
Note that we with the FILTER decomposition enabled (which is disabled for now by default), for q8 we experience a significant speedup from 240ms down to 40ms. See sheet 1.5.1_20150701.
EXPLORE ST: 3.6% dropdown in performace
EXPLORE MT16: 1% dropdown
EXPLORE MT32: 0.5% dropdown
Our guess here was that this is due to more time taken in parsing and/or static analysis. More concretely, the regressions might be caused by increased time through the changes in the optimizers. With the new static analysis stats we now have tooling to track this down and optimize for future releases. Unless we take the effort and migrate these performance measurement capabilities into the 1.5.1 release, I’d will be hard to trace down what exactly caused these changes.
More critical is the EXPLORE + UPDATE performance:
EXPLORE + UPDATE 16: 8.7% drop in performance
EXPLORE + UPDATE 32: 12.35% drop in performance
I also ran BSBM E+U16 locally and observed similar regressions, see https://docs.google.com/spreadsheets/d/1wV6ZlV7ABVFbR28gPFcUyn0zi1RwhaDvg0urFkA-mtM/edit#gid=1080792412. As discussed with Bryan before, I tried to track these things down, but wasn’t successful. Not sure how to best proceed here.