michaelschmidt added a comment - 15/Jul/15 9:54 AM - edited Here’s a summary of the benchmark results for the 1.5.2_RC1 branch (rev. 3b3d81885c0d7e8c98b39382b37694d62ea265e3), compared with the latest results available for the 1.5.1 release: LUBM: https://docs.google.com/spreadsheets/d/1rjXWP3H_CMcqC66lTy_yVRLr1qBG3Zn2edaLt7i8YPc/edit#gid=500653430 In overall, there’s a small drop in performance of 0.62%. Some queries are slightly faster, some queries slightly slower (+/- 5% for all queries). No concerns from my side here. Govtrack: https://docs.google.com/spreadsheets/d/1nOCsKxleFCA452LWE8fMvqaE_EhMzn5v6Aqqsjeul1k/edit#gid=695758081 In overall, there’s a drop in performance of 0.48%. In no query, we observe a drop of more than 4%, so also here no concerns from my side. SP2B: https://docs.google.com/spreadsheets/d/1fMLuawcsqey0kzT5wfS8BMxrVHDG04IsTRJC8vll2Zk/edit#gid=1796656657 In overall, there’s a drop in performance of 0.98%. Looking at the individual queries, we observe significant changes only for q10, q3a, q3c, and q6. As for q10, q3a, and q3c, these are short running queries — running these queries only 9 times, we simply do not get a representative result here and I would account the changes to statistics variance (also looked at some of the QEPs and there were no changes). Regarding q6: performance drops from 75-76s (in 1.5.1) to 77-78s (in 1.5.2 RC1), which is due to a correctness fix (additional distinct projection before projecting variables into subgroup). So also here, no concerns from my side. Note that we with the FILTER decomposition enabled (which is disabled for now by default), for q8 we experience a significant speedup from 240ms down to 40ms. See sheet 1.5.1_20150701. BSBM: https://docs.google.com/spreadsheets/d/1wV6ZlV7ABVFbR28gPFcUyn0zi1RwhaDvg0urFkA-mtM/edit#gid=1996430620 This is the most critical one, I’d guess, and I discussed these results already with Bryan. EXPLORE ST: 3.6% dropdown in performace EXPLORE MT16: 1% dropdown EXPLORE MT32: 0.5% dropdown Our guess here was that this is due to more time taken in parsing and/or static analysis. More concretely, the regressions might be caused by increased time through the changes in the optimizers. With the new static analysis stats we now have tooling to track this down and optimize for future releases. Unless we take the effort and migrate these performance measurement capabilities into the 1.5.1 release, I’d will be hard to trace down what exactly caused these changes. More critical is the EXPLORE + UPDATE performance: EXPLORE + UPDATE 16: 8.7% drop in performance EXPLORE + UPDATE 32: 12.35% drop in performance I also ran BSBM E+U16 locally and observed similar regressions, see https://docs.google.com/spreadsheets/d/1wV6ZlV7ABVFbR28gPFcUyn0zi1RwhaDvg0urFkA-mtM/edit#gid=1080792412 . As discussed with Bryan before, I tried to track these things down, but wasn’t successful. Not sure how to best proceed here.

michaelschmidt added a comment - 16/Jul/15 2:33 PM - edited Re-executed the benchmarks with correct RAM settings (the previous results were with 11GB rather than 4GB). Changes not significant for govtrack, lubm, SP2Bench (though the results are slightly better than before now). BSBM results are slightly better for EXPLORE ST, EXPLORE MT16, EXPLORE MT32, and EXPLORE+UPDATE 16, but still worse than those of 1.5.1. Only EXPLORE+UDPATE32 considerably improved (~4% performance gain), remaining 8% worse than the 1.5.1 results.

Brad Bebee added a comment - 31/Jul/15 3:56 PM - edited Final Benchmarks for 1.5.2 are here: BSBM: https://docs.google.com/spreadsheets/d/1bkBtgSuR1BpIo4jEtvbdpzOw5rHXVl4DoUOkaxgisKg/edit#gid=1831606474 . LUBM: https://docs.google.com/spreadsheets/d/12rbe77GOqnRmi4yFjWE1D1hXnd2jB-WPUV2uVJZEmwE/edit#gid=683632938 Govtrack: https://docs.google.com/spreadsheets/d/192kwVDT6p9fIdZQQP3TvNJDsyfoqTLRp2Bbg3e32l0M/edit#gid=2078277528 https://docs.google.com/spreadsheets/d/16XjMdI5PRjK7ODOWM_AS4u9Ma5tQBkWXOgZl8diaVYI/edit#gid=1294571085