Details

    • Type: Task
    • Status: Done
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_2_2_0
    • Component/s: None
    • Labels:
      None

      Description

      This is a follow-up to the benchmarks done as part of BLZG-1950: we need to benchmark the latest release candidate (called RC3 in the benchmark sheets), rev. 47dd29d5027c58dd5f33a9c06c02393afa57f126 of branch 2.2.0_RC from Aug 25.

        Activity

        Show
        michaelschmidt michaelschmidt added a comment - Benchmarks started, here are the templates for the results: SP2Bench: https://docs.google.com/spreadsheets/d/1xtOf9C-SycmynC8tZg6aDkjVWU0dcsj-l_da4o0gjtw/edit#gid=143382623 Govtrack: https://docs.google.com/spreadsheets/d/1MFF3kQmzQv7LzBFjbNX7bhc1eLgCVPQH0vQA6SxeHuQ/edit#gid=665371524 LUBM: https://docs.google.com/spreadsheets/d/12rbe77GOqnRmi4yFjWE1D1hXnd2jB-WPUV2uVJZEmwE/edit#gid=384598833 BSBM: https://docs.google.com/spreadsheets/d/1i-JnEy_W5Pt4AWg87oxg564GYkz3zaxxmIS9H4OKssE/edit#gid=435201657
        Hide
        michaelschmidt michaelschmidt added a comment -

        Here is a summary of the results (see sheets above):

        • SP2Bench: stable
        • Govtrack: ok from correctness point of view, 30% performance regression as we have seen it in the BLZG-533 branch
        • LUBM: 4% speedup, similiar to the speedups observed in previous runs (we actually had observed more in some BLZG-2041 branches, but lubm just seems to vary from time to time, probably we should increase the number of runs to get that more stable)
        • BSBM:
        • EXPLORE and EXPLORE + UPDATE single-threaded: 5% performance gain (as also seen in previous runs)
        • EXPLORE MT: stable
        • EXPLORE + UPDATE: between 20-32% speedup, which was previously explained by the solution set stream change for UPDATE queries + parser decoupling

        Loading performance has not changed significantly since the latest release.

        So from a benchmark perspective there should be green light for a release. The only regression we have is for govtrack / analytics mode, which we had seen before and (as I understood) we're willing to accept for now.

        Please close the ticket after review if you agree, bryanthompson.

        Show
        michaelschmidt michaelschmidt added a comment - Here is a summary of the results (see sheets above): SP2Bench: stable Govtrack: ok from correctness point of view, 30% performance regression as we have seen it in the BLZG-533 branch LUBM: 4% speedup, similiar to the speedups observed in previous runs (we actually had observed more in some BLZG-2041 branches, but lubm just seems to vary from time to time, probably we should increase the number of runs to get that more stable) BSBM: EXPLORE and EXPLORE + UPDATE single-threaded: 5% performance gain (as also seen in previous runs) EXPLORE MT: stable EXPLORE + UPDATE: between 20-32% speedup, which was previously explained by the solution set stream change for UPDATE queries + parser decoupling Loading performance has not changed significantly since the latest release. So from a benchmark perspective there should be green light for a release. The only regression we have is for govtrack / analytics mode, which we had seen before and (as I understood) we're willing to accept for now. Please close the ticket after review if you agree, bryanthompson .
        Hide
        bryanthompson bryanthompson added a comment -

        michaelschmidt I think we are good for a 2.2.0 RC. For the 2.2.0 release, I propose that we use a dynamic policy such that the Java managed object help is used until we have 200k solutions on the heap, at which point we use native memory. This policy could have a low water mark / high water mark. It could also be made more automatic by paying attention to the current GC burden.

        If we make this an analytic mode only option, then this change would only impact govtrack.

        If we make this a default mode behavior, then the 2.2.0 release would need a complete benchmark run.

        Brad Bebee

        Show
        bryanthompson bryanthompson added a comment - michaelschmidt I think we are good for a 2.2.0 RC. For the 2.2.0 release, I propose that we use a dynamic policy such that the Java managed object help is used until we have 200k solutions on the heap, at which point we use native memory. This policy could have a low water mark / high water mark. It could also be made more automatic by paying attention to the current GC burden. If we make this an analytic mode only option, then this change would only impact govtrack. If we make this a default mode behavior, then the 2.2.0 release would need a complete benchmark run. Brad Bebee
        Hide
        michaelschmidt michaelschmidt added a comment -

        Agreed, makes perfect sense to me.

        Show
        michaelschmidt michaelschmidt added a comment - Agreed, makes perfect sense to me.

          People

          • Assignee:
            michaelschmidt michaelschmidt
            Reporter:
            michaelschmidt michaelschmidt
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: