Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-533 Vector the query engine on the native heap
  3. BLZG-2055

Benchmark performance of BLZG-533 (managed heap vs native heap within the branch)

    Details

    • Type: Sub-task
    • Status: Done
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_2_2_0
    • Component/s: None
    • Labels:
      None

      Description

      This ticket is to assess the performance benefit/loss realized when using the native heap to store the intermediate solutions. The comparison might be analytic mode vs non-analytic mode, or we might compare non-analytic mode with native heap storage for the intermediate solutions.

      Note: For some benchmarks (govtrack) we are already using the analytic mode, so the comparison would be analytic mode in BLZG-533 (with native heap for intermediate solutions) vs analytic mode for master. This would be addressed by the other benchmarking ticket automatically.

      Note: I am expecting that performance of the analytic mode is slightly less overall, but that as increased pressure is exerted on the JVM, the analytic mode (with native heap storage for the intermediate solutions) will scale better than the managed object heap mode and might do better on an absolute basis as well. Certainly, in cases where the JVM would be driven into GC OH with the non-analytic mode, the analytic mode should significantly out-perform the non-analytic mode.

      BLZG-533 removes one of the key remaining sources of GC pressure, which are the intermediate solutions on the managed object heap.

      There is still a remaining source of GC pressure for queries which produce high cardinality outputs at a high rate of production. The final query outputs are still stored on the managed object heap. BLZG-1968 would address this. The challenge with that ticket is ensuring that we have a suitable producer / consumer pattern such that the query engine can swiftly unburden itself of the query, by streaming the results into native memory, while at the same time ensuring that the client does not observed increased latency for low cardinality queries.

        Activity

        Hide
        michaelschmidt michaelschmidt added a comment -

        Started BSBM in analytic mode on master (rev. 0aef50543bac7cea9487f92efed81740e36b4f97). In addition, I created an override-web.xml that sets queryThreadPoolSize from 16 to 64).

        Show
        michaelschmidt michaelschmidt added a comment - Started BSBM in analytic mode on master (rev. 0aef50543bac7cea9487f92efed81740e36b4f97). In addition, I created an override-web.xml that sets queryThreadPoolSize from 16 to 64).
        Hide
        michaelschmidt michaelschmidt added a comment - - edited

        Benchmarks for BSBM in analytic mode and queryThreadPoolSize=64 are now through:

        > https://docs.google.com/spreadsheets/d/1i-JnEy_W5Pt4AWg87oxg564GYkz3zaxxmIS9H4OKssE/edit#gid=2071655808

        We observe a ~4-10% regression for the explore cases (depending on the number of threads). As for the BSBM EXPLORE+UPDATE the results are a bit surprising:

        a.) The single threaded, 48 + 64 threaded versions failed with errors (see below)
        b.) The 16 + 32 threaded versions succeeded; they seem to be slighly slower than previous benchmarks in non-analytic mode (see https://docs.google.com/spreadsheets/d/1i-JnEy_W5Pt4AWg87oxg564GYkz3zaxxmIS9H4OKssE/edit#gid=2071655808) => running the benchmark with the same branch now in non-analytic mode as requested to confirm with the identical version

        The error showing up in the log is the following:

             [java] ERROR: 53144      com.bigdata.journal.Journal.executorService6 com.bigdata.util.concurrent.Haltable.logCause(Haltable.java:469): com.bigdata.bop.join.PipelineJoin$JoinTask{ joinOp=com.bigdata.bop.join.PipelineJoin[13](PipelineJoin[11])[ BOp.bopId=13, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[12](review=null, TermId(13248732U)[http://purl.org/stuff/rev#reviewer], reviewer=null)[ IPredicate.relationName=[BSBM_284826.spo], IPredicate.timestamp=1471631393592, BOp.bopId=12, AST2BOpBase.estimatedCardinality=2848260, AST2BOpBase.originalIndex=POS, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL]]]} : isFirstCause=true : java.lang.RuntimeException: java.lang.IllegalStateException: Check blob header assumptions
             [java] java.lang.RuntimeException: java.lang.IllegalStateException: Check blob header assumptions
             [java]     at com.bigdata.bop.engine.ChunkedRunningQuery.scheduleNext(ChunkedRunningQuery.java:748)
             [java]     at com.bigdata.bop.engine.ChunkedRunningQuery.acceptChunk(ChunkedRunningQuery.java:348)
             [java]     at com.bigdata.bop.engine.QueryEngine.acceptChunk(QueryEngine.java:1159)
             [java]     at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:113)
             [java]     at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699)
             [java]     at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputBufferedChunk(ChunkedRunningQuery.java:1716)
             [java]     at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.flush(ChunkedRunningQuery.java:1739)
             [java]     at com.bigdata.bop.join.PipelineJoin$JoinTask.flushAndCloseBuffersAndAwaitSinks(PipelineJoin.java:788)
             [java]     at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:633)
             [java]     at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:382)
             [java]     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
             [java]     at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
             [java]     at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1346)
             [java]     at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:926)
             [java]     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
             [java]     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
             [java]     at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
             [java]     at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:821)
             [java]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
             [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
             [java]     at java.lang.Thread.run(Thread.java:745)
             [java] Caused by: java.lang.IllegalStateException: Check blob header assumptions
             [java]     at com.bigdata.rwstore.sector.PSInputStream.<init>(PSInputStream.java:35)
             [java]     at com.bigdata.rwstore.sector.AllocationContext.getInputStream(AllocationContext.java:351)
             [java]     at com.bigdata.rwstore.sector.MemStrategy.getInputStream(MemStrategy.java:477)
             [java]     at com.bigdata.rwstore.sector.MemStore.getInputStream(MemStore.java:388)
             [java]     at com.bigdata.bop.solutions.SolutionSetStream.get(SolutionSetStream.java:236)
             [java]     at com.bigdata.bop.engine.LocalNativeChunkMessage$ChunkAccessor.<init>(LocalNativeChunkMessage.java:280)
             [java]     at com.bigdata.bop.engine.LocalNativeChunkMessage.getChunkAccessor(LocalNativeChunkMessage.java:263)
             [java]     at com.bigdata.bop.engine.ChunkedRunningQuery.scheduleNext(ChunkedRunningQuery.java:684)
             [java]     ... 20 more
        

        (if I remember right, that was a problem from BLZG-2041 and might be resolved once we merge the final version of that branch into master?) There might be other errors, did not check in detail. Will attach the server log file.

        Show
        michaelschmidt michaelschmidt added a comment - - edited Benchmarks for BSBM in analytic mode and queryThreadPoolSize=64 are now through: > https://docs.google.com/spreadsheets/d/1i-JnEy_W5Pt4AWg87oxg564GYkz3zaxxmIS9H4OKssE/edit#gid=2071655808 We observe a ~4-10% regression for the explore cases (depending on the number of threads). As for the BSBM EXPLORE+UPDATE the results are a bit surprising: a.) The single threaded, 48 + 64 threaded versions failed with errors (see below) b.) The 16 + 32 threaded versions succeeded; they seem to be slighly slower than previous benchmarks in non-analytic mode (see https://docs.google.com/spreadsheets/d/1i-JnEy_W5Pt4AWg87oxg564GYkz3zaxxmIS9H4OKssE/edit#gid=2071655808 ) => running the benchmark with the same branch now in non-analytic mode as requested to confirm with the identical version The error showing up in the log is the following: [java] ERROR: 53144 com.bigdata.journal.Journal.executorService6 com.bigdata.util.concurrent.Haltable.logCause(Haltable.java:469): com.bigdata.bop.join.PipelineJoin$JoinTask{ joinOp=com.bigdata.bop.join.PipelineJoin[13](PipelineJoin[11])[ BOp.bopId=13, JoinAnnotations.constraints= null , AST2BOpBase.simpleJoin= true , BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[12](review= null , TermId(13248732U)[http: //purl.org/stuff/rev#reviewer], reviewer= null )[ IPredicate.relationName=[BSBM_284826.spo], IPredicate.timestamp=1471631393592, BOp.bopId=12, AST2BOpBase.estimatedCardinality=2848260, AST2BOpBase.originalIndex=POS, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL]]]} : isFirstCause= true : java.lang.RuntimeException: java.lang.IllegalStateException: Check blob header assumptions [java] java.lang.RuntimeException: java.lang.IllegalStateException: Check blob header assumptions [java] at com.bigdata.bop.engine.ChunkedRunningQuery.scheduleNext(ChunkedRunningQuery.java:748) [java] at com.bigdata.bop.engine.ChunkedRunningQuery.acceptChunk(ChunkedRunningQuery.java:348) [java] at com.bigdata.bop.engine.QueryEngine.acceptChunk(QueryEngine.java:1159) [java] at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:113) [java] at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699) [java] at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputBufferedChunk(ChunkedRunningQuery.java:1716) [java] at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.flush(ChunkedRunningQuery.java:1739) [java] at com.bigdata.bop.join.PipelineJoin$JoinTask.flushAndCloseBuffersAndAwaitSinks(PipelineJoin.java:788) [java] at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:633) [java] at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:382) [java] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [java] at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) [java] at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1346) [java] at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:926) [java] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [java] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [java] at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) [java] at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:821) [java] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [java] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [java] at java.lang. Thread .run( Thread .java:745) [java] Caused by: java.lang.IllegalStateException: Check blob header assumptions [java] at com.bigdata.rwstore.sector.PSInputStream.<init>(PSInputStream.java:35) [java] at com.bigdata.rwstore.sector.AllocationContext.getInputStream(AllocationContext.java:351) [java] at com.bigdata.rwstore.sector.MemStrategy.getInputStream(MemStrategy.java:477) [java] at com.bigdata.rwstore.sector.MemStore.getInputStream(MemStore.java:388) [java] at com.bigdata.bop.solutions.SolutionSetStream.get(SolutionSetStream.java:236) [java] at com.bigdata.bop.engine.LocalNativeChunkMessage$ChunkAccessor.<init>(LocalNativeChunkMessage.java:280) [java] at com.bigdata.bop.engine.LocalNativeChunkMessage.getChunkAccessor(LocalNativeChunkMessage.java:263) [java] at com.bigdata.bop.engine.ChunkedRunningQuery.scheduleNext(ChunkedRunningQuery.java:684) [java] ... 20 more (if I remember right, that was a problem from BLZG-2041 and might be resolved once we merge the final version of that branch into master?) There might be other errors, did not check in detail. Will attach the server log file.
        Hide
        michaelschmidt michaelschmidt added a comment -

        SPARQL server log for analytic mode

        Show
        michaelschmidt michaelschmidt added a comment - SPARQL server log for analytic mode
        Hide
        michaelschmidt michaelschmidt added a comment -

        Re-benchmarked BLZG-533 in non-analytic mode with queryThreadPoolSize=64, see:

        > https://docs.google.com/spreadsheets/d/1i-JnEy_W5Pt4AWg87oxg564GYkz3zaxxmIS9H4OKssE/edit#gid=880940940

        Compared to the BLZG-533 analytic version, we have the following:

        • BSBM EXPLORE single threaded: ~15% faster
        • BSBM EXPLORE multi-threaded: all versions (16, 32, 48, 64) are around 5% faster
        • BSBM EXPLORE + UPDATE multi-threaded 16+43: same performance
        • BSBM EXPLORE + UPDATE single threaded + multi threaded 48+64: not comparable (failed in recent analytic mode run due to errors)
        Show
        michaelschmidt michaelschmidt added a comment - Re-benchmarked BLZG-533 in non-analytic mode with queryThreadPoolSize=64, see: > https://docs.google.com/spreadsheets/d/1i-JnEy_W5Pt4AWg87oxg564GYkz3zaxxmIS9H4OKssE/edit#gid=880940940 Compared to the BLZG-533 analytic version, we have the following: BSBM EXPLORE single threaded: ~15% faster BSBM EXPLORE multi-threaded: all versions (16, 32, 48, 64) are around 5% faster BSBM EXPLORE + UPDATE multi-threaded 16+43: same performance BSBM EXPLORE + UPDATE single threaded + multi threaded 48+64: not comparable (failed in recent analytic mode run due to errors)
        Hide
        martyncutcher martyncutcher added a comment -

        I have pushed a fix to BLZG-2041 that relaxes the restriction on BLOBS to only have single buffer for a header. This should remove the restriction on maximum blob size.

        Show
        martyncutcher martyncutcher added a comment - I have pushed a fix to BLZG-2041 that relaxes the restriction on BLOBS to only have single buffer for a header. This should remove the restriction on maximum blob size.
        Hide
        michaelschmidt michaelschmidt added a comment -

        This can be closed from my side. bryanthompson

        Show
        michaelschmidt michaelschmidt added a comment - This can be closed from my side. bryanthompson

          People

          • Assignee:
            michaelschmidt michaelschmidt
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: