Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-739

SIDs mode creates unnecessary hash join for GRAPH group patterns.

    Details

      Description

      Per a thread on the forum, it looks like the SIDs mode query plan for a GRAPH group pattern is creating an unnecessary hash join that is in turn leading to some strange NPEs on the JiniFederation.

      The unnecessary hash join can be observed in the query plan for this query (and probably for any such SIDs mode query).

      prefix sc: <http://www.ibsplc.com/sprout/social/2013/01/social#>
      select *
      where { 
            GRAPH ?sid {?a ?b ?c }
            ?sid sc:source ?p .
            ?p rdf:type sc:Person .
            ?p sc:userName "Ajesh JS" .
      } 
      

      The optimized AST for that query is:

      PREFIX sc: <http://www.ibsplc.com/sprout/social/2013/01/social#>
      QueryType: SELECT
      includeInferred=true
      SELECT VarNode(a) VarNode(b) VarNode(c) VarNode(sid)
        JoinGroupNode [context=VarNode(sid)] {
          StatementPatternNode(VarNode(a), VarNode(b), VarNode(c), VarNode(sid)) [scope=NAMED_CONTEXTS]
            com.bigdata.rdf.sparql.ast.eval.AST2BOpBase.estimatedCardinality=745
            com.bigdata.rdf.sparql.ast.eval.AST2BOpBase.originalIndex=SPO
        }
      

      Notice the JoinGroupNode was not eliminated. We appear to be missing an AST optimizer to always lift out the SIDs mode triple pattern that binds the statement identifier variable.

      The NPEs appear to be related to having an empty solution set flow into a hash join wrapping the GRAPH group pattern. A sample stack trace is below. This NPE was triggered by a null "key" in the code to range check the keys that were mapped onto a given index partition. The NPE will probably disappear if we correct the query plan. It has never by observed in valid query plans.

      Caused by: java.lang.RuntimeException: cause=java.lang.NullPointerException, state=JVMHashJoinUtility{open=true,joinType=Normal,joinVars=[sid],size=418,considered(left=0,right=0,joins=0)}
      	at com.bigdata.bop.join.JVMHashJoinUtility.launderThrowable(JVMHashJoinUtility.java:1680)
      	at com.bigdata.bop.join.JVMHashJoinUtility.outputSolutions(JVMHashJoinUtility.java:1174)
      	at com.bigdata.bop.join.HashIndexOp$ChunkTask.outputSolutions(HashIndexOp.java:444)
      	at com.bigdata.bop.join.HashIndexOp$ChunkTask.call(HashIndexOp.java:337)
      	at com.bigdata.bop.join.HashIndexOp$ChunkTask.call(HashIndexOp.java:1)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1271)
      	... 9 more
      Caused by: java.lang.NullPointerException
      	at com.bigdata.btree.BytesUtil.compareBytes(BytesUtil.java:242)
      	at com.bigdata.btree.BytesUtil.rangeCheck(BytesUtil.java:766)
      	at com.bigdata.bop.fed.shards.Algorithm_NestedLocatorScan.mapOverShards(Algorithm_NestedLocatorScan.java:62)
      	at com.bigdata.bop.fed.shards.MapBindingSetsOverShardsBuffer.handleChunk(MapBindingSetsOverShardsBuffer.java:328)
      	at com.bigdata.bop.fed.shards.MapBindingSetsOverShardsBuffer.handleChunk(MapBindingSetsOverShardsBuffer.java:1)
      	at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.overflow(AbstractUnsynchronizedArrayBuffer.java:275)
      	at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.flush(AbstractUnsynchronizedArrayBuffer.java:301)
      	at com.bigdata.bop.fed.shards.MapBindingSetsOverShardsBuffer.flush(MapBindingSetsOverShardsBuffer.java:355)
      	at com.bigdata.bop.fed.FederationChunkHandler.handleChunk(FederationChunkHandler.java:256)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1544)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1526)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1)
      	at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:59)
      	at com.bigdata.relation.accesspath.UnsyncLocalOutputBuffer.handleChunk(UnsyncLocalOutputBuffer.java:1)
      	at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.overflow(AbstractUnsynchronizedArrayBuffer.java:275)
      	at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add2(AbstractUnsynchronizedArrayBuffer.java:203)
      	at com.bigdata.relation.accesspath.AbstractUnsynchronizedArrayBuffer.add(AbstractUnsynchronizedArrayBuffer.java:161)
      	at com.bigdata.bop.join.JVMHashJoinUtility.outputSolutions(JVMHashJoinUtility.java:1161)
      	... 15 more
      

        Activity

        Hide
        varunshaji varunshaji added a comment -

        prefix sc:http://www.ibsplc.com/sprout/social/2013/01/social#
        construct { ?a ?b ?c }
        where{

        { GRAPH ?sid \{?a ?b ?c }

        ?sid sc:source ?p .

        ?p rdf:type sc:Person .

        ?p sc:username "varun.k" . \

        }
        }

        Could you try this query. As per the dataset those usernames are not there. but "varun.k" is thr and also the property is sc:username not sc:userName.

        Show
        varunshaji varunshaji added a comment - prefix sc: http://www.ibsplc.com/sprout/social/2013/01/social# construct { ?a ?b ?c } where{ { GRAPH ?sid \{?a ?b ?c } ?sid sc:source ?p . ?p rdf:type sc:Person . ?p sc:username "varun.k" . \ } } Could you try this query. As per the dataset those usernames are not there. but "varun.k" is thr and also the property is sc:username not sc:userName.
        Hide
        mikepersonick mikepersonick added a comment -

        added an ASTFlattenJoinGroupsOptimizer to fix this issue

        Show
        mikepersonick mikepersonick added a comment - added an ASTFlattenJoinGroupsOptimizer to fix this issue

          People

          • Assignee:
            mikepersonick mikepersonick
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: