Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1181

Consequently exploit new built-in projection capabilities of HashIndexOp

    Details

    • Type: New Feature
    • Status: Done
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: BIGDATA_RELEASE_1_5_0
    • Fix Version/s: BLAZEGRAPH_2_0_0
    • Component/s: Query Plan Generator
    • Labels:
      None

      Description

      The HashIndexOp now offers means to compute distinct projections over a set of variable (see ticket BLZG-1163). This is exposed via method AST2BOpUtility.addHashIndexOp(). Currently, this method is used for SPARQL 1.1 subqueries (AST2BOpUtility.addSparql11Subquery) as well as for join groups (AST2BOpUtility.addSubgroup).

      However, there are some more positions in the code where we calculate hash indices and where the built-in projection capabilites might be useful. Consider the following methods:


      - AST2BOpUtility.addNamedSubqueryInclude()
      - AST2BOpUtility.addExistsSubqueryFast()
      - AST2BOpUtility.doMergeJoin()

      Decide whether it makes sense to pass in projection variables in these positions. Even if not, it might be desirable to call method addHashIndexOp() (or a slightly modified version thererof) instead of the duplicated code blocks for hash index construction).

        Issue Links

          Activity

          Hide
          michaelschmidt michaelschmidt added a comment -

          Here's another example that's slow on the Wikidata dataset and might benefit from DISTINCT projection into the FILTER NOT EXISTS clause:

          SELECT *
            WHERE {
              <http://www.wikidata.org/entity/Q30> ?statementPred ?statement .
              ?statement <http://www.w3.org/ns/prov#wasDerivedFrom> ?ref .
              
              # Since references are shared we can only clear the values on them when they are no longer used
              # anywhere else.
              FILTER NOT EXISTS {
                ?otherStatement <http://www.w3.org/ns/prov#wasDerivedFrom> ?ref .
                ?otherEntity ?otherStatementPred ?otherStatement .
                FILTER ( ?otherEntity != <http://www.wikidata.org/entity/Q23>  ) .
              }
              
              ?ref ?expandedValuePred ?s .
              ?s ?p ?o .
            }
          

          See also https://phabricator.wikimedia.org/T96094, we should update this thread once this has been resolved.

          Show
          michaelschmidt michaelschmidt added a comment - Here's another example that's slow on the Wikidata dataset and might benefit from DISTINCT projection into the FILTER NOT EXISTS clause: SELECT * WHERE { <http://www.wikidata.org/entity/Q30> ?statementPred ?statement . ?statement <http://www.w3.org/ns/prov#wasDerivedFrom> ?ref . # Since references are shared we can only clear the values on them when they are no longer used # anywhere else. FILTER NOT EXISTS { ?otherStatement <http://www.w3.org/ns/prov#wasDerivedFrom> ?ref . ?otherEntity ?otherStatementPred ?otherStatement . FILTER ( ?otherEntity != <http://www.wikidata.org/entity/Q23> ) . } ?ref ?expandedValuePred ?s . ?s ?p ?o . } See also https://phabricator.wikimedia.org/T96094 , we should update this thread once this has been resolved.
          Hide
          michaelschmidt michaelschmidt added a comment -

          This should be fixed along with the pipelined hash join. Verify by means of the query above and close if true.

          Show
          michaelschmidt michaelschmidt added a comment - This should be fixed along with the pipelined hash join. Verify by means of the query above and close if true.
          Hide
          michaelschmidt michaelschmidt added a comment -

          Verified code: this has been resolved as part of the JVM pipelined hash join implementation/refactoring. Closing issue.

          Show
          michaelschmidt michaelschmidt added a comment - Verified code: this has been resolved as part of the JVM pipelined hash join implementation/refactoring. Closing issue.

            People

            • Assignee:
              michaelschmidt michaelschmidt
              Reporter:
              michaelschmidt michaelschmidt
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: