Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1475

variable visibility FILTER EXISTS inside FILTER NOT EXISTS

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_RELEASE_1_5_3
    • Component/s: None
    • Labels:
      None

      Description

      I have one triple in the store

      prefix : <http://localhost/>
      :unbound rdfs:label "unbound" .
      
      

      the following query correctly returns 0 results, but when uncommenting the FILTER EXISTS I get 1 result

      prefix : <http://localhost/>
      
      select *
      { :unbound rdfs:label ?A 
        FILTER NOT EXISTS {
          :unbound rdfs:label ?A .
          # FILTER EXISTS {
            :unbound rdfs:label ?A 
          # }
        }
      }            
                       
       

        Activity

        Hide
        bryanthompson bryanthompson added a comment - - edited

        I am using the 1.5.2 maintenance branch. I am not able to replicate your experience on ticket as written. Your query against BLAZEGRAPH_RELEASE_1_5_2_MAINTENANCE returns NO solutions for the original query on the ticket both with and without the "#" comment lines enabled.

        You report no solutions as given and 1 solution when the comment is removed.

        I observe a different behavior locally using the 1.5.2 maintenance branch.

        Can you please try to replicate against that branch?

        Show
        bryanthompson bryanthompson added a comment - - edited I am using the 1.5.2 maintenance branch. I am not able to replicate your experience on ticket as written. Your query against BLAZEGRAPH_RELEASE_1_5_2_MAINTENANCE returns NO solutions for the original query on the ticket both with and without the "#" comment lines enabled. You report no solutions as given and 1 solution when the comment is removed. I observe a different behavior locally using the 1.5.2 maintenance branch. Can you please try to replicate against that branch?
        Show
        bryanthompson bryanthompson added a comment - Try https://sourceforge.net/p/bigdata/git/ci/BLAZEGRAPH_RELEASE_1_5_2_MAINTENANCE/tree/
        Hide
        bryanthompson bryanthompson added a comment - - edited

        Jeremy and I have cleaned up the example and I am now able to replicate the problem. Jeremy has a workaround on his end (he can rewrite the inner FILTER EXISTS into a SELECT DISTINCT).

        I managed to capture the Optimized AST and query plan for one of these variations (FILTER EXISTS within a FILTER EXISTS).

        prefix : <http://localhost/>
        
        select *
        { :unbound rdfs:label ?A 
          FILTER EXISTS {
            :unbound rdfs:label ?A .
             FILTER EXISTS {
              :unbound rdfs:label ?B 
             }
          }
        }
        
        PREFIX : <http://localhost/>
        PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
        PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
        PREFIX sesame: <http://www.openrdf.org/schema/sesame#>
        PREFIX owl: <http://www.w3.org/2002/07/owl#>
        PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
        PREFIX fn: <http://www.w3.org/2005/xpath-functions#>
        PREFIX foaf: <http://xmlns.com/foaf/0.1/>
        PREFIX dc: <http://purl.org/dc/elements/1.1/>
        PREFIX hint: <http://www.bigdata.com/queryHints#>
        PREFIX bd: <http://www.bigdata.com/rdf#>
        PREFIX bds: <http://www.bigdata.com/rdf/search#>
        QueryType: SELECT
        includeInferred=true
        SELECT VarNode(A)
          JoinGroupNode {
            StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(A)) [scope=DEFAULT_CONTEXTS]
              AST2BOpBase.estimatedCardinality=1
              AST2BOpBase.originalIndex=SPOC
            QueryType: ASK
            SELECT VarNode(A) VarNode(-exists-1)[anonymous]
              JoinGroupNode {
                FILTER( ExistsNode(VarNode(-exists-2))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.bigdata.com/sparql-1.1-undefined-functionsexists, graphPattern=
                  JoinGroupNode {
                    StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(B)) [scope=DEFAULT_CONTEXTS]
                      AST2BOpBase.estimatedCardinality=1
                      AST2BOpBase.originalIndex=SPOC
                  } AST2BOpBase.estimatedCardinality=1, valueExpr=-exists-2] )
                StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(A)) [scope=DEFAULT_CONTEXTS]
                  AST2BOpBase.estimatedCardinality=1
                  AST2BOpBase.originalIndex=SPOC
              } AST2BOpBase.estimatedCardinality=1
            @askVar=-exists-1
            FILTER( ExistsNode(VarNode(-exists-1))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.bigdata.com/sparql-1.1-undefined-functionsexists, graphPattern=
              JoinGroupNode {
                FILTER( ExistsNode(VarNode(-exists-2))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.bigdata.com/sparql-1.1-undefined-functionsexists, graphPattern=
                  JoinGroupNode {
                    StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(B)) [scope=DEFAULT_CONTEXTS]
                      AST2BOpBase.estimatedCardinality=1
                      AST2BOpBase.originalIndex=SPOC
                  } AST2BOpBase.estimatedCardinality=1, valueExpr=-exists-2] )
                StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(A)) [scope=DEFAULT_CONTEXTS]
                  AST2BOpBase.estimatedCardinality=1
                  AST2BOpBase.originalIndex=SPOC
              } AST2BOpBase.estimatedCardinality=1, valueExpr=-exists-1] )
          }
        
        with static (exogeneous) bindings defined as follows: 
        {
          {  }
        }
        Query Plan
        
        com.bigdata.bop.solutions.ProjectionOp[17](DropOp[16])[ BOp.bopId=17, BOp.evaluationContext=CONTROLLER, PipelineOp.sharedState=true, JoinAnnotations.select=[A], QueryEngine.queryId=18e6fdf6-c3b9-478b-be10-8e5a6b413b80]
          com.bigdata.bop.solutions.DropOp[16](ConditionalRoutingOp[13])[ BOp.bopId=16, DropOp.dropVars=[-exists-1]]
            com.bigdata.bop.bset.ConditionalRoutingOp[13](ChunkedMaterializationOp[15])[ BOp.bopId=13, ConditionalRoutingOp.condition=com.bigdata.rdf.internal.constraints.SPARQLConstraint(com.bigdata.rdf.internal.constraints.EBVBOp(-exists-1))]
              com.bigdata.bop.rdf.join.ChunkedMaterializationOp[15](ConditionalRoutingOp[14])[ ChunkedMaterializationOp.vars=[-exists-1], IPredicate.relationName=[syapse.lex], IPredicate.timestamp=1441391722558, ChunkedMaterializationOp.materializeAll=false, PipelineOp.sharedState=true, PipelineOp.reorderSolutions=true, PipelineOp.maxParallel=5, BOp.bopId=15]
                com.bigdata.bop.bset.ConditionalRoutingOp[14](JVMSolutionSetHashJoinOp[12])[ BOp.bopId=14, ConditionalRoutingOp.condition=com.bigdata.rdf.internal.constraints.SPARQLConstraint(com.bigdata.rdf.internal.constraints.NeedsMaterializationBOp(com.bigdata.rdf.internal.constraints.EBVBOp(-exists-1))), PipelineOp.altSinkRef=13]
                  com.bigdata.bop.join.JVMSolutionSetHashJoinOp[12](PipelineJoin[11])[ BOp.bopId=12, BOp.evaluationContext=CONTROLLER, PipelineOp.maxParallel=1, PipelineOp.sharedState=true, SolutionSetHashJoinOp.release=true, PipelineOp.lastPass=true, namedSetRef=NamedSolutionSetRef{localName=--set-4,queryId=18e6fdf6-c3b9-478b-be10-8e5a6b413b80,joinVars=[A]}]
                    com.bigdata.bop.join.PipelineJoin[11](ConditionalRoutingOp[6])[ BOp.bopId=11, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[9](TermId(5U)[http://localhost/unbound], Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label], A=null, --anon-10=null)[ IPredicate.relationName=[syapse.spo], IPredicate.timestamp=1441391722558, BOp.bopId=9, AST2BOpBase.estimatedCardinality=1, AST2BOpBase.originalIndex=SPOC, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL], IPredicate.accessPathFilter=cutthecrap.utils.striterators.NOPFilter@62e3d8e8{annotations=null,filterChain=[com.bigdata.bop.rdf.filter.StripContextFilter(), com.bigdata.bop.ap.filter.DistinctFilter()]}]]
                      com.bigdata.bop.bset.ConditionalRoutingOp[6](ChunkedMaterializationOp[8])[ BOp.bopId=6, ConditionalRoutingOp.condition=com.bigdata.rdf.internal.constraints.SPARQLConstraint(com.bigdata.rdf.internal.constraints.EBVBOp(-exists-2))]
                        com.bigdata.bop.rdf.join.ChunkedMaterializationOp[8](ConditionalRoutingOp[7])[ ChunkedMaterializationOp.vars=[-exists-2], IPredicate.relationName=[syapse.lex], IPredicate.timestamp=1441391722558, ChunkedMaterializationOp.materializeAll=false, PipelineOp.sharedState=true, PipelineOp.reorderSolutions=true, PipelineOp.maxParallel=5, BOp.bopId=8]
                          com.bigdata.bop.bset.ConditionalRoutingOp[7](JVMHashIndexOp[5])[ BOp.bopId=7, ConditionalRoutingOp.condition=com.bigdata.rdf.internal.constraints.SPARQLConstraint(com.bigdata.rdf.internal.constraints.NeedsMaterializationBOp(com.bigdata.rdf.internal.constraints.EBVBOp(-exists-2))), PipelineOp.altSinkRef=6]
                            com.bigdata.bop.join.JVMHashIndexOp[5](PipelineJoin[3])[ BOp.bopId=5, BOp.evaluationContext=CONTROLLER, PipelineOp.maxParallel=1, PipelineOp.lastPass=true, PipelineOp.sharedState=true, JoinAnnotations.joinType=Exists, HashJoinAnnotations.joinVars=[A], JoinAnnotations.constraints=null, JoinAnnotations.select=null, HashJoinAnnotations.askVar=-exists-1, namedSetRef=NamedSolutionSetRef{localName=--set-4,queryId=18e6fdf6-c3b9-478b-be10-8e5a6b413b80,joinVars=[A]}]
                              com.bigdata.bop.join.PipelineJoin[3]()[ BOp.bopId=3, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[1](TermId(5U)[http://localhost/unbound], Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label], A=null, --anon-2=null)[ IPredicate.relationName=[syapse.spo], IPredicate.timestamp=1441391722558, BOp.bopId=1, AST2BOpBase.estimatedCardinality=1, AST2BOpBase.originalIndex=SPOC, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL], IPredicate.accessPathFilter=cutthecrap.utils.striterators.NOPFilter@416f86a0{annotations=null,filterChain=[com.bigdata.bop.rdf.filter.StripContextFilter(), com.bigdata.bop.ap.filter.DistinctFilter()]}]]
        

        michaelschmidt

        Show
        bryanthompson bryanthompson added a comment - - edited Jeremy and I have cleaned up the example and I am now able to replicate the problem. Jeremy has a workaround on his end (he can rewrite the inner FILTER EXISTS into a SELECT DISTINCT). I managed to capture the Optimized AST and query plan for one of these variations (FILTER EXISTS within a FILTER EXISTS). prefix : <http://localhost/> select * { :unbound rdfs:label ?A FILTER EXISTS { :unbound rdfs:label ?A . FILTER EXISTS { :unbound rdfs:label ?B } } } PREFIX : <http://localhost/> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> PREFIX sesame: <http://www.openrdf.org/schema/sesame#> PREFIX owl: <http://www.w3.org/2002/07/owl#> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX fn: <http://www.w3.org/2005/xpath-functions#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX dc: <http://purl.org/dc/elements/1.1/> PREFIX hint: <http://www.bigdata.com/queryHints#> PREFIX bd: <http://www.bigdata.com/rdf#> PREFIX bds: <http://www.bigdata.com/rdf/search#> QueryType: SELECT includeInferred=true SELECT VarNode(A) JoinGroupNode { StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(A)) [scope=DEFAULT_CONTEXTS] AST2BOpBase.estimatedCardinality=1 AST2BOpBase.originalIndex=SPOC QueryType: ASK SELECT VarNode(A) VarNode(-exists-1)[anonymous] JoinGroupNode { FILTER( ExistsNode(VarNode(-exists-2))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.bigdata.com/sparql-1.1-undefined-functionsexists, graphPattern= JoinGroupNode { StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(B)) [scope=DEFAULT_CONTEXTS] AST2BOpBase.estimatedCardinality=1 AST2BOpBase.originalIndex=SPOC } AST2BOpBase.estimatedCardinality=1, valueExpr=-exists-2] ) StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(A)) [scope=DEFAULT_CONTEXTS] AST2BOpBase.estimatedCardinality=1 AST2BOpBase.originalIndex=SPOC } AST2BOpBase.estimatedCardinality=1 @askVar=-exists-1 FILTER( ExistsNode(VarNode(-exists-1))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.bigdata.com/sparql-1.1-undefined-functionsexists, graphPattern= JoinGroupNode { FILTER( ExistsNode(VarNode(-exists-2))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.bigdata.com/sparql-1.1-undefined-functionsexists, graphPattern= JoinGroupNode { StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(B)) [scope=DEFAULT_CONTEXTS] AST2BOpBase.estimatedCardinality=1 AST2BOpBase.originalIndex=SPOC } AST2BOpBase.estimatedCardinality=1, valueExpr=-exists-2] ) StatementPatternNode(ConstantNode(TermId(5U)[http://localhost/unbound]), ConstantNode(Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label]), VarNode(A)) [scope=DEFAULT_CONTEXTS] AST2BOpBase.estimatedCardinality=1 AST2BOpBase.originalIndex=SPOC } AST2BOpBase.estimatedCardinality=1, valueExpr=-exists-1] ) } with static (exogeneous) bindings defined as follows: { { } } Query Plan com.bigdata.bop.solutions.ProjectionOp[17](DropOp[16])[ BOp.bopId=17, BOp.evaluationContext=CONTROLLER, PipelineOp.sharedState=true, JoinAnnotations.select=[A], QueryEngine.queryId=18e6fdf6-c3b9-478b-be10-8e5a6b413b80] com.bigdata.bop.solutions.DropOp[16](ConditionalRoutingOp[13])[ BOp.bopId=16, DropOp.dropVars=[-exists-1]] com.bigdata.bop.bset.ConditionalRoutingOp[13](ChunkedMaterializationOp[15])[ BOp.bopId=13, ConditionalRoutingOp.condition=com.bigdata.rdf.internal.constraints.SPARQLConstraint(com.bigdata.rdf.internal.constraints.EBVBOp(-exists-1))] com.bigdata.bop.rdf.join.ChunkedMaterializationOp[15](ConditionalRoutingOp[14])[ ChunkedMaterializationOp.vars=[-exists-1], IPredicate.relationName=[syapse.lex], IPredicate.timestamp=1441391722558, ChunkedMaterializationOp.materializeAll=false, PipelineOp.sharedState=true, PipelineOp.reorderSolutions=true, PipelineOp.maxParallel=5, BOp.bopId=15] com.bigdata.bop.bset.ConditionalRoutingOp[14](JVMSolutionSetHashJoinOp[12])[ BOp.bopId=14, ConditionalRoutingOp.condition=com.bigdata.rdf.internal.constraints.SPARQLConstraint(com.bigdata.rdf.internal.constraints.NeedsMaterializationBOp(com.bigdata.rdf.internal.constraints.EBVBOp(-exists-1))), PipelineOp.altSinkRef=13] com.bigdata.bop.join.JVMSolutionSetHashJoinOp[12](PipelineJoin[11])[ BOp.bopId=12, BOp.evaluationContext=CONTROLLER, PipelineOp.maxParallel=1, PipelineOp.sharedState=true, SolutionSetHashJoinOp.release=true, PipelineOp.lastPass=true, namedSetRef=NamedSolutionSetRef{localName=--set-4,queryId=18e6fdf6-c3b9-478b-be10-8e5a6b413b80,joinVars=[A]}] com.bigdata.bop.join.PipelineJoin[11](ConditionalRoutingOp[6])[ BOp.bopId=11, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[9](TermId(5U)[http://localhost/unbound], Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label], A=null, --anon-10=null)[ IPredicate.relationName=[syapse.spo], IPredicate.timestamp=1441391722558, BOp.bopId=9, AST2BOpBase.estimatedCardinality=1, AST2BOpBase.originalIndex=SPOC, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL], IPredicate.accessPathFilter=cutthecrap.utils.striterators.NOPFilter@62e3d8e8{annotations=null,filterChain=[com.bigdata.bop.rdf.filter.StripContextFilter(), com.bigdata.bop.ap.filter.DistinctFilter()]}]] com.bigdata.bop.bset.ConditionalRoutingOp[6](ChunkedMaterializationOp[8])[ BOp.bopId=6, ConditionalRoutingOp.condition=com.bigdata.rdf.internal.constraints.SPARQLConstraint(com.bigdata.rdf.internal.constraints.EBVBOp(-exists-2))] com.bigdata.bop.rdf.join.ChunkedMaterializationOp[8](ConditionalRoutingOp[7])[ ChunkedMaterializationOp.vars=[-exists-2], IPredicate.relationName=[syapse.lex], IPredicate.timestamp=1441391722558, ChunkedMaterializationOp.materializeAll=false, PipelineOp.sharedState=true, PipelineOp.reorderSolutions=true, PipelineOp.maxParallel=5, BOp.bopId=8] com.bigdata.bop.bset.ConditionalRoutingOp[7](JVMHashIndexOp[5])[ BOp.bopId=7, ConditionalRoutingOp.condition=com.bigdata.rdf.internal.constraints.SPARQLConstraint(com.bigdata.rdf.internal.constraints.NeedsMaterializationBOp(com.bigdata.rdf.internal.constraints.EBVBOp(-exists-2))), PipelineOp.altSinkRef=6] com.bigdata.bop.join.JVMHashIndexOp[5](PipelineJoin[3])[ BOp.bopId=5, BOp.evaluationContext=CONTROLLER, PipelineOp.maxParallel=1, PipelineOp.lastPass=true, PipelineOp.sharedState=true, JoinAnnotations.joinType=Exists, HashJoinAnnotations.joinVars=[A], JoinAnnotations.constraints=null, JoinAnnotations.select=null, HashJoinAnnotations.askVar=-exists-1, namedSetRef=NamedSolutionSetRef{localName=--set-4,queryId=18e6fdf6-c3b9-478b-be10-8e5a6b413b80,joinVars=[A]}] com.bigdata.bop.join.PipelineJoin[3]()[ BOp.bopId=3, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[1](TermId(5U)[http://localhost/unbound], Vocab(25)[http://www.w3.org/2000/01/rdf-schema#label], A=null, --anon-2=null)[ IPredicate.relationName=[syapse.spo], IPredicate.timestamp=1441391722558, BOp.bopId=1, AST2BOpBase.estimatedCardinality=1, AST2BOpBase.originalIndex=SPOC, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL], IPredicate.accessPathFilter=cutthecrap.utils.striterators.NOPFilter@416f86a0{annotations=null,filterChain=[com.bigdata.bop.rdf.filter.StripContextFilter(), com.bigdata.bop.ap.filter.DistinctFilter()]}]] michaelschmidt
        Hide
        michaelschmidt michaelschmidt added a comment -

        I was also able to reproduce this fix, even with the previous 1.5.1 release. The problem is that nested FILTER (NOT) EXISTS are not recursively rewritten – in the AST above, note that the ASK subquery is missing in the inner expression. As a matter of fact, the FILTER expression always evaluated to false.

        Proposed a fix in branch BLZG-1475, which is currently running through CI, including the queries above (plus some variants) as test cases.

        Show
        michaelschmidt michaelschmidt added a comment - I was also able to reproduce this fix, even with the previous 1.5.1 release. The problem is that nested FILTER (NOT) EXISTS are not recursively rewritten – in the AST above, note that the ASK subquery is missing in the inner expression. As a matter of fact, the FILTER expression always evaluated to false. Proposed a fix in branch BLZG-1475 , which is currently running through CI, including the queries above (plus some variants) as test cases.
        Hide
        michaelschmidt michaelschmidt added a comment -

        CI fine, merged in BLZG-1475.

        Show
        michaelschmidt michaelschmidt added a comment - CI fine, merged in BLZG-1475 .

          People

          • Assignee:
            michaelschmidt michaelschmidt
            Reporter:
            jjc Jeremy Carroll
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: