Details

      Description

      Take a non-empty quad store, the following query gives no results

      select DISTINCT ?x 
      { 
         { select DISTINCT ?x 
          {GRAPH ?x
           { ?s ?p ?o}
          }
         }
        FILTER ( ?x != <eg:b>)
        FILTER ( ?x != <eg:b>)
      }
      

      I have failed to simplify further.

      The problem seems to be in the AttachJoinFilters optimizer

      INFO : 27637      com.bigdata.rdf.sail.webapp.BigdataRDFContext.queryService1 com.bigdata.rdf.sparql.ast.optimizers.ASTOptimizerList.optimize(ASTOptimizerList.java:102): Applying: com.bigdata.rdf.sparql.ast.optimizers.ASTStaticJoinOptimizer@15071f28
      DEBUG: 27639      com.bigdata.rdf.sail.webapp.BigdataRDFContext.queryService1 com.bigdata.rdf.sparql.ast.optimizers.ASTOptimizerList.optimize(ASTOptimizerList.java:110): Rewritten AST:
      
      QueryType: SELECT
      includeInferred=true
      SELECT DISTINCT ( VarNode(x) AS VarNode(x) )
        JoinGroupNode {
          FILTER( FunctionNode(VarNode(x),ConstantNode(TermId(58585U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(x,TermId(58585U)[eg:b])[ CompareBOp.op=NE]] )
          FILTER( FunctionNode(VarNode(x),ConstantNode(TermId(58585U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(x,TermId(58585U)[eg:b])[ CompareBOp.op=NE]] )
          QueryType: SELECT
          SELECT DISTINCT ( VarNode(x) AS VarNode(x) )
            JoinGroupNode [context=VarNode(x)] {
              StatementPatternNode(VarNode(s), VarNode(p), VarNode(o), VarNode(x)) [scope=NAMED_CONTEXTS]
                AST2BOpBase.estimatedCardinality=83545
                AST2BOpBase.originalIndex=SPOC
            }
      
        }
      INFO : 27639      com.bigdata.rdf.sail.webapp.BigdataRDFContext.queryService1 com.bigdata.rdf.sparql.ast.optimizers.ASTOptimizerList.optimize(ASTOptimizerList.java:102): Applying: com.bigdata.rdf.sparql.ast.optimizers.ASTAttachJoinFiltersOptimizer@464826ae
      DEBUG: 27641      com.bigdata.rdf.sail.webapp.BigdataRDFContext.queryService1 com.bigdata.rdf.sparql.ast.optimizers.ASTOptimizerList.optimize(ASTOptimizerList.java:110): Rewritten AST:
      
      QueryType: SELECT
      includeInferred=true
      SELECT DISTINCT ( VarNode(x) AS VarNode(x) )
        JoinGroupNode {
          FILTER( FunctionNode(VarNode(x),ConstantNode(TermId(58585U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(x,TermId(58585U)[eg:b])[ CompareBOp.op=NE]] )
          QueryType: SELECT
          SELECT DISTINCT ( VarNode(x) AS VarNode(x) )
            JoinGroupNode [context=VarNode(x)] {
              StatementPatternNode(VarNode(s), VarNode(p), VarNode(o), VarNode(x)) [scope=NAMED_CONTEXTS]
                AST2BOpBase.estimatedCardinality=83545
                AST2BOpBase.originalIndex=SPOC
            }
      
            FILTER( FunctionNode(VarNode(x),ConstantNode(TermId(58585U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(x,TermId(58585U)[eg:b])[ CompareBOp.op=NE]] )
        }
      
      

        Activity

        Hide
        michaelschmidt michaelschmidt added a comment -

        Further simplified query:

        select ?s
        { 
          { SELECT ?s WHERE { ?s ?p ?o } }
          FILTER ( ?s != <eg:b>) 
          FILTER ( ?s != <eg:b>) 
        }
        

        with optimized AST

        QueryType: SELECT
        includeInferred=true
        SELECT ( VarNode(s) AS VarNode(s) )
          JoinGroupNode {
            FILTER( FunctionNode(VarNode(s),ConstantNode(TermId(0U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(s,TermId(0U)[eg:b])[ CompareBOp.op=NE]] )
            QueryType: SELECT
            SELECT ( VarNode(s) AS VarNode(s) )
              JoinGroupNode {
                StatementPatternNode(VarNode(s), VarNode(p), VarNode(o)) [scope=DEFAULT_CONTEXTS]
                  AST2BOpBase.estimatedCardinality=15
                  AST2BOpBase.originalIndex=SPOC
              }
            
              FILTER( FunctionNode(VarNode(s),ConstantNode(TermId(0U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(s,TermId(0U)[eg:b])[ CompareBOp.op=NE]] )
          }
        

        -> while the simpler query

        select ?s
        { 
          { SELECT ?s WHERE { ?s ?p ?o } }
          FILTER ( ?s != <eg:b>) 
        }
        

        works fine with optimized AST

        QueryType: SELECT
        includeInferred=true
        SELECT ( VarNode(s) AS VarNode(s) )
          JoinGroupNode {
            QueryType: SELECT
            SELECT ( VarNode(s) AS VarNode(s) )
              JoinGroupNode {
                StatementPatternNode(VarNode(s), VarNode(p), VarNode(o)) [scope=DEFAULT_CONTEXTS]
                  AST2BOpBase.estimatedCardinality=15
                  AST2BOpBase.originalIndex=SPOC
              }
            
              FILTER( FunctionNode(VarNode(s),ConstantNode(TermId(0U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(s,TermId(0U)[eg:b])[ CompareBOp.op=NE]] )
          }
        
        Show
        michaelschmidt michaelschmidt added a comment - Further simplified query: select ?s { { SELECT ?s WHERE { ?s ?p ?o } } FILTER ( ?s != <eg:b>) FILTER ( ?s != <eg:b>) } with optimized AST QueryType: SELECT includeInferred=true SELECT ( VarNode(s) AS VarNode(s) ) JoinGroupNode { FILTER( FunctionNode(VarNode(s),ConstantNode(TermId(0U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(s,TermId(0U)[eg:b])[ CompareBOp.op=NE]] ) QueryType: SELECT SELECT ( VarNode(s) AS VarNode(s) ) JoinGroupNode { StatementPatternNode(VarNode(s), VarNode(p), VarNode(o)) [scope=DEFAULT_CONTEXTS] AST2BOpBase.estimatedCardinality=15 AST2BOpBase.originalIndex=SPOC } FILTER( FunctionNode(VarNode(s),ConstantNode(TermId(0U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(s,TermId(0U)[eg:b])[ CompareBOp.op=NE]] ) } -> while the simpler query select ?s { { SELECT ?s WHERE { ?s ?p ?o } } FILTER ( ?s != <eg:b>) } works fine with optimized AST QueryType: SELECT includeInferred=true SELECT ( VarNode(s) AS VarNode(s) ) JoinGroupNode { QueryType: SELECT SELECT ( VarNode(s) AS VarNode(s) ) JoinGroupNode { StatementPatternNode(VarNode(s), VarNode(p), VarNode(o)) [scope=DEFAULT_CONTEXTS] AST2BOpBase.estimatedCardinality=15 AST2BOpBase.originalIndex=SPOC } FILTER( FunctionNode(VarNode(s),ConstantNode(TermId(0U)[eg:b]))[ FunctionNode.scalarVals=null, FunctionNode.functionURI=http://www.w3.org/2005/xpath-functions#not-equal-to, valueExpr=com.bigdata.rdf.internal.constraints.CompareBOp(s,TermId(0U)[eg:b])[ CompareBOp.op=NE]] ) }
        Hide
        michaelschmidt michaelschmidt added a comment -

        Fixed in branch bugfix-duplicate-filter. The problem was that the ASTAttachJoinFilterRewriter removed the outer filter expression only once, since in the SA phase it is optimized to a single FILTER expressions. Fixed by removing iteratively.

        Bugfix including test cases, pull request created.

        As a follow-up, I'll open a general enhancement request to implement a new Filter optimizer, which tries to identify redundant filter expressions (possibly also decomposing filters) and removes them before applying the ASTAttachJoinFilterRewriter optimization.

        Show
        michaelschmidt michaelschmidt added a comment - Fixed in branch bugfix-duplicate-filter. The problem was that the ASTAttachJoinFilterRewriter removed the outer filter expression only once, since in the SA phase it is optimized to a single FILTER expressions. Fixed by removing iteratively. Bugfix including test cases, pull request created. As a follow-up, I'll open a general enhancement request to implement a new Filter optimizer, which tries to identify redundant filter expressions (possibly also decomposing filters) and removes them before applying the ASTAttachJoinFilterRewriter optimization.

          People

          • Assignee:
            michaelschmidt michaelschmidt
            Reporter:
            jeremycarroll jeremycarroll
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: