Details

      Description

      Running the query over given data returns 3 results. In the middle of the query there is a commented "#optional" keyword. If I un-comment that and rerun the query I get 2 results. This means that changing a graph pattern to be optional somehow produces less solutions.

      select distinct ?p 
      ?age 
      ?name 
      ?type 
      ?post 
      ?postContent
      ?comment 
      ?cperson 
      where
      {
              
              {
                      select ?p
                      where  
                      {
                              ?p a <http://www.example.org/schema/Person> . 
                              optional{?p <http://www.example.org/schema/age> ?age.}
                      }
                      order by desc(?age)
                      LIMIT 1
              }
              optional{?p <http://www.example.org/schema/age> ?age.}
              
              {?p <http://www.example.org/schema/name> ?name.}
              {?p a ?type.}
              
             # OPTIONAL
              {       ?post a <http://www.example.org/schema/Post> . 
                      ?post <http://www.example.org/schema/postedBy> ?p.
                      ?post <http://www.example.org/schema/content> ?postContent.}
              
              OPTIONAL{      
                      ?comment a <http://www.example.org/schema/Comment> .
                      ?comment <http://www.example.org/schema/parentPost> ?post.
                      ?cperson a <http://www.example.org/schema/Person> .
                      ?comment <http://www.example.org/schema/postedBy> ?cperson .
              }
      }
      order by desc(?age)
      

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        The bug is in the ASTComplexOptimizer. If you comment out this line in DefaultOptimizers and then run the two versions of the query, you get the same number of results:

        Comment out this line in DefaultOptimizers.

                add(new ASTComplexOptionalOptimizer());
        

        It is turning the version of the query with the additional OPTIONAL into a very different query plan involving several named subqueries (bottom-up evaluation), a hash index build, and a merge join. I have not analyzed where the problem is in that query plan, but the two query plans are very different.

        Workaround: Disable the ASTComplexOptionalOptimizer.

        Show
        bryanthompson bryanthompson added a comment - The bug is in the ASTComplexOptimizer. If you comment out this line in DefaultOptimizers and then run the two versions of the query, you get the same number of results: Comment out this line in DefaultOptimizers. add(new ASTComplexOptionalOptimizer()); It is turning the version of the query with the additional OPTIONAL into a very different query plan involving several named subqueries (bottom-up evaluation), a hash index build, and a merge join. I have not analyzed where the problem is in that query plan, but the two query plans are very different. Workaround: Disable the ASTComplexOptionalOptimizer.
        Hide
        bryanthompson bryanthompson added a comment -

        We would like to take up this ticket this week. Can you please re-post the persons.nt file? The attachment was lost when we migrated to a privately host trac instance.

        Show
        bryanthompson bryanthompson added a comment - We would like to take up this ticket this week. Can you please re-post the persons.nt file? The attachment was lost when we migrated to a privately host trac instance.
        Hide
        michaelschmidt michaelschmidt added a comment -

        See com.bigdata.rdf.sparql.ast.eval.TestSubQuery.test_ticket_801b_complex_optionals() for test case and sample nt file.

        Show
        michaelschmidt michaelschmidt added a comment - See com.bigdata.rdf.sparql.ast.eval.TestSubQuery.test_ticket_801b_complex_optionals() for test case and sample nt file.
        Hide
        michaelschmidt michaelschmidt added a comment -

        Problem caused by two problems in the ASTComplexOptionalOptimizer:

        1.) Named subqueries generated by the optimizer (representing the OPTIONAL bodies) were joined together, ignoring the OPTIONAL semantics -> instead of simply joining them, subqueries are now encapsulated into optional join group nodes to fix the semantics
        2.) The second problem that popped up was concerning the DISTINCT operator, which was necessary (wrong multiplicity). To deal with this problem, the set of projected variables was corrected: it needs to include the projection vars as required by the "ancestor" query as well as the variables that will be bound in subsequent join groups (cf. code for details and comments)

        Running CI, will issue pull request if this goes through.

        Show
        michaelschmidt michaelschmidt added a comment - Problem caused by two problems in the ASTComplexOptionalOptimizer: 1.) Named subqueries generated by the optimizer (representing the OPTIONAL bodies) were joined together, ignoring the OPTIONAL semantics -> instead of simply joining them, subqueries are now encapsulated into optional join group nodes to fix the semantics 2.) The second problem that popped up was concerning the DISTINCT operator, which was necessary (wrong multiplicity). To deal with this problem, the set of projected variables was corrected: it needs to include the projection vars as required by the "ancestor" query as well as the variables that will be bound in subsequent join groups (cf. code for details and comments) Running CI, will issue pull request if this goes through.
        Hide
        bryanthompson bryanthompson added a comment -

        staged for 1.5.1 release.

        Show
        bryanthompson bryanthompson added a comment - staged for 1.5.1 release.

          People

          • Assignee:
            michaelschmidt michaelschmidt
            Reporter:
            feugen24 feugen24
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: