Details

      Description

      Change the bigdata 1.1.x dependency to openrdf 2.6.3.

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        Work on the class loader problem (this ticket) and on changing over to Sesame 2.6.3.

        https://sourceforge.net/apps/trac/bigdata/ticket/496 (Sesame 2.6.3)

        Downloaded the onejar from sourceforge and generated new JARs from an SVN checkout using:

        mvn -Dmaven.test.skip=true clean install
        

        to generate:

        ./testsuites/sparql/target/sesame-sparql-testsuite-2.5.0.jar
        ./testsuites/store/target/sesame-store-testsuite-2.5.0.jar
        

        Update the classpath for Eclipse and updated the sesame version in build.properties.

        Removed the overridden version of QueryResultUtil from our code base since http://www.openrdf.org/issues/browse/SES-853 (QueryResultUtil fails when solutions have too many bindings.) was fixed in openrdf 2.6.0.

        http://sourceforge.net/apps/trac/bigdata/ticket/439 (Class loader problem)

        Moved all SPARQL grammar files related files from org.openrdf.query.parser.sparql into com.bigdata.rdf.sail.sparql and imported those few files which were not already present (NegatedPropertySet, PropertySet, and UpdateExprBuilder) into the com.bigdata.rdf.sail.sparql package. Those files all represent features which we do not yet support (property paths and SPARQL 1.1 UPDATE). I will review their contents in depth when I work those issues.

        Renamed the org.openrdf.query.parser.sparql.ast package to com.bigdata.rdf.sail.sparql.ast. This ensures that class path problems will not arise. Of course, it makes it more difficult to keep our version synchronized with the openrdf changes....

        Recompiled the SPARQL grammar to fix errors created by the package name change. I needed to change the package name in many of the AST files and also in the sparql.jjt grammar. Quite a PITA.

        TCK: 3 errors / 4 failures (6/8 if we run it as
        TestBigdataSailWithQuads which runs the TCK twice).

        Still broken:

        sparql11-sum-02 : This is failing because openrdf still expects an
        empty solution set rather than a single solution with a ZERO for the
        sum.  This is http://www.openrdf.org/issues/browse/SES-884
        (Aggregation with an solution set as input should produce an empty
        solution as output), which is clearly not fixed yet.  Just checked
        jira. They do not have a fix version for this yet.
        

        Newly broken: None. All pre-existing tests which were passing are still passing.

        Newly fixed:

        sparql11-substr-01
        sparql11-substr-02
        sparql11-substr-03
        sparql11-minus-05
        sparql11-minus-06
        sparql11-minus-07
        sparql11-sum-03
        sparql11-sum-04
        

        The substr tests were known to be broken in openrdf 2.5.0.

        The minus tests also reflected a problem with openrdf 2.5.0. I need to go back and look at them again to recall precisely what was failing.

        There were at least two known problems with aggregation in 2.5.0 (see the links below, only one of which has been fixed. The issue which was fixed has to do with handling of type errors during aggregation and was demonstrated by sparql11-sum-03. See http://www.openrdf.org/issues/browse/SES-862 (Incorrect error handling for SPARQL aggregation; fix in 2.6.1)

        New tests which are broken:

        SPARQL 1.1 BINDINGS 01   (Failing on the BINDINGS clause.)
        SPARQL 1.1 BINDINGS 02   (ditto)
        sparql11-subquery-04     (Port into the AST TestTCK suite and debug there)
        BSBM BI use case query 5 (This appears to be part of the Sesame TCK; fails with named solution set not found error)
        

        Committed revision r6046.

        Show
        bryanthompson bryanthompson added a comment - Work on the class loader problem (this ticket) and on changing over to Sesame 2.6.3. https://sourceforge.net/apps/trac/bigdata/ticket/496 (Sesame 2.6.3) Downloaded the onejar from sourceforge and generated new JARs from an SVN checkout using: mvn -Dmaven.test.skip=true clean install to generate: ./testsuites/sparql/target/sesame-sparql-testsuite-2.5.0.jar ./testsuites/store/target/sesame-store-testsuite-2.5.0.jar Update the classpath for Eclipse and updated the sesame version in build.properties. Removed the overridden version of QueryResultUtil from our code base since http://www.openrdf.org/issues/browse/SES-853 (QueryResultUtil fails when solutions have too many bindings.) was fixed in openrdf 2.6.0. http://sourceforge.net/apps/trac/bigdata/ticket/439 (Class loader problem) Moved all SPARQL grammar files related files from org.openrdf.query.parser.sparql into com.bigdata.rdf.sail.sparql and imported those few files which were not already present (NegatedPropertySet, PropertySet, and UpdateExprBuilder) into the com.bigdata.rdf.sail.sparql package. Those files all represent features which we do not yet support (property paths and SPARQL 1.1 UPDATE). I will review their contents in depth when I work those issues. Renamed the org.openrdf.query.parser.sparql.ast package to com.bigdata.rdf.sail.sparql.ast. This ensures that class path problems will not arise. Of course, it makes it more difficult to keep our version synchronized with the openrdf changes.... Recompiled the SPARQL grammar to fix errors created by the package name change. I needed to change the package name in many of the AST files and also in the sparql.jjt grammar. Quite a PITA. TCK: 3 errors / 4 failures (6/8 if we run it as TestBigdataSailWithQuads which runs the TCK twice). Still broken: sparql11-sum-02 : This is failing because openrdf still expects an empty solution set rather than a single solution with a ZERO for the sum. This is http://www.openrdf.org/issues/browse/SES-884 (Aggregation with an solution set as input should produce an empty solution as output), which is clearly not fixed yet. Just checked jira. They do not have a fix version for this yet. Newly broken: None. All pre-existing tests which were passing are still passing. Newly fixed: sparql11-substr-01 sparql11-substr-02 sparql11-substr-03 sparql11-minus-05 sparql11-minus-06 sparql11-minus-07 sparql11-sum-03 sparql11-sum-04 The substr tests were known to be broken in openrdf 2.5.0. The minus tests also reflected a problem with openrdf 2.5.0. I need to go back and look at them again to recall precisely what was failing. There were at least two known problems with aggregation in 2.5.0 (see the links below, only one of which has been fixed. The issue which was fixed has to do with handling of type errors during aggregation and was demonstrated by sparql11-sum-03. See http://www.openrdf.org/issues/browse/SES-862 (Incorrect error handling for SPARQL aggregation; fix in 2.6.1) New tests which are broken: SPARQL 1.1 BINDINGS 01 (Failing on the BINDINGS clause.) SPARQL 1.1 BINDINGS 02 (ditto) sparql11-subquery-04 (Port into the AST TestTCK suite and debug there) BSBM BI use case query 5 (This appears to be part of the Sesame TCK; fails with named solution set not found error) Committed revision r6046.
        Hide
        bryanthompson bryanthompson added a comment -

        Per the commit notes above, the dependency change to Sesame 2.6.3 appears to be Ok. There are some new test failures, but they are for new tests. I will debug those while I work through the UPDATE and property path support.

        I have filed an issue [1] for STRBEFORE, STRAFTER, and REPLACE. These methods are new in the most current SPARQL 1.1 LCWD.

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/497 (Implement STRBEFORE, STRAFTER, and REPLACE)

        Show
        bryanthompson bryanthompson added a comment - Per the commit notes above, the dependency change to Sesame 2.6.3 appears to be Ok. There are some new test failures, but they are for new tests. I will debug those while I work through the UPDATE and property path support. I have filed an issue [1] for STRBEFORE, STRAFTER, and REPLACE. These methods are new in the most current SPARQL 1.1 LCWD. [1] https://sourceforge.net/apps/trac/bigdata/ticket/497 (Implement STRBEFORE, STRAFTER, and REPLACE)
        Hide
        bryanthompson bryanthompson added a comment -

        Captured changes in the SPARQL grammar from 2.5.0 through 2.6.3. I did this by comparing those Sesame SVN versions using an external diff tool (WinMerge). This made it pretty easy to identify the delta.

          - SERVICE keyword and extraction of the string content for the graph
            graph pattern.  The SERVICE stuff is still not live, but it is now
            part of the grammar.  Remaining steps include ASTVisitor actions
            for SERVICE, BINDINGS, etc. and integration with bi-direction
            SERVICE invocation for 3rd party operators. This will be handled
            under a different issue.
        
          - Updated sparql.jjt and recompile.
        
            - Picked up SERVICE, BINDINGS, STRBEFORE, STRAFTER, REPLACE.
        
            - In and NotIn are now marked Infix(2).  This goes back to the
              problems that I was having with IN in the 2.5.0 grammar. This
              change is already present in our grammar.  Note: Our AST visitor
              production for Infix and the use of that production in the
              grammar differs slightly from the Sesame AST visitor productions.
        
          - BaseDeclProcessor: Added ASTIRIFunc and ASTServiceGraphPattern to
            internal the private static RelativeIRIResolver class.
        
          - ASTIRIFunc: Added baseURI property with getter and settor.
        
          - ASTServiceGraphPattern: Class was absent since SERVICE keyword was
            not in the grammar.  Modified the grammar and recompiled to
            generate the class.  Compared generated class to the 2.6.3 class
            and accepted delta from 2.6.3.
        
          - ASTQuery: picked up getBindingsClause().
        
          - ASTBindingsClause: picked up (new in the grammar).
        
          - ASTOperationContainer: picked up the [source] property.
        
          - ASTRDFValue: Sesame 2.6.3 has accepted my proposed change to
            introduce an ASTRDFValue (extends ASTSimpleNode).  This change is
            already be present in our code base.
        
          - ASTVisitorBase: Brought across new methods.
        
          - TupleExprBuilder => BigdataExprBuilder
        
              Note: There are numerous small changes in how Sesame 2.6.3
              manages the top-level query constructs (Select, Ask, Construct,
              Describe).  Some of these changes undoubtedly have to do with
              the BINDINGS clause.  
        

        Committed revision r6048.

        Show
        bryanthompson bryanthompson added a comment - Captured changes in the SPARQL grammar from 2.5.0 through 2.6.3. I did this by comparing those Sesame SVN versions using an external diff tool (WinMerge). This made it pretty easy to identify the delta. - SERVICE keyword and extraction of the string content for the graph graph pattern. The SERVICE stuff is still not live, but it is now part of the grammar. Remaining steps include ASTVisitor actions for SERVICE, BINDINGS, etc. and integration with bi-direction SERVICE invocation for 3rd party operators. This will be handled under a different issue. - Updated sparql.jjt and recompile. - Picked up SERVICE, BINDINGS, STRBEFORE, STRAFTER, REPLACE. - In and NotIn are now marked Infix(2). This goes back to the problems that I was having with IN in the 2.5.0 grammar. This change is already present in our grammar. Note: Our AST visitor production for Infix and the use of that production in the grammar differs slightly from the Sesame AST visitor productions. - BaseDeclProcessor: Added ASTIRIFunc and ASTServiceGraphPattern to internal the private static RelativeIRIResolver class. - ASTIRIFunc: Added baseURI property with getter and settor. - ASTServiceGraphPattern: Class was absent since SERVICE keyword was not in the grammar. Modified the grammar and recompiled to generate the class. Compared generated class to the 2.6.3 class and accepted delta from 2.6.3. - ASTQuery: picked up getBindingsClause(). - ASTBindingsClause: picked up (new in the grammar). - ASTOperationContainer: picked up the [source] property. - ASTRDFValue: Sesame 2.6.3 has accepted my proposed change to introduce an ASTRDFValue (extends ASTSimpleNode). This change is already be present in our code base. - ASTVisitorBase: Brought across new methods. - TupleExprBuilder => BigdataExprBuilder Note: There are numerous small changes in how Sesame 2.6.3 manages the top-level query constructs (Select, Ask, Construct, Describe). Some of these changes undoubtedly have to do with the BINDINGS clause. Committed revision r6048.
        Hide
        bryanthompson bryanthompson added a comment -

        Several files did not make it into the last commit for some reason.

        Committed revision r6049.

        Show
        bryanthompson bryanthompson added a comment - Several files did not make it into the last commit for some reason. Committed revision r6049.
        Hide
        bryanthompson bryanthompson added a comment -

        I have added a ticket to bring the RDF/XML parser support up to date [1].

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/498 (Bring bigdata RDF/XML parser up to openrdf 2.6.3)

        Show
        bryanthompson bryanthompson added a comment - I have added a ticket to bring the RDF/XML parser support up to date [1] . [1] https://sourceforge.net/apps/trac/bigdata/ticket/498 (Bring bigdata RDF/XML parser up to openrdf 2.6.3)

          People

          • Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: