Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1956

SPARQL UPDATE should use a SolutionSetStream backed by native memory for DELETE/INSERT+WHERE to avoid OOM

    Details

    • Type: Improvement
    • Status: Done
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_2_2_0
    • Component/s: SPARQL UPDATE
    • Labels:
      None

      Description

      SPARQL UPDATE currently runs the query into a MutableTupleQueryResult and then replays it for both the DELETE and the INSERT aspect of a DELETE/INSERT+WHERE query. This ensures proper isolation of the query and the mutation. However, this means that large result sets can be materialized on the Java heap.

      This ticket is to replace the use of the MutableTupleQueryResult with a SolutionSetStream backed by the native heap (memstore).

        Issue Links

          Activity

          bryanthompson bryanthompson created issue -
          bryanthompson bryanthompson made changes -
          Field Original Value New Value
          Fix Version/s BLAZEGRAPH_2_2_0 [ 10302 ]
          bryanthompson bryanthompson made changes -
          Assignee Brad Bebee [ beebs ] michaelschmidt [ michaelschmidt ]
          Hide
          bryanthompson bryanthompson added a comment -

          See SolutionSetManager for patterns for creating, populating, playing back and destroying solution set stream objects. However, be sure to use a memstore instance that is scoped to the UPDATE request or even the specific DELETE/INSERT + WHERE operation.

              @Override
              public void putSolutions(final String solutionSet,
                      final ICloseableIterator<IBindingSet[]> src) {
          
                  if (src == null)
                      throw new IllegalArgumentException();
          
                  final String fqn = getFQN(solutionSet);
          
                  SolutionSetStream sset = getSolutionSet(fqn);
          
                  if (sset == null) {
          
                      sset = _create(fqn, getDefaultMetadata());
          
                  }
          
                  // write out the solutions.
                  writeSolutions(sset, src);
          
              }
          
          
          Show
          bryanthompson bryanthompson added a comment - See SolutionSetManager for patterns for creating, populating, playing back and destroying solution set stream objects. However, be sure to use a memstore instance that is scoped to the UPDATE request or even the specific DELETE/INSERT + WHERE operation. @Override public void putSolutions( final String solutionSet, final ICloseableIterator<IBindingSet[]> src) { if (src == null ) throw new IllegalArgumentException(); final String fqn = getFQN(solutionSet); SolutionSetStream sset = getSolutionSet(fqn); if (sset == null ) { sset = _create(fqn, getDefaultMetadata()); } // write out the solutions. writeSolutions(sset, src); }
          bryanthompson bryanthompson made changes -
          Status Open [ 1 ] Accepted [ 10101 ]
          bryanthompson bryanthompson made changes -
          Link This issue relates to BLZG-637 [ BLZG-637 ]
          Hide
          michaelschmidt michaelschmidt added a comment -

          Brad, as discussed last Friday, it would be great if you could trigger a manual stand-alone run of the failing tests on the CI server:

          Brad Bebee bryanthompson

          Show
          michaelschmidt michaelschmidt added a comment - Brad, as discussed last Friday, it would be great if you could trigger a manual stand-alone run of the failing tests on the CI server: Latest failing CI run: https://ci.blazegraph.com/job/bigdata-github-maven-PR-tester/949/ Branch: https://github.com/SYSTAP/bigdata/pull/434 (PR https://github.com/SYSTAP/bigdata/tree/BLZG-1956 ) Brad Bebee bryanthompson
          Hide
          bryanthompson bryanthompson added a comment -

          Brad Bebee We are now able to recreate the failure mode locally after merging in master.

          Show
          bryanthompson bryanthompson added a comment - Brad Bebee We are now able to recreate the failure mode locally after merging in master.
          Hide
          bryanthompson bryanthompson added a comment -

          It appears that LexiconRelation.getTerms() is simply undefined about whether or not the BigdataValue.setIV() will be called to set the IV on the BigdataValue. My inclination is that this should be part of the contract for getTerms(), but we need to review the code around assumptions for this behavior.

          mikepersonick
          michaelschmidt

          Show
          bryanthompson bryanthompson added a comment - It appears that LexiconRelation.getTerms() is simply undefined about whether or not the BigdataValue.setIV() will be called to set the IV on the BigdataValue. My inclination is that this should be part of the contract for getTerms(), but we need to review the code around assumptions for this behavior. mikepersonick michaelschmidt
          bryanthompson bryanthompson made changes -
          Status Accepted [ 10101 ] In Progress [ 3 ]
          Hide
          bryanthompson bryanthompson added a comment -

          Code review is done. Merged to master.

          Show
          bryanthompson bryanthompson added a comment - Code review is done. Merged to master.
          bryanthompson bryanthompson made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          bryanthompson bryanthompson made changes -
          Status Resolved [ 5 ] In Review [ 10100 ]
          bryanthompson bryanthompson made changes -
          Resolution Done [ 10000 ]
          Status In Review [ 10100 ] Done [ 10000 ]

            People

            • Assignee:
              michaelschmidt michaelschmidt
              Reporter:
              bryanthompson bryanthompson
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: