Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-2030

Analytic flag in workbench / query hint ignored for SPARQL UPDATE

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_2_2_0
    • Component/s: SPARQL UPDATE
    • Labels:
      None

      Description

      Referring back to BLZG-2027, it looks like the anaytic checkbox does not (or not fully) have an effect in the UPDATE tab. In fact, I when starting the server in non-analytic mode and setting the checkbox for the second update query there (after having executed the first one once), I end up in JVMDistinctBindingSetsOp rather than HTreeDistinctBindingSetsOp. We need to review what happens, the code path should actually be the same.

        Activity

        Hide
        bryanthompson bryanthompson added a comment - - edited

        michaelschmidt Can you work up a reproducer for this? I did some looking yesterday at QueryServlet (which also handles SPARQL UPDATE) and I did not see anything obvious where the flag was not being passed through. Do you know if a query hint for the analytic query mode is respected for SPARQL UPDATE? If not, then maybe the problem is in AST2BOpUpdate. Otherwise I would continue to suspect QueryServlet => BigdataRDFContext.getUpdateTask() => UpdateTask.setup() / call().

        We should have a unit test for this in which the analytic mode is requested for SPARQL UPDATE and we verify that the generated plan uses analytic mode operators.

        Show
        bryanthompson bryanthompson added a comment - - edited michaelschmidt Can you work up a reproducer for this? I did some looking yesterday at QueryServlet (which also handles SPARQL UPDATE) and I did not see anything obvious where the flag was not being passed through. Do you know if a query hint for the analytic query mode is respected for SPARQL UPDATE? If not, then maybe the problem is in AST2BOpUpdate. Otherwise I would continue to suspect QueryServlet => BigdataRDFContext.getUpdateTask() => UpdateTask.setup() / call(). We should have a unit test for this in which the analytic mode is requested for SPARQL UPDATE and we verify that the generated plan uses analytic mode operators.
        Hide
        bryanthompson bryanthompson added a comment -

        BigdataRdfContext: The same code picks up the analytic flag for both SPARQL QUERY and SPARQL UPDATE.

        this.analytic = getEffectiveBooleanValue(
                            req.getParameter(ANALYTIC), QueryHints.DEFAULT_ANALYTIC);
        
        

        workbench.js: Here are the methods in the JavaScript client which submit SPARQL QUERY and SPARQL UPDATE requests.

        • function submitUpdate(e) {...}
          * function submitQuery(e) {...}

        For SPARQL QUERY, the settings object is established using this code:

           var settings = {
              type: 'POST',
              data: $('#query-form').serialize(),
              headers: { 'Accept': 'application/sparql-results+json' },
              success: showQueryResults,
              error: queryResultsError
           };
        

        For SPARQL UPDATE, the settings object is established using this code:

           var settings = {
              type: 'POST',
              data: FILE_CONTENTS === null ? EDITORS.update.getValue() : FILE_CONTENTS,
              success: updateResponseXML,
              error: updateResponseError
           };
        

        The following logic then addresses the analytic mode for SPARQL UPDATE (in the same method)

              case 'sparql':
                 // see if monitor mode is on
                 if($('#update-monitor').is(':checked')) {
                    // create form and submit it, sending output to the iframe
                    var form = $('<form id="update-monitor-form" method="POST" target="update-response-container">')
                       .attr('action', url)
                       .append($('<input name="update">').val(settings.data))
                       .append('<input name="monitor" value="true">');
                    if($('#update-analytic').is(':checked')) {
                       form.append('<input name="analytic" value="true">');
                    }
                    form.appendTo($('body'));
                    form.submit();
                    $('#update-monitor-form').remove();
                    $('#update-response iframe, #update-clear-container').show();
                    $('#update-response span').hide();
                    return;
                 }
                 settings.data = 'update=' + encodeURIComponent(settings.data);
                 if($('#update-analytic').is(':checked')) {
                    settings.data += '&analytic=true';
                 }
                 settings.success = updateResponseHTML;
                 break;
        
        Show
        bryanthompson bryanthompson added a comment - BigdataRdfContext: The same code picks up the analytic flag for both SPARQL QUERY and SPARQL UPDATE. this .analytic = getEffectiveBooleanValue( req.getParameter(ANALYTIC), QueryHints.DEFAULT_ANALYTIC); workbench.js: Here are the methods in the JavaScript client which submit SPARQL QUERY and SPARQL UPDATE requests. function submitUpdate(e) {...} * function submitQuery(e) {...} For SPARQL QUERY, the settings object is established using this code: var settings = { type: 'POST', data: $('#query-form').serialize(), headers: { 'Accept': 'application/sparql-results+json' }, success: showQueryResults, error: queryResultsError }; For SPARQL UPDATE, the settings object is established using this code: var settings = { type: 'POST', data: FILE_CONTENTS === null ? EDITORS.update.getValue() : FILE_CONTENTS, success: updateResponseXML, error: updateResponseError }; The following logic then addresses the analytic mode for SPARQL UPDATE (in the same method) case 'sparql': // see if monitor mode is on if ($('#update-monitor').is(':checked')) { // create form and submit it, sending output to the iframe var form = $('<form id= "update-monitor-form" method= "POST" target= "update-response-container" >') .attr('action', url) .append($('<input name= "update" >').val(settings.data)) .append('<input name= "monitor" value= " true " >'); if ($('#update-analytic').is(':checked')) { form.append('<input name= "analytic" value= " true " >'); } form.appendTo($('body')); form.submit(); $('#update-monitor-form').remove(); $('#update-response iframe, #update-clear-container').show(); $('#update-response span').hide(); return ; } settings.data = 'update=' + encodeURIComponent(settings.data); if ($('#update-analytic').is(':checked')) { settings.data += '&analytic= true '; } settings.success = updateResponseHTML; break ;
        Hide
        bryanthompson bryanthompson added a comment - - edited

        I have verified under the debugger that checking [x] Analytic in the workbench UPDATE page and submitting the following request:

        PREFIX dc: <http://purl.org/dc/elements/1.1/>
        INSERT DATA
        {
           <http://example/book1> dc:title "A new book" ;
                                  dc:creator "A.N.Other" .
        };
        
        DELETE DATA
        {
           <http://example/book1> dc:title "A new book" ;
                                  dc:creator "A.N.Other" .
        }
        

        Correctly extracts the analytic flag and then invokes the following logic for the SPARQL UPDATE to set the query hint (note that exactly the same incantation is used to set the analytic mode query hint for SPARQL QUERY).

                    if (analytic) {
        
                        // Turn analytic query on/off as requested.
                        astContainer.setQueryHint(QueryHints.ANALYTIC, "true");
        
                    }
        

        This is pretty conclusive that the issue is not in workbench.js

        Show
        bryanthompson bryanthompson added a comment - - edited I have verified under the debugger that checking [x] Analytic in the workbench UPDATE page and submitting the following request: PREFIX dc: <http://purl.org/dc/elements/1.1/> INSERT DATA { <http://example/book1> dc:title "A new book" ; dc:creator "A.N.Other" . }; DELETE DATA { <http://example/book1> dc:title "A new book" ; dc:creator "A.N.Other" . } Correctly extracts the analytic flag and then invokes the following logic for the SPARQL UPDATE to set the query hint (note that exactly the same incantation is used to set the analytic mode query hint for SPARQL QUERY). if (analytic) { // Turn analytic query on/off as requested. astContainer.setQueryHint(QueryHints.ANALYTIC, " true " ); } This is pretty conclusive that the issue is not in workbench.js
        Hide
        bryanthompson bryanthompson added a comment -

        When the hint is set, the AnalyticQueryHint handles it as follows:

            public void handle(final AST2BOpContext context, final QueryRoot queryRoot,
                    final QueryHintScope scope, final ASTBase op, final Boolean value) {
        
                switch (scope) {
                case Query:
                    context.nativeHashJoins = value;
                    context.nativeDistinctSolutions = value;
                    context.nativeDistinctSPO = value;
                    return;
                }
        
                throw new QueryHintException(scope, op, getName(), value);
        
            }
        
        
        Show
        bryanthompson bryanthompson added a comment - When the hint is set, the AnalyticQueryHint handles it as follows: public void handle( final AST2BOpContext context, final QueryRoot queryRoot, final QueryHintScope scope, final ASTBase op, final Boolean value) { switch (scope) { case Query: context.nativeHashJoins = value; context.nativeDistinctSolutions = value; context.nativeDistinctSPO = value; return ; } throw new QueryHintException(scope, op, getName(), value); }
        Hide
        bryanthompson bryanthompson added a comment - - edited

        Now looking at whether the hint (which has been set on the ASTContainer) is ever applied for SPARQL UPDATE. I am using the following W3C query (from the SPARQL 1.1 UPDATE rec) since the analytic mode is only used for DELETE/INSERT code paths since those are the only code paths which actually execute a WHERE clause.

        PREFIX foaf:  <http://xmlns.com/foaf/0.1/>
        
        WITH <http://example/addresses>
        DELETE { ?person foaf:givenName 'Bill' }
        INSERT { ?person foaf:givenName 'William' }
        WHERE
          { ?person foaf:givenName 'Bill'
          } 
        

        When the [x] Analytic flag is set, the following wind up still being false on AST2BOpContext when we enter AST2BOpUpdate.convertUpdate(). This means that we have not yet interpreted the query hints along this code path.

        • nativeDistinctSolutions false
        • nativeDistinctSPO false
        • nativeHashJoins false

        Those flags are still false when we enter AST2BOpUpdate.convertDeleteInsert().

        Perhaps ASTEvalHelper.executeUpdate() is not propagating the analytic mode information? It is responsible for propagating a bunch of things:

                    final AST2BOpUpdateContext ctx = new AST2BOpUpdateContext(
                            astContainer, conn);
        
                    doSparqlLogging(ctx);
        
                    // Propagate attribute.
                    ctx.setIncludeInferred(includeInferred);
                    
                    // Batch resolve Values to IVs and convert to bigdata binding set.
                    final IBindingSet[] bindingSets = toBindingSet(resolved.bindingSet) ;
                    
                    // Propagate bindings
                    ctx.setQueryBindingSet(bs);
                    ctx.setBindings(bindingSets);
                    ctx.setDataset(dataset);
        
                    /*
                     * Generate and execute physical plans for the update operations.
                     */
                    AST2BOpUpdate.convertUpdate(ctx);
        
        

        The original ASTContainer had the following entry in its attributes:

        • queryHints= {analytic=true, queryId=4a3e1762-73cd-4c84-a20e-667681b277ae}

        I still see that entry when we enter into ASTEvalHelper.executeUpdate() and it is present on the AST2BOpUpdateContext passed into AST2BOpUpdate.convertUpdate().

        Ah! AST2BOpUpdate.convertUpdate() creates a new ASTContainer. But it does not propagate the query hints to the new ASTContainer.

        • final ASTContainer astContainer = new ASTContainer(queryRoot);
        Show
        bryanthompson bryanthompson added a comment - - edited Now looking at whether the hint (which has been set on the ASTContainer) is ever applied for SPARQL UPDATE. I am using the following W3C query (from the SPARQL 1.1 UPDATE rec) since the analytic mode is only used for DELETE/INSERT code paths since those are the only code paths which actually execute a WHERE clause. PREFIX foaf: <http: //xmlns.com/foaf/0.1/> WITH <http: //example/addresses> DELETE { ?person foaf:givenName 'Bill' } INSERT { ?person foaf:givenName 'William' } WHERE { ?person foaf:givenName 'Bill' } When the [x] Analytic flag is set, the following wind up still being false on AST2BOpContext when we enter AST2BOpUpdate.convertUpdate(). This means that we have not yet interpreted the query hints along this code path. nativeDistinctSolutions false nativeDistinctSPO false nativeHashJoins false Those flags are still false when we enter AST2BOpUpdate.convertDeleteInsert(). Perhaps ASTEvalHelper.executeUpdate() is not propagating the analytic mode information? It is responsible for propagating a bunch of things: final AST2BOpUpdateContext ctx = new AST2BOpUpdateContext( astContainer, conn); doSparqlLogging(ctx); // Propagate attribute. ctx.setIncludeInferred(includeInferred); // Batch resolve Values to IVs and convert to bigdata binding set. final IBindingSet[] bindingSets = toBindingSet(resolved.bindingSet) ; // Propagate bindings ctx.setQueryBindingSet(bs); ctx.setBindings(bindingSets); ctx.setDataset(dataset); /* * Generate and execute physical plans for the update operations. */ AST2BOpUpdate.convertUpdate(ctx); The original ASTContainer had the following entry in its attributes: queryHints= {analytic=true, queryId=4a3e1762-73cd-4c84-a20e-667681b277ae} I still see that entry when we enter into ASTEvalHelper.executeUpdate() and it is present on the AST2BOpUpdateContext passed into AST2BOpUpdate.convertUpdate(). Ah! AST2BOpUpdate.convertUpdate() creates a new ASTContainer. But it does not propagate the query hints to the new ASTContainer. final ASTContainer astContainer = new ASTContainer(queryRoot);
        Hide
        bryanthompson bryanthompson added a comment -

        Ok. The fix is the following one liner in AST2BOpUpdate:

        Old:

                    final ASTContainer astContainer = new ASTContainer(queryRoot);
                    /*
                     * Inherit 'RESOLVED' flag, so resolution will not be run
                     * for ASTContainer, constructed from parts of already resolved update.
                     */
                    astContainer.setProperty(ASTContainer.Annotations.RESOLVED, context.astContainer.getProperty(ASTContainer.Annotations.RESOLVED));
        
        

        New:

                    final ASTContainer astContainer = new ASTContainer(queryRoot);
        
                    // Propagate the analytic mode query hint into the WHERE clause query.
                    // See BLZG-2030 (Analytic flag in workbench not working for UPDATE?)
                    astContainer.setQueryHint(QueryHints.ANALYTIC, Boolean.toString(context.astContainer.getQueryHintAsBoolean(QueryHints.ANALYTIC, QueryHints.DEFAULT_ANALYTIC)));
        
                    /*
                     * Inherit 'RESOLVED' flag, so resolution will not be run
                     * for ASTContainer, constructed from parts of already resolved update.
                     */
                    astContainer.setProperty(ASTContainer.Annotations.RESOLVED, context.astContainer.getProperty(ASTContainer.Annotations.RESOLVED));
        
        
        Show
        bryanthompson bryanthompson added a comment - Ok. The fix is the following one liner in AST2BOpUpdate: Old: final ASTContainer astContainer = new ASTContainer(queryRoot); /* * Inherit 'RESOLVED' flag, so resolution will not be run * for ASTContainer, constructed from parts of already resolved update. */ astContainer.setProperty(ASTContainer.Annotations.RESOLVED, context.astContainer.getProperty(ASTContainer.Annotations.RESOLVED)); New: final ASTContainer astContainer = new ASTContainer(queryRoot); // Propagate the analytic mode query hint into the WHERE clause query. // See BLZG-2030 (Analytic flag in workbench not working for UPDATE?) astContainer.setQueryHint(QueryHints.ANALYTIC, Boolean .toString(context.astContainer.getQueryHintAsBoolean(QueryHints.ANALYTIC, QueryHints.DEFAULT_ANALYTIC))); /* * Inherit 'RESOLVED' flag, so resolution will not be run * for ASTContainer, constructed from parts of already resolved update. */ astContainer.setProperty(ASTContainer.Annotations.RESOLVED, context.astContainer.getProperty(ASTContainer.Annotations.RESOLVED));
        Hide
        bryanthompson bryanthompson added a comment -

        michaelschmidt I have created a PR with this fix to get feedback from CI: https://github.com/SYSTAP/bigdata/pull/463

        I have not added a unit test to verify that the generated query plan for SPARQL UPDATE is now reflecting the analytic mode query hint. I have only verified this under the debugger.

        Show
        bryanthompson bryanthompson added a comment - michaelschmidt I have created a PR with this fix to get feedback from CI: https://github.com/SYSTAP/bigdata/pull/463 I have not added a unit test to verify that the generated query plan for SPARQL UPDATE is now reflecting the analytic mode query hint. I have only verified this under the debugger.
        Hide
        michaelschmidt michaelschmidt added a comment -

        Fix looks good. Also verified that, for the original reproducer, we now end up in HTreeDistinctBindingSetOp as expected when checking the workbench box.

        Show
        michaelschmidt michaelschmidt added a comment - Fix looks good. Also verified that, for the original reproducer, we now end up in HTreeDistinctBindingSetOp as expected when checking the workbench box.
        Hide
        bryanthompson bryanthompson added a comment -

        PR is green. Merged to master.

        Brad Bebee This is for 2.2.0

        Show
        bryanthompson bryanthompson added a comment - PR is green. Merged to master. Brad Bebee This is for 2.2.0

          People

          • Assignee:
            bryanthompson bryanthompson
            Reporter:
            michaelschmidt michaelschmidt
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: