Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-503

Prune variable bindings during query evaluation

    Details

    • Type: New Feature
    • Status: Done
    • Resolution: Done
    • Affects Version/s: TERMS_REFACTOR_BRANCH
    • Fix Version/s: None
    • Component/s: Bigdata SAIL

      Description

      Variable bindings must be pruned at the earliest possible point during query evaluation in order to reduce the memory burden and network traffic during query on a cluster.

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        Sesame allows externally given bindings to flow through the query even if they are projected by the query. This means that the "projection" can not remove all bindings except those which are projected. Either we need to also not prune exogenous bindings or we need to strip out bindings introduced by anonymous variables which are part of query rewrites. The former seems easier.

        Show
        bryanthompson bryanthompson added a comment - Sesame allows externally given bindings to flow through the query even if they are projected by the query. This means that the "projection" can not remove all bindings except those which are projected. Either we need to also not prune exogenous bindings or we need to strip out bindings introduced by anonymous variables which are part of query rewrites. The former seems easier.
        Hide
        bryanthompson bryanthompson added a comment -

        Write an AST optimizer to prune intermediate variables once they are no longer required by adding a "Distinct". This can go after a sub-group or in the middle of a pipeline, but is not required for a sub-select as we can just rewrite the projection to be DISTINCT and remove variables which do not need to be projected. Handle each of these cases.

        Show
        bryanthompson bryanthompson added a comment - Write an AST optimizer to prune intermediate variables once they are no longer required by adding a "Distinct". This can go after a sub-group or in the middle of a pipeline, but is not required for a sub-select as we can just rewrite the projection to be DISTINCT and remove variables which do not need to be projected. Handle each of these cases.
        Hide
        bryanthompson bryanthompson added a comment -

        I have modified AST2BOpUtility to always use a ProjectionOp for a query or subquery. Variables which are not projected are no longer available in solutions outside of the query/subquery.

        Show
        bryanthompson bryanthompson added a comment - I have modified AST2BOpUtility to always use a ProjectionOp for a query or subquery. Variables which are not projected are no longer available in solutions outside of the query/subquery.
        Hide
        bryanthompson bryanthompson added a comment -

        I am going to take this issue back.

        Show
        bryanthompson bryanthompson added a comment - I am going to take this issue back.
        Hide
        bryanthompson bryanthompson added a comment -

        Variable bindings are now pruned when we lift a complex optional group into a named subquery.

        The code is not smart enough to know when it can use a DISTINCT on that projection.

        The code is not yet smart enough to know when to push down some joins into a sub-group (or directly into a named subquery with a projection which prunes the intermediate variables).

        Show
        bryanthompson bryanthompson added a comment - Variable bindings are now pruned when we lift a complex optional group into a named subquery. The code is not smart enough to know when it can use a DISTINCT on that projection. The code is not yet smart enough to know when to push down some joins into a sub-group (or directly into a named subquery with a projection which prunes the intermediate variables).

          People

          • Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: