Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-593

Parallelize aggregation operators on cluster

    XMLWordPrintable

    Details

      Description

      Many aggregation queries can be decomposed and parallelized. For example:

      SELECT (COUNT(*) as ?cnt) {?s ?p ?o}
      

      If this query is evaluated naively, then the solutions from the where clause will be materialized on the query controller, which will then count those solutions.

      However, the query can also be decomposed into a count of the solutions from the last join in the WHERE clause on whatever node those solutions are produced. For the query above, the solutions would be produced on each node having a shard for the SPO index. This is a degenerate case in which there is a perfect decomposition and no solutions would need to flow through the network for the WHERE clause. However, the same principle applies when the WHERE clause is complex. The solutions are counted on the each node on which the last join step is performed. Those counts are then combined on the query controller for a final answer.

      Bigdata can already recognize aggregations which can be decomposed along these lines. However, the query plan generator needs to be modified in order to actually decompose the aggregation.

      See https://sourceforge.net/apps/trac/bigdata/ticket/131 (unselective joins on cluster)
      See https://sourceforge.net/apps/trac/bigdata/ticket/472 (acceptTaskService pool size on cluster)

        Attachments

          Activity

            People

            Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: