Details

    • Type: Sub-task
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_RELEASE_1_5_2
    • Component/s: None
    • Labels:
      None

      Description

      This is a feature request to support a limit on the amount of native memory that can be used by the analytic query mode.

      1. Modify MemoryManager to accept the "blocks" boolean as a constructor argument. When blocks:=false, the MemoryManager will throw an out of memory exception. This will break queries that attempt to consume to much native memory.
      2. Add property (with a System property to override the default) that specifies the maximum amount of native memory that may be consumed by a query. When 0L there will be no limit.
      3. When the maximum amount of native memory is NOT 0L, then specify blocks:=false to AbstractRunningQuery.getMemoryManager(). This will cause queries that consume too much native memory to break with a MemoryManagerOutOfMemory exception.
      • See AbstractRunningQuery.getMemoryManager()
      • See PipelineOp.MAX_MEMORY (this is not quite the same concept since it specified the per-input-queue memory limit)

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        I have defined a new field in QueryHints to specify the maximum amount of native memory per query. When non-zero, this value is rounded up to the nearest multiple of a direct buffer capacity for the DirectBufferPool default instance (that is, multiples of 1MB). This implies that the minimum limit for native memory allocation by a query is 1M. The default remains ZERO (0) which implies no limit.

            /**
             * The maximum amount of native heap memory that may be allocated for a
             * single query when using the analytic query mode -or- ZERO (0L) if no
             * limit should be imposed. When non-zero, queries that exceed this limit
             * will be broken with a memory allocation exception. Together with a limit
             * on the number of concurrent queries, this may be used to limit the amount
             * of native memory consumed by the query engine.
             * <p>
             * The default is ZERO (0) which implies no limit. The default may be
             * overridden using the environment variable named
             * 
             * <pre>
             * com.bigdata.rdf.sparql.ast.QueryHints.analyticMaxMemoryPerQuery
             * </pre>
             * <p>
             * Native memory allocations are made using a {@link DirectBufferPool}. The
             * per-query limit will be rounded up to a multiple of buffers based on the
             * configured buffer capacity. The default is a 1MB buffer, so the
             * granularity of the limit is multiples of 1MB.
             * 
             * @see DirectBufferPool
             * @see <a href="http://jira.blazegraph.com/browse/BLZG-42" > Per query
             *      memory limit for analytic query mode. </a>
             */
            String ANALYTIC_MAX_MEMORY_PER_QUERY = "analyticMaxMemoryPerQuery";
            
            long DEFAULT_ANALYTIC_MAX_MEMORY_PER_QUERY = Long.valueOf(System
                    .getProperty(QueryHints.class.getName() + "."
                            + ANALYTIC_MAX_MEMORY_PER_QUERY, "0"));
        

        A new method on AbstractRunningQuery was created to provision the MemoryManager:

            /**
             * Allocate a memory manager for the query.
             * 
             * @see QueryHints#ANALYTIC_MAX_MEMORY_PER_QUERY
             * 
             * @see QueryHints#DEFAULT_ANALYTIC_MAX_MEMORY_PER_QUERY
             * 
             * @see <a href="http://jira.blazegraph.com/browse/BLZG-42" > Per query
             *      memory limit for analytic query mode. </a>
             */
            private IMemoryManager newMemoryManager() {
                
                // The native memory pool that will be used by this query.
                final DirectBufferPool pool = DirectBufferPool.INSTANCE;
                
                // Figure out how much memory may be allocated by this query.
                long maxMemoryBytesPerQuery = QueryHints.DEFAULT_ANALYTIC_MAX_MEMORY_PER_QUERY;
                if (maxMemoryBytesPerQuery < 0) {
                    // Ignore illegal values.
                    maxMemoryBytesPerQuery = 0L;
                }
        
                final boolean blocking;
                final int nsectors;
                if (maxMemoryBytesPerQuery == 0) {
                    /*
                     * Allocation are blocking IFF there is no bound on the memory for
                     * the query.
                     */
                    blocking = true; // block until allocation is satisfied.
                    nsectors = 0; // no limit
                } else {
                    /*
                     * Allocations do not block if we run out of native memory for this
                     * query. Instead a memory allocation exception will be thrown and
                     * the query will break.
                     * 
                     * The #of sectors is computed by dividing through by the size of
                     * the backing native ByteBuffers and then rounding up.
                     */
                    blocking = false; // throw exception if query uses too much RAM.
        
                    // The capacity of the buffers in this pool.
                    final int bufferCapacity = pool.getBufferCapacity();
        
                    // Figure out the maximum #of buffers (rounding up).
                    nsectors = (int) Math.ceil(maxMemoryBytesPerQuery
                            / (double) bufferCapacity);
        
                }
        
                return new MemoryManager(pool, nsectors, blocking, null/* properties */);
        
            }
        

        Commit c5d15d7d71f9b7cd8ac626b93775811bc9ea2f2d to SF GIT branch BLZG-42

        Show
        bryanthompson bryanthompson added a comment - I have defined a new field in QueryHints to specify the maximum amount of native memory per query. When non-zero, this value is rounded up to the nearest multiple of a direct buffer capacity for the DirectBufferPool default instance (that is, multiples of 1MB). This implies that the minimum limit for native memory allocation by a query is 1M. The default remains ZERO (0) which implies no limit. /** * The maximum amount of native heap memory that may be allocated for a * single query when using the analytic query mode -or- ZERO (0L) if no * limit should be imposed. When non-zero, queries that exceed this limit * will be broken with a memory allocation exception. Together with a limit * on the number of concurrent queries, this may be used to limit the amount * of native memory consumed by the query engine. * <p> * The default is ZERO (0) which implies no limit. The default may be * overridden using the environment variable named * * <pre> * com.bigdata.rdf.sparql.ast.QueryHints.analyticMaxMemoryPerQuery * </pre> * <p> * Native memory allocations are made using a {@link DirectBufferPool}. The * per-query limit will be rounded up to a multiple of buffers based on the * configured buffer capacity. The default is a 1MB buffer, so the * granularity of the limit is multiples of 1MB. * * @see DirectBufferPool * @see <a href= "http: //jira.blazegraph.com/browse/BLZG-42" > Per query * memory limit for analytic query mode. </a> */ String ANALYTIC_MAX_MEMORY_PER_QUERY = "analyticMaxMemoryPerQuery" ; long DEFAULT_ANALYTIC_MAX_MEMORY_PER_QUERY = Long .valueOf( System .getProperty(QueryHints.class.getName() + "." + ANALYTIC_MAX_MEMORY_PER_QUERY, "0" )); A new method on AbstractRunningQuery was created to provision the MemoryManager: /** * Allocate a memory manager for the query. * * @see QueryHints#ANALYTIC_MAX_MEMORY_PER_QUERY * * @see QueryHints#DEFAULT_ANALYTIC_MAX_MEMORY_PER_QUERY * * @see <a href= "http: //jira.blazegraph.com/browse/BLZG-42" > Per query * memory limit for analytic query mode. </a> */ private IMemoryManager newMemoryManager() { // The native memory pool that will be used by this query. final DirectBufferPool pool = DirectBufferPool.INSTANCE; // Figure out how much memory may be allocated by this query. long maxMemoryBytesPerQuery = QueryHints.DEFAULT_ANALYTIC_MAX_MEMORY_PER_QUERY; if (maxMemoryBytesPerQuery < 0) { // Ignore illegal values. maxMemoryBytesPerQuery = 0L; } final boolean blocking; final int nsectors; if (maxMemoryBytesPerQuery == 0) { /* * Allocation are blocking IFF there is no bound on the memory for * the query. */ blocking = true ; // block until allocation is satisfied. nsectors = 0; // no limit } else { /* * Allocations do not block if we run out of native memory for this * query. Instead a memory allocation exception will be thrown and * the query will break . * * The #of sectors is computed by dividing through by the size of * the backing native ByteBuffers and then rounding up. */ blocking = false ; // throw exception if query uses too much RAM. // The capacity of the buffers in this pool. final int bufferCapacity = pool.getBufferCapacity(); // Figure out the maximum #of buffers (rounding up). nsectors = ( int ) Math .ceil(maxMemoryBytesPerQuery / ( double ) bufferCapacity); } return new MemoryManager(pool, nsectors, blocking, null /* properties */); } Commit c5d15d7d71f9b7cd8ac626b93775811bc9ea2f2d to SF GIT branch BLZG-42
        Hide
        bryanthompson bryanthompson added a comment -
        • AbstractRunningQuery.newMemoryManager() now uses Integer.MAX_VALUE rather than ZERO (0) when there is no allocation limit for native memory for a query.
        • MemoryManager now sets the blocks argument on a private field. This field is used when making allocations as the default policy (either blocking or non-blocking).
        • Marked overrides in MemoryManager.
        • Added counter to report blocking/non-blocking policy to MemoryManager.
        • Added isBlocking() to report blocking/non-blocking policy to MemoryManager.

        Commit e9aebee9b2b29aca545f9656a6d59c3defb97b91 to SF GIT branch BLZG-42

        Show
        bryanthompson bryanthompson added a comment - AbstractRunningQuery.newMemoryManager() now uses Integer.MAX_VALUE rather than ZERO (0) when there is no allocation limit for native memory for a query. MemoryManager now sets the blocks argument on a private field. This field is used when making allocations as the default policy (either blocking or non-blocking). Marked overrides in MemoryManager. Added counter to report blocking/non-blocking policy to MemoryManager. Added isBlocking() to report blocking/non-blocking policy to MemoryManager. Commit e9aebee9b2b29aca545f9656a6d59c3defb97b91 to SF GIT branch BLZG-42
        Hide
        bryanthompson bryanthompson added a comment -

        I have migrated the changes for this ticket into github in the BLZG-42-GITHUB branch.

        Commit 600a2c6354e776d8ddd9d7b1a95041f28bc08aa4 to github branch BLZG-42-GITHUB.

        Show
        bryanthompson bryanthompson added a comment - I have migrated the changes for this ticket into github in the BLZG-42 -GITHUB branch. Commit 600a2c6354e776d8ddd9d7b1a95041f28bc08aa4 to github branch BLZG-42 -GITHUB.
        Hide
        bryanthompson bryanthompson added a comment -

        Issued and resolved pull request to bring these features into the github master.

        https://github.com/SYSTAP/bigdata/pull/68

        Show
        bryanthompson bryanthompson added a comment - Issued and resolved pull request to bring these features into the github master. https://github.com/SYSTAP/bigdata/pull/68

          People

          • Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: