Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1392

Assess impact of the indexCache timeout.

    Details

    • Type: Task
    • Status: Done
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_RELEASE_1_5_2
    • Component/s: None
    • Labels:
      None

      Description

      The indexCache in AbstractJournal is used to retain access to recently used indices (ICheckpointProtocol is an index object) by (namespace,timestamp). Under heavy concurrent update pressure (as demonstrated by 16 or 32 client BSBM EXPLORE+UPDATE), the pace of updates causes this cache to grow quite large - as large as 1M entries after 120 runs of the cited benchmark.

          final private ConcurrentWeakValueCacheWithTimeout<NT, ICheckpointProtocol> indexCache;
      

      The cache is not infinitely leaking memory for the indices. In those 1M entries, all but 14 (two commit points) will be weakly reachable and swept by GC once the timeout for those entries has expired. That timeout currently defaults to 60 seconds.

      I believe that this timeout can be reduced to 5 seconds for such a workload, and that this low timeout might be a good default.

      The use case for retaining a larger timeout is a workload where there are many namespaces and only occasional access to those namespaces. In this context, if a namespace is accessed again within 60 seconds then it will remain in memory and stay "hot".

      The timeouts are imposed by the cleaner service in SynchronizedHardReferenceQueueWithTimeout. This service runs every 5 seconds. When it runs it scans the entries on the hard reference queue and clears the hard reference for any entry that has not been touched within the last 60 seconds.

          private static final ScheduledExecutorService cleanerService;
          static {
              
              cleanerService = Executors
                      .newSingleThreadScheduledExecutor(new DaemonThreadFactory(
                              "StaleReferenceCleaner"));
              
              cleanerService.scheduleWithFixedDelay(new Cleaner(),
                      5000/* initialDelay */, 5000/* delay */, TimeUnit.MILLISECONDS);
         
          }
      

      These behaviors can be configured using com.bigdata.journal.Options. The relevant parameter is given below. While there are (many) other related parameters, I suspect that we only need to tune this one.

          /**
           * The timeout in milliseconds for stale entries in the historical index
           * cache -or- ZERO (0) to disable the timeout (default
           * {@value #DEFAULT_HISTORICAL_INDEX_CACHE_TIMEOUT}). When this timeout
           * expires, the reference for the entry in the backing
           * {@link HardReferenceQueue} will be cleared. Note that the entry will
           * remain in the historical index cache regardless as long as it is strongly
           * reachable.
           * 
           * @see AbstractJournal#getIndexWithCheckpointAddr(long)
           */
          String HISTORICAL_INDEX_CACHE_TIMEOUT = AbstractJournal.class.getName()
                  + ".historicalIndexCacheTimeout";
      
          String DEFAULT_HISTORICAL_INDEX_CACHE_TIMEOUT = "" + (60 * 1000);
      

      This ticket is to examine the performance impact of this parameter using BSBM EXPLORE+UPDATE with 1, 16, 32, and 64 threads and recommend a setting that provides higher throughput overall.

        Activity

        bryanthompson bryanthompson created issue -
        bryanthompson bryanthompson made changes -
        Field Original Value New Value
        Assignee bryanthompson [ bryanthompson ] michaelschmidt [ michaelschmidt ]
        bryanthompson bryanthompson made changes -
        Fix Version/s BLAZEGRAPH_RELEASE_1_5_2 [ 10164 ]
        beebs Brad Bebee made changes -
        Fix Version/s BLAZEGRAPH_RELEASE_1_5_3 [ 10165 ]
        Fix Version/s BLAZEGRAPH_RELEASE_1_5_2 [ 10164 ]
        beebs Brad Bebee made changes -
        Fix Version/s BLAZEGRAPH_RELEASE_1_5_2 [ 10164 ]
        Fix Version/s BLAZEGRAPH_RELEASE_1_5_3 [ 10165 ]
        beebs Brad Bebee made changes -
        Status Open [ 1 ] Accepted [ 10101 ]
        beebs Brad Bebee made changes -
        Status Accepted [ 10101 ] In Progress [ 3 ]
        beebs Brad Bebee made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        beebs Brad Bebee made changes -
        Status Resolved [ 5 ] In Review [ 10100 ]
        beebs Brad Bebee made changes -
        Resolution Done [ 10000 ]
        Status In Review [ 10100 ] Done [ 10000 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v6 [ 18710 ] Trac Import v7 [ 19813 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v7 [ 19813 ] Trac Import v8 [ 21449 ]

          People

          • Assignee:
            michaelschmidt michaelschmidt
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: