Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1979

Journal appears to leak storage when namespace is deleted

    Details

      Description

      michaelschmidt has identified a pattern in which the Journal appears to leak storage when a namespace is deleted. The namespace and the indices are removed from the Journal, but many of the slots associated with the indices remain allocated.

      This is likely a side-effect caused by the refactor of the NSS to support a task-based concurrency control pattern which is compatible with group commit. AbstractJournal.dropIndex() explicitly tests for the IRWStrategy and then invokes BTree.removeAll() / HTree.removeAll() on the index. (This method is defined on ICheckpointProtocol). This is necessary in order to actually free the backing slots in the RWStore.

      The refactor to support group commit has likely caused the IIndexManager to be an AbstractTask.IsolatedActionJournal, which delegates dropIndex() to AbstractTask.dropIndex() which fails to test for whether or not the backing buffer strategy is an IRWStrategy and hence removeAll() is not being called.

      martyncutcher is looking a this now.

        Activity

        Hide
        martyncutcher martyncutcher added a comment -

        The issue was with BTree.removeAll which checked for getStore() as instance of IRWStrategy, when if getStore() returned a Journal required a (delegated) call to getBufferStrategy().

        This could be more neatly managed with addition to IRawStore interface, canRecycle() perhaps?

        I will push local BTree fix to new branch BLZG-1979 for PR

        Show
        martyncutcher martyncutcher added a comment - The issue was with BTree.removeAll which checked for getStore() as instance of IRWStrategy, when if getStore() returned a Journal required a (delegated) call to getBufferStrategy(). This could be more neatly managed with addition to IRawStore interface, canRecycle() perhaps? I will push local BTree fix to new branch BLZG-1979 for PR
        Show
        beebs Brad Bebee added a comment - https://github.com/SYSTAP/bigdata/pull/436
        Hide
        bryanthompson bryanthompson added a comment -

        I have made some edits to clean up the PR. Brad Bebee Please merge to master on clean CI. The current PR is just those changes which fix the problem for the RWStore and BTree with unisolated transactions and without group commit. The fix does not cover:

        • BigdataSail.Options.ISOLATABLE_INDICES (full read/write transactions, which requires that we introduce a destroy() method on ICheckpointRecord).
        • Group commit, which requires that we modify AbstractTask.dropIndex()
        • Simplification of the test logic by creating an IRawStore.canRecycle() method and ensuring that the method is appropriate passed through on all delegating implementations of that interface.
        Show
        bryanthompson bryanthompson added a comment - I have made some edits to clean up the PR. Brad Bebee Please merge to master on clean CI. The current PR is just those changes which fix the problem for the RWStore and BTree with unisolated transactions and without group commit. The fix does not cover: BigdataSail.Options.ISOLATABLE_INDICES (full read/write transactions, which requires that we introduce a destroy() method on ICheckpointRecord). Group commit, which requires that we modify AbstractTask.dropIndex() Simplification of the test logic by creating an IRawStore.canRecycle() method and ensuring that the method is appropriate passed through on all delegating implementations of that interface.
        Hide
        beebs Brad Bebee added a comment -
        Show
        beebs Brad Bebee added a comment - Merged down https://github.com/SYSTAP/bigdata/pull/436 .

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: