Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-719

Coalesce write cache records and install reads in cache

    Details

    • Type: New Feature
    • Status: Done
    • Resolution: Done
    • Affects Version/s: BIGDATA_RELEASE_1_2_2
    • Fix Version/s: None
    • Component/s: Journal

      Description

      We currently evict write cache buffers to the disk as soon as they are dropped onto the dirty list. This behavior limits our ability to exploit very large write cache buffer pools since the dirty list is always very short.

      The write cache service should be modified to (A) defer eviction of write cache buffers to the disk as long as possible (without starving the application of buffers to install new writes) and (B) coalesce write cache buffers before they are evicted to the disk.

      By deferring write cache eviction to the disk, the write cache buffers will be significantly more sparse as records will remain in RAM and have a much higher likelihood that they will be recycled rather than written to the disk. Further, by deferring evictions should be able to drive up the write throughput remaining 100% in RAM for data sets whose final allocation count is LTE the capacity of the write cache buffer pool in bytes.

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        Merging changes from the dev branch into the READ_CACHE branch.

        From r6766 to HEAD (r6857) of /branches/BIGDATA_RELEASE_1_2_0
        
        To /branches/READ_CACHE
        

        SVN COMMAND:

        merge https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0 /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD
        
            Merge complete.
            ===== File Statistics: =====
            Deleted: 8
            Added: 25
            Updated: 139
            ==== Property Statistics: =====
            Updated: 50
            ==== Conflict Statistics: =====
            File conflicts: 3
        
            C   /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD/bigdata/src/test/com/bigdata/rwstore/sector/TestMemStore.java
            C   /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java
            C   /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java
        

        Accepted the incoming version for all three conflicts. These files had been modified in the same manner in the dev branch and the READ_CACHE branch.

        Committed revision r6858.

        Show
        bryanthompson bryanthompson added a comment - Merging changes from the dev branch into the READ_CACHE branch. From r6766 to HEAD (r6857) of /branches/BIGDATA_RELEASE_1_2_0 To /branches/READ_CACHE SVN COMMAND: merge https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0 /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD Merge complete. ===== File Statistics: ===== Deleted: 8 Added: 25 Updated: 139 ==== Property Statistics: ===== Updated: 50 ==== Conflict Statistics: ===== File conflicts: 3 C /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD/bigdata/src/test/com/bigdata/rwstore/sector/TestMemStore.java C /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD/bigdata/src/test/com/bigdata/journal/TestJournalShutdown.java C /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD/bigdata/src/java/com/bigdata/journal/WriteExecutorService.java Accepted the incoming version for all three conflicts. These files had been modified in the same manner in the dev branch and the READ_CACHE branch. Committed revision r6858.
        Hide
        bryanthompson bryanthompson added a comment -

        Tagging the READ_CACHE branch after the merge from the dev branch.

        https://bigdata.svn.sourceforge.net/svnroot/bigdata/tags/READ_CACHE_V2
        
        Show
        bryanthompson bryanthompson added a comment - Tagging the READ_CACHE branch after the merge from the dev branch. https://bigdata.svn.sourceforge.net/svnroot/bigdata/tags/READ_CACHE_V2
        Hide
        bryanthompson bryanthompson added a comment -

        This ticket is closed out. Further work on HA is continuing under [1].

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/621 (Journal HA)

        Show
        bryanthompson bryanthompson added a comment - This ticket is closed out. Further work on HA is continuing under [1] . [1] https://sourceforge.net/apps/trac/bigdata/ticket/621 (Journal HA)
        Hide
        bryanthompson bryanthompson added a comment -

        Merging from dev branch to READ_CACHE branch to capture changes in the development branch. See [1] for the previous merge from the development branch to the READ_CACHE branch (which captured the delta from r6766 to r6857 of /branches/BIGDATA_RELEASE_1_2_0).

            At revision 6889. (Before doing the merge)
        
        merge https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0 /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD
            --- Merging r6857 through r6889 into /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD
        
        Merge complete.
            ===== File Statistics: =====
            Deleted: 4
            Added: 5
            Updated: 43
        
        No conflicts.
        

        Committed revision r6890.

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/621 (Coalesce write cache records and install reads in cache)

        Show
        bryanthompson bryanthompson added a comment - Merging from dev branch to READ_CACHE branch to capture changes in the development branch. See [1] for the previous merge from the development branch to the READ_CACHE branch (which captured the delta from r6766 to r6857 of /branches/BIGDATA_RELEASE_1_2_0). At revision 6889. (Before doing the merge) merge https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_2_0 /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD --- Merging r6857 through r6889 into /Users/bryan/Documents/workspace/BIGDATA_READ_CACHE_HEAD Merge complete. ===== File Statistics: ===== Deleted: 4 Added: 5 Updated: 43 No conflicts. Committed revision r6890. [1] https://sourceforge.net/apps/trac/bigdata/ticket/621 (Coalesce write cache records and install reads in cache)
        Hide
        bryanthompson bryanthompson added a comment -

        See [1] for more work on this branch.

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/530 (Journal HA)

        Show
        bryanthompson bryanthompson added a comment - See [1] for more work on this branch. [1] https://sourceforge.net/apps/trac/bigdata/ticket/530 (Journal HA)

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: