Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-744

Compress write cache blocks for replication and in HALogs

    Details

      Description

      It would be advantageous to compress the write cache blocks before they are replicated and before they are written to the HALog files. The leader would do the compression in the WriteCacheService.WriteTask.call() thread, so this would not add latency to writes. The receiver would immediately replicate the buffer. A receiver that only needs to log the buffer would log the compressed buffer. If the receiver needs to write the buffer onto the backing store, then it would decompress the buffer into a 2nd buffer
      - that 2nd buffer could be allocated and held by the HAJournalServer in its life cycle.

      We need to add fields for the compressionScheme (None, Zip, etc) and the wireByteLength to the IHAWriteMessage interface. For non-compressed data, the wireByteLength and the data byte length would be the same. The compressionScheme should default to None and the wireByteLength should default to the actual byte length for backwards compatibility.

      The HAWriteMessages are written onto the HALog. The messages themselves would not be compressed. However, the replicated write cache blocks WOULD be compressed. This should present a substantial space savings for the HALog files.

        Activity

        beebs Brad Bebee created issue -
        beebs Brad Bebee made changes -
        Field Original Value New Value
        Workflow Trac Import v2 [ 12580 ] Trac Import v3 [ 14129 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v3 [ 14129 ] Trac Import v4 [ 15458 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v4 [ 15458 ] Trac Import v5 [ 16844 ]
        beebs Brad Bebee made changes -
        Labels Issue_patch_20150625
        beebs Brad Bebee made changes -
        Status Closed - Won't Fix [ 6 ] Open [ 1 ]
        beebs Brad Bebee made changes -
        Status Open [ 1 ] Accepted [ 10101 ]
        beebs Brad Bebee made changes -
        Status Accepted [ 10101 ] In Progress [ 3 ]
        beebs Brad Bebee made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        beebs Brad Bebee made changes -
        Status Resolved [ 5 ] In Review [ 10100 ]
        beebs Brad Bebee made changes -
        Resolution Fixed [ 1 ] Done [ 10000 ]
        Status In Review [ 10100 ] Done [ 10000 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v5 [ 16844 ] Trac Import v6 [ 18058 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v6 [ 18058 ] Trac Import v7 [ 19455 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v7 [ 19455 ] Trac Import v8 [ 21076 ]

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: