Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-641 Improve load performance
  3. BLZG-1556

Examine impact of dirty list threshold vs direct buffer size on write cache performance for bulk load

    XMLWordPrintable

    Details

    • Type: Sub-task
    • Status: Done
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_2_0_0
    • Component/s: RWStore
    • Labels:
      None

      Description

      We have observed some performance gains when using 10M direct buffers instead of 1M direct buffers. It is my hypothesis that this gain is entirely due to the amount of IO that can be driven by the write cache thread that actually writes on the disk and the network replication channel. My thought is that larger write cache buffers in effect allow us to schedule more write IOs for a fixed dirty list threshold. For example, if we have a dirty list threshold of 5 and 1MB buffers, then we can schedule up to 5MB of writes. However if we have the same dirty list threshold and 10M buffers, then we can schedule up to 50MB of writes.

      If this is true, then we should adjust the default dirty list threshold. This is much safer than playing around with the direct buffer pool size since HA requires the same size buffers on all nodes.

        Attachments

          Activity

            People

            Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: