Details

    • Type: Task
    • Status: Done
    • Resolution: Done
    • Affects Version/s: BIGDATA_RELEASE_1_2_0
    • Fix Version/s: None
    • Component/s: Journal

      Description

      Add a BufferMode which supports the MemoryManager. This will allow the use of the Journal against main memory for data sets of up to 4TB. The immediate use case is to support a cached solution sets with MVCC semantics for SPARQL Update.

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        This commit incorporates the memory manager journal mode (MemStrategy, which is enabled by BuffedMode.MemStore) [1].

        It also incorporates some progress on the bigdata extension for SPARQL Update [2]. I am beginning to work through the integration to support this feature. I have written code for writing solution sets onto the memory manager and reading them back. That code has not yet been tested. This functionality is currently disabled in QueryHints and in the test suite.

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/541 (MemoryManager Journal mode)
        [2] https://sourceforge.net/apps/trac/bigdata/ticket/531 (SPARQL UPDATE for Solution Sets).

        Committed revision r6266.

        Show
        bryanthompson bryanthompson added a comment - This commit incorporates the memory manager journal mode (MemStrategy, which is enabled by BuffedMode.MemStore) [1] . It also incorporates some progress on the bigdata extension for SPARQL Update [2] . I am beginning to work through the integration to support this feature. I have written code for writing solution sets onto the memory manager and reading them back. That code has not yet been tested. This functionality is currently disabled in QueryHints and in the test suite. [1] https://sourceforge.net/apps/trac/bigdata/ticket/541 (MemoryManager Journal mode) [2] https://sourceforge.net/apps/trac/bigdata/ticket/531 (SPARQL UPDATE for Solution Sets). Committed revision r6266.
        Hide
        bryanthompson bryanthompson added a comment -

        Allocation contexts do NOT provide session protection by themselves.


        - The MemStrategy and the RWStrategy both require the same hooks in terms of provide session protection when read-write or read-only transaction exist. We need to search on RWStrategy and generalize IBufferStrategy and/or BufferMode so we have better hooks for this.


        - PSOutputStream and PSInputStream should be raised at a minimum to the RW/Mem strategy implementations, and possibly to IRawStore.


        - The deferred free recycler logic needs to be used by both the RWStore and the MemStore.


        - MemStrategy#abort() is currently a NOP. It should revert to the most recent commit point and clear out the transient bits.


        - The MemStrategy needs more unit test coverage for abort() and recycling.

        The DirectFixedAllocator support was removed in r6034.

        Show
        bryanthompson bryanthompson added a comment - Allocation contexts do NOT provide session protection by themselves. - The MemStrategy and the RWStrategy both require the same hooks in terms of provide session protection when read-write or read-only transaction exist. We need to search on RWStrategy and generalize IBufferStrategy and/or BufferMode so we have better hooks for this. - PSOutputStream and PSInputStream should be raised at a minimum to the RW/Mem strategy implementations, and possibly to IRawStore. - The deferred free recycler logic needs to be used by both the RWStore and the MemStore. - MemStrategy#abort() is currently a NOP. It should revert to the most recent commit point and clear out the transient bits. - The MemStrategy needs more unit test coverage for abort() and recycling. The DirectFixedAllocator support was removed in r6034.
        Hide
        bryanthompson bryanthompson added a comment -

        It turns out that the Journal was creating (but not using) a backing file when configured for the MemStore buffer mode. The initialization of the Journal has been reorganized slightly to address this. This also fixes the problem where the MemStore backed Journal was requiring either the FILE or the CREATE_TEMP_FILE property. I also cleaned up some HTree unit tests which were explicitly specifying Integer.MAX_VALUE for the maximum #of buffers. They now use the default (which amounts to the same thing but is cleaner).

        Committed revision r6284.

        Show
        bryanthompson bryanthompson added a comment - It turns out that the Journal was creating (but not using) a backing file when configured for the MemStore buffer mode. The initialization of the Journal has been reorganized slightly to address this. This also fixes the problem where the MemStore backed Journal was requiring either the FILE or the CREATE_TEMP_FILE property. I also cleaned up some HTree unit tests which were explicitly specifying Integer.MAX_VALUE for the maximum #of buffers. They now use the default (which amounts to the same thing but is cleaner). Committed revision r6284.
        Hide
        bryanthompson bryanthompson added a comment -

        See [1] for an issue which is showing up in CI.

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/549 (BTree can not be case to Name2Addr for MemStore).

        Show
        bryanthompson bryanthompson added a comment - See [1] for an issue which is showing up in CI. [1] https://sourceforge.net/apps/trac/bigdata/ticket/549 (BTree can not be case to Name2Addr for MemStore).
        Hide
        bryanthompson bryanthompson added a comment -

        Martyn wrote:

        Update to MemoryManager providing improved compatibility for MemStrategy use with Journals in addition to raw MemoryManager access.

        There were two issues
        - one with an unprotected (no lock taken) commit
        and the second when a new SectorAllocator was created while session
        protection was active (but not flagged in the allocator), it was happily
        recycling allocations that should have been protected

        Lifted out IPSOutputStream interface.

        Committed Revision r6318.

        Show
        bryanthompson bryanthompson added a comment - Martyn wrote: Update to MemoryManager providing improved compatibility for MemStrategy use with Journals in addition to raw MemoryManager access. There were two issues - one with an unprotected (no lock taken) commit and the second when a new SectorAllocator was created while session protection was active (but not flagged in the allocator), it was happily recycling allocations that should have been protected Lifted out IPSOutputStream interface. Committed Revision r6318.
        Hide
        bryanthompson bryanthompson added a comment -

        The MemoryManager backed Journal can be enabled using

        com.bigdata.journal.Options.BUFFER_MODE=MemStore
        
        Show
        bryanthompson bryanthompson added a comment - The MemoryManager backed Journal can be enabled using com.bigdata.journal.Options.BUFFER_MODE=MemStore

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: