Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Affects Version/s: BIGDATA_RELEASE_1_3_0
    • Fix Version/s: None
    • Component/s: Journal

      Description

      It has been reported that multiple statement remove/inserts result in increasing store size even when the BTree data does not increase.

        Activity

        Hide
        martyncutcher martyncutcher added a comment -

        I ran a test program that demonstrated this behaviour. Even with zero history retention the store expansion is observed.

        With shorter runs, the store does not expand but the RWStore statistics indicate a large net increase of allocated data. I am focussing on simplifying the test program to isolate the extra RWStore allocations.

        With zero history retention it is as if the BTree is not calling free for all re-allocations.

        Show
        martyncutcher martyncutcher added a comment - I ran a test program that demonstrated this behaviour. Even with zero history retention the store expansion is observed. With shorter runs, the store does not expand but the RWStore statistics indicate a large net increase of allocated data. I am focussing on simplifying the test program to isolate the extra RWStore allocations. With zero history retention it is as if the BTree is not calling free for all re-allocations.
        Hide
        martyncutcher martyncutcher added a comment -
        allocations: 34891, frees: 630, net: 34261, commits: 4
        allocations: 36606, frees: 1728, net: 34878, commits: 6
        allocations: 38322, frees: 2826, net: 35496, commits: 8
        allocations: 40037, frees: 3925, net: 36112, commits: 10
        

        These figures indicate ~550 recycled allocations and a net increase of ~300 allocations per commit with 10K statement remove/inserts.

        The corresponding figures from dumpJournal indicate no corresponding increase index requirements.

        Show
        martyncutcher martyncutcher added a comment - allocations: 34891, frees: 630, net: 34261, commits: 4 allocations: 36606, frees: 1728, net: 34878, commits: 6 allocations: 38322, frees: 2826, net: 35496, commits: 8 allocations: 40037, frees: 3925, net: 36112, commits: 10 These figures indicate ~550 recycled allocations and a net increase of ~300 allocations per commit with 10K statement remove/inserts. The corresponding figures from dumpJournal indicate no corresponding increase index requirements.
        Hide
        martyncutcher martyncutcher added a comment -

        Adding counters from BTree it points to a lack of recycling from the BTree nodes.

        RW allocations: 34891, frees: 630, net: 34261, commits: 4, 
        BTree recycles: 622, allocations: 34804 (34182)
        
        RW allocations: 40895, frees: 4474, net: 36421, commits: 11, 
        BTree recycles: 4444, allocations: 40711 (36267)
        

        ...with a net increase of ~ 2100 allocations due to BTree updates

        Show
        martyncutcher martyncutcher added a comment - Adding counters from BTree it points to a lack of recycling from the BTree nodes. RW allocations: 34891, frees: 630, net: 34261, commits: 4, BTree recycles: 622, allocations: 34804 (34182) RW allocations: 40895, frees: 4474, net: 36421, commits: 11, BTree recycles: 4444, allocations: 40711 (36267) ...with a net increase of ~ 2100 allocations due to BTree updates
        Hide
        martyncutcher martyncutcher added a comment -

        Found it!

        The Node.removeChild code, called when merging nodes, does not recycle the old address.

        Show
        martyncutcher martyncutcher added a comment - Found it! The Node.removeChild code, called when merging nodes, does not recycle the old address.
        Hide
        martyncutcher martyncutcher added a comment -

        Committed r7887 for 1.2.0 branch and r7888 for 1.3.0

        Show
        martyncutcher martyncutcher added a comment - Committed r7887 for 1.2.0 branch and r7888 for 1.3.0

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            martyncutcher martyncutcher
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: