Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-601

Error releasing deferred frees using 1.0.6 against a 1.0.4 journal

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Affects Version/s: BIGDATA_RELEASE_1_0_6
    • Fix Version/s: None
    • Component/s: RWStore

      Description

      The following stack trace can appear when attempting to commit data using 1.0.6 against a journal which had been written on by 1.0.4.

      Note: This exception is only observed when using the recycler (minReleaseAge GT 0) (versus session protection).

      Caused by: java.lang.RuntimeException: Problem with entry at -176050709462618
              at com.bigdata.rwstore.RWStore.freeDeferrals(RWStore.java:3692) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.rwstore.RWStore.checkDeferredFrees(RWStore.java:2517) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.journal.RWStrategy.checkDeferredFrees(RWStrategy.java:673) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.journal.AbstractJournal.commitNow(AbstractJournal.java:2423) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2285) ~[bigdata-1.0.6.jar:na]
      

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        In 1.0.4, the journal was not pruning the commit record index when processing deferred frees in an attempt to work around [1] and only the commit records from "lastDeferredReleaseTime" forward were processed.

        When the journal is then opened under 1.0.6, all historical commit records are considered (the start time is 0L). This causes the RWStore to consider commit records whose deferred frees have already been release, which is the root cause of the thrown exception.

        Martyn is working on a migration utility which will simply prune the commit record index, removing all entries up to the "lastDeferredReleaseTime". This utility could then be applied to a 1.0.4 journal to make it compatible with 1.0.6.

        [1] http://sourceforge.net/apps/trac/bigdata/ticket/440 (BTree can not be cast to Name2Addr)

        Show
        bryanthompson bryanthompson added a comment - In 1.0.4, the journal was not pruning the commit record index when processing deferred frees in an attempt to work around [1] and only the commit records from "lastDeferredReleaseTime" forward were processed. When the journal is then opened under 1.0.6, all historical commit records are considered (the start time is 0L). This causes the RWStore to consider commit records whose deferred frees have already been release, which is the root cause of the thrown exception. Martyn is working on a migration utility which will simply prune the commit record index, removing all entries up to the "lastDeferredReleaseTime". This utility could then be applied to a 1.0.4 journal to make it compatible with 1.0.6. [1] http://sourceforge.net/apps/trac/bigdata/ticket/440 (BTree can not be cast to Name2Addr)
        Hide
        bryanthompson bryanthompson added a comment -

        Commit of journal migration utility by Martyn. The utility prunes entries in the commit record index whose deferred deletes have already been released. This utility is necessary when upgrading from bigdata 1.0.4 to 1.0.6 if deferred deletes were enabled for the journal (minReleaseAge > 0). When minReleaseAge is ZERO (0), the database uses session protection mode and the older commit record entries will not cause a problem.

        This utility might also be necessary when moving from 1.1.0 to 1.1.1.

        Committed revision r6008.

        Show
        bryanthompson bryanthompson added a comment - Commit of journal migration utility by Martyn. The utility prunes entries in the commit record index whose deferred deletes have already been released. This utility is necessary when upgrading from bigdata 1.0.4 to 1.0.6 if deferred deletes were enabled for the journal (minReleaseAge > 0). When minReleaseAge is ZERO (0), the database uses session protection mode and the older commit record entries will not cause a problem. This utility might also be necessary when moving from 1.1.0 to 1.1.1. Committed revision r6008.
        Hide
        bryanthompson bryanthompson added a comment -

        This issue and the existence of the migration utility have also been documented on the wiki.

        See https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=DataMigration

        Show
        bryanthompson bryanthompson added a comment - This issue and the existence of the migration utility have also been documented on the wiki. See https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=DataMigration
        Hide
        bryanthompson bryanthompson added a comment -

        Modified RWStore to NOT recycle commit records starting at commitTime := 0L. Instead, it starts recycling from lastDeferredReleaseTime+1. This is correct and also handles the case where the commit record index was not pruned when deferred deletes were recycled (bigdata 1.0.4).

        Commented AbstractJournal#getCommitRecord(long) to indicate that we must continue to check against the lastReleaseTime and refuse to return a commit record from the commit record index which is older than the lastReleaseTime.

        Committed Revision r6009 (1.0.x and 1.1.x maintenance branches)

        Show
        bryanthompson bryanthompson added a comment - Modified RWStore to NOT recycle commit records starting at commitTime := 0L. Instead, it starts recycling from lastDeferredReleaseTime+1. This is correct and also handles the case where the commit record index was not pruned when deferred deletes were recycled (bigdata 1.0.4). Commented AbstractJournal#getCommitRecord(long) to indicate that we must continue to check against the lastReleaseTime and refuse to return a commit record from the commit record index which is older than the lastReleaseTime. Committed Revision r6009 (1.0.x and 1.1.x maintenance branches)
        Hide
        bryanthompson bryanthompson added a comment -

        The data migration utility is:

        java ... com.bigdata.journal.VerifyCommitRecordIndex <journal-file>
        

        The only argument is the name of the journal file. The utility will prune the commit record index up to the lastDeferredReleaseTime and then commit the journal. After running this utility, the exception associated with this ticket should no longer be thrown.

        Show
        bryanthompson bryanthompson added a comment - The data migration utility is: java ... com.bigdata.journal.VerifyCommitRecordIndex <journal-file> The only argument is the name of the journal file. The utility will prune the commit record index up to the lastDeferredReleaseTime and then commit the journal. After running this utility, the exception associated with this ticket should no longer be thrown.

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: