Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-481

Stochastic assert in AbstractBTree#writeNodeOrLeaf() in CI

    Details

    • Type: Bug
    • Status: Closed - Won't Fix
    • Resolution: Incomplete
    • Affects Version/s: BIGDATA_RELEASE_1_3_1
    • Fix Version/s: None
    • Component/s: B+Tree
    • Labels:
      None

      Description

      There is a stochastic failure in AbstractBTree#writeNodeOfLeaf() in CI runs which is sometimes observed for TestMROWTransactions and also in StressTestUnisolatedReadWriteIndex (7/16/2014).

      The relevant line in AbstractBTree is:

          protected long writeNodeOrLeaf(final AbstractNode<?> node) {
      
              assert root != null; // i.e., isOpen().
              assert node != null;
              assert node.btree == this;
              assert node.dirty;
              assert !node.deleted;
              assert !node.isPersistent();
              assert !node.isReadOnly(); // FIXME Occasional CI errors on this assert for TestMROWTransactions
              assertNotReadOnly();
      

      This error is suggestive of a concurrent update during the commit protocol, which is something which we addressed in some depth a few months ago and which was corrected, in part, by protecting the commit using the same lock as the UnisolatedReadWriteIndex.

      The failure was observed in the QUADS_QUERY_BRANCH, but it is presumed to be present in the TERMS_REFACTOR_BRANCH as well. There were 20 clean builds between the exceptions listed below. These are the only builds in which this assert has been tripped since the fix to the "Record already in cache" (CI build #1431).

      This exception is from CI build #1451.

      junit.framework.AssertionFailedError: Test failed: 
      firstCause=java.lang.RuntimeException: Could not commit
      index: name=kb.spo.POS,
      commitList=[DirtyListener{name=kb.spo.OSP,checkpointAddr=0},
      DirtyListener{name=kb.spo.POS,needsCheckpoint},
      DirtyListener{name=kb.spo.SPO,needsCheckpoint}]: 
      lastRootBlock=rootBlock{ rootBlock=1, challisField=37, version=2, 
      nextOffset=4346506912097, localTime=1309190439952 [Monday, June 27, 
      2011 12:00:39 PM EDT],
      firstCommitTime=1309190381802 [Monday, June 27, 2011 11:59:41 AM EDT], 
      lastCommitTime=1309190439934 [Monday, June 27, 2011
      12:00:39 PM EDT], commitCounter=37,
      commitRecordAddr={off=NATIVE:-60227,len=422},
      commitRecordIndexAddr={off=NATIVE:-8544,len=220},
      quorumToken=0, metaBitsAddr=63174606902, metaStartAddr=1376, 
      storeType=RW, uuid=6f139bc9-46d8-4f33-b199-c105e7dadcce,
      offsetBits=42, checksum=1623663730, createTime=1309190381786 [Monday, 
      June 27, 2011 11:59:41 AM EDT], closeTime=0}
      	at
      com.bigdata.rdf.sail.TestMROWTransactions.test_multiple_csem_t
      ransaction(TestMROWTransactions.java:382)
      Caused by: java.lang.RuntimeException: Could not commit
      index: name=kb.spo.POS,
      commitList=[DirtyListener{name=kb.spo.OSP,checkpointAddr=0},
      DirtyListener{name=kb.spo.POS,needsCheckpoint},
      DirtyListener{name=kb.spo.SPO,needsCheckpoint}]: 
      lastRootBlock=rootBlock{ rootBlock=1, challisField=37, version=2, 
      nextOffset=4346506912097, localTime=1309190439952 [Monday, June 27, 
      2011 12:00:39 PM EDT],
      firstCommitTime=1309190381802 [Monday, June 27, 2011 11:59:41 AM EDT], 
      lastCommitTime=1309190439934 [Monday, June 27, 2011
      12:00:39 PM EDT], commitCounter=37,
      commitRecordAddr={off=NATIVE:-60227,len=422},
      commitRecordIndexAddr={off=NATIVE:-8544,len=220},
      quorumToken=0, metaBitsAddr=63174606902, metaStartAddr=1376, 
      storeType=RW, uuid=6f139bc9-46d8-4f33-b199-c105e7dadcce,
      offsetBits=42, checksum=1623663730, createTime=1309190381786 [Monday, 
      June 27, 2011 11:59:41 AM EDT], closeTime=0}
      	at
      com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2252)
      	at
      com.bigdata.rdf.store.LocalTripleStore.commit(LocalTripleStore
      .java:74)
      	at
      com.bigdata.rdf.sail.TestMROWTransactions$1Writer.call(TestMRO
      WTransactions.java:274)
      	at
      com.bigdata.rdf.sail.TestMROWTransactions$1Writer.call(TestMRO
      WTransactions.java:247)
      	at
      java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadP
      oolExecutor.java:886)
      	at
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE
      xecutor.java:908)
      	at java.lang.Thread.run(Thread.java:680)
      Caused by: java.lang.RuntimeException: Could not commit
      index: name=kb.spo.POS,
      commitList=[DirtyListener{name=kb.spo.OSP,checkpointAddr=0},
      DirtyListener{name=kb.spo.POS,needsCheckpoint},
      DirtyListener{name=kb.spo.SPO,needsCheckpoint}]
      	at
      com.bigdata.journal.Name2Addr.handleCommit(Name2Addr.java:571)
      	at
      com.bigdata.journal.AbstractJournal.notifyCommitters(AbstractJ
      ournal.java:1983)
      	at
      com.bigdata.journal.AbstractJournal.commitNow(AbstractJournal.
      java:2336)
      	at
      com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2250)
      Caused by: java.lang.AssertionError
      	at
      com.bigdata.btree.AbstractBTree.writeNodeOrLeaf(AbstractBTree.
      java:3561)
      	at
      com.bigdata.btree.AbstractBTree.writeNodeRecursive(AbstractBTr
      ee.java:3485)
      	at com.bigdata.btree.BTree.flush(BTree.java:770)
      	at com.bigdata.btree.BTree.writeCheckpoint2(BTree.java:910)
      	at com.bigdata.btree.BTree.writeCheckpoint(BTree.java:881)
      	at com.bigdata.btree.BTree.handleCommit(BTree.java:1210)
      	at
      com.bigdata.journal.Name2Addr.handleCommit(Name2Addr.java:563)
      

      This exception is from CI build #1471.

      junit.framework.AssertionFailedError: Test failed: firstCause=java.lang.RuntimeException: Could not commit index: name=kb.spo.POS, commitList=[DirtyListener{name=kb.spo.OSP,checkpointAddr=0}, DirtyListener{name=kb.spo.POS,needsCheckpoint}, DirtyListener{name=kb.spo.SPO,needsCheckpoint}]: lastRootBlock=rootBlock{ rootBlock=0, challisField=36, version=2, nextOffset=4273492468087, localTime=1309056345571 [Saturday, June 25, 2011 10:45:45 PM EDT], firstCommitTime=1309056286594 [Saturday, June 25, 2011 10:44:46 PM EDT], lastCommitTime=1309056345554 [Saturday, June 25, 2011 10:45:45 PM EDT], commitCounter=36, commitRecordAddr={off=NATIVE:-61228,len=422}, commitRecordIndexAddr={off=NATIVE:-8566,len=220}, quorumToken=0, metaBitsAddr=63635980342, metaStartAddr=1376, storeType=RW, uuid=80833dc5-f461-4f66-8ffd-54298387ded9, offsetBits=42, checksum=-1789582556, createTime=1309056286577 [Saturday, June 25, 2011 10:44:46 PM EDT], closeTime=0}
      	at com.bigdata.rdf.sail.TestMROWTransactions.test_multiple_csem_transaction(TestMROWTransactions.java:382)
      Caused by: java.lang.RuntimeException: Could not commit index: name=kb.spo.POS, commitList=[DirtyListener{name=kb.spo.OSP,checkpointAddr=0}, DirtyListener{name=kb.spo.POS,needsCheckpoint}, DirtyListener{name=kb.spo.SPO,needsCheckpoint}]: lastRootBlock=rootBlock{ rootBlock=0, challisField=36, version=2, nextOffset=4273492468087, localTime=1309056345571 [Saturday, June 25, 2011 10:45:45 PM EDT], firstCommitTime=1309056286594 [Saturday, June 25, 2011 10:44:46 PM EDT], lastCommitTime=1309056345554 [Saturday, June 25, 2011 10:45:45 PM EDT], commitCounter=36, commitRecordAddr={off=NATIVE:-61228,len=422}, commitRecordIndexAddr={off=NATIVE:-8566,len=220}, quorumToken=0, metaBitsAddr=63635980342, metaStartAddr=1376, storeType=RW, uuid=80833dc5-f461-4f66-8ffd-54298387ded9, offsetBits=42, checksum=-1789582556, createTime=1309056286577 [Saturday, June 25, 2011 10:44:46 PM EDT], closeTime=0}
      	at com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2252)
      	at com.bigdata.rdf.store.LocalTripleStore.commit(LocalTripleStore.java:74)
      	at com.bigdata.rdf.sail.TestMROWTransactions$1Writer.call(TestMROWTransactions.java:274)
      	at com.bigdata.rdf.sail.TestMROWTransactions$1Writer.call(TestMROWTransactions.java:247)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:680)
      Caused by: java.lang.RuntimeException: Could not commit index: name=kb.spo.POS, commitList=[DirtyListener{name=kb.spo.OSP,checkpointAddr=0}, DirtyListener{name=kb.spo.POS,needsCheckpoint}, DirtyListener{name=kb.spo.SPO,needsCheckpoint}]
      	at com.bigdata.journal.Name2Addr.handleCommit(Name2Addr.java:571)
      	at com.bigdata.journal.AbstractJournal.notifyCommitters(AbstractJournal.java:1983)
      	at com.bigdata.journal.AbstractJournal.commitNow(AbstractJournal.java:2336)
      	at com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2250)
      Caused by: java.lang.AssertionError
      	at com.bigdata.btree.AbstractBTree.writeNodeOrLeaf(AbstractBTree.java:3561)
      	at com.bigdata.btree.AbstractBTree.writeNodeRecursive(AbstractBTree.java:3485)
      	at com.bigdata.btree.BTree.flush(BTree.java:770)
      	at com.bigdata.btree.BTree.writeCheckpoint2(BTree.java:910)
      	at com.bigdata.btree.BTree.writeCheckpoint(BTree.java:881)
      	at com.bigdata.btree.BTree.handleCommit(BTree.java:1210)
      	at com.bigdata.journal.Name2Addr.handleCommit(Name2Addr.java:563) 
      

      See BLZG-933 (AssertionError: Child does not have persistent identity) which is pretty much a duplicate of this.

        Issue Links

          Activity

          Hide
          bryanthompson bryanthompson added a comment -

          Modified the test to explicitly set a flag if there are no errors by the time we interrupt the running tasks. This allows the test to report failure only for errors encountered during normal execution of tasks and filters out any failures which might arise from the interruption of a large number of concurrent tasks.

          Committed revision r4889. Changes were applied to the 1.0.0 release branch and to the development branch.

          Show
          bryanthompson bryanthompson added a comment - Modified the test to explicitly set a flag if there are no errors by the time we interrupt the running tasks. This allows the test to report failure only for errors encountered during normal execution of tasks and filters out any failures which might arise from the interruption of a large number of concurrent tasks. Committed revision r4889. Changes were applied to the 1.0.0 release branch and to the development branch.
          Hide
          bryanthompson bryanthompson added a comment -

          This is related to [1]

          [1] https://sourceforge.net/apps/trac/bigdata/ticket/440#comment:32 (BTree can not be cast to Name2Addr when accessing historical state on RWStore)

          Show
          bryanthompson bryanthompson added a comment - This is related to [1] [1] https://sourceforge.net/apps/trac/bigdata/ticket/440#comment:32 (BTree can not be cast to Name2Addr when accessing historical state on RWStore)
          Hide
          bryanthompson bryanthompson added a comment -

          An exception was observed 8 hours into a stress test run against the RWSTORE_DEBUG_1_1_0 branch r5931.

          The exception was:

          java.lang.RuntimeException: Could not commit index: name=kb.spo.POCS, commitList=[DirtyListener{name=kb.spo.CSPO,checkpointAddr=0}, DirtyListener{name=kb.spo.OCSP,checkpointAddr=0}, DirtyListener{name=kb.spo.PCSO,checkpointAddr=0}, DirtyListener{name=kb.spo.POCS,needsCheckpoint}, DirtyListener{name=kb.spo.SOPC,needsCheckpoint}, DirtyListener{name=kb.spo.SPOC,needsCheckpoint}]: lastRootBlock=rootBlock{ rootBlock=0, challisField=16, version=2, nextOffset=661424971997, localTime=1328061936805 [Tuesday, January 31, 2012 9:05:36 PM EST], firstCommitTime=1328061934263 [Tuesday, January 31, 2012 9:05:34 PM EST], lastCommitTime=1328061936793 [Tuesday, January 31, 2012 9:05:36 PM EST], commitCounter=16, commitRecordAddr={off=NATIVE:-42021,len=422}, commitRecordIndexAddr={off=NATIVE:-8412,len=220}, quorumToken=0, metaBitsAddr=60406366258, metaStartAddr=160, storeType=RW, uuid=fc891ff9-c773-4698-9362-08ef700202b0, offsetBits=42, checksum=202121150, createTime=1328061934235 [Tuesday, January 31, 2012 9:05:34 PM EST], closeTime=0}
          	at com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2278)
          	at com.bigdata.rdf.store.LocalTripleStore.commit(LocalTripleStore.java:78)
          	at com.bigdata.rdf.sail.TestMROWTransactions$1Writer.call(TestMROWTransactions.java:147)
          	at com.bigdata.rdf.sail.TestMROWTransactions$1Writer.call(TestMROWTransactions.java:1)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:662)
          Caused by: java.lang.RuntimeException: Could not commit index: name=kb.spo.POCS, commitList=[DirtyListener{name=kb.spo.CSPO,checkpointAddr=0}, DirtyListener{name=kb.spo.OCSP,checkpointAddr=0}, DirtyListener{name=kb.spo.PCSO,checkpointAddr=0}, DirtyListener{name=kb.spo.POCS,needsCheckpoint}, DirtyListener{name=kb.spo.SOPC,needsCheckpoint}, DirtyListener{name=kb.spo.SPOC,needsCheckpoint}]
          	at com.bigdata.journal.Name2Addr.handleCommit(Name2Addr.java:573)
          	at com.bigdata.journal.AbstractJournal.notifyCommitters(AbstractJournal.java:2001)
          	at com.bigdata.journal.AbstractJournal.commitNow(AbstractJournal.java:2364)
          	at com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2276)
          	... 8 more
          Caused by: java.lang.IllegalStateException
          	at com.bigdata.btree.NodeSerializer.encodeLive(NodeSerializer.java:419)
          	at com.bigdata.btree.AbstractBTree.writeNodeOrLeaf(AbstractBTree.java:3636)
          	at com.bigdata.btree.AbstractBTree.writeNodeRecursive(AbstractBTree.java:3493)
          	at com.bigdata.btree.BTree.flush(BTree.java:749)
          	at com.bigdata.btree.BTree._writeCheckpoint2(BTree.java:942)
          	at com.bigdata.btree.BTree.writeCheckpoint2(BTree.java:904)
          	at com.bigdata.btree.BTree.handleCommit(BTree.java:1248)
          	at com.bigdata.journal.Name2Addr.handleCommit(Name2Addr.java:565)
          	... 11 more
          

          The test was:

          public void test_multiple_csem_transaction_nohistory() throws Exception {
          		
          		for(int i=0; i<1000; i++) {
          			log.warn("Trial: " + i);
          			domultiple_csem_transaction2(0/* retentionMillis */,
          				2/* nreaderThreads */, 1000/* nwriters */, 20 * 1000/* nreaders */);
          		}
          
          	}
          
          Show
          bryanthompson bryanthompson added a comment - An exception was observed 8 hours into a stress test run against the RWSTORE_DEBUG_1_1_0 branch r5931. The exception was: java.lang.RuntimeException: Could not commit index: name=kb.spo.POCS, commitList=[DirtyListener{name=kb.spo.CSPO,checkpointAddr=0}, DirtyListener{name=kb.spo.OCSP,checkpointAddr=0}, DirtyListener{name=kb.spo.PCSO,checkpointAddr=0}, DirtyListener{name=kb.spo.POCS,needsCheckpoint}, DirtyListener{name=kb.spo.SOPC,needsCheckpoint}, DirtyListener{name=kb.spo.SPOC,needsCheckpoint}]: lastRootBlock=rootBlock{ rootBlock=0, challisField=16, version=2, nextOffset=661424971997, localTime=1328061936805 [Tuesday, January 31, 2012 9:05:36 PM EST], firstCommitTime=1328061934263 [Tuesday, January 31, 2012 9:05:34 PM EST], lastCommitTime=1328061936793 [Tuesday, January 31, 2012 9:05:36 PM EST], commitCounter=16, commitRecordAddr={off=NATIVE:-42021,len=422}, commitRecordIndexAddr={off=NATIVE:-8412,len=220}, quorumToken=0, metaBitsAddr=60406366258, metaStartAddr=160, storeType=RW, uuid=fc891ff9-c773-4698-9362-08ef700202b0, offsetBits=42, checksum=202121150, createTime=1328061934235 [Tuesday, January 31, 2012 9:05:34 PM EST], closeTime=0} at com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2278) at com.bigdata.rdf.store.LocalTripleStore.commit(LocalTripleStore.java:78) at com.bigdata.rdf.sail.TestMROWTransactions$1Writer.call(TestMROWTransactions.java:147) at com.bigdata.rdf.sail.TestMROWTransactions$1Writer.call(TestMROWTransactions.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: Could not commit index: name=kb.spo.POCS, commitList=[DirtyListener{name=kb.spo.CSPO,checkpointAddr=0}, DirtyListener{name=kb.spo.OCSP,checkpointAddr=0}, DirtyListener{name=kb.spo.PCSO,checkpointAddr=0}, DirtyListener{name=kb.spo.POCS,needsCheckpoint}, DirtyListener{name=kb.spo.SOPC,needsCheckpoint}, DirtyListener{name=kb.spo.SPOC,needsCheckpoint}] at com.bigdata.journal.Name2Addr.handleCommit(Name2Addr.java:573) at com.bigdata.journal.AbstractJournal.notifyCommitters(AbstractJournal.java:2001) at com.bigdata.journal.AbstractJournal.commitNow(AbstractJournal.java:2364) at com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2276) ... 8 more Caused by: java.lang.IllegalStateException at com.bigdata.btree.NodeSerializer.encodeLive(NodeSerializer.java:419) at com.bigdata.btree.AbstractBTree.writeNodeOrLeaf(AbstractBTree.java:3636) at com.bigdata.btree.AbstractBTree.writeNodeRecursive(AbstractBTree.java:3493) at com.bigdata.btree.BTree.flush(BTree.java:749) at com.bigdata.btree.BTree._writeCheckpoint2(BTree.java:942) at com.bigdata.btree.BTree.writeCheckpoint2(BTree.java:904) at com.bigdata.btree.BTree.handleCommit(BTree.java:1248) at com.bigdata.journal.Name2Addr.handleCommit(Name2Addr.java:565) ... 11 more The test was: public void test_multiple_csem_transaction_nohistory() throws Exception { for(int i=0; i<1000; i++) { log.warn("Trial: " + i); domultiple_csem_transaction2(0/* retentionMillis */, 2/* nreaderThreads */, 1000/* nwriters */, 20 * 1000/* nreaders */); } }
          Hide
          bryanthompson bryanthompson added a comment -

          Martyn and I looked at this some more. In working to bring forward the code for overwriting deleted records in the write cache, he wound up being able to trigger this quite quickly (however, we are not yet convinced that the problem is not due to the overwrite logic). Regardless, we tried wrapping the writeNodeRecursive() and writeNodeOrLeaf logic with (a) synchronized(btree); and (b) the UnisolatedReadWriteIndex's writeLock(). Taking those locks did not cure the observed exception, which is pretty clear evidence that this is not a concurrent modification problem.

          I am beginning to believe that the problem is either one of recursion (through the striterator pattern for the dirty node iterator in writeNodeRecursive()) or simply a bug in the maintenance of the various bits of state on the PO object.

          Show
          bryanthompson bryanthompson added a comment - Martyn and I looked at this some more. In working to bring forward the code for overwriting deleted records in the write cache, he wound up being able to trigger this quite quickly (however, we are not yet convinced that the problem is not due to the overwrite logic). Regardless, we tried wrapping the writeNodeRecursive() and writeNodeOrLeaf logic with (a) synchronized(btree); and (b) the UnisolatedReadWriteIndex's writeLock(). Taking those locks did not cure the observed exception, which is pretty clear evidence that this is not a concurrent modification problem. I am beginning to believe that the problem is either one of recursion (through the striterator pattern for the dirty node iterator in writeNodeRecursive()) or simply a bug in the maintenance of the various bits of state on the PO object.
          Hide
          bryanthompson bryanthompson added a comment -

          Made BTree#flush() private to ensure that it is never invoked outside of the checkpoint protocol, which ensures that the appropriate lock is held.

          Made PO#setDirty(boolean) final.

          Committed revision r5932

          Show
          bryanthompson bryanthompson added a comment - Made BTree#flush() private to ensure that it is never invoked outside of the checkpoint protocol, which ensures that the appropriate lock is held. Made PO#setDirty(boolean) final. Committed revision r5932
          Hide
          bryanthompson bryanthompson added a comment -

          Here is another one from that noHistory test. Child does not have persistent identity. Seems related to https://sourceforge.net/apps/trac/bigdata/ticket/343 since this is essentially an inconsistency in the PO state for the BTree.

          junit.framework.AssertionFailedError: Test failed: firstCause=java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416,
          keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]},
          childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067],
          childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22],
          versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3, retentionMillis=0, nreaderThreads=2, nwriters=1000, nreaders=20000
          	at junit.framework.TestCase2.fail(TestCase2.java:118)
          	at com.bigdata.rdf.sail.TestMROWTransactions.domultiple_csem_transaction2(TestMROWTransactions.java:291)
          	at com.bigdata.rdf.sail.TestMROWTransactionsNoHistory.test_multiple_csem_transaction_nohistory(TestMROWTransactionsNoHistory.java:66)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at junit.framework.TestCase.runTest(TestCase.java:154)
          	at junit.framework.TestCase.runBare(TestCase.java:127)
          	at junit.framework.TestResult$1.protect(TestResult.java:106)
          	at junit.framework.TestResult.runProtected(TestResult.java:124)
          	at junit.framework.TestResult.run(TestResult.java:109)
          	at junit.framework.TestCase.run(TestCase.java:118)
          	at junit.framework.TestSuite.runTest(TestSuite.java:208)
          	at junit.framework.TestSuite.run(TestSuite.java:203)
          	at junit.framework.TestSuite.runTest(TestSuite.java:208)
          	at junit.framework.TestSuite.run(TestSuite.java:203)
          	at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
          	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
          Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416,
          keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]},
          childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067],
          childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22],
          versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3
          	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1481)
          	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1658)
          	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1521)
          	at com.bigdata.striterator.AbstractChunkedResolverator._hasNext(AbstractChunkedResolverator.java:327)
          	at com.bigdata.striterator.AbstractChunkedResolverator.hasNext(AbstractChunkedResolverator.java:303)
          	at com.bigdata.rdf.sail.TestMROWTransactions$1Reader.call(TestMROWTransactions.java:196)
          	at com.bigdata.rdf.sail.TestMROWTransactions$1Reader.call(TestMROWTransactions.java:1)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:619)
          Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416,
          keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]},
          childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067],
          childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22],
          versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3
          	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
          	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
          	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1434)
          	... 11 more
          Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416,
          keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]},
          childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067],
          childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22],
          versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3
          	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1481)
          	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1658)
          	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1521)
          	at com.bigdata.relation.accesspath.ChunkConsumerIterator.hasNext(ChunkConsumerIterator.java:127)
          	at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:215)
          	at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:1)
          	... 5 more
          Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416,
          keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]},
          childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067],
          childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22],
          versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3
          	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
          	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
          	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1434)
          	... 10 more
          Caused by: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416,
          keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196],
          [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5],
          [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45],
          [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]},
          childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067],
          childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22],
          versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3
          	at com.bigdata.btree.Node._getChild(Node.java:2704)
          	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:340)
          	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:1)
          	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:79)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:94)
          	at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:482)
          	at com.bigdata.btree.Node.getChild(Node.java:2624)
          	at com.bigdata.btree.ChildIterator.next(ChildIterator.java:163)
          	at com.bigdata.btree.ChildIterator.next(ChildIterator.java:1)
          	at cutthecrap.utils.striterators.Expanderator.getNext(Expanderator.java:34)
          	at cutthecrap.utils.striterators.Prefetch.checkInit(Prefetch.java:12)
          	at cutthecrap.utils.striterators.Prefetch.hasNext(Prefetch.java:20)
          	at cutthecrap.utils.striterators.Appenderator.getNext(Appenderator.java:57)
          	at cutthecrap.utils.striterators.Prefetch.checkInit(Prefetch.java:12)
          	at cutthecrap.utils.striterators.Prefetch.hasNext(Prefetch.java:20)
          	at cutthecrap.utils.striterators.Appenderator.getNext(Appenderator.java:57)
          	at cutthecrap.utils.striterators.Prefetch.checkInit(Prefetch.java:12)
          	at cutthecrap.utils.striterators.Prefetch.hasNext(Prefetch.java:20)
          	at cutthecrap.utils.striterators.Appenderator.getNext(Appenderator.java:57)
          	at cutthecrap.utils.striterators.Prefetch.checkInit(Prefetch.java:12)
          	at cutthecrap.utils.striterators.Prefetch.hasNext(Prefetch.java:20)
          	at cutthecrap.utils.striterators.Striterator.hasNext(Striterator.java:80)
          	at cutthecrap.utils.striterators.Expanderator.availableTailOp(Expanderator.java:49)
          	at cutthecrap.utils.striterators.Striterator.next(Striterator.java:91)
          	at com.bigdata.btree.AbstractNode$PostOrderEntryIterator.next(AbstractNode.java:664)
          	at com.bigdata.btree.AbstractNode$PostOrderEntryIterator.next(AbstractNode.java:1)
          	at cutthecrap.utils.striterators.Resolverator.next(Resolverator.java:54)
          	at cutthecrap.utils.striterators.Striterator.next(Striterator.java:87)
          	at com.bigdata.striterator.ChunkedWrappedIterator.nextChunk(ChunkedWrappedIterator.java:246)
          	at com.bigdata.relation.accesspath.AccessPath$ChunkConsumerTask.call(AccessPath.java:1192)
          	at com.bigdata.relation.accesspath.AccessPath$ChunkConsumerTask.call(AccessPath.java:1)
          	... 5 more
          
          Show
          bryanthompson bryanthompson added a comment - Here is another one from that noHistory test. Child does not have persistent identity. Seems related to https://sourceforge.net/apps/trac/bigdata/ticket/343 since this is essentially an inconsistency in the PO state for the BTree. junit.framework.AssertionFailedError: Test failed: firstCause=java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416, keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [ [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]}, childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067], childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22], versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3, retentionMillis=0, nreaderThreads=2, nwriters=1000, nreaders=20000 at junit.framework.TestCase2.fail(TestCase2.java:118) at com.bigdata.rdf.sail.TestMROWTransactions.domultiple_csem_transaction2(TestMROWTransactions.java:291) at com.bigdata.rdf.sail.TestMROWTransactionsNoHistory.test_multiple_csem_transaction_nohistory(TestMROWTransactionsNoHistory.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:154) at junit.framework.TestCase.runBare(TestCase.java:127) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416, keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [ [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]}, childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067], childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22], versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3 at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1481) at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1658) at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1521) at com.bigdata.striterator.AbstractChunkedResolverator._hasNext(AbstractChunkedResolverator.java:327) at com.bigdata.striterator.AbstractChunkedResolverator.hasNext(AbstractChunkedResolverator.java:303) at com.bigdata.rdf.sail.TestMROWTransactions$1Reader.call(TestMROWTransactions.java:196) at com.bigdata.rdf.sail.TestMROWTransactions$1Reader.call(TestMROWTransactions.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416, keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [ [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]}, childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067], childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22], versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1434) ... 11 more Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416, keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [ [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]}, childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067], childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22], versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3 at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1481) at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1658) at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1521) at com.bigdata.relation.accesspath.ChunkConsumerIterator.hasNext(ChunkConsumerIterator.java:127) at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:215) at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:1) ... 5 more Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416, keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [ [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]}, childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067], childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22], versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1434) ... 10 more Caused by: java.lang.AssertionError: Child does not have persistent identity: this=com.bigdata.btree.Node@100c991#-1940569303547521{ isDirty=false, isDeleted=false, addr=-1940569303547521, parent=com.bigdata.btree.Node@8838dc#-1695872131792373, nkeys=17, minKeys=15, maxKeys=31, nchildren=18, spannedTupleCount=416, keys=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$CodedRabaImpl{ capacity=17, size=17, isKeys=true, isReadOnly=true, [ [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 13, 128, 128, 0, 0, 0, 0, 0, 5, 198], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 29], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 0, 196], [128, 128, 0, 0, 0, 0, 0, 2, 0, 128, 128, 0, 0, 0, 0, 0, 1, 10, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 37], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 70, 128, 128, 0, 0, 0, 0, 0, 6], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 105], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 135], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 196, 128, 128, 0, 0, 0, 0, 0, 7], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 0, 237], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 1, 51, 128, 128, 0, 0, 0, 0, 0, 5], [128, 128, 0, 0, 0, 0, 0, 2, 1, 128, 128, 0, 0, 0, 0, 0, 2], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 21], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 45], [128, 128, 0, 0, 0, 0, 0, 2, 2, 128, 128, 0, 0, 0, 0, 0, 0, 81, 128, 128, 0, 0, 0, 0, 0, 5]]}, childAddr=[-1939894993682002, -1030994014502496, -1923552643120523, 0, -1007315859799367, -1647429195660605, -1303840401915363, -1538633379085800, -967729146232333, -1610818894429707, -902445643333177, -653852936240708, -633546330865231, -1308397362216431, -857704969010607, -163238822018455, -2322864342564393, -1315458288451067], childEntryCount=[18, 17, 27, 32, 30, 30, 24, 23, 21, 22, 20, 18, 18, 23, 25, 27, 19, 22], versionTimestamps={min=0,max=1328116769750}, children=[L, L, L, U, U, U, L, U, U, U, U, U, U, U, U, U, U, U]}, index=3 at com.bigdata.btree.Node._getChild(Node.java:2704) at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:340) at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:1) at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:79) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:94) at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:482) at com.bigdata.btree.Node.getChild(Node.java:2624) at com.bigdata.btree.ChildIterator.next(ChildIterator.java:163) at com.bigdata.btree.ChildIterator.next(ChildIterator.java:1) at cutthecrap.utils.striterators.Expanderator.getNext(Expanderator.java:34) at cutthecrap.utils.striterators.Prefetch.checkInit(Prefetch.java:12) at cutthecrap.utils.striterators.Prefetch.hasNext(Prefetch.java:20) at cutthecrap.utils.striterators.Appenderator.getNext(Appenderator.java:57) at cutthecrap.utils.striterators.Prefetch.checkInit(Prefetch.java:12) at cutthecrap.utils.striterators.Prefetch.hasNext(Prefetch.java:20) at cutthecrap.utils.striterators.Appenderator.getNext(Appenderator.java:57) at cutthecrap.utils.striterators.Prefetch.checkInit(Prefetch.java:12) at cutthecrap.utils.striterators.Prefetch.hasNext(Prefetch.java:20) at cutthecrap.utils.striterators.Appenderator.getNext(Appenderator.java:57) at cutthecrap.utils.striterators.Prefetch.checkInit(Prefetch.java:12) at cutthecrap.utils.striterators.Prefetch.hasNext(Prefetch.java:20) at cutthecrap.utils.striterators.Striterator.hasNext(Striterator.java:80) at cutthecrap.utils.striterators.Expanderator.availableTailOp(Expanderator.java:49) at cutthecrap.utils.striterators.Striterator.next(Striterator.java:91) at com.bigdata.btree.AbstractNode$PostOrderEntryIterator.next(AbstractNode.java:664) at com.bigdata.btree.AbstractNode$PostOrderEntryIterator.next(AbstractNode.java:1) at cutthecrap.utils.striterators.Resolverator.next(Resolverator.java:54) at cutthecrap.utils.striterators.Striterator.next(Striterator.java:87) at com.bigdata.striterator.ChunkedWrappedIterator.nextChunk(ChunkedWrappedIterator.java:246) at com.bigdata.relation.accesspath.AccessPath$ChunkConsumerTask.call(AccessPath.java:1192) at com.bigdata.relation.accesspath.AccessPath$ChunkConsumerTask.call(AccessPath.java:1) ... 5 more
          Hide
          bryanthompson bryanthompson added a comment -

          Observed test failure in StressTestUnisolatedReadWriteIndex. This test suite had somehow been dropped from CI. This failure was observed against TestWORMStrategy with uncommitted code related to the group commit feature (BLZG-1046).

          junit.framework.AssertionFailedError: Not expecting: java.util.concurrent.ExecutionException: java.lang.AssertionError
          	at junit.framework.TestCase2.fail(TestCase2.java:90)
          	at com.bigdata.journal.StressTestUnisolatedReadWriteIndex.doConcurrentClientTest(StressTestUnisolatedReadWriteIndex.java:353)
          	at com.bigdata.journal.StressTestUnisolatedReadWriteIndex.test_concurrentClients(StressTestUnisolatedReadWriteIndex.java:135)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at junit.framework.TestCase.runTest(TestCase.java:154)
          	at junit.framework.TestCase.runBare(TestCase.java:127)
          	at junit.framework.TestResult$1.protect(TestResult.java:106)
          	at junit.framework.TestResult.runProtected(TestResult.java:124)
          	at junit.framework.TestResult.run(TestResult.java:109)
          	at junit.framework.TestCase.run(TestCase.java:118)
          	at junit.framework.TestSuite.runTest(TestSuite.java:208)
          	at junit.framework.TestSuite.run(TestSuite.java:203)
          	at junit.framework.TestSuite.runTest(TestSuite.java:208)
          	at junit.framework.TestSuite.run(TestSuite.java:203)
          	at junit.framework.TestSuite.runTest(TestSuite.java:208)
          	at junit.framework.TestSuite.run(TestSuite.java:203)
          	at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
          	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
          Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
          	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
          	at java.util.concurrent.FutureTask.get(FutureTask.java:188)
          	at com.bigdata.journal.StressTestUnisolatedReadWriteIndex.doConcurrentClientTest(StressTestUnisolatedReadWriteIndex.java:323)
          	... 23 more
          Caused by: java.lang.AssertionError
          	at com.bigdata.btree.AbstractBTree.writeNodeOrLeaf(AbstractBTree.java:3626)
          	at com.bigdata.btree.DefaultEvictionListener.evicted(DefaultEvictionListener.java:91)
          	at com.bigdata.btree.DefaultEvictionListener.evicted(DefaultEvictionListener.java:1)
          	at com.bigdata.cache.HardReferenceQueue.evict(HardReferenceQueue.java:226)
          	at com.bigdata.cache.HardReferenceQueue.beforeOffer(HardReferenceQueue.java:199)
          	at com.bigdata.cache.RingBuffer.add(RingBuffer.java:159)
          	at com.bigdata.cache.HardReferenceQueue.add(HardReferenceQueue.java:176)
          	at com.bigdata.btree.AbstractBTree.doTouch(AbstractBTree.java:3438)
          	at com.bigdata.btree.AbstractBTree.doSyncTouch(AbstractBTree.java:3404)
          	at com.bigdata.btree.AbstractBTree.touch(AbstractBTree.java:3371)
          	at com.bigdata.btree.Node.insert(Node.java:917)
          	at com.bigdata.btree.Node.insert(Node.java:923)
          	at com.bigdata.btree.Node.insert(Node.java:923)
          	at com.bigdata.btree.AbstractBTree.insert(AbstractBTree.java:2062)
          	at com.bigdata.btree.AbstractBTree.insert(AbstractBTree.java:2006)
          	at com.bigdata.btree.UnisolatedReadWriteIndex.insert(UnisolatedReadWriteIndex.java:527)
          	at com.bigdata.journal.StressTestUnisolatedReadWriteIndex$WriteTask.call(StressTestUnisolatedReadWriteIndex.java:493)
          	at com.bigdata.journal.StressTestUnisolatedReadWriteIndex$WriteTask.call(StressTestUnisolatedReadWriteIndex.java:1)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          	at java.lang.Thread.run(Thread.java:745)
          
          Show
          bryanthompson bryanthompson added a comment - Observed test failure in StressTestUnisolatedReadWriteIndex. This test suite had somehow been dropped from CI. This failure was observed against TestWORMStrategy with uncommitted code related to the group commit feature ( BLZG-1046 ). junit.framework.AssertionFailedError: Not expecting: java.util.concurrent.ExecutionException: java.lang.AssertionError at junit.framework.TestCase2.fail(TestCase2.java:90) at com.bigdata.journal.StressTestUnisolatedReadWriteIndex.doConcurrentClientTest(StressTestUnisolatedReadWriteIndex.java:353) at com.bigdata.journal.StressTestUnisolatedReadWriteIndex.test_concurrentClients(StressTestUnisolatedReadWriteIndex.java:135) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:154) at junit.framework.TestCase.runBare(TestCase.java:127) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at com.bigdata.journal.StressTestUnisolatedReadWriteIndex.doConcurrentClientTest(StressTestUnisolatedReadWriteIndex.java:323) ... 23 more Caused by: java.lang.AssertionError at com.bigdata.btree.AbstractBTree.writeNodeOrLeaf(AbstractBTree.java:3626) at com.bigdata.btree.DefaultEvictionListener.evicted(DefaultEvictionListener.java:91) at com.bigdata.btree.DefaultEvictionListener.evicted(DefaultEvictionListener.java:1) at com.bigdata.cache.HardReferenceQueue.evict(HardReferenceQueue.java:226) at com.bigdata.cache.HardReferenceQueue.beforeOffer(HardReferenceQueue.java:199) at com.bigdata.cache.RingBuffer.add(RingBuffer.java:159) at com.bigdata.cache.HardReferenceQueue.add(HardReferenceQueue.java:176) at com.bigdata.btree.AbstractBTree.doTouch(AbstractBTree.java:3438) at com.bigdata.btree.AbstractBTree.doSyncTouch(AbstractBTree.java:3404) at com.bigdata.btree.AbstractBTree.touch(AbstractBTree.java:3371) at com.bigdata.btree.Node.insert(Node.java:917) at com.bigdata.btree.Node.insert(Node.java:923) at com.bigdata.btree.Node.insert(Node.java:923) at com.bigdata.btree.AbstractBTree.insert(AbstractBTree.java:2062) at com.bigdata.btree.AbstractBTree.insert(AbstractBTree.java:2006) at com.bigdata.btree.UnisolatedReadWriteIndex.insert(UnisolatedReadWriteIndex.java:527) at com.bigdata.journal.StressTestUnisolatedReadWriteIndex$WriteTask.call(StressTestUnisolatedReadWriteIndex.java:493) at com.bigdata.journal.StressTestUnisolatedReadWriteIndex$WriteTask.call(StressTestUnisolatedReadWriteIndex.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
          Hide
          bryanthompson bryanthompson added a comment -

          I am able to hit this trace with either TestRWJournal or TestWORMStrategy iff the test is linked into the TestJournalBasics test suite. This is not deterministic. I am using 2GB heap with assertions enabled and server mode against the 1.3.x development branch. Martyn and I have discussed possible root causes, including whether the test or the BTree is at fault. We are continuing to look at ways in which the write retention queue might be the root cause of this exception (through the eviction of a parent of the node or leaf that is undergoing mutation).

          Show
          bryanthompson bryanthompson added a comment - I am able to hit this trace with either TestRWJournal or TestWORMStrategy iff the test is linked into the TestJournalBasics test suite. This is not deterministic. I am using 2GB heap with assertions enabled and server mode against the 1.3.x development branch. Martyn and I have discussed possible root causes, including whether the test or the BTree is at fault. We are continuing to look at ways in which the write retention queue might be the root cause of this exception (through the eviction of a parent of the node or leaf that is undergoing mutation).
          Hide
          bryanthompson bryanthompson added a comment -

          This has been traced to a test error. The problem does not appear if the test is run without a timeout. The timeout is causing the interrupt of the tasks, but the tasks are not all cancelled simultaneously. This means that some tasks can still acquire the necessary lock in the UnisolatedReadWriteIndex class and proceed with a mutation. However, if a mutation had been interrupted by an interrupt in another task then the write retention queue and the index state are now inconsistent and the error can arise.

          See BLZG-933 for a related bug (unisoalted connection iteration patterrns can not be combined with unisolated connection mutation)

          Show
          bryanthompson bryanthompson added a comment - This has been traced to a test error. The problem does not appear if the test is run without a timeout. The timeout is causing the interrupt of the tasks, but the tasks are not all cancelled simultaneously. This means that some tasks can still acquire the necessary lock in the UnisolatedReadWriteIndex class and proceed with a mutation. However, if a mutation had been interrupted by an interrupt in another task then the write retention queue and the index state are now inconsistent and the error can arise. See BLZG-933 for a related bug (unisoalted connection iteration patterrns can not be combined with unisolated connection mutation)

            People

            • Assignee:
              bryanthompson bryanthompson
              Reporter:
              bryanthompson bryanthompson
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: