Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1575 Small slot optimization meta-ticket
  3. BLZG-1602

BTree Eviction - Index Out Of Bounds when loading data (small slot optimization?)

    XMLWordPrintable

    Details

    • Type: Sub-task
    • Status: Done
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_2_0_0
    • Component/s: RWStore
    • Labels:
      None

      Description

      Encountered the following error message while loading a large dataset using the DataLoader. This happened in branch load-performance-geo-dteext, commit point: https://github.com/SYSTAP/bigdata/commit/be38367b5b56e2f85ac4648aa41c2373feb29421

           [java] ERROR: 1910959      com.bigdata.journal.Journal.executorService3 com.bigdata.rwstore.RWStore.alloc(RWStore.java:2841): java.lang.IndexOutOfBoundsException: Index: 28, Size: 28
           [java] java.lang.IndexOutOfBoundsException: Index: 28, Size: 28
           [java] 	at java.util.ArrayList.rangeCheck(ArrayList.java:635)
           [java] 	at java.util.ArrayList.get(ArrayList.java:411)
           [java] 	at com.bigdata.rwstore.FixedAllocator.allocFromIndex(FixedAllocator.java:1025)
           [java] 	at com.bigdata.rwstore.FixedAllocator.alloc(FixedAllocator.java:914)
           [java] 	at com.bigdata.rwstore.RWStore.alloc(RWStore.java:2818)
           [java] 	at com.bigdata.rwstore.RWStore.alloc(RWStore.java:3000)
           [java] 	at com.bigdata.journal.RWStrategy.write(RWStrategy.java:239)
           [java] 	at com.bigdata.journal.RWStrategy.write(RWStrategy.java:199)
           [java] 	at com.bigdata.journal.AbstractJournal.write(AbstractJournal.java:4311)
           [java] 	at com.bigdata.btree.AbstractBTree.writeNodeOrLeaf(AbstractBTree.java:3929)
           [java] 	at com.bigdata.btree.DefaultEvictionListener.evicted(DefaultEvictionListener.java:107)
           [java] 	at com.bigdata.btree.DefaultEvictionListener.evicted(DefaultEvictionListener.java:37)
           [java] 	at com.bigdata.cache.HardReferenceQueue.evict(HardReferenceQueue.java:226)
           [java] 	at com.bigdata.cache.HardReferenceQueue.beforeOffer(HardReferenceQueue.java:199)
           [java] 	at com.bigdata.cache.RingBuffer.add(RingBuffer.java:159)
           [java] 	at com.bigdata.cache.HardReferenceQueue.add(HardReferenceQueue.java:176)
           [java] 	at com.bigdata.btree.AbstractBTree.doTouch(AbstractBTree.java:3589)
           [java] 	at com.bigdata.btree.AbstractBTree.doSyncTouch(AbstractBTree.java:3555)
           [java] 	at com.bigdata.btree.AbstractBTree.touch(AbstractBTree.java:3520)
           [java] 	at com.bigdata.btree.AbstractNode.<init>(AbstractNode.java:298)
           [java] 	at com.bigdata.btree.Leaf.<init>(Leaf.java:348)
           [java] 	at com.bigdata.btree.BTree$NodeFactory.allocLeaf(BTree.java:1925)
           [java] 	at com.bigdata.btree.NodeSerializer.wrap(NodeSerializer.java:390)
           [java] 	at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:4097)
           [java] 	at com.bigdata.btree.Node._getChild(Node.java:2727)
           [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:370)
           [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:353)
           [java] 	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
           [java] 	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
           [java] 	at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:539)
           [java] 	at com.bigdata.btree.Node.getChild(Node.java:2628)
           [java] 	at com.bigdata.btree.Node.lookup(Node.java:938)
           [java] 	at com.bigdata.btree.Node.lookup(Node.java:940)
           [java] 	at com.bigdata.btree.Node.lookup(Node.java:940)
           [java] 	at com.bigdata.btree.Node.lookup(Node.java:940)
           [java] 	at com.bigdata.btree.AbstractBTree.lookup(AbstractBTree.java:2406)
           [java] 	at com.bigdata.btree.AbstractBTree.lookup(AbstractBTree.java:2334)
           [java] 	at com.bigdata.rdf.spo.SPOIndexWriteProc.applyOnce(SPOIndexWriteProc.java:232)
           [java] 	at com.bigdata.btree.proc.AbstractKeyArrayIndexProcedure.apply(AbstractKeyArrayIndexProcedure.java:381)
           [java] 	at com.bigdata.btree.UnisolatedReadWriteIndex.submit(UnisolatedReadWriteIndex.java:723)
           [java] 	at com.bigdata.rdf.spo.SPOIndexWriter.call(SPOIndexWriter.java:363)
           [java] 	at com.bigdata.rdf.spo.SPOIndexWriter.call(SPOIndexWriter.java:68)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
           [java] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
           [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
           [java] 	at java.lang.Thread.run(Thread.java:745)
           [java] INFO : 1911198      main com.bigdata.rdf.store.DataLoader$1.processingNotification(DataLoader.java:1525): 82200000 stmts buffered in 1907.585 secs, rate= 43091, baseURL=file:[ANONYMIZED], totalStatementsSoFar=82200000
           [java] ERROR: 1933188      com.bigdata.journal.Journal.executorService1 com.bigdata.rdf.spo.SPORelation.logFuture(SPORelation.java:2303): java.util.concurrent.ExecutionException: java.lang.RuntimeException: De-serialization problem: addr={off=56234739712,len=1419} from store=[ANONYMIZED] : cause=com.bigdata.btree.EvictionError: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 28, Size: 28
           [java] java.util.concurrent.ExecutionException: java.lang.RuntimeException: De-serialization problem: addr={off=56234739712,len=1419} from store=… : cause=com.bigdata.btree.EvictionError: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 28, Size: 28
           [java] 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
           [java] 	at java.util.concurrent.FutureTask.get(FutureTask.java:188)
           [java] 	at com.bigdata.rdf.spo.SPORelation.logFuture(SPORelation.java:2298)
           [java] 	at com.bigdata.rdf.spo.SPORelation.insert(SPORelation.java:2253)
           [java] 	at com.bigdata.rdf.store.AbstractTripleStore.addStatements(AbstractTripleStore.java:4221)
           [java] 	at com.bigdata.rdf.rio.StatementBuffer$Batch.writeSPOs(StatementBuffer.java:2003)
           [java] 	at com.bigdata.rdf.rio.StatementBuffer$Batch.addStatements(StatementBuffer.java:1854)
           [java] 	at com.bigdata.rdf.rio.StatementBuffer$Batch.writeNow(StatementBuffer.java:1740)
           [java] 	at com.bigdata.rdf.rio.StatementBuffer$Batch.access$800(StatementBuffer.java:1550)
           [java] 	at com.bigdata.rdf.rio.StatementBuffer$DrainQueueCallable.drainQueueAndMergeBatches(StatementBuffer.java:868)
           [java] 	at com.bigdata.rdf.rio.StatementBuffer$DrainQueueCallable.call(StatementBuffer.java:801)
           [java] 	at com.bigdata.rdf.rio.StatementBuffer$DrainQueueCallable.call(StatementBuffer.java:770)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
           [java] 	at com.bigdata.util.concurrent.LatchedExecutor$1.run(LatchedExecutor.java:121)
           [java] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
           [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
           [java] 	at java.lang.Thread.run(Thread.java:745)
           [java] Caused by: java.lang.RuntimeException: De-serialization problem: addr={off=56234739712,len=1419} from store=…: cause=com.bigdata.btree.EvictionError: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 28, Size: 28
           [java] 	at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:4110)
           [java] 	at com.bigdata.btree.Node._getChild(Node.java:2727)
           [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:370)
           [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:353)
           [java] 	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
           [java] 	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
           [java] 	at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:539)
           [java] 	at com.bigdata.btree.Node.getChild(Node.java:2628)
           [java] 	at com.bigdata.btree.Node.lookup(Node.java:938)
           [java] 	at com.bigdata.btree.Node.lookup(Node.java:940)
           [java] 	at com.bigdata.btree.Node.lookup(Node.java:940)
           [java] 	at com.bigdata.btree.Node.lookup(Node.java:940)
           [java] 	at com.bigdata.btree.AbstractBTree.lookup(AbstractBTree.java:2406)
           [java] 	at com.bigdata.btree.AbstractBTree.lookup(AbstractBTree.java:2334)
           [java] 	at com.bigdata.rdf.spo.SPOIndexWriteProc.applyOnce(SPOIndexWriteProc.java:232)
           [java] 	at com.bigdata.btree.proc.AbstractKeyArrayIndexProcedure.apply(AbstractKeyArrayIndexProcedure.java:381)
           [java] 	at com.bigdata.btree.UnisolatedReadWriteIndex.submit(UnisolatedReadWriteIndex.java:723)
           [java] 	at com.bigdata.rdf.spo.SPOIndexWriter.call(SPOIndexWriter.java:363)
           [java] 	at com.bigdata.rdf.spo.SPOIndexWriter.call(SPOIndexWriter.java:68)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
           [java] 	... 3 more
           [java] Caused by: com.bigdata.btree.EvictionError: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 28, Size: 28
           [java] 	at com.bigdata.btree.DefaultEvictionListener.evicted(DefaultEvictionListener.java:177)
           [java] 	at com.bigdata.btree.DefaultEvictionListener.evicted(DefaultEvictionListener.java:37)
           [java] 	at com.bigdata.cache.HardReferenceQueue.evict(HardReferenceQueue.java:226)
           [java] 	at com.bigdata.cache.HardReferenceQueue.beforeOffer(HardReferenceQueue.java:199)
           [java] 	at com.bigdata.cache.RingBuffer.add(RingBuffer.java:159)
           [java] 	at com.bigdata.cache.HardReferenceQueue.add(HardReferenceQueue.java:176)
           [java] 	at com.bigdata.btree.AbstractBTree.doTouch(AbstractBTree.java:3589)
           [java] 	at com.bigdata.btree.AbstractBTree.doSyncTouch(AbstractBTree.java:3555)
           [java] 	at com.bigdata.btree.AbstractBTree.touch(AbstractBTree.java:3520)
           [java] 	at com.bigdata.btree.AbstractNode.<init>(AbstractNode.java:298)
           [java] 	at com.bigdata.btree.Leaf.<init>(Leaf.java:348)
           [java] 	at com.bigdata.btree.BTree$NodeFactory.allocLeaf(BTree.java:1925)
           [java] 	at com.bigdata.btree.NodeSerializer.wrap(NodeSerializer.java:390)
           [java] 	at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:4097)
           [java] 	... 23 more
           [java] Caused by: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 28, Size: 28
           [java] 	at com.bigdata.rwstore.RWStore.alloc(RWStore.java:2843)
           [java] 	at com.bigdata.rwstore.RWStore.alloc(RWStore.java:3000)
           [java] 	at com.bigdata.journal.RWStrategy.write(RWStrategy.java:239)
           [java] 	at com.bigdata.journal.RWStrategy.write(RWStrategy.java:199)
           [java] 	at com.bigdata.journal.AbstractJournal.write(AbstractJournal.java:4311)
           [java] 	at com.bigdata.btree.AbstractBTree.writeNodeOrLeaf(AbstractBTree.java:3929)
           [java] 	at com.bigdata.btree.DefaultEvictionListener.evicted(DefaultEvictionListener.java:107)
           [java] 	... 36 more
           [java] Caused by: java.lang.IndexOutOfBoundsException: Index: 28, Size: 28
           [java] 	at java.util.ArrayList.rangeCheck(ArrayList.java:635)
           [java] 	at java.util.ArrayList.get(ArrayList.java:411)
           [java] 	at com.bigdata.rwstore.FixedAllocator.allocFromIndex(FixedAllocator.java:1025)
           [java] 	at com.bigdata.rwstore.FixedAllocator.alloc(FixedAllocator.java:914)
           [java] 	at com.bigdata.rwstore.RWStore.alloc(RWStore.java:2818)
           [java] 	... 42 more
      

      This might be related to the small slot optimization. After all, I restarted the load with small slot optimization disabled and could not observe any further problems. Will attach the properties file for reference. However, setup of the overall loading scenario is kind of complex, please ping me in case you need more infos/experiments.

      In overall, data has about 3.5GB triples, the problem occured after loading half of it. The DataLoader was called multiple times, for multiple files.

        Attachments

          Activity

            People

            Assignee:
            martyncutcher martyncutcher
            Reporter:
            michaelschmidt michaelschmidt
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: