Details

    • Type: Bug
    • Status: Closed - Won't Fix
    • Resolution: Cannot Reproduce
    • Affects Version/s: BIGDATA_RELEASE_1_5_0
    • Fix Version/s: None
    • Component/s: RWStore
    • Labels:
      None

      Description

      The machine was bigdata11. I loaded BSBM 100M using the ant scripts in bigdata-perf/bsbm3. The load was fine. I then launched the BSBM EXPLORE workload against the database. This resulted in a large number of checksum errors being reported by the server console:

      For example

      ERROR: Haltable.java:469: com.bigdata.bop.join.PipelineJoin$JoinTask{ joinOp=com.bigdata.bop.join.PipelineJoin[12](PipelineJoin[10])[ BOp.bopId=12, JoinAnnotations.constraints=null, AST2BOpBase.simpleJoin=true, BOp.evaluationContext=ANY, AccessPathJoinAnnotations.predicate=com.bigdata.rdf.spo.SPOPredicate[11](vendor=null, TermId(69857U)[http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/country], TermId(3U)[http://downlode.org/rdf/iso-3166/countries#DE])[ IPredicate.relationName=[BSBM_284826.spo], IPredicate.timestamp=1425835081052, BOp.bopId=11, AST2BOpBase.estimatedCardinality=7702, AST2BOpBase.originalIndex=POS, IPredicate.flags=[KEYS,VALS,READONLY,PARALLEL]]]} : isFirstCause=true : java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-1196892907
           [java] java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-1196892907
           [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:1966)
           [java] 	at com.bigdata.journal.RWStrategy.readFromLocalStore(RWStrategy.java:733)
           [java] 	at com.bigdata.journal.RWStrategy.read(RWStrategy.java:156)
           [java] 	at com.bigdata.journal.AbstractJournal.read(AbstractJournal.java:4284)
           [java] 	at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:4018)
           [java] 	at com.bigdata.btree.Node._getChild(Node.java:2727)
           [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:363)
           [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:346)
           [java] 	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
           [java] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
           [java] 	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
           [java] 	at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:532)
           [java] 	at com.bigdata.btree.Node.getChild(Node.java:2628)
           [java] 	at com.bigdata.btree.Node.indexOf(Node.java:971)
           [java] 	at com.bigdata.btree.Node.indexOf(Node.java:988)
           [java] 	at com.bigdata.btree.AbstractBTree.rangeCount(AbstractBTree.java:2631)
           [java] 	at com.bigdata.relation.accesspath.AccessPath.historicalRangeCount(AccessPath.java:1418)
           [java] 	at com.bigdata.relation.accesspath.AccessPath.rangeCount(AccessPath.java:1386)
           [java] 	at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.call(PipelineJoin.java:1612)
           [java] 	at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.executeTasks(PipelineJoin.java:1346)
           [java] 	at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:977)
           [java] 	at com.bigdata.bop.join.PipelineJoin$JoinTask.consumeSource(PipelineJoin.java:700)
           [java] 	at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:584)
           [java] 	at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:343)
           [java] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
           [java] 	at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
           [java] 	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1314)
           [java] 	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:894)
           [java] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
           [java] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
           [java] 	at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
           [java] 	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:789)
           [java] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
           [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
           [java] 	at java.lang.Thread.run(Thread.java:724)
           [java] Caused by: com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-1196892907
           [java] 	at com.bigdata.io.writecache.WriteCacheService._readFromLocalDiskIntoNewHeapByteBuffer(WriteCacheService.java:3711)
           [java] 	at com.bigdata.io.writecache.WriteCacheService._getRecord(WriteCacheService.java:3526)
           [java] 	at com.bigdata.io.writecache.WriteCacheService.access$2500(WriteCacheService.java:200)
           [java] 	at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3363)
           [java] 	at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3347)
           [java] 	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
           [java] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
           [java] 	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
           [java] 	at com.bigdata.io.writecache.WriteCacheService.loadRecord(WriteCacheService.java:3468)
           [java] 	at com.bigdata.io.writecache.WriteCacheService.read(WriteCacheService.java:3187)
           [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:1957)
           [java] 	... 37 more
      

      The exact sequence of operations was:

      # git checkout master
      # git pull
      # Build code.
      ant clean bundleJar
      
      ## Setup for BSBM.
      cd bigdata-perf/bsbm3
      # edit build.properties to specify location of bsbmtools
      # emacs build.properties
      # if you do not have the data, you need to generate it now.
      # Load BSBM 100M. Note: Will delete old journal if it exists.  Note: Will load into BSBM_284826 namespace by default (BSBM 100M)
      ant run-load
      # Load: 100000748 stmts added in 2304.494 secs, rate= 43393, commitLatency=0ms
      
      # Start NSS
      ant start-sparql-server
      
      ## Run benchmark
      # Open another terminal. In that terminal do:
      cd /root/workspace/bsbmtools/trunk
      # This runs 120 trials of the BSBM 100M EXPLORE benchmark with 16 threads. This is the ?warmup? procedure.
      for a in `seq 120`; do ./testdriver -o mt_16.xml -seed $RANDOM -w 50 -mt 16 -idir td_100m/td_data http://localhost:80/bigdata/namespace/BSBM_284826/sparql|grep QMpH;done;
      

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        Running DumpJournal as

        http://bigdata11/bigdata/status?dumpJournal&dumpPages}}}
        
        produces the output below and also causes errors to be reported on the server console (see below for those traces)
        
        {{{
        Dump Journal
        
        Running...
        
        magic=e6b4c275
        version=1
        extent=13575716864(12946M), userExtent=13575716176(12946M), bytesAvailable=-874008000174780(-833518982M), nextOffset=874021575890956
        rootBlock{ rootBlock=0, challisField=2, version=3, nextOffset=874021575890956, localTime=1425829363023 [Sunday, March 8, 2015 11:42:43 AM EDT], firstCommitTime=1425827057089 [Sunday, March 8, 2015 11:04:17 AM EDT], lastCommitTime=1425829361645 [Sunday, March 8, 2015 11:42:41 AM EDT], commitCounter=2, commitRecordAddr={off=NATIVE:-25440860,len=422}, commitRecordIndexAddr={off=NATIVE:-2431970,len=220}, blockSequence=3131, quorumToken=-1, metaBitsAddr=846224974413979, metaStartAddr=207149, storeType=RW, uuid=75711495-da41-47d3-af28-98245135b3eb, offsetBits=42, checksum=-1268307968, createTime=1425827056650 [Sunday, March 8, 2015 11:04:16 AM EDT], closeTime=0}
        rootBlock{ rootBlock=1, challisField=1, version=3, nextOffset=206158438413, localTime=1425827057129 [Sunday, March 8, 2015 11:04:17 AM EDT], firstCommitTime=1425827057089 [Sunday, March 8, 2015 11:04:17 AM EDT], lastCommitTime=1425827057089 [Sunday, March 8, 2015 11:04:17 AM EDT], commitCounter=1, commitRecordAddr={off=NATIVE:-57348,len=422}, commitRecordIndexAddr={off=NATIVE:-8195,len=220}, blockSequence=1, quorumToken=-1, metaBitsAddr=56044290095, metaStartAddr=3200, storeType=RW, uuid=75711495-da41-47d3-af28-98245135b3eb, offsetBits=42, checksum=-1550833551, createTime=1425827056650 [Sunday, March 8, 2015 11:04:16 AM EDT], closeTime=0}
        The current root block is #0
        
        -------------------------
        RWStore Allocator Summary
        -------------------------
        AllocatorSize      AllocatorCount   SlotsAllocated  %SlotsAllocated    SlotsRecycled        SlotChurn       SlotsInUse      %SlotsInUse   MeanAllocation    SlotsReserved     %SlotsUnused    BytesReserved     BytesAppData       %SlotWaste         %AppData       %StoreFile      %TotalWaste       %FileWaste 
        64                            154          2326224             8.23          1229776             2.12          1096448             4.70               29          1097728             0.12         70254592         47009299            33.09             0.44             0.53             0.85              0.17 
        128                          2347         16828366            59.54            19045             1.00         16809321            72.08               96         16821760             0.07       2153185280       1625746945            24.50            15.35            16.15            19.24              3.96 
        192                           188          1355532             4.80            18650             1.01          1336882             5.73              151          1343488             0.49        257949696        203087661            21.27             1.92             1.94             2.00              0.41 
        320                            15           143897             0.51            37321             1.35           106576             0.46              225           107520             0.88         34406400         22974189            33.23             0.22             0.26             0.42              0.09 
        512                            11           128264             0.45            53329             1.71            74935             0.32              438            76544             2.10         39190528         34355788            12.34             0.32             0.29             0.18              0.04 
        768                            79           636002             2.25            71528             1.13           564474             2.42              640           566272             0.32        434896896        362266536            16.70             3.42             3.26             2.65              0.54 
        1024                           80           635659             2.25            68427             1.12           567232             2.43              896           569600             0.42        583270400        508730102            12.78             4.80             4.38             2.72              0.56 
        2048                          277          2207412             7.81           228122             1.12          1979290             8.49             1489          1984256             0.25       4063756288       2944976131            27.53            27.81            30.49            40.80              8.39 
        3072                            9           232421             0.82           171522             3.82            60899             0.26             2616            64512             5.60        198180864        173514981            12.45             1.64             1.49             0.90              0.19 
        4096                           20           313587             1.11           178358             2.32           135229             0.58             3609           141312             4.30        578813952        483407345            16.48             4.57             4.34             3.48              0.72 
        8192                           84          3455567            12.23          2865907             5.86           589660             2.53             7271           600064             1.73       4915724288       4181683691            14.93            39.50            36.88            26.77              5.51 
        
        -------------------------
        BLOBS
        -------------------------
        Bucket(K)   Allocations    Allocated      Deletes      Deleted      Current         Data         Mean        Churn
        16              1131836  12783813674      1044103  11700905982        87733   1082907692        11294        12.90
        32                83310   1582810615        78646   1488844103         4664     93966512        18999        17.86
        64                87051   4060621662        79582   3692942259         7469    367679403        46646        11.65
        128                9951    700568541         9137    639776806          814     60791735        70401        12.22
        256                   0            0            0            0            0            0            0         0.00
        512                   0            0            0            0            0            0            0         0.00
        1024                  0            0            0            0            0            0            0         0.00
        2048                  0            0            0            0            0            0            0         0.00
        4096                  0            0            0            0            0            0            0         0.00
        8192                  0            0            0            0            0            0            0         0.00
        16384                 0            0            0            0            0            0            0         0.00
        32768                 0            0            0            0            0            0            0         0.00
        65536                 0            0            0            0            0            0            0         0.00
        2097151               0            0            0            0            0            0            0         0.00
        
        There are 2 commit points.
        CommitRecord{timestamp=1425829361645, commitCounter=2, roots=[-10445227319885604, -109267657387146924, -114775102540742596, -175939040313213, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}
        name=__globalRowStore
        	Checkpoint{indexType=BTree,height=1,nnodes=1,nleaves=4,nentries=95,counter=0,addrRoot=-175934745345874,addrMetadata=-70385924045614,addrBloomFilter=0,addrCheckpoint=-35205846925092}
        	addrMetadata=0, name=__globalRowStore, indexType=BTree, indexUUID=2001afcb-ccdd-491c-b66e-34812a763bd0, branchingFactor=32, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@6e16af5a{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.btree.DefaultTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.DefaultKeyBuilderFactory{ initialCapacity=0, collator=ICU, locale=en_US, strength=null, decomposition=null}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@1352762d{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.CanonicalHuffmanRabaCoder@acc8e44}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=com.bigdata.sparse.LogicalRowSplitHandler@7b9017e5, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
        	com.bigdata.btree.BTreePageStats{indexType=BTree,m=32,nnodes=1,nleaves=4,nrawRecs=0,nodeBytes=174,minNodeBytes=174,maxNodeBytes=174,leafBytes=6357,minLeafBytes=1003,maxLeafBytes=2160,rawRecBytes=0,bytesPerNode=174,bytesPerLeaf=1589,bytesPerRawRec=0,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.2,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.2,slot_2048=0.4,slot_3072=0.2,slot_4096=0.0,slot_8192=0.0,blobs=0.0,newM=668}
        name=BSBM_284826.lex.BLOBS
        	Checkpoint{indexType=BTree,height=2,nnodes=41,nleaves=10576,nentries=3179117,counter=0,addrRoot=-112194102073752973,addrMetadata=-21474835633,addrBloomFilter=0,addrCheckpoint=-10445132830605092}
        	addrMetadata=0, name=BSBM_284826.lex.BLOBS, indexType=BTree, indexUUID=ca18bce6-8e17-41cf-84eb-85e9ef4a538f, branchingFactor=400, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@3110fac3{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.lexicon.BlobsTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=8}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@51bf315f{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.SimpleRabaCoder@5f427de6}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=true, maxRecLen=0, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
        	com.bigdata.btree.BTreePageStats{indexType=BTree,m=400,nnodes=41,nleaves=10576,nrawRecs=3179117,nodeBytes=141846,minNodeBytes=627,maxNodeBytes=5244,leafBytes=59463561,minLeafBytes=3743,maxLeafBytes=7480,rawRecBytes=3820125292,bytesPerNode=3459,bytesPerLeaf=5622,bytesPerRawRec=1201,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.14,slot_8192=0.85,blobs=0.0,newM=611}
        name=BSBM_284826.lex.ID2TERM
        	Checkpoint{indexType=BTree,height=2,nnodes=120,nleaves=48304,nentries=19322065,counter=0,addrRoot=-113739405537048709,addrMetadata=-17179868333,addrBloomFilter=0,addrCheckpoint=-10443706901462820}
        	addrMetadata=0, name=BSBM_284826.lex.ID2TERM, indexType=BTree, indexUUID=44cc03a0-1021-495f-8a6b-c56e2491fbfb, branchingFactor=800, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@657b7e6{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.lexicon.Id2TermTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=9}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@68ea604a{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.SimpleRabaCoder@7357d015}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=true, maxRecLen=16, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
        	com.bigdata.btree.BTreePageStats{indexType=BTree,m=800,nnodes=120,nleaves=48304,nrawRecs=19231729,nodeBytes=623535,minNodeBytes=1915,maxNodeBytes=10005,leafBytes=314000524,minLeafBytes=6489,maxLeafBytes=12155,rawRecBytes=1815706974,bytesPerNode=5196,bytesPerLeaf=6500,bytesPerRawRec=94,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.0,slot_8192=0.99,blobs=0.0,newM=907}
        name=BSBM_284826.lex.TERM2ID
        	Checkpoint{indexType=BTree,height=2,nnodes=99,nleaves=45177,nentries=19322065,counter=19322065,addrRoot=-114687270459535695,addrMetadata=-70390219012891,addrBloomFilter=0,addrCheckpoint=-10445218729951012}
        	addrMetadata=0, name=BSBM_284826.lex.TERM2ID, indexType=BTree, indexUUID=7d1fb58d-43f1-47b1-89dd-d56ca3807bfb, branchingFactor=800, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@4b89abb4{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.lexicon.Term2IdTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.DefaultKeyBuilderFactory{ initialCapacity=0, collator=ICU, locale=en_US, strength=null, decomposition=null}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@3b67dd2a{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.FixedLengthValueRabaCoder@1fe90ede}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
        	com.bigdata.btree.BTreePageStats{indexType=BTree,m=800,nnodes=99,nleaves=45176,nrawRecs=0,nodeBytes=1196198,minNodeBytes=3761,maxNodeBytes=20854,leafBytes=968966798,minLeafBytes=10025,maxLeafBytes=86966,rawRecBytes=0,bytesPerNode=12082,bytesPerLeaf=21448,bytesPerRawRec=0,nerrors=1,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.0,slot_8192=0.0,blobs=0.99,newM=338}
        name=BSBM_284826.spo.JUST
        	Checkpoint{indexType=BTree,height=0,nnodes=0,nleaves=1,nentries=0,counter=0,addrRoot=-105596065939426,addrMetadata=-38654704873,addrBloomFilter=0,addrCheckpoint=-35235911696164}
        	addrMetadata=0, name=BSBM_284826.spo.JUST, indexType=BTree, indexUUID=98a1cb98-9369-4067-8858-f27dc7a6eada, branchingFactor=128, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@5c56efb7{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.JustificationTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@4f9406bd{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.EmptyRabaValueCoder@7f468b1b}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
        	com.bigdata.btree.BTreePageStats{indexType=BTree,m=128,nnodes=0,nleaves=1,nrawRecs=0,nodeBytes=0,minNodeBytes=0,maxNodeBytes=0,leafBytes=30,minLeafBytes=30,maxLeafBytes=30,rawRecBytes=0,bytesPerNode=0,bytesPerLeaf=30,bytesPerRawRec=0,nerrors=0,slot_64=1.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.0,slot_8192=0.0,blobs=0.0,newM=128}
        name=BSBM_284826.spo.OSP
        	Checkpoint{indexType=BTree,height=2,nnodes=298,nleaves=180819,nentries=100000748,counter=0,addrRoot=-114812859598235401,addrMetadata=-30064770301,addrBloomFilter=0,addrCheckpoint=-10445197255114532}
        	addrMetadata=0, name=BSBM_284826.spo.OSP, indexType=BTree, indexUUID=ff4127fa-de79-4880-8ca3-d9dfcd12f592, branchingFactor=1024, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@563ce85a{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.SPOTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@38ba98f1{ratio=8}, leafValuesCoder=com.bigdata.rdf.spo.FastRDFValueCoder2@628f134f}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
        	com.bigdata.btree.BTreePageStats{indexType=BTree,m=1024,nnodes=298,nleaves=180819,nrawRecs=0,nodeBytes=2858553,minNodeBytes=6391,maxNodeBytes=19703,leafBytes=1291109816,minLeafBytes=3415,maxLeafBytes=24756,rawRecBytes=0,bytesPerNode=9592,bytesPerLeaf=7140,bytesPerRawRec=0,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.22,slot_8192=0.52,blobs=0.25,newM=819}
        name=BSBM_284826.spo.POS
        	Checkpoint{indexType=BTree,height=2,nnodes=288,nleaves=182371,nentries=100000748,counter=0,addrRoot=-114812348497126940,addrMetadata=-34359737597,addrBloomFilter=0,addrCheckpoint=-10445192960147236}
        	addrMetadata=0, name=BSBM_284826.spo.POS, indexType=BTree, indexUUID=a1d2ce58-7dd5-4e6e-8429-4f0274f81441, branchingFactor=1024, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@a8a1279{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.SPOTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@6ec2cb56{ratio=8}, leafValuesCoder=com.bigdata.rdf.spo.FastRDFValueCoder2@36b3b3e9}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
        	com.bigdata.btree.BTreePageStats{indexType=BTree,m=1024,nnodes=288,nleaves=182371,nrawRecs=0,nodeBytes=3017187,minNodeBytes=6628,maxNodeBytes=21834,leafBytes=793789115,minLeafBytes=2527,maxLeafBytes=17386,rawRecBytes=0,bytesPerNode=10476,bytesPerLeaf=4352,bytesPerRawRec=0,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.28,slot_4096=0.41,slot_8192=0.26,blobs=0.03,newM=1090}
        name=BSBM_284826.spo.SPO
        	Checkpoint{indexType=BTree,height=2,nnodes=381,nleaves=195313,nentries=100000748,counter=0,addrRoot=-114828394494943415,addrMetadata=-25769802907,addrBloomFilter=0,addrCheckpoint=-10445223024918308}
        	addrMetadata=0, name=BSBM_284826.spo.SPO, indexType=BTree, indexUUID=53f9b734-dfcc-48f7-9ffc-fda4251c28cd, branchingFactor=1024, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@2da77f97{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.SPOTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@4fa9a9eb{ratio=8}, leafValuesCoder=com.bigdata.rdf.spo.FastRDFValueCoder2@bd2ac29}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=BloomFilterFactory{ n=1000000, p=0.02, maxP=0.15, maxN=1883227}, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
        	com.bigdata.btree.BTreePageStats{indexType=BTree,m=1024,nnodes=381,nleaves=195312,nrawRecs=0,nodeBytes=3464020,minNodeBytes=7330,maxNodeBytes=13898,leafBytes=1403679601,minLeafBytes=4721,maxLeafBytes=11511,rawRecBytes=0,bytesPerNode=9091,bytesPerLeaf=7186,bytesPerRawRec=0,nerrors=1,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.0,slot_8192=0.99,blobs=0.0,newM=835}
        name	indexType	m	height	nnodes	nleaves	nentries	nrawRecs	nerrors	nodeBytes	leafBytes	rawRecBytes	totalBytes	avgNodeBytes	avgLeafBytes	avgRawRecBytes	minNodeBytes	maxNodeBytes	minLeafBytes	maxLeafBytes	64	128	192	320	512	768	1024	2048	3072	4096	8192	blobs	newM	curM
        BSBM_284826.lex.BLOBS	BTree	400	2	41	10576	3179117	3179117	0	141846	59463561	3820125292	3879730699	3459	5622	1201	627	5244	3743	7480	0.0	0.0	0.0	0.0	0.0	9.418856550814731E-5	0.0	0.0	0.0016012056136385042	0.14288405387585948	0.8554205519449939	0.0	611	400
        BSBM_284826.lex.ID2TERM	BTree	800	2	120	48304	19322065	19231729	0	623535	314000524	1815706974	2130331033	5196	6500	94	1915	10005	6489	12155	0.0	0.0	0.0	0.0	0.0	0.0	0.0	2.065091690071039E-5	0.0	0.0	0.9998967454154964	8.260366760284156E-5	907	800
        BSBM_284826.lex.TERM2ID	BTree	800	2	99	45176	19322065	0	1	1196198	968966798	0	970162996	12082	21448	0	3761	20854	10025	86966	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	2.2087244616234126E-5	0.0	0.9999779127553837	338	800
        BSBM_284826.spo.JUST	BTree	128	0	0	1	0	0	0	0	30	0	30	0	30	0	0	0	30	30	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	128	128
        BSBM_284826.spo.OSP	BTree	1024	2	298	180819	100000748	0	0	2858553	1291109816	0	1293968369	9592	7140	0	6391	19703	3415	24756	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.2238221701993739	0.5202106925357641	0.2559671372648619	819	1024
        BSBM_284826.spo.POS	BTree	1024	2	288	182371	100000748	0	0	3017187	793789115	0	796806302	10476	4352	0	6628	21834	2527	17386	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.28471632933499036	0.4150082941437323	0.260633201758468	0.03964217476280939	1090	1024
        BSBM_284826.spo.SPO	BTree	1024	2	381	195312	100000748	0	1	3464020	1403679601	0	1407143621	9091	7186	0	7330	13898	4721	11511	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.9911953927835947	0.008804607216405289	835	1024
        __globalRowStore	BTree	32	1	1	4	95	0	0	174	6357	0	6531	174	1589	0	174	174	1003	2160	0.0	0.0	0.2	0.0	0.0	0.0	0.2	0.4	0.2	0.0	0.0	0.0	668	32
        
        Build Version=N/A
        
        Accepted query count=510161
        
        Running query count=0
        
        Show queries, query details.
        
        /blockedWorkQueueCount=0 /blockedWorkQueueRunningTotal=0 /deadlineQueueSize=0 /operatorActiveCount=0 /operatorHaltCount=0 /operatorStartCount=0 /operatorTasksPerQuery=0.0 /queriesPerSecond=0.0 /queryDoneCount=0 /queryErrorCount=0 /queryStartCount=0
        
        [java] ERROR: AbstractBTree.java:1602: Error reading child[i=50]: java.lang.RuntimeException: addr=-5588677 : cause=java.lang.IllegalStateException: Error reading from WriteCache addr: 2307604480 length: 6165, writeCacheDebug: No WriteCache debug info
             [java] java.lang.RuntimeException: addr=-5588677 : cause=java.lang.IllegalStateException: Error reading from WriteCache addr: 2307604480 length: 6165, writeCacheDebug: No WriteCache debug info
             [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:2190)
             [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:2052)
             [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:1941)
             [java] 	at com.bigdata.journal.RWStrategy.readFromLocalStore(RWStrategy.java:733)
             [java] 	at com.bigdata.journal.RWStrategy.read(RWStrategy.java:156)
             [java] 	at com.bigdata.journal.AbstractJournal.read(AbstractJournal.java:4284)
             [java] 	at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:4018)
             [java] 	at com.bigdata.btree.Node._getChild(Node.java:2727)
             [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:363)
             [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:346)
             [java] 	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
             [java] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
             [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
             [java] 	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
             [java] 	at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:532)
             [java] 	at com.bigdata.btree.Node.getChild(Node.java:2628)
             [java] 	at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1585)
             [java] 	at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1588)
             [java] 	at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1563)
             [java] 	at com.bigdata.journal.DumpJournal.dumpNamedIndicesMetadata(DumpJournal.java:763)
             [java] 	at com.bigdata.journal.DumpJournal.dumpJournal(DumpJournal.java:616)
             [java] 	at com.bigdata.rdf.sail.webapp.StatusServlet.doGet(StatusServlet.java:529)
             [java] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
             [java] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
             [java] 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
             [java] 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
             [java] 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
             [java] 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
             [java] 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
             [java] 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
             [java] 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
             [java] 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
             [java] 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
             [java] 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
             [java] 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
             [java] 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
             [java] 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
             [java] 	at org.eclipse.jetty.server.Server.handle(Server.java:497)
             [java] 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
             [java] 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
             [java] 	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
             [java] 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
             [java] 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
             [java] 	at java.lang.Thread.run(Thread.java:724)
             [java] Caused by: java.lang.IllegalStateException: Error reading from WriteCache addr: 2307604480 length: 6165, writeCacheDebug: No WriteCache debug info
             [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:2108)
             [java] 	... 43 more
             [java] Caused by: com.bigdata.util.ChecksumError: offset=2307604480,nbytes=6169,expected=-212098888,actual=-2072665828
             [java] 	at com.bigdata.io.writecache.WriteCacheService._readFromLocalDiskIntoNewHeapByteBuffer(WriteCacheService.java:3711)
             [java] 	at com.bigdata.io.writecache.WriteCacheService._getRecord(WriteCacheService.java:3526)
             [java] 	at com.bigdata.io.writecache.WriteCacheService.access$2500(WriteCacheService.java:200)
             [java] 	at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3363)
             [java] 	at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3347)
             [java] 	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
             [java] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
             [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
             [java] 	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
             [java] 	at com.bigdata.io.writecache.WriteCacheService.loadRecord(WriteCacheService.java:3468)
             [java] 	at com.bigdata.io.writecache.WriteCacheService.read(WriteCacheService.java:3187)
             [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:2106)
             [java] 	... 43 more
        
             [java] ERROR: AbstractBTree.java:1602: Error reading child[i=185]: java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620
             [java] java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620
             [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:1966)
             [java] 	at com.bigdata.journal.RWStrategy.readFromLocalStore(RWStrategy.java:733)
             [java] 	at com.bigdata.journal.RWStrategy.read(RWStrategy.java:156)
             [java] 	at com.bigdata.journal.AbstractJournal.read(AbstractJournal.java:4284)
             [java] 	at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:4018)
             [java] 	at com.bigdata.btree.Node._getChild(Node.java:2727)
             [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:363)
             [java] 	at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:346)
             [java] 	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
             [java] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
             [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
             [java] 	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
             [java] 	at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:532)
             [java] 	at com.bigdata.btree.Node.getChild(Node.java:2628)
             [java] 	at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1585)
             [java] 	at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1588)
             [java] 	at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1563)
             [java] 	at com.bigdata.journal.DumpJournal.dumpNamedIndicesMetadata(DumpJournal.java:763)
             [java] 	at com.bigdata.journal.DumpJournal.dumpJournal(DumpJournal.java:616)
             [java] 	at com.bigdata.rdf.sail.webapp.StatusServlet.doGet(StatusServlet.java:529)
             [java] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
             [java] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
             [java] 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
             [java] 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
             [java] 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
             [java] 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
             [java] 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
             [java] 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
             [java] 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
             [java] 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
             [java] 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
             [java] 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
             [java] 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
             [java] 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
             [java] 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
             [java] 	at org.eclipse.jetty.server.Server.handle(Server.java:497)
             [java] 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
             [java] 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
             [java] 	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
             [java] 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
             [java] 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
             [java] 	at java.lang.Thread.run(Thread.java:724)
             [java] Caused by: com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620
             [java] 	at com.bigdata.io.writecache.WriteCacheService._readFromLocalDiskIntoNewHeapByteBuffer(WriteCacheService.java:3711)
             [java] 	at com.bigdata.io.writecache.WriteCacheService._getRecord(WriteCacheService.java:3526)
             [java] 	at com.bigdata.io.writecache.WriteCacheService.access$2500(WriteCacheService.java:200)
             [java] 	at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3363)
             [java] 	at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3347)
             [java] 	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
             [java] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
             [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
             [java] 	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
             [java] 	at com.bigdata.io.writecache.WriteCacheService.loadRecord(WriteCacheService.java:3468)
             [java] 	at com.bigdata.io.writecache.WriteCacheService.read(WriteCacheService.java:3187)
             [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:1957)
             [java] 	... 41 more
        
        Show
        bryanthompson bryanthompson added a comment - Running DumpJournal as http://bigdata11/bigdata/status?dumpJournal&dumpPages}}} produces the output below and also causes errors to be reported on the server console (see below for those traces) {{{ Dump Journal Running... magic=e6b4c275 version=1 extent=13575716864(12946M), userExtent=13575716176(12946M), bytesAvailable=-874008000174780(-833518982M), nextOffset=874021575890956 rootBlock{ rootBlock=0, challisField=2, version=3, nextOffset=874021575890956, localTime=1425829363023 [Sunday, March 8, 2015 11:42:43 AM EDT], firstCommitTime=1425827057089 [Sunday, March 8, 2015 11:04:17 AM EDT], lastCommitTime=1425829361645 [Sunday, March 8, 2015 11:42:41 AM EDT], commitCounter=2, commitRecordAddr={off=NATIVE:-25440860,len=422}, commitRecordIndexAddr={off=NATIVE:-2431970,len=220}, blockSequence=3131, quorumToken=-1, metaBitsAddr=846224974413979, metaStartAddr=207149, storeType=RW, uuid=75711495-da41-47d3-af28-98245135b3eb, offsetBits=42, checksum=-1268307968, createTime=1425827056650 [Sunday, March 8, 2015 11:04:16 AM EDT], closeTime=0} rootBlock{ rootBlock=1, challisField=1, version=3, nextOffset=206158438413, localTime=1425827057129 [Sunday, March 8, 2015 11:04:17 AM EDT], firstCommitTime=1425827057089 [Sunday, March 8, 2015 11:04:17 AM EDT], lastCommitTime=1425827057089 [Sunday, March 8, 2015 11:04:17 AM EDT], commitCounter=1, commitRecordAddr={off=NATIVE:-57348,len=422}, commitRecordIndexAddr={off=NATIVE:-8195,len=220}, blockSequence=1, quorumToken=-1, metaBitsAddr=56044290095, metaStartAddr=3200, storeType=RW, uuid=75711495-da41-47d3-af28-98245135b3eb, offsetBits=42, checksum=-1550833551, createTime=1425827056650 [Sunday, March 8, 2015 11:04:16 AM EDT], closeTime=0} The current root block is #0 ------------------------- RWStore Allocator Summary ------------------------- AllocatorSize AllocatorCount SlotsAllocated %SlotsAllocated SlotsRecycled SlotChurn SlotsInUse %SlotsInUse MeanAllocation SlotsReserved %SlotsUnused BytesReserved BytesAppData %SlotWaste %AppData %StoreFile %TotalWaste %FileWaste 64 154 2326224 8.23 1229776 2.12 1096448 4.70 29 1097728 0.12 70254592 47009299 33.09 0.44 0.53 0.85 0.17 128 2347 16828366 59.54 19045 1.00 16809321 72.08 96 16821760 0.07 2153185280 1625746945 24.50 15.35 16.15 19.24 3.96 192 188 1355532 4.80 18650 1.01 1336882 5.73 151 1343488 0.49 257949696 203087661 21.27 1.92 1.94 2.00 0.41 320 15 143897 0.51 37321 1.35 106576 0.46 225 107520 0.88 34406400 22974189 33.23 0.22 0.26 0.42 0.09 512 11 128264 0.45 53329 1.71 74935 0.32 438 76544 2.10 39190528 34355788 12.34 0.32 0.29 0.18 0.04 768 79 636002 2.25 71528 1.13 564474 2.42 640 566272 0.32 434896896 362266536 16.70 3.42 3.26 2.65 0.54 1024 80 635659 2.25 68427 1.12 567232 2.43 896 569600 0.42 583270400 508730102 12.78 4.80 4.38 2.72 0.56 2048 277 2207412 7.81 228122 1.12 1979290 8.49 1489 1984256 0.25 4063756288 2944976131 27.53 27.81 30.49 40.80 8.39 3072 9 232421 0.82 171522 3.82 60899 0.26 2616 64512 5.60 198180864 173514981 12.45 1.64 1.49 0.90 0.19 4096 20 313587 1.11 178358 2.32 135229 0.58 3609 141312 4.30 578813952 483407345 16.48 4.57 4.34 3.48 0.72 8192 84 3455567 12.23 2865907 5.86 589660 2.53 7271 600064 1.73 4915724288 4181683691 14.93 39.50 36.88 26.77 5.51 ------------------------- BLOBS ------------------------- Bucket(K) Allocations Allocated Deletes Deleted Current Data Mean Churn 16 1131836 12783813674 1044103 11700905982 87733 1082907692 11294 12.90 32 83310 1582810615 78646 1488844103 4664 93966512 18999 17.86 64 87051 4060621662 79582 3692942259 7469 367679403 46646 11.65 128 9951 700568541 9137 639776806 814 60791735 70401 12.22 256 0 0 0 0 0 0 0 0.00 512 0 0 0 0 0 0 0 0.00 1024 0 0 0 0 0 0 0 0.00 2048 0 0 0 0 0 0 0 0.00 4096 0 0 0 0 0 0 0 0.00 8192 0 0 0 0 0 0 0 0.00 16384 0 0 0 0 0 0 0 0.00 32768 0 0 0 0 0 0 0 0.00 65536 0 0 0 0 0 0 0 0.00 2097151 0 0 0 0 0 0 0 0.00 There are 2 commit points. CommitRecord{timestamp=1425829361645, commitCounter=2, roots=[-10445227319885604, -109267657387146924, -114775102540742596, -175939040313213, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]} name=__globalRowStore Checkpoint{indexType=BTree,height=1,nnodes=1,nleaves=4,nentries=95,counter=0,addrRoot=-175934745345874,addrMetadata=-70385924045614,addrBloomFilter=0,addrCheckpoint=-35205846925092} addrMetadata=0, name=__globalRowStore, indexType=BTree, indexUUID=2001afcb-ccdd-491c-b66e-34812a763bd0, branchingFactor=32, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@6e16af5a{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.btree.DefaultTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.DefaultKeyBuilderFactory{ initialCapacity=0, collator=ICU, locale=en_US, strength=null, decomposition=null}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@1352762d{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.CanonicalHuffmanRabaCoder@acc8e44}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=com.bigdata.sparse.LogicalRowSplitHandler@7b9017e5, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0} com.bigdata.btree.BTreePageStats{indexType=BTree,m=32,nnodes=1,nleaves=4,nrawRecs=0,nodeBytes=174,minNodeBytes=174,maxNodeBytes=174,leafBytes=6357,minLeafBytes=1003,maxLeafBytes=2160,rawRecBytes=0,bytesPerNode=174,bytesPerLeaf=1589,bytesPerRawRec=0,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.2,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.2,slot_2048=0.4,slot_3072=0.2,slot_4096=0.0,slot_8192=0.0,blobs=0.0,newM=668} name=BSBM_284826.lex.BLOBS Checkpoint{indexType=BTree,height=2,nnodes=41,nleaves=10576,nentries=3179117,counter=0,addrRoot=-112194102073752973,addrMetadata=-21474835633,addrBloomFilter=0,addrCheckpoint=-10445132830605092} addrMetadata=0, name=BSBM_284826.lex.BLOBS, indexType=BTree, indexUUID=ca18bce6-8e17-41cf-84eb-85e9ef4a538f, branchingFactor=400, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@3110fac3{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.lexicon.BlobsTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=8}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@51bf315f{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.SimpleRabaCoder@5f427de6}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=true, maxRecLen=0, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0} com.bigdata.btree.BTreePageStats{indexType=BTree,m=400,nnodes=41,nleaves=10576,nrawRecs=3179117,nodeBytes=141846,minNodeBytes=627,maxNodeBytes=5244,leafBytes=59463561,minLeafBytes=3743,maxLeafBytes=7480,rawRecBytes=3820125292,bytesPerNode=3459,bytesPerLeaf=5622,bytesPerRawRec=1201,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.14,slot_8192=0.85,blobs=0.0,newM=611} name=BSBM_284826.lex.ID2TERM Checkpoint{indexType=BTree,height=2,nnodes=120,nleaves=48304,nentries=19322065,counter=0,addrRoot=-113739405537048709,addrMetadata=-17179868333,addrBloomFilter=0,addrCheckpoint=-10443706901462820} addrMetadata=0, name=BSBM_284826.lex.ID2TERM, indexType=BTree, indexUUID=44cc03a0-1021-495f-8a6b-c56e2491fbfb, branchingFactor=800, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@657b7e6{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.lexicon.Id2TermTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=9}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@68ea604a{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.SimpleRabaCoder@7357d015}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=true, maxRecLen=16, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0} com.bigdata.btree.BTreePageStats{indexType=BTree,m=800,nnodes=120,nleaves=48304,nrawRecs=19231729,nodeBytes=623535,minNodeBytes=1915,maxNodeBytes=10005,leafBytes=314000524,minLeafBytes=6489,maxLeafBytes=12155,rawRecBytes=1815706974,bytesPerNode=5196,bytesPerLeaf=6500,bytesPerRawRec=94,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.0,slot_8192=0.99,blobs=0.0,newM=907} name=BSBM_284826.lex.TERM2ID Checkpoint{indexType=BTree,height=2,nnodes=99,nleaves=45177,nentries=19322065,counter=19322065,addrRoot=-114687270459535695,addrMetadata=-70390219012891,addrBloomFilter=0,addrCheckpoint=-10445218729951012} addrMetadata=0, name=BSBM_284826.lex.TERM2ID, indexType=BTree, indexUUID=7d1fb58d-43f1-47b1-89dd-d56ca3807bfb, branchingFactor=800, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@4b89abb4{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.lexicon.Term2IdTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.DefaultKeyBuilderFactory{ initialCapacity=0, collator=ICU, locale=en_US, strength=null, decomposition=null}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@3b67dd2a{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.FixedLengthValueRabaCoder@1fe90ede}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0} com.bigdata.btree.BTreePageStats{indexType=BTree,m=800,nnodes=99,nleaves=45176,nrawRecs=0,nodeBytes=1196198,minNodeBytes=3761,maxNodeBytes=20854,leafBytes=968966798,minLeafBytes=10025,maxLeafBytes=86966,rawRecBytes=0,bytesPerNode=12082,bytesPerLeaf=21448,bytesPerRawRec=0,nerrors=1,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.0,slot_8192=0.0,blobs=0.99,newM=338} name=BSBM_284826.spo.JUST Checkpoint{indexType=BTree,height=0,nnodes=0,nleaves=1,nentries=0,counter=0,addrRoot=-105596065939426,addrMetadata=-38654704873,addrBloomFilter=0,addrCheckpoint=-35235911696164} addrMetadata=0, name=BSBM_284826.spo.JUST, indexType=BTree, indexUUID=98a1cb98-9369-4067-8858-f27dc7a6eada, branchingFactor=128, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@5c56efb7{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.JustificationTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@4f9406bd{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.EmptyRabaValueCoder@7f468b1b}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0} com.bigdata.btree.BTreePageStats{indexType=BTree,m=128,nnodes=0,nleaves=1,nrawRecs=0,nodeBytes=0,minNodeBytes=0,maxNodeBytes=0,leafBytes=30,minLeafBytes=30,maxLeafBytes=30,rawRecBytes=0,bytesPerNode=0,bytesPerLeaf=30,bytesPerRawRec=0,nerrors=0,slot_64=1.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.0,slot_8192=0.0,blobs=0.0,newM=128} name=BSBM_284826.spo.OSP Checkpoint{indexType=BTree,height=2,nnodes=298,nleaves=180819,nentries=100000748,counter=0,addrRoot=-114812859598235401,addrMetadata=-30064770301,addrBloomFilter=0,addrCheckpoint=-10445197255114532} addrMetadata=0, name=BSBM_284826.spo.OSP, indexType=BTree, indexUUID=ff4127fa-de79-4880-8ca3-d9dfcd12f592, branchingFactor=1024, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@563ce85a{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.SPOTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@38ba98f1{ratio=8}, leafValuesCoder=com.bigdata.rdf.spo.FastRDFValueCoder2@628f134f}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0} com.bigdata.btree.BTreePageStats{indexType=BTree,m=1024,nnodes=298,nleaves=180819,nrawRecs=0,nodeBytes=2858553,minNodeBytes=6391,maxNodeBytes=19703,leafBytes=1291109816,minLeafBytes=3415,maxLeafBytes=24756,rawRecBytes=0,bytesPerNode=9592,bytesPerLeaf=7140,bytesPerRawRec=0,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.22,slot_8192=0.52,blobs=0.25,newM=819} name=BSBM_284826.spo.POS Checkpoint{indexType=BTree,height=2,nnodes=288,nleaves=182371,nentries=100000748,counter=0,addrRoot=-114812348497126940,addrMetadata=-34359737597,addrBloomFilter=0,addrCheckpoint=-10445192960147236} addrMetadata=0, name=BSBM_284826.spo.POS, indexType=BTree, indexUUID=a1d2ce58-7dd5-4e6e-8429-4f0274f81441, branchingFactor=1024, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@a8a1279{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.SPOTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@6ec2cb56{ratio=8}, leafValuesCoder=com.bigdata.rdf.spo.FastRDFValueCoder2@36b3b3e9}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0} com.bigdata.btree.BTreePageStats{indexType=BTree,m=1024,nnodes=288,nleaves=182371,nrawRecs=0,nodeBytes=3017187,minNodeBytes=6628,maxNodeBytes=21834,leafBytes=793789115,minLeafBytes=2527,maxLeafBytes=17386,rawRecBytes=0,bytesPerNode=10476,bytesPerLeaf=4352,bytesPerRawRec=0,nerrors=0,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.28,slot_4096=0.41,slot_8192=0.26,blobs=0.03,newM=1090} name=BSBM_284826.spo.SPO Checkpoint{indexType=BTree,height=2,nnodes=381,nleaves=195313,nentries=100000748,counter=0,addrRoot=-114828394494943415,addrMetadata=-25769802907,addrBloomFilter=0,addrCheckpoint=-10445223024918308} addrMetadata=0, name=BSBM_284826.spo.SPO, indexType=BTree, indexUUID=53f9b734-dfcc-48f7-9ffc-fda4251c28cd, branchingFactor=1024, pmd=null, btreeClassName=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@2da77f97{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.SPOTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@4fa9a9eb{ratio=8}, leafValuesCoder=com.bigdata.rdf.spo.FastRDFValueCoder2@bd2ac29}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, maxRecLen=256, bloomFilterFactory=BloomFilterFactory{ n=1000000, p=0.02, maxP=0.15, maxN=1883227}, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0} com.bigdata.btree.BTreePageStats{indexType=BTree,m=1024,nnodes=381,nleaves=195312,nrawRecs=0,nodeBytes=3464020,minNodeBytes=7330,maxNodeBytes=13898,leafBytes=1403679601,minLeafBytes=4721,maxLeafBytes=11511,rawRecBytes=0,bytesPerNode=9091,bytesPerLeaf=7186,bytesPerRawRec=0,nerrors=1,slot_64=0.0,slot_128=0.0,slot_192=0.0,slot_320=0.0,slot_512=0.0,slot_768=0.0,slot_1024=0.0,slot_2048=0.0,slot_3072=0.0,slot_4096=0.0,slot_8192=0.99,blobs=0.0,newM=835} name indexType m height nnodes nleaves nentries nrawRecs nerrors nodeBytes leafBytes rawRecBytes totalBytes avgNodeBytes avgLeafBytes avgRawRecBytes minNodeBytes maxNodeBytes minLeafBytes maxLeafBytes 64 128 192 320 512 768 1024 2048 3072 4096 8192 blobs newM curM BSBM_284826.lex.BLOBS BTree 400 2 41 10576 3179117 3179117 0 141846 59463561 3820125292 3879730699 3459 5622 1201 627 5244 3743 7480 0.0 0.0 0.0 0.0 0.0 9.418856550814731E-5 0.0 0.0 0.0016012056136385042 0.14288405387585948 0.8554205519449939 0.0 611 400 BSBM_284826.lex.ID2TERM BTree 800 2 120 48304 19322065 19231729 0 623535 314000524 1815706974 2130331033 5196 6500 94 1915 10005 6489 12155 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.065091690071039E-5 0.0 0.0 0.9998967454154964 8.260366760284156E-5 907 800 BSBM_284826.lex.TERM2ID BTree 800 2 99 45176 19322065 0 1 1196198 968966798 0 970162996 12082 21448 0 3761 20854 10025 86966 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.2087244616234126E-5 0.0 0.9999779127553837 338 800 BSBM_284826.spo.JUST BTree 128 0 0 1 0 0 0 0 30 0 30 0 30 0 0 0 30 30 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 128 128 BSBM_284826.spo.OSP BTree 1024 2 298 180819 100000748 0 0 2858553 1291109816 0 1293968369 9592 7140 0 6391 19703 3415 24756 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2238221701993739 0.5202106925357641 0.2559671372648619 819 1024 BSBM_284826.spo.POS BTree 1024 2 288 182371 100000748 0 0 3017187 793789115 0 796806302 10476 4352 0 6628 21834 2527 17386 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.28471632933499036 0.4150082941437323 0.260633201758468 0.03964217476280939 1090 1024 BSBM_284826.spo.SPO BTree 1024 2 381 195312 100000748 0 1 3464020 1403679601 0 1407143621 9091 7186 0 7330 13898 4721 11511 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.9911953927835947 0.008804607216405289 835 1024 __globalRowStore BTree 32 1 1 4 95 0 0 174 6357 0 6531 174 1589 0 174 174 1003 2160 0.0 0.0 0.2 0.0 0.0 0.0 0.2 0.4 0.2 0.0 0.0 0.0 668 32 Build Version=N/A Accepted query count=510161 Running query count=0 Show queries, query details. /blockedWorkQueueCount=0 /blockedWorkQueueRunningTotal=0 /deadlineQueueSize=0 /operatorActiveCount=0 /operatorHaltCount=0 /operatorStartCount=0 /operatorTasksPerQuery=0.0 /queriesPerSecond=0.0 /queryDoneCount=0 /queryErrorCount=0 /queryStartCount=0 [java] ERROR: AbstractBTree.java:1602: Error reading child[i=50]: java.lang.RuntimeException: addr=-5588677 : cause=java.lang.IllegalStateException: Error reading from WriteCache addr: 2307604480 length: 6165, writeCacheDebug: No WriteCache debug info [java] java.lang.RuntimeException: addr=-5588677 : cause=java.lang.IllegalStateException: Error reading from WriteCache addr: 2307604480 length: 6165, writeCacheDebug: No WriteCache debug info [java] at com.bigdata.rwstore.RWStore.getData(RWStore.java:2190) [java] at com.bigdata.rwstore.RWStore.getData(RWStore.java:2052) [java] at com.bigdata.rwstore.RWStore.getData(RWStore.java:1941) [java] at com.bigdata.journal.RWStrategy.readFromLocalStore(RWStrategy.java:733) [java] at com.bigdata.journal.RWStrategy.read(RWStrategy.java:156) [java] at com.bigdata.journal.AbstractJournal.read(AbstractJournal.java:4284) [java] at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:4018) [java] at com.bigdata.btree.Node._getChild(Node.java:2727) [java] at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:363) [java] at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:346) [java] at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77) [java] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [java] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [java] at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92) [java] at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:532) [java] at com.bigdata.btree.Node.getChild(Node.java:2628) [java] at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1585) [java] at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1588) [java] at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1563) [java] at com.bigdata.journal.DumpJournal.dumpNamedIndicesMetadata(DumpJournal.java:763) [java] at com.bigdata.journal.DumpJournal.dumpJournal(DumpJournal.java:616) [java] at com.bigdata.rdf.sail.webapp.StatusServlet.doGet(StatusServlet.java:529) [java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [java] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769) [java] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [java] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [java] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [java] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [java] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [java] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [java] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [java] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [java] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [java] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) [java] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) [java] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [java] at org.eclipse.jetty.server.Server.handle(Server.java:497) [java] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [java] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [java] at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [java] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610) [java] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539) [java] at java.lang.Thread.run(Thread.java:724) [java] Caused by: java.lang.IllegalStateException: Error reading from WriteCache addr: 2307604480 length: 6165, writeCacheDebug: No WriteCache debug info [java] at com.bigdata.rwstore.RWStore.getData(RWStore.java:2108) [java] ... 43 more [java] Caused by: com.bigdata.util.ChecksumError: offset=2307604480,nbytes=6169,expected=-212098888,actual=-2072665828 [java] at com.bigdata.io.writecache.WriteCacheService._readFromLocalDiskIntoNewHeapByteBuffer(WriteCacheService.java:3711) [java] at com.bigdata.io.writecache.WriteCacheService._getRecord(WriteCacheService.java:3526) [java] at com.bigdata.io.writecache.WriteCacheService.access$2500(WriteCacheService.java:200) [java] at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3363) [java] at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3347) [java] at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77) [java] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [java] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [java] at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92) [java] at com.bigdata.io.writecache.WriteCacheService.loadRecord(WriteCacheService.java:3468) [java] at com.bigdata.io.writecache.WriteCacheService.read(WriteCacheService.java:3187) [java] at com.bigdata.rwstore.RWStore.getData(RWStore.java:2106) [java] ... 43 more [java] ERROR: AbstractBTree.java:1602: Error reading child[i=185]: java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620 [java] java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620 [java] at com.bigdata.rwstore.RWStore.getData(RWStore.java:1966) [java] at com.bigdata.journal.RWStrategy.readFromLocalStore(RWStrategy.java:733) [java] at com.bigdata.journal.RWStrategy.read(RWStrategy.java:156) [java] at com.bigdata.journal.AbstractJournal.read(AbstractJournal.java:4284) [java] at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:4018) [java] at com.bigdata.btree.Node._getChild(Node.java:2727) [java] at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:363) [java] at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:346) [java] at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77) [java] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [java] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [java] at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92) [java] at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:532) [java] at com.bigdata.btree.Node.getChild(Node.java:2628) [java] at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1585) [java] at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1588) [java] at com.bigdata.btree.AbstractBTree.dumpPages(AbstractBTree.java:1563) [java] at com.bigdata.journal.DumpJournal.dumpNamedIndicesMetadata(DumpJournal.java:763) [java] at com.bigdata.journal.DumpJournal.dumpJournal(DumpJournal.java:616) [java] at com.bigdata.rdf.sail.webapp.StatusServlet.doGet(StatusServlet.java:529) [java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [java] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769) [java] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [java] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [java] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [java] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [java] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [java] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [java] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [java] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [java] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [java] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) [java] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) [java] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [java] at org.eclipse.jetty.server.Server.handle(Server.java:497) [java] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [java] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [java] at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [java] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610) [java] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539) [java] at java.lang.Thread.run(Thread.java:724) [java] Caused by: com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620 [java] at com.bigdata.io.writecache.WriteCacheService._readFromLocalDiskIntoNewHeapByteBuffer(WriteCacheService.java:3711) [java] at com.bigdata.io.writecache.WriteCacheService._getRecord(WriteCacheService.java:3526) [java] at com.bigdata.io.writecache.WriteCacheService.access$2500(WriteCacheService.java:200) [java] at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3363) [java] at com.bigdata.io.writecache.WriteCacheService$1.compute(WriteCacheService.java:3347) [java] at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77) [java] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [java] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [java] at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92) [java] at com.bigdata.io.writecache.WriteCacheService.loadRecord(WriteCacheService.java:3468) [java] at com.bigdata.io.writecache.WriteCacheService.read(WriteCacheService.java:3187) [java] at com.bigdata.rwstore.RWStore.getData(RWStore.java:1957) [java] ... 41 more
        Hide
        bryanthompson bryanthompson added a comment -

        The first of those checksum errors appeared in both the dumpJournal and the query error reports on the console.

             [java] Caused by: java.lang.IllegalStateException: Error reading from WriteCache addr: 2307604480 length: 6165, writeCacheDebug: No WriteCache debug info
             [java] 	at com.bigdata.rwstore.RWStore.getData(RWStore.java:2108)
             [java] 	... 43 more
             [java] Caused by: com.bigdata.util.ChecksumError: offset=2307604480,nbytes=6169,expected=-212098888,actual=-2072665828
        

        The second one only appeared (within the retained data on the console) in the dumpJournal It could be that we hit this problem in queries whose errors had disappeared from the console due to limited buffering.

             [java] ERROR: AbstractBTree.java:1602: Error reading child[i=185]: java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620
             [java] java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620
        
        Show
        bryanthompson bryanthompson added a comment - The first of those checksum errors appeared in both the dumpJournal and the query error reports on the console. [java] Caused by: java.lang.IllegalStateException: Error reading from WriteCache addr: 2307604480 length: 6165, writeCacheDebug: No WriteCache debug info [java] at com.bigdata.rwstore.RWStore.getData(RWStore.java:2108) [java] ... 43 more [java] Caused by: com.bigdata.util.ChecksumError: offset=2307604480,nbytes=6169,expected=-212098888,actual=-2072665828 The second one only appeared (within the retained data on the console) in the dumpJournal It could be that we hit this problem in queries whose errors had disappeared from the console due to limited buffering. [java] ERROR: AbstractBTree.java:1602: Error reading child[i=185]: java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620 [java] java.lang.RuntimeException: addr=-5588679 : cause=com.bigdata.util.ChecksumError: offset=2307620864,nbytes=7726,expected=1208212629,actual=-919862620
        Hide
        bryanthompson bryanthompson added a comment -

        The original load was performed using "ant run-load" in a separate JVM from the "ant start-sparql-server". Therefore the write cache was not shared across the load and the query. Further, I then restarted the server process and observe the same address errors being reported by DumpJournal so the problem is durable.

        Show
        bryanthompson bryanthompson added a comment - The original load was performed using "ant run-load" in a separate JVM from the "ant start-sparql-server". Therefore the write cache was not shared across the load and the query. Further, I then restarted the server process and observe the same address errors being reported by DumpJournal so the problem is durable.
        Hide
        bryanthompson bryanthompson added a comment -

        I have made a compressed archive of the journal file:

        gzip -c /data/bsbm/bsbm3_284826/bigdata-bsbm.RW.jnl > /fast/ticket-1149.jnl.gz
        
        Show
        bryanthompson bryanthompson added a comment - I have made a compressed archive of the journal file: gzip -c /data/bsbm/bsbm3_284826/bigdata-bsbm.RW.jnl > /fast/ticket-1149.jnl.gz
        Hide
        bryanthompson bryanthompson added a comment -

        Assigned to Martyn for further analysis. The compressed file is 3.5GB. Since this was not an HA deployment, there are no transaction logs available.

        Note: I have not been able to figure out how to obtain a bad blocks scan on the ext4 file system /dev/sda. This makes it impossible to rule out a bad disk that corrupted the write. However, I was able to compress the file onto another disk without errors so the file system itself is not reporting errors during reads.

        Show
        bryanthompson bryanthompson added a comment - Assigned to Martyn for further analysis. The compressed file is 3.5GB. Since this was not an HA deployment, there are no transaction logs available. Note: I have not been able to figure out how to obtain a bad blocks scan on the ext4 file system /dev/sda. This makes it impossible to rule out a bad disk that corrupted the write. However, I was able to compress the file onto another disk without errors so the file system itself is not reporting errors during reads.
        Hide
        martyncutcher martyncutcher added a comment -

        Since looking into ticket 1136. I am now aware of the possibility of multiple Allocators mapping to the same file region. If this does happen then checksum errors are guaranteed.

        I will amend the showAllocators code to check for any region conflict.

        Show
        martyncutcher martyncutcher added a comment - Since looking into ticket 1136. I am now aware of the possibility of multiple Allocators mapping to the same file region. If this does happen then checksum errors are guaranteed. I will amend the showAllocators code to check for any region conflict.
        Hide
        martyncutcher martyncutcher added a comment -

        I have added this check to the GROUP_COMMIT_1136 branch. This branch could be used to read the problem store and confirm whether the error is due to a region conflict.

        Show
        martyncutcher martyncutcher added a comment - I have added this check to the GROUP_COMMIT_1136 branch. This branch could be used to read the problem store and confirm whether the error is due to a region conflict.
        Hide
        bryanthompson bryanthompson added a comment -

        I am seeing more odd behavior on this machine (seg fault of the JVM).

        I have checked the allocators with some new code per Martyn. They do not have overlapping regions.

        I am going to close this for new as likely being a machine failure.

        Show
        bryanthompson bryanthompson added a comment - I am seeing more odd behavior on this machine (seg fault of the JVM). I have checked the allocators with some new code per Martyn. They do not have overlapping regions. I am going to close this for new as likely being a machine failure.

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: