Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-602

PhysicalAddressResolutionException against 1.0.6

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Affects Version/s: BIGDATA_RELEASE_1_0_6
    • Fix Version/s: None
    • Component/s: RWStore

      Description

      java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1357207
              at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1481) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1658) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1521) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.striterator.AbstractChunkedResolverator._hasNext(AbstractChunkedResolverator.java:327) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.striterator.AbstractChunkedResolverator.hasNext(AbstractChunkedResolverator.java:303) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.rdf.sail.Bigdata2Sesame2BindingSetIterator.hasNext(Bigdata2Sesame2BindingSetIterator.java:70) ~[bigdata-1.0.6.jar:na]
              at info.aduna.iteration.ConvertingIteration.hasNext(ConvertingIteration.java:62) ~[openrdf-sesame-onejar-2.3.0.jar:na]
              at info.aduna.iteration.IterationWrapper.hasNext(IterationWrapper.java:57) ~[openrdf-sesame-onejar-2.3.0.jar:na]
              at info.aduna.iteration.FilterIteration.findNextElement(FilterIteration.java:68) ~[openrdf-sesame-onejar-2.3.0.jar:na]
              at info.aduna.iteration.FilterIteration.hasNext(FilterIteration.java:43) ~[openrdf-sesame-onejar-2.3.0.jar:na]
              at org.openrdf.query.impl.TupleQueryResultImpl.hasNext(TupleQueryResultImpl.java:90) ~[openrdf-sesame-onejar-2.3.0.jar:na]
      	...
      Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1357207
              at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) ~[na:1.6.0_29]
              at java.util.concurrent.FutureTask.get(FutureTask.java:83) ~[na:1.6.0_29]
              at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1434) ~[bigdata-1.0.6.jar:na]
              ... 52 common frames omitted
      Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1357207
              at com.bigdata.rdf.sail.RunningQueryCloseableIterator.checkFuture(RunningQueryCloseableIterator.java:58) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.rdf.sail.RunningQueryCloseableIterator.close(RunningQueryCloseableIterator.java:72) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.rdf.sail.RunningQueryCloseableIterator.hasNext(RunningQueryCloseableIterator.java:81) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.striterator.ChunkedWrappedIterator.hasNext(ChunkedWrappedIterator.java:193) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:215) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:191) ~[bigdata-1.0.6.jar:na]
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) ~[na:1.6.0_29]
              at java.util.concurrent.FutureTask.run(FutureTask.java:138) ~[na:1.6.0_29]
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_29]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_29]
              ... 1 common frames omitted
      Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1357207
              at com.bigdata.util.concurrent.Haltable.get(Haltable.java:251) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.engine.AbstractRunningQuery.get(AbstractRunningQuery.java:1185) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.engine.AbstractRunningQuery.get(AbstractRunningQuery.java:88) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.rdf.sail.RunningQueryCloseableIterator.checkFuture(RunningQueryCloseableIterator.java:45) ~[bigdata-1.0.6.jar:na]
              ... 10 common frames omitted
      Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1357207
              at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) ~[na:1.6.0_29]
              at java.util.concurrent.FutureTask.get(FutureTask.java:83) ~[na:1.6.0_29]
              at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1130) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:728) ~[bigdata-1.0.6.jar:na]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) ~[na:1.6.0_29]
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) ~[na:1.6.0_29]
              at java.util.concurrent.FutureTask.run(FutureTask.java:138) ~[na:1.6.0_29]
              at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:641) ~[bigdata-1.0.6.jar:na]
              ... 3 common frames omitted
      Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1357207
              at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:810) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:523) ~[bigdata-1.0.6.jar:na]
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) ~[na:1.6.0_29]
              at java.util.concurrent.FutureTask.run(FutureTask.java:138) ~[na:1.6.0_29]
              at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1129) ~[bigdata-1.0.6.jar:na]
              ... 9 common frames omitted
      Caused by: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1357207
              at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:1102) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.join.PipelineJoin$JoinTask.consumeSource(PipelineJoin.java:867) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.join.PipelineJoin$JoinTask.call(PipelineJoin.java:751) ~[bigdata-1.0.6.jar:na]
              ... 14 common frames omitted
      Caused by: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1357207
              at com.bigdata.rwstore.RWStore.getData(RWStore.java:1487) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.journal.RWStrategy.readFromLocalStore(RWStrategy.java:624) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.journal.RWStrategy.read(RWStrategy.java:135) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.journal.AbstractJournal.read(AbstractJournal.java:2738) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:3824) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.Node._getChild(Node.java:2723) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:340) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:325) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:79) ~[bigdata-1.0.6.jar:na]
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) ~[na:1.6.0_29]
              at java.util.concurrent.FutureTask.run(FutureTask.java:138) ~[na:1.6.0_29]
              at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:94) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:482) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.Node.getChild(Node.java:2624) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.Node.indexOf(Node.java:972) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.Node.indexOf(Node.java:989) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.Node.indexOf(Node.java:989) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.btree.AbstractBTree.rangeCount(AbstractBTree.java:2525) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.relation.accesspath.AccessPath.historicalRangeCount(AccessPath.java:1304) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.relation.accesspath.AccessPath.rangeCount(AccessPath.java:1272) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.join.PipelineJoin$JoinTask$AccessPathTask.call(PipelineJoin.java:1652) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.executeTasks(PipelineJoin.java:1401) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.bop.join.PipelineJoin$JoinTask$BindingSetConsumerTask.call(PipelineJoin.java:1092) ~[bigdata-1.0.6.jar:na]
              ... 16 common frames omitted
      Caused by: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1357207
              at com.bigdata.rwstore.RWStore.getBlock(RWStore.java:3140) ~[bigdata-1.0.6.jar:na]
              at com.bigdata.rwstore.RWStore.getData(RWStore.java:1382) ~[bigdata-1.0.6.jar:na]
      

      The output of DumpJournal (with the -pages option) is below.

      File: C:\journal.jnl
      Length: 2239234048
      Last Modified: Fri Feb 10 10:43:35 EST 2012
      Opening (read-only): C:\journal.jnl
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.historicalIndexCacheCapacity=60
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.historicalIndexCacheTimeout=60000
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.liveIndexCacheCapacity=60
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.liveIndexCacheTimeout=60000
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.initialExtent=10485760
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.maximumExtent=209715200
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.minimumExtension=33554432
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.readOnly=true
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.forceOnCommit=ForceMetadata
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.doubleSync=false
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.deleteOnClose=false
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.tmpDir=C:\Users\stanaka\AppData\Local\Temp\
      INFO : com.bigdata.config.Configuration.getProperty2(Configuration.java:256): com.bigdata.journal.AbstractJournal.bufferMode=Disk
      ERROR: com.bigdata.util.config.LogUtil : Could not initialize Log4J logging utility.
      Set system property '-Dlog4j.configuration=file:bigdata/src/resources/logging/log4j.properties'
        and / or 
      Set system property '-Dlog4j.primary.configuration=file:<installDir>/bigdata/src/resources/logging/log4j.properties'
      INFO : com.bigdata.counters.AbstractStatisticsCollector.<clinit>(AbstractStatisticsCollector.java:99): FQDN      : 10.238.52.125
      INFO : com.bigdata.counters.AbstractStatisticsCollector.<clinit>(AbstractStatisticsCollector.java:100): hostPrefix: /10.238.52.125/
      WARN : org.apache.system.SystemUtil.<clinit>(SystemUtil.java:71): No CPUParser for this platform - looking for class: [org.apache.system.Windows7]
      INFO : org.apache.system.SystemUtil.<clinit>(SystemUtil.java:90): architecture: amd64
      INFO : org.apache.system.SystemUtil.<clinit>(SystemUtil.java:91): operating system: Windows 7
      INFO : org.apache.system.SystemUtil.<clinit>(SystemUtil.java:92): operation system version: 6.1
      INFO : org.apache.system.SystemUtil.<clinit>(SystemUtil.java:93): #processors: 1
      INFO : org.apache.system.SystemUtil.<clinit>(SystemUtil.java:94): cpu info: amd64 Family n, Model n, Stepping n, Undeterminable
       
      BIGDATA(R)
       
                         Flexible
                         Reliable
                        Affordable
            Web-Scale Computing for the Enterprise
       
      Copyright SYSTAP, LLC 2006-2010.  All rights reserved.
      
      10.238.52.125
      Fri Feb 10 19:05:57 EST 2012
      Windows 7/6.1 amd64
      amd64 Family n, Model n, Stepping n, Undeterminable #CPU=1
      Sun Microsystems Inc. 1.6.0_22
      buildVersion=1.0.2
       
      Dependency     License                                                                         
      ICU            http://source.icu-project.org/repos/icu/icu/trunk/license.html                  
      colt           http://acs.lbl.gov/software/colt/license.html                                   
      cweb           http://www.cognitiveweb.org/legal/license/CognitiveWebOpenSourceLicense-1.1.html
      dsiutils       http://www.gnu.org/licenses/lgpl-2.1.html                                       
      fastutil       http://www.apache.org/licenses/LICENSE-2.0.html                                 
      flot           http://www.opensource.org/licenses/mit-license.php                              
      high-scale-lib http://creativecommons.org/licenses/publicdomain                                
      iris           http://www.gnu.org/licenses/lgpl-2.1.html                                       
      jetty          http://www.apache.org/licenses/LICENSE-2.0.html                                 
      jgrapht        http://www.gnu.org/licenses/lgpl-2.1.html                                       
      jquery         https://github.com/jquery/jquery/blob/master/MIT-LICENSE.txt                    
      log4j          http://www.apache.org/licenses/LICENSE-2.0.html                                 
      lucene         http://www.apache.org/licenses/LICENSE-2.0.html                                 
      nanohttp       http://elonen.iki.fi/code/nanohttpd/#license                                    
      nxparser       http://sw.deri.org/2006/08/nxparser/license.txt                                 
      river          http://www.apache.org/licenses/LICENSE-2.0.html                                 
      servlet-api    http://www.apache.org/licenses/LICENSE-2.0.html                                 
      sesame         http://www.openrdf.org/download.jsp                                             
      slf4j          http://www.slf4j.org/license.html                                               
      tuprolog       http://www.gnu.org/licenses/lgpl-2.1.html                                       
      zookeeper      http://www.apache.org/licenses/LICENSE-2.0.html                                 
      
      WARN : com.bigdata.Banner.setDefaultLogLevel(Banner.java:194): Defaulting log level to WARN: com.bigdata
      magic=e6b4c275
      version=1
      rootBlock{ rootBlock=0, challisField=63800, version=2, nextOffset=140638704115749, localTime=1328888455821 [Friday, February 10, 2012 10:40:55 AM EST], firstCommitTime=1328823717147 [Thursday, February 9, 2012 4:41:57 PM EST], lastCommitTime=1328888455764 [Friday, February 10, 2012 10:40:55 AM EST], commitCounter=63800, commitRecordAddr={off=NATIVE:-1398206,len=422}, commitRecordIndexAddr={off=NATIVE:-8228,len=220}, quorumToken=0, metaBitsAddr=56547606587, metaStartAddr=34168, storeType=RW, uuid=15875eb1-2290-460c-b00b-eef9f8b49154, offsetBits=42, checksum=192815664, createTime=1328823716451 [Thursday, February 9, 2012 4:41:56 PM EST], closeTime=0}
      rootBlock{ rootBlock=1, challisField=63801, version=2, nextOffset=140638704115745, localTime=1328888494536 [Friday, February 10, 2012 10:41:34 AM EST], firstCommitTime=1328823717147 [Thursday, February 9, 2012 4:41:57 PM EST], lastCommitTime=1328888494513 [Friday, February 10, 2012 10:41:34 AM EST], commitCounter=63801, commitRecordAddr={off=NATIVE:-694754,len=422}, commitRecordIndexAddr={off=NATIVE:-8215,len=220}, quorumToken=0, metaBitsAddr=56463720507, metaStartAddr=34168, storeType=RW, uuid=15875eb1-2290-460c-b00b-eef9f8b49154, offsetBits=42, checksum=1858347426, createTime=1328823716451 [Thursday, February 9, 2012 4:41:56 PM EST], closeTime=0}
      The current root block is BLZG-208
      extent=2239234048(2135M), userExtent=2239233360(2135M), bytesAvailable=-140636464882385(-134121384M), nextOffset=140638704115745
      There are 2 commit points.
      CommitRecord{timestamp=1328888494513, commitCounter=63801, roots=[-35274566401828, -2983941413797548, -211123412402040, -211119117434749, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}
      name=__globalRowStore, addr={off=852608,len=220}
            Checkpoint{indexType=BTree,height=1,nnodes=1,nleaves=5,nentries=92,counter=0,addrRoot=-35231616728810,addrMetadata=-70385924045659,addrBloomFilter=0,addrCheckpoint=-35205846925092}
            addrMetadata=0, name=__globalRowStore, indexUUID=6c6d81f8-00a2-4b70-a024-8af6275173ec, branchingFactor=32, pmd=null, class=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@48ff4cf{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.btree.DefaultTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.DefaultKeyBuilderFactory{ initialCapacity=0, collator=ICU, locale=en_US, strength=null, decomposition=null}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@7114460{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.CanonicalHuffmanRabaCoder@7e9ffe3f}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=com.bigdata.sparse.LogicalRowSplitHandler@39ed1b0b, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
            com.bigdata.btree.DumpIndex$PageStats{nnodes=1,nleaves=5,nodeBytes=278,minNodeBytes=278,maxNodeBytes=278,leafBytes=6244,minLeafBytes=832,maxLeafBytes=2354,bytesPerNode=278,bytesPerLeaf=1248}
      name=XXXX.kb.lex.ID2TERM, addr={off=853888,len=220}
            Checkpoint{indexType=BTree,height=3,nnodes=788,nleaves=50509,nentries=3232597,counter=0,addrRoot=-35214436859686,addrMetadata=-17179868319,addrBloomFilter=0,addrCheckpoint=-35223026794276}
            addrMetadata=0, name=XXXX.kb.lex.ID2TERM, indexUUID=28306d84-649a-4380-9d36-8c9fb0c76f09, branchingFactor=128, pmd=null, class=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@722b9406{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.lexicon.Id2TermTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=8}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@b035079{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.SimpleRabaCoder@6983d95}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
            com.bigdata.btree.DumpIndex$PageStats{nnodes=788,nleaves=50509,nodeBytes=649364,minNodeBytes=218,maxNodeBytes=1758,leafBytes=347277707,minLeafBytes=969,maxLeafBytes=18278,bytesPerNode=824,bytesPerLeaf=6875}
      name=XXXX.kb.lex.TERM2ID, addr={off=855488,len=220}
            Checkpoint{indexType=BTree,height=3,nnodes=508,nleaves=41862,nentries=3232597,counter=3232864,addrRoot=-2396192319208823,addrMetadata=-70390219012934,addrBloomFilter=0,addrCheckpoint=-35244501630756}
            addrMetadata=0, name=XXXX.kb.lex.TERM2ID, indexUUID=001c266c-ed9b-4e99-9596-87819ee03160, branchingFactor=128, pmd=null, class=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@1b27e267{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.lexicon.Term2IdTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.DefaultKeyBuilderFactory{ initialCapacity=0, collator=ICU, locale=en_US, strength=null, decomposition=null}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@5780e61e{ratio=8}, leafValuesCoder=com.bigdata.btree.raba.codec.CanonicalHuffmanRabaCoder@4f83e873}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
            com.bigdata.btree.DumpIndex$PageStats{nnodes=508,nleaves=41862,nodeBytes=1717275,minNodeBytes=649,maxNodeBytes=13578,leafBytes=252114106,minLeafBytes=968,maxLeafBytes=16734,bytesPerNode=3380,bytesPerLeaf=6022}
      name=XXXX.kb.spo.OSP, addr={off=854208,len=220}
            Checkpoint{indexType=BTree,height=3,nnodes=4791,nleaves=334418,nentries=23040757,counter=0,addrRoot=-6581539164912083,addrMetadata=-25769802999,addrBloomFilter=0,addrCheckpoint=-35227321761572}
            addrMetadata=0, name=XXXX.kb.spo.OSP, indexUUID=12d2d94e-5de2-4e85-9bab-bbe140771e54, branchingFactor=128, pmd=null, class=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@1ab23564{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.SPOTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@29cf45ed{ratio=8}, leafValuesCoder=com.bigdata.rdf.spo.FastRDFValueCoder2@66f7e00c}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
            com.bigdata.btree.DumpIndex$PageStats{nnodes=4791,nleaves=334418,nodeBytes=6175791,minNodeBytes=804,maxNodeBytes=3409,leafBytes=327749056,minLeafBytes=561,maxLeafBytes=3062,bytesPerNode=1289,bytesPerLeaf=980}
      name=XXXX.kb.spo.POS, addr={off=854848,len=220}
            Checkpoint{indexType=BTree,height=3,nnodes=4883,nleaves=337459,nentries=23040757,counter=0,addrRoot=-6581582114585166,addrMetadata=-30064770295,addrBloomFilter=0,addrCheckpoint=-35235911696164}
            addrMetadata=0, name=XXXX.kb.spo.POS, indexUUID=944b4f9c-788a-4629-ace4-d028c48348ed, branchingFactor=128, pmd=null, class=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@5ad08f6f{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.SPOTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@3278eefd{ratio=8}, leafValuesCoder=com.bigdata.rdf.spo.FastRDFValueCoder2@21923dee}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
            com.bigdata.btree.DumpIndex$PageStats{nnodes=4883,nleaves=337459,nodeBytes=5307729,minNodeBytes=872,maxNodeBytes=2918,leafBytes=222045791,minLeafBytes=399,maxLeafBytes=2054,bytesPerNode=1086,bytesPerLeaf=657}
      name=XXXX.kb.spo.SPO, addr={off=857088,len=220}
            Checkpoint{indexType=BTree,height=3,nnodes=4951,nleaves=330741,nentries=23040757,counter=0,addrRoot=-6581693783734438,addrMetadata=-21474835703,addrBloomFilter=0,addrCheckpoint=-35265976467236}
            addrMetadata=0, name=XXXX.kb.spo.SPO, indexUUID=70af9f41-30e8-4f78-8cce-a31763d79c00, branchingFactor=128, pmd=null, class=com.bigdata.btree.BTree, checkpointClass=com.bigdata.btree.Checkpoint, nodeKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@33d7dfb6{ratio=8}, btreeRecordCompressorFactory=N/A, tupleSerializer=com.bigdata.rdf.spo.SPOTupleSerializer{, keyBuilderFactory=com.bigdata.btree.keys.ASCIIKeyBuilderFactory{ initialCapacity=0}, leafKeysCoder=com.bigdata.btree.raba.codec.FrontCodedRabaCoder$DefaultFrontCodedRabaCoder@21a04051{ratio=8}, leafValuesCoder=com.bigdata.rdf.spo.FastRDFValueCoder2@1fde9716}, conflictResolver=N/A, deleteMarkers=false, versionTimestamps=false, versionTimestampFilters=false, isolatable=false, rawRecords=false, bloomFilterFactory=N/A, overflowHandler=N/A, splitHandler=N/A, indexSegmentBranchingFactor=512, indexSegmentBufferNodes=false, indexSegmentRecordCompressorFactory=N/A, asynchronousIndexWriteConfiguration=com.bigdata.btree.AsynchronousIndexWriteConfiguration{ masterQueueCapacity=5000, masterChunkSize=10000, masterChunkTimeoutNanos=50000000, sinkIdleTimeoutNanos=9223372036854775807, sinkPollTimeoutNanos=50000000, sinkQueueCapacity=5000, sinkChunkSize=10000, sinkChunkTimeoutNanos=9223372036854775807}, scatterSplitConfiguration=com.bigdata.btree.ScatterSplitConfiguration{enabled=true, percentOfSplitThreshold=0.25, dataServiceCount=0, indexPartitionCount=0}
            com.bigdata.btree.DumpIndex$PageStats{nnodes=4951,nleaves=330741,nodeBytes=8003465,minNodeBytes=796,maxNodeBytes=3665,leafBytes=282854602,minLeafBytes=455,maxLeafBytes=2169,bytesPerNode=1616,bytesPerLeaf=855}
      
      name  m     height      nnodes      nleaves     nentries    nodeBytes   leafBytes      totalBytes  avgNodeBytes      avgLeafBytes      minNodeBytes      maxNodeBytes      minLeafBytes      maxLeafBytes
      __globalRowStore  32    1     1     5     92    278   6244  6522  278   1248  278   278   832      2354
      XXXX.kb.lex.ID2TERM     128   3     788   50509 3232597     649364      347277707      347927071   824   6875  218   1758  969   18278
      XXXX.kb.lex.TERM2ID     128   3     508   41862 3232597     1717275     252114106      253831381   3380  6022  649   13578 968   16734
      XXXX.kb.spo.OSP   128   3     4791  334418      23040757    6175791     327749056      333924847   1289  980   804   3409  561   3062
      XXXX.kb.spo.POS   128   3     4883  337459      23040757    5307729     222045791      227353520   1086  657   872   2918  399   2054
      XXXX.kb.spo.SPO   128   3     4951  330741      23040757    8003465     282854602      290858067   1616  855   796   3665  455   2169
      

        Activity

        Hide
        martyncutcher martyncutcher added a comment -

        The exception from getBlock is something more than an unallocated address, it is an unregistered Allocator. It is not an address that could be returned from the store in its current state
        - but it looks like a "reasonable" address as opposed to random corrupted data. It would be interesting to determine how far out of range the address was. We could easily modify the exception message to provide some more information.

        If we can confirm that the range is off by a small number then it would appear as if the RWStore committed meta-data was out of sync with the data it referenced. This is not something we have ever seen in any test scenarios previously, so this could be a problem exposed by a specific live environment/usage.

        How this might be linked to 1.0.6 is odd, since the commit protocol is unchanged with regard to root blocks and allocation meta-data.

        Show
        martyncutcher martyncutcher added a comment - The exception from getBlock is something more than an unallocated address, it is an unregistered Allocator. It is not an address that could be returned from the store in its current state - but it looks like a "reasonable" address as opposed to random corrupted data. It would be interesting to determine how far out of range the address was. We could easily modify the exception message to provide some more information. If we can confirm that the range is off by a small number then it would appear as if the RWStore committed meta-data was out of sync with the data it referenced. This is not something we have ever seen in any test scenarios previously, so this could be a problem exposed by a specific live environment/usage. How this might be linked to 1.0.6 is odd, since the commit protocol is unchanged with regard to root blocks and allocation meta-data.
        Hide
        bryanthompson bryanthompson added a comment -

        The customer wrote:

        One way QE has found to constantly reproduce this issue is to do the following:

        1. obtain a write lock (while coping the journal) ~4 minutes

        2. issue queries onto the system and every so often you will hit an exception like the following:

        java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1909086
        java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1909086
                at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1481) ~[bigdata-1.0.6.jar:na]
                at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1658) ~[bigdata-1.0.6.jar:na]
                at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1521) ~[bigdata-1.0.6.jar:na]
                at com.bigdata.striterator.AbstractChunkedResolverator._hasNext(AbstractChunkedResolverator.java:327) ~[bigdata-1.0.6.jar:na]
                at com.bigdata.striterator.AbstractChunkedResolverator.hasNext(AbstractChunkedResolverator.java:303) ~[bigdata-1.0.6.jar:na]
                at com.bigdata.rdf.sail.Bigdata2Sesame2BindingSetIterator.hasNext(Bigdata2Sesame2BindingSetIterator.java:70) ~[bigdata-1.0.6.jar:na]
                at info.aduna.iteration.ConvertingIteration.hasNext(ConvertingIteration.java:62) ~[openrdf-sesame-onejar-2.3.0.jar:na]
                at info.aduna.iteration.IterationWrapper.hasNext(IterationWrapper.java:57) ~[openrdf-sesame-onejar-2.3.0.jar:na]
                at info.aduna.iteration.FilterIteration.findNextElement(FilterIteration.java:68) ~[openrdf-sesame-onejar-2.3.0.jar:na]
                at info.aduna.iteration.FilterIteration.hasNext(FilterIteration.java:43) ~[openrdf-sesame-onejar-2.3.0.jar:na]
                at org.openrdf.query.impl.TupleQueryResultImpl.hasNext(TupleQueryResultImpl.java:90) ~[openrdf-sesame-onejar-2.3.0.jar:na]
                at XXXX.impl.DDUIRowQuery.getRowCount(DDUIRowQuery.java:309) ~[ddui-query-1.5.0.0.1.jar:na]
                at XXXX.represent(DDUIHostResource.java:153) ~[ddui-rest-resource-1.5.0.0.1.jar:na]
        

        The code in getRowCount does the following:

        Try {
                - getReadOnlyConnection
                - executeRDFQuery
                - TupleQueryResult.hasNext()
        } catch (...) {
        } finally {
                - close result
                - close connection
        }
        
        Show
        bryanthompson bryanthompson added a comment - The customer wrote: One way QE has found to constantly reproduce this issue is to do the following: 1. obtain a write lock (while coping the journal) ~4 minutes 2. issue queries onto the system and every so often you will hit an exception like the following: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1909086 java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Halting join: java.lang.IllegalArgumentException: Unable to read data: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -1909086 at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1481) ~[bigdata-1.0.6.jar:na] at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1658) ~[bigdata-1.0.6.jar:na] at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1521) ~[bigdata-1.0.6.jar:na] at com.bigdata.striterator.AbstractChunkedResolverator._hasNext(AbstractChunkedResolverator.java:327) ~[bigdata-1.0.6.jar:na] at com.bigdata.striterator.AbstractChunkedResolverator.hasNext(AbstractChunkedResolverator.java:303) ~[bigdata-1.0.6.jar:na] at com.bigdata.rdf.sail.Bigdata2Sesame2BindingSetIterator.hasNext(Bigdata2Sesame2BindingSetIterator.java:70) ~[bigdata-1.0.6.jar:na] at info.aduna.iteration.ConvertingIteration.hasNext(ConvertingIteration.java:62) ~[openrdf-sesame-onejar-2.3.0.jar:na] at info.aduna.iteration.IterationWrapper.hasNext(IterationWrapper.java:57) ~[openrdf-sesame-onejar-2.3.0.jar:na] at info.aduna.iteration.FilterIteration.findNextElement(FilterIteration.java:68) ~[openrdf-sesame-onejar-2.3.0.jar:na] at info.aduna.iteration.FilterIteration.hasNext(FilterIteration.java:43) ~[openrdf-sesame-onejar-2.3.0.jar:na] at org.openrdf.query.impl.TupleQueryResultImpl.hasNext(TupleQueryResultImpl.java:90) ~[openrdf-sesame-onejar-2.3.0.jar:na] at XXXX.impl.DDUIRowQuery.getRowCount(DDUIRowQuery.java:309) ~[ddui-query-1.5.0.0.1.jar:na] at XXXX.represent(DDUIHostResource.java:153) ~[ddui-rest-resource-1.5.0.0.1.jar:na] The code in getRowCount does the following: Try { - getReadOnlyConnection - executeRDFQuery - TupleQueryResult.hasNext() } catch (...) { } finally { - close result - close connection }
        Hide
        bryanthompson bryanthompson added a comment -

        The pattern reported in https://sourceforge.net/apps/trac/bigdata/ticket/481#comment:2 suggests one way to me that this kind of exception could be triggered.

        Notice that each time the exception has appeared in a thread which was materializing RDF Values for computed query solutions. This thread is asynchronous with respect to the thread in which the solutions are being consumed, which would appear to be the thread in which the connection is being opened and closed.

        The internal IRunningQuery is a Future. When the IRunningQuery is done, it is cancelled. This causes all asynchronous tasks which are running in support of the query to be cancelled as well. This includes any running query operators and the RDF Value materialization behavior. The latter is where this exception is being observed.

        The IRunningQuery implementation uses a pattern where the first Throwable encountered will cancel() the IRunningQuery. This is known as the "firstCause" and is logged as firstCause=true. When the firstCause has a root cause which indicates an interrupt, the query is considered to have terminated normally. This pattern occurs when a LIMIT is used or a result set is closed before it has been fully drained. Only the firstCause exception should be reported out to the thread consuming results from the query. It appear in these stack traces that the exception was the firstCause.

        Assume for a moment that the connection was closed before the query was cancelled. If this were to occur and either if session protection was used or if there was a concurrent commit such that deferred frees could be recycled, then it is possible that some of the allocation slots on which the query had been reading would be marked as free. Any read against a free allocation slot would cause an exception to be thrown and might result in a PhysicalAddressResolutionException being thrown. Assuming that the connection close occurred first, it is quite possible that such read errors could be observed from asynchronous tasks supporting the query which had not yet noticed the interrupt when the IRunningQuery#cancel() was invoked. Further, the RDF Value materialization is not actually implemented inside of the query engine for a variety of reasons so a slightly different code path is followed to ensure that the value materialization operation halts when the result set is closed.

        However, based on the original stack trace and the stack trace immediately above, it seems as if the exception caused the query to be terminated rather than occurring after the query was done. Is it possible that "close connection" is occurring before all results have been fully materialized? If so, then this might point to a concurrency hole in the logic used to terminate the RDF Value materialization task resulting in a race condition between the that task reading on the RWStore and the RWStore marking allocation bits as free due to a concurrent commit after the query connection was closed.

        Show
        bryanthompson bryanthompson added a comment - The pattern reported in https://sourceforge.net/apps/trac/bigdata/ticket/481#comment:2 suggests one way to me that this kind of exception could be triggered. Notice that each time the exception has appeared in a thread which was materializing RDF Values for computed query solutions. This thread is asynchronous with respect to the thread in which the solutions are being consumed, which would appear to be the thread in which the connection is being opened and closed. The internal IRunningQuery is a Future. When the IRunningQuery is done, it is cancelled. This causes all asynchronous tasks which are running in support of the query to be cancelled as well. This includes any running query operators and the RDF Value materialization behavior. The latter is where this exception is being observed. The IRunningQuery implementation uses a pattern where the first Throwable encountered will cancel() the IRunningQuery. This is known as the "firstCause" and is logged as firstCause=true. When the firstCause has a root cause which indicates an interrupt, the query is considered to have terminated normally. This pattern occurs when a LIMIT is used or a result set is closed before it has been fully drained. Only the firstCause exception should be reported out to the thread consuming results from the query. It appear in these stack traces that the exception was the firstCause. Assume for a moment that the connection was closed before the query was cancelled. If this were to occur and either if session protection was used or if there was a concurrent commit such that deferred frees could be recycled, then it is possible that some of the allocation slots on which the query had been reading would be marked as free. Any read against a free allocation slot would cause an exception to be thrown and might result in a PhysicalAddressResolutionException being thrown. Assuming that the connection close occurred first, it is quite possible that such read errors could be observed from asynchronous tasks supporting the query which had not yet noticed the interrupt when the IRunningQuery#cancel() was invoked. Further, the RDF Value materialization is not actually implemented inside of the query engine for a variety of reasons so a slightly different code path is followed to ensure that the value materialization operation halts when the result set is closed. However, based on the original stack trace and the stack trace immediately above, it seems as if the exception caused the query to be terminated rather than occurring after the query was done. Is it possible that "close connection" is occurring before all results have been fully materialized? If so, then this might point to a concurrency hole in the logic used to terminate the RDF Value materialization task resulting in a race condition between the that task reading on the RWStore and the RWStore marking allocation bits as free due to a concurrent commit after the query connection was closed.
        Hide
        bryanthompson bryanthompson added a comment -

        It appears that there were two distinct root causes.

        1) It seems that there was code which was closing the query connection before closing the result set. This could give rise to a PhysicalAddressResolutionException through concurrent recycling while the query was still running.

        2) It appears that the backup procedure was calling BigdataSailConnection#abort() on the unisolated connection rather than BigdataSailConnection#close().

        Customer has fixed both issues and is no longer observing the PhysicalAddressResolutionException.

        Show
        bryanthompson bryanthompson added a comment - It appears that there were two distinct root causes. 1) It seems that there was code which was closing the query connection before closing the result set. This could give rise to a PhysicalAddressResolutionException through concurrent recycling while the query was still running. 2) It appears that the backup procedure was calling BigdataSailConnection#abort() on the unisolated connection rather than BigdataSailConnection#close(). Customer has fixed both issues and is no longer observing the PhysicalAddressResolutionException.
        Hide
        bryanthompson bryanthompson added a comment -

        Closing off this issue. The customer was able to fix the problem by the code changes described above on their end.

        Show
        bryanthompson bryanthompson added a comment - Closing off this issue. The customer was able to fix the problem by the code changes described above on their end.

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: