Details

    • Type: Sub-task
    • Status: Done
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_2_2_0
    • Component/s: RWStore
    • Labels:
      None

      Description

      This shows up in QA for govtrack query10.

      org.openrdf.query.QueryEvaluationException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=d4316351-0157-4284-9c5b-fad534b8a855,bopId=36,partitionId=-1,sinkId=39,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Check header size assumptions
      	at com.bigdata.rdf.sail.Bigdata2Sesame2BindingSetIterator.hasNext(Bigdata2Sesame2BindingSetIterator.java:188)
      	at info.aduna.iteration.IterationWrapper.hasNext(IterationWrapper.java:68)
      	at org.openrdf.query.QueryResults.report(QueryResults.java:155)
      	at org.openrdf.repository.sail.SailTupleQuery.evaluate(SailTupleQuery.java:76)
      	at com.bigdata.rdf.sail.webapp.BigdataRDFContext$TupleQueryTask.doQuery(BigdataRDFContext.java:1713)
      	at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.innerCall(BigdataRDFContext.java:1558)
      	at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:1526)
      	at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:739)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=d4316351-0157-4284-9c5b-fad534b8a855,bopId=36,partitionId=-1,sinkId=39,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Check header size assumptions
      	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1523)
      	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1710)
      	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1563)
      	at com.bigdata.striterator.AbstractChunkedResolverator._hasNext(AbstractChunkedResolverator.java:365)
      	at com.bigdata.striterator.AbstractChunkedResolverator.hasNext(AbstractChunkedResolverator.java:341)
      	at com.bigdata.rdf.sail.Bigdata2Sesame2BindingSetIterator.hasNext(Bigdata2Sesame2BindingSetIterator.java:134)
      	... 11 more
      Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=d4316351-0157-4284-9c5b-fad534b8a855,bopId=36,partitionId=-1,sinkId=39,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Check header size assumptions
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:188)
      	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1454)
      	... 16 more
      Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=d4316351-0157-4284-9c5b-fad534b8a855,bopId=36,partitionId=-1,sinkId=39,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Check header size assumptions
      	at com.bigdata.rdf.sail.RunningQueryCloseableIterator.checkFuture(RunningQueryCloseableIterator.java:59)
      	at com.bigdata.rdf.sail.RunningQueryCloseableIterator.close(RunningQueryCloseableIterator.java:74)
      	at com.bigdata.rdf.sail.RunningQueryCloseableIterator.hasNext(RunningQueryCloseableIterator.java:84)
      	at com.bigdata.striterator.ChunkedWrappedIterator.hasNext(ChunkedWrappedIterator.java:197)
      	at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:222)
      	at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:197)
      	... 4 more
      Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=d4316351-0157-4284-9c5b-fad534b8a855,bopId=36,partitionId=-1,sinkId=39,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Check header size assumptions
      	at com.bigdata.util.concurrent.Haltable.get(Haltable.java:273)
      	at com.bigdata.bop.engine.AbstractRunningQuery.get(AbstractRunningQuery.java:1516)
      	at com.bigdata.bop.engine.AbstractRunningQuery.get(AbstractRunningQuery.java:104)
      	at com.bigdata.rdf.sail.RunningQueryCloseableIterator.checkFuture(RunningQueryCloseableIterator.java:46)
      	... 9 more
      Caused by: java.lang.Exception: task=ChunkTask{query=d4316351-0157-4284-9c5b-fad534b8a855,bopId=36,partitionId=-1,sinkId=39,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Check header size assumptions
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1367)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:926)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:821)
      	... 3 more
      Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Check header size assumptions
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:188)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1347)
      	... 8 more
      Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Check header size assumptions
      	at com.bigdata.bop.engine.ChunkedRunningQuery.scheduleNext(ChunkedRunningQuery.java:748)
      	at com.bigdata.bop.engine.ChunkedRunningQuery.acceptChunk(ChunkedRunningQuery.java:348)
      	at com.bigdata.bop.engine.QueryEngine.acceptChunk(QueryEngine.java:1159)
      	at com.bigdata.bop.engine.StandaloneChunkHandler.handleChunk(StandaloneChunkHandler.java:113)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.outputChunk(ChunkedRunningQuery.java:1699)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.addReorderAllowed(ChunkedRunningQuery.java:1628)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1569)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$HandleChunkBuffer.add(ChunkedRunningQuery.java:1453)
      	at com.bigdata.bop.solutions.MemoryGroupByOp$GroupByTask.call(MemoryGroupByOp.java:486)
      	at com.bigdata.bop.solutions.MemoryGroupByOp$GroupByTask.call(MemoryGroupByOp.java:291)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1346)
      	... 8 more
      Caused by: java.lang.IllegalStateException: Check header size assumptions
      	at com.bigdata.rwstore.sector.PSInputStream.<init>(PSInputStream.java:30)
      	at com.bigdata.rwstore.sector.AllocationContext.getInputStream(AllocationContext.java:351)
      	at com.bigdata.rwstore.sector.MemStrategy.getInputStream(MemStrategy.java:477)
      	at com.bigdata.rwstore.sector.MemStore.getInputStream(MemStore.java:388)
      	at com.bigdata.bop.solutions.SolutionSetStream.get(SolutionSetStream.java:236)
      	at com.bigdata.bop.engine.LocalNativeChunkMessage$ChunkAccessor.<init>(LocalNativeChunkMessage.java:280)
      	at com.bigdata.bop.engine.LocalNativeChunkMessage.getChunkAccessor(LocalNativeChunkMessage.java:263)
      	at com.bigdata.bop.engine.ChunkedRunningQuery.scheduleNext(ChunkedRunningQuery.java:684)
      	... 19 more
      

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        Here is the explain showing the exception.

        martyncutcher

        Show
        bryanthompson bryanthompson added a comment - Here is the explain showing the exception. martyncutcher
        Hide
        martyncutcher martyncutcher added a comment -

        It looks like the maximum size of data allowed with a memory manager BLOB is 4G. Any more than that and the blob header cannot be stored in a single "page". Is this likely to have been exceeded?

        The limit could easily be increased by increasing the maximum page size from 4K to 8K. This would increase the maximum BLOB from 4G to 16G.

        Less easily, the MemoryManager could support "blob of blobs" to enable the blob header to itself be a blob, then, in theory, the limit is removed.

        Show
        martyncutcher martyncutcher added a comment - It looks like the maximum size of data allowed with a memory manager BLOB is 4G. Any more than that and the blob header cannot be stored in a single "page". Is this likely to have been exceeded? The limit could easily be increased by increasing the maximum page size from 4K to 8K. This would increase the maximum BLOB from 4G to 16G. Less easily, the MemoryManager could support "blob of blobs" to enable the blob header to itself be a blob, then, in theory, the limit is removed.
        Hide
        bryanthompson bryanthompson added a comment -

        Martyn wrote: I have pushed a fix to an "off by one" boundary calculation on BLOB data for the MemoryManager, and added a testBlobStreamBoundaries test to TestMemoryManager.

        @ctctechnology pushed 2 commits.

        Show
        bryanthompson bryanthompson added a comment - Martyn wrote: I have pushed a fix to an "off by one" boundary calculation on BLOB data for the MemoryManager, and added a testBlobStreamBoundaries test to TestMemoryManager. @ctctechnology pushed 2 commits. f2fe17f Add Blob Boundary tests and fix boundary "off by one" calculation error 80172c8 Merge branch ' BLZG-2041 ' of https://github.com/SYSTAP/bigdata into BLZG-2041
        Hide
        bryanthompson bryanthompson added a comment - - edited

        Brad Bebee has merged this into BLZG-2041-dbg in https://github.com/SYSTAP/bigdata/commit/de4543b2a09a7740a0ce80a845135fd3eb39755e

        michaelschmidt has verified that this addresses an error still appears in govtrack query10.

        Show
        bryanthompson bryanthompson added a comment - - edited Brad Bebee has merged this into BLZG-2041 -dbg in https://github.com/SYSTAP/bigdata/commit/de4543b2a09a7740a0ce80a845135fd3eb39755e michaelschmidt has verified that this addresses an error still appears in govtrack query10.
        Hide
        bryanthompson bryanthompson added a comment -

        The error with govtrack query10 has been resolved by the introduction of a blob of blobs pattern for the memstore PSInputStream by martyncutcher in https://github.com/SYSTAP/bigdata/pull/475/commits/27a39099524b62a9a9eaefcb2aea5bb214fe4b38

        Show
        bryanthompson bryanthompson added a comment - The error with govtrack query10 has been resolved by the introduction of a blob of blobs pattern for the memstore PSInputStream by martyncutcher in https://github.com/SYSTAP/bigdata/pull/475/commits/27a39099524b62a9a9eaefcb2aea5bb214fe4b38

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: