Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1658

MemoryManager should know when it has been closed

    XMLWordPrintable

    Details

      Description

      The MemoryManager associated with an AbstractRunningQuery is currently cleared (clear()) when releaseNativeResources() is invoked by cancel(...).

      [java] ERROR: 1395088      com.bigdata.journal.Journal.executorService197 com.bigdata.bop.join.HTreeHashJoinUtility.launderThrowable(HTreeHashJoinUtility.java:2056): cause=java.lang.IllegalStateException: Address: -9830430 yields index: 149 >= sector:size(): 0, state=HTreeHashJoinUtility{open=true,joinType=Optional,joinVars=[doc],outputDistinctJVs=true,size=861343,considered(left=841790,right=23092600,joins=55437602),joinSetSize=845935}
           [java] java.lang.IllegalStateException: Address: -9830430 yields index: 149 >= sector:size(): 0
           [java] 	at com.bigdata.rwstore.sector.MemoryManager.getSector(MemoryManager.java:905)
           [java] 	at com.bigdata.rwstore.sector.MemoryManager.getBuffer(MemoryManager.java:860)
           [java] 	at com.bigdata.rwstore.sector.MemoryManager.get(MemoryManager.java:750)
           [java] 	at com.bigdata.rwstore.sector.MemoryManager.getBlobHdr(MemoryManager.java:832)
           [java] 	at com.bigdata.rwstore.sector.MemoryManager.get(MemoryManager.java:759)
           [java] 	at com.bigdata.rwstore.sector.AllocationContext.get(AllocationContext.java:269)
           [java] 	at com.bigdata.rwstore.sector.MemoryManager.read(MemoryManager.java:802)
           [java] 	at com.bigdata.rwstore.sector.AllocationContext.read(AllocationContext.java:276)
           [java] 	at com.bigdata.rwstore.sector.MemStrategy.read(MemStrategy.java:361)
           [java] 	at com.bigdata.rwstore.sector.MemStore.read(MemStore.java:167)
           [java] 	at com.bigdata.htree.AbstractHTree.readNodeOrLeaf(AbstractHTree.java:2016)
           [java] 	at com.bigdata.htree.DirectoryPage._getChild(DirectoryPage.java:641)
           [java] 	at com.bigdata.htree.AbstractHTree$1.compute(AbstractHTree.java:435)
           [java] 	at com.bigdata.htree.AbstractHTree$1.compute(AbstractHTree.java:418)
           [java] 	at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
           [java] 	at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
           [java] 	at com.bigdata.htree.AbstractHTree.loadChild(AbstractHTree.java:554)
           [java] 	at com.bigdata.htree.DirectoryPage.getChild(DirectoryPage.java:437)
           [java] 	at com.bigdata.htree.DirectoryPage.getChildIfPresent(DirectoryPage.java:1777)
           [java] 	at com.bigdata.htree.HTree.lookupAll(HTree.java:1311)
           [java] 	at com.bigdata.bop.join.HTreeHashJoinUtility.outputOptionals(HTreeHashJoinUtility.java:1537)
           [java] 	at com.bigdata.bop.join.SolutionSetHashJoinOp$ChunkTask.doLastPass(SolutionSetHashJoinOp.java:365)
           [java] 	at com.bigdata.bop.join.SolutionSetHashJoinOp$ChunkTask.doHashJoin(SolutionSetHashJoinOp.java:330)
           [java] 	at com.bigdata.bop.join.SolutionSetHashJoinOp$ChunkTask.call(SolutionSetHashJoinOp.java:280)
           [java] 	at com.bigdata.bop.join.SolutionSetHashJoinOp$ChunkTask.call(SolutionSetHashJoinOp.java:191)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
           [java] 	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1315)
           [java] 	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:895)
           [java] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
           [java] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
           [java] 	at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
           [java] 	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:790)
           [java] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
           [java] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
           [java] 	at java.lang.Thread.run(Thread.java:745)
      

      We should invoke MemoryManager.close() rather than .clear().

          protected void releaseNativeMemoryForQuery() {
              
              assert lock.isHeldByCurrentThread();
              
              // clear reference, returning old value.
              final IMemoryManager memoryManager = this.memoryManager.getAndSet(null);
      
              if (memoryManager != null) {
                  
                  // release resources.
                  memoryManager.clear();
                  
              }
      
          }
      

      newMemoryManager() always returns a MemoryManager object, so we could change the reference to MemoryManager rather IMemoryManager (which does not declare close()).

              return new MemoryManager(pool, nsectors, blocking, null/* properties */);
      

      This would allow us to detect cases where the MemoryManager has been closed (because the query was terminated). Right now attempts to access data in the memory manager instead cause cryptic errors with an attempt to access beyond the end of the sector (see above).

        Attachments

          Activity

            People

            Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: