Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1953

multiple namespaces - Address did not resolve to physical address: -3085793

    Details

    • Type: Bug
    • Status: Done
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: BLAZEGRAPH_2_0_1
    • Fix Version/s: BLAZEGRAPH_2_2_0
    • Component/s: None
    • Labels:
      None

      Description

      Using 2.0.1 with concurrent operations on multiple namespaces, we get this stack trace, after which the journal is toast.

      Jun 16,2016 13:16:34 PDT - WARN : 5224632      com.bigdata.journal.Journal.executorService1411 com.bigdata.rwstore.FixedAllocator.getPhysicalAddress(FixedAllocator.java:205): Physical address 1012055040 not accessible for Allocator of size 512
      Jun 16,2016 13:16:34 PDT - ERROR: 5224659      com.bigdata.rdf.sail.webapp.BigdataRDFContext.queryService15 com.bigdata.rdf.sparql.ast.optimizers.ASTRangeCountOptimizer.attachRangeCounts(ASTRangeCountOptimizer.java:130): java.util.concurrent.ExecutionException: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -3085793
      Jun 16,2016 13:16:34 PDT - ERROR: 5224661      qtp710714889-1420 com.bigdata.rdf.sail.webapp.BigdataRDFServlet.launderThrowable(BigdataRDFServlet.java:214): cause=java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -3085793, query=SPARQL-QUERY: queryStr=base <http://localhost:8000/>
      prefix owl: <http://www.w3.org/2002/07/owl#>
      prefix s: </bdm/api/>
      prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
      prefix bds: <http://www.bigdata.com/rdf/search#>
      prefix xsd: <http://www.w3.org/2001/XMLSchema#>
      prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
      prefix skos: <http://www.w3.org/2004/02/skos/core#>
      prefix dc: <http://purl.org/dc/elements/1.1/>
      prefix syapse: </graph/syapse#>
      
      
      SELECT ?graph ?scheme ?id ?vType ?primaryScheme
      # returns the short id, and vocabulary type for each scheme
      {
         GRAPH ?graph {
          ?scheme rdf:type skos:ConceptScheme
         }
         ?scheme
               rdf:type / syapse:shortId ?vType  ;
               syapse:shortId ?id .
          OPTIONAL {
            ?scheme syapse:dependsOn / syapse:shortId ?primaryScheme
          }
      } ORDER BY ?primaryScheme
      
      java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -3085793
              at java.util.concurrent.FutureTask.report(FutureTask.java:122)
              at java.util.concurrent.FutureTask.get(FutureTask.java:192)
              at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:281)
              at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:636)
              at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:263)
              at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:269)
              at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doPost(MultiTenancyServlet.java:170)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
              at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
              at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
              at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
              at org.eclipse.jetty.server.Server.handle(Server.java:497)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
              at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -3085793
              at java.util.concurrent.FutureTask.report(FutureTask.java:122)
              at java.util.concurrent.FutureTask.get(FutureTask.java:192)
              at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:834)
              at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:653)
              at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              ... 1 more
      Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -3085793
              at com.bigdata.rdf.sparql.ast.optimizers.ASTRangeCountOptimizer.attachRangeCounts(ASTRangeCountOptimizer.java:141)
              at com.bigdata.rdf.sparql.ast.optimizers.ASTRangeCountOptimizer.optimizeJoinGroup(ASTRangeCountOptimizer.java:78)
              at com.bigdata.rdf.sparql.ast.optimizers.AbstractJoinGroupOptimizer.optimize(AbstractJoinGroupOptimizer.java:161)
              at com.bigdata.rdf.sparql.ast.optimizers.AbstractJoinGroupOptimizer.optimize(AbstractJoinGroupOptimizer.java:179)
              at com.bigdata.rdf.sparql.ast.optimizers.AbstractJoinGroupOptimizer.optimize(AbstractJoinGroupOptimizer.java:101)
              at com.bigdata.rdf.sparql.ast.optimizers.ASTOptimizerList.optimize(ASTOptimizerList.java:126)
              at com.bigdata.rdf.sparql.ast.eval.AST2BOpUtility.convert(AST2BOpUtility.java:268)
             at com.bigdata.rdf.sparql.ast.optimizers.ASTRangeCountOptimizer.optimizeJoinGroup(ASTRangeCountOptimizer.java:78)
              at com.bigdata.rdf.sparql.ast.optimizers.AbstractJoinGroupOptimizer.optimize(AbstractJoinGroupOptimizer.java:161)
              at com.bigdata.rdf.sparql.ast.optimizers.AbstractJoinGroupOptimizer.optimize(AbstractJoinGroupOptimizer.java:179)
              at com.bigdata.rdf.sparql.ast.optimizers.AbstractJoinGroupOptimizer.optimize(AbstractJoinGroupOptimizer.java:101)
              at com.bigdata.rdf.sparql.ast.optimizers.ASTOptimizerList.optimize(ASTOptimizerList.java:126)
              at com.bigdata.rdf.sparql.ast.eval.AST2BOpUtility.convert(AST2BOpUtility.java:268)
              at com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.optimizeQuery(ASTEvalHelper.java:426)
              at com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.evaluateTupleQuery(ASTEvalHelper.java:212)
              at com.bigdata.rdf.sail.BigdataSailTupleQuery.evaluate(BigdataSailTupleQuery.java:79)
              at com.bigdata.rdf.sail.BigdataSailTupleQuery.evaluate(BigdataSailTupleQuery.java:61)
              at org.openrdf.repository.sail.SailTupleQuery.evaluate(SailTupleQuery.java:75)
              at com.bigdata.rdf.sail.webapp.BigdataRDFContext$TupleQueryTask.doQuery(BigdataRDFContext.java:1711)
              at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.innerCall(BigdataRDFContext.java:1568)
              at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:1533)
              at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:705)
              ... 4 more
      Caused by: java.util.concurrent.ExecutionException: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -3085793
              at java.util.concurrent.FutureTask.report(FutureTask.java:122)
              at java.util.concurrent.FutureTask.get(FutureTask.java:192)
              at com.bigdata.rdf.sparql.ast.optimizers.ASTRangeCountOptimizer.attachRangeCounts(ASTRangeCountOptimizer.java:125)
              ... 19 more
      Caused by: com.bigdata.rwstore.PhysicalAddressResolutionException: Address did not resolve to physical address: -3085793
              at com.bigdata.rwstore.RWStore.getData(RWStore.java:2046)
              at com.bigdata.journal.RWStrategy.readFromLocalStore(RWStrategy.java:732)
              at com.bigdata.journal.RWStrategy.read(RWStrategy.java:155)
              at com.bigdata.journal.AbstractJournal.read(AbstractJournal.java:4304)
              at com.bigdata.btree.AbstractBTree.readNodeOrLeaf(AbstractBTree.java:4065)
              at com.bigdata.btree.Node._getChild(Node.java:2746)
              at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:370)
              at com.bigdata.btree.AbstractBTree$1.compute(AbstractBTree.java:353)
              at com.bigdata.util.concurrent.Memoizer$1.call(Memoizer.java:77)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at com.bigdata.util.concurrent.Memoizer.compute(Memoizer.java:92)
              at com.bigdata.btree.AbstractBTree.loadChild(AbstractBTree.java:539)
              at com.bigdata.btree.Node.getChild(Node.java:2644)
              at com.bigdata.btree.Node.indexOf(Node.java:971)
              at com.bigdata.btree.Node.indexOf(Node.java:988)
              at com.bigdata.btree.AbstractBTree.rangeCount(AbstractBTree.java:2663)
              at com.bigdata.relation.accesspath.AccessPath.historicalRangeCount(AccessPath.java:1420)
              at com.bigdata.relation.accesspath.AccessPath.rangeCount(AccessPath.java:1388)
              at com.bigdata.rdf.sparql.ast.optimizers.ASTRangeCountOptimizer.estimateCardinalities(ASTRangeCountOptimizer.java:209)
              at com.bigdata.rdf.sparql.ast.optimizers.ASTRangeCountOptimizer.estimateCardinality(ASTRangeCountOptimizer.java:193)
              at com.bigdata.rdf.sparql.ast.optimizers.ASTRangeCountOptimizer$RangeCountTask.call(ASTRangeCountOptimizer.java:172)
              at com.bigdata.rdf.sparql.ast.optimizers.ASTRangeCountOptimizer$RangeCountTask.call(ASTRangeCountOptimizer.java:155)
              ... 4 more
      Jun 16,2016 13:16:34 PDT - WARN : 5225124      com.bigdata.journal.Journal.executorService1407 com.bigdata.rwstore.RWStore.showWriteCacheDebug(RWStore.java:6653): WriteCacheDebug: 1012055040 - No WriteCache debug info
      Jun 16,2016 13:16:34 PDT - WARN : 5225124      com.bigdata.journal.Journal.executorService1407 com.bigdata.rwstore.FixedAllocator.getPhysicalAddress(FixedAllocator.java:205): Physical address 1012055040 not accessible for Allocator of size 512
      

      etc

        Issue Links

          Activity

          Hide
          bryanthompson bryanthompson added a comment -

          Things appear to be narrowing things down to tests that attempt to write bad rdf and prompt rollbacks. Cusrtomer is going to try to reduce things to (a) loading 500k statements from a file; and (b) running a sequence of tests in two separate processes against a single NSS instance where each client process uses a different namespace on the server. Once we get a reproducer, we can replicate locally and analyze. We should also make sure to recreate a version of the reproducer that we can run in CI.

          Show
          bryanthompson bryanthompson added a comment - Things appear to be narrowing things down to tests that attempt to write bad rdf and prompt rollbacks. Cusrtomer is going to try to reduce things to (a) loading 500k statements from a file; and (b) running a sequence of tests in two separate processes against a single NSS instance where each client process uses a different namespace on the server. Once we get a reproducer, we can replicate locally and analyze. We should also make sure to recreate a version of the reproducer that we can run in CI.
          Hide
          martyncutcher martyncutcher added a comment -

          Running the latest test scripts I have observed two failure exceptions:

          1) A checksum error on reading data
          2) An address error from a range check on a fixed allocator

          Both these errors are indicative of a problem around rollback. I will try and uncover more failure modes to get more clues.

          Show
          martyncutcher martyncutcher added a comment - Running the latest test scripts I have observed two failure exceptions: 1) A checksum error on reading data 2) An address error from a range check on a fixed allocator Both these errors are indicative of a problem around rollback. I will try and uncover more failure modes to get more clues.
          Hide
          bryanthompson bryanthompson added a comment -

          Fix is in PR: https://github.com/SYSTAP/bigdata/pull/448

          PR is in CI.

          This works for namespaces a .. k and 5 executions of the script.

          jeremycarroll

          The script should either be rewritten as a unit test or CI amended to run the script.

          Show
          bryanthompson bryanthompson added a comment - Fix is in PR: https://github.com/SYSTAP/bigdata/pull/448 PR is in CI. This works for namespaces a .. k and 5 executions of the script. jeremycarroll The script should either be rewritten as a unit test or CI amended to run the script.
          Hide
          bryanthompson bryanthompson added a comment -

          PR was merged. Fix is in master. Test still needs to be ported to CI

          Show
          bryanthompson bryanthompson added a comment - PR was merged. Fix is in master. Test still needs to be ported to CI
          Hide
          bryanthompson bryanthompson added a comment -

          Assigned to Brad Bebee for CI integration of the test.

          Show
          bryanthompson bryanthompson added a comment - Assigned to Brad Bebee for CI integration of the test.

            People

            • Assignee:
              beebs Brad Bebee
              Reporter:
              jjc Jeremy Carroll
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: