Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-769

HA deadlock under UPDATE + QUERY

    XMLWordPrintable

    Details

      Description

      A deadlock was observed in HA after running the BSBM 100M UPDATE mixture for several hours. The deadlock is related to the release time consensus protocol for HA. However, the deadlock is also the cause of a known hot spot for the standalone deployment model.

      The following method
      {{
      DefaultResourceLocator.locateResourceOn(final IIndexManager indexManager,final String namespace, final long timestamp)
      }}}
      uses

      AbstractJournal.getCommitRecord(long timestamp)
      

      to obtain the commitTime corresponding to the timestamp. This latter method is a hot spot since it requires the field read/write lock for the AbstractJournal and access to the CommitRecordIndex must be mutex for readers and mutation (concurrent readers are allow). However, this method is invoked most commonly using a read-only tx, and this is in fact true for the observed deadlock since the resource is being located for a QUERY as shown by the following stack traces:

      This thread was blocked on AbstractJournal.fieldReadWriteLock while holding the NamedLock in DefaultResourceLocator.

      	at com.bigdata.journal.AbstractJournal.getCommitRecord(AbstractJournal.java:4131) <== DEADLOCK
      	at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:655)
      	at com.bigdata.relation.locator.DefaultResourceLocator.locateResource(DefaultResourceLocator.java:560)
      	at com.bigdata.relation.locator.DefaultResourceLocator.cacheMiss(DefaultResourceLocator.java:381)
      	at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:333)
      	at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.newTx(BigdataSail.java:4104)
      	at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:3986)
      

      New read-only queries (such as the following trace) were unable to start since the NamedLock was held by the thread above:

      	at com.bigdata.concurrent.NamedLock.acquireLock(NamedLock.java:106) <== DEADLOCK
      	at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:313)
      	at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.newTx(BigdataSail.java:4104)
      	at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:3986)
      

      The HA commit protocol was causing the deadlock by (a):

      	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:399)
      	at com.bigdata.journal.Journal$BarrierState.messageFollowers(Journal.java:708)
      	at com.bigdata.journal.Journal$BarrierState.access$700(Journal.java:314)
      	at com.bigdata.journal.Journal$InnerJournalTransactionService.updateReleaseTimeConsensus(Journal.java:1090)
      	at com.bigdata.journal.AbstractJournal.commitNow(AbstractJournal.java:2937)
      	at com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2797)
      	at com.bigdata.rdf.store.LocalTripleStore.commit(LocalTripleStore.java:80) <== DEADLOCK : LocalTripleStore.this
      	- locked <0x000000070358f3a0> (a com.bigdata.rdf.store.LocalTripleStore)
      	at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.commit2(BigdataSail.java:2977) <== DEADLOCK: BigdataSail.this
      	- locked <0x0000000782cf3fd8> (a com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection)
      	at com.bigdata.rdf.sail.BigdataSailRepositoryConnection.commit2(BigdataSailRepositoryConnection.java:272)
      

      Also, new UPDATEs could not run since the monitor for the unisolated view of the LocalTripleStore was held by the thread requesting the commit (which is shown immediately above):

      	at com.bigdata.rdf.store.AbstractTripleStore.getLexiconRelation(AbstractTripleStore.java:1944)
      	- waiting to lock <0x000000070358f3a0> (a com.bigdata.rdf.store.LocalTripleStore) <== DEADLOCK : LocalTripleStore.this
      	at com.bigdata.rdf.sail.sparql.BigdataASTContext.<init>(BigdataASTContext.java:96)
      	at com.bigdata.rdf.sail.sparql.Bigdata2ASTSPARQLParser.<init>(Bigdata2ASTSPARQLParser.java:100)
      	at com.bigdata.rdf.sail.webapp.BigdataRDFContext.getQueryTask(BigdataRDFContext.java:1775)
      	at com.bigdata.rdf.sail.webapp.QueryServlet.doUpdate(QueryServlet.java:345)
      	at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:151)
      	at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:197)
      

      Looking at the followers, I see the 2nd follower at

      	at $Proxy2.notifyEarliestCommitTime(Unknown Source)
      	at com.bigdata.journal.Journal$InnerJournalTransactionService$GatherTask.call(Journal.java:1392)
      	at com.bigdata.journal.Journal$InnerJournalTransactionService$GatherTask.call(Journal.java:1337)
      

      and the first follower is not running a GatherTask at all.

      This last observation leads me to wonder if the problem had to do with a non-atomic decision concerning the #of parties that would participate in the release time consensus protocol or if the code that was running on the HA3 cluster was not monitoring the quorum state for a quorum break or service leave.

      Thus, I think that this issue was misdiagnosed. The problem is that new transaction starts are blocking on getCommitRecordIndex() in the DefaultResourceLocator during the commitNow(). However, that contention can be removed by using the LocalTransactionManager to resolve the ITx for the timestamp and then the readsOnCommitTime from the ITx. This should increase query throughput (on the leader) with concurrent UPDATEs on the leader.

      See https://sourceforge.net/apps/trac/bigdata/ticket/673 (DGC in release time consensus protocol causes native thread leak in HAJournalServer at each commit)

        Attachments

          Activity

            People

            Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: