Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-163

Resource leaks and deadlocks (CI for HA)

    Details

    • Type: Task
    • Status: Open
    • Resolution: Unresolved
    • Affects Version/s: BIGDATA_RELEASE_1_3_0
    • Fix Version/s: None
    • Component/s: CI, HAJournalServer
    • Labels:
      None

      Description

      • Several tests are not tearing down the HAReceiveService. Since these stack traces are in the Launcher thread, they are probably related to the TestHASendAndReceive and TestHASendAndReceive3Nodes. The following stack traces are from a single CI run:
        "com.bigdata.ha.pipeline.HAReceiveService@116886396{addrSelf=0.0.0.0/0.0.0.0:52838}" daemon prio=10 tid=0x00007fc124050000 nid=0x542a waiting on condition [0x00007fc1812ec000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x0000000701198b18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@1042839245{addrSelf=0.0.0.0/0.0.0.0:37898}" daemon prio=10 tid=0x00007fc124063000 nid=0x5427 waiting on condition [0x00007fc1804d0000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x0000000701196a38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@1153334631{addrSelf=0.0.0.0/0.0.0.0:40404}" daemon prio=10 tid=0x00007fc124049800 nid=0x53b5 waiting on condition [0x00007fc1030ef000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x000000070116fcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@484354515{addrSelf=0.0.0.0/0.0.0.0:38197}" daemon prio=10 tid=0x00007fc12408a800 nid=0x5416 waiting on condition [0x00007fc0f8948000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x0000000701193058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@218651010{addrSelf=0.0.0.0/0.0.0.0:56367}" daemon prio=10 tid=0x00007fc12403d800 nid=0x5413 waiting on condition [0x00007fc0f9251000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x0000000701190f78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@1472543566{addrSelf=0.0.0.0/0.0.0.0:36547}" daemon prio=10 tid=0x00007fc12407b800 nid=0x5403 waiting on condition [0x00007fc0f7a39000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x000000070118d9f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@1379383211{addrSelf=0.0.0.0/0.0.0.0:52078}" daemon prio=10 tid=0x00007fc124269800 nid=0x53f6 waiting on condition [0x00007fc0fbd7c000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x000000070118a800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@547975288{addrSelf=0.0.0.0/0.0.0.0:46974}" daemon prio=10 tid=0x00007fc124074000 nid=0x53e9 waiting on condition [0x00007fc0f9b5a000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x00000007011872c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@1042570477{addrSelf=0.0.0.0/0.0.0.0:47755}" daemon prio=10 tid=0x00007fc124062000 nid=0x53dc waiting on condition [0x00007fc1093e4000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x00000007011849c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@1065067423{addrSelf=0.0.0.0/0.0.0.0:59181}" daemon prio=10 tid=0x00007fc124010000 nid=0x53cf waiting on condition [0x00007fc0fad6c000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x000000070116ca20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@1951317107{addrSelf=0.0.0.0/0.0.0.0:48990}" daemon prio=10 tid=0x00007fc12400d000 nid=0x53c2 waiting on condition [0x00007fc0f8645000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x00000007011708b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        "com.bigdata.ha.pipeline.HAReceiveService@1153334631{addrSelf=0.0.0.0/0.0.0.0:40404}" daemon prio=10 tid=0x00007fc124049800 nid=0x53b5 waiting on condition [0x00007fc1030ef000]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0x000000070116fcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        	at com.bigdata.ha.pipeline.HAReceiveService.runNoBlock(HAReceiveService.java:540)
        	at com.bigdata.ha.pipeline.HAReceiveService.run(HAReceiveService.java:468)
        
        
      • TestWriteCache.test_writeCacheStressScatteredWrites() does not release its ".tmp" file.
      • TestWriteCache.test_writeCacheChecksums() does not release its ".tmp" file.
      • sparql-1.1-evaluation-XXXX and sparql-evaluation-XXXX. These should be cleaned up by build.xml....
      • Lots of things use File.deleteOnExit(). All should be fixed.
      • Deadlock in com.bigdata.quorum.TestHA3QuorumSemantics.test_pipelineReorganization. See https://www.dropbox.com/sh/apua6idtlhvkrsl/sHZ6YUsWyu
            Caused by: com.bigdata.quorum.QuorumException: Concurrent set of token: expected=0, actual=-1
            [junit] 	at com.bigdata.quorum.AbstractQuorum$QuorumActorBase.conditionalSetToken(AbstractQuorum.java:2227)
            [junit]     at com.bigdata.quorum.AbstractQuorum$QuorumActorBase.access$1600(AbstractQuorum.java:1519)
            [junit] 	at com.bigdata.quorum.AbstractQuorum$QuorumActorBase$SetTokenTask.doAction(AbstractQuorum.java:1827)
            [junit] 	at com.bigdata.quorum.AbstractQuorum$QuorumActorBase$ActorTask.call(AbstractQuorum.java:1576)
            [junit] 	at com.bigdata.quorum.AbstractQuorum$QuorumActorBase$ActorTask.call(AbstractQuorum.java:1568)
            [junit] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            [junit] 	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            [junit] 	at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
            [junit] 	... 3 more
        

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            martyncutcher martyncutcher
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated: