Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1167

Adapt blueprints integration for group commit?

    Details

    • Type: New Feature
    • Status: In Progress
    • Resolution: Unresolved
    • Affects Version/s: BIGDATA_RELEASE_1_5_0
    • Fix Version/s: None
    • Component/s: blueprints
    • Labels:
      None

      Description

      The current blueprints API implementation uses BigdataSail or BigdataRepository operations. These are not aware of the group commit API. They could be made aware by modifying them to use AbstractApiTask. Or this could remain an embedded interface without group commit support, similar to the direct use of the Sail APIs. In both cases you need to use AbstractApiTask to wrap up any requests within the MVCC concurrency manager.

      See BLZG-670 (NSS GROUP COMMIT)

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        Mike, would you please take a look at this branch and see if you can identify the problem with the test setup.

        >>> BLUEPRINTS_TEST_SUITE_REFACTOR

        Thanks,
        Bryan

        Show
        bryanthompson bryanthompson added a comment - Mike, would you please take a look at this branch and see if you can identify the problem with the test setup. >>> BLUEPRINTS_TEST_SUITE_REFACTOR Thanks, Bryan
        Hide
        bryanthompson bryanthompson added a comment -

        I have resolved the GRS index cache problem. See BLZG-1174. I have merged this changes into the BLUEPRINTS_TEST_SUITE_REFACTOR branch.

        @MikeP: Please review the test suite in the BLUEPRINTS_TEST_SUITE_REFACTOR.

        Here is the exception with the modified test suite (with group commit disabled). This looks like a failure to go through a commit point at some point. Or perhaps a re-opening issue:

        WARN : 298350      main com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:189): Testing testGraphDataPersists...
        	bigdatagraphclient: 1 graph loaded in 26.262939453125ms
        junit.framework.AssertionFailedError: expected:<0> but was:<2>
        	at junit.framework.Assert.fail(Assert.java:57)
        	at junit.framework.Assert.failNotEquals(Assert.java:329)
        	at junit.framework.Assert.assertEquals(Assert.java:78)
        	at junit.framework.Assert.assertEquals(Assert.java:234)
        	at junit.framework.Assert.assertEquals(Assert.java:241)
        	at junit.framework.TestCase.assertEquals(TestCase.java:409)
        	at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:191)
        	at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:165)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at junit.framework.TestCase.runTest(TestCase.java:176)
        	at junit.framework.TestCase.runBare(TestCase.java:141)
        	at junit.framework.TestResult$1.protect(TestResult.java:122)
        	at junit.framework.TestResult.runProtected(TestResult.java:142)
        	at junit.framework.TestResult.run(TestResult.java:125)
        	at junit.framework.TestCase.run(TestCase.java:129)
        	at junit.framework.TestSuite.runTest(TestSuite.java:255)
        	at junit.framework.TestSuite.run(TestSuite.java:250)
        	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
        	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
        	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
        

        The problem is identical when group commit is enabled @ line 109 of the test suite:

        WARN : 262253      main com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:189): Testing testGraphDataPersists...
        	bigdatagraphclient: 1 graph loaded in 29.2470703125ms
        junit.framework.AssertionFailedError: expected:<0> but was:<2>
        	at junit.framework.Assert.fail(Assert.java:57)
        	at junit.framework.Assert.failNotEquals(Assert.java:329)
        	at junit.framework.Assert.assertEquals(Assert.java:78)
        	at junit.framework.Assert.assertEquals(Assert.java:234)
        	at junit.framework.Assert.assertEquals(Assert.java:241)
        	at junit.framework.TestCase.assertEquals(TestCase.java:409)
        	at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:191)
        	at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:165)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at junit.framework.TestCase.runTest(TestCase.java:176)
        	at junit.framework.TestCase.runBare(TestCase.java:141)
        	at junit.framework.TestResult$1.protect(TestResult.java:122)
        	at junit.framework.TestResult.runProtected(TestResult.java:142)
        	at junit.framework.TestResult.run(TestResult.java:125)
        	at junit.framework.TestCase.run(TestCase.java:129)
        	at junit.framework.TestSuite.runTest(TestSuite.java:255)
        	at junit.framework.TestSuite.run(TestSuite.java:250)
        	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
        	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
        	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
        
        Show
        bryanthompson bryanthompson added a comment - I have resolved the GRS index cache problem. See BLZG-1174 . I have merged this changes into the BLUEPRINTS_TEST_SUITE_REFACTOR branch. @MikeP: Please review the test suite in the BLUEPRINTS_TEST_SUITE_REFACTOR. Here is the exception with the modified test suite (with group commit disabled). This looks like a failure to go through a commit point at some point. Or perhaps a re-opening issue: WARN : 298350 main com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:189): Testing testGraphDataPersists... bigdatagraphclient: 1 graph loaded in 26.262939453125ms junit.framework.AssertionFailedError: expected:<0> but was:<2> at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:191) at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:165) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) The problem is identical when group commit is enabled @ line 109 of the test suite: WARN : 262253 main com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:189): Testing testGraphDataPersists... bigdatagraphclient: 1 graph loaded in 29.2470703125ms junit.framework.AssertionFailedError: expected:<0> but was:<2> at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:191) at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:165) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
        Hide
        bryanthompson bryanthompson added a comment -

        I've pushed the changes above into a BLUEPRINTS_TEST_SUITE_REFACTOR branch. I am hitting an issue with durable data after refactoring the test suite. This only occurs for one of the suites and even then only for the embedded client. I suspect a life cycle issue with the test fixtures.

        	at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760)
        

        The full stack trace is:

        java.lang.reflect.InvocationTargetException
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:187)
        	at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:161)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at junit.framework.TestCase.runTest(TestCase.java:176)
        	at junit.framework.TestCase.runBare(TestCase.java:141)
        	at junit.framework.TestResult$1.protect(TestResult.java:122)
        	at junit.framework.TestResult.runProtected(TestResult.java:142)
        	at junit.framework.TestResult.run(TestResult.java:125)
        	at junit.framework.TestCase.run(TestCase.java:129)
        	at junit.framework.TestSuite.runTest(TestSuite.java:255)
        	at junit.framework.TestSuite.run(TestSuite.java:250)
        	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
        	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
        	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
        	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
        Caused by: junit.framework.AssertionFailedError: expected:<0> but was:<2>
        	at junit.framework.Assert.fail(Assert.java:57)
        	at junit.framework.Assert.failNotEquals(Assert.java:329)
        	at junit.framework.Assert.assertEquals(Assert.java:78)
        	at junit.framework.Assert.assertEquals(Assert.java:234)
        	at junit.framework.Assert.assertEquals(Assert.java:241)
        	at junit.framework.TestCase.assertEquals(TestCase.java:409)
        	at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760)
        	... 25 more
        

        Assigned to MikeP to review and discuss. I will then take back the issue to resolve the GSR index isolation problem.

        Show
        bryanthompson bryanthompson added a comment - I've pushed the changes above into a BLUEPRINTS_TEST_SUITE_REFACTOR branch. I am hitting an issue with durable data after refactoring the test suite. This only occurs for one of the suites and even then only for the embedded client. I suspect a life cycle issue with the test fixtures. at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760) The full stack trace is: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:187) at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:161) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) Caused by: junit.framework.AssertionFailedError: expected:<0> but was:<2> at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760) ... 25 more Assigned to MikeP to review and discuss. I will then take back the issue to resolve the GSR index isolation problem.
        Hide
        bryanthompson bryanthompson added a comment -

        I have refactored the test harness for the blueprints http client to not extent the sail proxy test suite mechanism.

        I have done a preliminary root cause analysis
        - see BLZG-1174. The underlying issue appears to be the cached of the GRS btree reference on the GlobalRowStoreHelper object tightly held by the Journal is not being updated when the CreateKBTask creates a GSR within the scope of an IsolatedActionJournal.

        A similar issue is documented in the constructor of the IsolatedActionJournal.

        We probably need to refactor the cache mechanisms for the GSR to support isolation by AbstractTask.

        Note: This does not imply that the blueprints http client is broken when group commit is enabled. It is probably fine. This problem is causing test harness failures.

        Show
        bryanthompson bryanthompson added a comment - I have refactored the test harness for the blueprints http client to not extent the sail proxy test suite mechanism. I have done a preliminary root cause analysis - see BLZG-1174 . The underlying issue appears to be the cached of the GRS btree reference on the GlobalRowStoreHelper object tightly held by the Journal is not being updated when the CreateKBTask creates a GSR within the scope of an IsolatedActionJournal. A similar issue is documented in the constructor of the IsolatedActionJournal. We probably need to refactor the cache mechanisms for the GSR to support isolation by AbstractTask. Note: This does not imply that the blueprints http client is broken when group commit is enabled. It is probably fine. This problem is causing test harness failures.

          People

          • Assignee:
            mikepersonick mikepersonick
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated: