[BLZG-1167] Adapt blueprints integration for group commit? - Blazegraph

Details

Type: New Feature
Status: In Progress
Resolution: Unresolved
Affects Version/s: BIGDATA_RELEASE_1_5_0
Fix Version/s: None
Component/s: blueprints
Labels:
None

Description

The current blueprints API implementation uses BigdataSail or BigdataRepository operations. These are not aware of the group commit API. They could be made aware by modifying them to use AbstractApiTask. Or this could remain an embedded interface without group commit support, similar to the direct use of the Sail APIs. In both cases you need to use AbstractApiTask to wrap up any requests within the MVCC concurrency manager.

See ~~BLZG-670~~ (NSS GROUP COMMIT)

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

bryanthompson added a comment - 01/Mar/15 3:45 PM

I have refactored the test harness for the blueprints http client to not extent the sail proxy test suite mechanism.

I have done a preliminary root cause analysis
- see ~~BLZG-1174~~. The underlying issue appears to be the cached of the GRS btree reference on the GlobalRowStoreHelper object tightly held by the Journal is not being updated when the CreateKBTask creates a GSR within the scope of an IsolatedActionJournal.

A similar issue is documented in the constructor of the IsolatedActionJournal.

We probably need to refactor the cache mechanisms for the GSR to support isolation by AbstractTask.

Note: This does not imply that the blueprints http client is broken when group commit is enabled. It is probably fine. This problem is causing test harness failures.

Show

bryanthompson added a comment - 01/Mar/15 3:45 PM I have refactored the test harness for the blueprints http client to not extent the sail proxy test suite mechanism. I have done a preliminary root cause analysis - see BLZG-1174 . The underlying issue appears to be the cached of the GRS btree reference on the GlobalRowStoreHelper object tightly held by the Journal is not being updated when the CreateKBTask creates a GSR within the scope of an IsolatedActionJournal. A similar issue is documented in the constructor of the IsolatedActionJournal. We probably need to refactor the cache mechanisms for the GSR to support isolation by AbstractTask. Note: This does not imply that the blueprints http client is broken when group commit is enabled. It is probably fine. This problem is causing test harness failures.

Hide

Permalink

bryanthompson added a comment - 01/Mar/15 3:57 PM

I've pushed the changes above into a BLUEPRINTS_TEST_SUITE_REFACTOR branch. I am hitting an issue with durable data after refactoring the test suite. This only occurs for one of the suites and even then only for the embedded client. I suspect a life cycle issue with the test fixtures.

	at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760)

The full stack trace is:

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:187)
	at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:161)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at junit.framework.TestCase.runBare(TestCase.java:141)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:129)
	at junit.framework.TestSuite.runTest(TestSuite.java:255)
	at junit.framework.TestSuite.run(TestSuite.java:250)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: junit.framework.AssertionFailedError: expected:<0> but was:<2>
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.failNotEquals(Assert.java:329)
	at junit.framework.Assert.assertEquals(Assert.java:78)
	at junit.framework.Assert.assertEquals(Assert.java:234)
	at junit.framework.Assert.assertEquals(Assert.java:241)
	at junit.framework.TestCase.assertEquals(TestCase.java:409)
	at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760)
	... 25 more

Assigned to MikeP to review and discuss. I will then take back the issue to resolve the GSR index isolation problem.

Show

bryanthompson added a comment - 01/Mar/15 3:57 PM I've pushed the changes above into a BLUEPRINTS_TEST_SUITE_REFACTOR branch. I am hitting an issue with durable data after refactoring the test suite. This only occurs for one of the suites and even then only for the embedded client. I suspect a life cycle issue with the test fixtures. at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760) The full stack trace is: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:187) at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:161) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) Caused by: junit.framework.AssertionFailedError: expected:<0> but was:<2> at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760) ... 25 more Assigned to MikeP to review and discuss. I will then take back the issue to resolve the GSR index isolation problem.

Hide

Permalink

bryanthompson added a comment - 03/Mar/15 6:03 PM

I have resolved the GRS index cache problem. See ~~BLZG-1174~~. I have merged this changes into the BLUEPRINTS_TEST_SUITE_REFACTOR branch.

@MikeP: Please review the test suite in the BLUEPRINTS_TEST_SUITE_REFACTOR.

Here is the exception with the modified test suite (with group commit disabled). This looks like a failure to go through a commit point at some point. Or perhaps a re-opening issue:

WARN : 298350      main com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:189): Testing testGraphDataPersists...
	bigdatagraphclient: 1 graph loaded in 26.262939453125ms
junit.framework.AssertionFailedError: expected:<0> but was:<2>
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.failNotEquals(Assert.java:329)
	at junit.framework.Assert.assertEquals(Assert.java:78)
	at junit.framework.Assert.assertEquals(Assert.java:234)
	at junit.framework.Assert.assertEquals(Assert.java:241)
	at junit.framework.TestCase.assertEquals(TestCase.java:409)
	at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:191)
	at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:165)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at junit.framework.TestCase.runBare(TestCase.java:141)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:129)
	at junit.framework.TestSuite.runTest(TestSuite.java:255)
	at junit.framework.TestSuite.run(TestSuite.java:250)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

The problem is identical when group commit is enabled @ line 109 of the test suite:

WARN : 262253      main com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:189): Testing testGraphDataPersists...
	bigdatagraphclient: 1 graph loaded in 29.2470703125ms
junit.framework.AssertionFailedError: expected:<0> but was:<2>
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.failNotEquals(Assert.java:329)
	at junit.framework.Assert.assertEquals(Assert.java:78)
	at junit.framework.Assert.assertEquals(Assert.java:234)
	at junit.framework.Assert.assertEquals(Assert.java:241)
	at junit.framework.TestCase.assertEquals(TestCase.java:409)
	at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:191)
	at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:165)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at junit.framework.TestCase.runBare(TestCase.java:141)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:129)
	at junit.framework.TestSuite.runTest(TestSuite.java:255)
	at junit.framework.TestSuite.run(TestSuite.java:250)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

Show

bryanthompson added a comment - 03/Mar/15 6:03 PM I have resolved the GRS index cache problem. See BLZG-1174 . I have merged this changes into the BLUEPRINTS_TEST_SUITE_REFACTOR branch. @MikeP: Please review the test suite in the BLUEPRINTS_TEST_SUITE_REFACTOR. Here is the exception with the modified test suite (with group commit disabled). This looks like a failure to go through a commit point at some point. Or perhaps a re-opening issue: WARN : 298350 main com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:189): Testing testGraphDataPersists... bigdatagraphclient: 1 graph loaded in 26.262939453125ms junit.framework.AssertionFailedError: expected:<0> but was:<2> at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:191) at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:165) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) The problem is identical when group commit is enabled @ line 109 of the test suite: WARN : 262253 main com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:189): Testing testGraphDataPersists... bigdatagraphclient: 1 graph loaded in 29.2470703125ms junit.framework.AssertionFailedError: expected:<0> but was:<2> at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) at com.tinkerpop.blueprints.GraphTestSuite.testGraphDataPersists(GraphTestSuite.java:760) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.bigdata.blueprints.TestBigdataGraphClient$BigdataGraphTest.doTestSuite(TestBigdataGraphClient.java:191) at com.bigdata.blueprints.TestBigdataGraphClient.testGraphSuite(TestBigdataGraphClient.java:165) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

Hide

Permalink

bryanthompson added a comment - 10/Apr/15 12:05 PM

Mike, would you please take a look at this branch and see if you can identify the problem with the test setup.

>>> BLUEPRINTS_TEST_SUITE_REFACTOR

Thanks,
Bryan

Show

bryanthompson added a comment - 10/Apr/15 12:05 PM Mike, would you please take a look at this branch and see if you can identify the problem with the test setup. >>> BLUEPRINTS_TEST_SUITE_REFACTOR Thanks, Bryan

People

Assignee:

mikepersonick

Reporter:

bryanthompson

Votes:

0 Vote for this issue

Watchers:

3 Start watching this issue

Dates

Created:

26/Feb/15 9:15 PM

Updated:

10/Apr/15 12:05 PM