Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-727

NotMaterializedException when a SERVICE call needs variables that are provided as query input bindings

    Details

      Description

      We are observing query failures due to NotMaterializedExceptions for values that need to flow into an (openrdf based) SERVICE call.

      I will attach a testcase to this issue that demonstrates the problem. The testcase failes (testing with -DtestClass=com.bigdata.rdf.sail.TestBigdataSailWithQuads) with the following exception:

      org.openrdf.query.QueryEvaluationException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=a67985ab-4c11-452e-b819-e9c7082f7d65,bopId=1,partitionId=-1,sinkId=3,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at com.bigdata.rdf.sail.Bigdata2Sesame2BindingSetIterator.hasNext(Bigdata2Sesame2BindingSetIterator.java:119)
      	at org.openrdf.query.impl.TupleQueryResultImpl.hasNext(TupleQueryResultImpl.java:90)
      	at com.bigdata.rdf.sail.TestNotMaterialized.executeQuery(TestNotMaterialized.java:87)
      	at com.bigdata.rdf.sail.TestNotMaterialized.testServiceWithBindingArg(TestNotMaterialized.java:68)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at junit.framework.TestCase.runTest(TestCase.java:154)
      	at junit.framework.TestCase.runBare(TestCase.java:127)
      	at junit.framework.TestResult$1.protect(TestResult.java:106)
      	at junit.framework.TestResult.runProtected(TestResult.java:124)
      	at junit.framework.TestResult.run(TestResult.java:109)
      	at junit.framework.TestCase.run(TestCase.java:118)
      	at junit.framework.TestSuite.runTest(TestSuite.java:208)
      	at junit.framework.TestSuite.run(TestSuite.java:203)
      	at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
      	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
      Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=a67985ab-4c11-452e-b819-e9c7082f7d65,bopId=1,partitionId=-1,sinkId=3,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1490)
      	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1667)
      	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.hasNext(BlockingBuffer.java:1530)
      	at com.bigdata.striterator.AbstractChunkedResolverator._hasNext(AbstractChunkedResolverator.java:352)
      	at com.bigdata.striterator.AbstractChunkedResolverator.hasNext(AbstractChunkedResolverator.java:328)
      	at com.bigdata.rdf.sail.Bigdata2Sesame2BindingSetIterator.hasNext(Bigdata2Sesame2BindingSetIterator.java:100)
      	... 21 more
      Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=a67985ab-4c11-452e-b819-e9c7082f7d65,bopId=1,partitionId=-1,sinkId=3,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
      	at java.util.concurrent.FutureTask.get(Unknown Source)
      	at com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator.checkFuture(BlockingBuffer.java:1443)
      	... 26 more
      Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=a67985ab-4c11-452e-b819-e9c7082f7d65,bopId=1,partitionId=-1,sinkId=3,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at com.bigdata.rdf.sail.RunningQueryCloseableIterator.checkFuture(RunningQueryCloseableIterator.java:58)
      	at com.bigdata.rdf.sail.RunningQueryCloseableIterator.close(RunningQueryCloseableIterator.java:72)
      	at com.bigdata.rdf.sail.RunningQueryCloseableIterator.hasNext(RunningQueryCloseableIterator.java:81)
      	at com.bigdata.striterator.ChunkedWrappedIterator.hasNext(ChunkedWrappedIterator.java:193)
      	at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:218)
      	at com.bigdata.striterator.AbstractChunkedResolverator$ChunkConsumerTask.call(AbstractChunkedResolverator.java:1)
      	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: task=ChunkTask{query=a67985ab-4c11-452e-b819-e9c7082f7d65,bopId=1,partitionId=-1,sinkId=3,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at com.bigdata.util.concurrent.Haltable.get(Haltable.java:251)
      	at com.bigdata.bop.engine.AbstractRunningQuery.get(AbstractRunningQuery.java:1401)
      	at com.bigdata.bop.engine.AbstractRunningQuery.get(AbstractRunningQuery.java:1)
      	at com.bigdata.rdf.sail.RunningQueryCloseableIterator.checkFuture(RunningQueryCloseableIterator.java:45)
      	... 10 more
      Caused by: java.lang.Exception: task=ChunkTask{query=a67985ab-4c11-452e-b819-e9c7082f7d65,bopId=1,partitionId=-1,sinkId=3,altSinkId=null}, cause=java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1292)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTaskWrapper.run(ChunkedRunningQuery.java:828)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at com.bigdata.concurrent.FutureTaskMon.run(FutureTaskMon.java:63)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkFutureTask.run(ChunkedRunningQuery.java:729)
      	... 3 more
      Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
      	at java.util.concurrent.FutureTask.get(Unknown Source)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1272)
      	... 9 more
      Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
      	at java.util.concurrent.FutureTask.get(Unknown Source)
      	at com.bigdata.bop.controller.ServiceCallJoin$ChunkTask.doServiceCallWithConstant(ServiceCallJoin.java:339)
      	at com.bigdata.bop.controller.ServiceCallJoin$ChunkTask.call(ServiceCallJoin.java:291)
      	at com.bigdata.bop.controller.ServiceCallJoin$ChunkTask.call(ServiceCallJoin.java:1)
      	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at com.bigdata.bop.engine.ChunkedRunningQuery$ChunkTask.call(ChunkedRunningQuery.java:1271)
      	... 9 more
      Caused by: java.lang.RuntimeException: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at com.bigdata.bop.controller.ServiceCallJoin$ChunkTask$ServiceCallTask.doServiceCall(ServiceCallJoin.java:722)
      	at com.bigdata.bop.controller.ServiceCallJoin$ChunkTask$ServiceCallTask.call(ServiceCallJoin.java:603)
      	at com.bigdata.bop.controller.ServiceCallJoin$ChunkTask$ServiceCallTask.call(ServiceCallJoin.java:1)
      	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	... 3 more
      Caused by: com.bigdata.rdf.internal.NotMaterializedException: var=bool1, val=XSDBoolean(true)
      	at com.bigdata.rdf.sparql.ast.service.ServiceCallUtility.bigdata2Openrdf(ServiceCallUtility.java:197)
      	at com.bigdata.rdf.sparql.ast.service.ServiceCallUtility.convert(ServiceCallUtility.java:290)
      	at com.bigdata.bop.controller.ServiceCallJoin$ChunkTask$ServiceCallTask.doNonBigdataServiceCall(ServiceCallJoin.java:778)
      	at com.bigdata.bop.controller.ServiceCallJoin$ChunkTask$ServiceCallTask.doExternalServiceCall(ServiceCallJoin.java:746)
      	at com.bigdata.bop.controller.ServiceCallJoin$ChunkTask$ServiceCallTask.doServiceCall(ServiceCallJoin.java:687)
      	... 7 more
      Caused by: com.bigdata.rdf.internal.NotMaterializedException: XSDBoolean(true)
      	at com.bigdata.rdf.internal.impl.AbstractIV.getValue(AbstractIV.java:887)
      	at com.bigdata.rdf.sparql.ast.service.ServiceCallUtility.bigdata2Openrdf(ServiceCallUtility.java:189)
      	... 11 more
      
      

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        Added unit test for [1] from the ticket.

        Note: The test is not yet integrated into CI since it is not yet known whether the test will pass once the NotMaterializedException problem has been resolved.

        The NotMaterializedException is being through due to a getValue() call without a proceeding setValue() call on an inline IV (XSD Boolean in this case). The logic that sets the IV Cache needs to be checked to decide whether we want to set the IV Cache for inline IVs or make the code invoke asValue() rather than getValue() when the IV is inline.

        Committed revision r6813.

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/632 (NotMaterializedException in SERVICE call)

        Show
        bryanthompson bryanthompson added a comment - Added unit test for [1] from the ticket. Note: The test is not yet integrated into CI since it is not yet known whether the test will pass once the NotMaterializedException problem has been resolved. The NotMaterializedException is being through due to a getValue() call without a proceeding setValue() call on an inline IV (XSD Boolean in this case). The logic that sets the IV Cache needs to be checked to decide whether we want to set the IV Cache for inline IVs or make the code invoke asValue() rather than getValue() when the IV is inline. Committed revision r6813. [1] https://sourceforge.net/apps/trac/bigdata/ticket/632 (NotMaterializedException in SERVICE call)
        Hide
        bryanthompson bryanthompson added a comment -

        I am assigning this to MikeP to review the IVCache pattern. We can then discuss whether setValue() should have been called or whether the code should be conditionally calling asValue() for inline IVs.

        Show
        bryanthompson bryanthompson added a comment - I am assigning this to MikeP to review the IVCache pattern. We can then discuss whether setValue() should have been called or whether the code should be conditionally calling asValue() for inline IVs.
        Hide
        bryanthompson bryanthompson added a comment -

        Here is a workaround:
        ServiceCallUtility:

            static public BindingSet bigdata2Openrdf(final LexiconRelation lex, final Set<IVariable<?>> vars,
                    final IBindingSet in) {
        
                final MapBindingSet out = new MapBindingSet();
                
                @SuppressWarnings("rawtypes")
                final Iterator<Map.Entry<IVariable,IConstant>> itr = in.iterator();
                
                while(itr.hasNext()) {
                
                    @SuppressWarnings("rawtypes")
                    final Map.Entry<IVariable,IConstant> e = itr.next();
        
                    final IVariable<?> var = e.getKey();
                    
                    if (vars != null && !vars.contains(var)) {
        
                        // This variable is not being projected.
                        continue;
                        
                    }
                    
                    final String name = var.getName();
        
                    @SuppressWarnings("rawtypes")
                    final IV iv = (IV) e.getValue().get();
        
                    final BigdataValue value;
                    
                    if (iv.isInline()) {
        
                        value = iv.asValue(lex);
                        
                    } else {
                        
                        try {
        
                            value = iv.getValue();
        
                        } catch (NotMaterializedException ex) {
        
                            /*
                             * Add the variable name to the stack trace.
                             */
        
                            throw new NotMaterializedException("var=" + name + ", val="
                                    + iv, ex);
        
                        }
                    }
        
                    out.addBinding(name, value);
        
                }
                
                return out;
        
            }
        

        You also need to modify the methods calling this method to pass down the LexiconRelation. In ServiceCallJoin:

                    private ICloseableIterator<IBindingSet> doNonBigdataServiceCall(
                            final ServiceCall<BindingSet> serviceCall,
                            final IBindingSet left[]) throws Exception {
        
                        final LexiconRelation lex = db.getLexiconRelation(); // <=== NEW LINE
                        
                        // Convert IBindingSet[] to openrdf BindingSet[].
                        final BindingSet[] left2 = ServiceCallUtility.convert(
                                lex, projectedVars, left);
        

        and also modify ServiceCallUtility.convert() to pass through the LexiconRelation.

        This workaround makes the test pass for me. The change is NOT committed because we still need to decide whether this should have been handled in the setup rather than in this method.

        Show
        bryanthompson bryanthompson added a comment - Here is a workaround: ServiceCallUtility: static public BindingSet bigdata2Openrdf(final LexiconRelation lex, final Set<IVariable<?>> vars, final IBindingSet in) { final MapBindingSet out = new MapBindingSet(); @SuppressWarnings("rawtypes") final Iterator<Map.Entry<IVariable,IConstant>> itr = in.iterator(); while(itr.hasNext()) { @SuppressWarnings("rawtypes") final Map.Entry<IVariable,IConstant> e = itr.next(); final IVariable<?> var = e.getKey(); if (vars != null && !vars.contains(var)) { // This variable is not being projected. continue; } final String name = var.getName(); @SuppressWarnings("rawtypes") final IV iv = (IV) e.getValue().get(); final BigdataValue value; if (iv.isInline()) { value = iv.asValue(lex); } else { try { value = iv.getValue(); } catch (NotMaterializedException ex) { /* * Add the variable name to the stack trace. */ throw new NotMaterializedException("var=" + name + ", val=" + iv, ex); } } out.addBinding(name, value); } return out; } You also need to modify the methods calling this method to pass down the LexiconRelation. In ServiceCallJoin: private ICloseableIterator<IBindingSet> doNonBigdataServiceCall( final ServiceCall<BindingSet> serviceCall, final IBindingSet left[]) throws Exception { final LexiconRelation lex = db.getLexiconRelation(); // <=== NEW LINE // Convert IBindingSet[] to openrdf BindingSet[]. final BindingSet[] left2 = ServiceCallUtility.convert( lex, projectedVars, left); and also modify ServiceCallUtility.convert() to pass through the LexiconRelation. This workaround makes the test pass for me. The change is NOT committed because we still need to decide whether this should have been handled in the setup rather than in this method.
        Hide
        bryanthompson bryanthompson added a comment -

        Applied fix as documented above.

        Committed revision r6816.

        Show
        bryanthompson bryanthompson added a comment - Applied fix as documented above. Committed revision r6816.

          People

          • Assignee:
            mikepersonick mikepersonick
            Reporter:
            gjdev gjdev
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: