Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-1285

Sticky index error after running concurrent update

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: BLAZEGRAPH_RELEASE_1_5_2
    • Component/s: None
    • Labels:
      None

      Description

      After running updater for a while, got this error:

      ERROR: AbstractTripleStore.java:1765: java.lang.IllegalStateException: Index is in error state
      java.lang.IllegalStateException: Index is in error state

      Full log attached.

      I also find this in /tmp:

      rw-rr- 1 blazegraph blazegraph 1.1G May 18 14:40 bigdata3104270295584281148.tmp
      rw-rr- 1 blazegraph blazegraph 33M May 18 14:40 bigdata3369438409198414383.tmp

      1. log.1431812556.gz
        32 kB
        stasmalyshev
      2. log.1431902533.gz
        518 kB
        stasmalyshev

        Issue Links

          Activity

          Hide
          bryanthompson bryanthompson added a comment -

          The root cause was the RDR truth maintenance logic for the statements about statements during retraction of assertions.

          Show
          bryanthompson bryanthompson added a comment - The root cause was the RDR truth maintenance logic for the statements about statements during retraction of assertions.
          Hide
          stasmalyshev stasmalyshev added a comment -

          Looks like it is indeed the case about the link to BLZG-1268: after patching out statementIdentifiers NPEs seem to be gone - or at least become much rarer, as I haven't seen one since the last restart of the service. I'll watch them for the next day or so but so far it appears like removing the RDR stuff eliminated them.

          Show
          stasmalyshev stasmalyshev added a comment - Looks like it is indeed the case about the link to BLZG-1268 : after patching out statementIdentifiers NPEs seem to be gone - or at least become much rarer, as I haven't seen one since the last restart of the service. I'll watch them for the next day or so but so far it appears like removing the RDR stuff eliminated them.
          Hide
          bryanthompson bryanthompson added a comment -

          Ah. Turn off RDR. It is the truth maintenance for the statements about statements that is creating those temporary files and is probably responsible for those triple store creates against the GRS (I will need to look at how this interaction is happening - I would not have expected it to be registering those temporary triple stores in the GRS).

          com.bigdata.rdf.sail.BigdataSailHelper has some code that you can look at for how to change the properties that are in effect for a triple store. You need to change from sids => triples).

          You can also just edit the code and make it return immediately.

              public IChunkedOrderedIterator<ISPO> computeClosureForStatementIdentifiers(
                      IChunkedOrderedIterator<ISPO> src) {
               
          *        if(!statementIdentifiers||true) {* // return immediately.
                      
                      // There will be no statement identifiers unless they were enabled.
                      
                      return src;
                      
                  }
          

          And change your properties file for your deployment to specify triples rather than RDR.

          We will track down the issue against the RDR mode and resolve it. But this will unblock you and will speed up the load significantly.

          Show
          bryanthompson bryanthompson added a comment - Ah. Turn off RDR. It is the truth maintenance for the statements about statements that is creating those temporary files and is probably responsible for those triple store creates against the GRS (I will need to look at how this interaction is happening - I would not have expected it to be registering those temporary triple stores in the GRS). com.bigdata.rdf.sail.BigdataSailHelper has some code that you can look at for how to change the properties that are in effect for a triple store. You need to change from sids => triples). You can also just edit the code and make it return immediately. public IChunkedOrderedIterator<ISPO> computeClosureForStatementIdentifiers( IChunkedOrderedIterator<ISPO> src) { * if (!statementIdentifiers|| true ) {* // return immediately. // There will be no statement identifiers unless they were enabled. return src; } And change your properties file for your deployment to specify triples rather than RDR. We will track down the issue against the RDR mode and resolve it. But this will unblock you and will speed up the load significantly.
          Hide
          stasmalyshev stasmalyshev added a comment -

          For the more info about the app - the updater app is pretty basic. The workflow is as follows:
          1. It downloads the list of recent changes from wikidata.org and extract list of entity IDs (looking like Q12345 etc.)
          2. It uses SELECT to extract info about current revision number for each entity.
          3. If the number is old, it starts the update sequence, as follows:
          3.1. Extracts some data on dependent values for the entity with SELECT
          3.2 Runs a series of DELETE queries to remove old data for the entity.
          3.3. Runs an INSERT query that inserts the new data for the entity.
          3.4. Runs another DELETE query to clean up data that may not be in use anymore like old value/reference nodes.

          All queries use REST API. Steps 3.2 and 3.3 are done as one big query sent to the API. Steps 3.x are done in parallel (10 threads currently) so that each entity ID is assigned to a thread that does all the operations for it. I can provide the details on the queries used if needed. The query that shows in backtrace leading to Store.create seems to be one of the queries run in step 3.2.

          Just occurred to me - can it happen that VALUES clause causes AbstractTripleStore.create? It seems to come from computeClosureForStatementIdentifiers which creates some kind of temp store, but not sure what it means.

          Show
          stasmalyshev stasmalyshev added a comment - For the more info about the app - the updater app is pretty basic. The workflow is as follows: 1. It downloads the list of recent changes from wikidata.org and extract list of entity IDs (looking like Q12345 etc.) 2. It uses SELECT to extract info about current revision number for each entity. 3. If the number is old, it starts the update sequence, as follows: 3.1. Extracts some data on dependent values for the entity with SELECT 3.2 Runs a series of DELETE queries to remove old data for the entity. 3.3. Runs an INSERT query that inserts the new data for the entity. 3.4. Runs another DELETE query to clean up data that may not be in use anymore like old value/reference nodes. All queries use REST API. Steps 3.2 and 3.3 are done as one big query sent to the API. Steps 3.x are done in parallel (10 threads currently) so that each entity ID is assigned to a thread that does all the operations for it. I can provide the details on the queries used if needed. The query that shows in backtrace leading to Store.create seems to be one of the queries run in step 3.2. Just occurred to me - can it happen that VALUES clause causes AbstractTripleStore.create? It seems to come from computeClosureForStatementIdentifiers which creates some kind of temp store, but not sure what it means.
          Hide
          stasmalyshev stasmalyshev added a comment -

          Restart of the DB seems to bring it back to normal.

          Show
          stasmalyshev stasmalyshev added a comment - Restart of the DB seems to bring it back to normal.

            People

            • Assignee:
              bryanthompson bryanthompson
              Reporter:
              stasmalyshev stasmalyshev
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: