Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-641 Improve load performance
  3. BLZG-1524

Expose more parallelism in lexicon index writes

    XMLWordPrintable

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The standard code path writes on the lexicon relation and then on the statement relation. The lexicon relation writes on the forward indices (TERM2ID, BLOBS) in parallel and then on the ID2TERM index. Then on the statement indices. However the ID2TERM index can be written in parallel with the statement indices.

      In fact, we are currently doing all the lexicon writes in sequence, which is bad:

      1. BLOBS (and SEARCH if enabled, in sequence).
      2. TERMS (TERM2ID then ID2TERM then SEARCH)
      3. SEARCH

      Note that the last SEARCH actually appears to be redundant?!? with the others. I've copied the specific code block from the end of the public addTerms() method below.

       public long addTerms(final BigdataValue[] values, final int numTerms,
                  final boolean readOnly) {
      
      // ....
      
              if (this.textIndex && textIndex.size() > 0) {
                  
                  try {
                      
                      stats.fullTextIndexTime
                          .addAndGet(new FullTextIndexWriterTask(
                                  getSearchEngine(), textIndex.size()/* capacity */, 
                                  textIndex.iterator())
                                  .call());
      
                  } catch (Exception ex) {
                      
                      throw new RuntimeException(ex);
                      
                  }
                  
              }
      
      

        Attachments

          Activity

            People

            Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: