Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-8851

Rest API bulk load slows down greatly

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Wikidata Query Service
    • Labels:
      None

      Description

      When trying to load data into Blazegraph from Wikidata RDF dump on one of the test machines, using REST API bulk load, the initial files are loaded reasonably fast, but then loading increasingly slows down. For example, first items produce:

      May 01 20:59:39 wdq-deploy bash[32644]: loading: 32541566 stmts added in 768.181 secs, rate= 42361, commitLatency=0ms, {failSet=0,goodSet=1}
      May 01 21:11:22 wdq-deploy bash[32644]: loading: 56495030 stmts added in 1471.33 secs, rate= 38397, commitLatency=0ms, {failSet=0,goodSet=4}
      

      but later items look like:

      May 05 22:06:00 wdq-deploy bash[32644]: loading: 1053973816 stmts added in 350340.908 secs, rate= 3008, commitLatency=0ms, {failSet=0,goodSet=216}
      May 06 02:06:18 wdq-deploy bash[32644]: loading: 1068153713 stmts added in 364759.295 secs, rate= 2928, commitLatency=0ms, {failSet=0,goodSet=217}
      May 06 06:34:30 wdq-deploy bash[32644]: loading: 1082622842 stmts added in 380850.778 secs, rate= 2842, commitLatency=0ms, {failSet=0,goodSet=218}
      

      The rate has slowed down almost 20 times, and for the last file the speed seems to be about 900 statement/second, while for the first one it was over 42k/s. Are there some settings that would allow to prevent this slowdown or decrease it? Should data be loaded in some other way?

        Attachments

          Activity

            People

            Assignee:
            bryanthompson bryanthompson
            Reporter:
            stasmalyshev stasmalyshev
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: