Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-505

DataLoader fails to load resources available from the classpath.

    Details

      Description

      yifeng wrote:
      I built the bigdata 1.0.1 (https://bigdata.svn.sourceforge.net/svnroot/bigdata/tags/BIGDATA_RELEASE_1_0_1) but encountered a regression.

      The problem is in the method loadData2 (line 735) in class com.bigdata.rdf.store.DataLoader:

      If the resource to be loaded is available from the package location/classpath, i.e. if rdfStream is NOT null on line 745, the method actually does nothing due to the misplaced closing bracket.

      This problem was not present in 0.84.0 branch. The closing bracket was apparently moved in 1.0.1 and hence the regression.

        Activity

        Hide
        yifengc yifengc added a comment -

        Hi Bryan, we are starting full P&S benchmark tests in our product with the 1.0.1 build. Since the fix should be trivial, can we get the fix checked in asap to unblock our P&S?
        Thanks!

        -Yifeng

        Show
        yifengc yifengc added a comment - Hi Bryan, we are starting full P&S benchmark tests in our product with the 1.0.1 build. Since the fix should be trivial, can we get the fix checked in asap to unblock our P&S? Thanks! -Yifeng
        Hide
        bryanthompson bryanthompson added a comment -

        Yifeng, I've made the change and reformatted the code block to avoid this problem in the future. Can you please verify that this fixes the problem for you? The change is checked in against [1] and is also in the development branch [2].

        Thanks,
        Bryan

        [1] https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_0_0
        [2] https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/TERMS_REFACTOR_BRANCH

        Committed revision r5081.

        Show
        bryanthompson bryanthompson added a comment - Yifeng, I've made the change and reformatted the code block to avoid this problem in the future. Can you please verify that this fixes the problem for you? The change is checked in against [1] and is also in the development branch [2] . Thanks, Bryan [1] https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BIGDATA_RELEASE_1_0_0 [2] https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/TERMS_REFACTOR_BRANCH Committed revision r5081.
        Hide
        bryanthompson bryanthompson added a comment -

        We will do a dot release for this change to support downstream CI. We plan to do the dot release next week. As a first step, I will have to compile a change log against the 1.0.0 maintenance branch. If the change set is relatively small, then we will be able to issue the dot release without performance testing.

        Show
        bryanthompson bryanthompson added a comment - We will do a dot release for this change to support downstream CI. We plan to do the dot release next week. As a first step, I will have to compile a change log against the 1.0.0 maintenance branch. If the change set is relatively small, then we will be able to issue the dot release without performance testing.
        Hide
        bryanthompson bryanthompson added a comment -

        Raised priority since we will be doing a 1.0.2 release which incorporates this change and rolls in any other changes in the maintenance branch since the 1.0.1 release.

        Show
        bryanthompson bryanthompson added a comment - Raised priority since we will be doing a 1.0.2 release which incorporates this change and rolls in any other changes in the maintenance branch since the 1.0.1 release.
        Hide
        bryanthompson bryanthompson added a comment -

        The following is a list of tickets which (in addition to this ticket) have been resolved since the 1.0.1 release and which will be part of the 1.0.2 release.

        ticket	summary
        356	Query not terminated by error
        359	NamedGraph pattern fails to bind graph variable if only one binding exists.
        361	Code review of openrdf CloseableIterations and layering of IRunningQuery.iterator()
        

        The changes sets for 361/356 were related and were moderately extensive. However, I believe that we have already benchmarked those change sets in order to validate them. I will check the notes on the benchmark machine to verify this. If true, then we can cut a release with relatively little overhead.

        Show
        bryanthompson bryanthompson added a comment - The following is a list of tickets which (in addition to this ticket) have been resolved since the 1.0.1 release and which will be part of the 1.0.2 release. ticket summary 356 Query not terminated by error 359 NamedGraph pattern fails to bind graph variable if only one binding exists. 361 Code review of openrdf CloseableIterations and layering of IRunningQuery.iterator() The changes sets for 361/356 were related and were moderately extensive. However, I believe that we have already benchmarked those change sets in order to validate them. I will check the notes on the benchmark machine to verify this. If true, then we can cut a release with relatively little overhead.
        Hide
        bryanthompson bryanthompson added a comment -

        Here are the performance results from benchmarking BSBM v3 on the 1.0.1 maintenance branch after change sets for ticketBLZG-115 [1].

        w/o Q5 (reduced query mix)

        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   35134.51 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   34288.52 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   34991.39 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   34481.22 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   34824.14 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   34954.71 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   35262.16 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   36590.29 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        

        w/ Q5 (full query mix)

        QMpH:                   6282.60 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   6354.30 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   6314.14 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   6387.82 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        QMpH:                   6378.44 query mixes per hour
        [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH
        

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/361 (IRunningQuery does not close source and iterator/iteration close patterns).

        Show
        bryanthompson bryanthompson added a comment - Here are the performance results from benchmarking BSBM v3 on the 1.0.1 maintenance branch after change sets for ticketBLZG-115 [1] . w/o Q5 (reduced query mix) [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 35134.51 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 34288.52 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 34991.39 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 34481.22 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 34824.14 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 34954.71 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 35262.16 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 36590.29 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH w/ Q5 (full query mix) QMpH: 6282.60 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 6354.30 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 6314.14 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 6387.82 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH QMpH: 6378.44 query mixes per hour [root@bigdata01 bsbm-trunk]# ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 8 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH [1] https://sourceforge.net/apps/trac/bigdata/ticket/361 (IRunningQuery does not close source and iterator/iteration close patterns).
        Hide
        bryanthompson bryanthompson added a comment -

        Here are the benchmark results for LUBM U50 for the same change set:

             [java] Load: 6890949 stmts added in 154.82 secs, rate= 44508, commitLatency=0ms
             [java] Computing closure.
             [java] INFO : 158003      main com.bigdata.rdf.store.DataLoader.main(DataLoader.java:1453): Computing closure.
             [java] Closure: ClosureStats{mutationCount=1699274, elapsed=82703ms, rate=20546}
             [java] INFO : 240752      main com.bigdata.rdf.store.DataLoader.main(DataLoader.java:1460): Closure: ClosureStats{mutationCount=1699274, elapsed=82703ms, rate=20546}
             [java] Wrote: 715784192 bytes.
             [java] Total elapsed=243479ms
             [java] INFO : 243350      main com.bigdata.rdf.store.DataLoader.main(DataLoader.java:1502): Total elapsed=243479ms
        BUILD SUCCESSFUL
        Total time: 4 minutes 4 seconds
        
        818M on the disk (note: 673M for TERMS branch).
        

        1st trial (immediately after load)

             [java] ### Finished testing BIGDATA_SPARQL_ENDPOINT ###
             [java] BIGDATA_SPARQL_ENDPOINT     #trials=10      #parallel=1
             [java] query       Time    Result#
             [java] query1      86      4
             [java] query3      46      6
             [java] query4      104     34
             [java] query5      82      719
             [java] query7      35      61
             [java] query8      273     6463
             [java] query10     37      0
             [java] query11     37      0
             [java] query12     36      0
             [java] query13     37      0
             [java] query14     2543    393730
             [java] query6      3053    430114
             [java] query2      710     130
             [java] query9      3640    8627
             [java] Total       10719
        

        2nd trial:

             [java] ### Finished testing BIGDATA_SPARQL_ENDPOINT ###
             [java] BIGDATA_SPARQL_ENDPOINT     #trials=10      #parallel=1
             [java] query       Time    Result#
             [java] query1      37      4
             [java] query3      27      6
             [java] query4      49      34
             [java] query5      47      719
             [java] query7      24      61
             [java] query8      164     6463
             [java] query10     36      0
             [java] query11     38      0
             [java] query12     40      0
             [java] query13     36      0
             [java] query14     2560    393730
             [java] query6      2811    430114
             [java] query2      590     130
             [java] query9      3583    8627
             [java] Total       10042
        
        Show
        bryanthompson bryanthompson added a comment - Here are the benchmark results for LUBM U50 for the same change set: [java] Load: 6890949 stmts added in 154.82 secs, rate= 44508, commitLatency=0ms [java] Computing closure. [java] INFO : 158003 main com.bigdata.rdf.store.DataLoader.main(DataLoader.java:1453): Computing closure. [java] Closure: ClosureStats{mutationCount=1699274, elapsed=82703ms, rate=20546} [java] INFO : 240752 main com.bigdata.rdf.store.DataLoader.main(DataLoader.java:1460): Closure: ClosureStats{mutationCount=1699274, elapsed=82703ms, rate=20546} [java] Wrote: 715784192 bytes. [java] Total elapsed=243479ms [java] INFO : 243350 main com.bigdata.rdf.store.DataLoader.main(DataLoader.java:1502): Total elapsed=243479ms BUILD SUCCESSFUL Total time: 4 minutes 4 seconds 818M on the disk (note: 673M for TERMS branch). 1st trial (immediately after load) [java] ### Finished testing BIGDATA_SPARQL_ENDPOINT ### [java] BIGDATA_SPARQL_ENDPOINT #trials=10 #parallel=1 [java] query Time Result# [java] query1 86 4 [java] query3 46 6 [java] query4 104 34 [java] query5 82 719 [java] query7 35 61 [java] query8 273 6463 [java] query10 37 0 [java] query11 37 0 [java] query12 36 0 [java] query13 37 0 [java] query14 2543 393730 [java] query6 3053 430114 [java] query2 710 130 [java] query9 3640 8627 [java] Total 10719 2nd trial: [java] ### Finished testing BIGDATA_SPARQL_ENDPOINT ### [java] BIGDATA_SPARQL_ENDPOINT #trials=10 #parallel=1 [java] query Time Result# [java] query1 37 4 [java] query3 27 6 [java] query4 49 34 [java] query5 47 719 [java] query7 24 61 [java] query8 164 6463 [java] query10 36 0 [java] query11 38 0 [java] query12 40 0 [java] query13 36 0 [java] query14 2560 393730 [java] query6 2811 430114 [java] query2 590 130 [java] query9 3583 8627 [java] Total 10042
        Hide
        bryanthompson bryanthompson added a comment -

        Based on these benchmarks we are Ok for a dot release. I've filed an issue for that here [1]. I have closed this issue since there is no more work to be done on the fix.

        [1] https://sourceforge.net/apps/trac/bigdata/ticket/374

        Show
        bryanthompson bryanthompson added a comment - Based on these benchmarks we are Ok for a dot release. I've filed an issue for that here [1] . I have closed this issue since there is no more work to be done on the fix. [1] https://sourceforge.net/apps/trac/bigdata/ticket/374

          People

          • Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: