Details

    • Type: New Feature
    • Status: Accepted
    • Resolution: Unresolved
    • Affects Version/s: BIGDATA_RELEASE_1_2_0
    • Fix Version/s: None
    • Component/s: Query Plan Generator
    • Labels:
      None

      Description

      Per [1], here are some interesting breakdowns of the total time:

      - 23% Parsing SPARQL queries
      - 17% Executing SPARQL queries (optimization plus evaluation)
      - 14% Query optimization (it looks like we spend MUCH more time optimizing queries than we do evaluating them).
      

      It looks like there is a lot of fat in the query optimization phase and perhaps in the query parser (which is known to drive the heap heavily).

      A big driver of the HEAP is the iterator() methods. That is coming out of AbstractList.listIterator(), Collections$SynchronizedCollection.iterator(), AbstractSequentialList.iterator(), and ModifiableBOpBase$NotifyingList.iterator().

      "Name","Objects","Size","Level"
      "java.util.AbstractList.listIterator()","12230","390928", "1"
      "java.util.Collections$SynchronizedCollection.iterator()","8016","256512", "2"
      "java.util.AbstractSequentialList.iterator()","2771","88672", "2"
      "com.bigdata.bop.ModifiableBOpBase$NotifyingList.iterator()","1321","42272", "2"
      

      Some of the big drivers for those iterator methods are:

      - com.bigdata.bop.BOpUtility.preOrderIterator2(int, BOp)
      - com.bigdata.bop.BOpUtility.annotationOpIterator(BOp)
      

      Striterator.addFilter() shows up BIG with back traces through BOpUtility$Expand, which is part of the same iteration pattern.

      This suggests that we could win big if we could improve our iteration patterns over the AST.

      [1] https://sourceforge.net/apps/trac/bigdata/ticket/546 (Index cache for Journal)

        Issue Links

          Activity

          beebs Brad Bebee created issue -
          bryanthompson bryanthompson made changes -
          Field Original Value New Value
          Assignee mikepersonick [ mikepersonick ] michaelschmidt [ michaelschmidt ]
          beebs Brad Bebee made changes -
          Workflow Trac Import v2 [ 12489 ] Trac Import v3 [ 13208 ]
          beebs Brad Bebee made changes -
          Workflow Trac Import v3 [ 13208 ] Trac Import v4 [ 14537 ]
          bryanthompson bryanthompson made changes -
          Link This issue relates to BLZG-1351 [ BLZG-1351 ]
          beebs Brad Bebee made changes -
          Workflow Trac Import v4 [ 14537 ] Trac Import v5 [ 15888 ]
          michaelschmidt michaelschmidt made changes -
          Status Open [ 1 ] Accepted [ 10101 ]
          beebs Brad Bebee made changes -
          Workflow Trac Import v5 [ 15888 ] Trac Import v6 [ 18411 ]
          beebs Brad Bebee made changes -
          Workflow Trac Import v6 [ 18411 ] Trac Import v7 [ 19830 ]
          beebs Brad Bebee made changes -
          Workflow Trac Import v7 [ 19830 ] Trac Import v8 [ 21520 ]

            People

            • Assignee:
              michaelschmidt michaelschmidt
              Reporter:
              bryanthompson bryanthompson
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: