Details

      Description

      There are three different problems:

      1. The client should not interrupt the http submit of the request since the outcome of interrupting an HTTP request or RMI is unpredictable (this is a client code issue
      - the semantics of interrupt within Java are locally defined and there is no general protocol around how interrupt of NIO is handled).

      2. Long running queries that do not return the first result until the query is done need to be cancelled before that result is ready.

      3. There should be an easier way to cancel all queries associated with a UX page, especially when using the HA load balancer. In particular, there are an issues around how to cause queries to be cancelled that (A) are submitted concurrent with or immediately before the cancel of the queries for a page or that are being processed by the query optimizer by are not yet associated with IRunningQuery objects on the QueryEngine; or (B) would run to completion before returning their first result.

      Currently, the application can associate a UUID with each query and issue a CANCEL with that UUID. There are two drawbacks with this approach.


      - The application needs to track all in-flight queries.


      - The CANCEL is effective as of when it is received by the QueryEngine. If the query is in an indeterminate state (for example, received by the REST API, but not yet an IRunningQuery), then it will not be cancelled.

      ---

      Here is a list of some possibilities that could be explored for the cancellation of the submit of query that does not return any results until all work is done:


      - Return http status code as early as possible so the client can terminate the submitted query. The servlet engine might not flush that status line through to the client in a timely basis unless we flush the response. However, flushing the response can hurt the performance of the servlet engine. This would need to be explored.


      - REST redirect for long-running queries. This is a common pattern for long running operations. A client POSTs a request. The server responds with a redirect to a URL. The client follows the redirect to the URL using a GET and awaits the outcome of the process. Often, the client is given a means to interact with the computation, e.g., to cancel the process.

        Activity

        beebs Brad Bebee created issue -
        Hide
        bryanthompson bryanthompson added a comment -

        It may be that we can modify the RemoteRepository code to create and return the TupleQueryResult object before the HTTP status line is received by the client and then check that status code within the BackgroundTupleResult object. I will investigate this. However, there is still a problem on the server side if the client issues the CANCEL before the query is scheduled as an IRunningQuery on the QueryEngine. If we return the status code before that occurs, then the CANCEL would fail to find the query on the server and the query would run to completion.

        Show
        bryanthompson bryanthompson added a comment - It may be that we can modify the RemoteRepository code to create and return the TupleQueryResult object before the HTTP status line is received by the client and then check that status code within the BackgroundTupleResult object. I will investigate this. However, there is still a problem on the server side if the client issues the CANCEL before the query is scheduled as an IRunningQuery on the QueryEngine. If we return the status code before that occurs, then the CANCEL would fail to find the query on the server and the query would run to completion.
        Hide
        bryanthompson bryanthompson added a comment -

        The server should keep a list of recently canceled queries that were not running at the time of cancellation and were not in the pool of recently terminated queries. We can then check this LRU before we start a query and not start if it was recently cancel.

        However, there is a real danger of this LRU being a hotspot since we must test it for each new query that we start. Thus, we need to do something more sophisticated. The test against this LRU MUST allow high concurrency. At the same time, we MUST eventually evict entries from the LRU. This suggests an approximate LRU with good concurrency for readers. It is Ok if there is a single writer since we will only add entries if a CANCEL fails to discover the query in any of:


        - running queries (getRunningQueries() also tests the recently done queries)
        - running UPDATE requests.

        Show
        bryanthompson bryanthompson added a comment - The server should keep a list of recently canceled queries that were not running at the time of cancellation and were not in the pool of recently terminated queries. We can then check this LRU before we start a query and not start if it was recently cancel. However, there is a real danger of this LRU being a hotspot since we must test it for each new query that we start. Thus, we need to do something more sophisticated. The test against this LRU MUST allow high concurrency. At the same time, we MUST eventually evict entries from the LRU. This suggests an approximate LRU with good concurrency for readers. It is Ok if there is a single writer since we will only add entries if a CANCEL fails to discover the query in any of: - running queries (getRunningQueries() also tests the recently done queries) - running UPDATE requests.
        Hide
        bryanthompson bryanthompson added a comment -

        I have integrated a high concurrency LRU into the QueryEngine. This is used to track the UUID of an operation that has been CANCELed through the REST API but which is neither (a) currently running; nor (b) recently done. In this case, the UUID is entered into the new "pendingCancelLRU".

        The pendingCancelLRU is checked before we start a new query (all code paths) and before we start a SPARQL UPDATE operation (REST API). If the operation is found in the pendingCancelLRU, then the Future of that operation is cancelled. This occurs before the Future is submitted for evaluation to avoid a race in which the operation might complete before it was cancelled.

        Changes are to:


        - QueryEngine: added pendingCancelLRU and associated access methods. The pendingCancelLRU is checked for QUERY in startEval().
        - BigdataRDFContext: Handle pending CANCEL of UPDATE requests.
        - StatusServlet: Queue UUID in the pendingCancelLRU iff not found when CANCEL request is processed.

        The AST, SPARQL, and NSS test suites are good locally (but see BLZG-1071 for memory leaks in the test suite).

        Committed revision r8656. (Committed to CI)

        TODO Re-run the query performance benchmarks to look for a performance regression associated with this change set.

        Show
        bryanthompson bryanthompson added a comment - I have integrated a high concurrency LRU into the QueryEngine. This is used to track the UUID of an operation that has been CANCELed through the REST API but which is neither (a) currently running; nor (b) recently done. In this case, the UUID is entered into the new "pendingCancelLRU". The pendingCancelLRU is checked before we start a new query (all code paths) and before we start a SPARQL UPDATE operation (REST API). If the operation is found in the pendingCancelLRU, then the Future of that operation is cancelled. This occurs before the Future is submitted for evaluation to avoid a race in which the operation might complete before it was cancelled. Changes are to: - QueryEngine: added pendingCancelLRU and associated access methods. The pendingCancelLRU is checked for QUERY in startEval(). - BigdataRDFContext: Handle pending CANCEL of UPDATE requests. - StatusServlet: Queue UUID in the pendingCancelLRU iff not found when CANCEL request is processed. The AST, SPARQL, and NSS test suites are good locally (but see BLZG-1071 for memory leaks in the test suite). Committed revision r8656. (Committed to CI) TODO Re-run the query performance benchmarks to look for a performance regression associated with this change set.
        Hide
        bryanthompson bryanthompson added a comment -

        CI is good for this change set.

        LUBM U50 performance against r8656 is good.

        BSBM is currently running.

        Show
        bryanthompson bryanthompson added a comment - CI is good for this change set. LUBM U50 performance against r8656 is good. BSBM is currently running.
        Hide
        bryanthompson bryanthompson added a comment -

        Converged query performance on BSBM 100M looks ok

        root@bigdata11:~/workspace/bsbmtools/trunk# for a in `seq 120`; do ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 16 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH; done
        QMpH:                   17836.47 query mixes per hour
        QMpH:                   27681.61 query mixes per hour
        QMpH:                   34620.65 query mixes per hour
        QMpH:                   44397.75 query mixes per hour
        QMpH:                   46979.70 query mixes per hour
        QMpH:                   47220.10 query mixes per hour
        QMpH:                   47318.41 query mixes per hour
        QMpH:                   47523.29 query mixes per hour
        QMpH:                   47966.19 query mixes per hour
        QMpH:                   47852.73 query mixes per hour
        QMpH:                   48170.39 query mixes per hour
        QMpH:                   49295.09 query mixes per hour
        QMpH:                   47719.32 query mixes per hour
        QMpH:                   48102.25 query mixes per hour
        QMpH:                   49151.71 query mixes per hour
        QMpH:                   47703.47 query mixes per hour
        QMpH:                   48361.68 query mixes per hour
        QMpH:                   49284.32 query mixes per hour
        QMpH:                   48084.53 query mixes per hour
        QMpH:                   49390.49 query mixes per hour
        QMpH:                   48149.64 query mixes per hour
        QMpH:                   49488.86 query mixes per hour
        QMpH:                   48557.35 query mixes per hour
        QMpH:                   48891.98 query mixes per hour
        QMpH:                   50112.33 query mixes per hour
        QMpH:                   47801.18 query mixes per hour
        QMpH:                   49753.10 query mixes per hour
        QMpH:                   48498.07 query mixes per hour
        QMpH:                   49278.52 query mixes per hour
        QMpH:                   49951.89 query mixes per hour
        QMpH:                   48035.99 query mixes per hour
        QMpH:                   49353.69 query mixes per hour
        QMpH:                   50121.31 query mixes per hour
        QMpH:                   48172.97 query mixes per hour
        QMpH:                   49482.34 query mixes per hour
        QMpH:                   50306.86 query mixes per hour
        QMpH:                   48221.52 query mixes per hour
        QMpH:                   48879.38 query mixes per hour
        QMpH:                   49563.24 query mixes per hour
        QMpH:                   50400.67 query mixes per hour
        QMpH:                   47650.62 query mixes per hour
        QMpH:                   49621.94 query mixes per hour
        QMpH:                   50179.12 query mixes per hour
        QMpH:                   50121.66 query mixes per hour
        QMpH:                   46918.69 query mixes per hour
        QMpH:                   49477.48 query mixes per hour
        QMpH:                   50088.84 query mixes per hour
        QMpH:                   50605.66 query mixes per hour
        QMpH:                   47901.35 query mixes per hour
        QMpH:                   48879.08 query mixes per hour
        QMpH:                   49814.42 query mixes per hour
        QMpH:                   50175.20 query mixes per hour
        QMpH:                   49980.21 query mixes per hour
        QMpH:                   47639.71 query mixes per hour
        QMpH:                   49486.24 query mixes per hour
        QMpH:                   50109.79 query mixes per hour
        QMpH:                   50383.81 query mixes per hour
        QMpH:                   50609.05 query mixes per hour
        QMpH:                   47402.14 query mixes per hour
        QMpH:                   49600.20 query mixes per hour
        QMpH:                   50394.44 query mixes per hour
        QMpH:                   50746.85 query mixes per hour
        QMpH:                   50755.83 query mixes per hour
        QMpH:                   50615.72 query mixes per hour
        QMpH:                   48624.19 query mixes per hour
        QMpH:                   49690.24 query mixes per hour
        QMpH:                   49695.86 query mixes per hour
        QMpH:                   50672.48 query mixes per hour
        QMpH:                   50594.22 query mixes per hour
        QMpH:                   47457.06 query mixes per hour
        QMpH:                   49348.70 query mixes per hour
        QMpH:                   49903.95 query mixes per hour
        QMpH:                   50236.92 query mixes per hour
        QMpH:                   50662.19 query mixes per hour
        QMpH:                   50166.32 query mixes per hour
        QMpH:                   50733.54 query mixes per hour
        QMpH:                   46197.30 query mixes per hour
        QMpH:                   49405.59 query mixes per hour
        QMpH:                   50127.05 query mixes per hour
        QMpH:                   50688.06 query mixes per hour
        QMpH:                   50262.21 query mixes per hour
        QMpH:                   50725.86 query mixes per hour
        QMpH:                   50496.62 query mixes per hour
        QMpH:                   46305.06 query mixes per hour
        QMpH:                   49602.56 query mixes per hour
        QMpH:                   50328.03 query mixes per hour
        QMpH:                   50246.48 query mixes per hour
        QMpH:                   50190.79 query mixes per hour
        QMpH:                   50538.40 query mixes per hour
        QMpH:                   51030.92 query mixes per hour
        QMpH:                   50251.89 query mixes per hour
        QMpH:                   48282.93 query mixes per hour
        QMpH:                   49828.76 query mixes per hour
        QMpH:                   50173.53 query mixes per hour
        QMpH:                   50309.44 query mixes per hour
        QMpH:                   50459.95 query mixes per hour
        QMpH:                   50173.58 query mixes per hour
        QMpH:                   50902.95 query mixes per hour
        QMpH:                   50980.06 query mixes per hour
        QMpH:                   46065.53 query mixes per hour
        QMpH:                   49693.76 query mixes per hour
        QMpH:                   50024.07 query mixes per hour
        QMpH:                   50185.72 query mixes per hour
        QMpH:                   50833.60 query mixes per hour
        QMpH:                   50445.08 query mixes per hour
        QMpH:                   50104.55 query mixes per hour
        QMpH:                   50536.78 query mixes per hour
        QMpH:                   47215.22 query mixes per hour
        QMpH:                   48753.42 query mixes per hour
        QMpH:                   49537.54 query mixes per hour
        QMpH:                   50315.51 query mixes per hour
        QMpH:                   50813.64 query mixes per hour
        QMpH:                   50454.16 query mixes per hour
        QMpH:                   50543.55 query mixes per hour
        QMpH:                   50878.96 query mixes per hour
        QMpH:                   50888.24 query mixes per hour
        QMpH:                   50754.96 query mixes per hour
        QMpH:                   46296.42 query mixes per hour
        QMpH:                   49833.32 query mixes per hour
        
        Show
        bryanthompson bryanthompson added a comment - Converged query performance on BSBM 100M looks ok root@bigdata11:~/workspace/bsbmtools/trunk# for a in `seq 120`; do ./testdriver -o mt_8.xml -seed $RANDOM -w 50 -mt 16 -idir td_100m/td_data http://localhost:80/sparql|grep QMpH; done QMpH: 17836.47 query mixes per hour QMpH: 27681.61 query mixes per hour QMpH: 34620.65 query mixes per hour QMpH: 44397.75 query mixes per hour QMpH: 46979.70 query mixes per hour QMpH: 47220.10 query mixes per hour QMpH: 47318.41 query mixes per hour QMpH: 47523.29 query mixes per hour QMpH: 47966.19 query mixes per hour QMpH: 47852.73 query mixes per hour QMpH: 48170.39 query mixes per hour QMpH: 49295.09 query mixes per hour QMpH: 47719.32 query mixes per hour QMpH: 48102.25 query mixes per hour QMpH: 49151.71 query mixes per hour QMpH: 47703.47 query mixes per hour QMpH: 48361.68 query mixes per hour QMpH: 49284.32 query mixes per hour QMpH: 48084.53 query mixes per hour QMpH: 49390.49 query mixes per hour QMpH: 48149.64 query mixes per hour QMpH: 49488.86 query mixes per hour QMpH: 48557.35 query mixes per hour QMpH: 48891.98 query mixes per hour QMpH: 50112.33 query mixes per hour QMpH: 47801.18 query mixes per hour QMpH: 49753.10 query mixes per hour QMpH: 48498.07 query mixes per hour QMpH: 49278.52 query mixes per hour QMpH: 49951.89 query mixes per hour QMpH: 48035.99 query mixes per hour QMpH: 49353.69 query mixes per hour QMpH: 50121.31 query mixes per hour QMpH: 48172.97 query mixes per hour QMpH: 49482.34 query mixes per hour QMpH: 50306.86 query mixes per hour QMpH: 48221.52 query mixes per hour QMpH: 48879.38 query mixes per hour QMpH: 49563.24 query mixes per hour QMpH: 50400.67 query mixes per hour QMpH: 47650.62 query mixes per hour QMpH: 49621.94 query mixes per hour QMpH: 50179.12 query mixes per hour QMpH: 50121.66 query mixes per hour QMpH: 46918.69 query mixes per hour QMpH: 49477.48 query mixes per hour QMpH: 50088.84 query mixes per hour QMpH: 50605.66 query mixes per hour QMpH: 47901.35 query mixes per hour QMpH: 48879.08 query mixes per hour QMpH: 49814.42 query mixes per hour QMpH: 50175.20 query mixes per hour QMpH: 49980.21 query mixes per hour QMpH: 47639.71 query mixes per hour QMpH: 49486.24 query mixes per hour QMpH: 50109.79 query mixes per hour QMpH: 50383.81 query mixes per hour QMpH: 50609.05 query mixes per hour QMpH: 47402.14 query mixes per hour QMpH: 49600.20 query mixes per hour QMpH: 50394.44 query mixes per hour QMpH: 50746.85 query mixes per hour QMpH: 50755.83 query mixes per hour QMpH: 50615.72 query mixes per hour QMpH: 48624.19 query mixes per hour QMpH: 49690.24 query mixes per hour QMpH: 49695.86 query mixes per hour QMpH: 50672.48 query mixes per hour QMpH: 50594.22 query mixes per hour QMpH: 47457.06 query mixes per hour QMpH: 49348.70 query mixes per hour QMpH: 49903.95 query mixes per hour QMpH: 50236.92 query mixes per hour QMpH: 50662.19 query mixes per hour QMpH: 50166.32 query mixes per hour QMpH: 50733.54 query mixes per hour QMpH: 46197.30 query mixes per hour QMpH: 49405.59 query mixes per hour QMpH: 50127.05 query mixes per hour QMpH: 50688.06 query mixes per hour QMpH: 50262.21 query mixes per hour QMpH: 50725.86 query mixes per hour QMpH: 50496.62 query mixes per hour QMpH: 46305.06 query mixes per hour QMpH: 49602.56 query mixes per hour QMpH: 50328.03 query mixes per hour QMpH: 50246.48 query mixes per hour QMpH: 50190.79 query mixes per hour QMpH: 50538.40 query mixes per hour QMpH: 51030.92 query mixes per hour QMpH: 50251.89 query mixes per hour QMpH: 48282.93 query mixes per hour QMpH: 49828.76 query mixes per hour QMpH: 50173.53 query mixes per hour QMpH: 50309.44 query mixes per hour QMpH: 50459.95 query mixes per hour QMpH: 50173.58 query mixes per hour QMpH: 50902.95 query mixes per hour QMpH: 50980.06 query mixes per hour QMpH: 46065.53 query mixes per hour QMpH: 49693.76 query mixes per hour QMpH: 50024.07 query mixes per hour QMpH: 50185.72 query mixes per hour QMpH: 50833.60 query mixes per hour QMpH: 50445.08 query mixes per hour QMpH: 50104.55 query mixes per hour QMpH: 50536.78 query mixes per hour QMpH: 47215.22 query mixes per hour QMpH: 48753.42 query mixes per hour QMpH: 49537.54 query mixes per hour QMpH: 50315.51 query mixes per hour QMpH: 50813.64 query mixes per hour QMpH: 50454.16 query mixes per hour QMpH: 50543.55 query mixes per hour QMpH: 50878.96 query mixes per hour QMpH: 50888.24 query mixes per hour QMpH: 50754.96 query mixes per hour QMpH: 46296.42 query mixes per hour QMpH: 49833.32 query mixes per hour
        beebs Brad Bebee made changes -
        Field Original Value New Value
        Workflow Trac Import v2 [ 12811 ] Trac Import v3 [ 14316 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v3 [ 14316 ] Trac Import v4 [ 15645 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v4 [ 15645 ] Trac Import v5 [ 17031 ]
        beebs Brad Bebee made changes -
        Labels Issue_patch_20150625
        beebs Brad Bebee made changes -
        Status Closed - Won't Fix [ 6 ] Open [ 1 ]
        beebs Brad Bebee made changes -
        Status Open [ 1 ] Accepted [ 10101 ]
        beebs Brad Bebee made changes -
        Status Accepted [ 10101 ] In Progress [ 3 ]
        beebs Brad Bebee made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        beebs Brad Bebee made changes -
        Status Resolved [ 5 ] In Review [ 10100 ]
        beebs Brad Bebee made changes -
        Resolution Fixed [ 1 ] Done [ 10000 ]
        Status In Review [ 10100 ] Done [ 10000 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v5 [ 17031 ] Trac Import v6 [ 18216 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v6 [ 18216 ] Trac Import v7 [ 19613 ]
        beebs Brad Bebee made changes -
        Workflow Trac Import v7 [ 19613 ] Trac Import v8 [ 21235 ]

          People

          • Assignee:
            bryanthompson bryanthompson
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: