I am looking at how to improve encapsulation of the REST API methods. There is some tension between providing the servlet response and request objects and hiding the access to the journal and the sail connection. We want the sail connection to be obtained from within the call() method so its scope is strictly limited to the period during which the ConcurrencyManager is holding the necessary lock. This is what will give us the ability to run concurrent mutation operations with group commit. This means pushing down the "getUnisolatedConnection()" (which should use a tx anyway if the caller has specified one in the update). However, some of the REST API methods require direct access to the servlet API in order to using efficient streaming operations on the servlet request and/or servlet response.
I have create a branch (NSS_GROUP_COMMIT) to explore this issue. Overall, the issue about interrupting NSS requests is quite viable. Support group commit is trickier, but looks possible. However, there are some methods where group commit would break the ACID semantics of the current implementation (this is discussed below). Those methods could be rewritten, e.g., as SPARQL UPDATE requests. There are also some complexities around the overhead of declaring the locks required to support group commit and the differences between the global index views used by the IBigdataFederation and the Journal local views (which can use the ConcurrencyManager). Details are below for all of this.
- REST API mutation methods should support read/write tx if the KB is so configured. Right now they are using the unisolated connection regardless.
- The RDFParser options are being configured from constants in the REST API code rather than using the configured properties on the KB instance. For example:
// Note: the setPreserveBNodeIDs() method is not being invoked.
- The REST API methods SHOULD be able to provide unnested http responses with simple status codes and explanations (status lines). This means that they need to either error check the API before entering the Callable or they need to directly operation on the HttpServletResponse and commit it before throwing out an exception. Right now, the preferred pattern is that we do not throw exceptions for API errors. Instead, we commit the response and return. This really needs to be done outside of the Callable or we need to have a well known exception type that is used for errors that have already been handled and the response committed.
- The IBigdataFederation can not use the ConcurrencyManager to run the tasks in precisely the same manner. There is an IConcurrencyManager available on the IDataService, but it is used to guard the individual named index partitions rather than the global index views. Further, the NSS can run against the global index views from any process that is connected to an IBigdataFederation, and such processes will not have an IConcurrencyManager unless they are IDataService or IMetadataService instances. Thus, we need to abstract out two patterns: one that can be used when the backing database is an IJournal instance and one that can be used when the backing database is an IBigdataFederation instance. This suggests that we need a pattern using a delegation model that is either wrapped by an AbstractTask (for the Journal) or by a simpler pattern (for an IBigdataFederation).
- The immediate concern is being able to interrupt the unisolated tasks for a leader fail scenario. A secondary concern is whether we can leverage the ConcurrencyManager to allow concurrent unisolated updates (also overriding the Journal's semaphore).
- I have code that runs the InsertServlet tasks on the ConcurrencyManager for the journal. There is some additional overhead involved because each task must discover and then declare all of the indices associated with a given KB. However, this approach should permit concurrent operations against different KB instances if we also disable the Journal semaphore. It would be more efficient if we could do the index name declarations based on the namespace along, but this requires an efficient prefix scan on Name2Addr and there is a known bug for that. Some of the test harness needs to be aware and operate using the same concurrency control mechanisms, especially the code that creates the KB instance (if this is done explicitly) and that destroys the KB instance. The tasks are calling through to BigdataSailConnection.commit() and .rollback(). Those methods call through to the LocalTripleStore.commit() and LocalTripleStore.abort() methods. Since those calls are being pass down to the AbstractTask.IsolatedActionJournal, I have had to make the methods into NOPs on the IsolatedActionJournal (they were throwing UnsupportedOperationExceptions). The general pattern for job-based concurrency is that the commit is applied automatically when the AbstractTask finishes and the write set of the job is discarded automatically if the task fails.
- There is a potential issue where the existing code commits the response and returns, e.g., from the InsertServlet. Any task that does not fail (thrown exception) will commit. This means that mutations operations that fail will still attempt to join a commit point. This is inappropriate and could cause resource leaks (e.g., if the operation failed after writing on the Journal). We really should throw out a typed exception, but in launderThrowable() ignore that typed exception if the response has already been committed. That way the task will not join a commit point.
- There is a flag in BigdataServlet that currently enables or disables the use of the ConcurrencyManager. Disable/Enable group commit on the Journal from the NSS API. There SHOULD be a global flag should control this and also disable the journal's semaphore and should disable the wrapping of BTree as an UnisolatedReadWriteIndex, and should disable the calls to commit() or abort() from the LocalTripleStore to the Journal.
- DELETE WITH QUERY, UPDATE WITH QUERY : allowing group commit for these methods on the REST API could break the ACID contract. The methods currently rely on reading from the lastCommitTime and writing on the unisolated connection. With group commit, there can be mutations that have been checkpointed and hence are visible to the unisolated connection, but which are not yet visible as of the lastCommitTime. Thus, if another request modified the KB since the last commit point and before the DELETE WITH QUERY or UPDATE WITH QUERY is executed, then those modifications will not be observed by the reader on the lastUpdateTime view.
- SPARQL QUERY/UPDATE: We need to refactor the code that manages the running queries in BigdataRDFServlet so we can separate out the concurrency control of the views from the control over the #of running queries and/or update requests and the metadata that we manage to track and report on those requests.
- MULTI TENANCY: We also need to refactor the MultiTenancyServlet to push down Callable tasks that can be executed with the appropriate concurrency controls.
See https://sourceforge.net/apps/trac/bigdata/ticket/743 (AbstractTripleStore.destroy() does not filter for correct prefix
- this notes the problem with prefix scans on Name2Addr; this issue might also be noted on the feature for durable named solution sets.)