Uploaded image for project: 'Blazegraph (by SYSTAP)'
  1. Blazegraph (by SYSTAP)
  2. BLZG-190

Workbench must provide editable window to configure/clone namespace

    Details

    • Type: New Feature
    • Status: Done
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: BIGDATA_RELEASE_1_5_0
    • Fix Version/s: BLAZEGRAPH_2_0_0
    • Component/s: Workbench
    • Labels:
      None

      Description

      We need to define a text input field in which someone can edit the properties before creating the new namespace.

      When cloning a namespace the properties should be filtered to remove anything that is specific to the Journal rather than the namespace. For example, given the following the marked lines should all be removed when cloning the properties. Also, some properties are namespace specific and need to be renamed when cloning into another namespace (this applies to the branching factors).

      com.bigdata.namespace.kb.spo.com.bigdata.btree.BTree.branchingFactor=1024 // rename (namespace)
      com.bigdata.relation.container=test // remove (old namespace, automatically injected)
      com.bigdata.journal.AbstractJournal.bufferMode=DiskRW // remove (applies to journal)
      com.bigdata.journal.AbstractJournal.file=bigdata.jnl // remove (applies to journal)
      com.bigdata.journal.AbstractJournal.initialExtent=209715200 // remove (applies to journal)
      com.bigdata.rdf.store.AbstractTripleStore.vocabularyClass=com.bigdata.rdf.vocab.DefaultBigdataVocabulary
      com.bigdata.rdf.store.AbstractTripleStore.textIndex=false
      com.bigdata.btree.BTree.branchingFactor=128 // remove (global btree default is not namespace specific)
      com.bigdata.namespace.test.lex.com.bigdata.btree.BTree.branchingFactor=400 // rename (namespace)
      com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axioms.NoAxioms
      com.bigdata.service.AbstractTransactionService.minReleaseAge=1 // remove (applies to Journal/transaction service)
      com.bigdata.rdf.sail.truthMaintenance=true
      com.bigdata.journal.AbstractJournal.maximumExtent=209715200 // remove (applies to Journal)
      com.bigdata.rdf.sail.namespace=test // remove/rename (old namespace)
      com.bigdata.relation.class=com.bigdata.rdf.store.LocalTripleStore // remove (com.bigdata.relation properties are generated when creating the namespace)
      com.bigdata.rdf.store.AbstractTripleStore.quads=false
      com.bigdata.btree.writeRetentionQueue.capacity=4000 // remove (global btree default is not namespace specific)
      com.bigdata.rdf.store.AbstractTripleStore.statementIdentifiers=false
      

      It is especially unfortunate the the workbench does not clone the branching factors into the new namespace. Contrary to reasonable expectations, the cloned namespace will use default branching factors (e.g., 128) rather than those configured in the cloned namespace.

        Issue Links

          Activity

          Hide
          bryanthompson bryanthompson added a comment -

          Igor, please review this ticket for our next sync call. I would like to give this some priority and get this feature into 2.0.
          Thanks,
          Bryan

          Brad Bebee

          Show
          bryanthompson bryanthompson added a comment - Igor, please review this ticket for our next sync call. I would like to give this some priority and get this feature into 2.0. Thanks, Bryan Brad Bebee
          Hide
          beebs Brad Bebee added a comment -

          igorkim thompsonbry Yes, I agree. Let's put this one into the priority list.

          Show
          beebs Brad Bebee added a comment - igorkim thompsonbry Yes, I agree. Let's put this one into the priority list.
          Hide
          igorkim igorkim added a comment -

          Per the example above, upon user action (clone namespace) we should take properties of the namespace which is being cloned, keep the options AbstractTripleStore.* and sail.truthMaintenance; rename namespace in the branchingFactor options; and remove everything else. After that user may adjust these properties in some text box on Workbench, another adjustment should be done to the user input (rename/remove incompatible options again), and then a new namespace will be created using these properties. Is this correct?

          Show
          igorkim igorkim added a comment - Per the example above, upon user action (clone namespace) we should take properties of the namespace which is being cloned, keep the options AbstractTripleStore.* and sail.truthMaintenance; rename namespace in the branchingFactor options; and remove everything else. After that user may adjust these properties in some text box on Workbench, another adjustment should be done to the user input (rename/remove incompatible options again), and then a new namespace will be created using these properties. Is this correct?
          Hide
          bryanthompson bryanthompson added a comment -

          The most important aspects are:

          • Correct removal of the old namespace and assert of the new namespace.
          • Correct migration of per-namespace parameterized options (the branching factors are the most important example of this, but it is a generalized pattern for override of index properties). A regex should handle this.
          • An opportunity to override / edit.

          I suggest that you develop a web service (part of the REST API) to rewrite the properties. This can also do the filtering. Pass it the target namespace and the properties. Have it rename properties as appropriate and return the rewritten properties. This way it can also invoke additional validation on the server. For example, the BigdataSail.checkProperties() method. Please add a test suite for this.

          It would also be good to automatically remove properties that are only valid for the Journal as a whole or that are unparameterized properties for indices (not specific to a given namespace). However, this is not a critical item. I think that a black list approach is appropriate here. The initial black list should include a list of those properties that are not namespace specific that appear in our default RWStore.properties files or that are commonly recommended overrides.

          A whitelist of properties to be retained would be problematic. First, it is possible to use a very wide list of properties. While many of the valid options are captured by the inheritance hierarchy for BigdataSail.Options and AbstractTripleStore.Options, there could be other options that are also valid and thus we should not make it impossible for people to specify such options (though a direct HTTP request would be a workaround for a whitelist).

              public static interface Options extends AbstractResource.Options,
                      InferenceEngine.Options, com.bigdata.journal.Options,
                      KeyBuilder.Options, DataLoader.Options, FullTextIndex.Options {...}
          
              public static interface Options extends com.bigdata.rdf.store.AbstractTripleStore.Options {}
          

          See com.bigdata.config.Configuration for how properties may be parameterized for a namespace, a container (such as namespace.lex), or an index (such as namespace.lex.TERM2ID). You may be able to lift some regex or other logic from this class.


          If there was an easy (2hr) hack to report the javadoc for an option back as an XML comment, that might be a nice way to deliver some added information to the user. Or just the link to the javadoc URL as a comment in case they want to follow it. This could be an option in the namespace create to preview the properties with the hot links to the documentation.


          The properties could be edited as XML or as Java plain text properties file. Please use whichever is simpler (probably the latter since a normal html widget will suffice and we do not need to identify a suitable OS dependency for the workbench for XML editing).

          Show
          bryanthompson bryanthompson added a comment - The most important aspects are: Correct removal of the old namespace and assert of the new namespace. Correct migration of per-namespace parameterized options (the branching factors are the most important example of this, but it is a generalized pattern for override of index properties). A regex should handle this. An opportunity to override / edit. I suggest that you develop a web service (part of the REST API) to rewrite the properties. This can also do the filtering. Pass it the target namespace and the properties. Have it rename properties as appropriate and return the rewritten properties. This way it can also invoke additional validation on the server. For example, the BigdataSail.checkProperties() method. Please add a test suite for this. It would also be good to automatically remove properties that are only valid for the Journal as a whole or that are unparameterized properties for indices (not specific to a given namespace). However, this is not a critical item. I think that a black list approach is appropriate here. The initial black list should include a list of those properties that are not namespace specific that appear in our default RWStore.properties files or that are commonly recommended overrides. A whitelist of properties to be retained would be problematic. First, it is possible to use a very wide list of properties. While many of the valid options are captured by the inheritance hierarchy for BigdataSail.Options and AbstractTripleStore.Options, there could be other options that are also valid and thus we should not make it impossible for people to specify such options (though a direct HTTP request would be a workaround for a whitelist). public static interface Options extends AbstractResource.Options, InferenceEngine.Options, com.bigdata.journal.Options, KeyBuilder.Options, DataLoader.Options, FullTextIndex.Options {...} public static interface Options extends com.bigdata.rdf.store.AbstractTripleStore.Options {} See com.bigdata.config.Configuration for how properties may be parameterized for a namespace, a container (such as namespace.lex), or an index (such as namespace.lex.TERM2ID). You may be able to lift some regex or other logic from this class. If there was an easy (2hr) hack to report the javadoc for an option back as an XML comment, that might be a nice way to deliver some added information to the user. Or just the link to the javadoc URL as a comment in case they want to follow it. This could be an option in the namespace create to preview the properties with the hot links to the documentation. The properties could be edited as XML or as Java plain text properties file. Please use whichever is simpler (probably the latter since a normal html widget will suffice and we do not need to identify a suitable OS dependency for the workbench for XML editing).
          Hide
          igorkim igorkim added a comment -

          Regarding a way to report the javadoc for an option in XML/properties comment. JavaDoc texts are effectively erased from the executable file.

          1. Some third-party tool may be used, for example http://apidocjs.com/ under MIT license, which compiles comments into apidoc.json, which is used to provide a documentation for RESTful web API.

          2. Another approach is to utilize

          {@value #comment}

          javadoc feature http://docs.oracle.com/javase/7/docs/technotes/tools/windows/javadoc.html#value
          public class B {
          public static final String OPTION_COMMENT = "This is some comment";
          /**

          • {@value #OPTION_COMMENT}

            */
            public static final String OPTION = "com.bigdata.....";
            }
            But in this case javadoc shows OPTION comment as a hyperlink to OPTION_COMMENT string.

          3. Approach which does not require existing javadocs changes would be to write our own compiler in the same way as #1 do, but it will collect all comments for all constants in Options classes across the code. Collected comments has to be placed into jar resources and served as a comments for xml/properties options.

          Both 1 and 2 approaches require massive change of javadocs across the code, thus may take significantly more that 2 hours. Option 3 is just an idea, and its implementation could probably take several days also.

          Quick and unstable hack could be to browse all Option classes at runtime, collect all their comment constants names and values, then find corresponding constant location from the properties option name, and construct a reference to javadoc, for example:
          for properties option com.bigdata.rdf.sail.bufferCapacity=1000000
          find com.bigdata.rdf.sail.BigdataSail.Options.QUEUE_CAPACITY constant
          and compose a link https://www.blazegraph.com/docs/api/com/bigdata/rdf/sail/BigdataSail.Options.html#QUEUE_CAPACITY
          which is actually is missing from the javadoc pages on website.

          Show
          igorkim igorkim added a comment - Regarding a way to report the javadoc for an option in XML/properties comment. JavaDoc texts are effectively erased from the executable file. 1. Some third-party tool may be used, for example http://apidocjs.com/ under MIT license, which compiles comments into apidoc.json, which is used to provide a documentation for RESTful web API. 2. Another approach is to utilize {@value #comment} javadoc feature http://docs.oracle.com/javase/7/docs/technotes/tools/windows/javadoc.html#value public class B { public static final String OPTION_COMMENT = "This is some comment"; /** {@value #OPTION_COMMENT} */ public static final String OPTION = "com.bigdata....."; } But in this case javadoc shows OPTION comment as a hyperlink to OPTION_COMMENT string. 3. Approach which does not require existing javadocs changes would be to write our own compiler in the same way as #1 do, but it will collect all comments for all constants in Options classes across the code. Collected comments has to be placed into jar resources and served as a comments for xml/properties options. Both 1 and 2 approaches require massive change of javadocs across the code, thus may take significantly more that 2 hours. Option 3 is just an idea, and its implementation could probably take several days also. Quick and unstable hack could be to browse all Option classes at runtime, collect all their comment constants names and values, then find corresponding constant location from the properties option name, and construct a reference to javadoc, for example: for properties option com.bigdata.rdf.sail.bufferCapacity=1000000 find com.bigdata.rdf.sail.BigdataSail.Options.QUEUE_CAPACITY constant and compose a link https://www.blazegraph.com/docs/api/com/bigdata/rdf/sail/BigdataSail.Options.html#QUEUE_CAPACITY which is actually is missing from the javadoc pages on website.
          Hide
          bryanthompson bryanthompson added a comment -

          I do not want to inject the java doc into the code. I was thinking along the lines of the manufactured link. The problem with QUEUE_CAPACITY is that the feature is new in master and the javadoc is from the last release, so new features will not be resolvable and documentation will be out of date. That should not be true once we actually do a new release.

          This javadoc lookup is a minor part of the ticket and could be left out.

          Show
          bryanthompson bryanthompson added a comment - I do not want to inject the java doc into the code. I was thinking along the lines of the manufactured link. The problem with QUEUE_CAPACITY is that the feature is new in master and the javadoc is from the last release, so new features will not be resolvable and documentation will be out of date. That should not be true once we actually do a new release. This javadoc lookup is a minor part of the ticket and could be left out.
          Hide
          igorkim igorkim added a comment -
          Show
          igorkim igorkim added a comment - PR https://github.com/SYSTAP/bigdata/pull/234

            People

            • Assignee:
              igorkim igorkim
              Reporter:
              bryanthompson bryanthompson
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: