XMLWordPrintable

    Details

    • Type: Sub-task
    • Status: Open
    • Resolution: Unresolved
    • Affects Version/s: BIGDATA_RELEASE_1_2_0
    • Fix Version/s: None
    • Component/s: Bigdata RDF Database
    • Labels:
      None

      Description

      Pack TermIds. This change would introduce a data migration requirement, but it could reduce the size on disk of tids by 50% or more. We would need to make sure that tids were 63 bits clear, which means also modifying the TermIdEncoder. See [1,2].

      Status: Packed TIDs have been implemented for the Journal. They are not enabled by default since they break binary compatibility. They have not been implemented for the cluster.

      Note: In addition to packing TIDs, the stride of the TID could be changed from 1+8 bytes (flags plus long term identifier) to 1+7 bytes. The 7 bytes provides for approximately 74 quadrillion term identifiers in a single graph. The same packing techniques could be applied after nulling out the flags byte. The advantage of this 8 byte stride (when not packed) is that it offers a better byte alignment and better compatibility with other processing architectures (GPUs).

      [1] BLZG-314 (TermIdEncoder)
      [2] BLZG-641 (Improve load performance)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              bryanthompson bryanthompson
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated: