Details

      Description

      The metadata service should be modified to use the RWStore. The RWStore will be allow the MDS to address petabytes of shards when deployed on a modest hardware configuration. Failover for the MDS will use the write replication mechanism for the RWStore. Clients can read from any MDS instance, except if they are requesting the unisolated view. All writes will go to the master MDS instance and be replicated to the failover nodes.

      This issue has a dependency on [1] (integrate and test the RWStore).

      [1] https://sourceforge.net/apps/trac/bigdata/ticket/27

        Activity

        Hide
        bryanthompson bryanthompson added a comment -

        A design has been put forward which would decentralize the metadata service (MDS) based on multicast or P2P gossip protocols. However, each DS (data service) must still be able to answer MDS queries for the shards on that DS.

        The decentralized MDS has the advantage that a decentralized MDS can scale to exabytes. As one consequence of that design, iterators over shards would be shard-wise ordered (rather than totally ordered) since otherwise the entire set of shard locators for the iterator must be materialized and ordered before the iterator can execute. This is of little consequence for high-level query since we nearly always want shard-wise parallelism for iterators used by high-level query (global access path scans).

        Show
        bryanthompson bryanthompson added a comment - A design has been put forward which would decentralize the metadata service (MDS) based on multicast or P2P gossip protocols. However, each DS (data service) must still be able to answer MDS queries for the shards on that DS. The decentralized MDS has the advantage that a decentralized MDS can scale to exabytes. As one consequence of that design, iterators over shards would be shard-wise ordered (rather than totally ordered) since otherwise the entire set of shard locators for the iterator must be materialized and ordered before the iterator can execute. This is of little consequence for high-level query since we nearly always want shard-wise parallelism for iterators used by high-level query (global access path scans).
        Hide
        bryanthompson bryanthompson added a comment -

        Closed. Not relevant to the new architecture.

        Show
        bryanthompson bryanthompson added a comment - Closed. Not relevant to the new architecture.
        Hide
        bryanthompson bryanthompson added a comment -

        Given that HDFS is an append only file structure and that the MDS is a DS instance, this ticket is actually a duplicate of BLZG-277 (HA DS).

        Show
        bryanthompson bryanthompson added a comment - Given that HDFS is an append only file structure and that the MDS is a DS instance, this ticket is actually a duplicate of BLZG-277 (HA DS).

          People

          • Assignee:
            Unassigned
            Reporter:
            bryanthompson bryanthompson
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: