There exists a dichotomy in modern storage products. Commodity storage is inexpensive, but unreliable. Enterprise storage is expensive, but reliable. Large capacities are present in both enterprise and commodity class. The problem, then, becomes how to leverage inexpensive commodity hardware to achieve high capacity enterprise class reliability at a fraction of the cost.

This problem space has been researched extensively, especially in the last few years: in academia, the commercial sector, and by open source community. Hibari uses techniques and algorithms from this research to create a solution which is reliable, cost effective, and scalable.

Hibari is key-value store. If a key-value store were represented as an SQL table, it would be defined as:

SQL-like definition of a generic key value store. 

CREATE TABLE foo (
    BLOB key;
    BLOB value;
) PRIMARY KEY key;

In truth, each key stored in Hibari has three additional fields associated with it. See Section 4.2, “The Hibari Data Model” and Hibari Contributor’s Guide for details.

Hibari was originally written by Cloudian, Inc. (formerly Gemini Mobile Technologies) to support mobile messaging and email services. Hibari was released outside of Cloudian under the Apache Public License version 2.0 in July 2010.

Hibari has been deployed by multiple telecom carriers in Asia and Europe. Hibari may lack some features such as monitoring, event and alarm management, and other "production environment" support services. Since telecom operator has its own data center support infrastructure, Hibari’s development has not included many services that would be redundant in a carrier environment. We hope that Hibari’s release to the open source community will close those functional gaps as Hibari spreads outside of carrier data centers.

Cloudian, Inc. provides full support, consulting, and development services for Hibari. Please see the Hibari NOSQL at Cloudian web site for more information.

	We strongly believe that "ACID" and "BASE" properties exist on a spectrum and are not exclusively one or the other (black-or-white) properties.

Most database users and administrators are familiar with the acronym ACID: Atomic, Consistent, Independent, and Durable. Now, consider an alternative method of storing and managing data, BASE:

For an exploration of ACID and BASE properties (at ACM Queue), see:

BASE: An Acid Alternative
Dan Pritchett
ACM Queue, volume 6, number 3 (May/June 2008)
ISSN: 1542-7730
http://queue.acm.org/detail.cfm?id=1394128

When both strict ACID and strict BASE properties are placed on a spectrum, they are at the opposite ends. However, a distributed database system can fit anywhere in the middle of the spectrum.

A Hibari cluster lies near the ACID end of the ACID/BASE spectrum. In general, Hibari’s design will always favors consistency and durability of updates at the expense of 100% availability in all situations.

Eric Brewer’s "CAP Theorem", and its proof by Gilbert and Lynch, is a tricky thing. It’s nearly impossible to cleanly apply the purity of logic to the dirty world of real, industrial computing systems. We strongly suggest that the reader consider the CAP properties as a spectrum, one of balances and trade-offs. The distributed database world is not black and white, and it is important to know where the gray areas are.

See the Wikipedia article about the CAP theorem for a summary of the theorem, its proof, and related links.

CAP Theorem (postulated by Eric Brewer, Inktomi, 2000)
Wikipedia
http://en.wikipedia.org/wiki/CAP_theorem

Hibari chooses the C and P of CAP. It utilizes chain replication technique and it always guarantees strong consistency. Hibari also includes an Erlang/OTP application specifically for detecting network partitions, so that when a network partition occurs, the brick nodes in the opposite side of the partition with the active master will be removed from the chains to keep the strong consistency guarantee.

See Section 6.4, “Admin Server and Network Partition” for details.

Copyright © 2005-2013 Hibari developers. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Multiple machines can participate in a single cluster. The maximum size of a Hibari cluster has not yet been determined. A practical limit of approximately 200-250 nodes is likely.

Any server node can handle any client request, forwarding a request to the correct server node when necessary. Clients maintain enough state to send their queries directly to the correct server node in all common cases.

The total storage and processing capacity of a Hibari cluster increases linearly as machines are added to the cluster.

Every key update is written and flushed to stable storage (via the fsync() system call) before sending acknowledgments to the client.

After a key’s update is acknowledged, no client in the cluster can see an older version of that key. Hibari uses the "chain replication" algorithm to maintain consistency across all replicas of a key.

All data written to disk include MD5 checksums; the checksums are validated on each read to avoid sending corrupted data to the client.

The Hibari client API requires that all operations (read queries operations and/or update operations) be self-contained within a single client request. Therefore, locks are not implemented because they are not required.

Inside Hibari, each key-value pair also contains a “timestamp” value. A timestamp is an integer. Each time the key is updated, the timestamp value must increase. (This requirement is enforced by all server nodes.)

In many database systems, if a client requires guarantees that a key has not changed since the last time it was read, then the client acquires a lock (or lease) on the key. In Hibari, the client’s update specifies the timestamp of the last read attempt of the key:

It is recommended that all Hibari nodes use NTP to synchronize their system clocks. The simplest Hibari client API uses timestamps based upon the OS system clock for timestamp values. This feature can be bypassed, however, by using a slightly more complex client API.

However, Hibari’s overload detection and work-dumping algorithms will use the OS system clock, regardless of which client API is used. All system clocks, client and server, be synchronized to be within roughly 1 second of each other.

Each key can be replicated multiple times (configurable on a per-table basis). As long as one copy of the key survives, all operations on that key are permitted. A cluster can survive multiple cluster node failures and still maintain full data integrity.

The cluster membership application, called the Hibari Admin Server, runs as an active/standby application on one or more of the server nodes. The Admin Server’s configuration and private state are also maintained in Hibari server nodes. Shared storage such as NFS, shared SCSI/Fibre Channel LUNs, or replicated block devices are not required.

If the Admin Server fails and is restarted on a standby node, the rest of the cluster can continue normal operation. If another brick fails while the Admin Server is restarting, then clients may see service interruptions (usually in the form of timeouts) until the Admin Server has finished restarting and can react to the failure.

Hibari supports many client protocols for queries and updates:

Protocols under development:

Most of the client access protocols are implemented using the Erlang/OTP application behavior. By separating each access protocol into separate OTP applications, Hibari’s packaging is quite flexible: packaging can add or remove protocol support as desired. Similarly, protocols can be stopped and started at runtime.

Hibari’s performance is competitive with other distributed, non-relational databases such as HBase and Cassandra, when used with similar replication and durability configurations. Despite the constraints of durable writes and strong consistency, Hibari’s performance can exceed those databases on some workloads.

	The metadata of all keys stored by the brick, called the “key catalog”, are stored in RAM to accelerate commonly-used operations. In addition, non-zero values of the "expiration_time" and non-empty values of "flags" are also stored in RAM (see SQL-like definition of a Hibari table). As a consequence, a multi-million key brick can require many gigabytes of RAM.

Replicas of keys are automatically repaired whenever a cluster node crashes and restarts.

The number of replicas per key can be changed without service interruption. Likewise, replication chains can be added or removed from the cluster without service interruption. This permits the cluster to grow (or shrink) as workloads change. See Section 9.4, “Chain Migration: Rebalancing Data Across Chains” for more details.

Keys will be automatically be rebalanced across the cluster without service interruption. See Section 9.4, “Chain Migration: Rebalancing Data Across Chains” for more details.

Each replication chain can be assigned a weighting factor that will increase or decrease the percentage of a table’s key space relative to all other chains. This feature can permit use of cluster nodes with different CPU, RAM, and/or disk capacities.

Under limited circumstances, operations on multiple keys can be given transactional commit/abort semantics. Such micro-transactions can considerably simplify the creation of robust applications that keep data consistent despite failures by both clients and servers.

Each Hibari table may be configured with the following options to enhance performance … though each of these options has a corresponding price to pay.

draft

draft

Hibari is a key-value database. Unlike a relational DBMS, Hibari applications do not need to create a schema. The only application requirement is that all its tables be created in advance, see Section 4.5, “Creating New Tables” below.

If a Hibari table were represented within an SQL database, it would look something like this:

SQL-like definition of a Hibari table. 

CREATE TABLE foo (
    BLOB key;
    BLOB value;
    INTEGER timestamp;                  -- Monotonically increasing
    INTEGER expiration_time;            -- Usually zero
    LIST OF ATOMS_AND_TWO_TUPLES flags; -- Metadata stored in RAM for speed
) PRIMARY KEY key;

Hibari table names use the Erlang data type “atom”. The types of all key-related attributes are presented below.

"Storage location = RAM" means that, during normal query handling, data is retrieved from a copy in RAM. All modifications of any/all attributes of a key are written to the write-ahead log to prevent data loss in case of a cluster-wide power failure. See Section 5.2, “Write-Ahead Logs” for more details.

"Store location = disk" means that the value of the attribute is not stored in RAM. Metadata in RAM contains a pointer to the attribute’s location:file #, byte offset, and length. A log sequence file inside the common log must be opened, call lseek(2), and then read(2) to retrieve the attribute.

The practical constraints on maximum value blob size are affected by total blob size and frequency of large blob access. For example, storing an occasional 64MB value blob is different than a 100% write workload of 100% 64MB value blobs. The Hibari client API does not have a method to update or fetch less than the entire value blob, so a brick can be blocked for many seconds if it tried to operate on (for example) even a single 4GB blob. In addition, other processes can be blocked by 'busy_dist_port' events while processing big value blobs.

Hibari’s basic client operations are enumerated below.

Each operation can be accompanied by operation-specific flags. Some of these flags include:

For details of these operations and lesser-used per-operation flags, see:

Hibari does not support automatic indexing of value blobs. If an application requires indexing, the application must build and maintain those indexes.

New tables can be created by two different methods:

For details on the Erlang shell API and detailed explanations of the table options presented in the Admin server’s HTTP interface, see the Hibari Contributor’s Guide

The word "brick" has two different meanings in a Hibari system:

By default, all logical bricks will record all updates to a “write-ahead log”. Used by many database systems, a write-ahead log (WAL) appears to be an infinitely-sized log where all important events (e.g. all write and delete operations) are appended to the end of the log. The log is considered “write-ahead” if a log entry is written prior to any significant processing by the application.

Write-ahead logs in the Hibari application

Two types of write-ahead logs are used by the Hibari application. These logs cooperate with each other to provide several benefits to the logical brick.

There are two types of write-ahead logs:

• The shared "common log". This single write-ahead log instance provides durability guarantees to all logical bricks within the server node via the fsync() system call.
• Individual “private logs”. Each logical brick maintains its own private write-ahead log instance. All metadata regarding keys in the logical brick are stored in the logical brick’s private log.

All updates are written first to the common log, usually in a synchronous manner. At a later time, update metadata is lazily copied from the common log to the corresponding brick’s private log. Value blobs (for bricks that store value blobs on disk) will remain in the common log and are managed by the “scavenger”, see Section 8.7, “The Scavenger”.

A chain is the unit of data replication used by the “chain replication” technique as described in this paper:

Chain Replication for Supporting High Throughput and Availability
Robbert van Renesse and Fred B. Schneider
USENIX OSDI 2004 conference proceedings
http://www.usenix.org/events/osdi04/tech/renesse.html

Data replication algorithms can be separated into two basic families:

The chain replication algorithm is from the state machine family of replication algorithms. It is a variation of the familiar “master/slave” replication algorithm, where all updates are sent to a master node and then copies are sent to zero or more slave nodes.

Chain replication requires a very specific ordering of nodes (which store copies of data) and the messages passed between them. The diagram below depicts the "key update" message flow in a chain of length three.

Figure 3. Message flow in a chain for a key update

If a chain is of length one, then the same brick assumes both “head” and “tail” roles simultaneously. In this case, the brick is called a “standalone” brick.

Figure 4. Message flow for a key update to a chain of length 1

To maintain the property strong consistency, a client must read data from the tail brick in the chain. A read processed by any other brick member would permit the client to read an update that has not yet been processed by all bricks and therefore could result in a strong consistency violation. Such a violation is frequently called a “dirty read” in other database systems.

Figure 5. Message flow for a read-only key query

A table is thing that divides the key namespace within Hibari. If you need to have two different keys called "foo" but have different values, you store each "foo" key in a separate table. The same is true in other database systems.

Hibari’s implementation uses one or more replication chains to store the data for one table.

Figure 6. Relationship between tables, chains, and bricks.

In a single request, a Hibari client may send multiple update operations to the cluster. The client has the option of requesting “micro-transaction” semantics for those updates: if there are no errors, then all updates will be applied atomically. This behaves like the “transaction commit” behavior supported by most relational databases.

On the other hand, if there is an error while processing one of the update operations, then all of update operations will fail. This behaves like the “transaction abort” behavior supported by most relational databases.

Unlike most relational databases, Hibari does not have a transaction manager that can coordinate ACID semantics for arbitrary read and write operations across any row in any table. In fact, Hibari has no transaction manager at all. For this reason, Hibari calls its limited transaction feature “micro-transactions”, to distinguish this feature from other database systems.

Hibari’s micro-transaction support has two important limitations:

Figure 7. Four keys in the "footab" table, distributed across two chains of length three.

In the diagram above, a micro-transaction can be permitted if it operates on only the keys "string1" & "string4" or only the keys "string2" and "string3". If a client were to send a micro-transaction that operates on keys "string1" and "string3", the micro-transaction will be rejected: key "string3" is not stored by the same chain as the key "string1".

Valid micro-transaction: all keys managed by same chain. 

[txn,
 {op = replace, key = "string1", value = "Hello, world!"},
 {op = delete, key = "string4"}
]

Invalid micro-transaction: keys managed by different chains. 

[txn,
 {op = replace, key = "string1", value = "Hello, world!"},
 {op = delete, key = "string2"}
]

The client does not have direct control over how keys are distributed across chains. When a table is defined and created, its configuration specifies the algorithm used to map a {TableName, Key} pair to a specific chain.

	See Hibari Contributor’s Guide, "Add a New Table" section for more information about table configuration.

The active/standby application failover is handled by the Erlang/OTP application controller. No extra third-party software is required. See Chapter 7, "Applications", and Chapter 9, "Distributed Applications", in the "OTP Design Principles User’s Guide" at http://www.erlang.org/doc/design_principles/distributed_applications.html.

On each active and standby node, there is a hint file called Schema.local which contains the name of the “bootstrap bricks”. These bricks operate outside of the chain replication algorithm to provide redundant, persistent state for the Admin Server application. See the section called “Bricks outside of chain replication” for a short summary of standalone bricks.

All of the Admin Server’s private state is stored in the bootstrap bricks. This includes:

With the help of the Erlang/OTP application controller and the Hibari Partition Detector application, only a single instance of the Admin Server is permitted to run at any one time. That single application instance has full control over the data stored in the bootstrap bricks and therefore does not have to manage concurrent updates to bootstrap brick data.

When the Admin Server application is stopped (e.g. node shutdown) or crashes (e.g. software bug, power failure), all of the tasks outlined at the beginning of Section 6, “The Admin Server Application” are halted. In theory, the 20-30 seconds that are required for the Admin Server to restart could mean 20-30 seconds of negative service impact to Hibari clients.

In practice, however, Hibari clients almost never notice when an Admin Server instance crashes and restarts. Hibari clients do not need the Admin Server when the cluster is stable. The Admin Server is only necessary when the state of the cluster changes. Furthermore, as far as clients are concerned, clients are only affected when bricks crash. Other cluster change events, such as when chain replication repair finished, do not directly impact clients and thus can wait for the Admin Server to finish restarting.

A Hibari client will only notice an Admin Server crash if another logical brick crashes while the Admin Server is temporarily out of service. The reason is due to the nature of the Admin Server’s responsibilities. When chain is broken by a brick failure, the remaining bricks must have their roles reconfigured to put the chain back into full service. The Admin Server is the only automated entity that is permitted to change the role of a brick. For more details, see:

One feature of the Erlang/OTP application controller is that it is not robust in event of a network partition. To prevent multiple Admin Server apps running simultaneously, another application is bundled with Hibari: the Partition Detector. See Section 10, “The Partition Detector Application” for an overview and explanation of the A and B physical networks.

As described briefly in Section 1.6, “The CAP Theorem and Hibari”, Hibari does support the "Partition tolerance" aspect of Eric Brewer’s CAP theorem. More specifically, if a network partition occurs, and a Hibari cluster is split into two or more pieces, not all clients on both/all sides of the network partition will be able to access Hibari services.

For the sake of discussion, we assume the cluster has been split into two fragments by a single partition, though any number of fragments may happen in real use. We also assume that nodes on both sides of the partition are configured in standby roles for the Admin Server.

If a network partition event happens, the following events will soon follow:

Note that all steps above will happen in parallel on nodes on both sides of the partition. If this situation is permitted to continue, the invariant of "Admin Server may only run on one node at a time" will be violated. However, with the help of the Partition Detector application, multiple Admin Server instances can be detected and halted.

UDP broadcasts on the A and B networks can help the Admin Server determine if it was restarted due to an Admin Server crash or by a network partition. In case of a network partition on network A, the broadcasts on network B can indicate that another Admin Server process remains alive.

If multiple Admin Server instances are detected, the following logic is used:

Importance of two physically separate networks

It is possible for both the A and B networks to partition simultaneously. The Admin Server and Partition Detector applications cannot always correctly react to such events. It is extremely important that the A and B networks be separate physical networks, including: separate physical network interfaces on each brick, separate cabling, separate network switches, and all other network-related equipment also be physically separate.

It is possible to reduce the reliance on multiple physical networks and the Partition Detector application, but such techniques have not been added to Hibari yet. Until an alternative network partition mitigation mechanism is implemented, we strongly recommend the proper configuration of the Partition Detector app and all of its hardware requirements.

When a network partition event occurs, there are two cases that affect a client’s ability to work with the cluster.

If the client machine is on the same side of the partition, the client may see no interruption of service at all. If the Admin Server is restarted in reaction to the partition event, there may be a small window of time (e.g. 20-30 seconds) where requests might fail because the Admin Server has not yet reconfigured chains on this side of the partition.

If the client machine is on the opposite side of the partition, then the client will not have access to the Admin Server and may not have access to properly configured chains. If a chain lies entirely entirely on the same side of the partition as the client, then the client can continue to use that chain successfully. However, any chain that is "cut in two" by the partition cannot support updates by any client.

Each line of the central.conf file has the form

parameter: value

where parameter is the name of the configuration option being set and value is the value that the configuration option is being set to.

Valid data types for configuration settings are INT (integer), STRING (string), and ATOM (one of a pre-defined set of option names, such as on or off). Apart from data type restrictions, no further valid range restrictions are enforced for central.conf parameters.

All time values in central.conf (such as delivery retry intervals or transaction timeouts) must be set as a number of seconds.

Blank lines and lines beginning with the pound sign (#) are ignored.

	To apply changes that you have made to the central.conf file, you must restart the server. There are exceptions to this rule, but it’s one of the cleanup/janitor tasks to access central.conf using a standard set of APIs, e.g. always use the gmt_config_svr API.

A detailed explanation of each of the items in central.conf can be found at Hibari central.conf Configuration Guide.

Configuration for the Hibari “Admin Server” is stored in three places:

All table and chain configuration parameters are stored within the Admin Server’s “schema”. The schema contains information on:

Much of this information can be seen in HTML form by pointing a Web browser at TCP port 23080 (default) of any Hibari server node. For example:

Admin Server Top-Level Status & Admin URL. 

http://hibari-server-node-hostname:23080/

Your Web browser should be redirected automatically to the Admin Server’s top-level status & admin page.

	The APIs that expose this are, for the most part, already written. We need more "friendly" wrapper funcs as part of the "try this first" set of APIs for administration.

The lifecycle of each Hibari logical brick goes through a set of states defined by a finite state machine (OTP gen_fsm behavior) that is executed by a process within the Admin Server application.

Figure 8. Logical brick lifecycle finite state machine

The chain FSM (OTP ‘gen_fsm` behavior) is executed by a process within the Admin Server application. All state transitions are triggered by changes in the state of each member bricks’ state, into or out of the 'ok' state. See Section 8.1, “Brick Lifecycle Finite State Machine” for details.

Figure 9. Chain replication finite state machine

Each brick within a chain has a role. The role will be changed by the Admin Server whenever it detects that the chain’s state has changed. These roles are:

There is one additional attribute that is given to a brick in a cluster. Its name “official tail”.

As far as Hibari clients are concerned, the chain member with the the "official tail" role is the brick that they consider the "tail" of the chain. Hibari clients are not aware of "tail" bricks that are undergoing repair. Any client request that is sent to a repairing state brick will be rejected.

See Figure 3, “Message flow in a chain for a key update” for an example of a healthy chain of length three.

A logical brick does not maintain an on-disk data structure, such as a binary tree or B-tree, to keep track of the keys it stores. Instead, each logical brick maintains that metadata entirely in RAM. Therefore, the only time that the metadata in the private write-ahead log is ever read is at brick initialization time, i.e. when the brick restarts.

The contents of the private write-ahead log are used to repopulate the brick’s “key catalog”, the list of all keys (and associated metadata) stored by the brick.

When a logical brick is started, all of the log sequence files in the private log are read, starting from the oldest and ending with the newest. (See the section called “Directories and files used by write-ahead logs”.) The total amount of data required at startup can be quite small or it can be hundreds of gigabytes. The factors that influence the amount of data in the private log are:

When the log scan is complete, construction of the brick’s in-RAM key catalog is finished.

See Section 8.6, “Brick Checkpoint Operations” for details on brick checkpoint operations.

When a chain is in the degraded state, new bricks that have entered their pre_init state can become eligible to join the chain. All new bricks are added to the end of the chain and undergo the chain repair process.

Figure 10. Chain of length 2 in degraded state, a third brick under repair

The protocol used between upstream and downstream bricks is an iterative protocol that has two phases in a single iteration.

When the repair is finished, the Admin Server will change the roles of some/all chain members to make the repairing brick the new tail of the chain.

Only one brick may be repaired at one time. In theory it is possible to repair multiple bricks simultaneously, but the extra code complexity that would be required to do so has been judged to be too expensive (so far).

Chain reordering when moving from degraded → healthy states

Figure 11. Chain order after a middle brick fails and is repaired (but not yet reordered)

After a middle brick fails and is repaired, the chain’s ordering is: brick 1 → brick 3 → brick 2. According to the algorithm in the original Chain Replication paper, the final chain ordering is expected. The Hibari implementation adds another step: reordering the chain.

For chains longer than length 1, when the Admin Server moves the chain from degraded → healthy state, the Admin Server will reorder the the chain to match the schema’s definition for the healthy chain order. The assumption is that the Hibari administrator wishes the chain use a very specific order when it is in the healthy state. For example, if the chain’s workload were extremely read-intensive, the machine for logical brick #3 could have faster CPU or faster disks than the other bricks in the chain. To take full advantage of the extra capacity, the chain should be reordered as soon as possible.

However, it is not easy to reorder the chain. The replication of a client update during the reordering could get lost and violate Hibari’s strong consistency guarantees. The following algorithm is used to preserve consistency:

1. Set all bricks to read-only mode.
2. Wait for all updates to sync to disk at each brick and to progress downstream fully from head → tail.
3. Set brick roles to reflect the final desired order.

4. Set all bricks to read-write mode.

   • Client do operations that contain updates will be resubmitted (via the client-side API function brick_server:do()) to the cluster.

Typically, executing this algorithm takes less than one second. However, because the head brick is forced temporarily into read-only mode, client update requests will be delayed until read-only mode is turned off.

Client update requests submitted during read-only mode will be queued by the head brick and will be processed when read-only mode is turned off. Client read-only requests are not affected by read-only mode.

As updates are received by a brick, those updates are written to the brick’s private write-ahead log. During normal operations, private write-ahead log is write-only: the data there is only read at logical brick initialization time.

The checkpoint operation is used to reclaim disk space in the brick’s private write-ahead log. See the section called “Directories and files used by write-ahead logs” for a description of log sequence files and Section 7.2, “Parameters in the central.conf File” for details on the central.conf configuration file.

	Each logical brick will checkpoint itself as its private log grows. It is possible that multiple logical bricks can schedule checkpoint operations simultaneously. The bandwidth limitation of the brick_check_checkpoint_throttle_bytes parameter is applied to the sum of all writes by all checkpoint operations.

As described in Section 5.2, “Write-Ahead Logs”, all updates from all logical bricks are first written to the “common log”. The most common of these updates are:

As explained in Section 5.2, “Write-Ahead Logs”, the write-ahead log provides infinite storage at a logical level. But in the physical level, disk space must be reclaimed somehow. Because the common log is shared by multiple logical bricks, the technique described in Section 8.6, “Brick Checkpoint Operations” cannot be used by the common log.

A process called the “scavenger” is used to reclaim disk space in the common log. By default, the scavenger runs at 03:00 daily. The steps it executes are described below.

	The value of the brick_skip_live_percentage_greater_than configuration parameter determines how much additional disk space is required to store X gigabytes of live data. If the parameter is N, then 100-N percent of all common log disk space may be wasted by storing dead data.

	Additional disk space is required to log all updates that are made after the scavenger has run. This includes space in the common log as well as in each logical brick private logs (subject to the general limit of the brick_check_checkpoint_max_mb configuration parameter.

The current implementation of Hibari requires that plenty of disk space always be available for write-ahead logs and for scavenger operations. We strongly recommend that the brick_scavenger_temp_dir configuration item use a different file system than the application_data_dir parameter. This directory stores temporary files required for sorting and other operations that would otherwise require large amounts of RAM.

A table can be added at any time, using either of two methods:

	The current Hibari implementation does not support removing a table.

In theory, most of the work of removing a table is already done: chains that are abandoned after a migration are shut down * Brick pinger processes are stopped. * Chain monitor processes are stopped. * Bricks are stopped. * Brick data directories are removed.

All that remains is to update the Admin Server’s schema to remove references to the table.

The Hibari Admin Server manages each chain as an independent data replication entity. Though Hibari clients view multiple chains that are associated with a single table, each chain is actually independent of the other chains. It is possible to change the length of one chain without changing any others. For long term operation, such differences do not make sense. But during short periods of cluster reconfiguration, such differences are possible.

A chain’s length is determined by specifying a list of bricks that are members of that chain. The order of the list specifies the exact chain order when the chain is in the healthy state. By adding or removing bricks from a chain definition, the length of the chain can be changed.

A chain is defined by the Erlang 2-tuple of {ChainName, ListOfBricks}, where each brick in ListOfBricks is a 2-tuple {BrickName, NodeName}. For example, a chain of length two called footab_ch1 could be defined as:

{footab_ch1, [{footab1_ch1_b1, 'gdss1@box-a'}, {footab1_ch1_b1, 'gdss1@box-b'}]}

The current definition of all chains for table TableName can be retrieved from the Admin Server using the brick_admin:get_table_chain_list() function, for example:

%% Get a list of all tables currently defined.
> brick_admin:get_tables().
[tab1]

%% Get list of chains in 'tab1' as they are currently in operation.
> brick_admin:get_table_chain_list(tab1).
{ok,[{tab1_ch1,[{tab1_ch1_b1,'gdss1@machine-1'},
                {tab1_ch1_b2,'gdss1@machine-2'}]},
     {tab1_ch2,[{tab1_ch2_b1,'gdss1@machine-2'},
                {tab1_ch2_b2,'gdss1@machine-1'}]}]}

This above chain list for table tab1 corresponds to the chain and brick layout below.

Figure 12. Table tab1: Two chains of length two across two Erlang nodes on two physical machines

	To change the definition of a chain, use the change_chain_length/2 or change_chain_length/3 functions. For documentation, see Hibari Contributor’s Guide, "Changing Chain Length" section

	When specifying a new chain definition, at least one brick from the current chain must be included.

Removing a brick from a chain

Removing a brick B permanently from a chain is a simple operation. Brick B is handled the same way that any other brick failure is handled: the chain is simply reconfigured to exclude B. See Figure 11, “Chain order after a middle brick fails and is repaired (but not yet reordered)” for an example.

	When a brick B is removed from a chain, all data from brick B will be deleted when the operation is successful. At this time, the API does not have an option to allow B's data to be preserved.

There are several cases where it is desirable to rebalance data across chains and bricks in a Hibari cluster:

The same technique is used in all of these cases: chain migration. This mirrors the same design philosophy that’s used for handling chain changes (see the section called “Chain changes: same algorithm, different tasks.”): use the same algorithm to handle multiple use cases.

Example: Migrating from three chains to four

Figure 13. Chain migration from 3 chains to 4 chains

In the example above, both the 3-chain and 4-chain configurations used equal weighting factors. When all chains use the same weighting factor (e.g. 100), then the consistent hashing map in the “before” and “after” cases look something like the figure below.

Figure 14. Migration from three chains to four chains

It doesn’t matter that chain #4’s total area within the unit interval is divided into three regions. What matters is that chain #4’s total area is equal to the regions of the other three chains.

It is not strictly necessary to formally configure a list of all Hibari client nodes that may use a Hibari cluster. However, practically speaking, it is useful to do so.

To bootstrap itself to be able to use Hibari servers, a Hibari client must be able to:

To solve both problems, the Admin Server maintains a list of Hibari client nodes. (Hibari server nodes do not need this mechanism.) For each client node, a monitor process on the Admin Server polls the node to see if the gdss or gdss_client application is running. If the client node is running, then problem #1 (connecting to other nodes in the cluster) is automatically solved by using net_adm:ping/1. Problem #2 is solved by the client monitor calling brick_admin:spam_gh_to_all_nodes/0.

The Admin Server’s client monitor runs approximately once per second, so there may be a delay of up to a couple of seconds before a newly-started Hibari client node is connected to the rest of the cluster and has all of the table info required to start work.

When a client node goes down, an OTP alarm is raised until the client is up and running again.

Two methods can be used to view and change the client node monitor list:

Through the partition monitoring application, Hibari nodes send heartbeat messages to one another at the configurable heartbeat_beacon_interval, and each node keeps track of heartbeat history from each of the other nodes in the cluster. The heartbeats are transmitted through both Network A and Network B. If node gdss1@machine1 detects that the incoming heartbeats from gdss1@machine2 are absent both on Network A and on Network B, then gdss1@machine2 might have a problem. If the incoming heartbeats from gdss1@machine2 fail on Network A but not on Network B, a partition on Network A might be the cause. If heartbeats fail on Network B but not Network A, then Network B might have a partition problem, but this is less serious because Hibari data communication does not take place on Network B.

Configurable timers on each Hibari node determine the interval at which the absence of incoming heartbeats from another node is considered a problem. If on node gdss1@machine1 no heartbeat has been received from gdss1@machine2 for the duration of the configurable heartbeat_warning_interval, then a warning message is written to the application log of node gdss1@machine1. This warning message can be triggered by missing heartbeats either on Network A or on Network B; the warning message will indicate which node has not been heard from, and over which network.

If on node gdss1@machine1 no heartbeat has been received from gdss1@machine2 via Network A for the duration of the configurable heartbeat_failure_interval, and if during that period heartbeats from gdss1@machine2 continue to be received via Network B, then a network partition is presumed to have occurred in Network A. In this scenario, node gdss1@machine1 will attempt to ping the configurable network_a_tiebreaker address. If gdss1@machine1 successfully pings the tiebreaker address, then gdss1@machine1 considers itself to be on the "correct" side of the Network A partition, and it continues running. If by contrast gdss1@machine1 cannot successfully ping the tiebreaker address, then gdss1@machine1 considers itself to be on the "wrong" side of the Network A partition and shuts itself down. Meanwhile, comparable calculations and decisions are being made by node gdss1@machine2.

In a scenario where the network monitoring application determines that a partition has occurred on Network B — that is, heartbeats are received through Network A but not through Network B — then warnings are written to the Hibari nodes' application logs but no node is shut down.

At the time of writing, Hibari’s largest cluster deployment is:

If a backup were made of all data in the cluster, the biggest question is, "Where would you store the backup?" Given the cluster’s purpose (real-time email/messaging services), the quality of the data center’s physical and software infrastructures, the length of the Hibari chains used for physical data redundancy, the business factors influencing the choice not to deploy a "hot backup" data center, and other factors, Cloudian has not developed the backup and recovery software for Hibari. Cloudian’s smaller Hibari deployments also resemble the largest deployment.

However, we expect that backup and recovery software will be high priorities for open source Hibari users. Together with the open source users and developers, we expect this software to be developed relatively quickly.

Each log entry in the Hibari application log is composed of these fields in this order, with vertical bar delimitation:

<PID>|<<ERLANGPID>>|<DATETIME>|<MODULE>|<LEVEL>|<MESSAGECODE>|<MESSAGE>

This Hibari application log entry format is not configurable. Each of these application log entry fields is described in the table that follows. The “Position” column indicates the position of the field within a log entry.

Items written to the Hibari application log come from multiple sources:

The <MESSAGE> field is free-form text. Application code can freely add newline characters and various white-space padding wherever it wishes. However, the file format dictates that a newline character (ASCII 10) appear only at the end of the entire app log message.

The Hibari error logger must therefore reformat the text of the <MESSAGE> field to remove newlines and to remove whitespace padding. The result is not nearly as readable as the formatting presented to the Erlang shell. For example, within the shell, a message can look like this:

=PROGRESS REPORT==== 12-Apr-2010::17:49:22 ===
          supervisor: {local,sasl_safe_sup}
             started: [{pid,<0.43.0>},
                       {name,alarm_handler},
                       {mfa,{alarm_handler,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]

Within the Hibari application log, however, the same message is reformatted as line #2 below. The reformatted version is much more difficult for a human to read than the version above, but the purpose of the app log file is to be machine-parsable, not human-parsable.

8955|<0.54.0>|20100412174922|gmt_app      |INFO|2190301|start: normal []
8955|<0.55.0>|20100412174922|SASL         |INFO|2199999|progress: [{supervisor,{local,gmt_sup}},{started,[{pid,<0.56.0>},{name,gmt_config_svr},{mfa,{gmt_config_svr,start_link,["../priv/central.conf"]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]
8955|<0.55.0>|20100412174922|SASL         |INFO|2199999|progress: [{supervisor,{local,gmt_sup}},{started,[{pid,<0.57.0>},{name,gmt_tlog_svr},{mfa,{gmt_tlog_svr,start_link,[]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]
8955|<0.36.0>|20100412174922|SASL         |INFO|2199999|progress: [{supervisor,{local,kernel_safe_sup}},{started,[{pid,<0.59.0>},{name,timer_server},{mfa,{timer,start_link,[]}},{restart_type,permanent},{shutdown,1000},{child_type,worker}]}]
[...skipping ahead...]
8955|<0.7.0>|20100412174923|SASL         |INFO|2199999|progress: [{application,gdss},{started_at,gdss_dev2@bb3}]
8955|<0.98.0>|20100412174923|DEFAULT      |INFO|2199999|brick_sb: Admin Server not registered yet, retrying
8955|<0.65.0>|20100412174923|SASL         |INFO|2199999|progress: [{supervisor,{local,brick_admin_sup}},{started,[{pid,<0.98.0>},{name,brick_sb},{mfa,{brick_sb,start_link,[]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]
8955|<0.105.0>|20100412174924|DEFAULT      |INFO|2199999|top of init: bootstrap_copy1, [{implementation_module,brick_ets},{default_data_dir,"."}]
8955|<0.105.0>|20100412174924|DEFAULT      |INFO|2199999|do_init_second_half: bootstrap_copy1
8955|<0.79.0>|20100412174924|SASL         |INFO|2199999|progress: [{supervisor,{local,brick_brick_sup}},{started,[{pid,<0.105.0>},{name,bootstrap_copy1},{mfa,{brick_server,start_link,[bootstrap_copy1,[{default_data_dir,"."}]]}},{restart_type,temporary},{shutdown,2000},{child_type,worker}]}]
8955|<0.105.0>|20100412174924|DEFAULT      |INFO|2199999|do_init_second_half: bootstrap_copy1 finished

Hibari works quite well using commodity "Gigabit Ethernet" interfaces. Lower latency (and higher cost) networking gear, such as Infiniband, is not required.

For production use, it is strongly recommended that all Hibari servers be configured with two physical network interfaces, cabling, switches, etc. For more details, see:

Client protocol load balancing

The native Erlang client, via the gdss or gdss_client OTP applications, do not require any load balancing. The Erlang client already is a participant in the consistent hashing algorithm (see the section called “Partitioning by consistent hashing”). The Admin Server distributes updates to a table’s consistent hash map each time cluster membership or chain/brick status changes.

All other client access protocols are "dumb", by comparison. Take for example the Amazon S3 protocol service. There is no easy way for a Hibari cluster to convey to a generic HTTP client how to calculate which brick to send a query to. The HTTP redirect mechanism could be used for this purpose, but other protocols don’t have an equivalent feature. Also, the latency overhead of sending a redirect is far higher than Hibari’s solution to this problem.

Hibari’s solution is simple: the Hibari server-side "dumb" protocol handler uses the same native Erlang client that any other Hibari client app written in Erlang users. That client is capable of making direct routing decisions. Therefore, the "dumb" protocol handler within a Hibari node acts as a translating proxy: it uses the "dumb" client access protocol on one side and uses the native Erlang client API on the other.

Figure 17. Hibari "dumb" protocol proxy

The deployed "state of the art" for such dumb protocols is to use a TCP load balancer (aka a "layer 4" load balancer) to spread dumb client workload across multiple Hibari dumb protocol servers.

Hibari servers operate on top of the Erlang virtual machine. In principle, any operating system that is supported by the Erlang virtual machine can support Hibari.

A typical "server" type installation of a Linux or FreeBSD OS is sufficient for Hibari. The following is an incomplete list of other software packages that are necessary for Hibari’s installation and/or runtime.

There are several reasons why disk I/O rates can temporarily increase within a Hibari physical brick:

The Hibari central.conf file contains parameters that can limit the amount of disk bandwidth used by most of these operations.

See also:

The Admin Server’s status page contains current status information regarding all tables, chains, and bricks in the cluster. By default, this service listens to TCP port 23080 and is reachable via HTTP at http://any-hibari-node-name:23080/. HTTP redirect will steer your browser to the Admin Server node.

As yet, Hibari does not have a method to delete a table. The only methods available now are:

Adding or removing bricks from a single chain changes the replication factor for the keys stored in that chain: more bricks increases the replication factor, and fewer bricks decreases it.

Data types for brick_admin:change_chain_length(). 

brick_admin:change_chain_length(ChainName, BrickList)

ChainName       = atom()
BrickList       = [brick()]

brick()         = {logical_brick(), node()}
logical_brick() = atom()
node()          = atom()

See also, brick_admin:change_chain_length() usage examples Section 14.9, “Changing Chain Length: Examples”.

Data types for brick_admin:start_migration(). 

brick_admin:start_migration(TableName, LH)
  equivalent to brick_admin:start_migration(TableName, LH, [])

brick_admin:start_migration(TableName, LH, Options)
-> {ok, cookie()} |
   {'EXIT', term()}

TableName           = atom()
LH                  = hash_r()
Options             = migration_options()

cookie()            = term()
migration_option()  = {'do_not_initiate_serial_ack', boolean()} |
                      {'interval', integer()} |
                      {'max_keys_per_chain', integer()} |
                      {'max_keys_per_iter', integer()} |
                      {'propagation_delay', integer()}
migration_options() = [migration_option()]

brick_admin:chash_init('via_proplist', ChainList, Options)
-> hash_r()

ChainList = chain_list()
Options   = brick_options()

See the section called “Types for brick_admin:add_table()” for definitions of chain_list() and brick_options() types.

The hash_r() type is an Erlang record, #hash_r as defined in the brick_hash.hrl header file. It is normally considered an opaque type that is created by a function such as brick_hash:chash_init/3.

	The options list passed in argument #3 to brick_admin:chash_init/3 is the same properties list that is used for brick_admin:add_table/3. The difference is that the options that are related strictly to brick behavior, such as the do_logging and do_sync properties, are ignored by chash_init/3.

Once a hash_r() term is created and brick_admin:start_migration/2 is called successfully, the data migration will start immediately.

The cookie() type is an opaque term that uniquely identifies the data migration that was triggered for the TableName table. Another data migration may not be triggered until the current migration has finished successfully.

The migration_option() properties are described below:

See also Section 14.10, “Creating and Rebalancing Chains: Examples”.

The functions to change chain weighting are the same for adding/removing chains, see Section 14.4, “Change a Table: Add/Remove Chains” for additional details.

When creating a hash_r() type record, follow these two bits of advice:

See also Section 14.10, “Creating and Rebalancing Chains: Examples”.

See EDoc documentation for brick_admin.erl API.

See EDoc documentation for brick_sb.erl API.

See EDoc documentation for brick_chainmon.erl API.

The Admin Server’s basic definition of a chain: the chains name, and the list of bricks. In turn, each brick is defined by a 2-tuple of brick name and node name.

Chain definition = {ChainName, [BrickDef1, BrickDef2, ...]}
Brick definition = {BrickName, NodeName}

Example chain definition, chain length=1
    {tab1_ch1, [{tab1_ch1_b1, hibari1@bb3}]}

The function brick_admin:get_table_chain_list/1 will retrieve the active chain definition list for a table. For example, we retrieve the chain definition list for the table tab1. The node bb3 is the hostname of my laptop.

(hibari1@bb3)23> {ok, Tab1ChList} = brick_admin:get_table_chain_list(tab1).
{ok,[{tab1_ch1,[{tab1_ch1_b1,hibari1@bb3}]}]}

(hibari1@bb3)24> Tab1ChList.
[{tab1_ch1,[{tab1_ch1_b1,hibari1@bb3}]}]

	The brick_admin:get_table_chain_list/1 function will retrieve the active chain definition list for a table: only bricks that are in ok state will be shown. If a chain has a brick that has crashed, that brick will not appear in the list returned by this function. The brick_admin:get_table_info() function can fetch the list of all bricks, in service and crashed, but the API is not as convenient.

To change the chain length, use the brick_admin:change_chain_length/2 function. The arguments are the chain name and brick list.

	Any bricks in the brick list that aren’t in the chain are automatically started. Any bricks in the current chain that are not in the new list are halted, and their persistent data will be deleted.

(hibari1@bb3)29> brick_admin:change_chain_length(tab1_ch1,
                 [{tab1_ch1_b1,hibari1@bb3}, {tab1_ch1_b2,hibari1@bb3}]).
ok

(hibari1@bb3)30> {ok, Tab1ChList2} = brick_admin:get_table_chain_list(tab1). {ok,[{tab1_ch1,[{tab1_ch1_b1,hibari1@bb3},
                {tab1_ch1_b2,hibari1@bb3}]}]}

Now the tab1_ch1 chain has length two. We’ll shorten it back down to length 1.

(hibari1@bb3)31> brick_admin:change_chain_length(tab1_ch1, [{tab1_ch1_b2,hibari1@bb3}]).
ok

(hibari1@bb3)32> {ok, Tab1ChList3} = brick_admin:get_table_chain_list(tab1).
{ok,[{tab1_ch1,[{tab1_ch1_b2,hibari1@bb3}]}]}

	A chain’s new brick list must contain at least one brick from the current chain’s definition. If the intersection of old brick list and new brick list is empty, the command will fail.

(hibari1@bb3)34> brick_admin:change_chain_length(tab1_ch1, [{tab1_ch1_b3,hibari1@bb3}]).
{'EXIT',{error,no_intersection}}

The procedure for creating new chains, deleting existing chains, and reweighing existing chains, and rehashing is done using the the brick_admin:start_migration() function. The chain definitions are specified in the same way as changing chain lengths, see Section 14.9, “Changing Chain Length: Examples” for details.

The data structure required by brick_admin:start_migration/2 is more complex than the relatively-simple brick list that brick_admin:change_chain_length/2 requires. This section will demonstrate the creation of this structure, the “local hash record”, step-by-step.

First, we create a new chain definition list. (Refer to Section 14.9, “Changing Chain Length: Examples” if necessary.) For this example, we’ll assume that we’ll be modifying the tab1 table and that we’ll be adding two more chains. Each chain will be of length one. We’ll place each chain on the same node as everything else, hibari1@bb3 (i.e. my laptop).

(hibari1@bb3)48> brick_admin:get_table_chain_list(tab1).
{ok,[{tab1_ch1,[{tab1_ch1_b1,hibari1@bb3}]}]}

(hibari1@bb3)49> NewCL = [{tab1_ch1, [{tab1_ch1_b1, hibari1@bb3}]},
                 {tab1_ch2, [{tab1_ch2_b1, hibari1@bb3}]},
                 {tab1_ch3, [{tab1_ch3_b1, hibari1@bb3}]}].
[{tab1_ch1,[{tab1_ch1_b1,hibari1@bb3}]},
 {tab1_ch2,[{tab1_ch2_b1,hibari1@bb3}]},
 {tab1_ch3,[{tab1_ch3_b1,hibari1@bb3}]}]

	Any bricks in the brick list that aren’t in a chain are automatically started. Any bricks in a current chains that are not in the chain definition are halted, and their persistent data will be deleted.

Next, we retrieve the table’s current hashing configuration. The data is returned to us in the form of an Erlang property list. (See the Erlang/OTP documentation for the proplists module, located in the "Basic Applications" area under "stdlib".) We then pick out several properties that we’ll need later; we use lists:keyfind/3 instead of a function in the proplists module because it will preserve the properties in 2-tuple form, which will save us some typing effort later.

(hibari1@bb3)51> {ok, TabInfo} = brick_admin:get_table_info(tab1).
{ok,[{name,tab1},
    ...lots of stuff omitted...

(hibari1@bb3)53> Opts = proplists:get_value(brick_options, TabInfo).
[{hash_init,#Fun<brick_hash.chash_init.3>},
 {old_float_map,[]},
 {new_chainweights,[{tab1_ch1,100}]},
 {hash_init,#Fun<brick_hash.chash_init.3>},
 {prefix_method,var_prefix},
 {prefix_separator,47},
 {num_separators,3},
 {bigdata_dir,"cwd"},
 {do_logging,true},
 {do_sync,true},
 {created_date,{2010,4,17}},
 {created_time,{17,21,58}}]

(hibari1@bb3)58> PrefixMethod = lists:keyfind(prefix_method, 1, Opts).
{prefix_method,var_prefix}

(hibari1@bb3)59> NumSep = lists:keyfind(num_separators, 1, Opts).
{num_separators,3}

(hibari1@bb3)60> PrefixSep = lists:keyfind(prefix_separator, 1, Opts).
{prefix_separator,47}

(hibari1@bb3)61> OldCWs = proplists:get_value(new_chainweights, Opts).
[{tab1_ch1,100}]

(hibari1@bb3)62> OldGH = proplists:get_value(ghash, TabInfo).

(hibari1@bb3)63> OldFloatMap = brick_hash:chash_extract_new_float_map(OldGH).

Next, we create a new property list.

(hibari1@bb3)71> NewCWs = OldCWs ++ [{tab1_ch2, 100}, {tab1_ch3, 100}].
[{tab1_ch1,100},{tab1_ch2,100},{tab1_ch3,100}]

(hibari1@bb3)72> NewOpts = [PrefixMethod, NumSep, PrefixSep,
                               {new_chainweights, NewCWs},
                               {old_float_map, OldFloatMap}].
[{prefix_method,var_prefix},
 {num_separators,3},
 {prefix_separator,47},
 {new_chainweights,[{tab1_ch1,100},
                    {tab1_ch2,100},
                    {tab1_ch3,100}]}
 {old_float_map, []}]

Next, we use the chain definition list, NewCL, and the table options list, NewOpts, to create a “local hash” record. This record will contain all of the configuration information required to change a table’s consistent hashing characteristics.

(hibari1@bb3)73> NewLH = brick_hash:chash_init(via_proplist, NewCL, NewOpts).
{hash_r,chash,brick_hash,chash_key_to_chain,
 ...lots of stuff omitted...

We’re just one step away from changing the tab1 table. Before we change the table, however, we’d like to see how the table change will affect the data in the table. First, we add 1,000 keys to the tab1 table. Then we use the brick_simple:chash_migration_pre_check/2 function to tell us how many keys will move and to where.

(hibari1@bb3)74> [brick_simple:set(tab1, "foo"++integer_to_list(X), "bar") || X <- lists:seq(1,1000)].
[ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,
 ok,ok,ok,ok,ok,ok,ok,ok,ok,ok|...]

(hibari1@bb3)75> brick_simple:chash_migration_pre_check(tab1, NewLH).
[{keys_before,[{tab1_ch1,1001}]},
 {keys_keep,[{tab1_ch1,348}]},
 {keys_moving,[{tab1_ch2,315},{tab1_ch3,338}]},
 {keys_moving_where,[{tab1_ch1,[{tab1_ch2,315},
                                {tab1_ch3,338}]}]},
 {errors,[]}]

The output above shows us that of the 1,001 keys in the tab1 table, 348 will remain in the tab1_ch1 chain, 315 keys will move to the tab1_ch2 chain, and 338 keys will move to the tab1_ch3 chain. That looks like what we want, so let’s reconfigure the table and start the data migration.

brick_admin:start_migration(tab1, NewLH).

Immediately, we’ll see a bunch of application messages sent to the console as new activities start:

=GMT INFO REPORT==== 20-Apr-2010::00:26:40 ===
Migration number 1 is starting with cookie {1271,741200,988900}

=GMT INFO REPORT==== 20-Apr-2010::00:26:41 ===
progress: [{supervisor,{local,brick_mon_sup}},
           {started,
               [{pid,<0.2937.0>},
                {name,chmon_tab1_ch2},
                ...stuff omitted...

[...lines skipped...]
=GMT INFO REPORT==== 20-Apr-2010::00:26:41 ===
Migration monitor: tab1: chains starting

[...lines skipped...]
=GMT INFO REPORT==== 20-Apr-2010::00:26:41 ===
brick_admin: handle_cast: chain tab1_ch2 in unknown state

[...lines skipped...]
=GMT INFO REPORT==== 20-Apr-2010::00:26:52 ===
Migration monitor: tab1: sweeps starting

[...lines skipped...]
=GMT INFO REPORT==== 20-Apr-2010::00:26:54 ===
Migration number 1 finished

[...lines skipped...]
=GMT INFO REPORT==== 20-Apr-2010::00:26:57 ===
Clearing final migration state for table tab1

For the sake of demonstration, now let’s see what brick_simple:chash_migration_pre_check() would say if we were to migrate from three chains to four chains.

(hibari_dev@bb3)24> {ok, TabInfo3} = brick_admin:get_table_info(tab1).

(hibari_dev@bb3)25> Opts3 = proplists:get_value(brick_options, TabInfo3).

(hibari_dev@bb3)26> GH3 = proplists:get_value(ghash, TabInfo3).

(hibari_dev@bb3)28> OldFloatMap = brick_hash:chash_extract_new_float_map(GH3).

(hibari_dev@bb3)31> NewOpts4 = [PrefixMethod, NumSep, PrefixSep,
                    {new_chainweights, NewCWs4}, {old_float_map, OldFloatMap}].

(hibari_dev@bb3)35> NewCL4 = [ {tab1_ch1, [{tab1_ch1_b1, hibari1@bb3}]},
                               {tab1_ch2, [{tab1_ch2_b1, hibari1@bb3}]},
                               {tab1_ch3, [{tab1_ch3_b1, hibari1@bb3}]},
                               {tab1_ch4, [{tab1_ch4_b1, hibari1@bb3}]} ].
(hibari_dev@bb3)36> NewLH4 = brick_hash:chash_init(via_proplist, NewCL4, NewOpts4).

(hibari_dev@bb3)37> brick_simple:chash_migration_pre_check(tab1, NewLH4).
[{keys_before,[{tab1_ch1,349},
               {tab1_ch2,315},
               {tab1_ch3,337}]},
 {keys_keep,[{tab1_ch1,250},{tab1_ch2,232},{tab1_ch3,232}]},
 {keys_moving,[{tab1_ch4,287}]},
 {keys_moving_where,[{tab1_ch1,[{tab1_ch4,99}]},
                     {tab1_ch2,[{tab1_ch4,83}]},
                     {tab1_ch3,[{tab1_ch4,105}]}]},
 {errors,[]}]

The output tells us that chain tab1_ch1 will lose 99 keys, tab1_ch2 will lose 83 keys, and tab1_ch3 will lose 105 keys. The final key distribution across the four chains would be 250, 232, 232, and 287 keys, respectively.