Reverbrain wiki

Site Tools


Atomic operations

This is a rather common task to perform server-side processing in Elliptics, for example one can parse input data and split message header and body and then upload both using different keys.

Such server-side operation is the same first class citizen as read, write and other commands. And thus it obeys its rules - node which handles given operation is selected through to the same process.

First, client uses its route table to find node in hash ring in given group (hash ring is created per group), which is responsible to cover ID of your operation (it can be hashed string, or manually created 64-byte ID). Client then sends command (IO or EXEC, which is acually a command to execute server-side application) to this node and attaches data to be processed. This data can be raw binary data or ID of the record client already wrote.

Either IO command or EXEC - it is processed by elliptics under lock derived from ID you are using. You can specify DNET_FLAGS_NOLOCK flag in dnet_cmd structure to prevent lock from being taken - in this case multiple commands (IO, EXEC or any other) with the same ID can be executed in parallel. We use this flag for statistics gathering or route reading for example.

This basically means that any operation in elliptics is atomic in single group (if no special flag is set).

But there no locks between groups are ever taken - this requires things like Paxos or Zookeeper broadcast protocol to be implemented, which definitely kills performance. Or there might be a single-master design, which has quite obvious problems.

Server-side processing and locks

When elliptics receives EXEC command and there is no NOLOCK flag, it will lock given key and no other locked (i.e. those which do not have NOLOCK flag) operation with the same key (ID) can be executed. This means locked READ or WRITE will be blocked waiting for original locked operation (EXEC in this example) to complete.

Elliptics server-side engine invokes cocaine engine, which uses 0mq to send your data to one of the processes (or it can be container in later versions) which were created when your application was started. Worker executes your code with data it received, and reply is being sent back, which is returned back to client.

Client's code (it is called 'application' in cocaine's terms) can read or write any keys in turn. It can even start execution of other commands. And if those operations are locked too and they operate on the same key as original locked EXEC command, then application will deadlock. Eventually it will timeout on client's side.

It is recommended to either lock EXEC command and then use nolocking (session.set_cflags(DNET_FLAGS_NOLOCK)) commands, or use nonlocked EXEC and locked commands inside. In the latter case EXEC command will not be atomic.

Compare and swap write

One can use compare-and-swap write in that case. One has to provide checksum of the original data, which is supposed to be overwritten, and new data. Server will checksum existing data on disk and compare checksum to what client provided, if they match, server will overwrite existing data with data provided by client, otherwise error is returned.

To enable CAS write client must use appropriate C++ method, where checksum and data are provided. Typical application uses session.read_data(), then checksum it using session.transform(std::string &data, struct dnet_id &csum_id), and then write_cas(), where new data and csum_id are provded.

elliptics/atomic-operations.txt ยท Last modified: 2013/01/23 02:33 by zbr