Reverbrain wiki

Site Tools

  1. callback structure
  2. dnet_id structure
  3. dnet_cmd structure
  4. dnet_io_attr structure
  5. dnet_file_info structure
  6. dnet_stat structure
  7. dnet_addr_stat structure
  8. sph structure
  9. dnet_addr structure
  10. dnet_node_status structure
  11. dnet_io_control structure
  12. dnet_cmd and cflags flags
  13. ioflags flags

callback structure

The network connection, the command with a reply (struct dnet_cmd) and optional data is passing to the callback (struct dnet_net_state). There are several types of events (valid response, error, the last / not the last call, etc.), that can be passed to the callback. They are defined by dnet_cmd flags, as well as by the presence of a non-zero dnet_net_state and dnet_cmd.

The supporting function is_trans_destroyed (struct dnet_net_state *, struct dnet_cmd *) must be called in the first callback. If it returns non-zero value, it means that this transaction guarantee there won’t be any more operations. However, this does not mean that the transaction was sent or processed, only that the transaction will be destroyed after the release from the callback.

For example, if a command was sent to the non-existent group at the routing table, the callback will be called immediately, and is_trans_destroyed() immediately will return the positive value. We recommend to leave the handler to «wake» synchronous client, etc. upon receipt of a positive value of is_trans_destroyed() .

If is_trans_destroyed() returns 0, then it is a logical response from the server. As a minimum, it is guaranteed that the transferred dnet_net_state and dnet_cmd are valid, so you can see the command status (dnet_cmd->status), look at the size of the passed data in response, etc.

There is an aknowledge that the operation is complete at the end of a transaction in case If the client has set DNET_FLAGS_NEED_ACK flag.

Acknowledge is dnet_cmd structure that was sent in response to some request. It does not cling to any additional data, there is only dnet_cmd, inside of which is the status: if it is 0 - all is right; if it is not equal to 0 - there was some error (with an errno-code).

Callback example:

static int stat_complete(struct dnet_net_state *state,
                         struct dnet_cmd *cmd,
                         void *priv)


  • state - is a structure that defines the network connection (collecting statistics from the nodes, etc.), dnet_net_state - is a structure that describes a network connection from the remote server. struct dnet_net_state contains an IP-address and the routing table.

  • dnet_cmd - the basic structure of a low-level messaging protocol - each message starts with this structure. This structure is arranged so that after dnet_cmd there is a response and the size of this response is written at dnet_cmd.size.

  • priv - pointer to user data: when you create a transaction (when you run the commands for reading, writting or requesting the statistics) you can set your own data, and its pointers are passing to the priv.

dnet_id structure

Elliptics is the indexed routing table. Assume that the user wants to write the data somewhere. By default we take its file called «abcd», take the title, do sha512 hash of it and get a 64-byte value, the key size (64 bytes, 512 bits) is set on the compile stage and theoretically can be changed as a hash function itself.

The resulting hash is the id record, which is placed in dnet_id. Also, you can do not make a hash of the file name, and write your own identifier in these 64 bytes.

struct dnet_id {
        uint8_t                        id[DNET_ID_SIZE];
        uint32_t                        group_id;
        int                                type;
} __attribute__ ((packed));


  • id - an identifier that stored here (by default — hash), DNET_ID_SIZE is 64 bytes by default;

  • group_id - replica identifier;

  • type - column number;

By default dnet_id contains a hash of the file name that the user wants to write, to read or to delete. But you can set an identifier all alone.

dnet_raw_id structure

This structure is a dnet_id substructure and does not include the group or type (column).

dnet_cmd structure

struct dnet_cmd
        struct dnet_id                id;
        int                             status;
        int                             cmd;
        uint64_t                     flags;
        uint64_t                     trans;
        uint64_t                     size;
        uint8_t                        data[0];
} __attribute__ ((packed));

dnet_id is the key with that you want to execute the command - the identifier that used inside Elliptics.


  • status - status of complition of operation: 0 - operation was successful, a negative value - errno-error code;

  • cmd — a command that user calls;

  • flags - flags associated with the command (DNET_FLAGS_ *);

  • trans - number of a transaction that Elliptics generates and send to the request, the top bit is 1, when a reply is recieved;

  • size - how many bytes after dnet_cmd belongs to the request or response.

dnet_io_attr structure

dnet_io_attr structure consists of a structure for storing the information about request.

struct dnet_io_attr
    uint8_t         parent[DNET_ID_SIZE];
    uint8_t         id[DNET_ID_SIZE];

    uint64_t        start, num;
    int         type;
    uint32_t        flags;
    uint64_t        offset;
    uint64_t        size;
} __attribute__ ((packed));

static inline void dnet_convert_io_attr(struct dnet_io_attr *a)
    a->start = dnet_bswap64(a->start);
    a->num = dnet_bswap64(a->num);

    a->flags = dnet_bswap32(a->flags);
    a->offset = dnet_bswap64(a->offset);
    a->size = dnet_bswap64(a->size);


  • id — id of the object key;

  • parent — an id that can be used in read_range(), write_range() and remove_range() requests;

  • flags — flags to make a request with (DNET_IO_FLAGS_ *);

  • offset — in the case of reading it is the offset from that we begin to read the data on the server, for writting it’s an offset by which we need to write the data;

  • size — how many bytes we read or write. If the size for reading is equal to 0, then all of the data will be read.

  • start, num — are used in the range-queries, for example, if we have 1,000 keys it may come to a request range, start = 100, num = 50, we’ll receive all id, starting with 100 and ending with the 149th. Also start is used in the write_cache() request, and then it is treated as an object lifetime in seconds - that is, if start = 0 it is considered that the subject does not "die" by time-out while it in the cache, otherwise it should be removed from the cache in start-seconds.

dnet_file_info structure

struct dnet_file_info {
    int         flen;       
    unsigned char       checksum[DNET_CSUM_SIZE];
    unsigned int        nlink;
    uint64_t        mode;
    uint64_t        dev;
    uint64_t        rdev;
    uint64_t        ino;
    uint64_t        uid;
    uint64_t        gid;
    uint64_t        blksize;
    uint64_t        blocks;
    uint64_t        size;
    uint64_t        offset;     
    struct dnet_time    atime;
    struct dnet_time    ctime;
    struct dnet_time    mtime;


  • flen tells us about the file name length to which we can apply via file_path() method at lookup_result_entry. If flen = 0, then when you call that method the exception will be thrown;

  • ckecksum keeps a check-sum of the data, you can use it in the write_cas() method;

  • nlink, mode, dev, rdev, ino, uid, gid, atime matche stat structure, you can read about it at the stat(2) (man 2 stat);

  • size tells us about file size;

  • offset tells us about the offset by which you can find a data record with that key in the file. Applies only to eblob backend;

  • blocks/blksize — see stat(2);

  • atime - access time to the file (not used and not updated);

  • mtime - file modification time;

  • ctime - file creation time.

dnet_time structure

dnet_time structure indicates the time.

struct dnet_time {
    uint64_t        tsec, tnsec;


  • tsec — the number of seconds (integer part of a second) since the epoch;

  • tnsec — the number of nanoseconds since the start of the second (the fractional part of a second).

dnet_stat structure

dnet_stat structure can be found here.


  • la — an average load on the system, it is written in the form of interests;

  • namemax — the maximum file name allowed in the system;

  • bsize — block size (man 2 statvfs);

  • frsize — size of the fragments;

  • blocks — the number of the blocks in the system;

  • bfree — the number of free blocks;

  • bavail — the number of blocks available for non-root;

  • files — the number of inodes (the files created directly on the disk);

  • ffree — the number of free inodes;

  • favail — number of free inodes available for no-root;

  • fsid — id of the file system

  • flags — the flags that were transferred to the operating system while the device was fixed;

  • vm_active — the number of active virtual memory (man proc -> /proc /meminfo);

  • vm_inactive — number of not active virtual memory;

  • vm_total — the total amount of the virtual memory;

  • vm_free — the amount of free virtual memory;

  • vm_cached — the amount of memory allocated for caching an information;

  • vm_buffers — the amount of memory allocated for the buffers.

dnet_addr_stat structure

struct dnet_addr_stat
    struct dnet_addr        addr;
    int             num;
    int             cmd_num;
    struct dnet_stat_count      count[0];
} __attribute__ ((packed));


  • addr — stores an address;

  • num — an amount of the client counters at the reply;

  • cmd_num — the number of server counters - the same as the num.

struct dnet_stat_count
    uint64_t            count;
    uint64_t            err;

After that structure there is num-structures of dnet_stat_count (part of dnet_addr_stat structure) in memory, which has two fields:

  • count — returns the number of elements corresponding to the given object, there is also an array of dnet_stat_count objects stored, each element of which corresponds to its index by enum dnet_counters. After them there is an array of the same objects at the memory, but the corresponding enum dnet_commands.

  • err — returns an amount of the errors.

sph structure

This is a low-level structure, used to communicate between the client and the server to process the tasks.


  • src — id that expecting to receive a response after the completion of all tasks.

  • flags — flags peculiar to a particular transaction, which include DNET_SPH_FLAGS_SRC_BLOCK, DNET_SPH_FLAGS_REPLY and DNET_SPH_FLAGS_FINISH.

  • DNET_SPH_FLAGS_SRC_BLOCK — if this flag is set at the request, then the performing of the command shown in this sph will block the client - he will not receive the final acknowledge until the application will send it itself.

  • DNET_SPH_FLAGS_REPLY — means that the data of sph is a response to a request that was sent previously.

  • DNET_SPH_FLAGS_FINISH — means that this is the last of the answers, and you need to unlock (to send the final acknowledge) the client that sent the request with DNET_SPH_FLAGS_SRC_BLOCK flag.

  • status — contains the information about a success or failure of the operation; negative errno code or zero on success.

  • key and src_key — internal fields that can be used for communication between the source of the request and a worker where the task was sent; then we will be able to understand from the answer to whom it applies in case if a worker don’t answer for a few times.

  • addr — includes the address of the node that originally sent sph data.

  • data — at the beginning of event_size byte is reserved for the name of the event (the event represents the application name, & and the name of an event), then data_size byte contains the data to be processed in the course of this request.

dnet_addr structure

struct dnet_addr
    uint8_t     addr[DNET_ADDR_SIZE];
    uint16_t    addr_len;
    uint16_t    family;
} __attribute__ ((packed));

Structure contains an address (IP-address, port, etc.).


  • addr — an array of bytes, that can be lead to the structure sockaddr_in or sockaddr_in6;

  • addr_len — how many bytes in addr array is used to store an address;

  • family — tells us about the family of network connections, that is ipv4 (2 - AF_INET) or ipv6 (10 - AF_INET6).

dnet_node_status structure

dnet_node_status is the node status.

struct dnet_node_status {
    int nflags;
    int status_flags;  
    uint32_t log_level;


  • nflags — flags that the node started with;

  • status_flags — either DNET_STATUS_EXIT (1) or DNET_STATUS_RO (2) flags should be specified here;

  • log_level — the level of logging that is enabled on this node.

You can use dnet_ioclient for changing these settings. For example, the following command will change «log-level» to «log-level-notice» (3):

$ dnet_ioclient -r address:port:family -U0 -M3

dnet_io_control structure

It represents a [structure] ( that describes the action that the client requests the server to execute.


  • id — stores the id of the element on which you need to do the action;

  • io — detailed parameters of actions to be performed;

  • complete — a pointer to a function that will be called when the server will receive some answers;

  • priv — a pointer that will be passed to the function specified at the «complete», the user can send his data specific to this request there;

  • data — stores a pointer to the data that you want to transfer;

  • fd — file descriptor from which the client can take the data for sending to the server;

  • local_offset — an offset to a local file, for example, if we want to send only a certain number of bytes from the file starting with the certain place (local_offset);

  • cmd — id of the command; for example, DNET_CMD_READ, DNET_CMD_WRITE and other​enum dnet_commands values;

  • cflags — flags that are passed in the command;

  • ts — timestamp of the operation. For example, if we want to write the file with a timestamp like «month ago», then ts must match this time.

dnet_commands — it's all possible commands that can be added to dnet_io_control (each command is described at the reference).

dnet_cmd and cflags flags

  • DNET_FLAGS_NEED_ACK — the server must send an acknowledge after the last packet of a response, which includes an error code (negative value - standard errno code, if successful — zero). If an error occurs you will receive only an acknowledge.

  • DNET_FLAGS_MORE — there is nothing stopping the server to split its response into the chunks and send multiple responses (with or without the data) to a user command. This flag is set when the server knows that after this message it has to be one or more subsequent messages (including an acknowledge). In the last (usually the final acknowledge) message in response to a command that flag is not set. This flag can be checked in the callback to determine if more data will be sent to the client.

  • DNET_FLAGS_DESTROY — this flag is set before the destruction of the transaction. This is the guaranteed last call to this callback for this transaction.

  • DNET_FLAGS_DIRECT — if a node has received a command with the id within a range that does not belong to the area of responsibility of this node, this flag makes it so that the node won’t send the command to the node that is responsible for the id zone and processes the command itself.

  • DNET_FLAGS_NOLOCK — by default, all transactions within the group are executed atomically. The appropriate local lock is taken on the node responsible for that id. In the presence of the flag the lock is not taken.

Description of additional flags is here.

Description of cflags is here.

ioflags flags

  • DNET_IO_FLAGS_COMPRESS - compresses the data on the disk.

  • DNET_IO_FLAGS_PREPARE - preallocation of the certain amount of space on the disk for the data to be sent in chunks.

  • DNET_IO_FLAGS_PLAIN_WRITE - an intermediate command between prepare and commit - writting of the data without updating the index.

  • DNET_IO_FLAGS_COMMIT - update the index for the key after a space has been reserved with DNET_IO_FLAGS_PREPARE flag and the data were recorded with DNET_IO_FLAGS_PLAIN_WRITE flag several times.

  • DNET_IO_FLAGS_APPEND - if you call «write» with this flag, the data will be appended rather than rewritten.

  • DNET_IO_FLAGS_REMOVED — to delete a file.

  • DNET_IO_FLAGS_OVERWRITE - overwrite a file if it already exists; if the file does not exist - uses simple write. It is relevant only for eblob backend, which by default uses the append-only mode. Overwrite mode must be enabled at the eblob config.

  • DNET_IO_FLAGS_NOCSUM - don’t receive, don’t generate and don’t verify a check-sum.

  • DNET_IO_FLAGS_NODATA - send the data statistics in the request range, not the data.

  • DNET_IO_FLAGS_META - request for an metadata operation, such as reading metadata and not the data by the specified key.

  • DNET_IO_FLAGS_CACHE - this flag indicates that we want to work with the cache too - reading and writing will not only go to the disk but also into the cache. When reading the data will be read from the cache at first, and if they are not there, then from the backend.

  • DNET_IO_FLAGS_CACHE_ONLY - this flag indicates that we want to work only with the cache.

  • DNET_IO_FLAGS_CACHE_REMOVE_FROM_DISK - write to the cache, and when the data is deleted from it, also remove it from the disc.

  • DNET_IO_FLAGS_COMPARE_AND_SWAP - check a check-sum of the file that is now really on the server with the check-sum, which is passed by the client and performs the action only if the check-sums are the same; this is necessary to make sure that we overwrite exactly the data that we assume to be there initially.

  • DNET_IO_FLAGS_CHECKSUM - the data checksum will be put into the dnet_io_attr.parent during the read operation.

elliptics/api-low-level-data-structures.txt · Last modified: 2013/06/21 17:58 by iidioteque