Reverbrain wiki

Site Tools



Assignance: rbtz@


ETA: 1 week

WIP repository:


Eblob should be able to copy all data modification operations and subsequent reads to separate binary log while preforming batch tasks(like sorting data in blob). This would result in non-consistent blob and binary log with all modifications. By locking blob and applying binlog onto it consistency will be restored.


Work is split into there parts:

  • Implement binary log.
  • Plug it into existing blob infrastructure.
  • Enable it on demand when async data-sort starts.

Technical information

General info

Begin with simple block based binary log implementation. As opposed to statement-based logging that stores records like “remove key 123” or “overwrite key 321 with following data” we store data like “at offset 000 following data was written”.

For now binlog supports redo-only operations (in other words no rollback data is saved).

Binlog is only used when associated blob(s) are sorted. Since we do not lock blob, all its updates have to be stored somewhere. Only 'closed' blob can be sorted, and 'closed' here means current operations go to other blobs, namely in append-only mode 'closed' blob can only be written, when record, stored in given blob, is removed. In overwrite mode some records in 'closed' blob can be updated, so we update data in place as well as ыещку appropriate record into binlog. When blob is finished sorting, we apply (rewind) all records from binlog on top of the sorted blob.

On disk format


File starts with 256 byte header which consists of magic, version, flags and reserved space for future use. Then records go: each record consist of record header, metadata passed by calling routine(for example original entry's on-disk header) and data itself(which may be null). This is quite simple approach that can be generalized even further like: append following IOVs to the binlog.


After binlog has been initialized(for example via data-sort routine) for some base it's →binlog member points to binlog control structure.

If eblob_ram_control instance has non-NULL →bctl→binlog following hook is enabled:

  • eblob_write_binlog - simple pwrite(2) wrapper that copies written data to binlog via binlog_append()

To apply this binlog to sorted blob data-sort routine keeps track of sorted-to-unsorted offset mapping while performs sort operation.


Very simple but powerful binlog API, for now it supports mostly append and apply functions read as available but used only internally because currently there is no way to obtain offset of appended record. In future this may be changed by utilizing binlog_append return value.

int binlog_open(struct eblob_binlog_cfg *bcfg);
int binlog_append(struct eblob_binlog_ctl *bctl);
int binlog_read(struct eblob_binlog_ctl *bctl, off_t offset);
int binlog_apply(struct eblob_binlog_cfg *bcfg, void *priv, int (*callback)(void *priv, struct eblob_binlog_ctl *bctl));
int binlog_close(struct eblob_binlog_cfg *bcfg);
int binlog_destroy(struct eblob_binlog_cfg *bcfg)

Work Items

  • basic operations like: init/open/close/destroy binlog, append data to binlog, read data from binlog
  • plug into existing codebase
  • unblock data-sort
blueprints/eblob/binlog.txt · Last modified: 2012/12/27 20:31 by savetherbtz