Reverbrain wiki

Site Tools


elliptics:configuration

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
elliptics:configuration [2014/09/24 19:47]
zbr [Elliptics backends section]
elliptics:configuration [2019/02/11 16:58] (current)
zbr [Elliptics logger section]
Line 16: Line 16:
 ==== Elliptics logger section ==== ==== Elliptics logger section ====
 ^ Property ​ ^ Description ^ ^ Property ​ ^ Description ^
-|**frontends** = [ ... ] | Describes frontends where logs have to be pushed, this section is described at [[http://3hren.github.io/blackhole/|blackhole'​s documentation]].|+|**frontends** = [ ... ] | Describes frontends where logs have to be pushed, this section is described at [[https://github.com/3hren/​blackhole|blackhole'​s documentation]].|
 | **level** = "​debug"​ | Sets verbosity level of elliptics logs.\\ It doesn'​t corresponds to syslog levels. When logging to syslog almost all messages will be written with INFO level. Elliptics supports follow levels:\\ "​error",​ "​warning",​ "​info",​ "​notice",​ "​debug"​. | | **level** = "​debug"​ | Sets verbosity level of elliptics logs.\\ It doesn'​t corresponds to syslog levels. When logging to syslog almost all messages will be written with INFO level. Elliptics supports follow levels:\\ "​error",​ "​warning",​ "​info",​ "​notice",​ "​debug"​. |
 ==== Elliptics options section ==== ==== Elliptics options section ====
 ^ Property ​ ^ Description ^ ^ Property ​ ^ Description ^
 | **join** = true | Specifies whether to join storage network.\\ Server nodes should use true. Client nodes don't use this parameter or set it to false.| | **join** = true | Specifies whether to join storage network.\\ Server nodes should use true. Client nodes don't use this parameter or set it to false.|
-| **flags** = 4 | Bitset flags influences to server behaviour.\\ Bits can be set in any variations, but in case of bits 2 and 5 set both, 2 will be used. It should include follow bits:\\ **bit 1 (flags=1)** - do not request remote route table\\ **bit 2 (flags=4)** - mix states before read operations according to state'​s weights\\ **bit 3 (flags=8)** - do not checksum data on upload and check it during data read\\ **bit 4 (flags=16)** - do not update metadata at all\\ **bit 5 (flags=32)** - randomize states for read requests\\ **bit 6 (flags=64)** - keeps ids in elliptics cluster | +| **flags** = 4 | Bitset flags influences to server behaviour.\\ Bits can be set in any variations, but in case of bits 2 and 5 set both, 2 will be used. It should include follow bits:\\ **bit 1 (flags=2)** - do not request remote route table\\ **bit 2 (flags=4)** - mix states before read operations according to state'​s weights\\ **bit 3 (flags=8)** - do not checksum data on upload and check it during data read\\ **bit 4 (flags=16)** - do not update metadata at all\\ **bit 5 (flags=32)** - randomize states for read requests\\ **bit 6 (flags=64)** - keeps ids in elliptics cluster | 
-| **remote** = [ "​1.2.3.4:​1025:​2",​ "​2.3.4.5:​2345:2", "​autodiscovery:​224.0.0.5:​1025:2" ] | List of remote nodes to connect.\\ Address should have follow format: `address:​port:​family` where family is either 2 (AF_INET) or 10 (AF_INET6) and address can be host name or IP of remote node.\\ Multicast doesn'​t used this time.\\ It is possible to autodiscover remote clusters via multicast. If you put `autodiscovery:​address:​port:​family` where `address:​port:​family` is valid multicast address, elliptics will broadcast information about itself and remote nodes with the same auth cookie will receive this information and connect to given node. Multicast TTL equals to 3.|+| **remote** = [ "​1.2.3.4:​1025:​2",​ "​2.3.4.5:​2345:​2"​ ] | List of remote nodes to connect.\\ Address should have follow format: `address:​port:​family` where family is either 2 (AF_INET) or 10 (AF_INET6) and address can be host name or IP of remote node.|
 | **address** = [ "​localhost:​1025:​2-0",​ "​10.10.0.1:​1025:​2-1"​ ] | List of node address which enables multiple interface support.\\ Elliptics server listens on port specified in this config option (format: addr:​port:​family-route_group)\\ If ports differ, the last one will be used.\\ Elliptics server will accept connections from any address, but it has to understand which addresses of the other connected joined servers it has to send to newly accepted client. Thus we '​join'​ multiple addresses on every node into '​logical route tables'​ which are indexed by the last number in addr config option. Thus, format becomes: local_address:​port:​family-route_group.\\ Addresses in the same route group on different servers will be put into the same route tables, thus when client in example below connects to localhost, it will receive (and connect to) addresses from the logical route group 0 (whatever addresses will be put into that route group on other servers).\\ Let's suppose we have 3 connected servers with the following addresses:​\\ srv0: [ "​10.0.0.17:​1025:​2-0",​ "​192.168.0.17:​1025:​2-1",​ "​20.20.20.17:​1025:​2-2"​ ]\\ srv1: [ "​10.0.0.34:​1025:​2-0",​ "​192.168.0.34:​1025:​2-1",​ "​20.20.20.34:​1025:​2-2"​]\\ srv2: [ "​15.14.13.12:​1025:​2-0",​ "​99.99.99.99:​1025:​2-1",​ "​111.111.111.111:​1025:​2-2"​ ]\\ When client connects to srv1 to IP address 192.168.0.34:​1025,​ it will receive (and connect to) following route table:\\ [ "​192.168.0.17:​1025",​ "​192.168.0.34:​1025",​ "​99.99.99.99:​1025"​ ]\\ Because above addresses are in the same logical route group and client connected to one of the addresses in that logical route group.\\ `addr` is a list of interfaces that server binds | | **address** = [ "​localhost:​1025:​2-0",​ "​10.10.0.1:​1025:​2-1"​ ] | List of node address which enables multiple interface support.\\ Elliptics server listens on port specified in this config option (format: addr:​port:​family-route_group)\\ If ports differ, the last one will be used.\\ Elliptics server will accept connections from any address, but it has to understand which addresses of the other connected joined servers it has to send to newly accepted client. Thus we '​join'​ multiple addresses on every node into '​logical route tables'​ which are indexed by the last number in addr config option. Thus, format becomes: local_address:​port:​family-route_group.\\ Addresses in the same route group on different servers will be put into the same route tables, thus when client in example below connects to localhost, it will receive (and connect to) addresses from the logical route group 0 (whatever addresses will be put into that route group on other servers).\\ Let's suppose we have 3 connected servers with the following addresses:​\\ srv0: [ "​10.0.0.17:​1025:​2-0",​ "​192.168.0.17:​1025:​2-1",​ "​20.20.20.17:​1025:​2-2"​ ]\\ srv1: [ "​10.0.0.34:​1025:​2-0",​ "​192.168.0.34:​1025:​2-1",​ "​20.20.20.34:​1025:​2-2"​]\\ srv2: [ "​15.14.13.12:​1025:​2-0",​ "​99.99.99.99:​1025:​2-1",​ "​111.111.111.111:​1025:​2-2"​ ]\\ When client connects to srv1 to IP address 192.168.0.34:​1025,​ it will receive (and connect to) following route table:\\ [ "​192.168.0.17:​1025",​ "​192.168.0.34:​1025",​ "​99.99.99.99:​1025"​ ]\\ Because above addresses are in the same logical route group and client connected to one of the addresses in that logical route group.\\ `addr` is a list of interfaces that server binds |
 | **wait_timeout** = 60 | Specifies number of seconds to wait for command completion | | **wait_timeout** = 60 | Specifies number of seconds to wait for command completion |
Line 54: Line 54:
 Elliptics can operate with several backends simultaneously. For that purpose '​backends'​ section of configuration file is an array of backends'​ configurations. Elliptics can operate with several backends simultaneously. For that purpose '​backends'​ section of configuration file is an array of backends'​ configurations.
  
-Each backend has parameters different from parameters of another backend, but some of them are equal for all of backends. At the moment Elliptics supports ​two backends: ​'​filesystem' ​and '​eblob'​. Config section for each are described below.+Each backend has parameters different from parameters of another backend, but some of them are equal for all of backends. At the moment Elliptics supports 'eblob' backend, there were number of others including ​filesystem, smack, tokyo cabinet ​and dynamic, but they were dropped due to performance reasonds. Config section for each are described below.
  
 === Generic backend options === === Generic backend options ===
Line 61: Line 61:
 | **enable** = true | Specifies if backend should be initialized at start, default value is true | | **enable** = true | Specifies if backend should be initialized at start, default value is true |
 | **type** = "​blob"​ | Specifies the type of backend | | **type** = "​blob"​ | Specifies the type of backend |
-| **enable** = true | Specifies if backend should be enabled at start | 
 | **group** = 1 | Backend will store keys for this group.\\ Group is a synonim for replica or copy. If multiple servers with the same group ID join elliptics network, group will contain multiple nodes and load will be spread to those servers. If multiple servers with different group IDs join elliptics network, there will be multiple groups and thus client will be able to write multiple copies. | | **group** = 1 | Backend will store keys for this group.\\ Group is a synonim for replica or copy. If multiple servers with the same group ID join elliptics network, group will contain multiple nodes and load will be spread to those servers. If multiple servers with different group IDs join elliptics network, there will be multiple groups and thus client will be able to write multiple copies. |
 | **history** = "/​path/​to/​history"​ | Specifies history environment directory.\\ It will host file with generated IDs for this backend. The history directory should be created manually before use. | | **history** = "/​path/​to/​history"​ | Specifies history environment directory.\\ It will host file with generated IDs for this backend. The history directory should be created manually before use. |
- 
-=== Filesystem backend section === 
- 
-Backend'​s type is **"​filesystem"​**. 
- 
-^ Property ​ ^ Description ^ 
-| **directory_bit_number** = 8 | Sets number of bits (from the beginning of the object ID) used for directory, which hosts given object. This option **must** be set in order for backend to work properly. | 
-| **root** = /​path/​to/​dir | Specifies root directory for data objects.\\ Should be created manually before use. If **root** option is not specified, current directory is used to store data and metadata. | 
-| **sync** = 0 | Sets number of seconds for the outcome of which metadata will be synced.\\ Zero here means 'sync on every write'​.\\ Positive number means file writes are never synced and metadata is synced every `sync` seconds. | 
- 
-Eblob is used to store object metadata (timestamp, user flags and so on), **blob_size**,​ **defrag_timeout** and **defrag_percentage** are described in [[#​eblob_backend_section|Eblob]] backend section. 
  
 === Eblob backend section === === Eblob backend section ===
Line 81: Line 69:
  
 ^ Property ​ ^ Description ^ ^ Property ​ ^ Description ^
-| **sync** = 0 | Sets number of seconds for the outcome of which metadata will be synced.\\ Zero here means 'sync on every write'​.\\ Positive number means file writes are never synced and metadata is synced every `sync` seconds. |+| **sync** = 0 | Sets number of seconds for the outcome of which metadata will be synced.\\ Zero here means 'sync on every write'​.\\ Positive number means file writes are never synced and metadata is synced every `sync` seconds. It is highly recommended not to specify this timeout at all (it is -1 by default), since it may heavily affect performance. Using default value relies on OS and kernel to sync data to disk. |
 | **data** = "/​path/​to/​blob/​data"​ | Specifies eblob objects prefix.\\ System will append .NNN and .NNN.index to new blobs. Path to blobs should be created manually before use.\\ For example, if prefix is `/​tmp/​blob/​data`,​ path `/tmp/blob` should be created. | | **data** = "/​path/​to/​blob/​data"​ | Specifies eblob objects prefix.\\ System will append .NNN and .NNN.index to new blobs. Path to blobs should be created manually before use.\\ For example, if prefix is `/​tmp/​blob/​data`,​ path `/tmp/blob` should be created. |
-| **blob_flags** = 1 | Specifies blob processing flags bitset (bits start from 0).\\ **bit 0** - if set, eblob reserves 10% of total space or size of the blob (which is bigger). By default it is turned off and eblob only reserves size of the blob. This is useful (required) to be able to run defragmentation.\\ **bit 1** - deprecated overwrite commits mode. Starting from eblob 0.20.0 it's default behaviour.\\ **bit 2** - deprecated overwrite mode. Starting from eblob 0.20.0 it's default behaviour.\\ **bit 3** - do not append checksum footer - this saves 72 bytes per written record. This also disables checksum.\\ **bit 4** - do not check whether system has enough space for the new blob.\\ **bit 5** - reserved for internal use, do not set.\\ **bit 6** - use second hashing layer - reduces memory usage for in-memory eblob index (costs some IOPS).\\ **bit 7** - auto data-sort. When set, eblob will perform data sorting as well as defragmentation and index sorting on startup and on blob close. This is prefered "set and forget"​ behaviour for small databases.\\ **bit 8** - timed data-sort. When set eblob will run data-sort on every non-sorted blob each `defrag_timeout` seconds. This is legacy behaviour for compatibility with old setups.\\ **bit 9** - scheduled data-sort. When set eblob will run data sort starting at `defrag_time` +/- `defrag_splay` hours. Probably one hould set this values to match clusters "​mostly-idle"​ hours. This option was introduced to "​spread"​ defragmentation load across nodes in cluster in time.\\ \\ This is prefered "set and forget"​ behaviour for not-so-big clusters. For very big clusters it's recommended to disable all `auto-data-sort` features and manually run '​dnet_ioclient ... -d' command with external synchronization. Also data-sort will try to defragment even already sorted blobs if they'​ve reached `defrag_percentage` fragmentation or they are way too small so there is good probability they will be merged into one. |+| **blob_flags** = 1 | Specifies blob processing flags bitset (bits start from 0).\\ **bit 0** - if set, eblob reserves 10% of total space or size of the blob (which is bigger). By default it is turned off and eblob only reserves size of the blob. This is useful (required) to be able to run defragmentation.\\ **bit 1** - deprecated overwrite commits mode. Starting from eblob 0.20.0 it's default behaviour.\\ **bit 2** - deprecated overwrite mode. Starting from eblob 0.20.0 it's default behaviour.\\ **bit 3** - do not append checksum footer - this saves 72 bytes per written record. This also disables checksum.\\ **bit 4** - do not check whether system has enough space for the new blob. If this bit is not set following checks are performed for every write call:\\ If **blob_size_limit** is specified in config and is not zero, total eblob size (sum of all blobs and indexes) plus size of the record to be written must be smaller than **blob_size_limit**\\ If **blob_size_limit** is not set (or is zero) eblob write checks available free space: if bit **0** is **not** set, then write call checks if available free space is large enough to hold 2 blobs (2 * **blob_size** bytes are reserved for eblob data sorting). If bit **0** is set at least 10% of the underlying storage must  empty.\\ **bit 5** - reserved for internal use, do not set.\\ **bit 6** - use second hashing layer - reduces memory usage for in-memory eblob index (costs some IOPS).\\ **bit 7** - auto data-sort. When set, eblob will perform data sorting as well as defragmentation and index sorting on startup and on blob close. This is prefered "set and forget"​ behaviour for small databases.\\ **bit 8** - timed data-sort. When set eblob will run data-sort on every non-sorted blob each `defrag_timeout` seconds. This is legacy behaviour for compatibility with old setups.\\ **bit 9** - scheduled data-sort. When set eblob will run data sort starting at `defrag_time` +/- `defrag_splay` hours. Probably one hould set this values to match clusters "​mostly-idle"​ hours. This option was introduced to "​spread"​ defragmentation load across nodes in cluster in time. \\ **bit 10** - disables starting permanent threads (sync, defrag, periodic). \\ **bit 11** - enables automatic index-only-sort which will kick-in on base's "​close"​. \\ \\ This is prefered "set and forget"​ behaviour for not-so-big clusters. For very big clusters it's recommended to disable all `auto-data-sort` features and manually run '​dnet_ioclient ... -d' command with external synchronization. Also data-sort will try to defragment even already sorted blobs if they'​ve reached `defrag_percentage` fragmentation or they are way too small so there is good probability they will be merged into one. |
 | **iterate_thread_num** = 4 | Sets number of threads used to populate data into RAM at startup.\\ This greatly speeds up data-sort/​defragmentation and somehow speeds up startup.\\ Also this threads are used for iterating by start_iterator request.\\ **Default: 1** | | **iterate_thread_num** = 4 | Sets number of threads used to populate data into RAM at startup.\\ This greatly speeds up data-sort/​defragmentation and somehow speeds up startup.\\ Also this threads are used for iterating by start_iterator request.\\ **Default: 1** |
 | **blob_size** = "​10G"​ | Specifies maximum blob size.\\ New file will be opened after current one grows beyond `blob_size` limit.\\ Supports K, M and G modifiers. | | **blob_size** = "​10G"​ | Specifies maximum blob size.\\ New file will be opened after current one grows beyond `blob_size` limit.\\ Supports K, M and G modifiers. |
Line 93: Line 81:
 | **index_block_size** = 40\\ **index_block_bloom_length** = "128 * 40" | Specifies bloom filter parameters:​\\ **index_block_size** - number of records from index file, which are hashed into one bloom filter. Eblob splits all records from sorted index file into chunks, each chunk has start and finish keys only and bloom filter which says whether requested entry can be found in given chunk.\\ **index_block_bloom_length** - number of bits per chunk, it should be at least as twice as number of records in chunk.\\ **Default values:\\ index_block_size = 40\\ index_block_bloom_length = 128 * 40** | | **index_block_size** = 40\\ **index_block_bloom_length** = "128 * 40" | Specifies bloom filter parameters:​\\ **index_block_size** - number of records from index file, which are hashed into one bloom filter. Eblob splits all records from sorted index file into chunks, each chunk has start and finish keys only and bloom filter which says whether requested entry can be found in given chunk.\\ **index_block_bloom_length** - number of bits per chunk, it should be at least as twice as number of records in chunk.\\ **Default values:\\ index_block_size = 40\\ index_block_bloom_length = 128 * 40** |
 | **periodic_timeout** | Specifies timeout for periodic process to start.\\ Periodic process writes eblob statistics to '​data.stat'​ file and caches json statistics.\\ This parameter allows to specify how often these statistics should be updated. Any time consumer of these statistics has an access to '​data.stat'​ file and cached json statistics generated by the last run of periodic process.| | **periodic_timeout** | Specifies timeout for periodic process to start.\\ Periodic process writes eblob statistics to '​data.stat'​ file and caches json statistics.\\ This parameter allows to specify how often these statistics should be updated. Any time consumer of these statistics has an access to '​data.stat'​ file and cached json statistics generated by the last run of periodic process.|
- 
 ==== Example of server node configuration file ==== ==== Example of server node configuration file ====
  
Line 112: Line 99:
                         "​type":​ "​files",​                         "​type":​ "​files",​
                         "​path":​ "/​dev/​stdout",​                         "​path":​ "/​dev/​stdout",​
-                        "​autoflush":​ true+                        "​autoflush":​ true
 +                        "​rotation":​ { 
 +                            "​move":​ 0 
 +                        }
                     }                     }
                 }                 }
Line 154: Line 144:
  "​backend_id":​ 1,  "​backend_id":​ 1,
  "​cache":​ {  "​cache":​ {
- "​size": ​0+ "​size": ​1073741824
  },  },
  "​type":​ "​blob",​  "​type":​ "​blob",​
Line 161: Line 151:
  "​data":​ "/​opt/​elliptics/​eblob.1/​data",​  "​data":​ "/​opt/​elliptics/​eblob.1/​data",​
  "​sync":​ "​-1",​  "​sync":​ "​-1",​
- "​blob_flags":​ "158",+ "​blob_flags":​ "0",
  "​blob_size":​ "​10G",​  "​blob_size":​ "​10G",​
  "​records_in_blob":​ "​1000000",​  "​records_in_blob":​ "​1000000",​
- "​periodic_timeout":​ 15 + "​periodic_timeout":​ 15, 
- }+ "​defrag_percentage":​ 10, 
- {+ "​defrag_timeout":​ 3600 
 + },{
  "​backend_id":​ 2,  "​backend_id":​ 2,
- "​type":​ "filesystem",+ "​cache":​ { 
 + "​size":​ 1073741824 
 + }, 
 + "​type":​ "blob",
  "​group":​ 2,  "​group":​ 2,
  "​history":​ "/​opt/​elliptics/​history.2",​  "​history":​ "/​opt/​elliptics/​history.2",​
- "root": "/​opt/​elliptics/​filesystem", + "data": "/​opt/​elliptics/​eblob.2/​data", 
- "​sync":​ "​-1"​+ "​sync":​ "​-1"​
 + "​blob_flags":​ "​0",​ 
 + "​blob_size":​ "​10G",​ 
 + "​records_in_blob":​ "​1000000",​ 
 + "​periodic_timeout":​ 15, 
 + "​defrag_percentage":​ 10, 
 + "​defrag_timeout":​ 3600
  }  }
  ]  ]
Line 186: Line 186:
 | **log_level** = 4 | Sets verbosity level of elliptics logs.\\ It doesn'​t corresponds to syslog levels. When logging to syslog almost all messages will be written with INFO level. Elliptics supports follow levels:\\- DNET_LOG_DATA = 0\\- DNET_LOG_ERROR = 1\\- DNET_LOG_INFO= 2\\- DNET_LOG_NOTICE = 3\\- DNET_LOG_DEBUG = 4| | **log_level** = 4 | Sets verbosity level of elliptics logs.\\ It doesn'​t corresponds to syslog levels. When logging to syslog almost all messages will be written with INFO level. Elliptics supports follow levels:\\- DNET_LOG_DATA = 0\\- DNET_LOG_ERROR = 1\\- DNET_LOG_INFO= 2\\- DNET_LOG_NOTICE = 3\\- DNET_LOG_DEBUG = 4|
 | **join** = 0 | Specifies whether to join storage network.\\ Client nodes set it to 0. Server nodes should use 1. | | **join** = 0 | Specifies whether to join storage network.\\ Client nodes set it to 0. Server nodes should use 1. |
-| **remote** = 1.2.3.4:​1025:​2 2.3.4.5:​2345:2 autodiscovery:​224.0.0.5:​1025:2 | List of remote nodes to connect.\\ Address should have follow format: `address:​port:​family` where family is either 2 (AF_INET) or 10 (AF_INET6) and address can be host name or IP of remote node.\\ Multicast doesn'​t used this time.\\ It is possible to autodiscover remote clusters via multicast. If you put `autodiscovery:​address:​port:​family` where `address:​port:​family` is valid multicast address, elliptics will broadcast information about itself and remote nodes with the same auth cookie will receive this information and connect to given node. Multicast TTL equals to 3.|+| **remote** = 1.2.3.4:​1025:​2 2.3.4.5:​2345:​2 | List of remote nodes to connect.\\ Address should have follow format: `address:​port:​family` where family is either 2 (AF_INET) or 10 (AF_INET6) and address can be host name or IP of remote node.|
 | **wait_timeout** = 60 | Specifies number of seconds to wait for command completion | | **wait_timeout** = 60 | Specifies number of seconds to wait for command completion |
 | **check_timeout** = 60 | Specifies number of seconds to wait before killing unacked transaction. This parameter is used also for some service operations such as update the routing server, check network connections an so on. | | **check_timeout** = 60 | Specifies number of seconds to wait before killing unacked transaction. This parameter is used also for some service operations such as update the routing server, check network connections an so on. |
elliptics/configuration.1411573676.txt.gz · Last modified: 2014/09/24 19:47 by zbr