Reverbrain wiki

Site Tools


backrunner:uri

Backrunner HTTP URI handlers

Handler is AJAX-friendly URI prefix used to separate REST API calls. For example in http://example.com/get/b12/test.txt URL /get/ is a handler - it is the first name after slash / before the next slash. Path parts after handler's name, b12, test.txt in this example, can be treated by per-handler fashion.

All handlers accept following URI key-value parameters after ? sign, following parameters are supported:

trace_id all IO requests to elliptics will gold this 64-bit value and all logs (on client and server) will contain this hexadecimal trace, also X-Request header can be used for the same purpose.
ioflags elliptics IO flags to be set for given request: https://github.com/reverbrain/elliptics/blob/master/include/elliptics/packet.h#L413
cflags elliptics command flags to be set for given request: https://github.com/reverbrain/elliptics/blob/master/include/elliptics/packet.h#L143

All handlers accept following headers (some of them may be not applicable for particular handler):

X-Request all IO requests to elliptics will gold this 64-bit value and all logs (on client and server) will contain this hexadecimal trace
Range RFC 2616 Range header, it also can be used to specify offset for upload request (only offset is accepted, size part will be discarded, content length is used instead) and redirect request
If-Modified-Since client-side caching

ACL support

Backrunner allows to limit access to Elliptics storage using access control lists, where each user (including wildcard one) has an associated secure token and set of actions it is allowed to do (reading, writing, allowing to pass through without secure checks).

Client has to set Authorization header according to authorization signature rules, otherwise it will be considered a wildcard user and appropriate ACL rule will be applied (it might prevent wildcard user from accessing storage for example).

More details are discussed at ACL section at main backrunner page.

/nobucket_upload/

Format: POST /nobucket_upload/key

When data is uploaded using this handler backrunner proxy automatically selects bucket to put data into. This selection is based on the storage performance, network delays, amount of free space, number of errors and so on.

Bucket is logical entity used to specify how and where replicas for given request has to go. It is possible that the same key will go into different buckets in subsequent handler invocation, thus it has to be used for unique keys, i.e. when you are not trying to overwrite data (which is anyway a very bad idea in distributed storages).

Bucket has a symbolic name like b12 in the example above, it is returned in the following reply message from the proxy:

{
	bucket: "b99991",
	key: "key",
	reply: {
		info: [
		{
			group: 266799,
			backend: 1,
			id: "5955ce2d0ae8776202f994f3020148d3a96a87d7643a6fb70e24a92276875556d35bd42ddae08c06a04b08630e4922f0749f996ed75b8a7546f15e9f012b63eb",
			csum: "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
			filename: "/mnt/elliptics/data/data-0.5",
			size: 5517,
			offset-within-data-file: 5159652595,
			mtime: "2014-12-05 04:45:07.495999 +0300 MSK",
			server: "address1:2025:2",
			error: null
		},
		{
			group: 266899,
			backend: 1,
			id: "5955ce2d0ae8776202f994f3020148d3a96a87d7643a6fb70e24a92276875556d35bd42ddae08c06a04b08630e4922f0749f996ed75b8a7546f15e9f012b63eb",
			csum: "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
			filename: "/mnt/elliptics/data/data-0.5",
			size: 5517,
			offset-within-data-file: 5159652595,
			mtime: "2014-12-05 04:45:07.495999 +0300 MSK",
			server: "address2:2025:2",
			error: null
		}
		],
		success-groups: [
			266799,
			266899
		],
		error-groups: [ ]
	}
}

Following fields are returned:

bucket name of the bucket this data was written into
key name of the key client used to write data
reply section which corresponds to how data was stored in the elliptics

Reply section

success-groups array of the elliptics group which host given key
error-groups array of the elliptics group which could not write data
info array of the objects each of which corresponds to single written replica of the data

Info section

group group ID this key was written to
server elliptics storage node address which hosts given data
backend backend with this ID on the elliptics storage node hosts given data
id elliptics ID of the written key
filename full absolute path name where given data has been stored
size size of the written data
offset-within-data-file offset in the abovementioned storage file where given data has been stored
mtime modification time, i.e. time when this data hit the disk
error error message, non-null if proxy failed to write data into given group

Client has to store bucket name near its key name, since it is only possible to read data from the storage using bucket name, i.e. client has to tell the proxy which bucket has to be looked into to find out given key.

/upload/

Format: POST /upload/bucket/key
Where bucket is a name of the existing bucket.

This handler will write data into particular bucket (if it exists), and no automatic bucket selection will be performed. It is possible that elliptics nodes in given bucket are already fully filled and this write may fail.

If you want to overwrite data, and some groups which belong to specified bucket are not accessible, upload still returns success, and when later missing servers come online, replicas will not be consistent. Elliptics does not automatically runs recovery when node goes online, since it usually adds more load to the system which was likely the reason nodes went offline at the first place. This will only be fixed after elliptics recovery.

It is generally a very bad idea to update data in eventually consistent systems, consider switching to the case when you only upload new keys and never overwrite existing, in this case missing data will be automatically recovered on the first read.

You still can use so called read-latest feature to read the latest data among all potentially inconsistent replicas, which adds noticeable overhead, basically it reads metadata from all the replicas, selects the most recently updated data and reads it.

/get/

Format: GET /get/bucket/key
Where bucket is a name obtained when data was written.

This handler is used to read data from the storage. Data will be read in chunks from the storage if object's total size is more than 10Mb, after each chunk has been sent to client, the next one will be read from the storage. No chunked encoding supported.

/delete/

Format: POST /delete/bucket/key
Where bucket is a name obtained when data was written.

Used to delete given key from specified bucket

/bulk_delete/

Format: POST /bulk_delete/bucket
Where bucket is a name obtained when data was written.

Used to delete set of keys from specified bucket. Keys are specified in the following uploaded form:

{
        "keys": ["key.1", "key.2", "key.3"]
}

/lookup/

Format: GET /lookup/bucket/key
Where bucket is a name obtained when data was written.

Lookup key in given bucket. Reply json object from nobucket_upload example is returned.

This method is used to find where given key lives on low-level storage devices. It is possible to use this information to find data on the disk on given storage nodes. Client may want to stream data from its own application running on elliptics storage nodes, and using information from this handler, client may tell that application where required data was placed.

/redirect/

Format: GET /redirect/bucket/key
Where bucket is a name obtained when data was written.

Simialar to /loookup/ handler, which instead randomly selects one of the storage node and generates 302 redirect according to redirect proxy config. Storage node must run nginx eblob streaming module which will handle this request.

/ping/ and /stat/

Format: GET /ping/

These handlers return internal data statistics if HTTP proxy is operational.

/proxy_stat/

Format: GET /proxy_stat/

Returns json with RPS and BPS statistics for all supported handlers for given proxy.

/ common handler

Format: GET /static.file

If no handler prefix matched, then backrunner will return files from its local root directory, specified as root proxy config option. It is commonly used to put files like crossdomain.xml and the like.

backrunner/uri.txt ยท Last modified: 2017/02/11 09:25 by zbr