Reverbrain wiki

Site Tools


elliptics:serverside-tutorial

Server-side scripting tutorial

This tutorial covers Elliptics server-side scripting issues. We will configure Cocaine engine and run a Python application under it. If Elliptics isn't installed yet, check the server setup tutorial.

To proceed with this tutorial please check that you have installed following packages:

  • cocaine-framework-python
  • cocaine-tools

If you don't now how to do this read the server setup tutorial.

Setting up server-side

Elliptics

Take ioserv.conf from server setup tutorial and uncomment the following:

srw_config = library_config.json

Also, to avoid problems, you need to change the next values if ioserv.conf:

io_thread_num = 4
nonblocking_io_thread_num = 4

Cocaine

Full description of Cocaine setting up you may find in the project documentation.

Here is a test example of cocaine config file:

cocaine.conf
{
    "version": 2,
    "paths": {
        "plugins": "/usr/lib64/cocaine",
	"runtime": "/var/tmp/cocaine/ipc",
        "spool": "/var/tmp/cocaine/spool"
    },
    "services": {
        "logging": {
            "type": "logging",
            "args": {
		"port": 12501
            }
        },
	"storage": {
	    "type": "storage"
	}
    },
    "storages": {
        "core": {
            "type": "elliptics",
            "args": {
                "nodes": [
                    "localhost:1026:2"
                ],
                "groups": [2],
                "verbosity": 2
            }
        }
    },
    "loggers": {
        "core": {
            "type": "files",
            "args": {
		"path": "/tmp/cocaine.log",
		"verbosity": "error"
            }
        },
        "slave": {
            "type": "remote",
            "args": {
	        "verbosity": "error"
	    }
        },
	"tools": {
		"type": "stdout"
	}
    },
    "port-mapper": {
        "range": [5100, 5200]
    }
}

Please notice, there is no cache section in storages - otherwise you will have gray hair in the process of experimentation with server-side scripting, application will be cached in cocaine directories, you will have to remove it every time you want to run updated app.

All directories listed in configuration file should exist and be accessible by user running dnet_ioserv.

mkdir -p /var/tmp/cocaine/ipc
mkdir /var/tmp/cocaine/spool
mkdir /var/tmp/cocaine/cache
mkdir /var/run/cocaine

Starting dnet_ioserv

After running dnet_ioserv -c ioserv.conf you should find following line in the log (if log level is 2 or higher):

2013-05-30 19:12:25.964511 2002/2002 2: srw: initialized: config: library_config.json

If some of above paths do not exist or user has no write permissions, you will find following error:

2012-09-11 02:33:02.468394 24684/24684 1: srw: init failed: config: library_config.json, exception: the configuration file doesn't exist
2012-09-11 02:33:02.468416 24684/24684 1: srw: initialization failure: Cannot allocate memory -12

Please double check that specified paths are correct, and if you are 110% sure that they are ok, drop us a mail to zbr@ioremap.net with full log and all config files as well as package versions (dpkg -l | egrep -v “elliptics|cocaine”).

Creating application

Cocaine engine executes user code, it supports any programming language but Python and C++ are the most preferable because of native support for them. How to create an application you may find in Cocaine documentation. In short, application is a simple noncompressed tarball which is uploaded into storage with associated configuration file, which describes what it is and how it should be executed.

Application

Let's create a Python application for Elliptics The application will consist of two files:

  • srw_app.py
  • srw_app.manifest

The srw_app.py file should be executable.

chmod u+x swr_app.py

Manifest has a JSON format. File names may be any other.

srw_app.py
#! /usr/bin/env python
 
from cocaine.worker import Worker
import elliptics
import socket
import struct
 
elog = elliptics.Logger("/dev/stderr", elliptics.log_level.info)
node = elliptics.Node(elog)
node.add_remotes(["localhost:1025:2"])
 
W = Worker()
 
def srw_example_function(request, response):
    data = yield request.read()
    s = elliptics.Session(node)
    s.set_groups([2])
    event_size = struct.unpack('<i', data[80:84])[0]
    s.write_data("TESTKEY", data[128 + event_size:], 0)
    response.close()
 
W.run({"srw_example_function" : srw_example_function})

Our application gets data on input and writes them into Elliptics group 2 with the TESTKEY identificator. After we start and execute this application we will read saved data with the dnet_ioclient command.

srw_example_function will be executed as a separate event.

srw_app.manifest
{
    "slave" : "srw_app.py"
}

Now we should upload cocaine profile, create tarball, upload it to the server and start the application.

cocaine.profile
{
  "isolate": {
    "type": "process",
    "args": {
	"spool": "/var/tmp/cocaine/spool"
    }
  }
}
install_app.sh
tar -cf srw_app.tar srw_app.py
cocaine-tool app upload --manifest srw_app.manifest --package srw_app.tar --name srw_app
cocaine-tool profile upload --profile cocaine.profile --name srw_app
dnet_ioclient -r localhost:1025:2 -g 2 -c "srw_app@start-task"

cocaine-tool lives in cocaine-tools repository.

Let's use our application with the simple Python script

client_app.py
#!/usr/bin/env python
 
from cocaine.services import Service
app = Service("srw_app")
for i in app.enqueue("srw_app@srw_example_function", "HELLO\n"):
    print i

With this code we send HELLO\n to application that writes it into the Elliptics storage under the id TESTKEY. Let's get this data with the dnet_ioclient:

dnet_ioclient -r localhost:1025:2-0 -g 2 -D TESTKEY
elliptics/serverside-tutorial.txt · Last modified: 2015/03/24 04:16 by zbr