README
Sloki
A NodeJS Server for LokiJS
:hammer: :hammer: :hammer: :hammer: :hammer: WORK IN PROGRESS :hammer: :hammer: :hammer: :hammer: :hammer:
Documentation
Introduction
Sloki is a nodejs server which embed LokiJS, a blazing fast in-memory documents database. Sloki help to make LokiJS scalable : you can now have multiple processes speaking with LokiJS through Sloki.
A possible architecture using sloki :
+----------------------------+ TCP / Binary +-----------------------------------+
| NodeJS app worker #1 |<------------------->| |
+----------------------------+ | Sloki |
| |
+----------------------------+ TCP / Dinary | +-------------------------+ |
| NodeJS app worker #2 |<------------------->| | | |
+----------------------------+ | | | |
| | LokiJS | |
+----------------------------+ TCP / JSONRPC | | fast in-memory database | |
| go/php/python/C/whatever |<------------------->| | | |
+----------------------------+ | | | |
| | | |
+----------------------------+ TCP / Binary | +-------------------------+ |
| sloki-cli |<------------------->| |
+----------------------------+ +-----------------------------------+
Goal
The goal is to have a complete in-memory database, supporting documents (mongodb like) AND key/value stores (like redis like).
For documents, lokiJS is used.
Key/value stores are not implemented at the moment.
Transports
For moment, only TCP transport is implemented. The advantage of TCP vs HTTP API is that the connection is persistent (i.e more fast). Websockets should be implemented before HTTP API.
By default, Sloki listens on the following ports:
| Port | Transport | TLS | Protocol |
|---|---|---|---|
| 6370 | TCP | NO | Binary |
| 6371 | TCP | YES | Binary |
| 6372 | TCP | NO | JSONRPC |
| 6373 | TCP | YES | JSONRPC |
| 6374 | TCP | NO | Dinary |
| 6375 | TCP | YES | Dinary (fastest) |
If somebody have an idea why TLS over TCP is fastest than raw TCP, i'd like to know .. :)
You will need a client to speak with sloki.
Protocols
Binary
The binary protocol has been made with performance in mind. Payloads looks like JSONRPC, but it's not.
REQUEST | RESPONSE
------------------------------------------- | --------------------------------------
{ | {
"m":"myMethod", | "r":true,
"p":["foo","bar"], | "id":"operation-uniq-id"
"id":"operation-uniq-id" | }
} |
- Payload is a little lighter compared to compliant JSONRPC protocol described below (i.e no
jsonrpcversion attribute,methodbecomem,paramsbecomep,resultbecomer) - Missive package is used both server and client side to transform payloads into a binary format. Missive support zlib compression, but it's not used here and it's not recommended because of performance crumble. Missive is based on fringe, an extensible message framing over streams for nodejs.
JSONRPC
The JSONRPC protocol has been chosen for interoperability.
REQUEST | RESPONSE
------------------------------------------- | --------------------------------------
{ | {
"jsonrpc":"2.0", | "jsonrpc":"2.0"
"method":"myMethod", | "result":true,
"params":["foo","bar"], | "id":"operation-uniq-id"
"id":"operation-uniq-id" | }
} |
- Raw and standard JSONRPC over TCP
- jayson package is used server side. Actually only TCP transport is implemented, but HTTP(s) JSON API and websocket API may be implemented in the future.
Dinary
It's not a typo. Dinary use 2 Binary clients, one socket for requests, the other one for responses. This is the fastest protocol, the one by default.
The underling protocol is the same as the Binary one.
Server
Installation
npm install -g sloki
Run
sloki
Help
sloki --help
=======================================================================
Sloki - a NodeJS Server for LokyJS
=======================================================================
Environment variable Default
SLOKI_TCP_BINARY_ENABLE true
SLOKI_TCP_BINARY_PORT 6370
SLOKI_TCP_BINARY_HOST localhost
SLOKI_TCP_BINARY_MAX_CLIENTS 64
SLOKI_TLS_BINARY_ENABLE true
SLOKI_TLS_BINARY_PORT 6371
SLOKI_TLS_BINARY_HOST localhost
SLOKI_TLS_BINARY_MAX_CLIENTS 64
SLOKI_TCP_JSONRPC_ENABLE true
SLOKI_TCP_JSONRPC_PORT 6372
SLOKI_TCP_JSONRPC_HOST localhost
SLOKI_TCP_JSONRPC_MAX_CLIENTS 64
SLOKI_TLS_JSONRPC_ENABLE true
SLOKI_TLS_JSONRPC_PORT 6373
SLOKI_TLS_JSONRPC_HOST localhost
SLOKI_TLS_JSONRPC_MAX_CLIENTS 64
SLOKI_TCP_DINARY_ENABLE true
SLOKI_TCP_DINARY_PORT 6374
SLOKI_TCP_DINARY_HOST localhost
SLOKI_TCP_DINARY_MAX_CLIENTS 64
SLOKI_TLS_DINARY_ENABLE true
SLOKI_TLS_DINARY_PORT 6375
SLOKI_TLS_DINARY_HOST localhost
SLOKI_TLS_DINARY_MAX_CLIENTS 64
SLOKI_DIR /home/franck/.sloki
SLOKI_SHOW_OPS_INTERVAL 0
SLOKI_GC_INTERVAL 3600000
SLOKI_MEM_LIMIT 26094 Mb
-----------------------------------------------------------------------
Command Line Options Default
--tcp-binary-enable true
--tcp-binary-port 6370
--tcp-binary-host localhost
--tcp-binary-max-clients 64
--tls-binary-enable true
--tls-binary-port 6371
--tls-binary-host localhost
--tls-binary-max-clients 64
--tcp-jsonrpc-enable true
--tcp-jsonrpc-port 6372
--tcp-jsonrpc-host localhost
--tcp-jsonrpc-max-clients 64
--tls-jsonrpc-enable true
--tls-jsonrpc-port 6373
--tls-jsonrpc-host localhost
--tls-jsonrpc-max-clients 64
--tcp-dinary-enable undefined
--tcp-dinary-port 6374
--tcp-dinary-host localhost
--tcp-dinary-max-clients 64
--tls-dinary-enable true
--tls-dinary-port 6375
--tls-dinary-host localhost
--tls-dinary-max-clients 64
--dir /home/franck/.sloki
--show-ops-interval 0
--gc-interval 3600000
--mem-limit 26094 Mb
-----------------------------------------------------------------------
Examples:
$ sloki # will use defaults
$ sloki --tcp-binary-port=6370 --tcp-binary-host=localhost
Clients
NodeJS
See https://github.com/sloki-project/sloki-node-client
The client will load every methods that sloki server have.
Benchmarks
> npm run bench
#################################################################################
# Benchmark suite using sloki v0.0.8 Intel® Core™ i7-6820HQ 2.70Ghz
#################################################################################
# x64 | 8 CPU(s) | linux (4.4.0-43-Microsoft Linux) | node v11.10.0
#################################################################################
> client connected (binary)
> client connected (binarys)
> client connected (jsonrpc)
> client connected (jsonrpcs)
> client connected (dinary)
> client connected (dinarys)
>>>>> test insert#nocallback
> run insert#nocallback@binary
> run insert#nocallback@binarys
> run insert#nocallback@jsonrpc
> run insert#nocallback@jsonrpcs
> run insert#nocallback@dinary
> run insert#nocallback@dinarys
>>>>> test insert#callback.fullDocument
> run insert#callback.fullDocument@binary
> run insert#callback.fullDocument@binarys
> run insert#callback.fullDocument@jsonrpc
> run insert#callback.fullDocument@jsonrpcs
> run insert#callback.fullDocument@dinary
> run insert#callback.fullDocument@dinarys
>>>>> test insert#callback.sret.01
> run insert#callback.sret.01@binary
> run insert#callback.sret.01@binarys
> run insert#callback.sret.01@jsonrpc
> run insert#callback.sret.01@jsonrpcs
> run insert#callback.sret.01@dinary
> run insert#callback.sret.01@dinarys
>>>>> gc done (rss before 206 MB, after 194 MB)
> client disconnected (binary)
> client disconnected (binarys)
> client disconnected (jsonrpc)
> client disconnected (jsonrpcs)
> client disconnected (dinary)
> client disconnected (dinarys)
# --------------------------------------------------------------------------------
# Test | Operations | ops/sec | exec time
# --------------------------------------------------------------------------------
# insert#nocallback@binary | 20000 | 12462 | 1.6s
# insert#nocallback@binarys | 20000 | 314297 | 64ms
# insert#nocallback@jsonrpc | 20000 | 19875 | 1s
# insert#nocallback@jsonrpcs | 20000 | 359086 | 56ms
# insert#nocallback@dinary | 20000 | 14190 | 1.4s
# insert#nocallback@dinarys | 20000 | 939276 | 22ms
# insert#callback.fullDocument@binary | 20000 | 11682 | 1.7s
# insert#callback.fullDocument@binarys | 20000 | 9542 | 2.1s
# insert#callback.fullDocument@jsonrpc | 20000 | 17443 | 1.1s
# insert#callback.fullDocument@jsonrpcs | 20000 | 17136 | 1.2s
# insert#callback.fullDocument@dinary | 20000 | 11089 | 1.8s
# insert#callback.fullDocument@dinarys | 20000 | 26836 | 746ms
# insert#callback.sret.01@binary | 20000 | 11939 | 1.7s
# insert#callback.sret.01@binarys | 20000 | 27308 | 733ms
# insert#callback.sret.01@jsonrpc | 20000 | 17117 | 1.2s
# insert#callback.sret.01@jsonrpcs | 20000 | 17031 | 1.2s
# insert#callback.sret.01@dinary | 20000 | 11184 | 1.8s
# insert#callback.sret.01@dinarys | 20000 | 26900 | 744ms
The winner is dinary protocol !
Development status
Not usable yet.