Delete all Redis keys

2011-11-01

redis-cli KEYS "*" | xargs redis-cli DEL

Recursively add empty directories to git

2011-10-03

for i in $(find . -type d -regex ``./[^.].*'' -empty); do touch $i"/.gitignore"; done;

Flash policy server using Node.js

2011-10-02

The following code serves the cross domain policy file, needed for flash socket connections.

flash_policy_server.js

var net = require("net"),
    domains = ["*:*"]; // Domain:Port

net.createServer(
    function(socket) {
        socket.write("<?xml version='1.0' ?>\n");
        socket.write("<!DOCTYPE cross-domain-policy SYSTEM 'http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd'>\n");
        socket.write("<cross-domain-policy>\n");
        domains.forEach(
            function(domain) {
                var parts = domain.split(':');
                socket.write("\t<allow-access-from domain='" + parts[0] + "' to-ports='" + parts[1] + "' />\n");
            }
        );
        socket.write("</cross-domain-policy>\n");
        socket.end();
    }
).listen(843);

Since 843 is a reserved port, you have to run the server as root.

sudo node flash_policy_server.js

Exploring Redis

2011-09-24

1. Introduction

1.1. What is Redis?

REmote DIctionary Server(Redis) is a key-value datastore created by Salvatore Sanfilippo. Redis can store rich set of datatypes including lists, sets, ordered sets and hashes in addition to strings. Redis provides many convenience methods that operate on these datatypes.

1.2. Advantages

Blazing fast – Redis can support 100K+ read/write operations per sec.
Rich Datatypes – Redis can store binary safe Strings, Lists, Hashes, Sets and Ordered Sets.
Atomicity – All operations in Redis are Atomic. Redis can run operations under transaction.
Mulitool – Redis supports publish/subscribe, notifications, key expiry etc in addition to data storage.

2. Data Structures

2.1. Strings

Redis can store binary safe strings of size limit upto 1GB.

redis 127.0.0.1:6379> SET name "John Doe"
OK
redis 127.0.0.1:6379> GET name
"John Doe"

Strings can be stored and retrived in batches.

redis 127.0.0.1:6379> MSET age 30 sex "male"
OK
redis 127.0.0.1:6379> MGET age sex
1) "30"
2) "male"

Strings can be treated as atomic counter.

redis 127.0.0.1:6379> INCR age
(integer) 31
redis 127.0.0.1:6379> INCRBY age 4
(integer) 35
redis 127.0.0.1:6379> GET age
"35"
redis 127.0.0.1:6379> DECR age
(integer) 34
redis 127.0.0.1:6379> DECRBY age 4
(integer) 30
redis 127.0.0.1:6379> GET age
"30"

Strings are mutable and can be modified in place.

redis 127.0.0.1:6379> APPEND name " Mr."
(integer) 12
redis 127.0.0.1:6379> GET name
"John Doe Mr."
redis 127.0.0.1:6379> STRLEN name
(integer) 12
redis 127.0.0.1:6379> SUBSTR name 0 3
"John"

2.2. Lists

Redis can store ordered collection of strings in a list.

redis 127.0.0.1:6379> LPUSH students "John Doe"
(integer) 1
redis 127.0.0.1:6379> LPUSH students "Captain Kirk"
(integer) 2
redis 127.0.0.1:6379> LPUSH students "Sheldon Cooper"
(integer) 3
redis 127.0.0.1:6379> LLEN students
(integer) 3
redis 127.0.0.1:6379> LRANGE students 0 2
1) "Sheldon Cooper"
2) "Captain Kirk"
3) "John Doe"
redis 127.0.0.1:6379> LPOP students
"Sheldon Cooper"
redis 127.0.0.1:6379> LLEN students
(integer) 2
redis 127.0.0.1:6379> LRANGE students 0 1
1) "Captain Kirk"
2) "John Doe"
redis 127.0.0.1:6379> LREM students 1 "John Doe"
(integer) 1
redis 127.0.0.1:6379> LLEN students
(integer) 1
redis 127.0.0.1:6379> LRANGE students 0 0
1) "Captain Kirk"

Lists are mutable and can be modified in place.

redis 127.0.0.1:6379> LINSERT students BEFORE "Captain Kirk" "Dexter Morgan"
(integer) 3
redis 127.0.0.1:6379> LRANGE students 0 2
1) "Dexter Morgan"
2) "Captain Kirk"
3) "John Doe"
redis 127.0.0.1:6379> LPUSH students "Peter Parker"
(integer) 4
redis 127.0.0.1:6379> LRANGE students 0 3
1) "Peter Parker"
2) "Dexter Morgan"
3) "Captain Kirk"
4) "John Doe"
redis 127.0.0.1:6379> LTRIM students 1 3
OK
redis 127.0.0.1:6379> LLEN students
(integer) 3
redis 127.0.0.1:6379> LRANGE students 0 2
1) "Dexter Morgan"
2) "Captain Kirk"
3) "John Doe"
redis 127.0.0.1:6379> LREM students 1 "John Doe"
(integer) 1
redis 127.0.0.1:6379> LLEN students
(integer) 1
redis 127.0.0.1:6379> LRANGE students 0 1
1) "Captain Kirk"

2.3. Sets

Redis can store unordered collection of non-duplicate strings in a set.

redis 127.0.0.1:6379> SADD birds crow
(integer) 1
redis 127.0.0.1:6379> SADD birds pigeon
(integer) 1
redis 127.0.0.1:6379> SADD birds bat
(integer) 1
redis 127.0.0.1:6379> SADD mammals dog
(integer) 1
redis 127.0.0.1:6379> SADD mammals cat
(integer) 1
redis 127.0.0.1:6379> SADD mammals bat
(integer) 1
redis 127.0.0.1:6379> SMEMBERS birds
1) "bat"
2) "crow"
3) "pigeon"
redis 127.0.0.1:6379> SMEMBERS mammals
1) "bat"
2) "cat"
3) "dog"

Sets are mutable and can be modified in place.

redis 127.0.0.1:6379> SREM mammals cat
(integer) 1
redis 127.0.0.1:6379> SMEMBERS mammals
1) "bat"
2) "dog"
redis 127.0.0.1:6379> SADD mammals human
(integer) 1
redis 127.0.0.1:6379> SMEMBERS mammals
1) "bat"
2) "human"
3) "dog"

Redis supports common Set related operations.

redis 127.0.0.1:6379> SINTER birds mammals
1) "bat"
redis 127.0.0.1:6379> SUNION birds mammals
1) "crow"
2) "bat"
3) "human"
4) "pigeon"
5) "dog"
redis 127.0.0.1:6379> SDIFF birds mammals
1) "crow"
2) "pigeon"

2.4. Ordered Sets

Ordered Sets are similar to Sets, but the members are sorted and stored according to an associated floating point score.

redis 127.0.0.1:6379> ZADD days 0 mon
(integer) 1
redis 127.0.0.1:6379> ZADD days 1 tue
(integer) 1
redis 127.0.0.1:6379> ZADD days 2 wed
(integer) 1
redis 127.0.0.1:6379> ZADD days 3 thu
(integer) 1
redis 127.0.0.1:6379> ZADD days 4 fri
(integer) 1
redis 127.0.0.1:6379> ZADD days 5 sat
(integer) 1
redis 127.0.0.1:6379> ZADD days 6 sun
(integer) 1
redis 127.0.0.1:6379> ZCARD days
(integer) 7
redis 127.0.0.1:6379> ZRANGE days 0 6
1) "mon"
2) "tue"
3) "wed"
4) "thu"
5) "fri"
6) "sat"
7) "sun"
redis 127.0.0.1:6379> ZSCORE days sat
"5"
redis 127.0.0.1:6379> ZCOUNT days 3 6
(integer) 4
redis 127.0.0.1:6379> ZRANGEBYSCORE days 3 6
1) "thu"
2) "fri"
3) "sat"
4) "sun"

2.5. Hashes

Redis can store key-value pairs in Hashes.

redis 127.0.0.1:6379> HKEYS student
1) "name"
2) "age"
3) "sex"
redis 127.0.0.1:6379> HVALS student
1) "Ganesh"
2) "30"
3) "Male"
redis 127.0.0.1:6379> HGETALL student
1) "name"
2) "Ganesh"
3) "age"
4) "30"
5) "sex"
6) "Male"
redis 127.0.0.1:6379> HDEL student sex
(integer) 1
redis 127.0.0.1:6379> HGETALL student
1) "name"
2) "Ganesh"
3) "age"
4) "30"

Hashes can be stored and retrived in batches.

redis 127.0.0.1:6379> HMSET kid name Akshi age 2 sex Female
OK
redis 127.0.0.1:6379> HMGET kid name age sex
1) "Akshi"
2) "2"
3) "Female"

3. Publish/Subscribe

Redis supports Publish/Subscribe pattern. Redis publisher send messages to different channels. And Redis receivers listen to the channels and retrieve the messages of its interest.

3.1. Channel Subscription

In one redis client session

redis 127.0.0.1:6379> SUBSCRIBE channelone
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "channelone"
3) (integer) 1

In another redis client session

redis 127.0.0.1:6379> PUBLISH channelone hello
(integer) 1
redis 127.0.0.1:6379> PUBLISH channelone world
(integer) 1

The messages are received in the first redis client session.

redis 127.0.0.1:6379> SUBSCRIBE channelone
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "channelone"
3) (integer) 1
1) "message"
2) "channelone"
3) "hello"
1) "message"
2) "channelone"
3) "world"

3.2. Pattern Subscription

Redis also supports pattern based subscriptions, wherein the receivers listen to channels matching a particular pattern.

In one redis client session.

redis 127.0.0.1:6379> PSUBSCRIBE channel*
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "channel*"
3) (integer) 1

In another redis client session.

redis 127.0.0.1:6379> PUBLISH channelone hello
(integer) 1
redis 127.0.0.1:6379> PUBLISH channeltwo world
(integer) 1

The messages sent to both the channels are received in the first redis client session.

redis 127.0.0.1:6379> PSUBSCRIBE channel*
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "channel*"
3) (integer) 1
1) "pmessage"
2) "channel*"
3) "channelone"
4) "hello"
1) "pmessage"
2) "channel*"
3) "channeltwo"
4) "world"

4. Key Expiry

Redis supports volatile keys that expire after a specified TTL(Time To Live).

Keys with TTL as -1 never expire.

redis 127.0.0.1:6379> SET name "John Doe"
OK
redis 127.0.0.1:6379> TTL name
(integer) -1

redis 127.0.0.1:6379> SET name "John Doe"
OK
redis 127.0.0.1:6379> EXISTS name
(integer) 1
redis 127.0.0.1:6379> EXPIRE name 5
(integer) 1

After 5 secs

redis 127.0.0.1:6379> EXISTS name
(integer) 0
redis 127.0.0.1:6379> GET name
(nil)

Keys can be expired at specified unix timestamp. The following expires at 2011-09-24 00:40:00

redis 127.0.0.1:6379> SET name "John Doe"
OK
redis 127.0.0.1:6379> EXPIREAT name 1316805000
(integer) 1
redis 127.0.0.1:6379> EXISTS name
(integer) 0

5. Transactions

All Redis operations run under transactions. Redis supports multi step commands that run as a single atomic unit.

*NX commands modify the keys only when the keys do not already exists. Checking for the keys and setting the value happen under transaction and are unaffected by simultaneous clients trying to alter the keys.

redis 127.0.0.1:6379> SET name "John Doe"
OK
redis 127.0.0.1:6379> SETNX name "Dexter Morgan"
(integer) 0
redis 127.0.0.1:6379> GET name
"John Doe"

redis 127.0.0.1:6379> GETSET name "Dexter Morgan"
"John Doe"
redis 127.0.0.1:6379> GET name
"Dexter Morgan"

Redis can also run multiple operations under transactions.

redis 127.0.0.1:6379> SET counter 0
OK
redis 127.0.0.1:6379> MULTI
OK
redis 127.0.0.1:6379> INCR counter
QUEUED
redis 127.0.0.1:6379> INCR counter
QUEUED
redis 127.0.0.1:6379> INCR counter
QUEUED
redis 127.0.0.1:6379> EXEC
1) (integer) 1
2) (integer) 2
3) (integer) 3
redis 127.0.0.1:6379> GET counter
"3"

Transactions can be explicitly aborted.

redis 127.0.0.1:6379> SET newcounter 0
OK
redis 127.0.0.1:6379> MULTI
OK
redis 127.0.0.1:6379> INCR newcounter
QUEUED
redis 127.0.0.1:6379> INCR newcounter
QUEUED
redis 127.0.0.1:6379> INCR newcounter
QUEUED
redis 127.0.0.1:6379> DISCARD
OK
redis 127.0.0.1:6379> GET newcounter
"0"

6. Persistance

Redis is in-memory datastore. But it supports configurable disk persistance.

6.1. Snapshotting

Snapshotting is the default persistance method of Redis. Redis stores the entire memory content in a file called dump.rdb at regular interval of time.

You can also explicity trigger snapshotting.

redis 127.0.0.1:6379> SET name "John Doe"
OK
redis 127.0.0.1:6379> SAVE
OK
redis 127.0.0.1:6379> SET name "Sheldon Cooper"
OK
redis 127.0.0.1:6379> BGSAVE
Background saving started

On redis installed through brew on Mac OSX, the dump.rdb is present at

/usr/local/var/db/redis/dump.rdb

6.2. Append-only file

Redis also supports fully-durable alternate to snapshotting. In the append-only mode all the commands altering the keys will be appended to a AOF(Append-Only File). The current state of Redis can be rebuilt by executing the commands in AOF.

To enable append-only file mode, change the following in redis.conf and restart the redis server.

appendonly yes

All the executed commands are appended to appendonly.aof file

redis 127.0.0.1:6379> GET name
(nil)
redis 127.0.0.1:6379> SET name "Ganesh Gunasegaran"
OK
redis 127.0.0.1:6379> EXIT

→ cat /usr/local/var/db/redis/appendonly.aof
*2
$6
SELECT
$1
0
*3
$3
SET
$4
name
$18
Ganesh Gunasegaran

7. Administration

Redis has 16 numbered databases. Data can be stored and moved among them.

redis 127.0.0.1:6379> SELECT 0
OK
redis 127.0.0.1:6379> SET name "John Doe"
OK
redis 127.0.0.1:6379> SELECT 1
OK
redis 127.0.0.1:6379[1]> GET name
(nil)
redis 127.0.0.1:6379[1]> SELECT 0
OK
redis 127.0.0.1:6379> MOVE name 1
(integer) 1
redis 127.0.0.1:6379> SELECT 1
OK
redis 127.0.0.1:6379[1]> GET name
"John Doe"

Redis supports various commands to query about the database.

redis 127.0.0.1:6379[1]> DBSIZE
(integer) 1
redis 127.0.0.1:6379[1]> INFO
redis_version:2.2.13
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:kqueue
......

Redis provides commands for clearing the database.

redis 127.0.0.1:6379> SET name "John Doe"
OK
redis 127.0.0.1:6379> DBSIZE
(integer) 1
redis 127.0.0.1:6379> SELECT 1
OK
redis 127.0.0.1:6379[1]> SET name "Sheldon Cooper"
OK
redis 127.0.0.1:6379[1]> DBSIZE
(integer) 1
redis 127.0.0.1:6379[1]> SELECT 0
OK
redis 127.0.0.1:6379> FLUSHDB
OK
redis 127.0.0.1:6379> DBSIZE
(integer) 0
redis 127.0.0.1:6379> SELECT 1
OK
redis 127.0.0.1:6379[1]> DBSIZE
(integer) 1
redis 127.0.0.1:6379[1]> FLUSHALL
OK
redis 127.0.0.1:6379[1]> DBSIZE
(integer) 0

8. Client Libraries

Redis has a wide range of client libraries.

8.1. Node.js Client

gg:~/Work/Node
→ npm install redis
redis@0.6.7 ./node_modules/redis

Simple storage and retrieval

var redis = require('redis');
var redisClient = redis.createClient();

redisClient.on('error', function(error) {
  console.log('Error: ' + error);
});

console.log('Storing Data');
redisClient.set('name', 'Ganesh Gunasegaran', redis.print);

console.log('Retreiving Data');
redisClient.get('name', redis.print);

redisClient.quit();

Output

gg:~/Work/Node
→ node redis_test.js
Storing Data
Retreiving Data
Reply: OK
Reply: Ganesh Gunasegaran

Using Node.js client to access Redis Publish/Subscribe

pubsub_client_one.js


var redis = require('redis');
var clientOne = redis.createClient();

clientOne.on('message', function(channel, message) {
  console.log('clientOne got : ' + message);
});

clientOne.subscribe('channelOne');