This tutorial explains how to implement simple moving window (rolling counter) in NodeJS and Redis. We will use sorted sets to count every request and then will show another approach.
We will count number of hits made in 24 hours. You can easily modify it to show weekly and monthly statistics.

At first install Redis and Redis-scripto modules as described in previous article.

Using sorted sets

Create demo.js. It runs simple-counter.lua and prints result which adds one hit to zset and returns number of hits made in last 24 hours as of running.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
var path = require('path');
redis = require('redis'),
scripto = require('redis-scripto'),
ts = Math.floor(Date.now() / 1000), // current time
redisClient = redis.createClient(), // creates a new redis client
scriptDir = path.resolve(path.dirname(__filename), 'lua'), // path to lua scripts
scriptManager = new scripto(redisClient); // creates new scripto manager
scriptManager.loadFromDir(scriptDir); // load lua scripts from directory
console.log(ts);
scriptManager.run('simple-counter', [], [ts], function(err, result) { //
console.log(result);
process.exit();
});

Create lua/simple-counter.lua

1
2
3
4
5
local ts = tonumber(ARGV[1])
redis.call('zremrangebyscore', 'stat:hits', 0, ts - 86400) -- remove hits older than 24 hours
redis.call('zadd', 'stat:hits', ts, ts) -- add 1 hit, then count and return
return redis.call('zcount', 'stat:hits', '-inf', '+inf')

It just adds one timestamp to the stat:hits sorted set, removes old timestamps and returns total count.

You can notice that score and value are both a timestamp

1
redis.call('zadd', 'stat:hits', ts, ts)

That leads to incorrect results in case of two requests made simultaneously. Indeed, timestamps will be the same and only one result will be stored in sorted set. To fix it, we need to use some unique value.

For simplicity, we can use some Redis key as a counter and just increment it with every request. Modify simple-counter.lua:

1
2
3
4
5
6
local ts = tonumber(ARGV[1])
local i = tonumber(redis.call('incr', 'stat:counter')) -- add one hit to the counter
redis.call('zremrangebyscore', 'stat:hits', 0, ts - 86400) -- remove hits older than 24 hours
redis.call('zadd', 'stat:hits', ts, i) -- add 1 hit, then count and return
return redis.call('zcount', 'stat:hits', '-inf', '+inf')

Here we used stat:counter as a unique value.

What if we don’t need stats per second and are only interested in minutes? We can save some memory by doing this

1
2
ts = Math.floor(Date.now() / (1000 * 60)), // timestamp in minutes

Or if we need hours:

1
2
ts = Math.floor(Date.now() / (1000 * 3600)), // timestamp in hours

Modify simple-counter.lua accordingly.

Using per-hour counters

If your site has millions of hits, removing and counting sorted sets might become an expensive operation. If you don’t need each individual hit information (you may want to store user’s request url, ip address and other information), you can use just several integer keys and let Redis expire and remove old ones.

Since there are only 24 hours, we can create 24 keys and store hits for each hour of the day accordingly. But then we can’t make it moving, as we need all 24 keys to get stats and we can’t delete ‘old’ ones.

To fix it, let’s pretend that there are 25 hours a day (timestamp in hours modulo 25). We will write hits to each one (of 25) continuously, but use only 24 last ones to get total. With each write we ask Redis to expire current hour key in 86400 seconds (24h).

Since we have 25 hours and each key expires in 24 hours, the least written one (that was 25 hours ago) will be removed by Redis. We will start counting from zero when we get to it again.

Thus we have the moving window (rolling counter).

demo.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
var path = require('path');
redis = require('redis'),
scripto = require('redis-scripto'),
ts = Math.floor(Date.now() / (3600 * 1000)), // number of hours passed since Unix epoch
redisClient = redis.createClient(), // creates a new redis client
scriptDir = path.resolve(path.dirname(__filename), 'lua'), // path to lua scripts
scriptManager = new scripto(redisClient); // creates new scripto manager
scriptManager.loadFromDir(scriptDir); // load lua scripts from directory
console.log(ts % 25); // current hour in batch of 25
scriptManager.run('key-counter', [], [ts % 25, 24], function(err, result) { // args: [hours modulo 25, max hour - 24 (i.e. 25 hours starting from 0)]
console.log(result);
process.exit();
});

key-counter.lua

1
2
3
4
5
6
7
8
9
10
11
12
13
local hour = tonumber(ARGV[1])
local maxHours = tonumber(ARGV[2])
local current = tonumber(redis.call('incr', 'stat:hour:' .. hour))
redis.call('expire', 'stat:hour:' .. hour, '86400')
for i = 1, maxHours - 1 do
hour = hour - 1
if hour < 0 then
hour = maxHours
end
current = current + (tonumber(redis.call('get', 'stat:hour:' .. hour) or 0 ))
end
return current

Using per-hour counters with a flag

Actually my spouse proposed another way of doing the same thing which might be more intuitive. We can use 24 keys and a flag (another key) indicating current hour we are writing to. When current hour changes, set corresponding hour stat to zero. Redis is single threaded and we won’t get any race conditions, like setting a flag multiple times.

Since we don’t need to expire anything, we can use Redis hashes.

demo.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
var path = require('path');
redis = require('redis'),
scripto = require('redis-scripto'),
hour = (new Date).getHours(), // current hour 0 - 23
redisClient = redis.createClient(), // creates a new redis client
scriptDir = path.resolve(path.dirname(__filename), 'lua'), // path to lua scripts
scriptManager = new scripto(redisClient); // creates new scripto manager
scriptManager.loadFromDir(scriptDir); // load lua scripts from directory
console.log(hour);
scriptManager.run('key-counter2', [], [hour], function(err, result) {
console.log(result);
process.exit();
});

key-counter2.lua

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
local hour = tonumber(ARGV[1])
local current = tonumber(redis.call('get', 'stat:hour:current') or 0)
local total = 0
if hour ~= current then
redis.call('hset', 'stat:hours', hour, 1)
redis.call('set', 'stat:hour:current', hour)
else
redis.call('hincrby', 'stat:hours', hour, 1)
end
for i = 0, 23 do
total = total + (tonumber(redis.call('hget', 'stat:hours', i) or 0))
end
return total