T••LBX: Blog

Dispatch Kemal websocket messages with Redis Pub/Sub

This tutorial is based on "Building a realtime Chat application with Crystal and Kemal" written by Serdar Doğruyol. Serdar is the author of Kemal, a small web framework written in Crystal.

We will go a little bit further and see how we can dispatch the websocket messages with Redis using Pub/Sub. This will allow us to scale across many server instances. There are other reasons for doing this but scaling via load balancing is one of them.

It is worth noting that I will use the most recent Crystal version, which currently is 0.30.0. You can find the whole code of this tutorial in this Github repository.

Let's start by creating a new Crystal application:

$ crystal init app kemal-redis-chat
$ cd kemal-redis-chat

The init should have put a target in your shard.yml, but if it didn't then add this:

targets:
  kemal-redis-chat:
    main: src/kemal-redis-chat.cr

Add the following dependencies to shard.yml:

dependencies:
  kemal:
    github: sdogruyol/kemal
    branch: master
  redis:
    github: stefanwille/crystal-redis
    branch: master
  slang:
    github: jeromegn/slang
    branch: master

And then install them:

$ shards install

We have Kemal, Redis and I've added Slang because I like this kind of templates better. But I'll put the ECR version as well.

Here is the template which we'll put in views/index.slang:

doctype html
html
  head
    title Chat
  body
    pre#chat
    form
      input#msg placeholder="message..."
      input type="submit" value="Send"

    script src="https://code.jquery.com/jquery-1.11.3.js"
    javascript:
      $(function() {
        var ws = new WebSocket("ws://" + location.host + "/chat");
        ws.onmessage = function(e) { $('#chat').append(e.data + '\\n') };

        $("form").submit(function(e) {
          e.preventDefault();
          var message = $('#msg').val();
          console.log(message);
          ws.send(message);
          $('#msg').val(''); $('#msg').focus();
        });
      });

There is actually no dynamic content, and it is pretty much the version of the original tutorial. Here is the HTML version which you can use in ECR:

<!DOCTYPE html>
<html>
  <head><title>Chat</title></head>
  <body>
    <pre id="chat"></pre>
    <form>
      <input id="msg" placeholder="message..." />
      <input type="submit" value="Send" />
    </form>
    <script src="https://code.jquery.com/jquery-1.11.3.js"></script>
    <script>

      $(function() {

        var ws = new WebSocket("ws://" + location.host + "/chat");
        ws.onmessage = function(e) { $('#chat').append(e.data + '\\n') };

        $("form").submit(function(e) {
          e.preventDefault();
          var message = $('#msg').val();
          console.log(message);
          ws.send(message);
          $('#msg').val(''); $('#msg').focus();
        });
      });

    </script>
  </body>
</html>

Not much but there is an empty pre element to display all messages received, a form for posting messages. And then in javascript we use jQuery. We connect to the local websocket under /chat. We set it to print in the pre element each message received.

We then override the submit mechanism on the form so that it sends messages to
the websocket. That's it.

Now for the most important part, src/kemal-redis-chat.cr:

require "kemal"
require "kilt/slang"
require "redis"

CHANNEL = "chat"
SOCKETS = [] of HTTP::WebSocket

REDIS = Redis.new
spawn do
  redis_sub = Redis.new
  redis_sub.subscribe(CHANNEL) do |on|
    on.message do |channel, message|
      SOCKETS.each {|ws| ws.send(message) }
    end
  end
end

get "/" do
  render "views/index.slang"
end

ws "/chat" do |socket|

  SOCKETS << socket

  socket.on_message do |message|
    REDIS.publish(CHANNEL, message)
  end

  socket.on_close do
    SOCKETS.delete socket
  end

end

Kemal.config.port = ENV["PORT"].to_i || 3000
Kemal.run

It's quite a lot to take, so let's break it down.

require "kemal"
require "kilt/slang"
require "redis"

CHANNEL = "chat"
SOCKETS = [] of HTTP::WebSocket

Here we require the necessary libraries and keep the channel name and the sockets in a constant.

REDIS = Redis.new
spawn do
  redis_sub = Redis.new
  redis_sub.subscribe(CHANNEL) do |on|
    on.message do |channel, message|
      SOCKETS.each {|ws| ws.send(message) }
    end
  end
end

This is the redis part. We create a first redis client which we'll use for publishing messages. And we create another one for subscriptions. This one has to be in a spawned block because it would block the execution otherwise. It needs to run in the background asynchronously. We set it so that we push to the websockets of this server instance each message received on our channel.

get "/" do
  render "views/index.slang"
end

This is just the handler for our web page.

ws "/chat" do |socket|

  SOCKETS << socket

  socket.on_message do |message|
    REDIS.publish(CHANNEL, message)
  end

  socket.on_close do
    SOCKETS.delete socket
  end

end

Here we have the websocket handler. On connection we add the new socket to the constant holding our websockets. When closing we remove the socket.

The most interesting part is when we receive messages. Instead of sending the message to our sockets, we publish it on the redis channel. Otherwise we would only send the message to the sockets of this particular server instance. Note that we use the first redis client we've created, the one dedicated to publishing.

What this does is that redis will send the message to all subscribing clients. That is to say from any server instance. Then each client forwards the message to the sockets they are responsible for.

Kemal.config.port = ENV["PORT"].to_i || 3000
Kemal.run

Finally we configure Kemal so that it can start on any port specified as an environment variable. There you go.

To test it, first make sure you have a Redis server running on its default port. Then start your application in two separate terminal windows. Pick a different port for each. This simulates load balancing.

$ PORT=3000 crystal src/kemal-redis-chat.cr
$ PORT=4000 crystal src/kemal-redis-chat.cr

Now in your browser, connect two different windows to each version http://localhost:3000/ and http://localhost:4000/. Type messages and you'll see they are sent to all browser windows. It doesn't matter which port they are on.


Tags:     crystal kemal websocket redis chat