I met some fantastic people at the fantastic Keeping It Realtime Conference last week. The speakers were great. The conversations were fantastic. The parties were fun. I will go again next year. There was so much information to consume that I am just now sorting some of it out. In my mind, the experience plays backs like ten thousand snapshots in rapid succession. A blur of events, conversations, people, code, and crappy wifi. A couple of faces stand out: the crew from Microsoft, Paul Batum, Glenn Block, and of course Scott Hanselman. They left a big impression on me. Not because I was in shock that Microsoft was actually represented at this event (though I was), or that these guys were deep behind enemy lines (you should have seen the mac/win ratio), or because they were all wickedly smart (they were). No, it was that Microsoft seems to be genuinely making a go of supporting node on Windows and it actually looks pretty good.

Okay. So what does this have to do with base jumping wingsuit insanity? Well, dear reader, not much. But I’ll try and tie it back. If you browse back through my archives or follow me on twitter, you will get a lot of Linux and node content. But every so often, something Windowsy slips through. You see, the truth is: I am .NET developer. Or at least I used to be. Now, I manage a team of developers–a mixed team. We are about 50/50 .NET and Django with a sprinkle of node on top. Our various products have to integrate with each other and I make platform decisions on a daily basis where I am pitting a Windows stack against a Linux one. In short, I “grind the crack” between them. But without the wingsuit and inevitable spectacular death.

Back to the conference. At the opening event, I met Glenn and Paul. I had a great discussion with them about where my team is and where we are going. I like node and I see some great use cases for it in our business. At one point, Glenn asked me, “What would it take to you get you choose windows?” I answered that it was probably too late for us, “We have node programs running on Linux. Why would we switch?” I told him that I wholeheartedly supported node on Windows for the good of the node community, but that we were already past the point of needing it ourselves.

Fast forward two days. I make sure to attend Glenn’s and Tomasz’s talk on node for Windows (the main point behind the recent 0.6.X release by the way). I didn’t expect to hear anything I wasn’t already aware of. And I was surprised. Floored even. They have actually done a great job creating a Windows “story” for node. They want you to run node with IIS. If you do that, you get some pretty awesome stuff for free:

• built-in process management ala forever
• load balancing between node processes
• graceful auto-refresh of the node process when code changes
• remote node-inspector (hell yes!)
• logs over http

Yes, I know you can get all these things on Linux. And, maybe there are arguments to be made about scaling (haven’t seen benchmarks, so I have no idea). But you know what? I don’t have scaling issues. We have a modest user base. Our biggest challenges are building a great experience for our users, integrating disparate systems, and maintaining our automated deployment and testing infrastructure. With iisnode, I can just include the node javascript along side our existing .NET app and ship it with the same TeamCity deployment. I won’t even need to restart IIS!

We were already planning on using node and WebSockets to bring some realtime features to our ASP.NET MVC app and Microsoft just made my life simpler. Glenn: you got me back (at least partly). I am excited to see what you guys do next.

Bravo.

Anyway, quick vim tip #1. If I have a chunk of a code with a string:

var blah = "Some string that I want to replace" + previouslyDefinedVariable

I want to replace it with “new string”. In vim, do:

1. Put your cursor on the first character inside the quotes of the string you want to end up with, in our case, the n from “new string”.
2. Type yi” – yank inbetween quotes
3. Navigate to the string you want to replace
4. Type di” – delete inbetween quotes
5. “0P – paste content from yank buffer
6. Marvel at your vim prowess.

Got any great vim tips? I would love to here about them. Send them to vimtips@fzysqr.com. Check out my full vim config fork of scrooloose’s config on github (here)[https://github.com/jslatts/vimfiles].

Updated Cory Schmitt wrote in with a helpful suggestion that saves two keystrokes!

1. Put your cursor on the first character inside the quotes of the string you want to end up with, in our case, the n from “new string”.
2. Type yi” – yank inbetween quotes
3. Navigate to the string you want to replace
4. Type vi”p – visual mode select inbetween quotes and paste over
5. Congratulate yourself for being awesome.

• Added batch mode to group callbacks
• Added second callback for file removal notices (this works in batch mode as well)
• Switched unit tests to vows.js and added additional coverage
• Made errors callback immediately in standard node-fashion.
• Fixed bugs

Get it on npm now or head to github.

Thank you for all the feedback.

Get it on npm: npm install stalker

You can test it by running node example/test.js and then dropping files and folders in the example directory.

To incorporate it in your program, simply call stalker.watch with your callback:

stalker.watch('/var/foo', function(err, file) {
  console.log('stalker saw file %s', file);
});

Code is at github: https://github.com/jslatts/stalker

node-inspector &
node --debug-brk --debug server.js 

Head to http://0.0.0.0:8080/debug?port=5858 to find your app nicely stopped on the first line.

Happy noding!

1. Find a colleague with Windows who has a properly functioning kiln client (or follow the process in this kiln stackexchange post).
2. Have them open up their user profile and look for an _hgookies folder. Open the cookie file inside.
3. There will be a line that looks something like (DBID’s have been changed to protect the innocent): servername FALSE / FALSE DBID o1sev3t3dn01dabdppin90wexczimb
4. Have them send you the DBID value
5. On your Mac, open the cookie in your ~/.hgcookies/ and add the missing DBID value.
6. You might have to hg logout, then try an hg get/in to put your credentials in. After that it should stop prompting you.

There are three main parts to my eventual solution:

• Socket.IO reconnection
• client versioning
• pushed client updates

Using the same techniques used in nodechat.js, we will write a simple node.js + socket.io application that displays it’s version number and can be remotely updated. If you are unfamiliar with Node.js or Socket.IO, you might want to checkout the original nodechat tutorial first.

Setup

Before the fun starts, we need to perform a little setup. I tried to keep this project as simple as possible, so please excuse any and all bad practices you see.

First, create a project directory. Within it, install these modules using npm:

npm install express
npm install socket.io
npm install jade

Then create the following directory structure and empty files:

project/
-server.js
-views/
--index.jade
-public/
--js/
---jquery-1.5.1.js

server.js will contain all of our server-side application code. index.jade will server up the client-side application code and the minimal layout. jQuery is included to make our life easier.

The contents of server.js should be as follows. Explanations are in-line:

//Include our various dependencies for this demo and set up socket.io to work with express
var express = require('express')
    , app = express.createServer()
    , io = require('socket.io')
    , socket = io.listen(app)
    , version = '1.1.awesome';

//Tell express where to find our jQuery library
app.use(express.static('./public'));

//Setup express to render index.jade to any client requesting '/'. 
app.get('/', function (req, res) {
    res.render('index.jade', { 
        locals: { version: version } //Send the version variable we defined above to the template
        , layout: false              //Tells jade we aren't going to use a layout.jade base template
    });
});

//Make express hang out and listen for incoming connections
app.listen(8000);

The contents of index.jade should be as follows:

!!! 5
html(lang='en')
  head
    title fzysqr.com client updating demo
    script(type='text/javascript', src='/js/jquery-1.5.1.js')
    script(type='text/javascript', src='/socket.io/socket.io.js')
    script
      $(document).ready(function () { 
        //Crockford compliance! Define all the need variables here
        var socket;

        //- Setup the socket library. The rememberTransport and tryTransportsOnConnectTimeout 
        //- are neccessary because the defaults will cause some browsers to degrade to 
        //- crappy transports when they are disconnected (think suspended laptop) and will 
        //- subsequently stick on said crappy transport even after the network is up again.
        socket = new io.Socket(null, {
            rememberTransport: false
            , tryTransportsOnConnectTimeout: false 
        });

        //- Simple connect event so we can see that we are connecting
        socket.on('connect', function () { 
            console.log('Connected! Oh hai!');
        }); 

        //- Start the connection
        socket.connect();
      });
  body
    p Hello fzysqr.com reader! 
      br
      br
      | This is version:
    //- Render the version variable
    h1 #{version}

Once you have all the pieces in place, run node server.js and point your browser to http://localhost:8000/. You should see something like:

Now we are ready to have some fun!

Reconnectionism

As web applications have become richer, user expectations of a traditional thick client software like experience has increased. You simply expect Gmail to reconnect when you open your laptop. Likewise, if nodechat detects that a connection is disrupted, it will display a notice to the user and attempt to reconnect every 30 seconds until it succeeds (or the user closes the tab). Let’s modify our new app to do the same.

All of the required changes are in index.jade. Here are the deltas:

....

//Crockford compliance! Define all the need variables here
var socket, connected, trying, tryConnect;

....

//- Log our connection even and hide the disconnectMessage element in case 
//- we are reconnecting after an interuption
socket.on('connect', function () { 
    console.log('Connected! Oh hai!');
    $('#disconnectMessage').hide();
}); 

//- This method checks to see if we are connected, if not, tell socket.io to connect 
//- and reset a timeout to check again in 30 seconds.
tryConnect = function () {
    if (!connected) {
        console.log('Trying to reconnect...');
        socket.connect();
        clearTimeout(trying);
        trying = setTimeout(tryConnect, 30000);
    }
};

//- Handle disconnection by displaying a notice to the user and
//- setting a timer to try connecting in 500ms. 
socket.on('disconnect', function () {
    console.log('Disconnected. Oh noes!');
    connected = false;
    trying = setTimeout(tryConnect, 500);
    $('#disconnectMessage').show();
});

....

We define a handler for the disconnect event from the socket object (which is an EventEmitter) and use a timer to try to reconnect the socket.io connection repeatedly until it succeeds. Go ahead and fire it up again (node server.js) and go back to http://localhost:8000/. Initially, you should see the same thing as before:

Then if you kill the node server with ctrl-c, you should see this:

Then if you restart the node server and wait patiently (up to 30 seconds), the client should reconnect again and you should see this again:

Client updates

Once nodechat was able to reconnect clients, I noticed a big spike in channel retention. People were able to stay connected/reconnected for weeks at a time. They were also able to reconnect after I shipped a new version of nodechat. This introduce a whole new problem. What happens if we deploy a breaking change to the code? The clients reconnect and promptly crash. Not good UX. Even if I am careful to maintain backwards compatibility, I still want to make sure that long running clients see improvements and bug fixes in the client-side code. To handle this, clients listen for a special upgrade message from the server and react accordingly.

This time we have some changes in server.js. Once again, deltas only:

//Handle the connect event for the socket.io server in order to send out the 
//current codebase version to any connecting clients.
socket.on('connection', function(client) {
    client.send({
        event: 'version'
        , data: version
    });
});

Then, in index.jade, we add a new variable to track the version and a handler for incoming socket messages on the client. If we received the version update message, we force a refresh:

....

//Crockford compliance! Define all the need variables here
var socket, connected, trying, tryConnect
    , version = '#{version}';       //Keep track of the code version at the time this page was rendered

....

//- Handle the message event to see if we are getting a version message from the server
//- If we do, check it's contents to make sure we are on the latest and greatest. If we
//- are out of date, force a reload to get the latest code.
socket.on('message', function (msg) {
    if (msg && msg.event === 'version' && version !== msg.data) {
        setTimeout(function() {
            window.location.reload();
        }, 5000);
    }
});

....

That’s it! All the pieces are in place. Fire up the server and try it out. When we first connect to http://localhost:8000/, you expect to see the same screen as before:

Then kill the node server and you should see this:

Now, go into server.js and update the version to something else:

, version = '1.2.awesomer';

Save the file and start up the node server again. Switch back to the web browser and wait. Do NOT reload the browser. You should first see the 1.1 version reconnect:

Then, after another view seconds, the browser should reload the whole page and display:

Wrapping up

The full source for this tutorial can be found here on github. The three different tags represent the three different stages in this tutorial.

This type of client versioning is going to be old hat for anyone who has done serious work in real time web applications, but it was not immediately obvious to me when I first dove into node.js and Socket.IO. Having the ability to pull all my clients along when I deploy a new version has made it easier to push bug fixes and feature to nodechat without dropping the whole channel off-line every time.

As always, I welcome any comments, questions, or suggestions. Find me on twitter (@jdslatts) or drop by nodechat.

Now back to your regularly scheduled programming:

One of the problems I have had with nodechat.js is that using sessions to handle the transition of authenticated users between express and socket.io has always been somewhat finicky.

Nodechat.js and the previous nodechat-tutorials have used sessions to manage this transition by storing a username and password in the session after initial login and making it available to the socket.io listener on new connections and during each message. While adequate most of the time, this method never worked correctly with all the various socket.io transports and it seemed like clients would frequently get stuck when reconnecting, requiring them to reload the page to get back into nodechat.js.

I now have a pretty decent way to address this issue and I am going to share it with you. A little technique I came up with that I call Purgatory…

The goal of this tutorial is to set up nodechat to authenticate clients initially through connect/express and then to pass that authentication along to a subsequent socket.io connection without relying on the session.

Before we get started, a few notes:

• The code can be found on the v0.3 tag on github if you want to follow along.
• This tutorial was written with npm 1.0RC, so I have been able to include all dependencies in the source code.
• I am going to assume you are comfortable with node/socket or have at least worked through the previous two nodechat tutorials.

In standard fzysqr tutorial style, we will just march through the important bits of code, explaining as we go.

Views

We start off by making a few views. First up, our login page:

login.jade

Crate views/login.jade as follows:

!!! 5
html(lang="en")
  head
    title nodechat login
  body
    #heading
      h1 nodechat login
    #content
      p
        | please login 
      form(method="post", action="/")
        div(style='width:200px; text-align:right')
          label username:
            input(type='text', name='username')
          br
          label password:
            input(type='password', name='password')
          br
          input(type='submit', value='login')
      p
        | or create an account 
      form(method="post", action="/")
        div(style='width:200px; text-align:right')
          label username:  
            input(type='text', name='username')
          br
          label password:  
            input(type='text', name='password')
          br
          label email:  
            input(type='text', name='email')
          br
          input(type='submit', value='signup & login')

We have a single login/signup page that uses two different forms to handle their respective duties. The password is not hidden when signing up for a new account. Both forms post to the same URI, and we use the existence of the email field to tell which one was used.

index.jade

Next make views/index.jade as follows:

!!! 5
html(lang="en")
  head
    title nodechat
    script(type="text/javascript", src="/lib/jquery-1.5.1.js")
    script(type="text/javascript", src="/lib/underscore.js")
    script(type="text/javascript", src="/lib/backbone.js")
    script(type="text/javascript", src="/socket.io/socket.io.js")
    script(type="text/javascript", src="/models/models.js")
    script(type="text/javascript", src="/controllers/controllers.js")
    script(type="text/javascript", src="/views/views.js")
    script
      //Fake out FF and IE8
      function log() {
          if (typeof console == 'undefined') {
              return;
          }
          console.log.apply(console, arguments);
      }

      $(document).ready(function () { 
          window.app = NodeChatController.init({hashpassword: !{locals.hashpassword}, userName: '!{locals.name}'}); 
      });
  body
    #heading
      h1 nodechat
    #content
      p 
        | connected clients
        span#client_count 0
      p 
        a(href='/logout') logout

      p
        label You are logged in as: 
          = locals.name 
      p
        | Fun Chat Messages
        ul#chat_list

      form(method="post", action="#", onsubmit="return false")#messageForm
        p
          label Message:
            input(type='text', name='message')
            input(type='submit', value='send')

This page serves up the actual chat UI. It includes the necessary dependencies, sets up the console logging and initializes the controller. The username and hashpassword are stored in the locals collection and replaced by express when the template is rendered. In turn, the parameters are passed to the controller for use later.

views.js and models.js

The backbone views are defined in the views/views.js folder. I won’t go over each view, as they are largely unchanged from the previous tutorials. Just pull the file from github and place in views.

We have the same story with models.js. It is basically the same as previous tutorials. Just pull the file from github and place in models.

clientauthrequest

We have seen how the username/hashpassword are passed into the controller during instantiation. Now let’s look at the controller:

var NodeChatController = {
    init: function(options) {
        var mySocket, user, view, hashpassword;

        this.socket = new io.Socket(null, {port: 8001});
        mySocket = this.socket;
        user = this.userName = options.userName;
        hashpassword = this.hashpassword = options.hashpassword

        this.model = new models.NodeChatModel();
        this.view = new NodeChatView({model: this.model, socket: this.socket, el: $('#content')});
        view = this.view;

        this.socket.on('connect', function () { 

            mySocket.send({
                event: 'clientauthrequest'
                , user: user
                , hashpassword: hashpassword
            });

            log('Connected! Oh hai!');
        }); 

        this.socket.on('message', function(msg) {view.msgReceived(msg)});
        this.socket.connect();

        this.view.render();

        return this;
    }
};

The controller stores the username and hashpassword and sets up an on connect event handler to send them back to the server as a “clientauthrequest” message.

Which brings us to Purgatory.

Purgatory

Previously, our session based authentication went something like:

1. [server] User submits username/password to login form.
2. [server] Check username/password by calculating the hash and comparing the information to what has been stored.
3. [server] If it matches, store a user object in the session.
4. [server] Redirect to ‘/’ and load the client-side code.
5. [client] The client fires up and connects back to the server.
6. [server] On incoming connection, the server checks the session for a valid user object and allows the connection.
7. [server] On incoming messages, the server checks the session for a valid user object and allows the message.

The process relies on the socket transport having access to the session cookie and that is what makes it unreliable.

The new, session-less process is:

1. [server] User submits username/password to login form.
2. [server] Check username/password by calculating the hash and comparing the information to what has been stored.
3. [server] If it matches, send the rendered index.jade template to the client with the username/hashpassword as part of the rendered html.
4. [client] The client fires up and connects back to the server.
5. [server] The server places client connection in purgatory.
6. [client] The client sends a “clientauthrequest” message back to the server.
7. [server] On incoming message, the server checks to see if the client is in purgatory. It is, so the credentials are verified.
8. [server] The authentication passes, so the client connection is removed from purgatory.
9. [server] On future incoming messages, the messages are known to be from an authenticated client and are allowed to pass.

Core.js

Let’s look at the pieces that make this happen in core.js:

routes

app.get('/', function (req, res) {
    res.render('login');
});

app.post('/', function (req, res) {
    signInAccount(req, res)
});

We only need to handle two routes: GET ‘/’ and POST ‘/’. For get, render login.jade. For POST, call signInAccount().

signInAccount

function signInAccount(req, res) {
    if (req.body.email) {
        auth.createNewUserAccount(req.body.username, req.body.password, req.body.email, function (err, user) {
            if ((err) || (!user)) {
                es.redirect('back');
            }
            else if (user) {
                res.render('index', {
                    locals: { name: user.name, hashpassword: JSON.stringify(user.hashpassword) }
                });
            }
        });
    }
    else {
        auth.authenticateUser(req.body.username, req.body.password, function (err, user) {
            if (err) {
                winston.error('[signInAccount][authenticateUser][fn] Error: ' + err);
            }

            if (user) {
                res.render('index', {
                    locals: { name: user.name, hashpassword: JSON.stringify(user.hashPass) }
                });
            } 
            else {
                res.redirect('back');
            }
        });
    }
}

signInAccount() determines whether we are logging in, or creating a new account by checking the request object for an email form field. Upon successful authentication or account creation, the username and hashed password are passed to express in the local variables collection and rendered into the index template.

socket event handler

socket.on('connection', function (client) {
    var clientPurgatory = purgatory();

    client.on('message', function(message) {
        if (clientPurgatory.stillInPurgatory()) {
            if(message.event === 'clientauthrequest') {
                //If we can get out of purgatory, set up the client for pubsub
                clientPurgatory.tryToGetOut(message, client, function () {
                    activeClients += 1;
                    winston.info('clients: ' + activeClients);
                    socket.broadcast({
                        event: 'update'
                        , data: activeClients
                    });

                    client.on('disconnect', function () {
                        clientDisconnect(client);
                    });
                });
            }
        }
        else {
            //If this is called, we are not in purgatory, so handle it normally
            chatMessage(client, socket, message);
        }
    });
});

The connection event handler creates a closure with a new instance of the purgatory() function. It then sets up an anonymous function to handle client message events. Each time a message is received from this client, stillInPurgatory() is called. If it returns true, check to see if a clientauthrequest message was received and call tryToGetOut(). We pass tryToGetOut() a callback to call if the client successfully gets out of purgatory.

If stillInPurgatory() returns false, then handle the message as a regular chat message.

purgatory

Note: if you don’t understand Javascript closures, read up a bit on them.

function purgatory() {
    var inPurgatory = true;
    return {
        tryToGetOut: function (message, client, cb) {
            if (!message || !message.user || !message.hashpassword) {
                winston.info('[purgatory][tryToGetOut] Client with no user/hash attempting message. Client still in purgatory');
                return;
            }
            auth.authenticateUserByHash(message.user, message.hashpassword, function(err, data) {
                if (err) {
                    winston.info('[purgatory] Bad auth. Client still in purgatory');
                    inPurgatory = true;
                }
                else {
                    winston.info('[purgatory] out of purgatory');
                    inPurgatory = false;

                    //Once we are sure the client is who s/he claims to be, attach name and hash for future use.
                    client.user = message.user;
                    client.hashpassword = message.hashpassword;

                    cb && cb();
                }
            });
        }
        , stillInPurgatory: function() {
            winston.info('[purgatory] status ' + inPurgatory);
            return inPurgatory;
        }
    }
}

The purgatory function returns a function that generates a closure for each client connection in order to keep track of whether the client has escaped purgatory or not. Subsequent calls to tryToGetOut() and stillInPurgatory() can reference the inPurgatory variable generated by the closure.

tryToGetOut() looks for a username and hashed password in the message. If it is found, it attempts to authenticate them. If authentication passes, the inPurgatory variable is set to false and the passed in callback is fired.

stillInPurgatory() returns the inPurgatory variable.

Wrapping it up

New clients are sent to purgatory. They stay in purgatory until they earn their release by successfully authenticating. Using the purgatory strategy gains us better compatibility with socket.io’s various transports and allows us to quit worrying about cookies and session expiration. Production nodechat has been running this code for the last week and I have already seen a big decrease in connection/reconnection issues.

You can get the code for this tutorial at the v0.3 tag at github. If you want to see how this is used in a more fully featured application, check out the production nodechat.js code-base on github.

If you have questions, comments, or suggestions, you can get a hold of me on twitter @jdslatts, at comments@fzysqr.com, or by stopping in at nodechat.no.de.

-jslatts

The first nodechat.js tutorial introduced some fancy ideas, like using backbone.js on the server and streaming models over socket.io which is all well and great, but it left some of the more practical questions unanswered. Issues like coordinating authentication between express/connect and socket.io and creating, storing, and retrieving user profiles, are less sexy, but still quite important for real world use. So that is what the next this tutorial will look at. Practical stuff. Besides, everything is more fun when you do it in node, right?

I will also sprinkle you with some other neat wisdom nuggets along the way. If you want.

Overview

First off, nodechat.js has advanced well passed the point that it is clean and clear enough to be used as a tutorial. I have decided to create a new nodechat project that will be used strictly for tutorials. It lives at github also. The code for this tutorial is at the v0.2 tag. nodechat.js and nodechat-tutorial share same underlying design, so if you want to see a demo, come visit nodechat.no.de. There will probably even be someone there to chat with.

Our goal for this tutorial is to put nodechat behind a simple login page, provide a way to sign up new accounts, and make sure that socket.io verifies the accounts before accepting traffic.

Install Stuff

Any good node.js adventure starts off with npm powerups. For this adventure, we will need everything from part one of this tutorial and:

• sudo npm install connect-redis
• sudo npm install joose
• sudo npm install joosex-namespace-depended
• sudo npm install hash

And just for fun, let’s do: sudo npm update

We will be using connect-redis as our connect session store and hash as our hashing library. The joose modules are there because hash requires them.

Authentication

To authenticate users and retrieve bits of information about them, we will need to:

1. Create user accounts
2. Check login information
3. Retrieve the profile if successful
4. Kick them out if we are not successful
5. Do something with the profile information

Let’s start with the easy stuff. We will need a login page and an account creation page. Create the following jade template files in the views/ directory:

login.jade

!!! 5
html(lang="en")
  head
    title nodechat login
  body
    #heading
      h1 nodechat login
    #content
      p
        | Please login or go 
        a(href='/signup') here
        |  to create a new account.
      form(method="post", action="/login")
        p
          label Username:
            input(type='text', name='username')
        p
          label Password:
            input(type='password', name='password')
        P
          input(type='submit', value='send')

Not much to this page. Just a simple login form with a link to our signup page.

signup.jade

!!! 5
html(lang="en")
  head
    title nodechat new user
  body
    #heading
      h1 nodechat new user
    #content
      p
      form(method="post", action="/signup")
        div(style='width:350px; text-align:right')
          label Username:  
            input(type='text', name='username')
          br
          label Password:  
            input(type='text', name='password1')
          br
          label Password (again):  
            input(type='text', name='password2')
          br
          label Email:  
            input(type='text', name='email')
          br
          label Favorite thing about ponies:  
            input(type='text', name='ponies')
          br
          input(type='submit', value='signup')

Again, pretty simple. Just collect the signup information and whatever special information we may *ahem* require from our users.

Routes

Now we need to set up routes to serve our new pages. In core.js, after we have set up express, we want to include several routes.

app.get('/login', function(req, res){
    res.render('login');
});

app.post('/login', function(req, res){
    auth.authenticateUser(req.body.username, req.body.password, function(err, user){
        if (user) {
            req.session.regenerate(function(){
                req.session.cookie.maxAge = 100 * 24 * 60 * 60 * 1000; //Force longer cookie age
                req.session.cookie.httpOnly = false;
                req.session.user = user;

                res.redirect('/');
            });
        } else {
            req.session.error = 'Authentication failed, please check your username and password.';
            res.redirect('back');
        }
    });
});

Setup the routes for GET and POST /login.

• GET returns the login.jade template
• POST calls the authentication module to verify login details.
  • Failures are redirected back to the login page.

*We will cover the authentication module in a minute. For now, just know that it does what it sounds like it might do :)

If the authentication module gives us a user object back, we ask connect to regenerate the session and send the client back to index. Note: we specify a long cookie age so users won’t have to log in frequently. We also set the httpOnly flag to false (I know, not so secure) to make the cookie available over Flash Sockets.

app.get('/signup', function(req, res){
    res.render('signup');
});

app.post('/signup', function(req, res) {
    auth.createNewUserAccount(req.body.username, req.body.password1, req.body.password2, req.body.email, req.body.ponies, function(err, user){
        if ((err) || (!user)) {
            req.session.error = 'New user failed, please check your username and password.';
            res.redirect('back');
        }
        else if(user) {
            res.redirect('/login');
        }
    });

});

Setup routes for GET and POST ‘/signup’:

• GET returns signup.jade
• POST calls createNewUserAccount() in the auth module.
  • Failures redirect to the same page
  • Success redirects back to /login for the user to formally log in

Next ‘/logout’:

app.get('/logout', function(req, res){
    req.session.destroy(function(){
        res.redirect('home');
    });
});

This route tells connect to destroy the session, which will cause nodechat to require the user to login again if they access the ‘/’ route.

app.get('/*.(js|css)', function(req, res){
    res.sendfile('./'+req.url);
});

app.get('/', restrictAccess, function(req, res){
    res.render('index', {
        locals: { name: req.session.user.name, hashPass: JSON.stringify(req.session.user.hashPass) }
    });
});

function restrictAccess(req, res, next) {
    if (req.session.user) {
        next();
    } else {
        req.session.error = 'Access denied!';
        res.redirect('/login');
    }
};

These last two routes already exist in nodechat v0.1. The only difference here is the reference to restrictAcess in the ‘/’ route. restrictAccess() is an express route middleware. Control is passed to the middleware function before the route function is called. We use restrictAccess() to verify that we have a valid user key in the session (implying that authentication has succeeded) before we send the client the requested route.

If we do not have a valid user object in the session, then we redirect the client to the ‘/login’ route. This effectively locks down our ‘/’ route from unauthenticated access. You could add the restrictAccess() to any route you want to protect.

model

That covers our new routes. Next we will need a backbone.js model for user accounts (well we don’t need a model, but nodechat is backbone.js on the server right?). Add a simple empty model to our models/models.js:

models.User = Backbone.Model.extend({});

That was easy.

auth.js

Let’s turn our attention to the authentication module touched on earlier. Create a new file, lib/auth.js. This module will handle the creation, verification, and profile loading of user accounts. It exposes two public methods: authenticateUser() and createNewUserAccount().

This will be a CommonJS module so we need to start off with some set up:

(function () {
    if (typeof exports !== 'undefined') {
        redis = require('redis');
        rc = redis.createClient();
        models = require('../models/models');

        //joose is required to support the hash lib we are using
        require('joose');
        require('joosex-namespace-depended');
        require('hash');
    } 
    else {
        throw new Error('auth.js must be loaded as a module.');
    }

Here we are checking to see if this code is included as a module. If it is, we go ahead and include our dependencies (in this case, our models lib, redis, and hash + friends). If we are not a module, we may as well explode because the rest of the code won’t run without redis and hash.

exports.authenticateUser = function(name, pass, fn) {
    console.log('[authenticate] Starting auth for ' + name + ' with password ' + pass);

    var rKey = 'user:' + name;
    rc.get(rKey, function(err, data){
        if(err) return fn(new Error('[authenticateUser] SET failed for key: ' + rKey + ' for value: ' + name));

        if (!data) {
            fn(new Error('[authenticateUser] invalid password'));
        }
        else {
            console.log('[authenticateUser] user: ' + name + ' found in store. Verifying password.');
            verifyUserAccount(data, pass, fn);
        }
    });
};

authenticateUser() takes a name, password, and callback. It checks to see if the user exists in redis. If it does, it calls verifyUserAccount().

CommonJS Modules

An aside: if you are unfamiliar with the CommonJS module specification, the basic idea is that you encapsulate your reusable methods and objects in a anonymous function that assigns its “public” methods to the global exports variable. This allows us to have private variables and methods, like verifyUserAccount() below, and to easily reuse our code through require() statements throughout our application. This is a pattern you will see constantly in node.js-world.

var verifyUserAccount = function(foundUserName, pass, fn) {
    var rKey = 'user:' + foundUserName;

    rc.get(rKey + '.salt', function(err, data){
        if(err) return fn(new Error('[verifyUserAccount] GET failed for key: ' + rKey + '.salt')); 

        if(data) {
            var calculatedHash = Hash.sha512(data + '_' + pass);
            rc.get(rKey + '.hashPass', function(err, data) {
                if(err) return fn(new Error('[verifyUserAccount] GET failed for key: ' + rKey + '.hashPass'));

                if (calculatedHash === data) {
                    console.log('[verifyUserAccount] Auth succeeded for ' + foundUserName + ' with password ' + pass);

                    rc.get(rKey + '.profile', function(err, data) {
                        if(err) return fn(new Error('[verifyUserAccount] GET failed for key: ' + rKey + '.profile' + ' for user profile'));

                        var foundUser = new models.User();
                        foundUser.mport(data);
                        foundUser.set({'hashPass': calculatedHash});

                        return fn(null, foundUser);
                    });
                }
                else {
                    return fn(new Error('[verifyUserAccount] invalid password'));
                }
            });
        }
        else {
            return fn(new Error('[verifyUserAccount] salt not found'));
        }
    });
}

verifyUserAccount() takes the same parameters as authenticateUser(). It assumes the passed in user exists in redis (remember, it is only accessible within the module, so that is a safe assumption to make) and then steps through the process of retrieving the salt, calculating the hash of the passed in password, then comparing it to the stored hash in redis. If the comparison is successful, create a new user model and pass it to the callback. Otherwise, any failure along the way means we callback with an error.

exports.createNewUserAccount = function(name, pass1, pass2, email, ponies, fn) {
    if (pass1 !== pass2) return fn(new Error('[createNewUserAccount] Passwords do not match'));

    var newUser = new models.User({name: name, email: email, ponies: ponies});

    var rKey = 'user:' + name;

    rc.set(rKey, name, function(err, data){
        if(err) return fn(new Error('[createNewUserAccount] SET failed for key: ' + rKey + ' for value: ' + name));

        var salt = new Date().getTime();
        rc.set(rKey + '.salt', salt, function(err, data) {
            if(err) return fn(new Error('[createNewUserAccount] SET failed for key: ' + rKey + '.salt' + ' for value: ' + salt));

            var hashPass = Hash.sha512(salt + '_' + pass1);
            rc.set(rKey + '.hashPass', hashPass, function(err, data) {
                if(err) return fn(new Error('[createNewUserAccount] SET failed for key: ' + rKey + '.hashPass' + ' for value: ' + hashPass));

                rc.set(rKey + '.profile', newUser.xport(), function(err, data) {
                    if(err) return fn(new Error('[createNewUserAccount] SET failed for key: ' + rKey + '.profile' + ' for user profile'));

                    newUser.set({'hashPass': hashPass});
                    return fn(null, newUser);
                });

            }); 
        }); 
    }); 
}

createNewUserAccount() verifies that the two passwords match, then uses the current timestamp to salt a hash of the password. It stores the hashed password and the passed in information about email and ponies in a user model which we will save as a poor man’s profile–if everything succeeds. Any failure along the way means we callback with an error. We are using SHA512 which has to be good, because it was invented by the NSA.

})()

If you have actually been cutting and pasting, you will need to end your file with this little snippet per the CommonJS spec.

Salting and Hashing

Another aside: if you are like me, you probably do not often mess around with actually storing usernames and passwords. Most frameworks will handle this for you quite nicely. I had to do some google-fu to figure out a decent way of doing this. Crypto experts: Please don’t email me telling me how the NSA could crack this in 5 minutes. It’s a chat program. I am also aware that there are probably five thousand node modules on npm that will do this for you as well. This is supposed to be fun dammit! Leave me alone.

The basic concept behind cryptographic hashing is to take a string of any length, in our case a password, and map it into a fixed length string, using an algorithm. This computation is inexpensive if you know the password up front, but doing the reverse–finding all the possible passwords that can map to the hash value–is computationally very expensive. Since we only store the hash, if our redis instance is ever compromised, an attacker would have to compute an enormous number potential passwords (and try each one) to figure out which one is the real password.

Hashing on its own can be susceptible to rainbow tables–precomputed reverse hash values. The attacker simply looks up the hash in the table and finds the list of corresponding possible passwords. The tables can be precomputed in advance and are easily obtained online by those who may be so inclined. This is where the salt comes in. The salt is a randomly generated string (I am just using a timestamp, but close enough) that is attached to the password before the cryptographic hash computation. We store the hash in the database as well; we will need it to verify passwords later on.

The salt mitigates the ability of an attacker to pre-compute the hash space. Since each user has a different salt, the attacker would have to compute the entire hash space for each user, and that is assuming the know the salt! If they don’t have the salt, the task becomes insurmountable.

nodechat.js uses the following salt/hash algorithm:

Account Creation

1. On account set up, get the current time in milliseconds. This is our salt.
2. Prepend the salt to the password separated by an underscore: salt_password.
3. Calculate the SHA512 hash of the salt_password string. This is our password hash.
4. Store the salt and the password hash for the user.

Account Verification

1. On an attempt to login to an existing account, retrieve the salt for the requested user.
2. Prepend the salt to the attempted password, separated by an underscore: salt_attemptedpassword.
3. Calculate the SHA512 hash of the salt_attemptedpassword string. This is our attempted password hash.
4. Retrieve the stored password hash and compare it to the attempted password hash. If both hash values match, then assume the passwords match. Otherwise, they do not.

Enough crypto for today. Back to node stuff.

Making it work with socket.io

We now have all the pieces in place to lock down our express routes and if we were not using socket.io, we would be done. But we are using socket.io, so we’re not. We need to be able to protect socket.io connections the same way we protect our routes. socket.io is not aware of the session and will not have access to the session store the same way express does, so we have to improvise.

Back in core.js we want to add some more code:

function disconnectAndRedirectClient(client, fn) {
    console.log('Disconnecting unauthenticated user');
    client.send({ event: 'disconnect' });
    client.connection.end();
    fn();
    return;
}

First we are going to give socket.io some teeth (get it, meaner socket.io? it bites? yes I am funny). When we have a client that shouldn’t be connected, kick ‘em off!. disconnectAndRedirectClient() takes a client socket.io object and a callback.

socket.on('connection', function(client){
    client.connectSession = function(fn) {
        if (!client.request || !client.request.headers || !client.request.headers.cookie) {
            disconnectAndRedirectClient(client,function() {
               console.log('Null request/header/cookie!');
            });
            return;
        }

        var match = client.request.headers.cookie.match(/connect\.sid=([^;]+)/);
        if (!match || match.length < 2) {
            disconnectAndRedirectClient(client,function() {
                console.log('Failed to find connect.sid in cookie');
            });
            return;
        }

        var sid = unescape(match[1]);

        rc.get(sid, function(err, data) {
            fn(err, JSON.parse(data));
        });
    };

    client.connectSession(function(err, data) {
        if(err) {
            console.log('Error on connectionSession: ' + err);
            return;
        }

        client.user = data.user;

        activeClients += 1;
        client.on('disconnect', function(){clientDisconnect(client)});
        client.on('message', function(msg){chatMessage(client, socket, msg)});

        socket.broadcast({
            event: 'update',
            clients: activeClients
        });

        var ponyWelcome = new models.ChatEntry({name: 'PonyBot', text: 'Hello ' + data.user.name + '. I also feel that ponies ' + data.user.ponies + '. Welcome to nodechat.js'});

        socket.broadcast({
            event: 'chat',
            data: ponyWelcome.xport()
        });
    });
});

There is a lot going on here, but much of this code is unchanged from v0.1. First, we are defining a helper method, connectSession, that will verify a client’s validity by checking for a cookie in the request header, then, if we find it, pulling their session out of redis! We then use the helper method in the ‘connection’ handler for our socket listener. Now instead of just accepting any old user connection, we are going to check that the client has a valid session (meaning they logged in). If they don’t, give them the boot! If they do, then we store a copy of the session data (yay we have access!) in the client object and then set up the rest of the socket events. Finally, send them a welcome message just to prove that we remembered their profile.

connect-redis

One last aside: connect-redis by visionmedia (I love his stuff!) is a connect session store the is backed by redis. It sets a cookie to track the session id, but it stores all other data in redis. This is really nice because a) we don’t have to worry about leaving cookies with critical data on the client, and b) we can grab the session data straight out of redis and give it to socket.io. We still have to parse the cookie to get the session id, but at least that is all we have to worry about.

function chatMessage(client, socket, msg){
    var chat = new models.ChatEntry();
    chat.mport(msg);

    rc.incr('next.chatentry.id', function(err, newId) {
        chat.set({id: newId});
        nodeChatModel.chats.add(chat);

        var expandedMsg = chat.get('id') + ' ' + client.user.name + ': ' + chat.get('text');

        rc.rpush('chatentries', chat.xport(), redis.print);

        socket.broadcast({
            event: 'chat',
            data:chat.xport()
        }); 
    }); 
}

The only material difference in chatMessage() is that we no longer simply believe the client when they tell us their name. Instead we use our newly found session power to tell them what their name is. And prevent fraud and stuff. Oh, and I also removed the call to rc.bgSave(). Because it was crashing during concurrent saves. If you want to persist your chats permanently, use the snapshotting method described by redis here.

Wrapping it up

That is about it. There may be some minor changes in some of the other files, so it never hurts to get the code from the v0.2 tag at github.

I want to close by saying that this method of authentication works and it has worked well enough to run in “production” on nodechat.no.de for several weeks. We do experience some funky issues once clients start falling back to xhr-polling and jsonp-polling, so be warned. This is the bleeding edge folks. Treat it as such.

My plan is to follow up with a different take on the authentication issue, one in which we ditch sessions altogether and use a shared key (like the hash) to sign each message over the socket. I’ll write it up if it works.

Finally, I want to say thanks to everyone who sent me email, tweets, or stopped by nodechat to… chat. It has been fantastic meeting you all and I look forward to seeing all the cool stuff you guys are cranking out.

Follow me on twitter @jdslatts, email me at comments@fzysqr.com, or stop by nodechat.no.de and say hi!

-jslatts

I have learned a few things over the last couple weeks. Good “lessons” I will share with you now:

1. Don’t actually put your crappy code online, publicly, for abuse. It will crash.
2. Sanitize inputs for your crappy demo code before you try to run it (in violation of rule number 1). XSS attacks will significantly lessen the quality of the experience you are trying to provide your audience.
3. Check for nulls. Everywhere. The internet is a cruel place and people are mean. They will look through your crappy demo code and figure out how to crash your crappy demo.
4. Running production-ish code on node.js requires a different mindset. If anything takes the process down, it is down for everyone. Coming from a typical web-development background, it took much crashing before I realized this. Writing robust, fault tolerant code is CRITICAL if you are hosting it on node.js
5. If you write a silly chat room and hang out it in all day, you will meet many interesting people.
6. There is a ton of interest in node.js. People are really excited to work with this stack.

People seemed to enjoy nodechat.js so I have been hacking out new features as fast as I can. Things that have changed since the demo:

• Authentication: reserve your username now, before they are all gone!
• Rightside up chat
• Mashtags: like #hashtags but #mashier
• Users online: now people are more than numbers
• Direct messages: now you can tell @jslatts how much you hate the UI in private
• Notifications: blinky tab thingies
• Linkify URLS / em / bold
• Flood control (yes, that one guy who knows who he is, I am thinking of you)
• Works in Firefox (sorry FF, I don’t test in you)
• Still Fun Chat Messages only

As always, you can find the latest code at github.

I am working on a follow up tutorial to cover authentication and some other fun tidbits I have discovered in the last couple weeks. Stay tuned.