Node in Production


Best practices and techniques


@eschoff - @funkytek - @uhduh - @wearefractal

Consulting, training, products, professional services

contact@wearefractal.com

Open source is at github.com/wearefractal

TODO List


  • Git workflow
  • NPM dependencies
  • Private NPM registry
  • Logging best practices
  • Error management
  • Process management






Workflow

Git


Use two different branches. One for development, one for production.

Use CI servers on each branch to deploy to staging and production.

NPM Shrinkwrap 


npm shrinkwrap

ALWAYS BEFORE PUSHING TO PRODUCTION!


This ensure package integrity via SHA hashes

This prevents security + stability issues

Private NPM


Simple answer: Use kappa


Private modules will be handled by this registry, request for other modules will be proxied to the main registry.

Kappa

Private NPM alternatives


You can use git or HTTP URLS to install packages!


{
  "name": "cool project",
  "dependencies": {
    "module1": "http://godaddy.com/secret/package.tar.gz",
    "module2": "git@secretgit.godaddy.com:myteam/module2.git"
  }
}






Application

Logging

Log all the things with Bunyan. Bunyan is a fast JSON logging library for Node.js.

You should be logging:

  • Every Request
  • Application Errors
  • Uncaught Exceptions
  • Process Metrics (Memory Usage, Heap Size, Uptime)

Setup

Create a logger and attach your streams and serializers.
                    var bunyan = require('bunyan');
var elasticsearch = require('bunyan-elasticsearch');

var logger = bunyan.createLogger({
  name: "My Application",
  streams: [
    { stream: process.stdout },
    { stream: new Elasticsearch() }
  ],
  serializers: bunyan.stdSerializers
});

logger.info('Starting application on port %d', app.get('port'));

Log Every Request

Create a log message for every request with response times and content length.
                  var requestLogger = function (req, res, next) {
  var start = new Date();
  var end = res.end;
  res.end = function (chunk, encoding) {
    var responseTime = (new Date()).getTime() - start.getTime();
    end.call(res, chunk, encoding);
    var contentLength = parseInt(res.getHeader('Content-Length'), 10);
    var data = {
      res: res,
      req: req,
      responseTime: responseTime,
      contentLength: isNaN(contentLength) ? 0 : contentLength
    };
    logger.info(data, '%s %s %d %dms - %d', data.req.method, data.req.url, data.res.statusCode, data.responseTime, data.contentLength);
  };
  next();
};
Attach the request logger middleware to your Express app
app.use(requestLogger);

Custom Error Handler

Log errors that would normally get caught by the Express error handler
                  var errorLogger = function (err, req, res, next) {
  logger.error({ req: req, res: res, error: err }, err.stack);
  next(err);
};
And after the express.router()
  app.use(errorLogger);

Log Exceptions

Wrap every request in a domain and log exceptions. This should be you very first function in your middleware chain.
                  app.use(function (req, res, next) {
  var requestDomain = domain.create();
  requestDomain.add(req);
  requestDomain.add(res);
  requestDomain.on('error', function (err) {
    var data = { req: req, res: res, error: err };
    logger.fatal(data, err.message);
  });
  next();
});

Log Process Metrics

It's a good idea to log your process matrics periodically for monitoring.
                  setInterval(function () {
  var startTime = Date.now();
  setImmediate(function () {
    var data = process.memoryUsage();
    data.uptime = process.uptime();
    data.pid = process.pid;
    data.tags = ['process-metrics']; 
    data.lag = Date.now()-startTime;
    logger.info(data,
       'process.pid: %d heapUsed: %d heapTotal: %d rss: %d uptime %d lag: %d',
       data.pid,
       data.heapUsed,
       data.heapTotal,
       data.rss,
       data.uptime,
       data.lag
       );
  });
}, 5000);
Set the interval to something sane based on how much log data you want to manage.

Set up Dashboards with Kibana

Processes


Use monit + upstart to keep your server online

Use node's cluster module to make sure an error doesn't bring your whole server down 

var cluster = require('cluster');
var os = require('os');

if (cluster.isMaster) {
  var children = os.cpus().map(cluster.fork);
  cluster.on('exit', function(worker) {
    cluster.fork();
  });
  return;
}

var server = require('./your-server');




Questions?