13 January 2013

Using salt to feed the graphite carbon daemon

Part #2 in the VM visualisations series using salt. Read Part #1

I deemed my experiment in Part #1 a success, but it had a huge drawback in not keeping any state, it was only refreshing fresh data all the time. It also would not scale for multi user, because the fetching would be too heavy for many users to be monitoring simultaneously. Thus I had a new goal, separate the metric retrieving from the metric displaying. The plan is to collect metrics using Salt and its Returner component. Returners makes the salt published commands return data using a different route than back to the publisher (master). But first I had to choose a backend.

Cubism the web frontend I am using comes with support for two metric storing backends:

Cube has a mongodb + node stack. Graphite is built using python components whisper, carbon, and graphite-web. Whisper is a RRD-alike format, Carbon is a network daemon and collector backend for whisper, and graphite-web is written in django. Since I'm a fan of both python and django I decided to go with the graphite stack.

A simple diagram of the flow of this setup:

 The salt server                 targetted minions                The graphite sever
+----------------+             +------------------+      salt    +------------------+
|                |             |                  |    returner  |                  |
|  salt master   +---zeromq---->   salt minion    +-   TCP:2033 ->   carbon daemon  |
|                |             |                  |              |                  |
+----------------+             +------------------+              +------------------+

Graphite server

I wrote a simple graphite state to install the graphite server. Two parts is not covered in the state, the configuration files and service starting. I used all defaults everywhere.


    - pkgs:
      - python-django
      - python-pip
      - python-django-tagging
      - python-dev
      - python-twisted
      - python-memcache
      - memcached
      - python-cairo

      - require:
        - pkg: graphite

      - require:
        - pkg: graphite

      - require:
        - pkg: graphite

The carbon returner

Carbon supports two protocols and pickle, I struggled abit with getting pickle to work, so for testing I am using the text line protocol. That also enabled me to debug easier. I'm also happy to say that my carbon returner got merged into saltstack upstream, so it can be used and improved by everyone that's interested.


Take data from salt and "return" it into a carbon receiver

Add the following configuration to your minion configuration files::

    carbon.host: <server ip address>
    carbon.port: 2003


import syslog
import pickle
import socket
import logging
import time
import struct

log = logging.getLogger(__name__)

def __virtual__():
    return 'carbon'

def _formatHostname(hostname, separator='_'):
    ''' carbon uses . as separator, so replace this in the hostname '''
    return hostname.replace('.', separator)

def returner(ret):
    Return data to a remote carbon server using the pickle format
    host = __salt__['config.option']('carbon.host')
    port = __salt__['config.option']('carbon.port')
    log.debug('Carbon minion configured with host: {0}:{1}'.format(host, port))
    if not host or not port:
        log.error('Host or port not defined')

        carbon_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP)
        carbon_sock.connect((host, port))
    except socket.error as e:
        log.error('Error connecting to {0}:{1}, {2}'.format(host, port, e))

    timestamp = int(time.time())

    saltdata = ret['return']
    metric_base = ret['fun']
    metrics = []
    for name, vals in saltdata.items():
        for key, val in vals.items():
            # XXX: force datatype, needs typechecks, etc
            val = int(val)
            metrics.append((metric_base + '.' + _formatHostname(name) + '.' + key, val, timestamp))

    def send_textmetrics(metrics):
        ''' Use text protorocol to send metric over socket '''
        data = []
        for metric in metrics:
            metric = '{0} {1} {2}'.format(metric[0], metric[1], metric[2])
        data = '\n'.join(data)
        total_sent_bytes = 0
        while total_sent_bytes < len(data):
            sent_bytes = carbon_sock.send(data[total_sent_bytes:])
            if sent_bytes == 0: 
                log.error('Bytes sent 0, Connection reset?')
            logging.debug("Sent %d bytes to carbon" % sent_bytes)
            total_sent_bytes += sent_bytes

    # Send metrics

    # Shut down and close socket

Metric pushing

The carbon returner has a weak point in its design, it only supports one type of data, dicts within dicts, since that was all I had to support to get my virt module metrics into carbon. One thing it could be easily extended to do is parse output from all the existing munin plugins, or collected plugins. But lets not get ahead of ourselves, lets see if the runner works.

I decided on a simple 10 second timer for pushing data, that seemed granular enough for my purposes. Turns out salt got support for running scheduled command this weekend, and it will be featured in the upcoming 0.12-release. Eager to get my data scheduled I fetched latest git and wrote a quick runner to be scheduled on the master:


     function: carbonmon.pollpush
     seconds: 10

The runner

This runner just publishes the commands to be run in an async fashion. No data is returned to the master. I use a simple list to specify my virt hosts.


import salt.client
import sys

def pollpush():
    Run the monitoring command and return to carbon
    client = salt.client.LocalClient(__opts__['conf_file'])
    cmds = ('virt.vm_diskstats', 'virt.vm_netstats', 'virt.vm_cputime')
    nodes = 'chani,harkonnen,fremen'
    for cmd in cmds:
        jid = client.cmd_async(nodes, cmd, expr_form='list', ret='carbon', timeout=__opts__['timeout'])

et voilĂ ! metrics are now persisted by the graphite stack. And I can now return to my client and rewrite cubism client to use graphite backend.

The JavaScript

The graphite queries include wildcard support, so I could easily get graphs for rxbytes, txbytes, disk_counters and cputime using one simple function. If I want multicolored graphs per host I would have to make some changes.

var graphite_cubism = function(metric_name, metric_vals) {
    var seconds = 10;

    // Create cubism context
    context = cubism.context()
        .serverDelay(seconds * 1000) // allow 10 seconds of collection lag
        .step(seconds * 1000)         // 10 seconds per value
        .size(700);          // fetch 1080 values (pixels)
    graphite = context.graphite('/graphite/');
    // Search graphite for CPU metrics
    graphite.find('virt.'+metric_name+'.*.'+metric_vals, function(error, results) {

        var metrics = []; // list of metrics

        $.each(results, function(idx, metric) {
            var alias = metric.split('.')[2] + ' ' + metric.split('.').slice(-1);

        // Create the graphs and add to #graphs container
        d3.select("#graphs").call( function(div) {

          // An axis at the top
              .attr("class", "axis")

          // All the horizon graphs
              .attr("class", "horizon")

              .attr("class", "rule")


And now I have persistent metrics with a very simple way to visualize them in my VM monitor web front end.

Screenshot of the CPU usage monitor

CPU Usage Screenshot

Also read