CodeLab
1May/1017

Monitoring memcached with a Zenoss Zenpack

Zenoss is an open source enterprise-grade monitoring tool that quickly is becoming a standard among the monitoring applications.

This step-by-step guide shows how to monitor memcached instances using a custom Zenoss Zenpack. The Zenoss.community.Memcached zenpack extends Zenoss to provide a custom datasource type, MemcachedStats, which automatically brings to Zenoss all the sections of the memcached stats as datapoints.

I. Installing the Zenpack

The zenpack can be downloaded here.

To install the zenpack, log into your zenoss instance as an adminitrator and go to settings -> zenpacks -> add new zenpack. Then select the downloaded egg file and click ok.

I suggest restarting Zenoss before going to the next step.

II. Monitorig memcached

  1. Create a template to hold the datasources & graph definitions at Device level:

    Click on Devices under Classes on the left menu, followed by Templates tab, then on the drop down arrow to expand the menu and select Add Template

    Add Device Template

    Type the template name MemcachedStats in the popup div and click OK:

    Name the template MemcachedStats

    1. Add a MemcachedStats datasource

      In the template page displayed, set the description of the template. Then click the drop-down arrow to add a datasource:

      Add a datasource to the new template

      Then type in a name for the datasource: MemcachedStats and select MemcachedStats as the datasource type:

      Select MemcachedSt datasource type

      On the next page you can change the settings to match your memcached connection settings, typically you may need to change the port to connect to the appropiate instance. Here you will also see a list of all the datapoints provided by the MemcachedStats datasource:

      set memcached instance settings

    2. Define Graphs to visualize memcached's performance

      To finish the template, add graphs definitions using the datapoints provided by the above datasource. To do this, go back to /Devices/Templates/MemcachedStats and click on the drop-down arrow in the Graphs section, select Add Graph, give it a name and click OK:

      Add a graph to the template

      On the resulting Graph settings page, click the drop-down arrow next to Graph points, and click on Add Datapoint, select the points that you would like to appear in the current Graph and click OK:

      select your datapoints for the graph

      select your datapoints for the graph

  2. Bind the created template to the device where your memcached instance is running

    Once the performance template has been completed, the final step is to bind the template to the devices were the memcached instances are running to start collecting data. In my example memcached is running on localhost, default port 11211. So, let's go to /Devices/localhost, click the drop-down arrow next to Status and select More and Templates:

    edit localhost templates

    Select the drop-down arrow next to Performance Templates for localhost and click on Bind Templates option. Then make sure the MemcachedStats template is selected (Use Ctrl+click to select multiple templates):

    bind templates

    select templates

  3. View your performance data

    Now your setup is complete and Zenoss will start monitoring your memcached instance. Wait some time to allow the collector to fetch data several times and then go to the perf tab of the localhost device:

    Performance graph

III. Hints & Tips

  • If you have more than one memcached runing in the same server you can add further datasources to the template (1 datasource = 1 memcached instance / port).
  • Keep the timeout in the datasource config low to avoid the overall running of the zencommand timing out.
  • Group your datapoints by info type so your graphs are easier to read, or even better, create multigraph reports!
Comments (17) Trackbacks (0)
  1. getting an error installing this ZenPack on Zenoss 2.5.2: ERROR: zenpack command failed. Reason: OSError: [Errno 20] Not a directory: ‘/opt/zenoss/Products/ZenPacks.community.Memcached-1.0-py2.4.egg/skins’

    • Eric, I have just tested the zenpack installation in a fresh zenoss install (using the zenoss 2.5.2 rpm under centos in a VM) and all works fine. Did you install zenoss using the stack or an rpm? If you installed using the stack zenoss will likely go under /usr/local while the rpm tends go put it under /opt. Can you check the paths set for the zenoss user env?

      =====

      Output:
      zenpack –install /tmp/ZenPacks.community.Memcached-1.0-py2.4.egg
      2010-05-25 07:18:33,058 INFO zen.ZPLoader: Loading /opt/zenoss/ZenPacks/ZenPacks.community.Memcached-1.0-py2.4.egg/ZenPacks/community/Memcached/objects/objects.xml
      2010-05-25 07:18:33,789 INFO zen.AddToPack: End loading objects
      2010-05-25 07:18:33,797 INFO zen.AddToPack: Processing links
      2010-05-25 07:18:34,081 INFO zen.AddToPack: Loaded 1 objects into the ZODB database
      2010-05-25 07:18:34,310 INFO zen.HookReportLoader: loading reports from:/opt/zenoss/ZenPacks/ZenPacks.community.Memcached-1.0-py2.4.egg/ZenPacks/community/Memcached/reports
      Done installing ZenPack.

  2. uploading the egg uncompressed worked.

  3. Hello,

    I did everything in this guide but the command times out. Trying to issue the command from cli:

    # /usr/local/zenoss/zenoss/ZenPacks/ZenPacks.community.Memcached-1.0-py2.4.egg/ZenPacks/community/Memcached/libexec/check_memcached.py -H live-web1 -p 11211 -t 3

    Traceback (most recent call last):
    File \/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.community.Memcached-1.0-py2.4.egg/ZenPacks/community/Memcached/libexec/check_memcached.py\, line 12, in ?
    import Globals
    ImportError: No module named Globals

    Any help on this ?
    Thanks in advance :)

    • Angie, the zencommand is designed to be run under the zenoss user environment. So to test with different timeouts log in as a zenoss user and try again. Also might be worth trying with a simple telnet to test that your memcached server is visible from the zenoss box… Usually is best to have low (2 to 5s) timeouts.

  4. Followed the directions, but the graphs are being shown as broken image links.

    Tried running check_memcached.py from the command line on the zenoss server, and it returned the expected results. Any idea why these might not be getting passed to the graphs?

    • It usually takes a couple of collections in zenoss before the graphs are available. Should it be a problem with the zencommand you will get a CMD FAIL alert in your zenoss interface. You could also check that the RRD files under $ZENHOME/perf/your-device are being updated. Are you using distributed collectors? Check that your render server config is correct?

  5. I tried the command as zenoss user with 2 hosts: one behind VPN and the other is not.
    The command timed out on both.

    Here is my configs for memcached on the host WITHOUT vpn:
    ==========================================
    # cat /etc/sysconfig/memcached
    PORT=”11211″
    USER=”memcached”
    MAXCONN=”1024″
    CACHESIZE=”32″
    OPTIONS=”"

    # netstat -antpue | grep -i memcache
    tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 102 10613 2755/memcached
    udp 0 0 0.0.0.0:11211 0.0.0.0:* 102 10628 2755/memcached

    Is there anything special I can do to get the command NOT to time out? I am almost positive if the command runs from CLI as user zenoss, I’ll get graphs in 15 mins.

    Thanks

  6. I just did the telnet test from zenoss machine to the host without vpn and I am in!

    On zenoss machine:
    ==============
    $ telnet my.host.com 11211
    Trying 196.x.x.x…
    Connected to my.host.com (196.x.x.x).
    Escape character is ‘^]’.

    On memcache host:
    =============
    # netstat -antpue | grep -i memcache
    tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 102 10613 2755/memcached
    tcp 0 0 196.x.x.x:11211 41.x.x.x:55207 ESTABLISHED 102 137122 2755/memcached
    udp 0 0 0.0.0.0:11211 0.0.0.0:* 102 10628 2755/memcached

    If it helps, I just tried to execute the command with -t3, I’ll be raising timeout limits gradually and posting my results here tomorrow, because it’s too late here now.

    Thanks for the help.

  7. Finally it works!!!!
    :)

    Here is what happens:

    $ /usr/local/zenoss/zenoss/ZenPacks/ZenPacks.community.Memcached-1.0-py2.4.egg/ZenPacks/community/Memcached/libexec/check_memcached.py -H host-device-name-in-zenoss -p 11211 -t 3

    CMD FAIL| “Operation failed during connection to internal-host: (110, ‘Connection timed out’)”

    BUT

    $ /usr/local/zenoss/zenoss/ZenPacks/ZenPacks.community.Memcached-1.0-py2.4.egg/ZenPacks/community/Memcached/libexec/check_memcached.py -H 196.x.x.x -p 11211 -t 3

    CMD OK|hit_percent=0 usage_percent=0.0 uptime=179.0 time=1275436136.0 rusage_user=0.0 rusage_system=0.000999 curr_items=0.0 total_items=0.0 bytes=0.0 curr_connections=5.0 total_connections=7.0 connection_structures=6.0 cmd_flush=0.0 cmd_get=0.0 cmd_set=0.0 get_hits=0.0 get_misses=0.0 evictions=0.0 bytes_read=16.0 bytes_written=534.0 limit_maxbytes=33554432.0 threads=5.0 accepting_conns=1.0 listen_disabled_num=0.0 get_set_ratio=0 missed_percent=0

    so, it works when -H is supplied with IP or real hostname but not with device name used in zenoss. So, I edited /etc/hosts and added a record for the host with all possible names I may use including the name I used under zenoss as a device name.

    Now issuing the command with the -H works like a charm & I am getting graphs.

    Thanks for your patience reading my long posts :D .. Hopefully it benefits some one else.

    • Of course the hostname should be “DNS-resolvable”. On next version of the zenpack the zencommand will use the ipaddress by default to avoid this.

  8. It’s possible to monitor repcache with this Zenpack? theorically repcache it’s only a patched version of memcached but seems that something works different for repcache because don’t works with standard config. I’d checked the config of port and all seems right.

    • It will work provided that you can still telnet to the repcached port, issue the stats command and its output is identical to that of the memcached stats

  9. Hi brau, I’ve checked the output from check_memcached.py script and now I can see whats happening, the output:

    CMD FAIL| invalid literal for float(): MASTER

    Maybe the reason is that repcached have more info than memcached and the output from repcached have extra data that the memcached script doesn’t understand.

  10. Shouldn’t following stats be DERIVE instead of GAUGE?
    ‘cmd_get’, ‘cmd_set’, ‘bytes_read’, ‘bytes_written’, ‘get_hits’, ‘get_misses’

    Otherwise, Zenoss draws accumulation of those values instead of rate at each interval.


Leave a comment


 

No trackbacks yet.