TCollector

tcollector is a client-side process that gathers data from local collectors and pushes the data to OpenTSDB. You run it on all your hosts, and it does the work of sending each host's data to the TSD.

OpenTSDB is designed to make it easy to collect and write data to it. It has a simple protocol, simple enough for even a shell script to start sending data. However, to do so reliably and consistently is a bit harder. What do you do when your TSD server is down? How do you make sure your collectors stay running? This is where tcollector comes in.

Tcollector does several things for you:

• Runs all of your data collectors and gathers their data
• Does all of the connection management work of sending data to the TSD
• You don't have to embed all of this code in every collector you write
• Does de-duplication of repeated values
• Handles all of the wire protocol work for you, as well as future enhancements

Deduplication

Typically you want to gather data about everything in your system. This generates a lot of datapoints, the majority of which don't change very often over time (if ever). However, you want fine-grained resolution when they do change. Tcollector remembers the last value and timestamp that was sent for all of the time series for all of the collectors it manages. If the value doesn't change between sample intervals, it suppresses sending that datapoint. Once the value does change (or 10 minutes have passed), it sends the last suppressed value and timestamp, plus the current value and timestamp. In this way all of your graphs and such are correct. Deduplication typically reduces the number of datapoints TSD needs to collect by a large fraction. This reduces network load and storage in the backend. A future OpenTSDB release however will improve on the storage format by using RLE (among other things), making it essentially free to store repeated values.

Collecting lots of metrics with tcollector

Collectors in tcollector can be written in any language. They just need to be executable and output the data to stdout. Tcollector will handle the rest. The collectors are placed in the collectors directory. Tcollector iterates over every directory named with a number in that directory and runs all the collectors in each directory. If you name the directory 60, then tcollector will try to run every collector in that directory every 60 seconds. The shortest supported interval is 15 seconds, you should use a long-running collector in the 0 folder for intervals shorter than 15 seconds. TCollector will sleep for 15 seconds after each time it runs the collectors so intervals of 15 seconds are the only actually supported intervals. For example, this would allow you to run a collector every 15, 30, 45, 60, 75, or 90 seconds, but not 80 or 55 seconds. Use the directory 0 for any collectors that are long-lived and run continuously. Tcollector will read their output and respawn them if they die. Generally you want to write long-lived collectors since that has less overhead. OpenTSDB is designed to have lots of datapoints for each metric (for most metrics we send datapoints every 15 seconds).

If there any non-numeric named directories in the collectors directory, then they are ignored. We've included a lib and etc directory for library and config data used by all collectors.

Installation of tcollector

You need to clone tcollector from GitHub:

git clone git://github.com/OpenTSDB/tcollector.git

and edit 'tcollector/startstop' script to set following variable: TSD_HOST=dns.name.of.tsd

To avoid having to run mkmetric for every metric that tcollector tracks you can to start TSD with the --auto-metric flag. This is useful to get started quickly, but it's not recommended to keep this flag in the long term, to avoid accidental metric creation.