/api/query/exp

This endpoint allows for querying data using expressions. The query is broken up into different sections.

Two set operations (or Joins) are allowed. The union of all time series ore the intersection.

For example we can compute "a + b" with a group by on the host field. Both metrics queried alone would emit a time series per host, e.g. maybe one for "web01", "web02" and "web03". Lets say metric "a" has values for all 3 hosts but metric "b" is missing "web03".

With the intersection operator, the expression will effectively add "a.web01 + b.web01" and "a.web02 + b.web02" but will skip emitting anything for "web03". Be aware of this if you see fewer outputs that you expected or you see errors about no series available after intersection.

With the union operator the expression will add the web01 and web02 series but for metric "b", it will substitute the metric's fill policy value for the results.

Note

Supported as of version 2.3

Verbs

POST

Requests

The various sections implemented include:

"time"

The time section is required and is a single JSON object. This affects the time range and optional reductions for all metrics requested.

Name	Data Type	Required	Description	Default	Example
start	Integer	Required	The start time for the query. This may be relative, absolute human readable or absolute Unix Epoch.		1h-ago, 2015/05/05-00:00:00
aggregator	String	Required	The global aggregation function to use for all metrics. It may be overridden on a per metric basis.		sum
end	Integer	Optional	The end time for the query. If left out, the end is now	now	1h-ago, 2015/05/05-00:00:00
downsampler	Object	Optional	Reduces the number of data points returned. The format is defined below	None	See below
rate	Boolean	Optional	Whether or not to calculate all metrics as rates, i.e. value per second. This is computed before expressions.	false	true

E.g.

"time":{ "start":"1h-ago", "end":"10m-ago", "downsampler":{"interval":"15m","aggregator":"max"}

Downsampler

Name	Data Type	Required	Description	Default	Example
interval	String	Required	A downsampling interval, i.e. what time span to rollup raw values into. The format is `<#><unit>`, e.g. `15m`		1h
aggregator	String	Required	The aggregation function to use for reducing the data points		avg
fillPolicy	Object	Optional	A policy to use for filling buckets that are missing data points	None	See Below

Fill Policies

These are used to replace "missing" values, i.e. when a data point was expected but couldn't be found in storage.

Name	Data Type	Required	Description	Default	Example
policy	String	Required	The name of a policy to use. The values are listed in the table below		zero
value	Double	Optional	For scalar fills, an optional value that can be used during substitution	NaN	42

Name	Description
nan	Emits a NaN if all values in the aggregation function were NaN or "missing". For aggregators, NaNs are treated as "sentinel" values that cause the function to skip over the values. Note that if a series emits a NaN in an expression, the NaN is infectious and will cause the output of that expression to be NaN. At serialization the NaN will be emitted.
null	Emits a Null at serialization time. During computation the values are treated as NaNs.
zero	Emits a zero when the value is missing
scalar	Emits a user defined value when a data point is missing. Must specify the value with `value`. The value can be an integer or floating point.

Note that if you try to supply a value that is incompatible with the type the query will throw an exception. E.g. supplying a value with the NaN that isn't NaN will throw an error.

E.g.

{"policy":"scalar","value":"1"}

"filters"

Filters are for selecting various time series based on the tag keys and values. At least one filter must be specified (for now) with at least an aggregation function supplied. Fields include:

Name	Data Type	Required	Description	Default	Example
id	String	Required	A unique ID for the filter. Cannot be the same as any metric or expression ID		f1
tags	Array	Optional	A list of filters on tag values	None	See below

E.g.

"filters":[
  "id":"f1",
  "tags":[
  {
    "type":"wildcard",
    "tagk":"host",
    "filter":"*",
    "groupBy":true
  },
  {
    "type":"literal_or",
    "tagk":"colo",
    "filter":"lga",
    "groupBy":false
  }
   ]
  ]

Filter Fields

Within the "tags" field you can have one or more filter. The list of filters can be found via the /api/config/filters endpoint.

Name	Data Type	Required	Description	Default	Example
type	String	Required	The name of the filter from the API		regexp
tagk	String	Required	The tag key name such as host or colo that we filter on		host
filter	String	Required	The value to filter on. This depends on the filter in use. See the API for details		web.*mysite.com
groupBy	Boolean	Optional	Whether or not to group results by the tag values matching this filter. E.g. grouping by host will return one result per host. Not grouping by host would aggregate (using the aggregation function) all results for the metric into one series	false	true

"metrics"

The metrics list determines which metrics are included in the expression. There must be at least one metric.

Name	Data Type	Required	Description	Default	Example
id	String	Required	A unique ID for the metric. This MUST be a simple string, no punctuation or spaces		cpunice
filter	String	Required	The filter to use when fetching this metric. It must match a filter in the filters array		f1
metric	String	Required	The name of a metric in OpenTSDB		system.cpu.nice
aggregator	String	Optional	An optional aggregation function to overload the global function in `time` for just this metric	`time`'s aggregator	count
fillPolicy	Object	Optional	If downsampling is not used, this can be included to determine what to emit in calculations. It will also override the downsampling policy	zero fill	See above

E.g.

{"id":"cpunice", "filter":"f1", "metric":"system.cpu.nice"}

"expressions"

A list of one or more expressions over the metrics. The variables in an expression MUST refer to either a metric ID field or an expression ID field. Nested expressions are supported but exceptions will be thrown if a self reference or circular dependency is detected. So far only basic operations are supported such as addition, subtraction, multiplication, division, modulo

Name	Data Type	Required	Description	Default	Example
id	String	Required	A unique ID for the expression		cpubusy
expr	String	Required	The expression to execute		a + b / 1024
join	Object	Optional	The set operation or "join" to perform for series across sets.	union	See below
fillPolicy	Object	Optional	An optional fill policy for the expression when it is used in a nested expression and doesn't have a value	NaN	See above

E.g.

{
  "id": "cpubusy",
  "expr": "(((a + b + c + d + e + f + g) - g) / (a + b + c + d + e + f + g)) * 100",
  "join": {
    "operator": "intersection",
    "useQueryTags": true,
    "includeAggTags": false
  }
}

Joins

The join object controls how the various time series for a given metric are merged within an expression. The two basic operations supported at this time are the union and intersection operators. Additional flags control join behavior.

Name	Data Type	Required	Description	Default	Example
operator	String	Required	The operator to use, either union or intersection		intersection
useQueryTags	Boolean	Optional	Whether or not to use just the tags explicitly defined in the filters when computing the join keys	false	true
includeAggTags	Boolean	Optional	Whether or not to include the tag keys that were aggregated out of a series in the join key	true	false

"outputs"

These determine the output behavior and allow you to eliminate some expressions from the results or include the raw metrics. By default, if this section is missing, all expressions and only the expressions will be serialized. The field is a list of one or more output objects. More fields will be added later with flags to affect the output.

Name	Data Type	Required	Description	Default	Example
id	String	Required	The ID of the metric or expression		e
alias	String	Optional	An optional descriptive name for series		System Busy

E.g.

{"id":"e", "alias":"System Busy"}

Note

The id field for all objects can not contain spaces, special characters or periods at this time.

Complete Example

{
   "time": {
     "start": "1y-ago",
     "aggregator":"sum"
   },
   "filters": [
     {
       "tags": [
         {
           "type": "wildcard",
           "tagk": "host",
           "filter": "web*",
           "groupBy": true
         }
       ],
       "id": "f1"
     }
   ],
   "metrics": [
     {
       "id": "a",
       "metric": "sys.cpu.user",
       "filter": "f1",
       "fillPolicy":{"policy":"nan"}
     },
     {
       "id": "b",
       "metric": "sys.cpu.iowait",
       "filter": "f1",
       "fillPolicy":{"policy":"nan"}
     }
   ],
   "expressions": [
     {
       "id": "e",
       "expr": "a + b"
     },
     {
     "id":"e2",
     "expr": "e * 2"
     },
     {
     "id":"e3",
     "expr": "e2 * 2"
     },
     {
     "id":"e4",
     "expr": "e3 * 2"
     },
     {
     "id":"e5",
     "expr": "e4 + e2"
     }
  ],
  "outputs":[
    {"id":"e5", "alias":"Mega expression"},
    {"id":"a", "alias":"CPU User"}
  ]
 }

Response

The output will contain a list of objects in the outputs array with the results in an array of arrays representing each time series followed by meta data for each series and the query overall. Also included is the original query and some summary statistics. The fields include:

Name	Description
id	The expression ID the output matches
dps	The array of results. Each sub array starts with the timestamp in ms as the first (offset 0) value. The remaining values are the results for each series when a group by was applied.
dpsMeta	Meta data around the query including the first and last timestamps, number of result "sets", or sub arrays, and the number of series represented.
datapoints	The total number of data points returned to the user after aggregation
meta	Data about each time series in the result set. The fields are below

The meta section contains ordered information about each time series in the output arrays. The first element in the array will always have a metrics value of timestamp and no other data.

Name	Description
index	The index in the data point arrays that the meta refers to
metrics	The different metric names included in the expression
commonTags	Tag keys and values that were common across all time series that were aggregated in the resulting series
aggregatedTags	Tag keys that appeared in all series in the resulting series but had different values
dps	The number of data points emitted
rawDps	The number of raw values wrapped into the result

Example Responses

{
  "outputs": [
    {
      "id": "Mega expression",
      "dps": [
        [
          1431561600000,
          1010,
          1030
        ],
        [
          1431561660000,
          "NaN",
          "NaN"
        ],
        [
          1431561720000,
          "NaN",
          "NaN"
        ],
        [
          1431561780000,
          1120,
          1140
        ]
      ],
      "dpsMeta": {
        "firstTimestamp": 1431561600000,
        "lastTimestamp": 1431561780000,
        "setCount": 4,
        "series": 2
      },
      "meta": [
        {
          "index": 0,
          "metrics": [
            "timestamp"
          ]
        },
        {
          "index": 1,
          "metrics": [
            "sys.cpu",
            "sys.iowait"
          ],
          "commonTags": {
            "host": "web01"
          },
          "aggregatedTags": []
        },
        {
          "index": 2,
          "metrics": [
            "sys.cpu",
            "sys.iowait"
          ],
          "commonTags": {
            "host": "web02"
          },
          "aggregatedTags": []
        }
      ]
    },
    {
      "id": "sys.cpu",
      "dps": [
        [
          1431561600000,
          1,
          2
        ],
        [
          1431561660000,
          3,
          0
        ],
        [
          1431561720000,
          5,
          0
        ],
        [
          1431561780000,
          7,
          8
        ]
      ],
      "dpsMeta": {
        "firstTimestamp": 1431561600000,
        "lastTimestamp": 1431561780000,
        "setCount": 4,
        "series": 2
      },
      "meta": [
        {
          "index": 0,
          "metrics": [
            "timestamp"
          ]
        },
        {
          "index": 1,
          "metrics": [
            "sys.cpu"
          ],
          "commonTags": {
            "host": "web01"
          },
          "aggregatedTags": []
        },
        {
          "index": 2,
          "metrics": [
            "sys.cpu"
          ],
          "commonTags": {
            "host": "web02"
          },
          "aggregatedTags": []
        }
      ]
    }
  ],
  "statsSummary": {
    "datapoints": 0,
    "rawDatapoints": 0,
    "aggregationTime": 0,
    "serializationTime": 33,
    "storageTime": 77,
    "timeTotal": 148.63
  },
  "query": {
    "name": null,
    "time": {
      "start": "1y-ago",
      "end": null,
      "timezone": null,
      "downsampler": null,
      "aggregator": "sum"
    },
    "filters": [
      {
        "id": "f1",
        "tags": [
          {
            "tagk": "host",
            "filter": "web*",
            "group_by": true,
            "type": "wildcard"
          }
        ]
      }
    ],
    "metrics": [
      {
        "metric": "sys.cpu",
        "id": "a",
        "filter": "f1",
        "aggregator": null,
        "fillPolicy": {
          "policy": "nan",
          "value": "NaN"
        },
        "timeOffset": null
      },
      {
        "metric": "sys.iowait",
        "id": "b",
        "filter": "f1",
        "aggregator": null,
        "fillPolicy": {
          "policy": "nan",
          "value": "NaN"
        },
        "timeOffset": null
      }
    ],
    "expressions": [
      {
        "id": "e",
        "expr": "a + b"
      },
      {
        "id": "e2",
        "expr": "e * 2"
      },
      {
        "id": "e3",
        "expr": "e2 * 2"
      },
      {
        "id": "e4",
        "expr": "e3 * 2"
      },
      {
        "id": "e5",
        "expr": "e4 + e2"
      }
    ],
    "outputs": [
      {
        "id": "e5",
        "alias": "Woot!"
      },
      {
        "id": "a",
        "alias": "Woot!2"
      }
    ]
   }
}

© 2010–2016 The OpenTSDB Authors
Licensed under the GNU LGPLv2.1+ and GPLv3+ licenses.
http://opentsdb.net/docs/build/html/api_http/query/exp.html