Tyk Pump Configuration

The Tyk Pump is our Open Source analytics purger that moves the data generated by your Tyk nodes to any back-end. By moving the analytics into MongoDB, it allows the Tyk Dashboard to display traffic analytics across all your gateways.

Supported Backends

MongoDB (Used by Tyk Dashboard)
CSV
Elasticsearch (2.0+)
Graylog
InfluxDB
Moesif
Splunk
StatsD
DogStatsD
Hybrid (Tyk RPC)
Prometheus
Logz.io
Kafka
Syslog (FluentD)

Configuration

Please visit the public GitHub Readme for instructions on installing and configuring the various Pump backends.

Tyk Dashboard

The Tyk Dashboard uses the mongo-pump-aggregate collection to display analytics. This is different than the standard mongo pump plugin that will store individual analytic items into MongoDB. The aggregate functionality was built to be fast, as querying raw analytics is expensive in large data sets. See Pump Dashboard Config for more details.

Capping analytics data

Tyk Gateways can generate a lot of analytics data. Be sure to read about capping your Dashboard analytics

Sharding analytics to different data sinks

In a multi-organisation deployment, each organisation, team, or environment might have their preferred analytics tooling. This capability allows Tyk-Pump to send analytics for different organisations or APIs to different places. E.g. Org A can send their analytics to MongoDB + DataDog. But Org B can send their analytics to DataDog + expose the Prometheus metrics endpoint.

It is also possible to put a blacklist in-place, meaning that some data sinks can receive information for all orgs, whereas other data sinks will not receive OrgA’s analytics if blacklisted.

Configuration

This feature adds a new configuration field in each pump called filters and its structure is the following:

"filters":{
  "api_ids":[],
  "org_ids":[],
  "skip_api_ids":[],
  "skip_org_ids":[]
     }

The fields api_ids and org_ids works as whitelists (APIs and orgs where we want to send the analytics records) and the fields skip_api_ids and skip_org_ids works as the opposite (blackslits).

The priority is always blacklisted configurations over whitelisted.

An example of configuration would be:

"csv": {
 "type": "csv",
 "filters": {
   "org_ids": ["org1","org2"]
 },
 "meta": {
   "csv_dir": "./bar"
 }
},
"elasticsearch": {
 "type": "elasticsearch",
 "filters": {
   "skip_api_ids": ["api_id_1"],
   },
 "meta": {
   "index_name": "tyk_analytics",
   "elasticsearch_url": "https://elasticurl:9243",
   "enable_sniffing": false,
   "document_type": "tyk_analytics",
   "rolling_index": false,
   "extended_stats": false,
   "version": "6"
 }
}

With this configuration, all the analytics records related to org1 or org2 will go to the csv backend and everything but analytics records from api_id_1 to elasticsearch.

Return to Top