Overview

Last updated: Jun 11, 2024 2 minutes read.

Tyk streams configuration is specified using YAML. The configuration consists of several main sections: input, pipeline, output and optionally resources, logger, and metrics.

Input

The input section defines the publisher source of the data stream. Tyk Streams supports various input types such as Kafka, HTTP, MQTT etc. Each input type has specific configuration parameters.

input:
  kafka:
    addresses:
      - localhost:9092
    topics:
      - example_topic
    consumer_group: example_group
    client_id: example_client

Pipeline

The pipeline section defines the processing steps applied to the data. It includes processors for filtering, mapping, enriching and transforming the data. Processors can be chained together.

pipeline:
  processors:
    - bloblang: |
        root = this
        root.foo = this.bar.uppercase()        
    - json_schema:
        schema_path: "./schemas/example_schema.json"

Output

The output section specifies the destination of the processed data. Similar to inputs, Tyk Streams supports various output types like Kafka, HTTP etc.

output:
  kafka:
    addresses:
      - localhost:9092
    topic: output_topic
    client_id: example_output_client

Resources (Optional)

The resources section allows you to define shared resources such as caches, rate limits and conditions that can be used across inputs, outputs and processors.

resources:
  caches:
    my_cache:
      memcached:
        addresses:
          - localhost:11211
  rate_limits:
    my_rate_limit:
      local:
        count: 1000
        interval: 1s

Logger (Optional)

The logger section is used to configure logging options, such as log level and output format.

logger:
  level: INFO
  format: json

Metrics (Optional)

The metrics section allows you to configure how metrics are collected and reported, supporting various backends like Prometheus, StatsD and Graphite.

metrics:
  prometheus:
    prefix: tyk
    listen_address: ":8080"

A complete example combining all the sections is given below:

input:
  kafka:
    addresses:
      - localhost:9092
    topics:
      - example_topic
    consumer_group: example_group
    client_id: example_client

pipeline:
  processors:
    - bloblang: |
        root = this
        root.foo = this.bar.uppercase()        
    - text:
        operator: trim
    - bloblang: |
                root.processed = this.foo.contains("example")

output:
  kafka:
    addresses:
      - localhost:9092
    topic: output_topic
    client_id: example_output_client

resources:
  caches:
    my_cache:
      memcached:
        addresses:
          - localhost:11211
  rate_limits:
    my_rate_limit:
      local:
        count: 1000
        interval: 1s

logger:
  level: INFO
  format: json

metrics:
  prometheus:
    prefix: tyk
    listen_address: ":8080"

This overview provides a foundational understanding of how to configure Tyk Streams for various streaming and processing tasks. Each section can be customized to fit specific use cases and deployment environments.