Inputs

An input is a source of data piped through an array of optional processors:

input:
  label: my_redis_input

  redis_streams:
    url: tcp://localhost:6379
    streams:
      - tyk_stream
    body_key: body
    consumer_group: tyk_group

  # Optional list of processing steps
  processors:
   - mapping: |
       root.document = this.without("links")
       root.link_count = this.links.length()       

Some inputs have a logical end ends once the last row is consumed, when this happens the input gracefully terminates and Tyk Streams will shut itself down once all messages have been processed fully.

Brokering

Only one input is configured at the root of a Tyk Streams config. However, the root input can be a broker which combines multiple inputs and merges the streams:

input:
  broker:
    inputs:
      - kafka:
          addresses: [ TODO ]
          topics: [ foo, bar ]
          consumer_group: foogroup

      - redis_streams:
          url: tcp://localhost:6379
          streams:
            - tyk_stream
          body_key: body
          consumer_group: tyk_group

Labels

Inputs have an optional field label that can uniquely identify them in observability data such as metrics and logs.

Generating Messages

It’s possible to generate data with Tyk Streams using the generate input, which is also a convenient way to trigger scheduled pipelines.