> ## Documentation Index > Fetch the complete documentation index at: https://tyk.io/docs/llms.txt > Use this file to discover all available pages before exploring further. # Using external Key Value storage with Tyk > Learn how to use external Key-Value storage with Tyk Gateway, including Consul and Vault, to manage configuration data and secrets. ## Introduction With Tyk Gateway you can store configuration data (typically authentication secrets or upstream server locations) in Key-Value (KV) systems such as [Vault](#vault), [Consul](#consul), and [local files](#local-files) and then reference these values during configuration of the Tyk Gateway or APIs deployed on the Gateway. ## When to use external Key-Value storage ### Simplify migration of APIs between environments Easily manage and update secrets and other configuration across multiple environments (e.g., development, staging, production) without modifying the configuration files. ### Ensure separation of concerns Securely store sensitive information such as API keys, passwords, and certificates in a centralized location. Not everybody needs access to these secrets: authorized people can maintain them and just pass along the reference used to access them from the KV store. ### Support per-machine variables Storing local settings within the Tyk Gateway's configuration file allows you to have per instance variables, such as a machine ID, and inject these into API requests and responses using [transformation middleware](/api-management/traffic-transformation). ## How external Key-Value storage works There are two parts to external Key-Value storage - the KV store and the Tyk configuration object (API definition or Tyk Gateway config file). 1. The key-value data that you wish to reference should be added to the storage 2. References should be included within the Tyk configuration object that identify the location (KV store) and Key 3. When Tyk Gateway initialises it will resolve any external KV references in its configuration, retrieving and applying the values from those references 4. When Tyk Gateway loads (or reloads) APIs it will resolve any external KV references in the API definitions, retrieving and applying the values from those references KV references in API definitions are retrieved when the API is loaded or reloaded. Changes to the externally stored value are picked up on the next gateway hot-reload. KV references in the Tyk Gateway configuration file (`tyk.conf`) are retrieved at gateway startup only. A full gateway restart is required to pick up changes to those values. The exception is `external_services.oauth.mtls.cert_file`, `external_services.oauth.mtls.key_file`, and `external_services.oauth.mtls.ca_file`, which are resolved again on each hot-reload. The exception to both of the above is for specific [transformation middleware](#transformation-middleware) where the value will be retrieved for each call to the API, during the processing of the API request or response. If a KV reference cannot be resolved at load time (for example, because the KV store is unreachable or the file cannot be read), Tyk Gateway logs an error and continues to load. The unresolved reference string (such as `consul://mykey` or `file://path/to/file`) remains in place as the literal value of the field. For [transformation middleware](#transformation-middleware), if a `$secret_*` reference cannot be resolved at request time, Tyk Gateway replaces it with an empty string and continues to process the request. ## Supported storage options Tyk Gateway supports the following locations for storage of key-value data, providing flexibility to choose the most suitable approach for your deployment and the data you are storing: ### Consul HashiCorp [Consul](https://www.consul.io) is a service networking solution that is used to connect and configure applications across dynamic, distributed infrastructure. Consul KV is a simple Key-Value store provided as a core feature of Consul that can be used to store and retrieve Tyk Gateway configuration across multiple data centers. * to retrieve the value assigned to a `KEY` in Consul you will use `consul://KEY` or `$secret_consul.KEY` notation depending on the [location](#how-to-access-the-externally-stored-data) of the reference. ### Vault [Vault](https://vaultproject.io) from Hashicorp is a tool for securely accessing secrets. It provides a unified interface to any secret while providing tight access control and recording a detailed audit log. Tyk Gateway can use Vault to manage and retrieve sensitive secrets such as API keys and passwords. * to retrieve the value assigned to a `KEY` in Vault you will use `vault://KEY` or `$secret_vault.KEY` notation depending on the [location](#how-to-access-the-externally-stored-data) of the reference. ### Tyk Gateway Config File The [`secrets`](/tyk-oss-gateway/configuration#secrets) section in the Tyk Gateway configuration file allows you to store settings that are specific to a single Tyk Gateway instance. This is useful for storing instance-specific configuration to be injected into API middleware or if you prefer using configuration files. * to retrieve the value assigned to a `KEY` in the `secrets` config you will use `secrets://KEY` or `$secret_conf.KEY` notation depending on the [location](/tyk-oss-gateway/configuration#secrets) of the reference. ### Environment Variables Tyk Gateway can access data declared in environment variables. This is a simple and straightforward way to manage secrets, especially in containerized environments such as Docker or Kubernetes. * if you want to set the local "secrets" section (equivalent to the `secrets` section in the [Gateway config file](#how-to-access-the-externally-stored-data)) using environment variables, you should use the following notation: `TYK_GW_SECRETS=key:value,key2:value2`. * if you’re using a different key value secret store not explicitly supported by Tyk but can map it to `TYK_GW_SECRETS`, this will allow you to access those KV data. * to retrieve the value assigned to an environment variable `VAR_NAME` you will use `env://VAR_NAME` or `$secret_env.VAR_NAME` notation depending on the [location](#how-to-access-the-externally-stored-data) of the reference. ### Local Files From **Tyk Gateway 5.14.0**, KV values can be read directly from files on the local filesystem. This is particularly useful in Kubernetes deployments where secrets are mounted as files on the data plane pod (for example, via a [Kubernetes secret volume](https://kubernetes.io/docs/concepts/configuration/secret/)). The Control Plane never needs to store or transmit the secret material. * to retrieve the contents of a file at relative path `PATH` you will use `file://PATH` or `$secret_file.PATH` notation depending on the [location](#how-to-access-the-externally-stored-data) of the reference. * all file paths are relative to `kv.file.base_path` configured in `tyk.conf`; absolute paths in references are not permitted. ## How to access the externally stored data You can configure Tyk Gateway to retrieve values from KV stores in the following places: * Tyk Gateway configuration file (`tyk.conf`) * API definitions You can use keys from different KV stores (e.g. Consul and environment variables) in the same configuration object (Gateway config or API definition). ### From the Tyk Gateway configuration file In Tyk Gateway's configuration file (`tyk.conf`), you can retrieve values from KV stores for the following [fields](/tyk-oss-gateway/configuration): * `secret` * `node_secret` * `storage.password` * `cache_storage.password` * `security.private_certificate_encoding_secret` * `db_app_conf_options.connection_string` * `policies.policy_connection_string` * `slave_options.api_key` From Tyk 5.13.0 the following fields can also contain KV references. Unlike other Tyk Gateway config fields, these are resolved again on each hot-reload. * `external_services.oauth.mtls.cert_file` * `external_services.oauth.mtls.key_file` * `external_services.oauth.mtls.ca_file` To reference the *Value* assigned to a *Key* in one of the KV stores from the Gateway configuration file use the following notation: * Consul: `consul://path/to/key` * Vault: `vault://path/to/secret.key` * `tyk.conf` secrets: `secrets://key` * Environment variables: `env://key` * Local files: `file://relative/path/to/file` For example, if you create a Key-Value pair in [Vault](#vault) with the *key* `shared-secret` in *secret* `gateway-dashboard` within directory `tyk-secrets/` then you could use the *Value* as the `node_secret` in your Gateway config by including the following in your `tyk.conf` file: ```.json theme={null} { "node_secret":"vault://tyk-secrets/gateway-dashboard.shared-secret" } ``` When the Gateway starts, Tyk will read the *Value* from Vault and use this as the `node_secret`, which is used to [secure connection to the Tyk Dashboard](/tyk-oss-gateway/configuration#node_secret). Note that all of these references are read (and replaced with the values read from the KV location) on Gateway start when loading the `tyk.conf` file. ### From API Definitions From Tyk Gateway v5.3.0 onwards, you can store [any string field](#other-string-fields) from the API definition in any of the supported KV storage options; for earlier versions of Tyk Gateway only the [Target URL and Listen Path](#target-url-and-listen-path) fields and [certain transformation middleware](#transformation-middleware) configurations were supported. #### Target URL and Listen Path To reference the *Value* assigned to a *Key* in one of the KV stores for **Target URL** or **Listen Path** use the following notation: * Consul: `consul://path/to/key` * Vault: `vault://path/to/secret.key` * Tyk config secrets: `secrets://key` * Environment variables: `env://key` * Local files: `file://relative/path/to/file` These references are read (and replaced with the values read from the KV location) when the API is loaded to the Gateway (either when Gateway restarts or when there is a hot-reload). For example, if you define an environment variable (*Key*) `UPSTREAM_SERVER_URL` with the *Value* `http://httpbin.org/` then within your API definition you could use the *Value* for the Target URL for your Tyk OAS API as follows: ```json theme={null} { "x-tyk-api-gateway": { "upstream": { "url": "env://UPSTREAM_SERVER_URL" } } } ``` When the Gateway starts, Tyk will read the *Value* from the environment variable and use this as the [Target URL](/api-management/gateway-config-tyk-oas#upstream). Prior to Tyk Gateway v5.3.0, **environment variables** used for the Target URL or Listen Path had to be named `TYK_SECRET_{KEY_NAME}`. They were referred to in the API definition using `env://{KEY_NAME}` excluding the `TYK_SECRET_` prefix.

From v5.3.0 onward, environment variables can have any `KEY_NAME`, and the full name should be provided in the API definition reference. The pre-v5.3.0 naming convention is still supported for backward compatibility, but only for these two fields. #### Transformation Middleware Key-value references can be included in the following middleware, with the values retrieved dynamically when the middleware is called (during processing of an API request or response): * [request body transform](/api-management/traffic-transformation/request-body) * [request header transform](/api-management/traffic-transformation/request-headers) * [URL rewrite](/transform-traffic/url-rewriting) * [response body transform](/api-management/traffic-transformation/response-body) * [response header transform](/api-management/traffic-transformation/response-headers) To reference the *Value* assigned to a *Key* in one of the KV stores from these middleware use the following notation: * Consul: `$secret_consul.key` * Vault: `$secret_vault.key` * Tyk config secrets: `$secret_conf.key` * Environment variables: `$secret_env.key` or `env://key` (see [here](#using-environment-variables-with-transformation-middleware)) * Local files: `$secret_file.relative/path/to/file` This notation is used to avoid ambiguity in the middleware parsing (for example where a KV secret is used in a URL rewrite path). For example, if you create a Key-Value pair in Consul with the *Key* `user_id` then you could use the *Value* in the `rewriteTo` upstream address in the URL rewrite middleware for your Tyk OAS API by including the following in your API definition: ```json theme={null} { "x-tyk-api-gateway": { "middleware": { "operations": { "anythingget": { "urlRewrite": { "enabled": true, "pattern": ".*", "rewriteTo": "/api/v1/users/$secret_consul.user_id", } } } } } } ``` When a call is made to `GET /anything`, Tyk will retrieve the *Value* assigned to the `user_id` *Key* in Consul and rewrite the Target URL for the request to `/api/v1/users/{user_id}`. These references are read (and replaced with the values read from the KV location) during the processing of the API request or response. ##### Using Environment Variables with Transformation Middleware There are some subtleties with the use of environment variables as KV storage for the transformation middleware. * **Request and Response Body Transforms** support only the `$secret_env.{KEY_NAME}` format. * **Request and Response Header Transforms** support both `env://{KEY_NAME}` and `$secret_env.{KEY_NAME}` formats. * **URL Rewrite** supports the `env://{KEY_NAME}` format for both the `pattern` and `rewriteTo` fields. The `rewriteTo` field also supports `$secret_env.{KEY_NAME}` format. **Notes** Due to the way that Tyk Gateway processes the API definition, when you use transformation middleware with the `$secret_env` format, it expects the environment variable to be named `TYK_SECRET_{KEY_NAME}` and to be referenced from the API definition using `$secret_env.{KEY_NAME}`.

For example, if you create a Gateway environment variable `TYK_SECRET_NEW_UPSTREAM=http://new.upstream.com`, then in a request body transform middleware, you would reference it as follows: ```json theme={null} { "custom_url": "$secret_env.NEW_UPSTREAM" } ``` To configure the URL rewrite `rewriteTo` field using this variable you could use either: ```json theme={null} { "rewriteTo": "env://TYK_SECRET_NEW_UPSTREAM" } ``` or ```json theme={null} { "rewriteTo": "$secret_env.NEW_UPSTREAM" } ``` #### Other `string` fields To reference the *Value* assigned to a *Key* in one of the KV stores from the API Definition use the following notation: * Consul: `consul://key` * Vault: `vault://secret.key` * Tyk config secrets: `secrets://key` * Environment variables: `env://key` * Local files: `file://relative/path/to/file` These references are read (and replaced with the values read from the KV location) when the API is loaded to the Gateway (either when Gateway restarts or when there is a hot-reload). **Notes** When accessing KV references from the `/tyk-apis` directory on Consul or Vault, you should not provide the `path/to/` the key except for Target URL and Listen Path (as described [above](#target-url-and-listen-path)). For example, if you create a Key-Value pair in the `secrets` section of the `tyk.conf` file with the *Key* `auth_header_name`: ```json theme={null} { "secrets": { "auth_header_name": "Authorization" } } ``` Then within your API definition you could use the *Value* for the authentication header name as follows: ```json theme={null} { "x-tyk-api-gateway": { "components": { "securitySchemes": { "authToken": { "type": "apiKey", "in": "header", "name": "secrets://auth_header_name" } } } } } ``` When the Gateway starts, Tyk will read the *Value* from the `secrets` section in the Gateway config file and use this to identify the header where Tyk Gateway should look for the [Authentication](/api-management/gateway-config-tyk-oas#authentication) token in requests to your Tyk OAS API. ## Using Consul as a KV store HashiCorp [Consul](https://www.consul.io) is a service networking solution that is used to connect and configure applications across dynamic, distributed infrastructure. Consul KV is a simple Key-Value (KV) store provided as a core feature of Consul that can be used to store and retrieve Tyk Gateway configuration across multiple data centers. ### How to Configure Tyk to Access Consul Configuring Tyk Gateway to read values from Consul is straightforward - you simply configure the connection in your Tyk Gateway config file (`tyk.conf`) by adding the `kv` section as follows: ```json theme={null} { "kv": { "consul": { "address": "localhost:8025", "scheme": "http", "datacenter": "dc-1", "http_auth": { "username": "", "password": "" }, "wait_time": 10, "token": "", "tls_config": { "address": "", "ca_path": "", "ca_file": "", "cert_file": "", "key_file": "", "insecure_skip_verify": false } } } } ``` | Key | Description | | :---------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | | address | The location of the Consul server | | scheme | The URI scheme for the Consul server, e.g. `http` | | datacenter | Consul datacenter (agent) identifier | | http\_auth | Username and password for Tyk to log into Consul using HTTP Basic Auth (if required by your Consul service) | | wait\_time | Limits how long a [watch will block](https://developer.hashicorp.com/consul/api-docs/features/blocking) in milliseconds (if enabled in your Consul service) | | token | Used to provide a per-request access token to Consul (if required by your Consul service) | | tls\_config | Configuration for TLS connection to Consul (if enabled in your Consul service) | Alternatively, you can configure it using the equivalent [environment variables](/tyk-oss-gateway/configuration#kv-consul-address). ### Where to Store Data in Consul When you want to reference KV data from Tyk Gateway config or transform middleware, you can store your KV pairs wherever you like within the Consul KV store. You can provide the Consul path to the key in the reference using the notation appropriate to the calling [location](#consul). From Tyk Gateway 5.3.0, you can reference KV data from any `string` field in the API definition. For these you should create a folder named `tyk-apis` in the root of your Consul KV store and store all keys in a flat structure there (sub-directories not currently supported). You should not include the `tyk-apis` path in the reference so, for example, given a key-value pair `"foo":"bar"` stored in `tyk-apis` in Consul, you would reference this from the API definition using `consul://foo`. ### How to Access Data Stored in Consul The notation used to refer to a KV pair stored in Consul depends upon the location of the reference as follows. **Tyk Gateway configuration file** As described [here](#from-the-tyk-gateway-configuration-file), from Tyk Gateway's configuration file (`tyk.conf`) you can retrieve values from Consul using the following notation: * `consul://path/to/KEY` **API definition** The **Target URL** and **Listen Path** key-value pairs can be stored in any directory in the Consul KV store as they are accessed using a different mechanism than other fields in the API definition. If storing these in a sub-directory, you can retrieve the values from Consul by providing the directory path within Consul KV using the following notation: * `consul://path/to/KEY` For [certain transformation middleware](#transformation-middleware) because the secret resolution happens during the request context, a different notation is used to retrieve values from Consul: * `$secret_consul.KEY` From Tyk Gateway v5.3.0 onwards, you can store KV pairs to be used in **any `string` field** in the API definition in the Consul KV store. You can retrieve these values from Consul, noting that you do not provide the directory path (`/tyk-apis`) when accessing data for *these* fields, using the following notation: * `consul://KEY` ## Using Vault as a KV store [Vault](https://vaultproject.io) from Hashicorp is a tool for securely accessing secrets. It provides a unified interface to any secret while providing tight access control and recording a detailed audit log. Tyk Gateway can use Vault to manage and retrieve sensitive secrets such as API keys and passwords. ### How to Configure Tyk to Access Vault Configuring Tyk Gateway to read values from Vault is straightforward - you simply configure the connection in your Tyk Gateway config file (`tyk.conf`) by adding the `kv` section as follows: ```json theme={null} { "kv": { "vault": { "address": "http://localhost:1023", "agent_address": "", "max_retries": 3, "timeout": 30, "token": "", "kv_version": 2 } } } ``` | Key | Description | | :------------- | :-------------------------------------------------------------------------------------------------------------------------------------- | | address | The address of the Vault server, which must be a complete URL such as `http://www.vault.example.com` | | agent\_address | The address of the local Vault agent, if different from the Vault server, must be a complete URL | | max\_retries | The maximum number of attempts Tyk will make to retrieve the value if Vault returns an error | | timeout | The maximum time that Tyk will wait for a response from Vault (in nanoseconds, if set to 0 (default) will be interpreted as 60 seconds) | | token | The Vault root access token | | kv\_version | The version number of Vault, usually defaults to 2 | Alternatively, you can configure it using the equivalent [environment variables](/tyk-oss-gateway/configuration#kv-vault-token). ### How Key-Value Data Is Stored in Vault In traditional systems secrets are typically stored individually, each with their own unique key. Vault, however, allows for a more flexible approach where multiple *keys* can be grouped together and stored under a single *secret*. This grouping allows for better organization and management of related secrets, making it easier to retrieve and manage them collectively. When retrieving data from Vault, you use the dot notation (`secret.key`) to access the *value* from a specific *key* within a *secret*. **Example of storing key value data in Vault** If you want to store a *secret* named `tyk` with a *key* `gw` and *value* `123` in Vault then, from the command line, you would: 1. Enable the `kv` secrets engine in Vault under the path `my-secret` using:\ `vault secrets enable -version=2 -path=my-secret kv` 2. Create a secret `tyk` with the key `gw` and value `123` in Vault:\ `vault kv put my-secret/tyk gw=123` To retrieve the secret from Vault using the command line you would use the following command (there is no need to append `/data` to the secret path): ```curl theme={null} curl \ --header "X-Vault-Token: " \ --request GET \ https://vault-server.example.com/v1/my-secret/tyk?lease=true ``` This would return a response along these lines, note that the response contains all the keys stored in the secret (here there are also keys called `excited` and `foo`): ```yaml theme={null} { "request_id": "0c7e44e1-b71d-2102-5349-b5c60c13fb02", "lease_id": "", "lease_duration": 0, "renewable": false, "data": { "gw": "123", "excited": "yes", "foo": "world", }, "metadata":{ "created_time": "2019-08-28T14:18:44.477126Z", "deletion_time": "", "destroyed": false, "version": 1 }, "auth": ... } ``` As explained [below](#vault), you could retrieve this value from within your Tyk Gateway config file using: `TYK_GW_SECRET=vault://my-secret/tyk.gw` ### Where to Store Data in Vault When you want to reference KV data from Tyk Gateway config or transform middleware, you can store your Vault *secrets* wherever you like within the KV store. You can provide the Vault path to the key in the reference using the notation appropriate to the calling [location](#vault). From Tyk Gateway 5.3.0, you can reference KV data from any `string` field in the API definition. For these you should create a folder named `tyk-apis` in the root of your Vault KV store and store all *secrets* in a flat structure there (sub-directories not currently supported). You should not include the `tyk-apis` path in the reference so, for example, given a key-value pair `"foo":"bar"` stored in a secret named `my-secret` in `/tyk-apis` in Vault, you would reference this from the API definition using `vault://my-secret.foo`. ### How to Access Data Stored in Vault The notation used to refer to a key-value pair stored in Vault depends upon the location of the reference as follows. **Tyk Gateway configuration file** As described [here](#from-the-tyk-gateway-configuration-file), from Tyk Gateway's configuration file (`tyk.conf`) you can retrieve values from Vault using the following notation: * `vault://path/to/secret.KEY` **API definition** The **Target URL** and **Listen Path** key-value pairs can be stored in any directory in the Vault KV store as they are accessed using a different mechanism than other fields in the API definition. If storing these in a sub-directory, you can retrieve the values from Vault by providing the directory path within Consul KV using the following notation: * `vault://path/to/secret.KEY` For [certain transformation middleware](#transformation-middleware) because the secret resolution happens during the request context, a different notation is used to retrieve values from Vault: * `$secret_vault.KEY` From Tyk Gateway v5.3.0 onwards, you can store KV pairs to be used in **any `string` field** in the API definition in the Vault KV store. You can retrieve these values from Vault, noting that you do not provide the directory path (`/tyk-apis`) when accessing data for *these* fields, using the following notation: * `vault://KEY` ## Using Local Files as a KV Store From Tyk Gateway 5.14.0, Tyk Gateway can resolve KV references to the contents of files on the local filesystem. This is designed for Kubernetes deployments where secrets (such as TLS certificates or API keys) are mounted as files on the data plane pod via a Kubernetes secret volume. The Control Plane never needs to store or transmit the secret material. ### How to Configure Tyk to Access Local Files You must configure a `base_path` in the `kv.file` section of your Tyk Gateway config file (`tyk.conf`): ```json theme={null} { "kv": { "file": { "base_path": "/etc/tyk/secrets" } } } ``` | Key | Description | | :---------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `base_path` | The absolute path to the directory from which file references are resolved. This field is required to enable file-based KV references; if not set, all `file://` and `$secret_file.` references are rejected. | `base_path` must be an absolute path. File references are resolved relative to this directory. Tyk Gateway enforces the following security constraints: * Absolute paths in references are rejected. * Path traversal using `..` is rejected. * Symlinks are followed, but only if the resolved path remains within `base_path`. This allows Kubernetes AtomicWriter symlinks (such as `..data`) to work correctly while preventing symlink escape attacks. ### How to Access Data Stored in Local Files The notation used to refer to a file depends on the location of the reference. **Tyk Gateway configuration file** From Tyk Gateway's configuration file (`tyk.conf`) you can retrieve the contents of a file using the following notation: * `file://relative/path/to/file` **API definition** For **Target URL**, **Listen Path**, and any other `string` field in the API definition: * `file://relative/path/to/file` For [transformation middleware](#transformation-middleware), where resolution happens at request time: * `$secret_file.relative/path/to/file` All paths are relative to `kv.file.base_path`. ### Content Handling Trailing newlines are stripped from file contents. This matches the behavior of Kubernetes secret mounts and avoids injecting unexpected whitespace into configuration values. Files containing multi-line content such as PEM certificates are returned intact except for the trailing newline. ### Kubernetes Secret Mounts When Kubernetes mounts a secret as a volume, it uses an AtomicWriter pattern that creates a `..data` symlink pointing to a timestamped directory. Tyk Gateway resolves these symlinks before reading the file, so secret rotations are picked up on the next gateway hot-reload without a restart. A common use case for file-based KV references is storing TLS certificates in Kubernetes secrets and referencing them directly in API definitions. From Tyk Gateway 5.14.0, certificate fields in API definitions accept inline PEM content, which means a `file://` reference can supply a certificate mounted from a Kubernetes secret without the Control Plane ever storing the private key. See [Certificate Management](/api-management/certificates#certificate-management) for details.