We get asked, and quite often, how to transition an API configuration from one system to another, especially in the world of continuous deployment and integration. We’re finding more and more cases where APIs need to be migrated from a dev environment, through a test pipeline and into a pre-production or production environment.
The thing is, the further away you get from the dev environment, the less reliant you want to be on people to implement the thing that’s been built. It becomes a real pain to manage and you have to deal with human error.
We’ve experimented with improving this process by creating a Tyk integration with Git for a future release, and because we like to run proof of concepts before we actually build larger features, we created a small tool that integrates with the Tyk APIs and Git.
We gave this tool the extremely imaginative name: ‘Tyk-Git‘, and this article outlines how to use Tyk and Git to source control your APIs.
Warning: Tyk-Git is in an early stage of development; we know it works, but you use it at your own risk!
Source control: usually a bit of a PITA
Tyk has a really fancy dashboard, and a great API designer that lets you configure, tweak and model your services, but if you want to move your API from your Dev to QA environment, you need to export the API Definition, or use the Dashboard REST API to move things around.
Assuming you’ve pulled all the relevant data, you also need to store it and then deploy it to the next system. This is fine for new services, but what about modified ones? For example, what if your team has finished v3 of your API, and you’ve added it to your dev gateway, but you need to transfer that change across to the other environments? Doing it manually is possible, but probably not a great idea.
Again, you could export and import the data using the REST API (that’s what it’s there for after all), but there is still some massaging to do to the data to get things just right (like aligning API IDs, Organisation IDs and Policy IDs).
And throughout all this wrangling, you would be taking your configuration files and storing them in a version control system like Git, where you can move everything you need to, and with a single process.
So basically, in order for you to remove human error, store your API gateway settings alongside your services and other infrastructure configs, and migrate configurations between environments, you need to jump through quite a few hoops…
Enter Tyk-Git
Reminder: Tyk-Git is in an early stage of development; we know it works, but you use it at your own risk!
The Tyk-Git tool is very simple to use – we’ll walk through a use case here. Let’s say you want to move your configuration from your dev environment to your integration environment. To do this, you would:
martin@tyk-git>./tyk-git dump --dashboard="https://localhost:3000" --secret="b2d420ca5302442b6f20100f76de7d83" --target="./tmp" Extracting APIs and Policies from https://localhost:3000 > Fetching policies --> Identified 1 policies --> Fetching and cleaning policy objects > Fetching APIs --> Fetched 3 APIs > Creating spec file in: tmp/.tyk.json Done.
Here we’ve told Tyk-Git to dump the API Definitions and the Policies from the dashboard located at https://localhost:3000 using our API Key (secret). We want this data to be dumped in the ./tmp directory.
If you look at the tmp directory, it will now look like this:
martin@tyk-git> ls -al tmp total 18592 drwxrwxrwx 0 root root 512 Aug 15 10:30 ./ drwxrwxrwx 0 root root 512 Aug 15 15:12 ../ -rwxrwxrwx 1 root root 4481 Aug 15 14:20 api-99991.json* -rwxrwxrwx 1 root root 4481 Aug 15 14:20 api-99992.json* -rwxrwxrwx 1 root root 4490 Aug 15 14:20 api-99993.json* -rwxrwxrwx 1 root root 643 Aug 15 14:20 policy-5925078a97f43e086720ce38.json* -rwxrwxrwx 1 root root 308 Aug 15 14:20 .tyk.json*
(ignore the file flags here, this demo was written on a windows box with the linux subsystem)
What you can see here are three APIs and one policy. You’ll notice that the APIs have odd ID’s, they are really short: 9991, 9992, and 9993.
The reason for this is because the Tyk Dashboard (On-Premises) actually supports arbitrary API IDs. If you look at the file contents for an API Definition, you will see a database ID *and* an API ID, it is the latter that is transportable:
martin@tyk-git> head -10 tmp/api-99991.json { "api_definition": { "id": "598ec94f9695f201730d835b", "name": "Dash 1", "slug": "dash-1", "api_id": "99991", "org_id": "591ea6b10668f001852ae77a", "use_keyless": false, "use_oauth2": false,
The Tyk Gateway will operate only on the API ID, not the database ID. The same thing applies to policies, though it isn’t quite as clear cut.
With a policy, there are two ID objects:
martin@tyk-git> head -3 tmp/policy-5925078a97f43e086720ce38.json { "_id": "5990cf459695f201730d836d", "id": "5925078a97f43e086720ce38",
In Tyk, we call this ID masking, and it is a bit of a sleight of hand to make policy-IDs portable. Back when the portal and policies were first created, they were a dashboard-only construct, and so weren’t designed with immutable IDs in mind. So we introduced ID masking, where the masked id is replaced for the DB ID if it is present, but it is not always present.
However, with Tyk-Git, you don’t need to worry about that, because it will make your policy ID immutable by default (i.e. if the policy is dumped, and does not have a masked ID, Tyk-Git will make sure it does by copying the existing ID over).
This is important, because it means that once these files are created you can use them in other environments safely.
When I say safely, I mean that if you are modifying an existing, live policy, and then migrating it through the flow, you do not want the ID to vanish, because it could affect all keys that make use of it. The immutable ID enforcement mean that you can retain your configuration and make it fully portable between systems.
Restoring and synchronising with your other dashboard
So you now have your files, your tooling and your git repo, you are ready to deploy to your production environment. To go about doing that is surprisingly simple:
martin@tyk-git> ./tyk-git sync --dashboard="https://localhost:3001" --secret="b2d420ca5302442b6f20100f76de7d83" --branch="refs/heads/prod" https://github.com/myname/my-apis.git Using publisher: Dashboard Publisher Fetched 3 definitions Fetched 1 policies Processing APIs... Deleting: 0 Updating: 3 Creating: 0 SYNC Updating: 598ec94f9695f201730d835b SYNC Updating: 598ec9589695f201730d835c SYNC Updating: 5990cfee9695f201730d836e Processing Policies... Deleting policies: 0 Updating policies: 1 Creating policies: 0 SYNC Updating Policy: Test policy 1
There’s quite a lot going on here, so let’s break it down step by step:
./tyk-git sync
This invokes the synchronisation mode. This is a one-way push from the repository to your dashboard, it will:
- Remove APIs and Policies that are not in Git
- Update APIs and Policies that are in Git
- Create APIs and Policies that are in Git but not found in the dashboard
–dashboard=”https://localhost:3001″
Target a dashboard. You can also target a gateway, but this tool is probably most useful for dashboard users.
–secret=”b2d420ca5302442b6f20100f76de7d83″
The secret to use with your dashboard. If you do not want to use a flag, you can actually also set an environment variable: TYKGIT_DB_SECRET.
–branch=”refs/heads/prod”
If you are pipelining your builds, you might have release branches, with Tyk-Git you can target the specific branch using this flag.
https://github.com/myname/my-apis.git
Your repository address, pretty self explanatory stuff.
If we examine the output of the command, we can see a few things have happened:
- It fetched 3 API Definitions from our git repo
- It fetched 1 policy
- When syncing with the dashboard, it actually updated 3 APIs, (i.e. they were already present on the system, and therefore needed updating, not creating)
- It also only updated an existing policy, again, the policy was found, and changed as opposed to created
- No policies or API Definitions on the dashboard needed to be deleted in order to satisfy the synchronisation process
This means that in our target environment, the APIs and the policy already existed and Tyk-Git just updated those, and it will have done this using masked IDs and the API ID matching so that it is completely transparent to any API user or UI user.
How you can use this
So now, if you ever end up in a dev cycle where you need to make changes to your API Gateway, you can make those interactively in the development environment. Then you can dump those into a branch, move that branch through your CI pipeline to QA and Testing, and once it has been validated, you can guarantee that those changes make it to production, the same way, every time.
Now that you can version control your API Definitions easily, you can also deploy those API Definitions selectively to environments that are not necessarily part of CI or CD systems.
We’ll be improving and tweaking Tyk-Git, and we expect this functionality to make it into the core applications in future. In the meantime, feel free to give Tyk-Git a go and let us know what you think.
