As you may know, platform engineering has been around for several years. Yet, there is still a bit of confusion on what platform engineering is, how it differs from DevOps, how it can shape an API program, and what it looks like in practice.
Let’s look at these areas and then explore a case study in the banking industry to see how years of focus have led to a successful platform engineering approach.
What is platform engineering?
At its core, platform engineering provides self-service delivery capabilities to support software engineering teams within your organisation. This involves creating workflows and toolchains that are provisioned and made available in a repeatable, scalable manner.
Platform engineering evolved out of the complexity of modern software applications that often leverage distributed services, web-based APIs, data streams, events, and the tools that support them as part of a cloud-native solution. This results in an internal developer platform (IDP) that supports the provisioning of delivery pipelines through an API call or point-and-click dashboard interface.
How is platform engineering related to DevOps?
DevOps is a cultural shift and tooling that broke down the silo of developers and operations staff by creating a culture of cooperation. This brought people, processes, and technology together to better support customer needs alongside the needs of the developer.
However, problems began to emerge as the operational needs of development teams shifted from dedicated ops teams to the development team. Developers became responsible for operational details, often needing more training and toolchains to accomplish their needs.
This resulted in a higher cognitive load as developers had to address their daily developer role. They were also required to gain experience in the operational details of their solutions quickly. Platform engineering emerged as a means to address these needs at scale.
Focusing on the developer experience
A key aspect of platform engineering is a focus on the developer experience (DX). DX has been a concern for API providers for many years, focusing on the rapid delivery of value through well-designed and documented APIs. Likewise, there is a focus on DX for platform engineering to ensure that developers can provision their infrastructure, deployment pipelines, and other resources quickly and efficiently.
Self-service provisioning by development teams is essential in delivering a great developer experience. Teams should be able to go to their IDP, discover their needs, provision them, and get started without the involvement of the platform engineering team. This scales the effort across larger organisations, with multiple teams collaborating to deliver solutions.
Now that we understand more about platform engineering let’s look at how APIs are involved in all aspects of platform engineering teams.
Using APIs for integration
Platform engineering teams are required to integrate a number of different third-party solutions. This involves writing the glue code necessary to connect each tool in the developer workflow into a unified experience. Sometimes, code is written to call an API to start a process. In other cases, platform engineering teams must react to notifications when specific workflow steps have been completed or generate an error.
The integration tasks require that platform engineers understand how to consume REST, GraphQL and/or gRPC-based APIs. They must also design APIs that receive callbacks from third-party systems due to an event notification. Platform engineers are also responsible for designing their own APIs to orchestrate integration workflows.
Platform engineers are API consumers as part of their integration and automation responsibilities. They learn first-hand what kinds of APIs deliver a great developer experience and which could be better designed, resulting in a better developer experience.
Leveraging infrastructure APIs
The primary reason that cloud-based delivery has gained so much traction is due to the ability to provision resources via APIs elastically. Platform engineers, in turn, use these APIs to establish and maintain shared infrastructure and team-based resources.
Kubernetes has become an industry standard as the “platform for platforms, ” meaning it is a foundational infrastructure element of most organisations. While Kubernetes can be configured declaratively using YAML-based documents, customisations require a deeper understanding of the Kubernetes APIs. Therefore, platform engineering teams must become experienced in using cloud vendors and Kubernetes APIs to support the automation of deployment pipelines for development teams.
Platform engineers are responsible for creating new APIs designed to orchestrate the creation of new delivery pipelines, which development teams may use directly or via an internal management dashboard. These APIs must be well-designed to support a great developer experience.
Supporting the discovery of APIs
An internal developer portal makes it easy for developers to discover the APIs they need. These may be APIs built and managed by a platform engineering team to provision infrastructure and delivery pipelines. An internal developer portal often includes APIs built by development teams themselves.
Discovering and consuming an API becomes a self-service process, improving the likelihood that a team will be able to reuse an existing API. As the portal is promoted through various communication channels, developers shift their behaviour from a build-first approach to a consumer-first approach. Platform engineering teams, through providing an internal developer portal, help to accelerate the delivery of solutions across the organisation.
Platform engineers must encourage the discoverability of their self-service APIs to reduce the time they spend supporting new delivery teams. This makes them both experienced in cataloguing their APIs and encouraging other teams to consume first and build only when necessary.
Case Study: North American Banking Institution
Let’s explore a case study in platform engineering by looking at a North American banking institution.
The banking institution has over 2000 software delivery teams and over 10,000 developers. As you can imagine, this requires considerable effort to ensure that every team can focus on delivering value without being concerned about their infrastructure concerns.
No platform engineering team in place
When they began their journey, there were no platform engineering teams. Every delivery team was forced to stand up their delivery pipeline. Initial investments were made to offer shared infrastructure and delivery pipelines but little else.
Over time, more investments were made to grow their IDP. Several platform engineering teams exist to support this large number of developers. This includes teams that support the following processes:
- Automated deployment of code into various environments, including integration, staging, and production.
- Automated configuration of API gateways based on the purpose of an API (internal access, external access, etc).
- Infrastructure automation to prevent code from being exposed externally without proper architectural, security, and design reviews.
- An internal developer portal that supports the API lifecycle, including discovery, consumer onboarding, registration of new assets, formal review and approval processes, and integration with API gateway configuration processes.
- Support for REST and GraphQL-based APIs, dataset discovery, and message streaming on the internal developer portal, making the portal a one-stop location for all digital assets
- Chaos engineering tools to test for resiliency in the face of cloud outages, including failover to other regions automatically.
These teams have existed for as long as ten years to as few as two years. Tremendous investment has been made to build, support, and promote their internal developer portal. Every new employee knows the portal when they onboard, encouraging a consume-first development model.
API, stream, and data governance have evolved as well. Processes that were previously manual have been automated whenever possible. This has resulted in automated reviews that offer reports on compliance based on their standards and practices. Reviewers still perform some checks manually, but the automation has helped speed up the process considerably.
This banking institution recognises the value of improving the developer experience of its staff. Many initiatives undertaken over the years contribute to the success of hundreds or even thousands of teams across the organisation. They have demonstrated leadership in platform engineering over the last decade.
A core discipline
Platform engineering is becoming a core discipline of any mid-to-large-sized organisation. Its focus on developer experience and self-service support for development teams has become essential for organisations seeking to automate their cloud-native initiatives.
It involves APIs at all levels and supports API programs through an internal developer portal and platform. The result is platform engineering teams are in a position to help shape their API platform.
