A step-by-step guide to Redpanda and GraphQL integration

Publish on 24 Jul, 2023 - by Carlos Villanua Fernandez
Last updated: 27 Dec, 2023

Integrating Kafka’s distributed streaming platform with front end applications can be complex, so developers must invest time understanding Kafka’s unique concepts and technologies. This means that handling the raw and unstructured nature of Kafka data requires some effort in data transformation for proper display in the front end. 

To ensure smooth integration and transformation and address potential concerns regarding latency, data compliance and security, companies choose to use intermediate layers like APIs or microservices to preprocess and present Kafka data in a more user-friendly and secure manner for their front-end applications.

Some examples of these companies include Airbnb, Netflix and Paypal, which were able to abstract the complexities of Kafka’s distributed streaming platform, saving time and effort in understanding its unique concepts and technologies. They managed to transform raw and unstructured Kafka data into a user-friendly format for seamless display on the front-end, enhancing the overall user experience and smoother integration.

GraphQL and Kafka

The best way to do this is to integrate GraphQL and Kafka. GraphQL is a powerful intermediary between Kafka and front-end applications, offering significant benefits:

  • Its flexible nature allows front-end developers to precisely request the data they need, reducing complexity and the learning curve associated with directly working with Kafka.
  • Its schema-based approach facilitates easy data transformation, enabling developers to format Kafka data for display in the front end seamlessly. 
  • The ability to fetch multiple data points in a single query minimises the potential latency issues when accessing Kafka topics.
  •  It ensures a smoother and more secure integration of Kafka data into front-end applications by handling data processing and transformation, providing a user-friendly and structured data delivery mechanism.

Here, we will guide you in integrating Redpanda, a Kafka API-compatible, high-performance, cost-efficient streaming data platform that simplifies Kafka APIs with GraphQL using Universal Data Graph from Tyk. 

Let’s start! 

How to integrate UDG from Tyk and Redpanda Kafka

In this section, you will find the configuration steps for using Universal Data Graph to read the data from the Redpanda Kafka database. It includes a Tyk-Pro installation, Redpanda console, Kafka broker and a mock server with e-commerce data.

Tyk prerequisites

Redpanda prerequisites

Tyk + Redpanda integration

  • Open the docker-compose.yaml from the tyk-pro-docker-demo folder.
  • Download the docker-compose.yaml from the Redpanda repository. https://github.com/redpanda-data/console/blob/master/docs/local/docker-compose.yaml. 
  • Copy the services redpanda, console, connect and owl-shop from the Redpanda docker-compose.yaml
  • Paste the copied services in on the docker-compose.yaml from the tyk-pro-docker-demo folder. 
  • Add the following at the end of each service:
networks:

      - tyk

For example:

Tyk + Redpanda deployment

  • Open a terminal and go to the tyk-pro-docker-demo folder.
  • Run docker-compose up.
  • Access the Redpanda console – http://localhost:8090/:

 

  • Access the Tyk Dashboard and bootstrap your installation http://localhost:3000

Configuring UDG + Kafka

Look for a topic:

  • Select the owlshop-customers topic.
  • Expand one of the topics to check the objects values:

Configuring UDG

  • Add a name of choice and select Design data graph.
  • Click on Next.

  • Change the authentication to open-keyless for now:

  • Go to Schema and copy and paste the following code. It has the objects from the owlshop-customers topic:
typeQuery {

customers:Customers

}

typeCustomers{

version:Int

id:String

firstName:String

lastName:String

gender:String

email:String

customerType:String

revision:Int

{

type Subscription {

owlShopCustomers:Customers

}
  • Click save, and you will see something like this:

  • Click on owlShopCustomers in Type|Subscription:

  • Click on KAFKA in the new external data source section:

Enter the following:

  • Data source name: Redpanda Kafka (it could be anything). 
  • Broker Addresses: redpanda:29092 (for this example).

Go to the Redpanda console broker section: http://localhost:8090/brokers. 

  • Look for the advertised.listeners and choose the PLAINTEXT (this is because we have all the containers in the same network). If Redpanda was outside, we would use OUTSIDE:

  • Topics: owlshop-customers 
  • GroupID: tyk-owlshop-customers-group
  • ClientID: tyk-owlshop-customers-client
  • Kafka Version: V2_8_0_0
  • Balance Strategy: BalanceStrategySticky
  • Select Use default field mapping
  • Leave the rest by default
  • Click save and update.

Testing with the Playground

  • Go to the Playground tab.
  • Copy and paste the following request:
subscription Customers {

owlShopCustomers {

customerType

email

firstName

gender

id

lastName

revision

version

}

}

 

  • Click on the play button (and let the fun start!). 

Of course, in that schema, you can see all the data coming from Kafka. Now let’s assume we have an issue, because “gender” is considered sensitive data and it should not be available for internal or external consumption.

Rather than making changes to the Kafka topic and the message structure, you can simply remove that field from the schema and not expose it at all. No changes to the Kafka topic, no changes to the Kafka message:

customers:Customers

}

type Customers {

version:Int

id:String

firstName:String

lastName:String

email:String

customerType:String

revision:Int

}

type Subscription {

owlShopCustomers:Customers

}

Redpanda and Tyk integration – complete!

Companies may face challenges when attempting to use Kafka data directly in their front-end applications, but that doesn’t mean it isn’t possible! If you follow this guide, you can leverage Redpanda’s high-data streaming capabilities and Tyk’s robust API gateway features for a powerful and effective synergy. 

This integration paves the way for building scalable, high-performance, and resilient API architectures, empowering businesses to deliver exceptional user experiences while managing their data flows effectively. Whether handling event-driven systems, real-time analytics, or ensuring data integrity, the Redpanda and Tyk integration is a winning combination in the ever-evolving API landscape.

Contact our team of expert engineers for more information about the integration of Redpanda and Tyk as a powerful duo, here.

Create, secure & test APIs