Skip to content

feat: migration to platformatic-kafka#166

Open
Tony133 wants to merge 4 commits intomainfrom
feat/platformatic-kafka-migration
Open

feat: migration to platformatic-kafka#166
Tony133 wants to merge 4 commits intomainfrom
feat/platformatic-kafka-migration

Conversation

@Tony133
Copy link
Member

@Tony133 Tony133 commented Mar 5, 2026

This PR migrates @fastify/kafka from node-rdkafka to @platformatic/kafka ( see issue: #158 )

Motivation

node-rdkafka relies on native C++ bindings which complicate cross-platform builds and CI pipelines. @platformatic/kafka is a pure JavaScript implementation with a modern async API, better maintained and aligned with the Fastify ecosystem.

Changes

  • lib/producer.js — rewritten using @platformatic/kafka Producer; push() is now fully async
  • lib/consumer.js — rewritten using @platformatic/kafka Consumer; supports both flow mode (event-based streaming) and batch mode (consume(n, cbk)) with configurable timeout
  • index.js — plugin structure unchanged; wiring updated to match new producer/consumer API
  • index.d.ts — types updated to reference @platformatic/kafka
  • docker-compose.yml — switched to KRaft mode (no Zookeeper) using confluentinc/cp-kafka:7.6.1
  • test/ — full rewrite using node:test; covers producer, consumer, error handling, batch timeout, stream destruction
  • README.md — updated usage examples and added migration guide from node-rdkafka
  • examples/basic.js — added a working end-to-end example showing producer and consumer setup, topic subscription, flow mode consumption, and graceful shutdown

Thanks to the @platformatic team for creating @platformatic/kafka 🙏

@Tony133
Copy link
Member Author

Tony133 commented Mar 5, 2026

I have used Kafka very little in my work experience 😅, but I believe the important thing was to start laying the groundwork and try to make this conversion, so if you find any errors or parts that could be improved, please feel free to leave feedback. We could compile a list as a reminder so we can update the code.

@Tony133 Tony133 force-pushed the feat/platformatic-kafka-migration branch from 88668ed to f384b28 Compare March 5, 2026 10:26
@Tony133 Tony133 requested a review from mcollina March 8, 2026 09:09
Copy link

@ShogunPanda ShogunPanda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for considering @platformatic/kafka. A few nits for you but in general it looks fine!

@Tony133 Tony133 marked this pull request as ready for review March 11, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants