lsm/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-11-24 16:53:14 +08:00
Code Issues Actions 17 Packages Projects Releases Wiki Activity
Page: Kafka to Kafka Gateway to SMQ to SQL
Pages
AWS CLI with SeaweedFS AWS IAM CLI Actual Users Admin UI Amazon IAM API Amazon S3 API Applications Async Backup Async Filer Metadata Backup Async Replication to Cloud Async Replication to another Filer Benchmark SeaweedFS as a GlusterFS replacement Benchmarks from jinleileiking Benchmarks Cache Remote Storage Choosing a Filer Store Client Libraries Cloud Drive Architecture Cloud Drive Benefits Cloud Drive Quick Setup Cloud Monitoring Cloud Tier Components Configure Remote Storage Customize Filer Store Data Backup Data Structure for Large Files Deployment to Kubernetes and Minikube Directories and Files Docker Compose for S3 Docker Image Registry with SeaweedFS Environment Variables Erasure Coding for warm storage Error reporting to sentry FAQ FIO benchmark FUSE Mount Failover Master Server File Operations Quick Reference Filer Active Active cross cluster continuous synchronization Filer Cassandra Setup Filer Change Data Capture Filer Commands and Operations Filer Data Encryption Filer JWT Use Filer Metadata Events Filer Redis Setup Filer Server API Filer Setup Filer Store Replication Filer Stores Filer as a Key Large Value Store Gateway to Remote Object Storage Getting Started HDFS via S3 connector Hadoop Benchmark Hadoop Compatible File System Hardware Hobbyest Tinkerer scale on premises tutorial Home Independent Benchmarks Kafka to Kafka Gateway to SMQ to SQL Keycloak Integration Kubernetes Backups and Recovery with K8up Large File Handling Load Command Line Options from a file Master Server API Migrate to Filer Store Mount Remote Storage Optimization Path Specific Configuration Path Specific Filer Store PostgreSQL compatible Server weed db Production Setup Pub Sub to SMQ to SQL Replication Run Blob Storage on Public Internet Run Presto on SeaweedFS S3 API Audit log S3 API Benchmark S3 API FAQ S3 Bucket Quota S3 CORS S3 Conditional Operations S3 Credentials S3 Nginx Proxy S3 Object Lock and Retention S3 Object Versioning SQL Queries on Message Queue SQL Quick Reference SRV Service Discovery Seaweed Message Queue SeaweedFS Java Client SeaweedFS in Docker Swarm Security Configuration Security Overview Server Side Encryption SSE C Server Side Encryption SSE KMS Server Side Encryption Server Startup via Systemd Store file with a Time To Live Structured Data Lake with SMQ and SQL Super Large Directories System Metrics TensorFlow with SeaweedFS Tiered Storage UrBackup with SeaweedFS Use Cases Volume Files Structure Volume Management Volume Server API WebDAV Words from SeaweedFS Users Worker fstab nodejs with Seaweed S3 rclone with SeaweedFS restic with SeaweedFS run HBase on SeaweedFS run Spark on SeaweedFS s3cmd with SeaweedFS weed shell
Clone
1
Kafka to Kafka Gateway to SMQ to SQL
chrislusf edited this page 2025-09-15 22:24:39 -07:00
Table of Contents

Kafka client → Kafka gateway → SMQ → SQL

Bring your existing Kafka clients. Point them at the SeaweedFS Kafka gateway. Messages flow into Seaweed Message Queue (SMQ) for streaming, while SeaweedFS persists them into Parquet for SQL analytics.

See the end-to-end picture: Structured Data Lake with SMQ and SQL.

Why use the Kafka gateway

  • Keep your Kafka tooling and clients
  • Scale stateless brokers and storage independently
  • Get streaming + Parquet-based analytics without changing producers

Architecture

Kafka Clients  <=>  SeaweedFS Kafka Gateway  <=>  SMQ Brokers  =>  Subscribers
                                                  \
                                                   +--> SeaweedFS (Parquet) => SQL Engines

The gateway speaks the Kafka protocol to clients and maps topics/partitions, offsets, and consumer groups to SMQ semantics.

What stays the same

  • Kafka client libraries and tooling (producers/consumers)
  • Topic/partition concepts
  • Consumer groups and offsets

What you gain

  • Durable Parquet storage for batch analytics
  • One pipeline for both streaming and SQL
  • Simple, scalable operations (stateless brokers, disaggregated storage)

Getting started

  1. Start SMQ and the Kafka gateway (example ports):
weed mq.broker -port=17777 -master=localhost:9333
weed mq.agent  -port=16777 -broker=localhost:17777
weed mq.kafka  -port=19092 -broker=localhost:17777

  1. Point your Kafka producer/consumer at localhost:19092.

  2. Query the resulting Parquet data with your SQL engine of choice (Trino, Spark, DuckDB, etc.).

Next steps

Introduction

API

Configuration

Filer

Filer Stores

Management

Advanced Filer Configurations

FUSE Mount

WebDAV

Cloud Drive

AWS S3 API

Server-Side Encryption

AWS IAM

Machine Learning

HDFS

Replication and Backup

Metadata Change Events

Messaging

Use Cases

Operations

Advanced

Security

Misc Use Case Examples