lsm/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-11-24 08:46:54 +08:00
Code Issues Actions 17 Packages Projects Releases Wiki Activity
Page: Structured Data Lake with SMQ and SQL
Pages
AWS CLI with SeaweedFS AWS IAM CLI Actual Users Admin UI Amazon IAM API Amazon S3 API Applications Async Backup Async Filer Metadata Backup Async Replication to Cloud Async Replication to another Filer Benchmark SeaweedFS as a GlusterFS replacement Benchmarks from jinleileiking Benchmarks Cache Remote Storage Choosing a Filer Store Client Libraries Cloud Drive Architecture Cloud Drive Benefits Cloud Drive Quick Setup Cloud Monitoring Cloud Tier Components Configure Remote Storage Customize Filer Store Data Backup Data Structure for Large Files Deployment to Kubernetes and Minikube Directories and Files Docker Compose for S3 Docker Image Registry with SeaweedFS Environment Variables Erasure Coding for warm storage Error reporting to sentry FAQ FIO benchmark FUSE Mount Failover Master Server File Operations Quick Reference Filer Active Active cross cluster continuous synchronization Filer Cassandra Setup Filer Change Data Capture Filer Commands and Operations Filer Data Encryption Filer JWT Use Filer Metadata Events Filer Redis Setup Filer Server API Filer Setup Filer Store Replication Filer Stores Filer as a Key Large Value Store Gateway to Remote Object Storage Getting Started HDFS via S3 connector Hadoop Benchmark Hadoop Compatible File System Hardware Hobbyest Tinkerer scale on premises tutorial Home Independent Benchmarks Kafka to Kafka Gateway to SMQ to SQL Keycloak Integration Kubernetes Backups and Recovery with K8up Large File Handling Load Command Line Options from a file Master Server API Migrate to Filer Store Mount Remote Storage Optimization Path Specific Configuration Path Specific Filer Store PostgreSQL compatible Server weed db Production Setup Pub Sub to SMQ to SQL Replication Run Blob Storage on Public Internet Run Presto on SeaweedFS S3 API Audit log S3 API Benchmark S3 API FAQ S3 Bucket Quota S3 CORS S3 Conditional Operations S3 Credentials S3 Nginx Proxy S3 Object Lock and Retention S3 Object Versioning SQL Queries on Message Queue SQL Quick Reference SRV Service Discovery Seaweed Message Queue SeaweedFS Java Client SeaweedFS in Docker Swarm Security Configuration Security Overview Server Side Encryption SSE C Server Side Encryption SSE KMS Server Side Encryption Server Startup via Systemd Store file with a Time To Live Structured Data Lake with SMQ and SQL Super Large Directories System Metrics TensorFlow with SeaweedFS Tiered Storage UrBackup with SeaweedFS Use Cases Volume Files Structure Volume Management Volume Server API WebDAV Words from SeaweedFS Users Worker fstab nodejs with Seaweed S3 rclone with SeaweedFS restic with SeaweedFS run HBase on SeaweedFS run Spark on SeaweedFS s3cmd with SeaweedFS weed shell
Clone
1
Structured Data Lake with SMQ and SQL
chrislusf edited this page 2025-09-15 22:24:39 -07:00
Table of Contents

Structured Data Lake with SMQ and SQL

SeaweedFS + Seaweed Message Queue (SMQ) gives you a unified pipeline: produce structured messages, process them in real time, and query the same data with SQL. Whether producers speak Kafka or a simple pub/sub gRPC API, they both write schematized messages into the same data lake.

Core ideas

  • Schematized messages can be queried directly with SQL (no ETL required)
  • SMQ brokers are computation-only nodes and scale linearly with demand
  • Structured data is written as messages and can be queried in real time
  • Together, SeaweedFS + SMQ form a data lake for structured data (hot streams + Parquet)

Ingestion paths

Two equivalent ways to ingest structured messages:

Both end up with the same outcomes: live streams for subscribers and Parquet files in SeaweedFS for SQL engines.

Architecture

Producers (Kafka or Pub/Sub)  ==>  SMQ Brokers  ==>  Subscribers (real-time)
                                        \
                                         +--> SeaweedFS (Parquet) ==> SQL Engines

Querying the lake

Point your SQL engines at the Parquet paths:

  • Trino/Presto
  • Spark SQL
  • DuckDB
  • ClickHouse (file table engines)

Examples are available in the ingestion pages.

Operate at scale

  • Scale SMQ brokers horizontally; they are stateless computation nodes
  • Storage is disaggregated (SeaweedFS) for durable, efficient Parquet files

Learn more

Introduction

API

Configuration

Filer

Filer Stores

Management

Advanced Filer Configurations

FUSE Mount

WebDAV

Cloud Drive

AWS S3 API

Server-Side Encryption

AWS IAM

Machine Learning

HDFS

Replication and Backup

Metadata Change Events

Messaging

Use Cases

Operations

Advanced

Security

Misc Use Case Examples