Table of Contents
Getting Started
Quick Start with weed mini (Recommended for Beginners)
⚠️ Warning: The
weed minicommand is for learning and development only. It may not guarantee backward compatibility and should not be used in production. For production use, see Multi-Component Setup below.
The easiest way to get started with SeaweedFS is using the weed mini command, which starts all components in one process with sensible defaults:
weed mini -dir=/data
This gives you a complete SeaweedFS setup including S3 compatibility, all accessible via simple web interfaces. See Quick-Start-with-weed-mini.md for detailed instructions.
Perfect for:
- Learning SeaweedFS
- Development and testing
- S3 beginners
- Prototyping
Multi-Component Setup (For Production and Advanced Users)
For production deployments or if you prefer separating components, follow the manual setup instructions below.
Decompress the downloaded file. You will only find one executable file, either "weed" on most systems or "weed.exe" on windows.
Put the file "weed" to all related computers, in any folder you want. Use
./weed -h # to check available options
Set up Weed Master
./weed master -h # to check available options
If no replication is required, this will be enough. The "mdir" option is to configure a folder where the maximum of generated volume id are saved.
./weed master -mdir="."
./weed master -mdir="." -ip=xxx.xxx.xxx.xxx # usually set the ip instead the default "localhost"
Set up Weed Volume Server
./weed volume -h # to check available options
Usually volume servers are spread on different computers. They can have different disk space, or even different operating system.
Usually you would need to specify the available disk space, the Weed Master location, and the storage folder.
./weed volume -max=100 -master="localhost:9333" -dir="./data"
If you are using a custom gRPC port for master, the address format for -master is <host>:<port>.<grpcPort>. By default the gRPC port is port + 10000.
Cheat Sheet: Setup One Master Server and One Volume Server
Actually, forget about previous commands. You can setup one master server and one volume server in one shot:
./weed server -dir="./data"
# same, just specifying the default values
# use "weed server -h" to find out more
./weed server -master.port=9333 -volume.port=8080 -dir="./data"
Testing SeaweedFS
With the master and volume server up, now what? Let's pump in a lot of files into the system!
./weed upload -dir="/some/big/folder"
This command would recursively upload all files. Or you can specify what files you want to include.
./weed upload -dir="/some/big/folder" -include=*.txt
Then, you can simply check "du -m -s /some/big/folder" to see the actual disk usage by OS, and compare it with the file size under "/data". Usually if you are uploading a lot of textual files, the consumed disk size would be much smaller since textual files are gzipped automatically.
Now you can use your tools to hit SeaweedFS as hard as you can.
Running with Docker
Use with docker is easy as run locally, you can pass all args like above. But you don't need to worry about "-ip". It'll be treated by the entrypoint script.
docker run -p 9333:9333 --name master chrislusf/seaweedfs master -ip=master
docker run -p 8080:8080 -p 18080:18080 --name volume --link master chrislusf/seaweedfs volume -max=5 -master="master:9333" -port=8080
With Compose
But with Compose it's easiest. To startup just run:
docker compose -f docker/seaweedfs-compose.yml -p seaweedfs up
Using SeaweedFS in docker
You can use image "chrislusf/seaweedfs" or build your own with dockerfile in the root of repo.
Using pre-built Docker image
docker run --name weed chrislusf/seaweedfs server
And in another terminal
IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' weed)
curl "http://$IP:9333/cluster/status?pretty=y"
{
"IsLeader": true,
"Leader": "localhost:9333"
}
# use $IP as host for api queries
Building image from dockerfile
Make a local copy of seaweedfs from github
git clone https://github.com/seaweedfs/seaweedfs.git
Minimal Image (~19.6 MB)
docker build --no-cache -t 'chrislusf/seaweedfs' .
Go-Build Docker Image (~764 MB)
mv Dockerfile Dockerfile.minimal
mv Dockerfile.go_build Dockerfile
docker build --no-cache -t 'chrislusf/seaweedfs' .
Building image from source code
Create an executable for docker and move it to the docker folder
cd weed
make build_docker
mv weed ../docker/
cd ..
Build the image
cd docker
docker build -f Dockerfile.local --no-cache -t localregistry/seaweedfs .
In production
You can use docker volumes to persist data:
# start our weed server daemonized
docker run --name weed -d -p 9333:9333 -p 8080:8080 -p 18080:8080 \
-v seaweedvolume:/data chrislusf/seaweedfs server -dir="/data"
Alternatively, you can mount a directory on the host machine into the container:
# start our weed server daemonized
docker run --name weed -d -p 9333:9333 -p 8080:8080 -p 18080:8080 \
-v /opt/weedfs/data:/data chrislusf/seaweedfs server -dir="/data"
Note that according to Docker's documentation, volumes are the preferred mechanism for persisting data.
Now our SeaweedFS server will be persistent and accessible by localhost:9333, :8080 and :18080 on host machine. Dont forget to specify "-publicIp" for correct connectivity.
Introduction
API
Configuration
- Replication
- Store file with a Time To Live
- Failover Master Server
- Erasure coding for warm storage
- Server Startup via Systemd
- Environment Variables
Filer
- Filer Setup
- Directories and Files
- File Operations Quick Reference
- Data Structure for Large Files
- Filer Data Encryption
- Filer Commands and Operations
- Filer JWT Use
- TUS Resumable Uploads
Filer Stores
- Filer Cassandra Setup
- Filer Redis Setup
- Super Large Directories
- Path-Specific Filer Store
- Choosing a Filer Store
- Customize Filer Store
Management
Advanced Filer Configurations
- Migrate to Filer Store
- Add New Filer Store
- Filer Store Replication
- Filer Active Active cross cluster continuous synchronization
- Filer as a Key-Large-Value Store
- Path Specific Configuration
- Filer Change Data Capture
FUSE Mount
WebDAV
Cloud Drive
- Cloud Drive Benefits
- Cloud Drive Architecture
- Configure Remote Storage
- Mount Remote Storage
- Cache Remote Storage
- Cloud Drive Quick Setup
- Gateway to Remote Object Storage
AWS S3 API
- Amazon S3 API
- S3 Conditional Operations
- S3 CORS
- S3 Object Lock and Retention
- S3 Object Versioning
- S3 API Benchmark
- S3 API FAQ
- S3 Bucket Quota
- S3 Rate Limiting
- S3 API Audit log
- S3 Nginx Proxy
- Docker Compose for S3
S3 Authentication & IAM
- S3 Configuration - Start Here
- S3 Credentials (
-s3.config) - OIDC Integration (
-s3.iam.config) - Amazon IAM API
- AWS IAM CLI
Server-Side Encryption
S3 Client Tools
- AWS CLI with SeaweedFS
- s3cmd with SeaweedFS
- rclone with SeaweedFS
- restic with SeaweedFS
- nodejs with Seaweed S3
Machine Learning
HDFS
- Hadoop Compatible File System
- run Spark on SeaweedFS
- run HBase on SeaweedFS
- run Presto on SeaweedFS
- Hadoop Benchmark
- HDFS via S3 connector
Replication and Backup
- Async Replication to another Filer [Deprecated]
- Async Backup
- Async Filer Metadata Backup
- Async Replication to Cloud [Deprecated]
- Kubernetes Backups and Recovery with K8up
Metadata Change Events
Messaging
- Structured Data Lake with SMQ and SQL
- Seaweed Message Queue
- SQL Queries on Message Queue
- SQL Quick Reference
- PostgreSQL-compatible Server weed db
- Pub-Sub to SMQ to SQL
- Kafka to Kafka Gateway to SMQ to SQL
Use Cases
Operations
Advanced
- Large File Handling
- Optimization
- Volume Management
- Tiered Storage
- Cloud Tier
- Cloud Monitoring
- Load Command Line Options from a file
- SRV Service Discovery
- Volume Files Structure
Security
- Security Overview
- Security Configuration
- Cryptography and FIPS Compliance
- Run Blob Storage on Public Internet