2

S3 Configuration

chrislusf edited this page 2025-12-14 13:54:08 -08:00

Table of Contents

• S3 Configuration Overview
• Quick Reference
• Basic Credentials (-s3.config)
• Configuration Format
• Available Actions
• Advanced IAM (-s3.iam.config)
• Configuration Format
• Key Components
• Using Both Together
• Common Mistakes
• Wrong: Using identities in -s3.iam.config
• Correct: Using identities in -s3.config
• Configuration Methods Summary
• Embedded IAM API
• Disabling Embedded IAM
• Related Documentation

S3 Configuration Overview

SeaweedFS S3 gateway has two separate configuration systems for different purposes. Understanding the difference is crucial for proper setup.

Quick Reference

Option	Purpose	Use When
-s3.config / -config	Basic S3 credentials (identities, access keys, actions)	You need simple user authentication with access keys
-s3.iam.config / -iam.config	Advanced IAM (STS, OIDC, policies, roles)	You need OIDC integration, role-based access, or AWS IAM-style policies

Basic Credentials (-s3.config)

Use this for: Simple username/password style authentication with access keys.

Documentation: S3 Credentials

weed s3 -config=/path/to/s3.json -filer=localhost:8888
# OR
weed server -s3 -s3.config=/path/to/s3.json

Configuration Format

{
  "identities": [
    {
      "name": "admin_user",
      "credentials": [
        {
          "accessKey": "your_access_key",
          "secretKey": "your_secret_key"
        }
      ],
      "actions": ["Admin", "Read", "Write", "List", "Tagging"]
    },
    {
      "name": "readonly_user",
      "credentials": [
        {
          "accessKey": "readonly_key",
          "secretKey": "readonly_secret"
        }
      ],
      "actions": ["Read", "List"]
    }
  ]
}

Available Actions

Action	Description
Admin	Full access, create/delete buckets
Read	Read objects
Write	Write/upload objects
List	List buckets and objects
Tagging	Manage object tags
Read:bucket1	Read access to specific bucket
Write:bucket1	Write access to specific bucket

---

Advanced IAM (-s3.iam.config)

Use this for: Enterprise features like OIDC/Keycloak integration, STS (Security Token Service), IAM policies, and role-based access control.

Documentation: OIDC Integration

weed s3 -filer=localhost:8888 -iam.config=/path/to/iam.json
# OR
weed server -s3 -s3.iam.config=/path/to/iam.json

Configuration Format

{
  "sts": {
    "tokenDuration": "1h",
    "maxSessionLength": "12h",
    "issuer": "seaweedfs-sts",
    "signingKey": "base64-encoded-32-byte-key"
  },
  "providers": [
    {
      "name": "keycloak",
      "type": "oidc",
      "enabled": true,
      "config": {
        "issuer": "https://keycloak.example.com/realms/myrealm",
        "clientId": "seaweedfs-s3",
        "jwksUri": "https://keycloak.example.com/realms/myrealm/protocol/openid-connect/certs"
      }
    }
  ],
  "policies": [
    {
      "name": "ReadOnlyPolicy",
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          { "Effect": "Allow", "Action": ["s3:Get*", "s3:List*"], "Resource": ["*"] }
        ]
      }
    }
  ],
  "roles": [
    {
      "roleName": "ReadOnlyRole",
      "roleArn": "arn:aws:iam::role/ReadOnlyRole",
      "attachedPolicies": ["ReadOnlyPolicy"],
      "trustPolicy": { ... }
    }
  ]
}

Key Components

Component	Description
sts	Security Token Service configuration for temporary credentials
providers	OIDC identity providers (Keycloak, Okta, Auth0, etc.)
policies	AWS IAM-style policy documents
roles	IAM roles with trust policies for role assumption

Important

: The -s3.iam.config does NOT support the identities field. For basic user credentials, use -s3.config instead.

---

Using Both Together

You can use both configuration options together:

weed s3 \
  -config=/path/to/s3-credentials.json \
  -iam.config=/path/to/iam-advanced.json \
  -filer=localhost:8888

This allows:

• Basic users to authenticate with access keys (from -s3.config)
• OIDC users to authenticate with JWT tokens (from -s3.iam.config)

---

Common Mistakes

Wrong: Using identities in -s3.iam.config

# This will NOT load identities!
weed s3 -iam.config=/path/to/config.json

With config file:

{
  "identities": [...]  // This is IGNORED by -iam.config
}

Correct: Using identities in -s3.config

weed s3 -config=/path/to/config.json

With config file:

{
  "identities": [...]  // This works with -config
}

---

Configuration Methods Summary

Method	Priority	Auto-Reload	Best For
-config file	Highest	SIGHUP	Production static config
Filer storage	Medium	Yes	Dynamic management
Admin UI	Medium	Yes	Web-based management
Environment variables	Fallback	No	Development/testing

See S3 Credentials for detailed information on each method.

---

Embedded IAM API

Starting with SeaweedFS 3.x, the IAM API is embedded in the S3 server by default. This allows managing users, access keys, and policies using AWS IAM CLI commands on the same endpoint as S3.

# Start S3 with embedded IAM (default)
weed s3 -filer=localhost:8888

# IAM and S3 use the same endpoint
aws --endpoint http://localhost:8333 iam create-user --user-name bob
aws --endpoint http://localhost:8333 s3 ls

Disabling Embedded IAM

If you don't need IAM API functionality, you can disable it:

weed s3 -iam=false -filer=localhost:8888

See Amazon IAM API for detailed IAM usage.

---

Related Documentation

• S3 Credentials - Detailed documentation for basic credentials
• OIDC Integration - OIDC/STS integration guide
• Amazon S3 API - S3 API compatibility reference
• Amazon IAM API - IAM API support (embedded in S3)
• AWS IAM CLI - AWS CLI examples for IAM
• Admin UI - Web-based credential management

Introduction

• Quick Start with weed mini
• Components
• Getting Started
• Production Setup
• Benchmarks
• FAQ
• Applications

API

• Master Server API
• Volume Server API
• Filer Server API
• Client Libraries
• SeaweedFS Java Client

Configuration

• Replication
• Store file with a Time To Live
• Failover Master Server
• Erasure coding for warm storage
• Server Startup via Systemd
• Environment Variables

Filer

• Filer Setup
• Directories and Files
• File Operations Quick Reference
• Data Structure for Large Files
• Filer Data Encryption
• Filer Commands and Operations
• Filer JWT Use
• TUS Resumable Uploads

Filer Stores

• Filer Cassandra Setup
• Filer Redis Setup
• Super Large Directories
• Path-Specific Filer Store
• Choosing a Filer Store
• Customize Filer Store

Management

• Admin UI
• Worker

Advanced Filer Configurations

• Migrate to Filer Store
• Add New Filer Store
• Filer Store Replication
• Filer Active Active cross cluster continuous synchronization
• Filer as a Key-Large-Value Store
• Path Specific Configuration
• Filer Change Data Capture

FUSE Mount

• FIO benchmark
• fstab

WebDAV

Cloud Drive

• Cloud Drive Benefits
• Cloud Drive Architecture
• Configure Remote Storage
• Mount Remote Storage
• Cache Remote Storage
• Cloud Drive Quick Setup
• Gateway to Remote Object Storage

AWS S3 API

• Amazon S3 API
• S3 Conditional Operations
• S3 CORS
• S3 Object Lock and Retention
• S3 Object Versioning
• S3 API Benchmark
• S3 API FAQ
• S3 Bucket Quota
• S3 Rate Limiting
• S3 API Audit log
• S3 Nginx Proxy
• Docker Compose for S3

S3 Authentication & IAM

• S3 Configuration - Start Here
• S3 Credentials (-s3.config)
• OIDC Integration (-s3.iam.config)
• Amazon IAM API
• AWS IAM CLI

Server-Side Encryption

• Server-Side Encryption
• Server-Side Encryption SSE-KMS
• Server-Side Encryption SSE-C

S3 Client Tools

• AWS CLI with SeaweedFS
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3

Machine Learning

• TensorFlow with SeaweedFS

HDFS

• Hadoop Compatible File System
• run Spark on SeaweedFS
• run HBase on SeaweedFS
• run Presto on SeaweedFS
• Hadoop Benchmark
• HDFS via S3 connector

Replication and Backup

• Async Replication to another Filer [Deprecated]
• Async Backup
• Async Filer Metadata Backup
• Async Replication to Cloud [Deprecated]
• Kubernetes Backups and Recovery with K8up

Metadata Change Events

• Filer Metadata Events
• Filer Notification Webhook

Messaging

• Structured Data Lake with SMQ and SQL
• Seaweed Message Queue
• SQL Queries on Message Queue
• SQL Quick Reference
• PostgreSQL-compatible Server weed db
• Pub-Sub to SMQ to SQL
• Kafka to Kafka Gateway to SMQ to SQL

Use Cases

• Use Cases
• Actual Users

Operations

• System Metrics
• weed shell
• Data Backup
• Deployment to Kubernetes and Minikube

Advanced

• Large File Handling
• Optimization
• Volume Management
• Tiered Storage
• Cloud Tier
• Cloud Monitoring
• Load Command Line Options from a file
• SRV Service Discovery
• Volume Files Structure

Security

• Security Overview
• Security Configuration
• Cryptography and FIPS Compliance
• Run Blob Storage on Public Internet

Misc Use Case Examples

• UrBackup with SeaweedFS
• Docker Image Registry with SeaweedFS
• SeaweedFS in Docker Swarm
• Words from SeaweedFS Users
• Independent Benchmarks
• Hardware