mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2025-09-23 07:25:01 +08:00
setup integration test for postgres
This commit is contained in:
chrislu
326
test/postgres/README.md
Normal file
326
test/postgres/README.md
Normal file
@@ -0,0 +1,326 @@
|
||||
# SeaweedFS PostgreSQL Protocol Test Suite
|
||||
|
||||
This directory contains a comprehensive Docker Compose test setup for the SeaweedFS PostgreSQL wire protocol implementation.
|
||||
|
||||
## Overview
|
||||
|
||||
The test suite includes:
|
||||
- **SeaweedFS Cluster**: Full SeaweedFS server with MQ broker and agent
|
||||
- **PostgreSQL Server**: SeaweedFS PostgreSQL wire protocol server
|
||||
- **MQ Data Producer**: Creates realistic test data across multiple topics and namespaces
|
||||
- **PostgreSQL Test Client**: Comprehensive Go client testing all functionality
|
||||
- **Interactive Tools**: psql CLI access for manual testing
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Run Complete Test Suite (Automated)
|
||||
```bash
|
||||
./run-tests.sh all
|
||||
```
|
||||
|
||||
This will automatically:
|
||||
1. Start SeaweedFS and PostgreSQL servers
|
||||
2. Create test data in multiple MQ topics
|
||||
3. Run comprehensive PostgreSQL client tests
|
||||
4. Show results
|
||||
|
||||
### 2. Manual Step-by-Step Testing
|
||||
```bash
|
||||
# Start the services
|
||||
./run-tests.sh start
|
||||
|
||||
# Create test data
|
||||
./run-tests.sh produce
|
||||
|
||||
# Run automated tests
|
||||
./run-tests.sh test
|
||||
|
||||
# Connect with psql for interactive testing
|
||||
./run-tests.sh psql
|
||||
```
|
||||
|
||||
### 3. Interactive PostgreSQL Testing
|
||||
```bash
|
||||
# Connect with psql
|
||||
./run-tests.sh psql
|
||||
|
||||
# Inside psql session:
|
||||
postgres=> SHOW DATABASES;
|
||||
postgres=> USE analytics;
|
||||
postgres=> SHOW TABLES;
|
||||
postgres=> SELECT COUNT(*) FROM user_events;
|
||||
postgres=> SELECT user_type, COUNT(*) FROM user_events GROUP BY user_type;
|
||||
postgres=> \q
|
||||
```
|
||||
|
||||
## Test Data Structure
|
||||
|
||||
The producer creates realistic test data across multiple namespaces:
|
||||
|
||||
### Analytics Namespace
|
||||
- **`user_events`** (1000 records): User interaction events
|
||||
- Fields: id, user_id, user_type, action, status, amount, timestamp, metadata
|
||||
- User types: premium, standard, trial, enterprise
|
||||
- Actions: login, logout, purchase, view, search, click, download
|
||||
|
||||
- **`system_logs`** (500 records): System operation logs
|
||||
- Fields: id, level, service, message, error_code, timestamp
|
||||
- Levels: debug, info, warning, error, critical
|
||||
- Services: auth-service, payment-service, user-service, etc.
|
||||
|
||||
- **`metrics`** (800 records): System metrics
|
||||
- Fields: id, name, value, tags, timestamp
|
||||
- Metrics: cpu_usage, memory_usage, disk_usage, request_latency, etc.
|
||||
|
||||
### E-commerce Namespace
|
||||
- **`product_views`** (1200 records): Product interaction data
|
||||
- Fields: id, product_id, user_id, category, price, view_count, timestamp
|
||||
- Categories: electronics, books, clothing, home, sports, automotive
|
||||
|
||||
- **`user_events`** (600 records): E-commerce specific user events
|
||||
|
||||
### Logs Namespace
|
||||
- **`application_logs`** (2000 records): Application logs
|
||||
- **`error_logs`** (300 records): Error-specific logs with 4xx/5xx error codes
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||||
│ PostgreSQL │ │ PostgreSQL │ │ SeaweedFS │
|
||||
│ Clients │◄──►│ Wire Protocol │◄──►│ SQL Engine │
|
||||
│ (psql, Go) │ │ Server │ │ │
|
||||
└─────────────────┘ └──────────────────┘ └─────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────┐ ┌─────────────────┐
|
||||
│ Session │ │ MQ Broker │
|
||||
│ Management │ │ & Topics │
|
||||
└──────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
## Services
|
||||
|
||||
### SeaweedFS Server
|
||||
- **Ports**: 9333 (master), 8888 (filer), 8333 (S3), 8085 (volume), 9533 (metrics), 26777→16777 (MQ agent), 27777→17777 (MQ broker)
|
||||
- **Features**: Full MQ broker, S3 API, filer, volume server
|
||||
- **Data**: Persistent storage in Docker volume
|
||||
- **Health Check**: Cluster status endpoint
|
||||
|
||||
### PostgreSQL Server
|
||||
- **Port**: 5432 (standard PostgreSQL port)
|
||||
- **Protocol**: Full PostgreSQL 3.0 wire protocol
|
||||
- **Authentication**: Trust mode (no password for testing)
|
||||
- **Features**: Real-time MQ topic discovery, database context switching
|
||||
|
||||
### MQ Producer
|
||||
- **Purpose**: Creates realistic test data
|
||||
- **Topics**: 7 topics across 3 namespaces
|
||||
- **Data Types**: JSON messages with varied schemas
|
||||
- **Volume**: ~4,400 total records with realistic distributions
|
||||
|
||||
### Test Client
|
||||
- **Language**: Go with standard `lib/pq` PostgreSQL driver
|
||||
- **Tests**: 8 comprehensive test categories
|
||||
- **Coverage**: System info, discovery, queries, aggregations, context switching
|
||||
|
||||
## Available Commands
|
||||
|
||||
```bash
|
||||
./run-tests.sh start # Start services
|
||||
./run-tests.sh produce # Create test data
|
||||
./run-tests.sh test # Run client tests
|
||||
./run-tests.sh psql # Interactive psql
|
||||
./run-tests.sh logs # Show service logs
|
||||
./run-tests.sh status # Service status
|
||||
./run-tests.sh stop # Stop services
|
||||
./run-tests.sh clean # Complete cleanup
|
||||
./run-tests.sh all # Full automated test
|
||||
```
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. System Information
|
||||
- PostgreSQL version compatibility
|
||||
- Current user and database
|
||||
- Server settings and encoding
|
||||
|
||||
### 2. Database Discovery
|
||||
- `SHOW DATABASES` - List MQ namespaces
|
||||
- Dynamic namespace discovery from filer
|
||||
|
||||
### 3. Table Discovery
|
||||
- `SHOW TABLES` - List topics in current namespace
|
||||
- Real-time topic discovery
|
||||
|
||||
### 4. Data Queries
|
||||
- Basic `SELECT * FROM table` queries
|
||||
- Sample data retrieval and display
|
||||
- Column information
|
||||
|
||||
### 5. Aggregation Queries
|
||||
- `COUNT(*)`, `SUM()`, `AVG()`, `MIN()`, `MAX()`
|
||||
- `GROUP BY` operations
|
||||
- Statistical analysis
|
||||
|
||||
### 6. Database Context Switching
|
||||
- `USE database` commands
|
||||
- Session isolation testing
|
||||
- Cross-namespace queries
|
||||
|
||||
### 7. System Columns
|
||||
- `_timestamp_ns`, `_key`, `_source` access
|
||||
- MQ metadata exposure
|
||||
|
||||
### 8. Complex Queries
|
||||
- `WHERE` clauses with comparisons
|
||||
- `ORDER BY` and `LIMIT`
|
||||
- Multi-condition filtering
|
||||
|
||||
## Expected Results
|
||||
|
||||
After running the complete test suite, you should see:
|
||||
|
||||
```
|
||||
=== Test Results ===
|
||||
✅ Test PASSED: System Information
|
||||
✅ Test PASSED: Database Discovery
|
||||
✅ Test PASSED: Table Discovery
|
||||
✅ Test PASSED: Data Queries
|
||||
✅ Test PASSED: Aggregation Queries
|
||||
✅ Test PASSED: Database Context Switching
|
||||
✅ Test PASSED: System Columns
|
||||
✅ Test PASSED: Complex Queries
|
||||
|
||||
Test Results: 8/8 tests passed
|
||||
🎉 All tests passed!
|
||||
```
|
||||
|
||||
## Manual Testing Examples
|
||||
|
||||
### Connect with psql
|
||||
```bash
|
||||
./run-tests.sh psql
|
||||
```
|
||||
|
||||
### Basic Exploration
|
||||
```sql
|
||||
-- Check system information
|
||||
SELECT version();
|
||||
SELECT current_user, current_database();
|
||||
|
||||
-- Discover data structure
|
||||
SHOW DATABASES;
|
||||
USE analytics;
|
||||
SHOW TABLES;
|
||||
DESCRIBE user_events;
|
||||
```
|
||||
|
||||
### Data Analysis
|
||||
```sql
|
||||
-- Basic queries
|
||||
SELECT COUNT(*) FROM user_events;
|
||||
SELECT * FROM user_events LIMIT 5;
|
||||
|
||||
-- Aggregations
|
||||
SELECT
|
||||
user_type,
|
||||
COUNT(*) as events,
|
||||
AVG(amount) as avg_amount
|
||||
FROM user_events
|
||||
WHERE amount IS NOT NULL
|
||||
GROUP BY user_type
|
||||
ORDER BY events DESC;
|
||||
|
||||
-- Time-based analysis
|
||||
SELECT
|
||||
action,
|
||||
COUNT(*) as count
|
||||
FROM user_events
|
||||
WHERE status = 'active'
|
||||
GROUP BY action
|
||||
ORDER BY count DESC;
|
||||
```
|
||||
|
||||
### Cross-Namespace Analysis
|
||||
```sql
|
||||
-- Switch between namespaces
|
||||
USE ecommerce;
|
||||
SELECT category, COUNT(*) FROM product_views GROUP BY category;
|
||||
|
||||
USE logs;
|
||||
SELECT level, COUNT(*) FROM application_logs GROUP BY level;
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Services Not Starting
|
||||
```bash
|
||||
# Check service status
|
||||
./run-tests.sh status
|
||||
|
||||
# View logs
|
||||
./run-tests.sh logs seaweedfs
|
||||
./run-tests.sh logs postgres-server
|
||||
```
|
||||
|
||||
### No Test Data
|
||||
```bash
|
||||
# Recreate test data
|
||||
./run-tests.sh produce
|
||||
|
||||
# Check producer logs
|
||||
./run-tests.sh logs mq-producer
|
||||
```
|
||||
|
||||
### Connection Issues
|
||||
```bash
|
||||
# Test PostgreSQL server health
|
||||
docker-compose exec postgres-server nc -z localhost 5432
|
||||
|
||||
# Test SeaweedFS health
|
||||
curl http://localhost:9333/cluster/status
|
||||
```
|
||||
|
||||
### Clean Restart
|
||||
```bash
|
||||
# Complete cleanup and restart
|
||||
./run-tests.sh clean
|
||||
./run-tests.sh all
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Modifying Test Data
|
||||
Edit `producer.go` to change:
|
||||
- Data schemas and volume
|
||||
- Topic names and namespaces
|
||||
- Record generation logic
|
||||
|
||||
### Adding Tests
|
||||
Edit `client.go` to add new test functions:
|
||||
```go
|
||||
func testNewFeature(db *sql.DB) error {
|
||||
// Your test implementation
|
||||
return nil
|
||||
}
|
||||
|
||||
// Add to tests slice in main()
|
||||
{"New Feature", testNewFeature},
|
||||
```
|
||||
|
||||
### Custom Queries
|
||||
Use the interactive psql session:
|
||||
```bash
|
||||
./run-tests.sh psql
|
||||
```
|
||||
|
||||
## Production Considerations
|
||||
|
||||
This test setup demonstrates:
|
||||
- **Real MQ Integration**: Actual topic discovery and data access
|
||||
- **Universal PostgreSQL Compatibility**: Works with any PostgreSQL client
|
||||
- **Production-Ready Features**: Authentication, session management, error handling
|
||||
- **Scalable Architecture**: Direct SQL engine integration, no translation overhead
|
||||
|
||||
The test validates that SeaweedFS can serve as a drop-in PostgreSQL replacement for read-only analytics workloads on MQ data.
|
||||
Reference in New Issue
Block a user