Files
seaweedfs/weed/server/postgres/protocol.go
Chris Lu a7fdc0d137 Message Queue: Add sql querying (#7185)
* feat: Phase 1 - Add SQL query engine foundation for MQ topics

Implements core SQL infrastructure with metadata operations:

New Components:
- SQL parser integration using github.com/xwb1989/sqlparser
- Query engine framework in weed/query/engine/
- Schema catalog mapping MQ topics to SQL tables
- Interactive SQL CLI command 'weed sql'

Supported Operations:
- SHOW DATABASES (lists MQ namespaces)
- SHOW TABLES (lists MQ topics)
- SQL statement parsing and routing
- Error handling and result formatting

Key Design Decisions:
- MQ namespaces ↔ SQL databases
- MQ topics ↔ SQL tables
- Parquet message storage ready for querying
- Backward-compatible schema evolution support

Testing:
- Unit tests for core engine functionality
- Command integration tests
- Parse error handling validation

Assumptions (documented in code):
- All MQ messages stored in Parquet format
- Schema evolution maintains backward compatibility
- MySQL-compatible SQL syntax via sqlparser
- Single-threaded usage per SQL session

Next Phase: DDL operations (CREATE/ALTER/DROP TABLE)

* feat: Phase 2 - Add DDL operations and real MQ broker integration

Implements comprehensive DDL support for MQ topic management:

New Components:
- Real MQ broker connectivity via BrokerClient
- CREATE TABLE → ConfigureTopic gRPC calls
- DROP TABLE → DeleteTopic operations
- DESCRIBE table → Schema introspection
- SQL type mapping (SQL ↔ MQ schema types)

Enhanced Features:
- Live topic discovery from MQ broker
- Fallback to cached/sample data when broker unavailable
- MySQL-compatible DESCRIBE output
- Schema validation and error handling
- CREATE TABLE with column definitions

Key Infrastructure:
- broker_client.go: gRPC communication with MQ broker
- sql_types.go: Bidirectional SQL/MQ type conversion
- describe.go: Table schema introspection
- Enhanced engine.go: Full DDL routing and execution

Supported SQL Operations:
 SHOW DATABASES, SHOW TABLES (live + fallback)
 CREATE TABLE table_name (col1 INT, col2 VARCHAR(50), ...)
 DROP TABLE table_name
 DESCRIBE table_name / SHOW COLUMNS FROM table_name

Known Limitations:
- SQL parser issues with reserved keywords (e.g., 'timestamp')
- Requires running MQ broker for full functionality
- ALTER TABLE not yet implemented
- DeleteTopic method needs broker-side implementation

Architecture Decisions:
- Broker discovery via filer lock mechanism (same as shell commands)
- Graceful fallback when broker unavailable
- ConfigureTopic for CREATE TABLE with 6 default partitions
- Schema versioning ready for ALTER TABLE support

Testing:
- Unit tests updated with filer address parameter
- Integration tests for DDL operations
- Error handling for connection failures

Next Phase: SELECT query execution with Parquet scanning

* fix: Resolve High Priority TODOs - Real MQ Broker Integration

 COMPLETED HIGH PRIORITY TODOs:

🔧 **Real FilerClient Integration** (engine.go:131)
- Implemented GetFilerClient() method in BrokerClient
- Added filerClientImpl with full FilerClient interface compliance
- Added AdjustedUrl() and GetDataCenter() methods
- Real filerClient connection replaces nil fallback

🔧 **Partition Discovery via MQ Broker** (hybrid_message_scanner.go:116)
- Added ListTopicPartitions() method using topic configuration
- Implemented discoverTopicPartitions() in HybridMessageScanner
- Reads actual partition count from BrokerPartitionAssignments
- Generates proper partition ranges based on topic.PartitionCount

📋 **Technical Fixes:**
- Fixed compilation errors with undefined variables
- Proper error handling with filerClientErr variable
- Corrected ConfigureTopicResponse field usage (BrokerPartitionAssignments vs PartitionCount)
- Complete FilerClient interface implementation

🎯 **Impact:**
- SQL engine now connects to real MQ broker infrastructure
- Actual topic partition discovery instead of hardcoded defaults
- Production-ready broker integration with graceful fallbacks
- Maintains backward compatibility with sample data when broker unavailable

 All tests passing - High priority TODO resolution complete!

Next: Schema-aware message parsing and time filter optimization.

* feat: Time Filter Extraction - Complete Performance Optimization

 FOURTH HIGH PRIORITY TODO COMPLETED!

 **Time Filter Extraction & Push-Down Optimization** (engine.go:198-199)
- Replaced hardcoded StartTimeNs=0, StopTimeNs=0 with intelligent extraction
- Added extractTimeFilters() with recursive WHERE clause analysis
- Smart time column detection (\_timestamp_ns, created_at, timestamp, etc.)
- Comprehensive time value parsing (nanoseconds, ISO dates, datetime formats)
- Operator reversal handling (column op value vs value op column)

🧠 **Intelligent WHERE Clause Processing:**
- AND expressions: Combine time bounds (intersection) 
- OR expressions: Skip extraction (safety) 
- Parentheses: Recursive unwrapping 
- Comparison operators: >, >=, <, <=, = 
- Multiple time formats: nanoseconds, RFC3339, date-only, datetime 

🚀 **Performance Impact:**
- Push-down filtering to hybrid scanner level
- Reduced data scanning at source (live logs + Parquet files)
- Time-based partition pruning potential
- Significant performance gains for time-series queries

📊 **Comprehensive Testing (21 tests passing):**
-  Time filter extraction (6 test scenarios)
-  Time column recognition (case-insensitive)
-  Time value parsing (5 formats)
-  Full integration with SELECT queries
-  Backward compatibility maintained

💡 **Real-World Query Examples:**
Before: Scans ALL data, filters in memory
  SELECT * FROM events WHERE \_timestamp_ns > 1672531200000000000;

After: Scans ONLY relevant time range at source level
  → StartTimeNs=1672531200000000000, StopTimeNs=0
  → Massive performance improvement for large datasets!

🎯 **Production Ready Features:**
- Multiple time column formats supported
- Graceful fallbacks for invalid dates
- OR clause safety (avoids incorrect optimization)
- Comprehensive error handling

**ALL MEDIUM PRIORITY TODOs NOW READY FOR NEXT PHASEtest ./weed/query/engine/ -v* 🎉

* feat: Extended WHERE Operators - Complete Advanced Filtering

 **EXTENDED WHERE OPERATORS IMPLEMENTEDtest ./weed/query/engine/ -v | grep -E PASS

* feat: Enhanced SQL CLI Experience

 COMPLETE ENHANCED CLI IMPLEMENTATION:

🚀 **Multiple Execution Modes:**
- Interactive shell with enhanced prompts and context
- Single query execution: --query 'SQL' --output format
- Batch file processing: --file queries.sql --output csv
- Database context switching: --database dbname

📊 **Multi-Format Output:**
- Table format (ASCII) - default for interactive
- JSON format - structured data for programmatic use
- CSV format - spreadsheet-friendly output
- Smart auto-detection based on execution mode

⚙️ **Enhanced Interactive Shell:**
- Database context switching: USE database_name;
- Output format switching: \format table|json|csv
- Command history tracking (basic implementation)
- Enhanced help with WHERE operator examples
- Contextual prompts: seaweedfs:dbname>

🛠️ **Production Features:**
- Comprehensive error handling (JSON + user-friendly)
- Query execution timing and performance metrics
- 30-second timeout protection with graceful handling
- Real MQ integration with hybrid data scanning

📖 **Complete CLI Interface:**
- Full flag support: --server, --interactive, --file, --output, --database, --query
- Auto-detection of execution mode and output format
- Structured help system with practical examples
- Batch processing with multi-query file support

💡 **Advanced WHERE Integration:**
All extended operators (<=, >=, !=, LIKE, IN) fully supported
across all execution modes and output formats.

🎯 **Usage Examples:**
- weed sql --interactive
- weed sql --query 'SHOW DATABASES' --output json
- weed sql --file queries.sql --output csv
- weed sql --database analytics --interactive

Enhanced CLI experience complete - production ready! 🚀

* Delete test_utils_test.go

* fmt

* integer conversion

* show databases works

* show tables works

* Update describe.go

* actual column types

* Update .gitignore

* scan topic messages

* remove emoji

* support aggregation functions

* column name case insensitive, better auto column names

* fmt

* fix reading system fields

* use parquet statistics for optimization

* remove emoji

* parquet file generate stats

* scan all files

* parquet file generation remember the sources also

* fmt

* sql

* truncate topic

* combine parquet results with live logs

* explain

* explain the execution plan

* add tests

* improve tests

* skip

* use mock for testing

* add tests

* refactor

* fix after refactoring

* detailed logs during explain. Fix bugs on reading live logs.

* fix decoding data

* save source buffer index start for log files

* process buffer from brokers

* filter out already flushed messages

* dedup with buffer start index

* explain with broker buffer

* the parquet file should also remember the first buffer_start attribute from the sources

* parquet file can query messages in broker memory, if log files do not exist

* buffer start stored as 8 bytes

* add jdbc

* add postgres protocol

* Revert "add jdbc"

This reverts commit a6e48b7690.

* hook up seaweed sql engine

* setup integration test for postgres

* rename to "weed db"

* return fast on error

* fix versioning

* address comments

* address some comments

* column name can be on left or right in where conditions

* avoid sample data

* remove sample data

* de-support alter table and drop table

* address comments

* read broker, logs, and parquet files

* Update engine.go

* address some comments

* use schema instead of inferred result types

* fix tests

* fix todo

* fix empty spaces and coercion

* fmt

* change to pg_query_go

* fix tests

* fix tests

* fmt

* fix: Enable CGO in Docker build for pg_query_go dependency

The pg_query_go library requires CGO to be enabled as it wraps the libpg_query C library.
Added gcc and musl-dev dependencies to the Docker build for proper compilation.

* feat: Replace pg_query_go with lightweight SQL parser (no CGO required)

- Remove github.com/pganalyze/pg_query_go/v6 dependency to avoid CGO requirement
- Implement lightweight SQL parser for basic SELECT, SHOW, and DDL statements
- Fix operator precedence in WHERE clause parsing (handle AND/OR before comparisons)
- Support INTEGER, FLOAT, and STRING literals in WHERE conditions
- All SQL engine tests passing with new parser
- PostgreSQL integration tests can now build without CGO

The lightweight parser handles the essential SQL features needed for the
SeaweedFS query engine while maintaining compatibility and avoiding CGO
dependencies that caused Docker build issues.

* feat: Add Parquet logical types to mq_schema.proto

Added support for Parquet logical types in SeaweedFS message queue schema:
- TIMESTAMP: UTC timestamp in microseconds since epoch with timezone flag
- DATE: Date as days since Unix epoch (1970-01-01)
- DECIMAL: Arbitrary precision decimal with configurable precision/scale
- TIME: Time of day in microseconds since midnight

These types enable advanced analytics features:
- Time-based filtering and window functions
- Date arithmetic and year/month/day extraction
- High-precision numeric calculations
- Proper time zone handling for global deployments

Regenerated protobuf Go code with new scalar types and value messages.

* feat: Enable publishers to use Parquet logical types

Enhanced MQ publishers to utilize the new logical types:
- Updated convertToRecordValue() to use TimestampValue instead of string RFC3339
- Added DateValue support for birth_date field (days since epoch)
- Added DecimalValue support for precise_amount field with configurable precision/scale
- Enhanced UserEvent struct with PreciseAmount and BirthDate fields
- Added convertToDecimal() helper using big.Rat for precise decimal conversion
- Updated test data generator to produce varied birth dates (1970-2005) and precise amounts

Publishers now generate structured data with proper logical types:
-  TIMESTAMP: Microsecond precision UTC timestamps
-  DATE: Birth dates as days since Unix epoch
-  DECIMAL: Precise amounts with 18-digit precision, 4-decimal scale

Successfully tested with PostgreSQL integration - all topics created with logical type data.

* feat: Add logical type support to SQL query engine

Extended SQL engine to handle new Parquet logical types:
- Added TimestampValue comparison support (microsecond precision)
- Added DateValue comparison support (days since epoch)
- Added DecimalValue comparison support with string conversion
- Added TimeValue comparison support (microseconds since midnight)
- Enhanced valuesEqual(), valueLessThan(), valueGreaterThan() functions
- Added decimalToString() helper for precise decimal-to-string conversion
- Imported math/big for arbitrary precision decimal handling

The SQL engine can now:
-  Compare TIMESTAMP values for filtering (e.g., WHERE timestamp > 1672531200000000000)
-  Compare DATE values for date-based queries (e.g., WHERE birth_date >= 12345)
-  Compare DECIMAL values for precise financial calculations
-  Compare TIME values for time-of-day filtering

Next: Add YEAR(), MONTH(), DAY() extraction functions for date analytics.

* feat: Add window function foundation with timestamp support

Added comprehensive foundation for SQL window functions with timestamp analytics:

Core Window Function Types:
- WindowSpec with PartitionBy and OrderBy support
- WindowFunction struct for ROW_NUMBER, RANK, LAG, LEAD
- OrderByClause for timestamp-based ordering
- Extended SelectStatement to support WindowFunctions field

Timestamp Analytics Functions:
 ApplyRowNumber() - ROW_NUMBER() OVER (ORDER BY timestamp)
 ExtractYear() - Extract year from TIMESTAMP logical type
 ExtractMonth() - Extract month from TIMESTAMP logical type
 ExtractDay() - Extract day from TIMESTAMP logical type
 FilterByYear() - Filter records by timestamp year

Foundation for Advanced Window Functions:
- LAG/LEAD for time-series access to previous/next values
- RANK/DENSE_RANK for temporal ranking
- FIRST_VALUE/LAST_VALUE for window boundaries
- PARTITION BY support for grouped analytics

This enables sophisticated time-series analytics like:
- SELECT *, ROW_NUMBER() OVER (ORDER BY timestamp) FROM user_events WHERE EXTRACT(YEAR FROM timestamp) = 2024
- Trend analysis over time windows
- Session analytics with LAG/LEAD functions
- Time-based ranking and percentiles

Ready for production time-series analytics with proper timestamp logical type support! 🚀

* fmt

* fix

* fix describe issue

* fix tests, avoid panic

* no more mysql

* timeout client connections

* Update SQL_FEATURE_PLAN.md

* handling errors

* remove sleep

* fix splitting multiple SQLs

* fixes

* fmt

* fix

* Update weed/util/log_buffer/log_buffer.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update SQL_FEATURE_PLAN.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* code reuse

* fix

* fix

* feat: Add basic arithmetic operators (+, -, *, /, %) with comprehensive tests

- Implement EvaluateArithmeticExpression with support for all basic operators
- Handle type conversions between int, float, string, and boolean
- Add proper error handling for division/modulo by zero
- Include 14 comprehensive test cases covering all edge cases
- Support mixed type arithmetic (int + float, string numbers, etc.)

All tests passing 

* feat: Add mathematical functions ROUND, CEIL, FLOOR, ABS with comprehensive tests

- Implement ROUND with optional precision parameter
- Add CEIL function for rounding up to nearest integer
- Add FLOOR function for rounding down to nearest integer
- Add ABS function for absolute values with type preservation
- Support all numeric types (int32, int64, float32, double)
- Comprehensive test suite with 20+ test cases covering:
  - Positive/negative numbers
  - Integer/float type preservation
  - Precision handling for ROUND
  - Null value error handling
  - Edge cases (zero, large numbers)

All tests passing 

* feat: Add date/time functions CURRENT_DATE, CURRENT_TIMESTAMP, EXTRACT with comprehensive tests

- Implement CURRENT_DATE returning YYYY-MM-DD format
- Add CURRENT_TIMESTAMP returning TimestampValue with microseconds
- Add CURRENT_TIME returning HH:MM:SS format
- Add NOW() as alias for CURRENT_TIMESTAMP
- Implement comprehensive EXTRACT function supporting:
  - YEAR, MONTH, DAY, HOUR, MINUTE, SECOND
  - QUARTER, WEEK, DOY (day of year), DOW (day of week)
  - EPOCH (Unix timestamp)
- Support multiple input formats:
  - TimestampValue (microseconds)
  - String dates (multiple formats)
  - Unix timestamps (int64 seconds)
- Comprehensive test suite with 15+ test cases covering:
  - All date/time constants
  - Extract from different value types
  - Error handling for invalid inputs
  - Timezone handling

All tests passing 

* feat: Add DATE_TRUNC function with comprehensive tests

- Implement comprehensive DATE_TRUNC function supporting:
  - Time precisions: microsecond, millisecond, second, minute, hour
  - Date precisions: day, week, month, quarter, year, decade, century, millennium
  - Support both singular and plural forms (e.g., 'minute' and 'minutes')
- Enhanced date/time parsing with proper timezone handling:
  - Assume local timezone for non-timezone string formats
  - Support UTC formats with explicit timezone indicators
  - Consistent behavior between parsing and truncation
- Comprehensive test suite with 11 test cases covering:
  - All supported precisions from microsecond to year
  - Multiple input types (TimestampValue, string dates)
  - Edge cases (null values, invalid precisions)
  - Timezone consistency validation

All tests passing 

* feat: Add comprehensive string functions with extensive tests

Implemented String Functions:
- LENGTH: Get string length (supports all value types)
- UPPER/LOWER: Case conversion
- TRIM/LTRIM/RTRIM: Whitespace removal (space, tab, newline, carriage return)
- SUBSTRING: Extract substring with optional length (SQL 1-based indexing)
- CONCAT: Concatenate multiple values (supports mixed types, skips nulls)
- REPLACE: Replace all occurrences of substring
- POSITION: Find substring position (1-based, 0 if not found)
- LEFT/RIGHT: Extract leftmost/rightmost characters
- REVERSE: Reverse string with proper Unicode support

Key Features:
- Robust type conversion (string, int, float, bool, bytes)
- Unicode-safe operations (proper rune handling in REVERSE)
- SQL-compatible indexing (1-based for SUBSTRING, POSITION)
- Comprehensive error handling with descriptive messages
- Mixed-type support (e.g., CONCAT number with string)

Helper Functions:
- valueToString: Convert any schema_pb.Value to string
- valueToInt64: Convert numeric values to int64

Comprehensive test suite with 25+ test cases covering:
- All string functions with typical use cases
- Type conversion scenarios (numbers, booleans)
- Edge cases (empty strings, null values, Unicode)
- Error conditions and boundary testing

All tests passing 

* refactor: Split sql_functions.go into smaller, focused files

**File Structure Before:**
- sql_functions.go (850+ lines)
- sql_functions_test.go (1,205+ lines)

**File Structure After:**
- function_helpers.go (105 lines) - shared utility functions
- arithmetic_functions.go (205 lines) - arithmetic operators & math functions
- datetime_functions.go (170 lines) - date/time functions & constants
- string_functions.go (335 lines) - string manipulation functions
- arithmetic_functions_test.go (560 lines) - tests for arithmetic & math
- datetime_functions_test.go (370 lines) - tests for date/time functions
- string_functions_test.go (270 lines) - tests for string functions

**Benefits:**
 Better organization by functional domain
 Easier to find and maintain specific function types
 Smaller, more manageable file sizes
 Clear separation of concerns
 Improved code readability and navigation
 All tests passing - no functionality lost

**Total:** 7 focused files (1,455 lines) vs 2 monolithic files (2,055+ lines)

This refactoring improves maintainability while preserving all functionality.

* fix: Improve test stability for date/time functions

**Problem:**
- CURRENT_TIMESTAMP test had timing race condition that could cause flaky failures
- CURRENT_DATE test could fail if run exactly at midnight boundary
- Tests were too strict about timing precision without accounting for system variations

**Root Cause:**
- Test captured before/after timestamps and expected function result to be exactly between them
- No tolerance for clock precision differences, NTP adjustments, or system timing variations
- Date boundary race condition around midnight transitions

**Solution:**
 **CURRENT_TIMESTAMP test**: Added 100ms tolerance buffer to account for:
  - Clock precision differences between time.Now() calls
  - System timing variations and NTP corrections
  - Microsecond vs nanosecond precision differences

 **CURRENT_DATE test**: Enhanced to handle midnight boundary crossings:
  - Captures date before and after function call
  - Accepts either date value in case of midnight transition
  - Prevents false failures during overnight test runs

**Testing:**
- Verified with repeated test runs (5x iterations) - all pass consistently
- Full test suite passes - no regressions introduced
- Tests are now robust against timing edge cases

**Impact:**
🚀 **Eliminated flaky test failures** while maintaining function correctness validation
🔧 **Production-ready testing** that works across different system environments
 **CI/CD reliability** - tests won't fail due to timing variations

* heap sort the data sources

* int overflow

* Update README.md

* redirect GetUnflushedMessages to brokers hosting the topic partition

* Update postgres-examples/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* clean up

* support limit with offset

* Update SQL_FEATURE_PLAN.md

* limit with offset

* ensure int conversion correctness

* Update weed/query/engine/hybrid_message_scanner.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* avoid closing closed channel

* support string concatenation ||

* int range

* using consts; avoid test data in production binary

* fix tests

* Update SQL_FEATURE_PLAN.md

* fix "use db"

* address comments

* fix comments

* Update mocks_test.go

* comment

* improve docker build

* normal if no partitions found

* fix build docker

* Update SQL_FEATURE_PLAN.md

* upgrade to raft v1.1.4 resolving race in leader

* raft 1.1.5

* Update SQL_FEATURE_PLAN.md

* Revert "raft 1.1.5"

This reverts commit 5f3bdfadbf.

* Revert "upgrade to raft v1.1.4 resolving race in leader"

This reverts commit fa620f0223.

* Fix data race in FUSE GetAttr operation

- Add shared lock to GetAttr when accessing file handle entries
- Prevents concurrent access between Write (ExclusiveLock) and GetAttr (SharedLock)
- Fixes race on entry.Attributes.FileSize field during concurrent operations
- Write operations already use ExclusiveLock, now GetAttr uses SharedLock for consistency

Resolves race condition:
Write at weedfs_file_write.go:62 vs Read at filechunks.go:28

* Update weed/mq/broker/broker_grpc_query.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* clean up

* Update db.go

* limit with offset

* Update Makefile

* fix id*2

* fix math

* fix string function bugs and add tests

* fix string concat

* ensure empty spaces for literals

* add ttl for catalog

* fix time functions

* unused code path

* database qualifier

* refactor

* extract

* recursive functions

* add cockroachdb parser

* postgres only

* test SQLs

* fix tests

* fix count *

* fix where clause

* fix limit offset

* fix  count fast path

* fix tests

* func name

* fix database qualifier

* fix tests

* Update engine.go

* fix tests

* fix jaeger

https://github.com/advisories/GHSA-2w8w-qhg4-f78j

* remove order by, group by, join

* fix extract

* prevent single quote in the string

* skip control messages

* skip control message when converting to parquet files

* psql change database

* remove old code

* remove old parser code

* rename file

* use db

* fix alias

* add alias test

* compare int64

* fix _timestamp_ns comparing

* alias support

* fix fast path count

* rendering data sources tree

* reading data sources

* reading parquet logic types

* convert logic types to parquet

* go mod

* fmt

* skip decimal types

* use UTC

* add warning if broker fails

* add user password file

* support IN

* support INTERVAL

* _ts as timestamp column

* _ts can compare with string

* address comments

* is null / is not null

* go mod

* clean up

* restructure execution plan

* remove extra double quotes

* fix converting logical types to parquet

* decimal

* decimal support

* do not skip decimal logical types

* making row-building schema-aware and alignment-safe

Emit parquet.NullValue() for missing fields to keep row shapes aligned.
Always advance list level and safely handle nil list values.
Add toParquetValueForType(...) to coerce values to match the declared Parquet type (e.g., STRING/BYTES via byte array; numeric/string conversions for INT32/INT64/DOUBLE/FLOAT/BOOL/TIMESTAMP/DATE/TIME).
Keep nil-byte guards for ByteArray.

* tests for growslice

* do not batch

* live logs in sources can be skipped in execution plan

* go mod tidy

* Update fuse-integration.yml

* Update Makefile

* fix deprecated

* fix deprecated

* remove deep-clean all rows

* broker memory count

* fix FieldIndex

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-09-09 01:01:03 -07:00

894 lines
26 KiB
Go

package postgres
import (
"context"
"encoding/binary"
"fmt"
"io"
"strconv"
"strings"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/schema_pb"
"github.com/seaweedfs/seaweedfs/weed/query/engine"
"github.com/seaweedfs/seaweedfs/weed/query/sqltypes"
"github.com/seaweedfs/seaweedfs/weed/util/sqlutil"
"github.com/seaweedfs/seaweedfs/weed/util/version"
)
// mapErrorToPostgreSQLCode maps SeaweedFS SQL engine errors to appropriate PostgreSQL error codes
func mapErrorToPostgreSQLCode(err error) string {
if err == nil {
return "00000" // Success
}
// Use typed errors for robust error mapping
switch err.(type) {
case engine.ParseError:
return "42601" // Syntax error
case engine.TableNotFoundError:
return "42P01" // Undefined table
case engine.ColumnNotFoundError:
return "42703" // Undefined column
case engine.UnsupportedFeatureError:
return "0A000" // Feature not supported
case engine.AggregationError:
// Aggregation errors are usually function-related issues
return "42883" // Undefined function (aggregation function issues)
case engine.DataSourceError:
// Data source errors are usually access or connection issues
return "08000" // Connection exception
case engine.OptimizationError:
// Optimization failures are usually feature limitations
return "0A000" // Feature not supported
case engine.NoSchemaError:
// Topic exists but no schema available
return "42P01" // Undefined table (treat as table not found)
}
// Fallback: analyze error message for backward compatibility with non-typed errors
errLower := strings.ToLower(err.Error())
// Parsing and syntax errors
if strings.Contains(errLower, "parse error") || strings.Contains(errLower, "syntax") {
return "42601" // Syntax error
}
// Unsupported features
if strings.Contains(errLower, "unsupported") || strings.Contains(errLower, "not supported") {
return "0A000" // Feature not supported
}
// Table/topic not found
if strings.Contains(errLower, "not found") ||
(strings.Contains(errLower, "topic") && strings.Contains(errLower, "available")) {
return "42P01" // Undefined table
}
// Column-related errors
if strings.Contains(errLower, "column") || strings.Contains(errLower, "field") {
return "42703" // Undefined column
}
// Multi-table or complex query limitations
if strings.Contains(errLower, "single table") || strings.Contains(errLower, "join") {
return "0A000" // Feature not supported
}
// Default to generic syntax/access error
return "42000" // Syntax error or access rule violation
}
// handleMessage processes a single PostgreSQL protocol message
func (s *PostgreSQLServer) handleMessage(session *PostgreSQLSession) error {
// Read message type
msgType := make([]byte, 1)
_, err := io.ReadFull(session.reader, msgType)
if err != nil {
return err
}
// Read message length
length := make([]byte, 4)
_, err = io.ReadFull(session.reader, length)
if err != nil {
return err
}
msgLength := binary.BigEndian.Uint32(length) - 4
msgBody := make([]byte, msgLength)
if msgLength > 0 {
_, err = io.ReadFull(session.reader, msgBody)
if err != nil {
return err
}
}
// Process message based on type
switch msgType[0] {
case PG_MSG_QUERY:
return s.handleSimpleQuery(session, string(msgBody[:len(msgBody)-1])) // Remove null terminator
case PG_MSG_PARSE:
return s.handleParse(session, msgBody)
case PG_MSG_BIND:
return s.handleBind(session, msgBody)
case PG_MSG_EXECUTE:
return s.handleExecute(session, msgBody)
case PG_MSG_DESCRIBE:
return s.handleDescribe(session, msgBody)
case PG_MSG_CLOSE:
return s.handleClose(session, msgBody)
case PG_MSG_FLUSH:
return s.handleFlush(session)
case PG_MSG_SYNC:
return s.handleSync(session)
case PG_MSG_TERMINATE:
return io.EOF // Signal connection termination
default:
return s.sendError(session, "08P01", fmt.Sprintf("unknown message type: %c", msgType[0]))
}
}
// handleSimpleQuery processes a simple query message
func (s *PostgreSQLServer) handleSimpleQuery(session *PostgreSQLSession, query string) error {
glog.V(2).Infof("PostgreSQL Query (ID: %d): %s", session.processID, query)
// Add comprehensive error recovery to prevent crashes
defer func() {
if r := recover(); r != nil {
glog.Errorf("Panic in handleSimpleQuery (ID: %d): %v", session.processID, r)
// Try to send error message
s.sendError(session, "XX000", fmt.Sprintf("Internal error: %v", r))
// Try to send ReadyForQuery to keep connection alive
s.sendReadyForQuery(session)
}
}()
// Handle USE database commands for session context
parts := strings.Fields(strings.TrimSpace(query))
if len(parts) >= 2 && strings.ToUpper(parts[0]) == "USE" {
// Re-join the parts after "USE" to handle names with spaces, then trim.
dbName := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(query), parts[0]))
// Unquote if necessary (handle quoted identifiers like "my-database")
if len(dbName) > 1 && dbName[0] == '"' && dbName[len(dbName)-1] == '"' {
dbName = dbName[1 : len(dbName)-1]
} else if len(dbName) > 1 && dbName[0] == '`' && dbName[len(dbName)-1] == '`' {
// Also handle backtick quotes for MySQL/other client compatibility
dbName = dbName[1 : len(dbName)-1]
}
session.database = dbName
s.sqlEngine.GetCatalog().SetCurrentDatabase(dbName)
// Send command complete for USE
err := s.sendCommandComplete(session, "USE")
if err != nil {
return err
}
// Send ReadyForQuery and exit (don't continue processing)
return s.sendReadyForQuery(session)
}
// Set database context in SQL engine if session database is different from current
if session.database != "" && session.database != s.sqlEngine.GetCatalog().GetCurrentDatabase() {
s.sqlEngine.GetCatalog().SetCurrentDatabase(session.database)
}
// Split query string into individual statements to handle multi-statement queries
queries := sqlutil.SplitStatements(query)
// Execute each statement sequentially
for _, singleQuery := range queries {
cleanQuery := strings.TrimSpace(singleQuery)
if cleanQuery == "" {
continue // Skip empty statements
}
// Handle PostgreSQL-specific system queries directly
if systemResult := s.handleSystemQuery(session, cleanQuery); systemResult != nil {
err := s.sendSystemQueryResult(session, systemResult, cleanQuery)
if err != nil {
return err
}
continue // Continue with next statement
}
// Execute using PostgreSQL-compatible SQL engine for proper dialect support
ctx := context.Background()
var result *engine.QueryResult
var err error
// Execute SQL query with panic recovery to prevent crashes
func() {
defer func() {
if r := recover(); r != nil {
glog.Errorf("Panic in SQL execution (ID: %d, Query: %s): %v", session.processID, cleanQuery, r)
err = fmt.Errorf("internal error during SQL execution: %v", r)
}
}()
// Use the main sqlEngine (now uses CockroachDB parser for PostgreSQL compatibility)
result, err = s.sqlEngine.ExecuteSQL(ctx, cleanQuery)
}()
if err != nil {
// Send error message but keep connection alive
errorCode := mapErrorToPostgreSQLCode(err)
sendErr := s.sendError(session, errorCode, err.Error())
if sendErr != nil {
return sendErr
}
// Send ReadyForQuery to keep connection alive
return s.sendReadyForQuery(session)
}
if result.Error != nil {
// Send error message but keep connection alive
errorCode := mapErrorToPostgreSQLCode(result.Error)
sendErr := s.sendError(session, errorCode, result.Error.Error())
if sendErr != nil {
return sendErr
}
// Send ReadyForQuery to keep connection alive
return s.sendReadyForQuery(session)
}
// Send results for this statement
if len(result.Columns) > 0 {
// Send row description
err = s.sendRowDescription(session, result)
if err != nil {
return err
}
// Send data rows
for _, row := range result.Rows {
err = s.sendDataRow(session, row)
if err != nil {
return err
}
}
}
// Send command complete for this statement
tag := s.getCommandTag(cleanQuery, len(result.Rows))
err = s.sendCommandComplete(session, tag)
if err != nil {
return err
}
}
// Send ready for query after all statements are processed
return s.sendReadyForQuery(session)
}
// SystemQueryResult represents the result of a system query
type SystemQueryResult struct {
Columns []string
Rows [][]string
}
// handleSystemQuery handles PostgreSQL system queries directly
func (s *PostgreSQLServer) handleSystemQuery(session *PostgreSQLSession, query string) *SystemQueryResult {
// Trim and normalize query
query = strings.TrimSpace(query)
query = strings.TrimSuffix(query, ";")
queryLower := strings.ToLower(query)
// Handle essential PostgreSQL system queries
switch queryLower {
case "select version()":
return &SystemQueryResult{
Columns: []string{"version"},
Rows: [][]string{{fmt.Sprintf("SeaweedFS %s (PostgreSQL 14.0 compatible)", version.VERSION_NUMBER)}},
}
case "select current_database()":
return &SystemQueryResult{
Columns: []string{"current_database"},
Rows: [][]string{{s.config.Database}},
}
case "select current_user":
return &SystemQueryResult{
Columns: []string{"current_user"},
Rows: [][]string{{"seaweedfs"}},
}
case "select current_setting('server_version')":
return &SystemQueryResult{
Columns: []string{"server_version"},
Rows: [][]string{{fmt.Sprintf("%s (SeaweedFS)", version.VERSION_NUMBER)}},
}
case "select current_setting('server_encoding')":
return &SystemQueryResult{
Columns: []string{"server_encoding"},
Rows: [][]string{{"UTF8"}},
}
case "select current_setting('client_encoding')":
return &SystemQueryResult{
Columns: []string{"client_encoding"},
Rows: [][]string{{"UTF8"}},
}
}
// Handle transaction commands (no-op for read-only)
switch queryLower {
case "begin", "start transaction":
return &SystemQueryResult{
Columns: []string{"status"},
Rows: [][]string{{"BEGIN"}},
}
case "commit":
return &SystemQueryResult{
Columns: []string{"status"},
Rows: [][]string{{"COMMIT"}},
}
case "rollback":
return &SystemQueryResult{
Columns: []string{"status"},
Rows: [][]string{{"ROLLBACK"}},
}
}
// If starts with SET, return a no-op
if strings.HasPrefix(queryLower, "set ") {
return &SystemQueryResult{
Columns: []string{"status"},
Rows: [][]string{{"SET"}},
}
}
// Return nil to use SQL engine
return nil
}
// sendSystemQueryResult sends the result of a system query
func (s *PostgreSQLServer) sendSystemQueryResult(session *PostgreSQLSession, result *SystemQueryResult, query string) error {
// Add panic recovery to prevent crashes in system query results
defer func() {
if r := recover(); r != nil {
glog.Errorf("Panic in sendSystemQueryResult (ID: %d, Query: %s): %v", session.processID, query, r)
// Try to send error and continue
s.sendError(session, "XX000", fmt.Sprintf("Internal error in system query: %v", r))
}
}()
// Create column descriptions for system query results
columns := make([]string, len(result.Columns))
for i, col := range result.Columns {
columns[i] = col
}
// Convert to sqltypes.Value format
var sqlRows [][]sqltypes.Value
for _, row := range result.Rows {
sqlRow := make([]sqltypes.Value, len(row))
for i, cell := range row {
sqlRow[i] = sqltypes.NewVarChar(cell)
}
sqlRows = append(sqlRows, sqlRow)
}
// Send row description (create a temporary QueryResult for consistency)
tempResult := &engine.QueryResult{
Columns: columns,
Rows: sqlRows,
}
err := s.sendRowDescription(session, tempResult)
if err != nil {
return err
}
// Send data rows
for _, row := range sqlRows {
err = s.sendDataRow(session, row)
if err != nil {
return err
}
}
// Send command complete
tag := s.getCommandTag(query, len(result.Rows))
err = s.sendCommandComplete(session, tag)
if err != nil {
return err
}
// Send ready for query
return s.sendReadyForQuery(session)
}
// handleParse processes a Parse message (prepared statement)
func (s *PostgreSQLServer) handleParse(session *PostgreSQLSession, msgBody []byte) error {
// Parse message format: statement_name\0query\0param_count(int16)[param_type(int32)...]
parts := strings.Split(string(msgBody), "\x00")
if len(parts) < 2 {
return s.sendError(session, "08P01", "invalid Parse message format")
}
stmtName := parts[0]
query := parts[1]
// Create prepared statement
stmt := &PreparedStatement{
Name: stmtName,
Query: query,
ParamTypes: []uint32{},
Fields: []FieldDescription{},
}
session.preparedStmts[stmtName] = stmt
// Send parse complete
return s.sendParseComplete(session)
}
// handleBind processes a Bind message
func (s *PostgreSQLServer) handleBind(session *PostgreSQLSession, msgBody []byte) error {
// For now, simple implementation
// In full implementation, would parse parameters and create portal
// Send bind complete
return s.sendBindComplete(session)
}
// handleExecute processes an Execute message
func (s *PostgreSQLServer) handleExecute(session *PostgreSQLSession, msgBody []byte) error {
// Parse portal name
parts := strings.Split(string(msgBody), "\x00")
if len(parts) == 0 {
return s.sendError(session, "08P01", "invalid Execute message format")
}
portalName := parts[0]
// For now, execute as simple query
// In full implementation, would use portal with parameters
glog.V(2).Infof("PostgreSQL Execute portal (ID: %d): %s", session.processID, portalName)
// Send command complete
err := s.sendCommandComplete(session, "SELECT 0")
if err != nil {
return err
}
return nil
}
// handleDescribe processes a Describe message
func (s *PostgreSQLServer) handleDescribe(session *PostgreSQLSession, msgBody []byte) error {
if len(msgBody) < 2 {
return s.sendError(session, "08P01", "invalid Describe message format")
}
objectType := msgBody[0] // 'S' for statement, 'P' for portal
objectName := string(msgBody[1:])
glog.V(2).Infof("PostgreSQL Describe %c (ID: %d): %s", objectType, session.processID, objectName)
// For now, send empty row description
tempResult := &engine.QueryResult{
Columns: []string{},
Rows: [][]sqltypes.Value{},
}
return s.sendRowDescription(session, tempResult)
}
// handleClose processes a Close message
func (s *PostgreSQLServer) handleClose(session *PostgreSQLSession, msgBody []byte) error {
if len(msgBody) < 2 {
return s.sendError(session, "08P01", "invalid Close message format")
}
objectType := msgBody[0] // 'S' for statement, 'P' for portal
objectName := string(msgBody[1:])
switch objectType {
case 'S':
delete(session.preparedStmts, objectName)
case 'P':
delete(session.portals, objectName)
}
// Send close complete
return s.sendCloseComplete(session)
}
// handleFlush processes a Flush message
func (s *PostgreSQLServer) handleFlush(session *PostgreSQLSession) error {
return session.writer.Flush()
}
// handleSync processes a Sync message
func (s *PostgreSQLServer) handleSync(session *PostgreSQLSession) error {
// Reset transaction state if needed
session.transactionState = PG_TRANS_IDLE
// Send ready for query
return s.sendReadyForQuery(session)
}
// sendParameterStatus sends a parameter status message
func (s *PostgreSQLServer) sendParameterStatus(session *PostgreSQLSession, name, value string) error {
msg := make([]byte, 0)
msg = append(msg, PG_RESP_PARAMETER)
// Calculate length
length := 4 + len(name) + 1 + len(value) + 1
lengthBytes := make([]byte, 4)
binary.BigEndian.PutUint32(lengthBytes, uint32(length))
msg = append(msg, lengthBytes...)
// Add name and value
msg = append(msg, []byte(name)...)
msg = append(msg, 0) // null terminator
msg = append(msg, []byte(value)...)
msg = append(msg, 0) // null terminator
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// sendBackendKeyData sends backend key data
func (s *PostgreSQLServer) sendBackendKeyData(session *PostgreSQLSession) error {
msg := make([]byte, 13)
msg[0] = PG_RESP_BACKEND_KEY
binary.BigEndian.PutUint32(msg[1:5], 12)
binary.BigEndian.PutUint32(msg[5:9], session.processID)
binary.BigEndian.PutUint32(msg[9:13], session.secretKey)
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// sendReadyForQuery sends ready for query message
func (s *PostgreSQLServer) sendReadyForQuery(session *PostgreSQLSession) error {
msg := make([]byte, 6)
msg[0] = PG_RESP_READY
binary.BigEndian.PutUint32(msg[1:5], 5)
msg[5] = session.transactionState
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// sendRowDescription sends row description message
func (s *PostgreSQLServer) sendRowDescription(session *PostgreSQLSession, result *engine.QueryResult) error {
msg := make([]byte, 0)
msg = append(msg, PG_RESP_ROW_DESC)
// Calculate message length
length := 4 + 2 // length + field count
for _, col := range result.Columns {
length += len(col) + 1 + 4 + 2 + 4 + 2 + 4 + 2 // name + null + tableOID + attrNum + typeOID + typeSize + typeMod + format
}
lengthBytes := make([]byte, 4)
binary.BigEndian.PutUint32(lengthBytes, uint32(length))
msg = append(msg, lengthBytes...)
// Field count
fieldCountBytes := make([]byte, 2)
binary.BigEndian.PutUint16(fieldCountBytes, uint16(len(result.Columns)))
msg = append(msg, fieldCountBytes...)
// Field descriptions
for i, col := range result.Columns {
// Field name
msg = append(msg, []byte(col)...)
msg = append(msg, 0) // null terminator
// Table OID (0 for no table)
tableOID := make([]byte, 4)
binary.BigEndian.PutUint32(tableOID, 0)
msg = append(msg, tableOID...)
// Attribute number
attrNum := make([]byte, 2)
binary.BigEndian.PutUint16(attrNum, uint16(i+1))
msg = append(msg, attrNum...)
// Type OID (determine from schema if available, fallback to data inference)
typeOID := s.getPostgreSQLTypeFromSchema(result, col, i)
typeOIDBytes := make([]byte, 4)
binary.BigEndian.PutUint32(typeOIDBytes, typeOID)
msg = append(msg, typeOIDBytes...)
// Type size (-1 for variable length)
typeSize := make([]byte, 2)
binary.BigEndian.PutUint16(typeSize, 0xFFFF) // -1 as uint16
msg = append(msg, typeSize...)
// Type modifier (-1 for default)
typeMod := make([]byte, 4)
binary.BigEndian.PutUint32(typeMod, 0xFFFFFFFF) // -1 as uint32
msg = append(msg, typeMod...)
// Format (0 for text)
format := make([]byte, 2)
binary.BigEndian.PutUint16(format, 0)
msg = append(msg, format...)
}
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// sendDataRow sends a data row message
func (s *PostgreSQLServer) sendDataRow(session *PostgreSQLSession, row []sqltypes.Value) error {
msg := make([]byte, 0)
msg = append(msg, PG_RESP_DATA_ROW)
// Calculate message length
length := 4 + 2 // length + field count
for _, value := range row {
if value.IsNull() {
length += 4 // null value length (-1)
} else {
valueStr := value.ToString()
length += 4 + len(valueStr) // field length + data
}
}
lengthBytes := make([]byte, 4)
binary.BigEndian.PutUint32(lengthBytes, uint32(length))
msg = append(msg, lengthBytes...)
// Field count
fieldCountBytes := make([]byte, 2)
binary.BigEndian.PutUint16(fieldCountBytes, uint16(len(row)))
msg = append(msg, fieldCountBytes...)
// Field values
for _, value := range row {
if value.IsNull() {
// Null value
nullLength := make([]byte, 4)
binary.BigEndian.PutUint32(nullLength, 0xFFFFFFFF) // -1 as uint32
msg = append(msg, nullLength...)
} else {
valueStr := value.ToString()
valueLength := make([]byte, 4)
binary.BigEndian.PutUint32(valueLength, uint32(len(valueStr)))
msg = append(msg, valueLength...)
msg = append(msg, []byte(valueStr)...)
}
}
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// sendCommandComplete sends command complete message
func (s *PostgreSQLServer) sendCommandComplete(session *PostgreSQLSession, tag string) error {
msg := make([]byte, 0)
msg = append(msg, PG_RESP_COMMAND)
length := 4 + len(tag) + 1
lengthBytes := make([]byte, 4)
binary.BigEndian.PutUint32(lengthBytes, uint32(length))
msg = append(msg, lengthBytes...)
msg = append(msg, []byte(tag)...)
msg = append(msg, 0) // null terminator
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// sendParseComplete sends parse complete message
func (s *PostgreSQLServer) sendParseComplete(session *PostgreSQLSession) error {
msg := make([]byte, 5)
msg[0] = PG_RESP_PARSE_COMPLETE
binary.BigEndian.PutUint32(msg[1:5], 4)
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// sendBindComplete sends bind complete message
func (s *PostgreSQLServer) sendBindComplete(session *PostgreSQLSession) error {
msg := make([]byte, 5)
msg[0] = PG_RESP_BIND_COMPLETE
binary.BigEndian.PutUint32(msg[1:5], 4)
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// sendCloseComplete sends close complete message
func (s *PostgreSQLServer) sendCloseComplete(session *PostgreSQLSession) error {
msg := make([]byte, 5)
msg[0] = PG_RESP_CLOSE_COMPLETE
binary.BigEndian.PutUint32(msg[1:5], 4)
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// sendError sends an error message
func (s *PostgreSQLServer) sendError(session *PostgreSQLSession, code, message string) error {
msg := make([]byte, 0)
msg = append(msg, PG_RESP_ERROR)
// Build error fields
fields := fmt.Sprintf("S%s\x00C%s\x00M%s\x00\x00", "ERROR", code, message)
length := 4 + len(fields)
lengthBytes := make([]byte, 4)
binary.BigEndian.PutUint32(lengthBytes, uint32(length))
msg = append(msg, lengthBytes...)
msg = append(msg, []byte(fields)...)
_, err := session.writer.Write(msg)
if err == nil {
err = session.writer.Flush()
}
return err
}
// getCommandTag generates appropriate command tag for query
func (s *PostgreSQLServer) getCommandTag(query string, rowCount int) string {
queryUpper := strings.ToUpper(strings.TrimSpace(query))
if strings.HasPrefix(queryUpper, "SELECT") {
return fmt.Sprintf("SELECT %d", rowCount)
} else if strings.HasPrefix(queryUpper, "INSERT") {
return fmt.Sprintf("INSERT 0 %d", rowCount)
} else if strings.HasPrefix(queryUpper, "UPDATE") {
return fmt.Sprintf("UPDATE %d", rowCount)
} else if strings.HasPrefix(queryUpper, "DELETE") {
return fmt.Sprintf("DELETE %d", rowCount)
} else if strings.HasPrefix(queryUpper, "SHOW") {
return fmt.Sprintf("SELECT %d", rowCount)
} else if strings.HasPrefix(queryUpper, "DESCRIBE") || strings.HasPrefix(queryUpper, "DESC") {
return fmt.Sprintf("SELECT %d", rowCount)
}
return "SELECT 0"
}
// getPostgreSQLTypeFromSchema determines PostgreSQL type OID from schema information first, fallback to data
func (s *PostgreSQLServer) getPostgreSQLTypeFromSchema(result *engine.QueryResult, columnName string, colIndex int) uint32 {
// Try to get type from schema if database and table are available
if result.Database != "" && result.Table != "" {
if tableInfo, err := s.sqlEngine.GetCatalog().GetTableInfo(result.Database, result.Table); err == nil {
if tableInfo.Schema != nil && tableInfo.Schema.RecordType != nil {
// Look for the field in the schema
for _, field := range tableInfo.Schema.RecordType.Fields {
if field.Name == columnName {
return s.mapSchemaTypeToPostgreSQL(field.Type)
}
}
}
}
}
// Handle system columns
switch columnName {
case "_timestamp_ns":
return PG_TYPE_INT8 // PostgreSQL BIGINT for nanosecond timestamps
case "_key":
return PG_TYPE_BYTEA // PostgreSQL BYTEA for binary keys
case "_source":
return PG_TYPE_TEXT // PostgreSQL TEXT for source information
}
// Fallback to data-based inference if schema is not available
return s.getPostgreSQLTypeFromData(result.Columns, result.Rows, colIndex)
}
// mapSchemaTypeToPostgreSQL maps SeaweedFS schema types to PostgreSQL type OIDs
func (s *PostgreSQLServer) mapSchemaTypeToPostgreSQL(fieldType *schema_pb.Type) uint32 {
if fieldType == nil {
return PG_TYPE_TEXT
}
switch kind := fieldType.Kind.(type) {
case *schema_pb.Type_ScalarType:
switch kind.ScalarType {
case schema_pb.ScalarType_BOOL:
return PG_TYPE_BOOL
case schema_pb.ScalarType_INT32:
return PG_TYPE_INT4
case schema_pb.ScalarType_INT64:
return PG_TYPE_INT8
case schema_pb.ScalarType_FLOAT:
return PG_TYPE_FLOAT4
case schema_pb.ScalarType_DOUBLE:
return PG_TYPE_FLOAT8
case schema_pb.ScalarType_BYTES:
return PG_TYPE_BYTEA
case schema_pb.ScalarType_STRING:
return PG_TYPE_TEXT
default:
return PG_TYPE_TEXT
}
case *schema_pb.Type_ListType:
// For list types, we'll represent them as JSON text
return PG_TYPE_JSONB
case *schema_pb.Type_RecordType:
// For nested record types, we'll represent them as JSON text
return PG_TYPE_JSONB
default:
return PG_TYPE_TEXT
}
}
// getPostgreSQLTypeFromData determines PostgreSQL type OID from data (legacy fallback method)
func (s *PostgreSQLServer) getPostgreSQLTypeFromData(columns []string, rows [][]sqltypes.Value, colIndex int) uint32 {
if len(rows) == 0 || colIndex >= len(rows[0]) {
return PG_TYPE_TEXT // Default to text
}
// Sample first non-null value to determine type
for _, row := range rows {
if colIndex < len(row) && !row[colIndex].IsNull() {
value := row[colIndex]
switch value.Type() {
case sqltypes.Int8, sqltypes.Int16, sqltypes.Int32:
return PG_TYPE_INT4
case sqltypes.Int64:
return PG_TYPE_INT8
case sqltypes.Float32, sqltypes.Float64:
return PG_TYPE_FLOAT8
case sqltypes.Bit:
return PG_TYPE_BOOL
case sqltypes.Timestamp, sqltypes.Datetime:
return PG_TYPE_TIMESTAMP
default:
// Try to infer from string content
valueStr := value.ToString()
if _, err := strconv.ParseInt(valueStr, 10, 32); err == nil {
return PG_TYPE_INT4
}
if _, err := strconv.ParseInt(valueStr, 10, 64); err == nil {
return PG_TYPE_INT8
}
if _, err := strconv.ParseFloat(valueStr, 64); err == nil {
return PG_TYPE_FLOAT8
}
if valueStr == "true" || valueStr == "false" {
return PG_TYPE_BOOL
}
return PG_TYPE_TEXT
}
}
}
return PG_TYPE_TEXT // Default to text
}