mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2025-11-08 23:46:14 +08:00
fix: Reduce manual commit frequency from every 10 to every 100 messages
PROBLEM: Consumer throughput still 45.46 msgs/sec vs producer 50.29 msgs/sec (10% gap). ROOT CAUSE: Manual session.Commit() every 10 messages creates excessive overhead: - 1,880 messages consumed → 188 commit operations - Each commit is SYNCHRONOUS and blocks message processing - Auto-commit is already enabled (5s interval) - Double-committing reduces effective throughput ANALYSIS: - Test showed consumer lag at 0 at end (not falling behind) - Only ~1,880 of 12,200 messages consumed during 2-minute window - Consumers start 2s late, need ~262s to consume all at current rate - Commit overhead: 188 RPC round trips = significant latency FIX: Reduce manual commit frequency from every 10 to every 100 messages: - Only 18-20 manual commits during entire test - Auto-commit handles primary offset persistence (5s interval) - Manual commits serve as backup for edge cases - Unblocks message processing loop for higher throughput EXPECTED IMPACT: - Consumer throughput: 45.46 → ~49+ msgs/sec (match producer!) - Latency reduction: Fewer synchronous commits - Test duration: Should consume all messages before test ends
This commit is contained in:
@@ -647,9 +647,10 @@ func (h *ConsumerGroupHandler) ConsumeClaim(session sarama.ConsumerGroupSession,
|
||||
// Mark message as processed
|
||||
session.MarkMessage(message, "")
|
||||
|
||||
// Commit offset immediately every 10 messages to reduce duplicate window
|
||||
// This supplements auto-commit and ensures offsets are committed more frequently
|
||||
if msgCount%10 == 0 {
|
||||
// Commit offset less frequently to reduce overhead
|
||||
// Auto-commit is enabled (5s interval by default), so manual commits are just backups
|
||||
// Reduced from every 10 messages to every 100 messages to match throughput
|
||||
if msgCount%100 == 0 {
|
||||
session.Commit()
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user