lsm/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-11-08 16:51:59 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

tune: Adjust commit frequency to every 20 messages for optimal balance

Browse Source 
Testing showed every 50 messages too aggressive (43.6% duplicates).
Every 10 messages creates too much overhead.

Every 20 messages provides good middle ground:
  - ~600 commits per 12k messages (manageable overhead)
  - ~20 message loss window if consumer crashes
  - Balanced duplicate/missing ratio
This commit is contained in:
chrislu
2025-10-16 23:36:48 -07:00
parent ebdabdc8fe
commit 133698a9c4
1 changed files with 6 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
test/kafka/kafka-client-loadtest/internal/consumer/consumer.go
View File

@@ -647,12 +647,12 @@ func (h *ConsumerGroupHandler) ConsumeClaim(session sarama.ConsumerGroupSession,
// Mark message as processed // Mark message as processed
session.MarkMessage(message, "") session.MarkMessage(message, "")
// Commit offset periodically to minimize message loss // Commit offset frequently to minimize both message loss and duplicates
// Every 50 messages provides good balance: // Every 20 messages balances:
// - 10s overhead reduction vs every 10 messages // - ~600 commits per 12k messages (reasonable overhead)
// - ~50 message loss window vs 100 if consumer fails // - ~20 message loss window if consumer fails
// Auto-commit (100ms) helps in background, manual commits are failsafe // - Reduces duplicate reads from rebalancing
if msgCount%50 == 0 { if msgCount%20 == 0 {
session.Commit() session.Commit()
} }
} }