Commit Graph

7624 Commits

Author SHA1 Message Date
chrislu
96a2749355 adding EcVacuumTaskConfig 2025-08-11 16:59:55 -07:00
chrislu
f9e7ca4308 Merge branch 'master' into add-ec-vacuum 2025-08-11 16:45:45 -07:00
chrislu
7889e78f4d use one http client 2025-08-11 16:45:13 -07:00
Chris Lu
af3300e063 filer: server side copying (#7121)
* copy

* address comments

* remove unused functions, reuse http clients

* address hardlink, checking existing directory

* destination is directory

* check for the key's existence in the map first before accessing its members

* address comments

* deep copy remote entry

* address comments

* copying chunks in parallel

* handle manifest chunks

* address comments

* errgroup

* there could be large chunks

* address comments

* address comments
2025-08-11 16:40:46 -07:00
Chris Lu
605b3333c1 fix sftp start with filer (#7120)
* fix sftp start with filer

* add bindIp
2025-08-11 10:56:40 -07:00
Chris Lu
9265e81fe9 S3 API: unsigned streaming (no cred) but chunks contain signatures (#7118)
* This handles the case where we have unsigned streaming (no cred) but chunks contain signatures

* Update chunked_reader_v4.go

* address comments
2025-08-11 10:31:01 -07:00
chrislu
a265a07922 fix ttl objects
fix https://github.com/seaweedfs/seaweedfs/discussions/7107#discussioncomment-14069018
2025-08-11 02:20:02 -07:00
chrislu
f03f4383af fix missing grpc dial option 2025-08-11 02:19:22 -07:00
chrislu
c57ccd554a worker_pb.TaskParams_VacuumParams 2025-08-11 02:11:39 -07:00
chrislu
9df006b49d fix min volume age 2025-08-11 01:54:10 -07:00
chrislu
424bb7fe11 showing criteria 2025-08-11 01:34:33 -07:00
chrislu
97d58e77c6 getting ec volume deletions 2025-08-11 01:03:21 -07:00
chrislu
553a229fd3 reduce lock scope 2025-08-11 00:21:04 -07:00
chrislu
57d025910d cancel context 2025-08-11 00:20:52 -07:00
chrislu
a31fc1ac01 Bypass compaction revision check to handle volumes compacted after task creation 2025-08-10 20:39:48 -07:00
chrislu
91d641e685 avoid dead lock 2025-08-10 20:23:18 -07:00
chrislu
0e649f710a collect deletion for ec shards 2025-08-10 20:08:45 -07:00
chrislu
772ee0f967 address comments 2025-08-10 19:58:52 -07:00
chrislu
0c6980182c ec shards with generation 2025-08-10 19:40:35 -07:00
chrislu
47ea1ac228 unmount 2025-08-10 19:39:24 -07:00
chrislu
6446893e3c metrics 2025-08-10 19:39:11 -07:00
chrislu
06c012ea60 Update maintenance_scanner.go 2025-08-10 19:37:54 -07:00
chrislu
5a6954be1b sort 2025-08-10 18:42:27 -07:00
chrislu
0f1ca16457 more accurate estimation 2025-08-10 18:42:22 -07:00
chrislu
72f0a47563 CRITICAL: Check ALL task states for volume conflicts
Fix major scheduling bug where only active tasks were checked for conflicts.

Changes:
- Check PENDING tasks: Prevent scheduling if task is queued for same volume
- Check ASSIGNED/ACTIVE tasks: Prevent scheduling if task is running on same volume
- Check RECENT tasks: Prevent immediate re-scheduling on same volume after completion

This prevents dangerous scenarios like:
 Scheduling vacuum while another vacuum is pending on same volume
 Scheduling balance while erasure coding is queued for same volume
 Immediately re-scheduling failed tasks without cooldown period

Critical safety improvement ensuring comprehensive volume-level task isolation.
2025-08-10 18:04:31 -07:00
chrislu
751cfac7d7 Implement volume-aware task conflict checking
MAJOR IMPROVEMENT: Tasks now conflict by volume ID, not globally by task type

Changes:
- PRIMARY RULE: Tasks on the same volume ID always conflict (prevents race conditions)
- SECONDARY RULE: Minimal global task type conflicts (currently none)
- Add isDiskAvailableForVolume() for volume-specific availability checking
- Add GetAvailableDisksForVolume() and GetDisksWithEffectiveCapacityForVolume()
- Remove overly restrictive global task type conflicts
- Update planning functions to focus on capacity, not conflicts

Benefits:
 Multiple vacuum tasks can run on different volumes simultaneously
 Balance and erasure coding can run on different volumes
 Still prevents dangerous concurrent operations on same volume
 Much more efficient resource utilization
 Maintains data integrity and prevents race conditions

This addresses the user feedback that task conflicts should be volume-specific,
not global task type restrictions.
2025-08-10 18:02:42 -07:00
chrislu
5c1e6e904d CRITICAL: Restore task conflict definitions to prevent data integrity issues
- Restore conflicts between vacuum, balance, erasure_coding, and ec_vacuum tasks
- Prevent dangerous concurrent operations on same volumes/resources
- Add comprehensive task conflict matrix to avoid race conditions
- This addresses a serious safety regression where all conflicts were removed

Critical conflicts restored:
- vacuum ↔ balance, erasure_coding, ec_vacuum
- balance ↔ vacuum, erasure_coding, ec_vacuum
- erasure_coding ↔ vacuum, balance, ec_vacuum
- ec_vacuum ↔ vacuum, balance, erasure_coding
- replication ↔ vacuum, balance (destructive ops)
2025-08-10 17:56:13 -07:00
chrislu
c220ad1e69 Replace bubble sort with idiomatic sort.Slice in EC shard management
- Replace O(n²) bubble sort implementation with efficient sort.Slice
- More concise, readable, and performant for larger slices
- Uses idiomatic Go sorting pattern
2025-08-10 17:52:30 -07:00
chrislu
c9ca213d4c Fix linter error in master_grpc_ec_generation_test.go
Remove orphaned code that was causing syntax error
2025-08-10 17:51:28 -07:00
chrislu
4ec743583d Address PR #7116 review comments
- Fix CodeQL security issue: Add bounds checking for int64 to uint8 conversion in disk_location_ec.go
- Replace goto with idiomatic map approach in ec_shard_management.go
- Fix EC volume handling in maintenance_scanner.go: add support for EC-only volumes
- Fix test failures in master_grpc_ec_generation_test.go: handle raft leadership issues
2025-08-10 17:50:10 -07:00
chrislu
5bb475c572 Fixed CodeQL Security Issue 2025-08-10 17:48:29 -07:00
chrislu
cd75202da8 Replaced goto with Idiomatic Map Approach 2025-08-10 17:48:18 -07:00
chrislu
cdba261913 add safety check 2025-08-10 17:41:55 -07:00
chrislu
179bcb777e Delete ec_vacuum_integration_test.go 2025-08-10 17:34:03 -07:00
chrislu
802267b281 validates all critical upgrade scenarios for the generation-aware EC vacuum system 2025-08-10 17:33:08 -07:00
chrislu
d6e335d6e2 Integration test: end-to-end vacuum G→G+1 with live reads (zero downtime) 2025-08-10 17:23:17 -07:00
chrislu
884da0496c testing mixed generation 2025-08-10 17:12:20 -07:00
chrislu
60f975aa20 add unit tests 2025-08-10 17:07:29 -07:00
chrislu
a3d0509c92 logs 2025-08-10 16:57:08 -07:00
chrislu
2e51e1dab2 ec volume UI rendering version 2025-08-10 16:54:44 -07:00
chrislu
a187f103d1 normal volume CompactionRevision 2025-08-10 16:22:36 -07:00
chrislu
3087da07db metrics with generation 2025-08-10 16:17:46 -07:00
chrislu
3ef8a9f3b2 Mixed-version cluster compatibility 2025-08-10 15:54:30 -07:00
chrislu
d326affc4f default generation 2025-08-10 15:44:08 -07:00
chrislu
d341db8861 Update ec_vacuum_task.go 2025-08-10 15:39:26 -07:00
chrislu
cf5f0b8e74 cleanupGracePeriod 2025-08-10 15:13:12 -07:00
chrislu
0bcb64140f Add activation step to EC vacuum: call ActivateEcGeneration when G+1 ready 2025-08-10 15:07:57 -07:00
chrislu
62d89fa60b EC vacuum distribution updated for generation-aware mount/copy RPCs 2025-08-10 15:04:55 -07:00
chrislu
de9399761b EC vacuum task updated for generation-aware operation 2025-08-10 15:00:22 -07:00
chrislu
8797e73523 cachedLookupEcShardLocations updated for generation-specific caching 2025-08-10 14:51:55 -07:00