NyaMisty
f894e7b7a5
Support filtering source disk type in volume.tier.upload ( #6868 )
2025-06-15 20:30:04 -07:00
NyaMisty
53e5c84523
Fix wrong error handling in volume.tier.upload when stream == nil but copyErr != nil ( #6867 )
2025-06-15 20:28:40 -07:00
NyaMisty
cdc543aa9e
Correctly sort in volume.list to ensure output consistency ( #6866 )
2025-06-15 20:27:48 -07:00
Aleksey Kosov
283d9e0079
Add context with request ( #6824 )
2025-05-28 11:34:02 -07:00
Aleksey Kosov
165af32d6b
added context to filer_client method calls ( #6808 )
...
Co-authored-by: akosov <a.kosov@kryptonite.ru >
2025-05-22 09:46:49 -07:00
Lisandro Pin
0be020b0fa
Nit: unify the default --maxParallelization value for weed shell commands supporting this option ( #6788 )
2025-05-13 07:59:26 -07:00
Lisandro Pin
ba1d82db90
Move shell.ErrorWaitGroup into a common file, to cleanly reuse across weed shell commands. ( #6780 )
...
Move `shell.ErrorWaitGroup` into a dedicated common file, to cleanly reuse across `weed shell` commands.
2025-05-12 14:38:55 -07:00
Lisandro Pin
848d1f7c34
Improve safety for weed shell's ec.encode. ( #6773 )
...
Improve safety for weed shells `ec.encode`.
The current process for `ec.encode` is:
1. EC shards for a volume are generated and added to a single server
2. The original volume is deleted
3. EC shards get re-balanced across the entire topology
It is then possible to lose data between #2 and #3 , if the underlying volume storage/server/rack/DC
happens to fail, for whatever reason. As a fix, this MR reworks `ec.encode` so:
* Newly created EC shards are spread across all locations for the source volume.
* Source volumes are deleted only after EC shards are converted and balanced.
2025-05-09 09:01:32 -07:00
Lisandro Pin
97dad06ed8
Improve parallelization for ec.encode ( #6769 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Improve parallelization for `ec.encode`.
Instead of processing one volume at at time, perform all EC conversion
steps (mark readonly -> generate EC shards -> delete volume -> remount) in
parallel for all of them.
This should substantially improve performance when EC encoding
entire collections.
2025-05-08 17:14:14 -07:00
NyaMisty
8d0e6f1ead
fix: volume.list volume info output not in order ( #6737 )
2025-04-27 08:52:49 -07:00
dependabot[bot]
216c52e377
chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 ( #6679 )
...
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0
Bumps [gocloud.dev](https://github.com/google/go-cloud ) from 0.40.0 to 0.41.0.
- [Release notes](https://github.com/google/go-cloud/releases )
- [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0 )
---
updated-dependencies:
- dependency-name: gocloud.dev
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
* fix error
* fix printing errors
* Update go.mod
---------
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com >
2025-03-31 21:42:54 -07:00
chrislu
c45b8bd6ac
add more help message
...
fix https://github.com/seaweedfs/seaweedfs/issues/6625
2025-03-13 09:11:16 -07:00
Lisandro Pin
c07596691c
ec.encode: Fix resolution of target collections. (#6585 )
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
* Don't ignore empty (`""`) collection names when computing collections for a given volume ID.
* `ec.encode`: Fix resolution of target collections.
When no `volumeId` parameter is provided, compute volumes
based on the provided collection name, even if it's empty (`""`).
This restores behavior to before recent EC rebalancing rework. See also
ec30a504ba/weed/shell/command_ec_encode.go (L99)
2025-02-28 11:42:19 -08:00
Lisandro Pin
76a111f0a2
Fix calculation of node's free EC shard slots. ( #6584 )
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
2025-02-28 07:35:28 -08:00
Changrui Chen
be74548cb5
fix: error info size bug in command_fs_merge_volumes.go ( #6567 )
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
2025-02-23 06:36:47 -08:00
Lisandro Pin
392656d59e
ec.encode: Explictly mount EC shards after volume conversion. (#6528 )
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
This guarantees EC shards are immediately available after encoding,
even if not affected by subsequent re-balancing.
2025-02-10 09:49:58 -08:00
Lisandro Pin
e8d8bfcccc
Nit: remove missing newlines on weed shell commands output. ( #6524 )
...
Nit: remove missing newlines on `weed` commands output.
2025-02-07 10:27:04 -08:00
Lisandro Pin
29c2d9b965
Remove warning on EC balancing if no replica placement settings are found. ( #6516 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Effectively undoes c9399a68; with ff8bd862chrislusf@users.noreply.github.com >
2025-02-06 09:19:28 -08:00
Lisandro Pin
68f547bdf2
Nit: fix missing newline on EC balancing warnings regarding replica settings ( #6509 )
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
Nit: fix missing newline on EC balancing warnings regarding replica settings.
See 79136812
2025-02-04 10:59:25 -08:00
Lisandro Pin
331c1f0f3f
Improve EC shards balancing logic regarding replica placement settings. ( #6491 )
...
The replica placement type specifies numebr of _replicas_ on the same/different rack;
that means we can have one EC shard copy on each, even if the replica setting is zero.
This PR reworks replica placement parsing for EC rebalancing, so we check allow
(replica placement + 1) when selecting racks and nodes to balance EC shards into.
2025-01-30 09:26:45 -08:00
Lisandro Pin
250fbbb3db
ec.balance: Allow EC balancing without collections. (#6488 )
2025-01-29 08:51:59 -08:00
Lisandro Pin
7913681297
ec.encode: Display a warning on EC balancing if no replica placement settings are found. (#6487 )
2025-01-29 08:50:19 -08:00
Chris Lu
cc05874d06
Add message queue agent ( #6463 )
...
* scaffold message queue agent
* adjust proto, add mq_agent
* add agent client implementation
* remove unused function
* agent publish server implementation
* adding agent
2025-01-20 22:19:27 -08:00
Hadi Zamani
b2f56d9add
Add JWT authentication to fs.mergeVolumes command ( #6461 )
...
Add jwt authentication to fs.mergeVolumes command
2025-01-20 18:16:46 -08:00
Lisandro Pin
eab2e0e112
ec.encode: Fix bug causing source volumes not being deleted after EC conversion. (#6447 )
...
This logic was originally part of `spreadEcShards()`, which got removed during
the unification effort with `ec.balance` (https://github.com/seaweedfs/seaweedfs/pull/6344 ),
accidentally breaking functionality in the process.
The commit restores the deletion code for EC'd volumes - with parallelization support.
2025-01-17 01:02:30 -08:00
ftong2020
e7f2936dcc
fix force arg dropped during volume balance command ( #6432 )
2025-01-12 23:30:18 -08:00
dsd
da2a234b00
[weed] change -n to -force ( #6421 )
2025-01-08 09:57:18 -08:00
Brad Murray
bc3640ee64
Update command_fs_merge_volumes.go ( #6406 )
2025-01-02 08:57:26 -08:00
Guang Jiong Lou
3b1ac77e1f
worm grace period and retention time support ( #6404 )
...
Signed-off-by: lou <alex1988@outlook.com >
2024-12-31 18:41:43 -08:00
dsd
20cbc9e4eb
skip error while executing volume.fix.replication ( #6382 )
2024-12-20 07:36:13 -08:00
chrislu
ec155022e7
"golang.org/x/exp/slices" => "slices" and go fmt
2024-12-19 19:25:06 -08:00
Lisandro Pin
4d91ec359b
Fix volume replica parallelization within ec.encode. ( #6377 )
...
See 826edd5d
2024-12-19 17:46:11 -08:00
Lisandro Pin
ba0707af64
Allow configuring the maximum number of concurrent tasks for EC parallelization. ( #6376 )
...
Follow-up to b0210df0
2024-12-18 13:26:26 -08:00
Lisandro Pin
44c48c929a
Parallelize volume replica operations within ec.encode. ( #6374 )
2024-12-18 11:59:48 -08:00
Lisandro Pin
9fbc4ea417
Rework shell.EcBalance()'s waitgroup code into a standalone type. ( #6373 )
...
Rework `shell.EcBalance()`'s waitgroup with errors code into a standalone type.
We'll re-use this for other EC jobs - for example, volume creation. Also fixes
potential concurrency issues when collecting error results.
2024-12-17 09:39:51 -08:00
Lisandro Pin
9b48ce0613
Parallelize EC shards balancing within racks ( #6354 )
...
Parallelize EC shards balancing within racks.
2024-12-15 13:36:23 -08:00
Lisandro Pin
926cfea3dc
Parallelize EC shards balancing across racks. ( #6352 )
2024-12-13 06:05:32 -08:00
Lisandro Pin
b81def5e5c
Parallelize EC balancing for racks. ( #6351 )
2024-12-13 05:33:53 -08:00
Konstantin Lebedev
0a4b1909a2
[shell] only apply the balancing for writable volumes ( #6346 )
2024-12-13 01:10:00 -08:00
Lisandro Pin
b0210df081
Begin implementing EC balancing parallelization support. ( #6342 )
...
* Begin implementing EC balancing parallelization support.
Impacts both `ec.encode` and `ec.balance`,
* Nit: improve type naming.
* Make the goroutine workgroup handler for `EcBalance()` a bit smarter/error-proof.
* Nit: unify naming for `ecBalancer` wait group methods with the rest of the module.
* Fix concurrency bug.
* Fix whitespace after Gitlab automerge.
* Delete stray TODO.
2024-12-12 09:14:44 -08:00
Lisandro Pin
23ffbb083c
Limit EC re-balancing for ec.encode to relevant collections when a volume ID argument is provided. ( #6347 )
...
Limit EC re-balancing for `ec.encode` to relevant collections when a volume ID is provided.
2024-12-12 08:41:33 -08:00
Lisandro Pin
6320036c56
Delete legacy balancing code for ec.encode. ( #6344 )
2024-12-12 07:42:03 -08:00
Konstantin Lebedev
700b95304b
[shell] volume.list show only writable volumes ( #6338 )
...
* show only writable volumes
* fix import
2024-12-11 09:06:15 -08:00
Konstantin Lebedev
c37281735e
volume.list avoid output empty data center and rack and disk info ( #6341 )
2024-12-11 09:03:16 -08:00
Lisandro Pin
8c82c037b9
Unify the re-balancing logic for ec.encode with ec.balance. ( #6339 )
...
Among others, this enables recent changes related to topology aware
re-balancing at EC encoding time.
2024-12-10 13:30:13 -08:00
Konstantin Lebedev
ff1392f7f4
[shell] use constant for hdd of type ( #6337 )
...
use constant for hdd of type
2024-12-10 08:43:59 -08:00
Lisandro Pin
522a25790a
Remove average constraints when selecting nodes/racks to balance EC shards into. ( #6325 )
2024-12-06 09:00:06 -08:00
Lisandro Pin
34cdbdd279
Share common parameters for EC re-balancing functions under a single struct. ( #6319 )
...
TODO cleanup for https://github.com/seaweedfs/seaweedfs/discussions/6179 .
2024-12-05 09:00:46 -08:00
Lisandro Pin
edef485333
Account for replication placement settings when balancing EC shards within the same rack. ( #6317 )
...
* Account for replication placement settings when balancing EC shards within racks.
* Update help contents for `ec.balance`.
* Add a few more representative test cases for `pickEcNodeToBalanceShardsInto()`.
2024-12-04 10:47:51 -08:00
Lisandro Pin
351efa134d
Account for replication placement settings when balancing EC shards across racks. ( #6316 )
2024-12-04 09:00:55 -08:00
