Lisandro Pin
29c2d9b965
Remove warning on EC balancing if no replica placement settings are found. ( #6516 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Effectively undoes c9399a68; with ff8bd862chrislusf@users.noreply.github.com >
2025-02-06 09:19:28 -08:00
Tom Crasset
a250699225
use X-Forwarded-Host and X-Forwarded-Port to determine correct host for AWS signature ( #6514 )
2025-02-06 07:54:50 -08:00
Lisandro Pin
68f547bdf2
Nit: fix missing newline on EC balancing warnings regarding replica settings ( #6509 )
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
Nit: fix missing newline on EC balancing warnings regarding replica settings.
See 79136812
2025-02-04 10:59:25 -08:00
chrislu
95d8f4415a
3.84
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
helm: lint and test charts / lint-test (push) Has been cancelled
2025-02-03 09:25:42 -08:00
chrislu
7fd42f5d06
Revert "Add content length to filer's requests ( #6489 )"
...
This reverts commit 9f42ce3f01
2025-02-03 09:24:14 -08:00
chrislu
5c3d7e6ae6
3.83
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
helm: lint and test charts / lint-test (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
2025-02-01 17:41:39 -08:00
chrislu
42efade0dc
adjust fix
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
fix https://github.com/seaweedfs/seaweedfs/issues/6497
2025-02-01 14:11:57 -08:00
chrislu
a75271dd43
ensure correct auto bucket creation
...
fix https://github.com/seaweedfs/seaweedfs/issues/6497
2025-02-01 13:26:12 -08:00
Lisandro Pin
fc4df944a0
Remove rate limit semaphore on master's leader selection logic. ( #6494 )
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
This was introduced by 054374c7
2025-01-30 13:08:36 -08:00
Lisandro Pin
331c1f0f3f
Improve EC shards balancing logic regarding replica placement settings. ( #6491 )
...
The replica placement type specifies numebr of _replicas_ on the same/different rack;
that means we can have one EC shard copy on each, even if the replica setting is zero.
This PR reworks replica placement parsing for EC rebalancing, so we check allow
(replica placement + 1) when selecting racks and nodes to balance EC shards into.
2025-01-30 09:26:45 -08:00
Tom Crasset
551baa44b2
verify if bucket exists in GetBucketLocationHandler ( #6485 )
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
2025-01-29 08:55:08 -08:00
Hadi Zamani
9f42ce3f01
Add content length to filer's requests ( #6489 )
2025-01-29 08:53:39 -08:00
Lisandro Pin
250fbbb3db
ec.balance: Allow EC balancing without collections. (#6488 )
2025-01-29 08:51:59 -08:00
Lisandro Pin
7913681297
ec.encode: Display a warning on EC balancing if no replica placement settings are found. (#6487 )
2025-01-29 08:50:19 -08:00
chrislu
33ba88df9c
fix from ensure() before actual deletion, within the b2 client library
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
fix https://github.com/seaweedfs/seaweedfs/issues/6483
2025-01-28 23:59:42 -08:00
Tom Crasset
7c3a0ed874
return error on invalid action in PutUserPolicy ( #6482 )
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
2025-01-28 05:42:03 -08:00
Hadi Zamani
a2330f624b
Add metrics for uploaded and deleted s3 objects ( #6475 )
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
2025-01-25 21:55:06 -08:00
chrislu
be15fee8e7
show error if backend is mis-configured
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
related to https://github.com/seaweedfs/seaweedfs/discussions/6472
2025-01-23 09:18:54 -08:00
MaratKarimov
4fe1248aa8
filer store YDB: fix upsert with TTL ( #6469 )
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
2025-01-22 07:45:54 -08:00
ludwigxia
5452405a81
cloud tier: support for Alibaba Cloud OSS ( #6466 )
2025-01-22 00:48:55 -08:00
chrislu
ca4dca14d8
3.82
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
helm: lint and test charts / lint-test (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
2025-01-20 22:33:23 -08:00
Chris Lu
cc05874d06
Add message queue agent ( #6463 )
...
* scaffold message queue agent
* adjust proto, add mq_agent
* add agent client implementation
* remove unused function
* agent publish server implementation
* adding agent
2025-01-20 22:19:27 -08:00
Hadi Zamani
b2f56d9add
Add JWT authentication to fs.mergeVolumes command ( #6461 )
...
Add jwt authentication to fs.mergeVolumes command
2025-01-20 18:16:46 -08:00
Tom Crasset
c5f21b2b01
fix S3 per-user-directory Policy ( #6443 )
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
* fix S3 per-user-directory Policy
* Delete docker/config.json
* add tests
* remove logs
* undo modifications of weed/shell/command_volume_balance.go
* remove modifications of docker-compose
* fix failing test
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com >
2025-01-17 01:03:17 -08:00
Lisandro Pin
eab2e0e112
ec.encode: Fix bug causing source volumes not being deleted after EC conversion. (#6447 )
...
This logic was originally part of `spreadEcShards()`, which got removed during
the unification effort with `ec.balance` (https://github.com/seaweedfs/seaweedfs/pull/6344 ),
accidentally breaking functionality in the process.
The commit restores the deletion code for EC'd volumes - with parallelization support.
2025-01-17 01:02:30 -08:00
Hadi Zamani
c7ae969c06
Add bucket's traffic metrics ( #6444 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
helm: lint and test charts / lint-test (push) Has been cancelled
* Add bucket's traffic metrics
* Add bucket traffic to dashboards
* Fix bucket metrics help messages
* Fix variable names
2025-01-16 08:23:35 -08:00
Tom Crasset
aa299462f2
improve iam error handling ( #6446 )
...
* improve iam error handling
* Delete docker/test.py
2025-01-16 08:23:19 -08:00
ftong2020
2304d2b472
Bugfix s3 audit missing requester for PUT requests ( #6434 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
fix s3 audit missing requster for PUT
2025-01-15 11:58:43 -08:00
dependabot[bot]
1609682ac2
chore(deps): bump github.com/getsentry/sentry-go from 0.30.0 to 0.31.1 ( #6437 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
helm: lint and test charts / lint-test (push) Has been cancelled
* chore(deps): bump github.com/getsentry/sentry-go from 0.30.0 to 0.31.1
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go ) from 0.30.0 to 0.31.1.
- [Release notes](https://github.com/getsentry/sentry-go/releases )
- [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md )
- [Commits](https://github.com/getsentry/sentry-go/compare/v0.30.0...v0.31.1 )
---
updated-dependencies:
- dependency-name: github.com/getsentry/sentry-go
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
* fix compilation
---------
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com >
2025-01-13 12:50:33 -08:00
ftong2020
e7f2936dcc
fix force arg dropped during volume balance command ( #6432 )
2025-01-12 23:30:18 -08:00
dsd
da2a234b00
[weed] change -n to -force ( #6421 )
2025-01-08 09:57:18 -08:00
chrislu
39c1dde9df
3.81
2025-01-06 11:38:50 -08:00
Brad Murray
bc3640ee64
Update command_fs_merge_volumes.go ( #6406 )
2025-01-02 08:57:26 -08:00
Guang Jiong Lou
3b1ac77e1f
worm grace period and retention time support ( #6404 )
...
Signed-off-by: lou <alex1988@outlook.com >
2024-12-31 18:41:43 -08:00
Trim21
0594006c81
use same time format in volume volume UI and filter UI ( #6395 )
...
use same time format in volume server UI with filter UI
2024-12-26 19:23:29 -08:00
dsd
20cbc9e4eb
skip error while executing volume.fix.replication ( #6382 )
2024-12-20 07:36:13 -08:00
chrislu
d7dfb11be9
fix compilation
2024-12-19 19:26:28 -08:00
chrislu
de09a82a32
go fmt
2024-12-19 19:25:27 -08:00
chrislu
bfd59dd579
adjust import
2024-12-19 19:25:06 -08:00
chrislu
ec155022e7
"golang.org/x/exp/slices" => "slices" and go fmt
2024-12-19 19:25:06 -08:00
Warren Hodgkinson
a1a76ccb8c
Fix for DeleteMultipleObjectsHandler wrongly deleting parent folders ( #6380 )
...
What problem are we solving?
Fix : #6379
How are we solving the problem?
We check for the AllowEmptyFolders option prior to cascade
deleting parent folders in S3 DeleteMultipleObjectsHandler.
How is the PR tested?
We ran SeaweedFS in a Kubernetes Cluster with a joint Filer
and S3 server in one container, with leveldb2 as the filer storage,
and AllowEmptyFolders set to true.
When using the Distribution Registry as the S3 client, it calls the
DeleteMultipleObjectsHandler as part of the artifact upload process
(uploads to a temp location, then performs a copy and delete).
Without this fix, the deletion cascade deleted parent folder until
the entire contents of the bucket were gone.
With this fix, the existing content of the bucket remained, and the
newly uploaded content was added.
Checks
[ ] I have added unit tests if possible.
[ ] I will add related wiki document changes and link to this PR after merging.
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com >
2024-12-19 18:00:08 -08:00
Lisandro Pin
4d91ec359b
Fix volume replica parallelization within ec.encode. ( #6377 )
...
See 826edd5d
2024-12-19 17:46:11 -08:00
Lisandro Pin
ba0707af64
Allow configuring the maximum number of concurrent tasks for EC parallelization. ( #6376 )
...
Follow-up to b0210df0
2024-12-18 13:26:26 -08:00
Lisandro Pin
44c48c929a
Parallelize volume replica operations within ec.encode. ( #6374 )
2024-12-18 11:59:48 -08:00
dsd
72af97162f
[shell] feat:stop vacuum immediately once volume.vacuum.disable was executed ( #6375 )
...
stop vacuum immediately once volume.vacuum.disable was executed
Co-authored-by: dsd <dsd2019@foxmail.com >
2024-12-18 11:56:40 -08:00
Lisandro Pin
9fbc4ea417
Rework shell.EcBalance()'s waitgroup code into a standalone type. ( #6373 )
...
Rework `shell.EcBalance()`'s waitgroup with errors code into a standalone type.
We'll re-use this for other EC jobs - for example, volume creation. Also fixes
potential concurrency issues when collecting error results.
2024-12-17 09:39:51 -08:00
zemul
e77e50886e
mount metacache add ttl ( #6360 )
...
* fix:mount deadlock
* fix
* feat: metaCache ttl
* Update weed/command/mount.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com >
* fix InodeEntry
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com >
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com >
2024-12-16 20:19:32 -08:00
coffeecloudgit
c531e2eb0e
Update disk_location_ec.go ( #6359 )
...
Add lock for l.ecVolumes
2024-12-15 22:05:47 -08:00
coffeecloudgit
7eb8257126
Update store.go ( #6358 )
...
Add lock for location.ecVolumes And single delete
2024-12-15 22:05:22 -08:00
Lisandro Pin
9b48ce0613
Parallelize EC shards balancing within racks ( #6354 )
...
Parallelize EC shards balancing within racks.
2024-12-15 13:36:23 -08:00
