lsm/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-08-20 07:02:09 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
11,804 Commits 43 Branches 312 Tags 258 MiB
6bf5a6871c
Commit Graph

11804 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
chrislu
6bf5a6871c fix presigned signature 2025-08-11 23:57:59 -07:00
dependabot[bot]
3864f89f92
chore(deps): bump golang.org/x/crypto from 0.40.0 to 0.41.0 (#7132) 
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.40.0 to 0.41.0.
- [Commits](https://github.com/golang/crypto/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.41.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-11 23:35:09 -07:00
dependabot[bot]
d1f18f08d3
chore(deps): bump google.golang.org/protobuf from 1.36.6 to 1.36.7 (#7131) 
Bumps google.golang.org/protobuf from 1.36.6 to 1.36.7.

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-version: 1.36.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-11 23:34:58 -07:00
chrislu
7889e78f4d use one http client 2025-08-11 16:45:13 -07:00
Chris Lu
af3300e063
filer: server side copying (#7121) 
* copy

* address comments

* remove unused functions, reuse http clients

* address hardlink, checking existing directory

* destination is directory

* check for the key's existence in the map first before accessing its members

* address comments

* deep copy remote entry

* address comments

* copying chunks in parallel

* handle manifest chunks

* address comments

* errgroup

* there could be large chunks

* address comments

* address comments
 2025-08-11 16:40:46 -07:00
dependabot[bot]
7e86045e22
chore(deps): bump golang.org/x/net from 0.42.0 to 0.43.0 (#7126) 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.42.0 to 0.43.0.
- [Commits](https://github.com/golang/net/compare/v0.42.0...v0.43.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.43.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-11 15:21:46 -07:00
dependabot[bot]
529c68a798
chore(deps): bump google.golang.org/api from 0.244.0 to 0.246.0 (#7127) 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.244.0 to 0.246.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.244.0...v0.246.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-version: 0.246.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-11 15:06:25 -07:00
dependabot[bot]
e3b15073a0
chore(deps): bump actions/checkout from 4 to 5 (#7125) 
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-11 15:06:14 -07:00
dependabot[bot]
5f62b11889
chore(deps): bump golang.org/x/sys from 0.34.0 to 0.35.0 (#7124) 
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.34.0 to 0.35.0.
- [Commits](https://github.com/golang/sys/compare/v0.34.0...v0.35.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-version: 0.35.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-11 15:06:01 -07:00
dependabot[bot]
524c1916b6
chore(deps): bump github.com/redis/go-redis/v9 from 9.11.0 to 9.12.0 (#7123) 
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.11.0 to 9.12.0.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.11.0...v9.12.0)

---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
  dependency-version: 9.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-11 15:05:53 -07:00
dependabot[bot]
412fd6ee86
chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.113.4 to 3.113.5 (#7122) 
chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.113.4 to 3.113.5.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.113.4...v3.113.5)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-version: 3.113.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-11 15:05:44 -07:00
Chris Lu
605b3333c1
fix sftp start with filer (#7120) 
* fix sftp start with filer

* add bindIp
 2025-08-11 10:56:40 -07:00
Chris Lu
9265e81fe9
S3 API: unsigned streaming (no cred) but chunks contain signatures (#7118) 
* This handles the case where we have unsigned streaming (no cred) but chunks contain signatures

* Update chunked_reader_v4.go

* address comments
 2025-08-11 10:31:01 -07:00
chrislu
a265a07922 fix ttl objects 
fix https://github.com/seaweedfs/seaweedfs/discussions/7107#discussioncomment-14069018
 2025-08-11 02:20:02 -07:00
Chris Lu
25bbf4c3d4
Admin UI: Fetch task logs (#7114) 
* show task details

* loading tasks

* task UI works

* generic rendering

* rendering the export link

* removing placementConflicts from task parameters

* remove TaskSourceLocation

* remove "Server ID" column

* rendering balance task source

* sources and targets

* fix ec task generation

* move info

* render timeline

* simplified worker id

* simplify

* read task logs from worker

* isValidTaskID

* address comments

* Update weed/worker/tasks/balance/execution.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/worker/tasks/erasure_coding/ec_task.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/worker/tasks/task_log_handler.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix shard ids

* plan distributing shard id

* rendering planned shards in task details

* remove Conflicts

* worker logs correctly

* pass in dc and rack

* task logging

* Update weed/admin/maintenance/maintenance_queue.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* display log details

* logs have fields now

* sort field keys

* fix link

* fix collection filtering

* avoid hard coded ec shard counts

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-09 21:47:29 -07:00
Chris Lu
3ac2a2e22d
fix tikv listing due to expired entries (#7115) 
* fix tikv listing due to expired entries

When there are many entries with empty fileName values (which can happen after TTL cleanup), the continue statements prevent the loop counter from incrementing, creating an infinite loop.

* address comments

* Update weed/filer/tikv/tikv_store.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* address comments

Update weed/filer/tikv/tikv_store.go

Co-Authored-By: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-08 12:38:55 -07:00
Devin Lauderdale
fae416586b
Move helm templates into folders (#7113) 
* refactor: move helm templates into respective service folders

* fix: update template path reference in filer-statefulset for s3-secret
 2025-08-08 10:36:01 -07:00
Devin Lauderdale
92cebe12f0
chore: remove default replica count for all-in-one deployment (#7111) 2025-08-07 21:18:17 -07:00
Chris Lu
535985adb6
Shell: add verbose ec encoding mode (#7105) 
* add verbose ec encoding mode

* address comments
 2025-08-07 00:12:05 -07:00
Chris Lu
cde2d65c16
ec candidate selection needs to adjust same rack count compare (#7106) 
ec needs to adjust same rack count compare
 2025-08-07 00:09:51 -07:00
Chris Lu
b4d9618efc
volume server UI: fix ec volume ui (#7104) 
* fix ec volume ui

* Update weed/storage/erasure_coding/ec_volume.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-07 00:07:03 -07:00
chrislu
dd4880d55a fix for baidu cloud storage 2025-08-06 20:53:05 -07:00
Chris Lu
4af182f880
Context cancellation during reading range reading large files (#7093) 
* context cancellation during reading range reading large files

* address comments

* cancellation for fuse read

* fix cancellation

* pass in context for each function to avoid racing condition

* Update reader_at_test.go

* remove dead code

* Update weed/filer/reader_at.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/filer/filechunk_group.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/filer/filechunk_group.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* address comments

* Update weed/mount/weedfs_file_read.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/mount/weedfs_file_lseek.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/mount/weedfs_file_read.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/filer/reader_at.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/mount/weedfs_file_lseek.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* test cancellation

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-06 10:09:26 -07:00
Chris Lu
e446234e9c
remove spoof-able request header (#7103) 
* remove spoof-able request header

https://github.com/seaweedfs/seaweedfs/issues/7094#issuecomment-3158320497

* Update weed/security/guard.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-06 10:08:30 -07:00
Chris Lu
0703308270
remote address parsing should handle special cases (#7101) 
* remote address parsing should handle special cases

* handling ipv6

* simplify

* Update weed/security/guard.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/security/guard.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* x-real-ip

* Update guard.go

* fixes

 Hostname Whitelisting: Fully restored - supports localhost, example.com, etc.
 IP Whitelisting: Still works - supports exact IPs and CIDR ranges
 Header Support: Consistent handling of X-Forwarded-For, X-Real-IP

* simplify

* Update weed/security/guard.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/security/guard.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update guard.go

* adjust function signature

* Update weed/security/guard.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* indention

* skip empty host

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-06 01:03:00 -07:00
Chris Lu
c6d9756933
fix signature hashing for iam (#7100) 
* fix signature hashing for iam

* add tests

* address comments

* Update weed/s3api/auto_signature_v4_test.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* indention

* fix test

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-05 22:54:54 -07:00
dependabot[bot]
b01b5e0f34
chore(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.29.18 to 1.30.2 (#7099) 
chore(deps): bump github.com/aws/aws-sdk-go-v2/config

Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.29.18 to 1.30.2.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.29.18...v1.30.2)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.30.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-05 22:06:58 -07:00
dependabot[bot]
315e089d41
chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.71 to 1.18.2 (#7084) 
chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials

Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.17.71 to 1.18.2.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/config/v1.18.2/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.71...config/v1.18.2)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.18.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-05 21:22:07 -07:00
dependabot[bot]
ec56e7e8d9
chore(deps): bump github.com/getsentry/sentry-go from 0.34.1 to 0.35.0 (#7098) 
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.34.1 to 0.35.0.
- [Release notes](https://github.com/getsentry/sentry-go/releases)
- [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-go/compare/v0.34.1...v0.35.0)

---
updated-dependencies:
- dependency-name: github.com/getsentry/sentry-go
  dependency-version: 0.35.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-05 14:31:32 -07:00
dependabot[bot]
bc19d144b5
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.84.1 to 1.85.1 (#7083) 
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3

Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.84.1 to 1.85.1.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.84.1...service/s3/v1.85.1)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.85.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-05 14:31:19 -07:00
dependabot[bot]
69fb524e36
chore(deps): bump google.golang.org/api from 0.243.0 to 0.244.0 (#7096) 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.243.0 to 0.244.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.243.0...v0.244.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-version: 0.244.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-05 14:31:08 -07:00
chrislu
2d11e9bc1a Pin helm-gh-pages action to stable version v1.7.0 
- Use stable release instead of @master to avoid potential regressions
- This should help resolve the 'gzip: invalid magic' error
 2025-08-05 08:42:05 -07:00
chrislu
4f38b7c3b9 Revert "Fix helm-gh-pages action configuration" 
This reverts commit 26ef76b7b9.
 2025-08-05 08:41:18 -07:00
chrislu
26ef76b7b9 Fix helm-gh-pages action configuration 
- Add missing commit_username and commit_email parameters
- This should resolve the 'gzip: invalid magic' error in the Helm GitHub Pages action
 2025-08-05 08:39:53 -07:00
Chris Lu
a834327755
context cancellation during reading range reading large files (#7092) 
* context cancellation during reading range reading large files

* address comments
 2025-08-04 23:22:11 -07:00
dependabot[bot]
375dfe18a4
chore(deps): bump modernc.org/sqlite from 1.38.1 to 1.38.2 (#7091) 
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.38.1 to 1.38.2.
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.38.1...v1.38.2)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-version: 1.38.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-04 21:30:55 -07:00
dependabot[bot]
ca9da29504
chore(deps): bump github.com/prometheus/client_golang from 1.22.0 to 1.23.0 (#7090) 
chore(deps): bump github.com/prometheus/client_golang

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.22.0 to 1.23.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.22.0...v1.23.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-version: 1.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-04 21:30:48 -07:00
dependabot[bot]
b376eccd22
chore(deps): bump github.com/golang-jwt/jwt/v5 from 5.2.3 to 5.3.0 (#7089) 
Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) from 5.2.3 to 5.3.0.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.2.3...v5.3.0)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-version: 5.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-08-04 21:30:39 -07:00
dependabot[bot]
b1ce55291f
chore(deps): bump docker/metadata-action from 5.7.0 to 5.8.0 (#7082) 2025-08-04 19:15:32 -07:00
dependabot[bot]
deb771817f
chore(deps): bump docker/login-action from 3.4.0 to 3.5.0 (#7085) 2025-08-04 19:15:07 -07:00
dependabot[bot]
16a16d79ce
chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.113.2 to 3.113.4 (#7086) 2025-08-04 19:14:35 -07:00
dependabot[bot]
20694a84f0
chore(deps): bump github.com/aws/aws-sdk-go from 1.55.7 to 1.55.8 (#7087) 2025-08-04 19:14:24 -07:00
dependabot[bot]
fd568cd796
chore(deps): bump cloud.google.com/go/pubsub from 1.49.0 to 1.50.0 (#7088) 2025-08-04 19:14:03 -07:00
Chris Lu
72176601c1
S3: Fix iam payload hash (#7081) 
* fix iam payload hash

* streaming hash

* Update weed/s3api/auto_signature_v4_test.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/s3api/auto_signature_v4_test.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* address comments

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-04 09:10:01 -07:00
Chris Lu
365d03ff32
mount ec shards correctly (#7079) 2025-08-03 23:10:28 -07:00
Chris Lu
513ac58504
Filer: fix filer range read (#7078) 
* fix filer range read

Only return true if we're reading the ENTIRE chunk from the beginning.
	// This prevents bandwidth amplification when range requests happen to align
	// with chunk boundaries but don't actually want the full chunk.

* Update weed/filer/filechunks.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-03 14:26:15 -07:00
Chris Lu
4fb7bbb215
Filer Store: postgres backend support pgbouncer (#7077) 
support pgbouncer
 2025-08-03 11:56:04 -07:00
Chris Lu
d49b44f2a4
Postgres (CockroachDB) with full certificate verification (#7076) 
* Postgres (CockroachDB) with full certificate verification

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* remove duplicated comments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-03 09:43:33 -07:00
Chris Lu
8c23952326
separate context for filer store (#7075) 
* separate context for filer store

* clone request id

* context.WithoutCancel
 2025-08-03 09:20:17 -07:00
Chris Lu
0ecb466eda
Admin: refactoring active topology (#7073) 
* refactoring

* add ec shard size

* address comments

* passing task id

There seems to be a disconnect between the pending tasks created in ActiveTopology and the TaskDetectionResult returned by this function. A taskID is generated locally and used to create pending tasks via AddPendingECShardTask, but this taskID is not stored in the TaskDetectionResult or passed along in any way.

This makes it impossible for the worker that eventually executes the task to know which pending task in ActiveTopology it corresponds to. Without the correct taskID, the worker cannot call AssignTask or CompleteTask on the master, breaking the entire task lifecycle and capacity management feature.

A potential solution is to add a TaskID field to TaskDetectionResult and worker_pb.TaskParams, ensuring the ID is propagated from detection to execution.

* 1 source multiple destinations

* task supports multi source and destination

* ec needs to clean up previous shards

* use erasure coding constants

* getPlanningCapacityUnsafe getEffectiveAvailableCapacityUnsafe  should return StorageSlotChange for calculation

* use CanAccommodate to calculate

* remove dead code

* address comments

* fix Mutex Copying in Protobuf Structs

* use constants

* fix estimatedSize

The calculation for estimatedSize only considers source.EstimatedSize and dest.StorageChange, but omits dest.EstimatedSize. The TaskDestination struct has an EstimatedSize field, which seems to be ignored here. This could lead to an incorrect estimation of the total size of data involved in tasks on a disk. The loop should probably also include estimatedSize += dest.EstimatedSize.

* at.assignTaskToDisk(task)

* refactoring

* Update weed/admin/topology/internal.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* fail fast

* fix compilation

* Update weed/worker/tasks/erasure_coding/detection.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* indexes for volume and shard locations

* dedup with ToVolumeSlots

* return an additional boolean to indicate success, or an error

* Update abstract_sql_store.go

* fix

* Update weed/worker/tasks/erasure_coding/detection.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/admin/topology/task_management.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* faster findVolumeDisk

* Update weed/worker/tasks/erasure_coding/detection.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/admin/topology/storage_slot_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* refactor

* simplify

* remove unused GetDiskStorageImpact function

* refactor

* add comments

* Update weed/admin/topology/storage_impact.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/admin/topology/storage_slot_test.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update storage_impact.go

* AddPendingTask

The unified AddPendingTask function now serves as the single entry point for all task creation, successfully consolidating the previously separate functions while maintaining full functionality and improving code organization.

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-03 01:35:38 -07:00