lsm/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-08-20 07:02:09 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
11,815 Commits 43 Branches 312 Tags 258 MiB
master
Commit Graph

1699 Commits

Author SHA1 Message Date
chrislu
d5ded63ce3 day-advance fix 2025-08-15 10:35:33 -07:00
Lisandro Pin
18a22177b9
Fix volume server's status code returned for missing needles on EC-encoded shards (#7137) 2025-08-14 06:35:55 -07:00
chrislu
7889e78f4d use one http client 2025-08-11 16:45:13 -07:00
Chris Lu
af3300e063
filer: server side copying (#7121) 
* copy

* address comments

* remove unused functions, reuse http clients

* address hardlink, checking existing directory

* destination is directory

* check for the key's existence in the map first before accessing its members

* address comments

* deep copy remote entry

* address comments

* copying chunks in parallel

* handle manifest chunks

* address comments

* errgroup

* there could be large chunks

* address comments

* address comments
 2025-08-11 16:40:46 -07:00
Chris Lu
b4d9618efc
volume server UI: fix ec volume ui (#7104) 
* fix ec volume ui

* Update weed/storage/erasure_coding/ec_volume.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-07 00:07:03 -07:00
Chris Lu
4af182f880
Context cancellation during reading range reading large files (#7093) 
* context cancellation during reading range reading large files

* address comments

* cancellation for fuse read

* fix cancellation

* pass in context for each function to avoid racing condition

* Update reader_at_test.go

* remove dead code

* Update weed/filer/reader_at.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/filer/filechunk_group.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/filer/filechunk_group.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* address comments

* Update weed/mount/weedfs_file_read.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/mount/weedfs_file_lseek.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/mount/weedfs_file_read.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/filer/reader_at.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/mount/weedfs_file_lseek.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* test cancellation

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-06 10:09:26 -07:00
Chris Lu
e446234e9c
remove spoof-able request header (#7103) 
* remove spoof-able request header

https://github.com/seaweedfs/seaweedfs/issues/7094#issuecomment-3158320497

* Update weed/security/guard.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-06 10:08:30 -07:00
Chris Lu
0703308270
remote address parsing should handle special cases (#7101) 
* remote address parsing should handle special cases

* handling ipv6

* simplify

* Update weed/security/guard.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/security/guard.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* x-real-ip

* Update guard.go

* fixes

 Hostname Whitelisting: Fully restored - supports localhost, example.com, etc.
 IP Whitelisting: Still works - supports exact IPs and CIDR ranges
 Header Support: Consistent handling of X-Forwarded-For, X-Real-IP

* simplify

* Update weed/security/guard.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/security/guard.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update guard.go

* adjust function signature

* Update weed/security/guard.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* indention

* skip empty host

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-06 01:03:00 -07:00
Chris Lu
fd447465c2
fix parsing s3 tag (#7069) 
* fix parsing s3 tag

fix https://github.com/seaweedfs/seaweedfs/issues/7040#issuecomment-3145615630

* url.ParseQuery
 2025-08-01 15:45:23 -07:00
Chris Lu
5a4067d36e
filer: remove filer support for image resizing (#7057) 
remove filer support for image resizing

fix https://github.com/seaweedfs/seaweedfs/issues/6544

only volume server still support it
 2025-07-30 16:18:28 -07:00
Chris Lu
891a2fb6eb
Admin: misc improvements on admin server and workers. EC now works. (#7055) 
* initial design

* added simulation as tests

* reorganized the codebase to move the simulation framework and tests into their own dedicated package

* integration test. ec worker task

* remove "enhanced" reference

* start master, volume servers, filer

Current Status
✅ Master: Healthy and running (port 9333)
✅ Filer: Healthy and running (port 8888)
✅ Volume Servers: All 6 servers running (ports 8080-8085)
🔄 Admin/Workers: Will start when dependencies are ready

* generate write load

* tasks are assigned

* admin start wtih grpc port. worker has its own working directory

* Update .gitignore

* working worker and admin. Task detection is not working yet.

* compiles, detection uses volumeSizeLimitMB from master

* compiles

* worker retries connecting to admin

* build and restart

* rendering pending tasks

* skip task ID column

* sticky worker id

* test canScheduleTaskNow

* worker reconnect to admin

* clean up logs

* worker register itself first

* worker can run ec work and report status

but:
1. one volume should not be repeatedly worked on.
2. ec shards needs to be distributed and source data should be deleted.

* move ec task logic

* listing ec shards

* local copy, ec. Need to distribute.

* ec is mostly working now

* distribution of ec shards needs improvement
* need configuration to enable ec

* show ec volumes

* interval field UI component

* rename

* integration test with vauuming

* garbage percentage threshold

* fix warning

* display ec shard sizes

* fix ec volumes list

* Update ui.go

* show default values

* ensure correct default value

* MaintenanceConfig use ConfigField

* use schema defined defaults

* config

* reduce duplication

* refactor to use BaseUIProvider

* each task register its schema

* checkECEncodingCandidate use ecDetector

* use vacuumDetector

* use volumeSizeLimitMB

* remove

remove

* remove unused

* refactor

* use new framework

* remove v2 reference

* refactor

* left menu can scroll now

* The maintenance manager was not being initialized when no data directory was configured for persistent storage.

* saving config

* Update task_config_schema_templ.go

* enable/disable tasks

* protobuf encoded task configurations

* fix system settings

* use ui component

* remove logs

* interface{} Reduction

* reduce interface{}

* reduce interface{}

* avoid from/to map

* reduce interface{}

* refactor

* keep it DRY

* added logging

* debug messages

* debug level

* debug

* show the log caller line

* use configured task policy

* log level

* handle admin heartbeat response

* Update worker.go

* fix EC rack and dc count

* Report task status to admin server

* fix task logging, simplify interface checking, use erasure_coding constants

* factor in empty volume server during task planning

* volume.list adds disk id

* track disk id also

* fix locking scheduled and manual scanning

* add active topology

* simplify task detector

* ec task completed, but shards are not showing up

* implement ec in ec_typed.go

* adjust log level

* dedup

* implementing ec copying shards and only ecx files

* use disk id when distributing ec shards

🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk
📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId
🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest
💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId])
📂 File System: EC shards and metadata land in the exact disk directory planned

* Delete original volume from all locations

* clean up existing shard locations

* local encoding and distributing

* Update docker/admin_integration/EC-TESTING-README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* check volume id range

* simplify

* fix tests

* fix types

* clean up logs and tests

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-07-30 12:38:03 -07:00
Chris Lu
69553e5ba6
convert error fromating to %w everywhere (#6995) 2025-07-16 23:39:27 -07:00
chrislu
89706d36dc less logs 2025-07-16 16:30:22 -07:00
chrislu
215c5de579 minor 2025-07-16 09:22:25 -07:00
Ibrahim Konsowa
93bbaa1fb4
[Notifications] Support webhook notifications (#6962) 
Add webhook notification support
 2025-07-10 09:22:05 -07:00
chrislu
592b6a1e98 less aggressive volume server shutdown on same uuid 
related to https://github.com/seaweedfs/seaweedfs/issues/5439
 2025-07-07 01:22:17 -07:00
chrislu
798f797158 use float for sleep seconds 
fix https://github.com/seaweedfs/seaweedfs/pull/6795
 2025-07-06 14:16:41 -07:00
Konstantin Lebedev
93007c1842
[volume] refactor and add metrics for flight upload and download data limit condition (#6920) 
* refactor concurrentDownloadLimit

* fix loop

* fix cmdServer

* fix: resolve conversation pr 6920

* Changes logging function (#6919)

* updated logging methods for stores

* updated logging methods for stores

* updated logging methods for filer

* updated logging methods for uploader and http_util

* updated logging methods for weed server

---------

Co-authored-by: akosov <a.kosov@kryptonite.ru>

* Improve lock ring (#6921)

* fix flaky lock ring test

* add more tests

* fix: build

* fix: rm import util/version

* fix: serverOptions

* refactoring

---------

Co-authored-by: Aleksey Kosov <rusyak777@list.ru>
Co-authored-by: akosov <a.kosov@kryptonite.ru>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2025-07-02 18:03:49 -07:00
chrislu
1733d0ce68 remove features and deployments fields 2025-06-28 20:03:06 -07:00
Chris Lu
a1aab8a083
add telemetry (#6926) 
* add telemetry

* fix go mod

* add default telemetry server url

* Update README.md

* replace with broker count instead of s3 count

* Update telemetry.pb.go

* github action to deploy
 2025-06-28 14:11:55 -07:00
Aleksey Kosov
4511c2cc1f
Changes logging function (#6919) 
* updated logging methods for stores

* updated logging methods for stores

* updated logging methods for filer

* updated logging methods for uploader and http_util

* updated logging methods for weed server

---------

Co-authored-by: akosov <a.kosov@kryptonite.ru>
 2025-06-24 08:44:06 -07:00
Aleksey Kosov
90c128e7a6
Context-based logging with request ID (#6899) 2025-06-20 06:23:53 -07:00
chrislu
da728750be follow grow volume option version 2025-06-19 13:54:54 -07:00
chrislu
2f1b3d68d7 pass volume version when creating a volume 2025-06-19 01:15:25 -07:00
chrislu
87927d068b display volume version when listing 2025-06-19 00:33:01 -07:00
chrislu
9a115068af adding cassandra2 
fix https://github.com/seaweedfs/seaweedfs/issues/6888
 2025-06-17 11:25:37 -07:00
chrislu
c602f53a6e tail-volume-uses-the-source-volume-version 2025-06-16 22:46:13 -07:00
chrislu
d2be5822a1 refactoring 2025-06-16 22:25:22 -07:00
chrislu
96632a34b1 add version to volume proto 2025-06-16 22:05:06 -07:00
SmoothDenis
c99530ec66
fix(filer): add missing return after KvDelete in KvPut with empty value (#6846) 2025-06-06 05:58:03 -07:00
chrislu
bd4891a117 change version directory 2025-06-03 22:46:10 -07:00
Aleksey Kosov
283d9e0079
Add context with request (#6824) 2025-05-28 11:34:02 -07:00
Aleksey Kosov
165af32d6b
added context to filer_client method calls (#6808) 
Co-authored-by: akosov <a.kosov@kryptonite.ru>
 2025-05-22 09:46:49 -07:00
Aleksey Kosov
5182d46e22
Added middleware for processing request_id grpc and http requests (#6805)
Some checks are pending
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Details
2025-05-21 07:57:39 -07:00
Bruce Zou
582206b7d2
load configuration before get max_file_name_length (#6766)
Some checks are pending
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Details
2025-05-07 21:52:36 -07:00
MaratKarimov
ba3afd1803
Tarantool filer store (#6669) 
Co-authored-by: Marat Karimov <m.karimov@digitalms.ru>
 2025-03-29 21:12:06 -07:00
alingse
46b3a71a1f
fix call writeJsonError with a nil value error after check another err (#6673) 
the dbErr != nil and err is nil

when call writeJsonError with a nil value error, it may panic
 2025-03-29 08:17:36 -07:00
hoverseu
14cb8a24c6
[filer] UploadRequestToChunks: terminate the upload of remaining chunks as early as possible might be more reasonable when one chunk upload goroutine encounters a failure. (#6610) 2025-03-09 20:38:26 -07:00
tiaoxizhan
533b675ec8
fix: fix the wrong error return value (#6603) 2025-03-06 06:36:33 -08:00
chrislu
b977e0b3b2 minor
Some checks are pending
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
2025-02-25 19:54:40 -08:00
Lisandro Pin
fc4df944a0
Remove rate limit semaphore on master's leader selection logic. (#6494)
Some checks failed
go: build dev binaries / cleanup (push) Has been cancelled
Details
docker: build dev containers / build-dev-containers (push) Has been cancelled
Details
End to End / FUSE Mount (push) Has been cancelled
Details
go: build binary / Build (push) Has been cancelled
Details
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
Details 
This was introduced by 054374c7 (2024-03-12) and serves no practical purpose,
yet it caps the maximum QPS master servers can handle.
 2025-01-30 13:08:36 -08:00
Guang Jiong Lou
3b1ac77e1f
worm grace period and retention time support (#6404) 
Signed-off-by: lou <alex1988@outlook.com>
 2024-12-31 18:41:43 -08:00
Trim21
0594006c81
use same time format in volume volume UI and filter UI (#6395) 
use same time format in volume server UI with filter UI
 2024-12-26 19:23:29 -08:00
chrislu
ec155022e7 "golang.org/x/exp/slices" => "slices" and go fmt 2024-12-19 19:25:06 -08:00
dsd
72af97162f
[shell] feat:stop vacuum immediately once volume.vacuum.disable was executed (#6375) 
stop vacuum immediately once volume.vacuum.disable was executed

Co-authored-by: dsd <dsd2019@foxmail.com>
 2024-12-18 11:56:40 -08:00
Konstantin Lebedev
b65eb2ec45
[security] reload whiteList on http seerver (#6302) 
* reload whiteList

* white_list add to scaffold
 2024-12-02 10:38:10 -08:00
Konstantin Lebedev
e2e97db917
[master] avoid timeout when assigning for main request with filter by DC or rack (#6291) 
* avoid timeout when assigning for main request with filter by DC or rack

https://github.com/seaweedfs/seaweedfs/issues/6290

* use constant NoWritableVolumes
 2024-11-26 08:33:31 -08:00
Konstantin Lebedev
fec88e64eb
[master] update LastLeaderChangeTime for hashicorp raft (#6292) 2024-11-26 08:02:45 -08:00
Konstantin Lebedev
8836fa19b6
use ShouldGrowVolumesByDcAndRack (#6280) 2024-11-25 09:30:37 -08:00
chrislu
ccf1795e6f wait a bit before getting the next volume id if the leader is recently elected 2024-11-23 19:58:45 -08:00