lsm/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-11-24 16:53:14 +08:00
Code Issues Actions 17 Packages Projects Releases Wiki Activity
11,587 Commits 51 Branches 316 Tags
2d0d429d2f2f05019aade98edd5089b2528b3a7a
Commit Graph

929 Commits

Author SHA1 Message Date
chrislu
2d0d429d2f fix disk space calculation 2025-06-30 10:11:30 -07:00
chalet
877b9b788a update s3 session cache key (#6923) 2025-06-26 03:21:35 -07:00
chrislu
2f1b3d68d7 pass volume version when creating a volume 2025-06-19 01:15:25 -07:00
chrislu
d2be5822a1 refactoring 2025-06-16 22:25:22 -07:00
chrislu
96632a34b1 add version to volume proto 2025-06-16 22:05:06 -07:00
chrislu
e71d681fee refactor 2025-06-11 20:46:13 -07:00
chrislu
7c4d98446b refactor 2025-06-11 20:13:06 -07:00
chrislu
f27e195354 refactoring 2025-06-11 20:13:06 -07:00
chrislu
33ecc8442e refactor
Some checks failed
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Details
helm: lint and test charts / lint-test (push) Has been cancelled
Details
2025-06-08 22:11:09 -07:00
chrislu
60f11f6510 add a readme file for volume needle data layout
Some checks failed
go: build dev binaries / cleanup (push) Has been cancelled
Details
docker: build dev containers / build-dev-containers (push) Has been cancelled
Details
End to End / FUSE Mount (push) Has been cancelled
Details
go: build binary / Build (push) Has been cancelled
Details
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
Details
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
Details
2025-06-07 15:52:51 -07:00
chrislu
61c4f01e05 refactor
Some checks failed
go: build dev binaries / cleanup (push) Has been cancelled
Details
docker: build dev containers / build-dev-containers (push) Has been cancelled
Details
End to End / FUSE Mount (push) Has been cancelled
Details
go: build binary / Build (push) Has been cancelled
Details
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
Details
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
Details
2025-06-06 08:55:32 -07:00
chrislu
a489d99333 fix tests 2025-06-06 08:25:04 -07:00
Lisandro Pin
00c621abb8 Fix dumb typo in 08556257 (#6844) 2025-06-06 05:59:11 -07:00
chrislu
7439af0eca refactoring 2025-06-06 01:35:48 -07:00
chrislu
cc135c63f7 a bit refactoring 2025-06-06 01:26:54 -07:00
chrislu
c4695fc3b3 refactor needle write for different versions 2025-06-06 00:35:13 -07:00
Lisandro Pin
bed0a64693 New needle_map.CompactMap() implementation for reduced memory usage (#6842)
Some checks are pending
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Details 
* Rework `needle_map.CompactMap()` to maximize memory efficiency.

* Use a memory-efficient structure for `CompactMap` needle value entries.

This slightly complicates the code, but makes a **massive** difference
in memory efficiency - preliminary results show a ~30% reduction in
heap usage, with no measurable performance impact otherwise.

* Clean up type for `CompactMap` chunk IDs.

* Add a small comment description for `CompactMap()`.

* Add the old version of `CompactMap()` for comparison purposes.
 2025-06-05 14:03:29 -07:00
chrislu
35f0daa198 the isFsync parameter is essentially IsAsyncWrite and it needs to be turned off if s.isStopping
Some checks are pending
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Details 
d8c574a5ef (r159132764)
 2025-06-05 00:19:10 -07:00
chrislu
bd4891a117 change version directory 2025-06-03 22:46:10 -07:00
Chris Lu
7151a54b28 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
Some checks failed
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Details
helm: lint and test charts / lint-test (push) Has been cancelled
Details
2025-06-02 23:57:54 -07:00
Chris Lu
b25561d0d7 3.89 2025-06-02 23:56:58 -07:00
Chris Lu
d40746f34e fix insert beyond look back window (#6838) 2025-06-02 23:43:01 -07:00
Lisandro Pin
7204731749 Minor fix for the CompactMap() performance test. (#6836)
Some checks are pending
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
helm: lint and test charts / lint-test (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Details 
Per-entry memory usage is based on `TotalAllocs`, which is incorrect - that
value is a cummulative of heap usage, which doesn't decrease when objects
are freeed.

`Allocs` is instead an accurate represeentation of actual memory usage
at the time metrics are reported.
 2025-06-02 17:09:01 -07:00
Chris Lu
90802cb201 revert part of d8c574a5ef (#6829) 2025-06-01 12:27:49 -07:00
Lisandro Pin
9ffc8bcb54 Further improve memory usage of needle_map.CompactMap(). (#6825) 2025-05-28 11:42:00 -07:00
Lisandro Pin
2e1506c31e Rewrite needle_map.CompactMap() for more efficient memory usage (#6813) 2025-05-23 07:05:08 -07:00
dongxufeng
ff878a542d correctly report volume with input/output error to master (#6790)
Some checks are pending
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Details 
* correctly capture io error and report to master

* code fix

* check io error by error.Is

---------

Co-authored-by: dongxu_feng <dongxu_feng@intsig.net>
 2025-05-15 00:56:43 -07:00
chrislu
d8c574a5ef fix fsync logic
Some checks are pending
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details
test s3 over https using aws-cli / awscli-tests (push) Waiting to run
Details
2025-05-14 01:33:36 -07:00
Lisandro Pin
dddb0f0ae5 Fix update of SeaweedFS_volumeServer_volumes gauge metrics when EC shards are unmounted (#6776)
Some checks failed
go: build dev binaries / cleanup (push) Has been cancelled
Details
docker: build dev containers / build-dev-containers (push) Has been cancelled
Details
End to End / FUSE Mount (push) Has been cancelled
Details
go: build binary / Build (push) Has been cancelled
Details
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
Details
test s3 over https using aws-cli / awscli-tests (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
Details
2025-05-09 10:15:34 -07:00
Quentin D.
2ae5b480a6 Use the correct constant when computing the offset in SearchNeedleFromSortedIndex (#6771) 
NeedleHeaderSize happen to have the same size as NeedleMapEntrySize, except when running the 5 bytes offset variant of Seaweedfs, because it does not contain OffsetSize. This causes ECX corruption on deletes, due to the drifting offset computation (offset is always computed on a basis of 16 bytes per record instead of 17 bytes)

Signed-off-by: Quentin Devos <4972091+Okhoshi@users.noreply.github.com>
 2025-05-09 08:47:53 -07:00
bwlfhu
0e08b83521 Fix volume ttl (#6683) 2025-04-02 07:59:21 -07:00
dependabot[bot]
216c52e377 chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679) 
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0

Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: gocloud.dev
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix error

* fix printing errors

* Update go.mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2025-03-31 21:42:54 -07:00
NinaLua
271505717e chore: make function comment match function name (#6607)
Some checks failed
go: build dev binaries / cleanup (push) Has been cancelled
Details
docker: build dev containers / build-dev-containers (push) Has been cancelled
Details
End to End / FUSE Mount (push) Has been cancelled
Details
go: build binary / Build (push) Has been cancelled
Details
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
Details
2025-03-07 00:45:47 -08:00
Aleksey Kosov
ef4eda0761 added re-generating and writing the Volume UUID if it is empty (#6568) 2025-02-24 07:58:43 -08:00
Lisandro Pin
7913681297 ec.encode: Display a warning on EC balancing if no replica placement settings are found. (#6487) 2025-01-29 08:50:19 -08:00
chrislu
be15fee8e7 show error if backend is mis-configured
Some checks failed
go: build dev binaries / cleanup (push) Has been cancelled
Details
docker: build dev containers / build-dev-containers (push) Has been cancelled
Details
End to End / FUSE Mount (push) Has been cancelled
Details
go: build binary / Build (push) Has been cancelled
Details
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
Details 
related to https://github.com/seaweedfs/seaweedfs/discussions/6472
 2025-01-23 09:18:54 -08:00
ludwigxia
5452405a81 cloud tier: support for Alibaba Cloud OSS (#6466) 2025-01-22 00:48:55 -08:00
chrislu
de09a82a32 go fmt 2024-12-19 19:25:27 -08:00
chrislu
bfd59dd579 adjust import 2024-12-19 19:25:06 -08:00
chrislu
ec155022e7 "golang.org/x/exp/slices" => "slices" and go fmt 2024-12-19 19:25:06 -08:00
coffeecloudgit
c531e2eb0e Update disk_location_ec.go (#6359) 
Add lock for l.ecVolumes
 2024-12-15 22:05:47 -08:00
coffeecloudgit
7eb8257126 Update store.go (#6358) 
Add lock for location.ecVolumes And single delete
 2024-12-15 22:05:22 -08:00
Konstantin Lebedev
ff1392f7f4 [shell] use constant for hdd of type (#6337) 
use constant for hdd of type
 2024-12-10 08:43:59 -08:00
chrislu
f3dde99796 adjust error message 2024-12-05 09:33:50 -08:00
Trim21
fd6c787c02 fix EcVolumes sorting in volume UI (#6275) 
* Update store_ec.go

* fix: EcVolume sorting not work

* use stdlib

* revert
 2024-11-23 22:43:08 -08:00
chrislu
75f5afa571 fix compilation 2024-11-21 00:41:22 -08:00
chrislu
1c143cfcea display actual offset value 2024-11-21 00:38:03 -08:00
chrislu
c9f3448692 ReadAt may return io.EOF t end of file 
related to https://github.com/seaweedfs/seaweedfs/issues/6219
 2024-11-21 00:37:38 -08:00
chrislu
be09d08eb6 adjust error message 2024-11-20 23:49:01 -08:00
chrislu
96f1a2e01a adjust error message 2024-11-19 14:16:06 -08:00