seaweedfs

mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-11-08 12:36:13 +08:00

Author	SHA1	Message	Date
Chris Lu	69553e5ba6	convert error fromating to %w everywhere (#6995 )	2025-07-16 23:39:27 -07:00
Lisandro Pin	00c621abb8	Fix dumb typo in `08556257` (#6844 )	2025-06-06 05:59:11 -07:00
Lisandro Pin	bed0a64693	New `needle_map.CompactMap()` implementation for reduced memory usage (#6842 ) Some checks are pending go: build dev binaries / cleanup (push) Waiting to run Details go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions Details go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions Details go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions Details go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions Details docker: build dev containers / build-dev-containers (push) Waiting to run Details End to End / FUSE Mount (push) Waiting to run Details go: build binary / Build (push) Waiting to run Details Ceph S3 tests / Ceph S3 tests (push) Waiting to run Details test s3 over https using aws-cli / awscli-tests (push) Waiting to run Details * Rework `needle_map.CompactMap()` to maximize memory efficiency. * Use a memory-efficient structure for `CompactMap` needle value entries. This slightly complicates the code, but makes a massive difference in memory efficiency - preliminary results show a ~30% reduction in heap usage, with no measurable performance impact otherwise. * Clean up type for `CompactMap` chunk IDs. * Add a small comment description for `CompactMap()`. * Add the old version of `CompactMap()` for comparison purposes.	2025-06-05 14:03:29 -07:00
Chris Lu	7151a54b28	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs Some checks failed go: build dev binaries / cleanup (push) Waiting to run Details go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions Details go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions Details go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions Details go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions Details docker: build dev containers / build-dev-containers (push) Waiting to run Details End to End / FUSE Mount (push) Waiting to run Details go: build binary / Build (push) Waiting to run Details Ceph S3 tests / Ceph S3 tests (push) Waiting to run Details test s3 over https using aws-cli / awscli-tests (push) Waiting to run Details helm: lint and test charts / lint-test (push) Has been cancelled Details	2025-06-02 23:57:54 -07:00
Chris Lu	b25561d0d7	3.89	2025-06-02 23:56:58 -07:00
Chris Lu	d40746f34e	fix insert beyond look back window (#6838 )	2025-06-02 23:43:01 -07:00
Lisandro Pin	7204731749	Minor fix for the `CompactMap()` performance test. (#6836 ) Some checks are pending go: build dev binaries / cleanup (push) Waiting to run Details go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions Details go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions Details go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions Details go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions Details docker: build dev containers / build-dev-containers (push) Waiting to run Details End to End / FUSE Mount (push) Waiting to run Details go: build binary / Build (push) Waiting to run Details helm: lint and test charts / lint-test (push) Waiting to run Details Ceph S3 tests / Ceph S3 tests (push) Waiting to run Details test s3 over https using aws-cli / awscli-tests (push) Waiting to run Details Per-entry memory usage is based on `TotalAllocs`, which is incorrect - that value is a cummulative of heap usage, which doesn't decrease when objects are freeed. `Allocs` is instead an accurate represeentation of actual memory usage at the time metrics are reported.	2025-06-02 17:09:01 -07:00
Lisandro Pin	9ffc8bcb54	Further improve memory usage of `needle_map.CompactMap()`. (#6825 )	2025-05-28 11:42:00 -07:00
Lisandro Pin	2e1506c31e	Rewrite `needle_map.CompactMap()` for more efficient memory usage (#6813 )	2025-05-23 07:05:08 -07:00
chrislu	cb50b720fd	Revert "Fix weed fix" This reverts commit `6c1ce18541`.	2024-01-25 08:09:36 -08:00
chrislu	81f11883e3	go fmt	2023-11-26 11:47:20 -08:00
SmsS4	6c1ce18541	Fix weed fix	2023-11-20 11:14:19 -08:00
Konstantin Lebedev	bf8a9d2db1	[volume.chek.disk] sync of deletions the fix (#3923 ) * sync of deletions the fix * avoid return if only partiallyDeletedNeedles * refactor sync deletions	2022-10-30 20:32:46 -07:00
Konstantin Lebedev	764d9cb105	[voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922 ) needles older than the cutoff time are not missing yet https://github.com/seaweedfs/seaweedfs/issues/3919	2022-10-28 12:12:20 -07:00
Konstantin Lebedev	1f7e52c63e	vacuum metrics and force sync dst files (#3832 )	2022-10-13 00:51:20 -07:00
binbinshi	b7de4a967e	fix: compact_map get error mismatching cokie (#3748 ) * fix: compact_map get error * fix: CompactSection delete lock and move test to compact_map Co-authored-by: shibinbin <shibinbin@megvii.com>	2022-10-09 16:56:40 -07:00
chrislu	21c0587900	go fmt	2022-09-14 23:06:44 -07:00
chrislu	26dbc6c905	move to https://github.com/seaweedfs/seaweedfs	2022-07-29 00:17:28 -07:00
guol-fnst	ac694f0c8f	rename parameter and reuse functions rename milestone to watermark	2022-07-20 17:00:40 +08:00
chrislu	509a9047db	test compact map with snowflake sequencer	2022-06-23 21:45:51 -07:00
chrislu	625fd16a2e	reduce upfront memory usage for low density volume	2022-06-16 15:39:29 -07:00
Chris Lu	7ce97b59d8	go fmt	2021-09-01 02:45:42 -07:00
Chris Lu	24e11d1e90	look back when adding to sorted values look back when adding to sorted values, before adding it to overflow	2021-06-28 22:46:49 -07:00
Chris Lu	fc8dd58aea	volume: large_volume version has bug when using in memory index fix https://github.com/chrislusf/seaweedfs/issues/2162	2021-06-28 15:48:07 -07:00
Chris Lu	b465095db1	shell: add volume.check.disk to fix inconsistency for replicated volumes fix https://github.com/chrislusf/seaweedfs/issues/1923	2021-03-22 00:03:16 -07:00
Chris Lu	f2723c1bc8	do not idx file format revert `c9ab8d05fa`	2020-09-12 12:42:36 -07:00
Chris Lu	c9ab8d05fa	fixes for reading deleted fid	2020-09-10 14:42:52 -07:00
Chris Lu	99ecf63276	go fmt	2020-08-29 22:28:33 -07:00
Chris Lu	3b4b1d4a77	fix tests	2020-08-19 01:37:56 -07:00
Chris Lu	fe01191b5b	support read option readDeleted=true	2020-08-18 19:22:16 -07:00
Chris Lu	6ccd7f0a4d	refactoring	2020-08-18 18:01:37 -07:00
Chris Lu	ee11d98650	refactoring	2020-08-18 17:35:19 -07:00
Chris Lu	6a92f0bc7a	refactoring to typed Size Go is amazing with refactoring!	2020-08-18 17:04:28 -07:00
Chris Lu	f2e8ad6241	refactoring	2020-07-11 06:35:54 -07:00
Chris Lu	0871d2cff0	volume: fix memory leak during compaction fix https://github.com/chrislusf/seaweedfs/issues/1222	2020-03-09 22:29:02 -07:00
Chris Lu	acf7ca7b93	volume: fix compaction	2020-01-08 09:45:03 -08:00
Chris Lu	9ff72f616a	go fmt	2019-12-24 14:56:16 -08:00
Chris Lu	d8b39fe92a	testing	2019-12-24 11:29:26 -08:00
Chris Lu	abffe857a1	change btree map to in memory level db	2019-12-24 10:18:56 -08:00
Chris Lu	ec75b2d761	volume: fix bug with 8000GB version if using in memory index fix https://github.com/chrislusf/seaweedfs/issues/994	2019-06-27 12:18:45 -07:00
Chris Lu	3f9ecee40f	working with reading remote intervals	2019-05-28 21:29:07 -07:00
Chris Lu	87f63b9c08	generate ec01~ec14, generate ecx file with sorted needle values	2019-05-18 22:46:24 -07:00
Chris Lu	ac2727853f	fix needle map entry size	2019-04-19 00:39:34 -07:00
Chris Lu	e5506152c0	refactoring	2019-04-18 21:43:36 -07:00

44 Commits