lsm/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-11-24 08:46:54 +08:00
Code Issues Actions 17 Packages Projects Releases Wiki Activity
11,684 Commits 51 Branches 316 Tags
bb81894078097785a48e74f01c884208a7b89448
Commit Graph

28 Commits

Author SHA1 Message Date
chrislu
bb81894078 Update .gitignore 2025-07-16 01:18:23 -07:00
Chris Lu
dde1cf63c2 S3 Object Lock: ensure x-amz-bucket-object-lock-enabled header (#6990) 
* ensure x-amz-bucket-object-lock-enabled header

* fix tests

* combine 2 metadata changes into one

* address comments

* Update s3api_bucket_handlers.go

* Update weed/s3api/s3api_bucket_handlers.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update test/s3/retention/object_lock_reproduce_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update test/s3/retention/object_lock_validation_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update test/s3/retention/s3_bucket_object_lock_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_bucket_handlers.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_bucket_handlers.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update test/s3/retention/s3_bucket_object_lock_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_bucket_handlers.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* package name

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-07-15 23:21:58 -07:00
Chris Lu
4b040e8a87 adding cors support (#6987) 
* adding cors support

* address some comments

* optimize matchesWildcard

* address comments

* fix for tests

* address comments

* address comments

* address comments

* path building

* refactor

* Update weed/s3api/s3api_bucket_config.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* address comment

Service-level responses need both Access-Control-Allow-Methods and Access-Control-Allow-Headers. After setting Access-Control-Allow-Origin and Access-Control-Expose-Headers, also set Access-Control-Allow-Methods: * and Access-Control-Allow-Headers: * so service endpoints satisfy CORS preflight requirements.

* Update weed/s3api/s3api_bucket_config.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_handlers.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_handlers.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix

* refactor

* Update weed/s3api/s3api_bucket_config.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_handlers.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_server.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* simplify

* add cors tests

* fix tests

* fix tests

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-07-15 00:23:54 -07:00
Chris Lu
1549ee2e15 implement PubObjectRetention and WORM (#6969) 
* implement PubObjectRetention and WORM

* Update s3_worm_integration_test.go

* avoid previous buckets

* Update s3-versioning-tests.yml

* address comments

* address comments

* rename to ExtObjectLockModeKey

* only checkObjectLockPermissions if versioningEnabled

* address comments

* comments

* Revert "comments"

This reverts commit 6736434176.

* Update s3api_object_handlers_skip.go

* Update s3api_object_retention_test.go

* add version id to ObjectIdentifier

* address comments

* add comments

* Add proper error logging for timestamp parsing failures

* address comments

* add version id to the error

* Update weed/s3api/s3api_object_retention_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_retention.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* constants

* fix comments

* address comments

* address comment

* refactor out handleObjectLockAvailabilityCheck

* errors.Is ErrBucketNotFound

* better error checking

* address comments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-07-12 21:58:55 -07:00
Chris Lu
d892538d32 More efficient copy object (#6665) 
* it compiles

* refactored

* reduce to 4 concurrent chunk upload

* CopyObjectPartHandler

* copy a range of the chunk data, fix offset size in copied chunks

* Update s3api_object_handlers_copy.go

What the PR Accomplishes:
CopyObjectHandler - Now copies entire objects by copying chunks individually instead of downloading/uploading the entire file
CopyObjectPartHandler - Handles copying parts of objects for multipart uploads by copying only the relevant chunk portions
Efficient Chunk Copying - Uses direct chunk-to-chunk copying with proper volume assignment and concurrent processing (limited to 4 concurrent operations)
Range Support - Properly handles range-based copying for partial object copies

* fix compilation

* fix part destination

* handling small objects

* use mkFile

* copy to existing file or part

* add testing tools

* adjust tests

* fix chunk lookup

* refactoring

* fix TestObjectCopyRetainingMetadata

* ensure bucket name not conflicting

* fix conditional copying tests

* remove debug messages

* add custom s3 copy tests
 2025-07-11 18:51:32 -07:00
Chris Lu
51543bbb87 Admin UI: Add message queue to admin UI (#6958) 
* add a menu item "Message Queue"

* add a menu item "Message Queue"
  * move the "brokers" link under it.
  * add "topics", "subscribers". Add pages for them.

* refactor

* show topic details

* admin display publisher and subscriber info

* remove publisher and subscribers from the topic row pull down

* collecting more stats from publishers and subscribers

* fix layout

* fix publisher name

* add local listeners for mq broker and agent

* render consumer group offsets

* remove subscribers from left menu

* topic with retention

* support editing topic retention

* show retention when listing topics

* create bucket

* Update s3_buckets_templ.go

* embed the static assets into the binary

fix https://github.com/seaweedfs/seaweedfs/issues/6964
 2025-07-11 10:19:27 -07:00
Chris Lu
02773a6107 Accumulated changes for message queue (#6600)
Some checks are pending
go: build dev binaries / cleanup (push) Waiting to run
Details
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
Details
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
Details
docker: build dev containers / build-dev-containers (push) Waiting to run
Details
End to End / FUSE Mount (push) Waiting to run
Details
go: build binary / Build (push) Waiting to run
Details
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
Details 
* rename

* set agent address

* refactor

* add agent sub

* pub messages

* grpc new client

* can publish records via agent

* send init message with session id

* fmt

* check cancelled request while waiting

* use sessionId

* handle possible nil stream

* subscriber process messages

* separate debug port

* use atomic int64

* less logs

* minor

* skip io.EOF

* rename

* remove unused

* use saved offsets

* do not reuse session, since always session id is new after restart

remove last active ts from SessionEntry

* simplify printing

* purge unused

* just proxy the subscription, skipping the session step

* adjust offset types

* subscribe offset type and possible value

* start after the known tsns

* avoid wrongly set startPosition

* move

* remove

* refactor

* typo

* fix

* fix changed path
 2025-03-09 23:49:42 -07:00
chrislu
f5f3b60a13 ignore 2024-05-05 12:19:21 -07:00
chrislu
928a4e8dff rename 2024-05-02 08:35:06 -07:00
chrislu
dbdb7c8abe Update .gitignore 2024-04-24 23:28:45 -07:00
Chris Lu
c471265837 build with pub sub clients for testing 2024-02-05 16:47:11 -08:00
Varun Upadhyay
77626666c5 Minor cleanup & gitignore update (#5144) 2023-12-28 20:25:43 -08:00
Chris Lu
580940bf82 Merge accumulated changes related to message queue (#5098) 
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
 2023-12-11 12:05:54 -08:00
chrislu
984b6c54cf ack interval 128 2023-09-06 23:15:29 -07:00
a
7d981a1c0e del 2022-03-28 17:26:43 +00:00
elee
921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
Lei Liu
1d9b75b536 weed.go: remove unused parameter 
Signed-off-by: Lei Liu <liul.stone@gmail.com>
 2019-06-26 10:46:32 +08:00
Chris Lu
4119c61df8 HCFS can read files 2018-12-03 20:25:57 -08:00
Chris Lu
1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu
acd8836d27 add ignore .class files 2018-09-02 14:19:47 -07:00
Chris Lu
2d13382c68 add releasing configs 2017-01-03 21:14:46 -08:00
Chris Lu
5ce6bbf076 directory structure change to work with glide 
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
 2016-06-02 18:09:14 -07:00
Chris Lu
c24c1ffd1a skip vendor folder 2016-05-23 14:28:38 -07:00
tnextday
662915e691 Delete all chunks when delete a ChunkManifest 
LoadChunkManifest can uncompress buffer
move compress.go from storage to operation because of import cycle
MakeFile add cross complete command
 2015-12-02 21:27:29 +08:00
tnextday
a4f64c0116 edit git ignore 2015-11-26 23:30:08 +08:00
Chris Lu
2495ce6707 Adjust .gitignore 2015-01-09 19:59:54 -08:00
yanyiwu
7304c840e3 update .gitignore 2015-01-08 17:38:01 +08:00
wyy
d39c62bbed add .gitignore 2014-09-25 16:34:26 +08:00