Commit Graph

37 Commits

Author SHA1 Message Date
Chris Lu
12f50d37fa
test versioning also (#7000)
* test versioning also

* fix some versioning tests

* fall back

* fixes

Never-versioned buckets: No VersionId headers, no Status field
Pre-versioning objects: Regular files, VersionId="null", included in all operations
Post-versioning objects: Stored in .versions directories with real version IDs
Suspended versioning: Proper status handling and null version IDs

* fixes

Bucket Versioning Status Compliance
Fixed: New buckets now return no Status field (AWS S3 compliant)
Before: Always returned "Suspended" 
After: Returns empty VersioningConfiguration for unconfigured buckets 
2. Multi-Object Delete Versioning Support
Fixed: DeleteMultipleObjectsHandler now fully versioning-aware
Before: Always deleted physical files, breaking versioning 
After: Creates delete markers or deletes specific versions properly 
Added: DeleteMarker field in response structure for AWS compatibility
3. Copy Operations Versioning Support
Fixed: CopyObjectHandler and CopyObjectPartHandler now versioning-aware
Before: Only copied regular files, couldn't handle versioned sources 
After: Parses version IDs from copy source, creates versions in destination 
Added: pathToBucketObjectAndVersion() function for version ID parsing
4. Pre-versioning Object Handling
Fixed: getLatestObjectVersion() now has proper fallback logic
Before: Failed when .versions directory didn't exist 
After: Falls back to regular objects for pre-versioning scenarios 
5. Enhanced Object Version Listings
Fixed: listObjectVersions() includes both versioned AND pre-versioning objects
Before: Only showed .versions directories, ignored pre-versioning objects 
After: Shows complete version history with VersionId="null" for pre-versioning 
6. Null Version ID Handling
Fixed: getSpecificObjectVersion() properly handles versionId="null"
Before: Couldn't retrieve pre-versioning objects by version ID 
After: Returns regular object files for "null" version requests 
7. Version ID Response Headers
Fixed: PUT operations only return x-amz-version-id when appropriate
Before: Returned version IDs for non-versioned buckets 
After: Only returns version IDs for explicitly configured versioning 

* more fixes

* fix copying with versioning, multipart upload

* more fixes

* reduce volume size for easier dev test

* fix

* fix version id

* fix versioning

* Update filer_multipart.go

* fix multipart versioned upload

* more fixes

* more fixes

* fix versioning on suspended

* fixes

* fixing test_versioning_obj_suspended_copy

* Update s3api_object_versioning.go

* fix versions

* skipping test_versioning_obj_suspend_versions

* > If the versioning state has never been set on a bucket, it has no versioning state; a GetBucketVersioning request does not return a versioning state value.

* fix tests, avoid duplicated bucket creation, skip tests

* only run s3tests_boto3/functional/test_s3.py

* fix checking filer_pb.ErrNotFound

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_handlers_copy.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_bucket_config.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update test/s3/versioning/s3_versioning_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-19 21:43:34 -07:00
Chris Lu
d892538d32
More efficient copy object (#6665)
* it compiles

* refactored

* reduce to 4 concurrent chunk upload

* CopyObjectPartHandler

* copy a range of the chunk data, fix offset size in copied chunks

* Update s3api_object_handlers_copy.go

What the PR Accomplishes:
CopyObjectHandler - Now copies entire objects by copying chunks individually instead of downloading/uploading the entire file
CopyObjectPartHandler - Handles copying parts of objects for multipart uploads by copying only the relevant chunk portions
Efficient Chunk Copying - Uses direct chunk-to-chunk copying with proper volume assignment and concurrent processing (limited to 4 concurrent operations)
Range Support - Properly handles range-based copying for partial object copies

* fix compilation

* fix part destination

* handling small objects

* use mkFile

* copy to existing file or part

* add testing tools

* adjust tests

* fix chunk lookup

* refactoring

* fix TestObjectCopyRetainingMetadata

* ensure bucket name not conflicting

* fix conditional copying tests

* remove debug messages

* add custom s3 copy tests
2025-07-11 18:51:32 -07:00
Chris Lu
1defee3d68
Add admin component (#6928)
* init version

* relocate

* add s3 bucket link

* refactor handlers into weed/admin folder

* fix login logout

* adding favicon

* remove fall back to http get topology

* grpc dial option, disk total capacity

* show filer count

* fix each volume disk usage

* add filers to dashboard

* adding hosts, volumes, collections

* refactor code and menu

* remove "refresh" button

* fix data for collections

* rename cluster hosts into volume servers

* add masters, filers

* reorder

* adding file browser

* create folder and upload files

* add filer version, created at time

* remove mock data

* remove fields

* fix submenu item highlighting

* fix bucket creation

* purge files

* delete multiple

* fix bucket creation

* remove region from buckets

* add object store with buckets and users

* rendering permission

* refactor

* get bucket objects and size

* link to file browser

* add file size and count for collections page

* paginate the volumes

* fix possible SSRF

https://github.com/seaweedfs/seaweedfs/pull/6928/checks?check_run_id=45108469801

* Update weed/command/admin.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/command/admin.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix build

* import

* remove filer CLI option

* remove filer option

* remove CLI options

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-01 01:28:09 -07:00
MaratKarimov
ba3afd1803
Tarantool filer store (#6669)
Co-authored-by: Marat Karimov <m.karimov@digitalms.ru>
2025-03-29 21:12:06 -07:00
Konstantin Lebedev
5bddf0c085
[shell] volume.balance collect volume servers by dc rack node (#6191)
* chore: balance by rack

* fix: rm check lock

* fix: selected racks

* fix: selected nodes

* fix: containts

* fix: one collectVolumeServersByDcRackNode

* fix: revert lock and add lock

* fix: panic test

* revert noLock
2024-11-03 11:08:45 -08:00
Konstantin Lebedev
33537ae29f
[s3] fix s3 test_multipart_get_part (#5476)
* try fix s3  test_multipart_get_part

* add passed s3 tests

* fix SeaweedFSUploadId

* rm spaces

* convert part request to range

* add passed s3 tests of multipart
2024-04-14 10:41:32 -07:00
Konstantin Lebedev
35cba720a5
[s3] add s3 pass test_multipart_upload (#5474)
add s3 pass test_multipart_upload
2024-04-07 11:51:22 -07:00
Konstantin Lebedev
a7fc723ae0
chore: add status code for request_total metrics (#5188) 2024-01-10 10:05:27 -08:00
chrislu
7a50762b9e reduce binary size by skipping rclone 2023-11-23 00:48:27 -08:00
Konstantin Lebedev
6175102ccf
add make benchmark with WARP (#4914)
add make benchmark

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-15 23:02:27 -07:00
Konstantin Lebedev
f8b94cac0e
[s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859)
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface

https://github.com/seaweedfs/seaweedfs/issues/4519

* fix: test bucket acl default and
adjust the variable names

* fix: s3 api config test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-25 08:34:12 -07:00
Konstantin Lebedev
7e09a548a6 exclude directories to sync on filer 2022-07-27 19:22:57 +05:00
yulai.li
824cbe32be Make tikv filer enable/disable by build tags 2022-06-27 10:57:24 +08:00
Konstantin Lebedev
5b388ed6c1 initial advancedtls 2022-06-23 23:32:15 +05:00
chrislu
1d0c53ea56 remote storage: stop supporting hdfs as a remote storage 2022-06-20 14:15:59 -07:00
Konstantin Lebedev
f58adaab25 build tag ydb since the growth of the binary is 5 mb 2022-05-03 23:20:09 +05:00
chrislu
ad01c63b84 conditionally skip hdfs related code 2022-04-21 01:43:01 -07:00
chrislu
4aa39ef33f conditionally compile sqlite 2022-04-21 01:22:38 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
Chris Lu
93741a6752 remove unused travis related files
fix https://github.com/chrislusf/seaweedfs/issues/2277
2021-08-23 13:33:33 -07:00
Konstantin Lebedev
1f301e4ba4 gccgo build 2021-08-17 20:27:47 +05:00
Chris Lu
6e8c2bfdd0 fix README and makefile
fix https://github.com/chrislusf/seaweedfs/issues/1548
2020-10-20 18:24:10 -07:00
mwpeterson
80fb139a10 add 5_byte_arm63 to release 2020-07-29 14:07:04 -04:00
mwpeterson
9f6ecfbd42 add 5-byte large disk builds for arm and arm64 2020-07-28 15:57:49 -04:00
Chris Lu
bc2ec6774d inject git version into build 2020-06-02 00:10:38 -07:00
Chris Lu
d9a997cb9e work with travis 2019-09-03 11:26:11 -07:00
Chris Lu
6361e2b2fc fix travis 2019-09-03 10:20:16 -07:00
Chris Lu
e33f5f5b33 travis build 2019-09-03 09:24:35 -07:00
Chris Lu
88ba08cb9e add large disk support 2019-04-08 22:32:42 -07:00
Michael Peick
57e159b369 Build statically linked release executables. 2018-11-29 13:26:48 +01:00
Chris Lu
19159ec479 adjust makefile and travis 2017-01-03 22:20:15 -08:00
Chris Lu
9cb034d49e adjust makefile 2017-01-03 22:11:33 -08:00
Chris Lu
d1562e48e1 default to release 2017-01-03 22:08:53 -08:00
Chris Lu
2d13382c68 add releasing configs 2017-01-03 21:14:46 -08:00
Chris Lu
bb839ba268 fix 2016-06-05 10:57:14 -07:00
tnextday
662915e691 Delete all chunks when delete a ChunkManifest
LoadChunkManifest can uncompress buffer
move compress.go from storage to operation because of import cycle
MakeFile add cross complete command
2015-12-02 21:27:29 +08:00
euforia
94757a209b Added basic makefile 2015-11-17 18:03:23 -08:00