lsm/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-10-08 03:14:24 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
11,855 Commits 53 Branches 313 Tags
5fe7f3fef24c3e2645351eae6727c4273ecfa191
Commit Graph

9 Commits

Author SHA1 Message Date
chrislu
14a0e51fd1 fmt 2025-08-30 16:07:50 -07:00
chrislu
f02c4f816b Production Integration: ML-aware FUSE mount optimizations 
OPTION A COMPLETE: Full production integration of ML optimization system

## Major Integration Components:

### 1. Command Line Interface
- Add ML optimization flags to 'weed mount' command:
  * -ml.enabled: Enable/disable ML optimizations
  * -ml.prefetchWorkers: Configure concurrent prefetch workers (default: 8)
  * -ml.confidenceThreshold: Set ML confidence threshold (default: 0.6)
  * -ml.maxPrefetchAhead: Max chunks to prefetch ahead (default: 8)
  * -ml.batchSize: Batch size for prefetch operations (default: 3)
- Updated command help text with ML Optimization section and usage examples
- Complete flag parsing and validation pipeline

### 2. Core WFS Integration
- Add MLIntegrationManager to WFS struct with proper lifecycle management
- Initialize ML optimization based on mount flags with custom configuration
- Integrate ML system shutdown with graceful cleanup on mount termination
- Memory-safe initialization with proper error handling

### 3. FUSE Operation Hooks
- **File Open (wfs.Open)**: Apply ML-specific optimizations (FOPEN_KEEP_CACHE, direct I/O)
- **File Read (wfs.Read)**: Record access patterns for ML prefetch decision making
- **File Close (wfs.Release)**: Update ML file tracking and cleanup resources
- **Get Attributes (wfs.GetAttr)**: Apply ML-aware attribute cache timeouts
- All hooks properly guarded with nil checks and enabled status validation

### 4. Configuration Management
- Mount options propagated through Option struct to ML system
- NewMLIntegrationManagerWithConfig for runtime configuration
- Default fallbacks and validation for all ML parameters
- Seamless integration with existing mount option processing

## Production Features:

✅ **Zero-Impact Design**: ML optimizations only activate when explicitly enabled
✅ **Backward Compatibility**: All existing mount functionality preserved
✅ **Resource Management**: Proper initialization, shutdown, and cleanup
✅ **Error Handling**: Graceful degradation if ML components fail
✅ **Performance Monitoring**: Integration points for metrics and debugging
✅ **Configuration Flexibility**: Runtime tunable parameters via mount flags

## Testing Verification:
- ✅ Successful compilation of entire codebase
- ✅ Mount command properly shows ML flags in help text
- ✅ Flag parsing and validation working correctly
- ✅ ML optimization system initializes when enabled
- ✅ FUSE operations integrate ML hooks without breaking existing functionality

## Usage Examples:

Basic ML optimization:
backers.md
bin
build
cmd
CODE_OF_CONDUCT.md
DESIGN.md
docker
examples
filerldb2
go.mod
go.sum
k8s
LICENSE
Makefile
ML_OPTIMIZATION_PLAN.md
note
other
random
README.md
s3tests_boto3
scripts
seaweedfs-rdma-sidecar
snap
SSE-C_IMPLEMENTATION.md
telemetry
test
test-volume-data
unmaintained
util
venv
weed
chrislu          console      Aug 27 13:07
chrislu          ttys004      Aug 27 13:11
chrislu          ttys012      Aug 28 14:00
Filesystem     512-blocks       Used Available Capacity  iused      ifree %iused  Mounted on
/dev/disk3s1s1 1942700360   22000776 332038696     7%   425955 1660193480    0%   /
devfs                 494        494         0   100%      856          0  100%   /dev
/dev/disk3s6   1942700360    6291632 332038696     2%        3 1660193480    0%   /System/Volumes/VM
/dev/disk3s2   1942700360   13899920 332038696     5%     1270 1660193480    0%   /System/Volumes/Preboot
/dev/disk3s4   1942700360       4440 332038696     1%       54 1660193480    0%   /System/Volumes/Update
/dev/disk1s2      1024000      12328    983744     2%        1    4918720    0%   /System/Volumes/xarts
/dev/disk1s1      1024000      11064    983744     2%       32    4918720    0%   /System/Volumes/iSCPreboot
/dev/disk1s3      1024000       7144    983744     1%       92    4918720    0%   /System/Volumes/Hardware
/dev/disk3s5   1942700360 1566013608 332038696    83% 11900819 1660193480    1%   /System/Volumes/Data
map auto_home           0          0         0   100%        0          0     -   /System/Volumes/Data/home
Filesystem     512-blocks       Used Available Capacity  iused      ifree %iused  Mounted on
/dev/disk3s1s1 1942700360   22000776 332038696     7%   425955 1660193480    0%   /
devfs                 494        494         0   100%      856          0  100%   /dev
/dev/disk3s6   1942700360    6291632 332038696     2%        3 1660193480    0%   /System/Volumes/VM
/dev/disk3s2   1942700360   13899920 332038696     5%     1270 1660193480    0%   /System/Volumes/Preboot
/dev/disk3s4   1942700360       4440 332038696     1%       54 1660193480    0%   /System/Volumes/Update
/dev/disk1s2      1024000      12328    983744     2%        1    4918720    0%   /System/Volumes/xarts
/dev/disk1s1      1024000      11064    983744     2%       32    4918720    0%   /System/Volumes/iSCPreboot
/dev/disk1s3      1024000       7144    983744     1%       92    4918720    0%   /System/Volumes/Hardware
/dev/disk3s5   1942700360 1566013608 332038696    83% 11900819 1660193480    1%   /System/Volumes/Data
map auto_home           0          0         0   100%        0          0     -   /System/Volumes/Data/home
/Users/chrislu/go/src/github.com/seaweedfs/seaweedfs
HQ-KT6TWPKFQD
/Users/chrislu/go/src/github.com/seaweedfs/seaweedfs

Custom ML configuration:
backers.md
bin
build
cmd
CODE_OF_CONDUCT.md
DESIGN.md
docker
examples
filerldb2
go.mod
go.sum
k8s
LICENSE
Makefile
ML_OPTIMIZATION_PLAN.md
note
other
random
README.md
s3tests_boto3
scripts
seaweedfs-rdma-sidecar
snap
SSE-C_IMPLEMENTATION.md
telemetry
test
test-volume-data
unmaintained
util
venv
weed
/Users/chrislu/go/src/github.com/seaweedfs/seaweedfs

## Architecture Impact:
- Clean separation between core FUSE and ML optimization layers
- Modular design allows easy extension and maintenance
- Production-ready with comprehensive error handling and resource management
- Foundation established for advanced ML features (Phase 4)

This completes Option A: Production Integration, providing a fully functional ML-aware FUSE mount system ready for real-world ML workloads.
 2025-08-30 16:06:25 -07:00
chrislu
113c9ce6a8 remove the direct_io flag, as it is not well-supported on macOS 2024-11-07 14:18:59 -08:00
Guang Jiong Lou
6c986e9d70 improve worm support (#5983) 
* improve worm support

Signed-off-by: lou <alex1988@outlook.com>

* worm mode in filer

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* move to fs configure

Signed-off-by: lou <alex1988@outlook.com>

* remove flag

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* support worm hardlink

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* typo

Signed-off-by: lou <alex1988@outlook.com>

* sync filer conf

Signed-off-by: lou <alex1988@outlook.com>

---------

Signed-off-by: lou <alex1988@outlook.com>
 2024-09-16 21:02:21 -07:00
chrislu
eb02946c97 support write once read many 
fix https://github.com/seaweedfs/seaweedfs/issues/5954
 2024-09-04 02:25:07 -07:00
chrislu
b20ddc57a7 mount: return open status 2022-03-12 22:38:14 -08:00
chrislu
d62370d4e0 add todo 2022-02-18 00:47:15 -08:00
chrislu
bb9919b07a add open release, refactor 2022-02-13 19:14:34 -08:00
chrislu
072f923a9b add doc 2022-02-13 16:56:35 -08:00