lsm/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2025-11-24 08:46:54 +08:00
Code Issues Actions 17 Packages Projects Releases Wiki Activity
11,528 Commits 51 Branches 316 Tags
c26299b05a67135636664fbbe57d9328a67288ad
Commit Graph

30 Commits

Author SHA1 Message Date
dependabot[bot]
216c52e377 chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679) 
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0

Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: gocloud.dev
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix error

* fix printing errors

* Update go.mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2025-03-31 21:42:54 -07:00
Chris Lu
cc05874d06 Add message queue agent (#6463) 
* scaffold message queue agent

* adjust proto, add mq_agent

* add agent client implementation

* remove unused function

* agent publish server implementation

* adding agent
 2025-01-20 22:19:27 -08:00
Chris Lu
dc784bf217 merge current message queue code changes (#6201) 
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
 2024-11-04 12:08:25 -08:00
chrislu
554ae09f82 rename 2024-05-21 10:02:07 -07:00
chrislu
5038577f7e Merge branch 'master' into mq 2024-05-20 11:04:53 -07:00
chrislu
d218fe54fa go fmt 2024-05-20 11:03:56 -07:00
chrislu
6e5075e14e move read write topic config into filer client accessor 2024-05-14 08:50:17 -07:00
chrislu
53d1d2b78a save schema when configuring topic 2024-04-12 22:29:53 -07:00
Chris Lu
656b78d1af proxy to broker leader 2024-02-05 18:00:50 -08:00
Chris Lu
4dc6681833 proxy requests to lock owner 2024-02-05 16:46:40 -08:00
chrislu
9e6ea80dff read existing topic conf 2024-01-28 14:27:52 -08:00
chrislu
d268fbe18a when configure, cancel existing assignments 2024-01-22 11:57:17 -08:00
chrislu
c3f8530f97 refactor 2024-01-22 11:06:53 -08:00
chrislu
5f35857cd0 no need to assign partitions explicitly 2024-01-22 11:04:47 -08:00
chrislu
e8b05ecc91 add/remove assigned partitions 2024-01-22 10:47:39 -08:00
chrislu
861ad732ca refactor 2024-01-22 09:00:22 -08:00
chrislu
47924afa1c refactor 2024-01-22 08:52:44 -08:00
chrislu
7121ae9617 separate out read topic conf and ensure topic active assignments 2024-01-22 00:51:31 -08:00
chrislu
a5b906c0dd refactor 2024-01-21 01:23:43 -08:00
chrislu
bc8d2a01cc create local topic partition from config on filer 2024-01-17 00:15:52 -08:00
chrislu
be0c426dc7 simplify to LookupTopicPartitions(topic) 2024-01-16 09:30:46 -08:00
chrislu
db3670a3a5 simplify api 2024-01-16 08:55:47 -08:00
chrislu
34a78ffad0 remove isForPublish from LookupTopicBrokers 
also adds a return parameter: whether the topic exists or not
 2024-01-16 08:52:42 -08:00
chrislu
ba73199174 reuse local partition 2024-01-15 21:22:41 -08:00
chrislu
fa59a5d67e read from disk if not in memory 2024-01-15 00:20:12 -08:00
chrislu
6a7a679137 passing partition generation timestamp 2024-01-10 22:36:17 -08:00
chrislu
8af61dec91 avoid empty spaces in file name 2024-01-09 09:00:54 -08:00
chrislu
aed54eda61 refactor 2024-01-09 08:01:01 -08:00
chrislu
d51efddf5c flush to disk 
Need to do: read from disk
 2024-01-08 23:27:02 -08:00
Chris Lu
580940bf82 Merge accumulated changes related to message queue (#5098) 
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
 2023-12-11 12:05:54 -08:00