46 Commits

Author	SHA1	Message	Date
Chris Lu	02773a6107	Accumulated changes for message queue (#6600) ... * rename * set agent address * refactor * add agent sub * pub messages * grpc new client * can publish records via agent * send init message with session id * fmt * check cancelled request while waiting * use sessionId * handle possible nil stream * subscriber process messages * separate debug port * use atomic int64 * less logs * minor * skip io.EOF * rename * remove unused * use saved offsets * do not reuse session, since always session id is new after restart remove last active ts from SessionEntry * simplify printing * purge unused * just proxy the subscription, skipping the session step * adjust offset types * subscribe offset type and possible value * start after the known tsns * avoid wrongly set startPosition * move * remove * refactor * typo * fix * fix changed path	2025-03-09 23:49:42 -07:00
Chris Lu	cc05874d06	Add message queue agent (#6463) ... * scaffold message queue agent * adjust proto, add mq_agent * add agent client implementation * remove unused function * agent publish server implementation * adding agent	2025-01-20 22:19:27 -08:00
chrislu	ec155022e7	"golang.org/x/exp/slices" => "slices" and go fmt	2024-12-19 19:25:06 -08:00
Chris Lu	dc784bf217	merge current message queue code changes (#6201) ... * listing files to convert to parquet * write parquet files * save logs into parquet files * pass by value * compact logs into parquet format * can skip existing files * refactor * refactor * fix compilation * when no partition found * refactor * add untested parquet file read * rename package * refactor * rename files * remove unused * add merged log read func * parquet wants to know the file size * rewind by time * pass in stop ts * add stop ts * adjust log * minor * adjust log * skip .parquet files when reading message logs * skip non message files * Update subscriber_record.go * send messages * skip message data with only ts * skip non log files * update parquet-go package * ensure a valid record type * add new field to a record type * Update read_parquet_to_log.go * fix parquet file name generation * separating reading parquet and logs * add key field * add skipped logs * use in memory cache * refactor * refactor * refactor * refactor, and change compact log * refactor * rename * refactor * fix format * prefix v to version directory	2024-11-04 12:08:25 -08:00
chrislu	25b2850e7d	refactor out FilerClientAccessor	2024-06-06 19:44:19 -07:00
chrislu	d1f0c40402	remove per-message debug logs	2024-05-30 09:49:08 -07:00
chrislu	df9d889489	fix size calculation	2024-05-30 09:41:04 -07:00
chrislu	f7b818483a	fix cases when buffer wraps around	2024-05-30 09:10:30 -07:00
chrislu	ad2b878130	fix inflight message tracker	2024-05-29 23:33:37 -07:00
chrislu	f0b9e163a0	avoid nil stopChan	2024-05-29 16:18:02 -07:00
chrislu	a5874954dc	less logs	2024-05-29 16:17:37 -07:00
chrislu	edc87c16f9	balance subscribers ... need to ensure the offsets are persisted	2024-05-27 17:30:16 -07:00
chrislu	6366898af2	add market	2024-05-26 14:35:22 -07:00
chrislu	96b326a304	use type ConsumerGroupInstanceId	2024-05-26 14:09:51 -07:00
chrislu	37d1ee562d	refactor	2024-05-23 09:21:48 -07:00
chrislu	d40b350f54	coordinator receives unassignment ack	2024-05-23 08:23:35 -07:00
chrislu	554ae09f82	rename	2024-05-21 10:02:07 -07:00
chrislu	6ef2d010aa	rename file	2024-05-21 09:57:59 -07:00
chrislu	67e6051585	rename Coordinator to SubCoordinator	2024-05-21 09:57:45 -07:00
chrislu	d5abffa42c	rename Balancer to PubBalancer	2024-05-21 09:56:30 -07:00
chrislu	2190d59bc3	refactor ... ToDo 1. handle unassignment on client side 2. get notified of completed unassignment 3. in consumer_group.go, balance and send unassignment	2024-05-21 08:05:56 -07:00
chrislu	3148dec1f6	avoid concurrent processing for the same key	2024-05-20 13:29:37 -07:00
chrislu	3622196881	go fmt	2024-05-20 11:05:18 -07:00
chrislu	d8ab22012a	track offset	2024-05-20 09:33:37 -07:00
chrislu	2b07a40da5	add InflightMessageTracker	2024-05-20 09:19:39 -07:00
chrislu	c6db3f31a1	start consuming ASAP	2024-05-19 14:52:38 -07:00
chrislu	b1871427c3	persist consumer group offset ... 1. use one follower 2. read write consumer group offset	2024-05-19 00:46:12 -07:00
chrislu	759a8e50cf	remove logs	2024-05-14 23:22:51 -07:00
chrislu	972e9faaa2	move initial assignment to rebalance logic	2024-05-14 09:09:36 -07:00
chrislu	6e5075e14e	move read write topic config into filer client accessor	2024-05-14 08:50:17 -07:00
chrislu	ac63f2b5a1	refactor	2024-05-13 22:03:57 -07:00
chrislu	844a1a0192	rename	2024-05-13 19:24:47 -07:00
chrislu	372bd8d71d	consumer instance passing MaxPartitionCount to coordinator	2024-05-13 18:30:10 -07:00
chrislu	2142842f82	refactor	2024-05-13 18:25:38 -07:00
chrislu	205d2285f3	setup follower by publisher ... * the subscriber would getOrGen a local partition and wait * the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.	2024-03-24 12:57:09 -07:00
chrislu	1b4484bf0a	go fmt	2024-02-29 09:38:52 -08:00
chrislu	34839237ab	refactor	2024-01-20 12:16:40 -08:00
chrislu	2828ccbb30	refactor	2024-01-20 11:41:11 -08:00
chrislu	f750a5e03b	passing timestamp	2024-01-11 23:03:55 -08:00
chrislu	6a7a679137	passing partition generation timestamp	2024-01-10 22:36:17 -08:00
chrislu	35869b5c80	subscriber can be notified of the assignment change when topic is just configured ... Next: Subscriber needs to read by the timestamp offset.	2024-01-03 13:30:30 -08:00
chrislu	2845230329	passing broker into the assignments	2023-12-28 21:13:49 -08:00
chrislu	093fdc1621	subscriber can get assignments	2023-12-28 20:35:15 -08:00
chrislu	c950a40aad	subscriber keep connected to the balancer	2023-12-28 11:56:37 -08:00
chrislu	e0727071c8	go fmt	2023-12-22 11:33:50 -08:00
Chris Lu	580940bf82	Merge accumulated changes related to message queue (#5098) ... * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * tracking topic=>broker * merge * comment	2023-12-11 12:05:54 -08:00