Chris Lu
cc05874d06
Add message queue agent ( #6463 )
...
* scaffold message queue agent
* adjust proto, add mq_agent
* add agent client implementation
* remove unused function
* agent publish server implementation
* adding agent
2025-01-20 22:19:27 -08:00
Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
...
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
ff47956530
add helper functions
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
2024-10-03 00:13:46 -07:00
chrislu
d218fe54fa
go fmt
2024-05-20 11:03:56 -07:00
chrislu
fcedf97e94
fix test
2024-05-05 22:19:59 -07:00
chrislu
30f8600a24
working
2024-05-03 21:27:06 -07:00
chrislu
7a9b115cc2
rename
2024-05-02 11:14:58 -07:00
chrislu
f2f4630029
rename field types
2024-05-02 08:59:22 -07:00
chrislu
b8af997e41
consistent with protobuf data types
2024-05-02 08:52:10 -07:00
chrislu
5bc1b70a4c
rename to WithField()
2024-05-02 08:32:15 -07:00
chrislu
b570e015ce
Update write_parquet_test.go
2024-04-28 13:56:55 -07:00
chrislu
5dc186ef01
record value API
2024-04-28 13:42:45 -07:00
chrislu
189a7fc90e
record type begin and end
2024-04-28 13:00:52 -07:00
chrislu
0da1b7b0f9
improve api
2024-04-28 12:42:58 -07:00
chrislu
1a2332c257
rename api
2024-04-28 12:40:00 -07:00
chrislu
5ce8f50546
Delete schema_builder_test.go
2024-04-28 12:39:27 -07:00
chrislu
8a0e5b6d54
struct to schema correctly
2024-04-28 12:23:54 -07:00
chrislu
f37862bffe
compile test
2024-04-28 12:23:33 -07:00
chrislu
7310d022ad
recursive struct to schema
2024-04-28 11:14:13 -07:00
chrislu
a2a872ca03
rename
2024-04-27 23:44:11 -07:00
chrislu
60103171b6
rename
2024-04-27 23:30:15 -07:00
chrislu
72b50980f4
a little bit more efficient
2024-04-25 23:59:30 -07:00
chrislu
63036133ff
minor
2024-04-25 22:02:03 -07:00
chrislu
2a9b0a9332
use parquet levels to construct row
2024-04-25 21:29:45 -07:00
chrislu
977e7988e6
toRow conversion with levels info
2024-04-25 09:14:37 -07:00
chrislu
9cb9d27b5b
file rename
2024-04-25 08:56:31 -07:00
chrislu
313097aca0
printout test values
2024-04-25 08:54:57 -07:00
chrislu
827e293010
add visitor
2024-04-25 08:54:41 -07:00
chrislu
138a027a2e
add field types but without value
2024-04-24 23:33:11 -07:00
chrislu
30b5aef62e
record type is also optional
2024-04-24 23:32:35 -07:00
chrislu
ce4597c623
delete the generated file
2024-04-24 23:29:55 -07:00
chrislu
f91484ca07
add some notes
2024-04-24 23:28:20 -07:00
chrislu
f93c4cf48c
adjust test
2024-04-24 23:24:53 -07:00
chrislu
d88c1872ac
works with single level of list
2024-04-24 23:04:47 -07:00
chrislu
d7e5f6b2a5
fix ordering
2024-04-24 01:32:02 -07:00
chrislu
9ef238b610
has a bug
2024-04-22 00:42:18 -07:00
chrislu
cc1ce3de87
all record fields are optional by default
2024-04-18 23:17:52 -07:00
chrislu
384ed7a381
add test
2024-04-18 22:41:20 -07:00
chrislu
57949f9959
support list type
2024-04-18 22:41:12 -07:00
chrislu
05d18130b6
add schema builder
2024-04-18 22:27:18 -07:00
chrislu
0847a17484
convert to parquet schema and value
2024-04-17 23:49:21 -07:00
chrislu
8415476c2d
fix test
2024-04-12 23:35:54 -07:00
chrislu
6f80dbdfe5
Create schema.go
2024-04-12 22:31:28 -07:00
chrislu
d45a372615
add schema.proto
2024-04-12 01:30:29 -07:00
