Maciej Szymczyk
Jun 27, 2022

--

Thx. There are two options:

1. Filebeat -> Logstash -> both S3 and Elasticsearch - but files on S3 will be JSONs

2. Filebeat -> Kafka -> Kafka Connect -> S3 - files will be Parquets (better option for Athena)

Splitting stream on Logstash/Kafka is a popular way to backup logs/events in colder/cheaper storage.

--

--

Maciej Szymczyk
Maciej Szymczyk

Written by Maciej Szymczyk

Software Developer, Big Data Engineer, Blogger (https://wiadrodanych.pl), Amateur Cyclists & Triathlete, @maciej_szymczyk

No responses yet