ksqlDB —real-time SQL magic in the cybersecurity scenario— part 1

ksqlDB is a solution from the Apache Kafka and Confluent family. It allows you to use SQL to define stream processing jobs. This story starts a series about ksqlDB. We’ll try to do something cool with Packetbeat data (network traffic monitoring).

What is ksqlDB?

ksqlDB (formerly ksql) is a platform for building applications that process data streams from Kafka. It’s built on top of Kafka Streams, a library for building applications that process data to and from Kafka.

The DB word suggests that this is a streaming database. There’s something to that. We have streams, tables, materialized views, queries, and… we operate with SQL.

Why is SQL such a big advantage? You can find a series of posts about Kafka Streams on my blog. Creating a Kafka Streams application requires having an IDE, programming knowledge, building the application, so it is a complex and complicated process. Writing a few lines of SQL in ksqlDB is much faster. Spark SQL, Flink SQL, Beam SQL… turns out SQL is cool 😊.

ksqlDB in Cybersecurity?

ksqlDB caught my attention for a reason. It turns out that you can translate sigma rules to ksqlDB queries. Sigma rules are generic detection rules that can be translated to…