Member-only story

ksqlDB —real-time SQL magic in the cybersecurity scenario— part 1

Maciej Szymczyk
9 min readFeb 4, 2022

ksqlDB is a solution from the Apache Kafka and Confluent family. It allows you to use SQL to define stream processing jobs. This story starts a series about ksqlDB. We’ll try to do something cool with Packetbeat data (network traffic monitoring).

What is ksqlDB?

ksqlDB (formerly ksql) is a platform for building applications that process data streams from Kafka. It’s built on top of Kafka Streams, a library for building applications that process data to and from Kafka.

The DB word suggests that this is a streaming database. There’s something to that. We have streams, tables, materialized views, queries, and… we operate with SQL.

Why is SQL such a big advantage? You can find a series of posts about Kafka Streams on my blog. Creating a Kafka Streams application requires having an IDE, programming knowledge, building the application, so it is a complex and complicated process. Writing a few lines of SQL in ksqlDB is much faster. Spark SQL, Flink SQL, Beam SQL… turns out SQL is cool 😊.

ksqlDB in Cybersecurity?

ksqlDB caught my attention for a reason. It turns out that you can translate sigma rules to ksqlDB queries. Sigma rules are generic detection rules that can be translated to…

Create an account to read the full story.

The author made this story available to Medium members only.
If you’re new to Medium, create a new account to read this story on us.

Or, continue in mobile web

Already have an account? Sign in

Maciej Szymczyk
Maciej Szymczyk

Written by Maciej Szymczyk

Software Developer, Big Data Engineer, Blogger (https://wiadrodanych.pl), Amateur Cyclists & Triathlete, @maciej_szymczyk

Responses (1)

Write a response

Nice write up - have you thought of sending packetbeat events to Amazon S3 where they can be queried using Athena?