PinnedMember-onlyEfficient SIEM and Detection Engineering in 10 stepsSIEM systems and detection engineering are not just about data and detection rules. Planning and processes are becoming increasingly important over time. In 10 steps, you will learn how to approach detection in cybersecurity efficiently. 1. Just start If you have ever been programming, you will certainly be familiar with software engineering. We…Detection Engineering8 min readDetection Engineering8 min read
Apr 5Member-only10 best MITRE ATT&CK sources in one click with PandasMITRE ATT&CK is a source of knowledge about adversarial tactics and techniques. It is a common domain language in the world of cyber security. Which data source will give us the best cost effect? It depends. Every organization is different. We use different operating systems, network devices, and applications. We…Pandas4 min readPandas4 min read
Mar 19Member-onlyHow To Clean Data with Python Pandas — Vehicles registered in PolandThanks to the Open Data project, we have sources made available by Polish public entities. In this article, we will prepare and clean the Vehicles registered in Poland by province using Python and Pandas. Data We have data available in csv and xlsx (Excel) formats. …Python5 min readPython5 min read
Feb 4, 2022Member-onlyksqlDB —real-time SQL magic in the cybersecurity scenario— part 1ksqlDB is a solution from the Apache Kafka and Confluent family. It allows you to use SQL to define stream processing jobs. This story starts a series about ksqlDB. We’ll try to do something cool with Packetbeat data (network traffic monitoring). What is ksqlDB? ksqlDB (formerly ksql) is a platform for building applications…Ksqldb9 min readKsqldb9 min read
Mar 27, 2021Thanks. Sometime after the day you shared this post, this problem (SPARK-26995) was solved by the…1Kadir AlanI see that the problem was solved with symbolic link.I see that the problem was solved with symbolic link. I didn't do any updates to Dockerfile. Currently I'm busy with other projects (not Kafka Streams related)1 min read1 min read
Jan 30, 2021Member-onlyChange Data Capture — Convert your database into a stream with DebeziumHave you ever thought about creating a stream from database operations? In this story, you will learn what Change Data Capture is and how to use it while planning your system architecture. In the practical part, we will see the Debezium in action. What is Change Data Capture? Change Data Capture is a process of…Debezium6 min readDebezium6 min read
Dec 11, 2020Member-onlyHow to use Variables and XCom in Apache Airflow?It is said that Apache Airflow is CRON on steroids. It is gaining popularity among tools for ETL orchestration (Scheduling, managing and monitoring tasks). The tasks are defined as Directed Acyclic Graph (DAG), in which they exchange information. …Apache Airflow4 min readApache Airflow4 min read
Oct 31, 2020Member-onlyReadable Scala Code in Apache Spark (4 attempts)Jupyter and Apache Zeppelin is a good place to experiment with data. Unfortunately, the specifics of notebooks do not encourage to organize the code, including its decomposition and readability. We can copy cells to Intellij IDEA and build JAR, but the effect will not be stunning. We can copy cells…Scala3 min readScala3 min read
Published inThe Startup·Oct 18, 2020Member-onlyTwitter Data Analysis for the Lazy in Elastic Stack (Xbox VS PlayStation)Twitter data can be obtained in many ways, but who wants to write the code 😉. Especially one that will work 24/7. In Elastic Stack, you can easily collect and analyze data from Twitter. Logstash has input to collect tweets. …Elastic Stack6 min readElastic Stack6 min read
Published inITNEXT·Oct 6, 2020Member-onlyKafka Connect in a nutshellKafka Connect is part of the Apache Kafka platform. It is used to connect Kafka with external services such as file systems and databases. In this story you will learn what problem it solves and how to run it. Why Kafka Connect? Apache Kafka is used in microservices architecture, log aggregation, Change data…Kafka5 min readKafka5 min read
