To send logs from your local computer, run the command below. You’ll see topics being created in Confluent Control Center and also have the option of running osquery locally on your host to add to the logs being generated from host1 and host2. Logs from host1 and host2 containers will begin sending osquery logs to Confluent Platform. Make ps wait 30 seconds for everything to start Git clone change directory the location which builds the Confluent PlatformĬd demo-scene/osquery/cp build and create the Kafka cluster Osquery (only if you want to send logs from your local computer).Make (only if you would like to utilize the Makefile in the repository).See this configuration for details.Īpache Avro™: By using librdkafka directly in your extension, you also have the option to convert the JSON messages that osquery produces into Avro and utilize Confluent Schema Registry. NET), which has the ability to publish osquery logs to Kafka, Confluent Platform, and Confluent Cloud without the issues and limitations that come with the prepackaged osquery Kafka producer logger plugin.Ĭloud: You can easily configure the osquery Python extension to publish to Confluent Cloud just by adding the required Confluent Cloud properties. In this demonstration, I’ve written an extension in Python that uses librdkafka (the Kafka-native library for Python, C++, and. The extensions feature in osquery enables you to create plugins in different languages (i.e., Python) and load it into osqueryd using an Apache Thrift™ based API. This, however, requires a recompile of the source and may not be an option especially if the OS’s in the enterprise already have an existing osquery daemon installed and running. In osquery, you have the option to add a custom plugin in C++. A better alternative is to use a custom osquery extension. It only supports a limited set of tunable Kafka producer configurations, and several issues related to the Kafka producer have also been reported, which are well documented in the osquery GitHub repository. It is not cloud ready and therefore cannot publish logs to Confluent Cloud. Since it doesn’t require a Kafka connector, there is no requirement for building a Connect cluster to simplify the architecture. The osquery Kafka producer logger plugin is a simple way to submit logs to Apache Kafka or Confluent Platform. Only one of the prepackaged plugins works without a Kafka connector, and that’s the Kafka producer. There are many ways to get osquery logs into Kafka using the prepackaged logger plugins paired with a Kafka connector from Confluent Hub. Using Kafka Connect to capture osquery logs The following logger plugins are built into osquery by default: Users have the option to build their own osquery logger plugin and recompile the project, but most users will use the default logger plugins packaged with it. Osquery comes with a daemon ( osqueryd) that can output its log results through components called logger plugins. I will be using a few of these packs to send logs to Confluent Platform. The osquery packs repository includes hardware-monitoring, incident-response, it-compliance, osx-attacks, unwanted-chrome-extensions, windows-attacks, etc. Fortunately, osquery has published a set of packs, which are prewritten queries (with descriptions) that gather events related to a specific behavioral category. If you are new to osquery, it can be difficult to determine which queries to use to begin inspecting logs. The full working implementation is provided at the end which you can clone and modify yourself. You can download/install osquery to follow along. Supported operating systems are Windows, macOS (OS X), CentOS, and FreeBSD. For this use case, I’ll use the Confluent Platform to curate all streams of osquery traffic and send it to Apache Kafka ®. Osquery is a powerful tool that can be used in modern security information and event management (SIEM) implementations to predict and detect anomalous behavior in real time using Confluent Platform or Confluent Cloud. The daemon that comes with osquery provides integration solutions to enable more modern techniques for publishing and searching logs for anomalous behavior. The SQL syntax makes it simpler for users familiar with SQL to look up OS information where it previously required knowledge of many terminal commands. It enables users to easily query important, low-level analytics on the OS. What’s unique about osquery is that it uses basic SQL commands against a relational data model that describes a device. Osquery (developed by Facebook) is an open source tool used to gather audit log events from an operating system (OS).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |