Multiple field query in ELK from Python

Morning, There are a lot of pages on how to query ELK stack from Python client library, however, it’s still hard to grab a useful pattern. What I wanted is to translate some simple query in Kibana like redis.info.replication.role:master AND beat.hostname:*test AND tags:test into a useful Query DSL JSON. It’s worth mentioning that the Python …

Kafka cluster nodes and controller using golang

Hi, Using the golang library for zookeeper from here you can get very easily the nodes that are registered in the cluster controller node. In order to install this module, beside needing to setup the GOPATH you will have also to install packages from linux distro repo called: bzr, gcc, libzookeeper-mt-dev Once all of this …

Golang example for kafka service restart script

Hi, Not much to say, a pretty decent script for Kafka service restart(i tried to write it for our rolling upgrade procedure) that it’s still work in progress. If there are any changes that needed to be made to it, i will post it. Here is the script: package main import ( “os/exec” “strings” “fmt” …

Command to start sysdig container – redundant but useful

Hi, This is more like a easier way to find the command without searching the net: docker run -it –rm –name=sysdig –privileged=true \ –volume=/var/run/docker.sock:/host/var/run/docker.sock \ –volume=/dev:/host/dev \ –volume=/proc:/host/proc:ro \ –volume=/boot:/host/boot:ro \ –volume=/lib/modules:/host/lib/modules:ro \ –volume=/usr:/host/usr:ro \ sysdig/sysdig The actual command on starting a sysdig container. I will get more in depth with some Kafka cluster aggregated …

Consumer group coordinator in Kafka using some scala script

Morning, Just a small post regarding returning a consumer group coordinator for a specific consumer group. We had the issue that consumer groups are re-balancing and we didn’t knew if it’s related to application logic or the Consumer Group Coordinator was changing and the Kafka cluster was reassign a different one each time. So, a …

Apache Kafka technical logs to ELK stack

Morning, Just wanted to share with you the following article: https://www.elastic.co/blog/monitoring-kafka-with-elastic-stack-1-filebeat I will try to share my experience once i have all i need to start working on it. As far as i understood from managing Kafka for a while, monitoring is not enough, this is mandatory in order to manage you clusters in a …

List differences between two sftp hosts using golang

Hi, Just as a intermediate post as i wanted to play a little bit with golang, let me show you what i managed to put together in some days. I created a virtual machine on which i installed docker and grabbed a sftp image. You can try first two from Docker Hub, it should work. …

Memory debug by Heroku guys on Apache Kafka – nice one

Hi, I know, i should write more about my experience with Apache Kafka, have patience, it’s still building, but until then please check this article: https://blog.heroku.com/fixing-kafka-memory-leak Be aware of the things that you want to include in functionalities and code that is written beside Apache Kafka functionalities, it might get you in to trouble. I …

Configure Jupyter Notebook on Raspberry PI 2 for remote access and scala kernel install

Hi, This is a continuation of the previous article regarding Jupyter Notebook (http://log-it.tech/2017/09/02/installing-jupyter-notebook-raspberry-pi-2/) Let’s start with my modification in order to have an remote connection to it. It first needs a password in the form of password hash. To generate this pass run python cli and execute this code from IPython.lib import passwd;passwd(“your_custom_password”). Once you …