Ubuntu – change ulimit for kafka, do not ignore

Hi, Wanna share with you what managed to take me half a day to clarify. I just read in the following article https://docs.confluent.io/current/kafka/deployment.html#file-descriptors-and-mmap and learned that in order to optimize kafka, you will need to also change the maximum number of open files. It is nice, but our clusters are deployed on Ubuntu and the …

Python dictionary construction from process list

Hi, This is out of my expertise but i wanted to shared it anyways. One colleague wanted to help him with the creation of a pair key:value from one command that lists the processes, in python. With a little bit of testing i came to the following form: import os import subprocess from subprocess import …

Securing kafka-manager endpoints with iptables rules behind traefik

Hi, One extra addition to my traefik balancing article from http://log-it.tech/2017/08/19/puppet-implementation-traefik-load-balancer-kafka-manager/ is that even so now we have the balancing capability we still need to restrict access to unsecured endpoint. I thought all the code to be deployable on all of the nodes. If this is taken in consideration, our issue with the firewall rules …

Small Mirror Maker test between different Kafka clusters

Hi, Today i am trying to show you what i have been playing with for the last day. There was a business case in which some colleagues from Analytics wanted to replicate all the data from other systems in their cluster. We will start with this, two independent configured clusters with 3 servers each (on …

Check Kafka JMX node stats using JConsole

Hi, As you probably know, Kafka is already publishing a lot of performance data on JMX to be collected. In order to do this, you will need to install jconsole (for Windows it’s already embedded in the jdk installation, for Linux you can use this article to check it out https://www.garron.me/en/linux/find-which-package-library-belongs.html. After you have done that, …

Small Vagrant config file for Rancher deploy

Hi, Just wanted to post this also, if it’s not that nice the config using a jumpserver, surely we can convert that to code (Puppet/Ansible), you can also use Vagrant. The main issue that i faced when i tried to create my setup is that for a reason (not really sure why, Vagrant on Windows …

Register RancherOs to the Rancher framework

Hi, After we were able to login via ssh on our machines, it’s time to put them to use by subscribing them to a orchestrating framework. One free and pretty powerful framework of such kind is given by the same company. In order to use it you will need to have Docker installed, more info …