Installation

Kafka NB server deployment on Rocky Linux 8.7

How to deploy a single-node Kafka cluster with schema registry in Docker on a Rocky Linux 8.7

Read before proceeding!

This procedure is intended for testing SANnav monitoring in STOR2RRD.
The solution described in this guide comes without guarantee or support.

Kafka deployment
Management Scripts
Basic Troubleshooting

Kafka deployment

Install Rocky Linux 8.7

The following steps require root priviledges.

Install minimal system
Configure VM networking
Create DNS records for Kafka VM
Update OS
```
dnf clean all; dnf -y update
```

Disable SELinux and reboot

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
reboot

Disable or configure firewall
Kafka is using ports: 9093, 9092 and 8081 if you need to configure the firewall.
```
systemctl stop firewalld; systemctl disable firewalld
```

Install additional packages

dnf install -y bash-completion wget bind-utils tar jq curl nmap-ncat

Install docker and docker-compose

The following steps require root priviledges.

Docker

dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y docker-ce docker-ce-cli containerd.io
systemctl enable --now docker

Docker Compose

curl -L "https://github.com/docker/compose/releases/download/v2.6.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

Create user and install helper scripts

Create user stor2rrd
This step requires root priviledges.

groupadd stor2rrd
useradd -m -g stor2rrd -G docker stor2rrd
passwd stor2rrd

Download and unpack helper scripts

cd /home/stor2rrd
wget https://download.stor2rrd.com/patch/stor4kafka.tgz
tar zxvf stor4kafka.tgz
chown -R stor2rrd:stor2rrd stor2rrd

Firewall Settings

Make sure all required ports are configured in the firewall:

STOR2RRD VM ➡ SANnav

API TCP/443

SANnav ➡ Kafka VM

Kafka TCP/9092 and 9003
Schema registry TCP/8081

STOR2RRD VM ➡ Kafka VM

Kafka TCP/9092 and 9003
Schema registry TCP/8081

Deploy and register Kafka NB

Perform the following steps as stor2rrd user.

Create configuration
Following script will create /home/stor2rrd/stor2rrd/etc/kafkanb direcotry and populate it with necessary configuration files.
You may be asked to enter the path to STOR2RRD directory.
You will have to confirm or enter the local host's IP address.
```
cd /home/stor2rrd/stor2rrd/bin
./kafka_configure.sh
```
Deploy containers
This script will create /home/stor2rrd/stor2rrd/kafkanb-run direcotry, pull required images from Docker Hub and start Kafka NB containers.
```
cd /home/stor2rrd/stor2rrd/bin
./kafka_deploy.sh
```

Check containers' status after a minute
All three containers have to be up and running.

docker ps

CONTAINER ID   IMAGE                                    COMMAND                  CREATED      STATUS      PORTS                                                           NAMES
1f76f3f3b0df   confluentinc/cp-schema-registry:latest   "/etc/confluent/dock…"   3 days ago   Up 3 days   0.0.0.0:8081->8081/tcp, :::8081->8081/tcp                       kafkanb_schema_registry
55338a58eb4e   confluentinc/cp-kafka:latest             "/etc/confluent/dock…"   3 days ago   Up 3 days   0.0.0.0:9092-9093->9092-9093/tcp, :::9092-9093->9092-9093/tcp   kafkanb_broker
057f93a746a0   confluentinc/cp-zookeeper:latest         "/etc/confluent/dock…"   3 days ago   Up 3 days   2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp   kafkanb_zookeeper

Register Kafka NB in SANnav
Following script will register Kafks NB server in SANnav.
It requires SANnav IP addres and administrator credentials.
It may also require confirmation of the local host's IP address.
```
cd /home/stor2rrd/stor2rrd/bin
./kafka_register.sh
    
```

Management Scripts

The /home/stor2rrd/stor2rrd/bin directory contains several useful scripts for administering the Kafka NB server.

kafka_deploy.sh
Creates and starts docker containers with Kafka northbound server during initial installation.
Creates /home/stor2rrd/stor2rrd/kafkanb-run directory.
You can use this script to re-deploy Kafka NB server if necessary.
kafka_register.sh
Registers Kafka NB server to SANnav.
Requires SANnav IP addres and administrator credentials.
It may also require confirmation of the local host's IP address.
kafka_listnb.sh
Lists northbound servers registered to SANnav.
kafka_stop.sh
Stops Kafka NB containers without removing them.
kafka_start.sh
Starts previously stopped Kafka NB containers.
kafka_unregister.sh
Removes previously registered Kafka NB server from SANnav.
kafka_delete.sh
Force stops and deletes Kafka NB containers.
Removes /home/stor2rrd/stor2rrd/kafkanb-run directory.

Basic Troubleshooting

Check log the file: /home/stor2rrd/stor2rrd/logs/kafkanb.log

Check Kafka NB containers' status

All three containers must be Up

docker ps

CONTAINER ID   IMAGE                                    COMMAND                  CREATED      STATUS      PORTS                                                           NAMES
1f76f3f3b0df   confluentinc/cp-schema-registry:latest   "/etc/confluent/dock…"   3 days ago   Up 3 days   0.0.0.0:8081->8081/tcp, :::8081->8081/tcp                       kafkanb_schema_registry
55338a58eb4e   confluentinc/cp-kafka:latest             "/etc/confluent/dock…"   3 days ago   Up 3 days   0.0.0.0:9092-9093->9092-9093/tcp, :::9092-9093->9092-9093/tcp   kafkanb_broker
057f93a746a0   confluentinc/cp-zookeeper:latest         "/etc/confluent/dock…"   3 days ago   Up 3 days   2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 3888/tcp   kafkanb_zookeeper

List northbound servers registered in SANnav

kafkaClusterUrl and schemaRegistryUrl must match STOR2RRD host's IP address
FC Port and Switch streams must be enabled: streamState: 1

/home/stor2rrd/stor2rrd/bin/kafka_listnb.sh

# SANnav IP address, user and password
Make sure user SANnav user has a priviledge to register and manage Northbound servers
SANnav IP address [192.168.0.10]:
SANnav username [Administrator]:
SANnav password:
[
  {
    "name": "kafka4stor",
    "kafkaClusterUrl": "192.168.0.24:9093",
    "schemaRegistryUrl": "http://192.168.0.24:8081",
    "caPublicCertificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMVENDQWhXZ0F3SUJBZ0lVYmxJa0Z5R3pnWTRUelNJZTVvU2tTZTA3b1Rzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0pqRU9NQXdHQTFVRUNnd0ZXRzl5ZFhneEZEQVNCZ05WQkFNTUMxaHZjblY0SUV0aFptdGhNQjRYRFRJeQpNRFl5TWpFME1EWTFORm9YRFRNeU1EWXhPVEUwTURZMU5Gb3dKakVPTUF3R0ExVUVDZ3dGV0c5eWRYZ3hGREFTCkJnTlZCQU1NQzFodmNuVjRJRXRoWm10aE1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0MKQVFFQW96cHl1WmlCcjdFK3NMeDgxWkcvdXZ1V2M2eHlsbXY4WE1ZVlgvSnlkU2Y4TE82czlkSkd0WTBBVXZOYwpuK3V6VFZITnNZbDhwczN1eW1sVDF6MXJzY3JYQXlhRDBMTmtKZFRTVG02SEpCT2czSzE5Yk1hMXBFUVd1b0k1CkFBTHFtK1ZDWnRodXZGVkdLZWI0MncxR2NlNlhsVTdWWEpkcGx2eFdRUDFmdjN6QVF6ckRQS0RTcStQMUhablYKZ2l4T2V1S1FkMFFIS3NQb1lSeStoS1Iyc2NjUis3a0xYTXJtRFIvMnA3OWlZR2U4SzN1US9INmc3MTNGakdlSgpNWi9ocHZoWHFxL2J2cjhlQm92VUdRSjRJOXRGSHQyWG9SWVpHaWVjU25GdE1rOFNCZUV6am52SzRFTTlQcm1zClBHM0RRMTdtOUZQclpKMkYvYk1tcERpQ2h3SURBUUFCbzFNd1VUQWRCZ05WSFE0RUZnUVU2QmhYUXdqbGdjZ28KWmhGeHlSMnpoM3pLczlBd0h3WURWUjBqQkJnd0ZvQVU2QmhYUXdqbGdjZ29aaEZ4eVIyemgzektzOUF3RHdZRApWUjBUQVFIL0JBVXdBd0VCL3pBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQW5UbkVuVk81RUljTWNlaitiMkhNCkFLVGxmd3c3bXBLc2N0QTN0SXNnOEFuZnVFSTZaaFBueTFKdGkwVC9abHY2clJnanBnaUF3TmV4bFdwK25vUVcKUjQybm92aDRNanZqTFBrU1d1OHdlR0w3SUM3aFhiTlYrYVhpWjZUNFAydGFDVk1UNyQ0R6djY4SUo4NkRGZApkMWJW42NUNRVGtEWFdiZW9PZTempXcUhVbnM0TDIrY2VJMDZDFndkdlMTRoWlMkR3OFdZemJpWHpxCm9TlYTmRMTTlFYkgwV2aXNCVllTUnc1M1RJMmZ0dOVlZBYmZxYnNFdnNE55b0daNldsaEx0ZraVgKQzRIMEhiVaZHFyNW13MzV6YUwxdFQld6TFlONWNQbkFQNEUDJ4ZUpyNmV2TmFaa3ZmVnVyRjJ5F2MworUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0=",
    "id": 18,
    "connectionState": 0,
    "connectionStateReason": "SANnav successfully connected to Northbound Server.",
    "streamDetails": [
      {
        "streamType": 1,
        "streamName": "FC Port",
        "streamState": 1
      },
      {
        "streamType": 2,
        "streamName": "Eth/GigE Port",
        "streamState": 0
      },
      {
        "streamType": 3,
        "streamName": "Extension Tunnel/Circuit",
        "streamState": 0
      },
      {
        "streamType": 4,
        "streamName": "Switch",
        "streamState": 1
      },
      {
        "streamType": 5,
        "streamName": "Flow",
        "streamState": 0
      }
    ]
  }
]