As a transitional step, this site will temporarily be made Read-Only from July 8th until the new community launch. During this time, you can still search and read articles and discussions.

While the community is read-only, if you have questions or issues requiring TIBCO review/response, please access the new TIBCO Community and select "Ask A Question."

You will need to register or log in or register to engage in the new community.

Hawk RedTail Container Edition 7.0.1 Kubernetes Deployment - Quickest Deployment with minimal Security

By:
Last updated:
7:54am Sep 02, 2021

Back to HomePage


TIBCO Operational Intelligence Hawk® (Formerly TIBCO Hawk®) and TIBCO Operational Intelligence Hawk® RedTail provide the industry's best, sophisticated tool for monitoring and managing distributed applications and systems throughout the enterprise. With Hawk, system administrators can monitor application parameters, behavior, and loading activities for all nodes in a local or wide-area network and take action when pre-defined conditions occur. In many cases, runtime failures or slowdowns can be repaired automatically within seconds of their discovery, reducing unscheduled outages and slowdowns of critical business systems.


Overview of TIBCO Hawk RedTail Deployment 

TIBCO OI Hawk RedTail - Container Edition helps you to monitor and manage distributed applications by providing distributed rules, alerts, and remediation actions. You can enable additional monitoring capabilities in TIBCO OI Hawk RedTail - Container Edition with the “TIBCO Operational Intelligence Hawk® RedTail - Standard Edition License”, such as centralized storage and processing of metrics, customizable dashboards, Tag-Based Rulebases, Fine-grained Role-Based Access Control, Collect and Forward logs to TIBCO Loglogic LMI etc.

OI Hawk RedTail Container Edition deployment consists of a set of microservices where each component provides internal/ external services and interacts with other components over HTTP/REST/ JDBC.

(Refer to TIBCO® Operational Intelligence Hawk® RedTail -Container Edition 7.0.1 Installation, Configuration, and Administration section for a description of the components.)

The components Hawk Console, Webapp, and QueryNode expose REST APIs over HTTPS and so certificates and keys must be configured for these components including the client certificates and keys for other components which interact with these 3 components.

Note that OI Hawk RedTail consists of multiple components. All these components talk to each other in a secured manner and for most of the components, security is not optional. The security is enforced using server and client certificate validation. The certificates, keys can be configured for each component using environment variables. As a convenience, self-signed certificates and keys are auto-generated at the time of the creation of RedTail Container images and placed within the containers at a certain location. You can use these self-signed certificates and keys or choose to use more secured CA-signed certificates.

Another configuration aspect for OI Hawk RedTail is the storage requirements for some of the components. You will need to create Persistent Volume Claims (PVCs) and mount the volumes for persisting data for components like Prometheus, Grafana, MySQL Database, Hawk Console, etc.

With the above-mentioned considerations, Kubernetes deployment configuration for RedTail components can be complex and requires careful planning and execution.

This document describes Kubernetes deployment of RedTail Container Edition components with the bare minimum secure communication.

TIBCO OI Hawk RedTail Container Edition provides the ‘build-all.sh’ script to build Docker images for each of the components. The script also generates the necessary certificates and keys at the time of building the images. These are self-signed certificates, you can change the certificate validity and password by configuring the details in cert.conf file.

Refer to TIBCO® Operational Intelligence Hawk® RedTail - Container Edition Security Guidelines Certification Management section for more information.

The certificates and keys are placed into each of the container images at the location: /loglogic/conf/certs/

Note:

  1. The steps mentioned in this document are only for reference purposes. Your actual production deployment may vary depending on your corporate security requirements.
  2. The sample YML files attached with this document are for reference only. You may compose your own YML/ Helm charts manifests for your production deployment.

RedTail Kubernetes Deployment with Minimal Security

We will use the default self-signed server and client certificates which are auto-generated and available at each of the container images.

Note that Hawk Console, WebApp, and Query Node offer public-facing REST APIs, so access to these components must be secured. At a minimum, we need to configure these 3 components and other components which interact with these using certs and keys via environment variables in their respective YML files.

Step 1: Download Hawk RedTail and build images

Download TIBCO OI Hawk RedTail Container Edition 7.0.1 binaries (TIB_oihrce_7.0.1.zip). Extract it (refer to it as <OIHRCE_HOME>) to find the following directory structure:

drwx------  12 ntamhankar  staff  384 Jun  5 22:26 .
drwxr-xr-x  10 ntamhankar  staff  320 Jun  6 11:10 ..
drwxr-xr-x@  4 ntamhankar  staff  128 Jan 29 04:33 docker
drwxr-xr-x@  5 ntamhankar  staff  160 Jun  8 16:36 hawkconsolenode
drwxr-xr-x@  8 ntamhankar  staff  256 Jun  8 16:36 hkceagent
drwxr-xr-x@  6 ntamhankar  staff  192 Jun  8 16:33 mysql-connector
drwxr-xr-x@  5 ntamhankar  staff  160 Jun  8 16:35 querynode
drwxr-xr-x@  5 ntamhankar  staff  160 Jun  8 16:36 redtail-grafana-plugin
drwxr-xr-x@  3 ntamhankar  staff   96 Jan 29 04:33 redtail-prometheus-discoveryservice
drwxr-xr-x@  4 ntamhankar  staff  128 Jan 29 04:33 samples
drwxr-xr-x@  9 ntamhankar  staff  288 Jun  8 16:36 webapp
drwxr-xr-x@  5 ntamhankar  staff  160 Jan 29 04:33 zookeeper

 

Go to <OIHRCE_HOME>/mysql-connector and download the mysql connector from the publicly available MySQL portal:

$ wget https://cdn.mysql.com//archives/mysql-connector-java-8.0/mysql-connector-java-8.0.17.tar.gz

Now go to <OIHRCE_HOME>/docker/build-images and run:

$ ./build-all.sh

The script will prompt you if you want to proceed with the default TLS key password. The default password is “changeit”. Type yes and proceed. You can change the default password and other configurations. However, if you change the password, then you need to change it in the sample YML files provided.

You will notice the certificates and keys will be generated and copied into the images as they are being created.

This will create all the images for various RedTail components.

 

$ docker images | grep redtail
redtail/zookeeper                                7.0.1     d8f445572e61   5 hours ago   254MB
redtail/prometheus_discoveryservice              7.0.1     3f331a20aa7e   7 hours ago   18.6MB
redtail/grafana                                  7.0.1     201289c3812b   7 hours ago   159MB
redtail/hkceagent                                7.0.1     b6bf5cf65c1a   7 hours ago   454MB
redtail/hawkconsolenode                          7.0.1     b771f39f86e1   7 hours ago   370MB
redtail/webapp                                   7.0.1     15a86460c839   7 hours ago   239MB
redtail/querynode                                7.0.1     002c10fd574a   7 hours ago   356MB
redtail/base                                     7.0.1     b6c1a8ef3ff8   7 hours ago   267MB
redtail/mysql                                    7.0.1     3717ea8b7df9   7 hours ago   447MB

 

Note:

1. build-all script does not build the Prometheus image.  This is because we refer to the official prom/prometheus:v2.22.0 image, we do not modify this image, however, we configure Prometheus to securely connect to Hawk Console for the targets using the certificates and keys configured using ConfigMap. More about this in Step 3.

2. You may need to tag and push these images to the container registry of your preferred Kubernetes platform. The sample YML files attached here refer to Google Cloud Container Registry, so the image tag needs to be of the format:

gcr.io/<GCP_Project>/<folder>/component_name:7.0.1

E.g.

gcr.io/hawk-engineering/ntamhank/redtail/querynode:7.0.1

Step 2: Identify the certificates and keys

Though all the self-signed server and client certificates and keys are present in every RedTail component image, they need not be explicitly configured as environment variables for each of the components. The only exception is the Prometheus component. The Prometheus image is not customized, rather a standard docker image is used prom/prometheus:v2.22.0.

Since Prometheus needs to connect to the Hawk Console component as a secure target for pulling the metrics, it needs to initialize the target with the Hawk Console server certificate, Prometheus client key, and certificate.

This can be done by configuring the cacert, client certificate, and key as “Secrets” and volume mounting them to the Prometheus container.

In order to configure the secrets, we need to get the actual cacert, Prometheus client certificate, and key and use the Base64 encoded content of these files.

Note that all the certificates and keys are available within <OIHRCE_HOME>.

$ cd <OIHRCE_HOME>/docker/build-images/build-context/loglogic/conf/certs
$ base64 -i cacert -o cacert-base64.txt
$ base64 -i prometheus-client-certificate -o prometheus-client-cert-base64.txt
$ base64 -i prometheus-client-key -o prometheus-client-key-base64.txt

 

These base64 encoded content will be useful in configuring redtail_prometheus.yaml file in the step below.

Step 3: Deploy RedTail components

We are all set to start deploying each of the YAML files one by one. Download all the YAML files attached. You will have to change the image name as per your Kubernetes requirements.

Make sure your kubectl context points to the right cluster you want to deploy RedTail.

1. Deploy Zookeeper component:

$kubectl apply -f redtail_zk.yaml

This will create a couple of services and a stateful set for Zookeeper. This will also create a Persistent Volume Claim and a persistent volume of 50 MB for persisting the zookeeper data directory.

The startup scripts in the container will create necessary nodes on the Zookeeper which is required for RedTail functioning.

Note: Access to zookeeper is unsecured. And you cannot configure secured access to Zookeeper.



2. Deploy MySQL component:

$kubectl apply -f redtail_db.yaml

This will create the necessary tables and populate them.

Note: Access to MySQL is not secured. You can configure TLS for MySQL access and configure client certificates for other components which communicate with MySQL.



3. Deploy Grafana component:

$kubectl apply -f redtail_grafana.yaml

This will deploy the customized grafana component which provides the dashboarding features. Grafana also connects to MySQL DB to store it’s configuration.

Note: By default, Grafana is unsecured. You can protect the access to Grafana by using a side-car reverse proxy such as nginx and configuring secure communication for nginx.

4. Deploy Hawk Console component:

$kubectl apply -f redtail_console.yaml

Hawk Console is the pivotal component of RedTail. It provides Authentication, connectivity to Hawk Agent, RBAC, and other features such as public-facing REST APIs. These REST APIs are secured.

5. Deploy Hawk QueryNode component:

$kubectl apply -f redtail_querynode.yaml

QueryNode component is responsible for providing query capability to Hawk microagent methods as well as Time-series metrics stored in Prometheus. QueryNode provides public-facing REST APIs. These REST APIs are secured.

6. Deploy Hawk Agent component:

$kubectl apply -f redtail_hkceagent.yaml

A single Hawk Agent is required for monitoring all the Applications and infrastructure components in the Kubernetes cluster. Hawk Agent connects to the Hawk Console using Console API via TCP transport and also connects to the microagents (Applications) using the TCP AMI API. Note that both the Console API and AMI (Microagent) API can be configured with secured communication using TLS. The sample YML files do not configure the secured transport communication.

In addition, the Hawk Console also scrapes from various Prometheus targets and facilitates to collects metrics. It also exposes metrics for the Hawk Microagent methods to be collected as time series data.

7. Deploy Prometheus component:

Note: This needs an edit in redtail_prometheus.yaml file. Copy the base64 encoded content for cacert, Prometheus client certificate, and Prometheus client key from Step 2 above.

Replace the values at the three places in the secret section of redtail_prometheus.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: prometheus-hkc-certs
type: Opaque
data:
  hkc-cacert: <base64_encoded_value_of_cacert>
  prom-certificate: <base64_encoded_value_of_prometheus_client_certificate>
  prom-key: <base64_encoded_value_of_prometheus_client_key>

Save redtail_prometheus file.

$kubectl apply -f redtail_prometheus.yaml

The Prometheus server with help of the side-car container for service discovery will start collecting metrics from the targets added by the Hawk Agent.

Note: By default, Prometheus is unsecured. You can protect the access to Prometheus by using a side-car reverse proxy such as nginx and configuring secure communication for nginx.



8. Deploy Webapp component:

$kubectl apply -f redtail_webapp.yaml

Webapp is responsible for providing the UI for the entire RedTail.

9. Finally, create Load Balancers:

Create Load Balancers for Webapp, Hawk Console, and Query node. This is required so that the Ui and REST API Swagger pages can be accessed from outside:

$ kubectl apply -f redtail_webapp_lb.yaml

$ kubectl apply -f redtail_console_lb.yaml

$ kubectl apply -f redtail_querynode_lb.yaml

Step 4: Check the Kubernetes services running and access the UI

$ kubectl get svc
NAME                  TYPE         CLUSTER-IP     EXTERNAL-IP      PORT(S)                 AGE
redtail-agent         ClusterIP    None           <none>           2571/TCP                143m
redtail-console-cmgr     ClusterIP    None           <none>           2561/TCP             143m
redtail-console-connect ClusterIP      None         <none>           9687/TCP              143m
redtail-console-rest    LoadBalancer   10.7.245.61  35.200.234.246   9687:32303/TCP        5h23m
redtail-db              ClusterIP    None         <none>           3306/TCP                143m
redtail-grafana         ClusterIP    10.7.246.253 <none>           3000/TCP                143m
redtail-prometheus      NodePort     10.7.248.0   <none>           9090:30244/TCP          143m
redtail-querynode       LoadBalancer 10.7.250.45  35.244.17.167    9681:31695/TCP          5h23m
redtail-webapp          LoadBalancer 10.7.245.183   34.93.241.12     9680:31773/TCP        5h23m
redtail-zk              ClusterIP    None           <none>           <none>                143m
redtail-zk-client       ClusterIP    10.7.248.200   <none>           9600/TCP              143m

 

Access the RedTail UI:

https://<WebApp_LB_IP>:9680/

(Username/ Password: admin/ admin)

Note:

When you log in for the first time, the UI will show the classic Hawk. Go to Administration → About and change the license to RedTail standard. On login, you will see the full view.

Go to Dashboards → Grafana settings and configure the RedTail plugin with the correct Webapp URL. For e.g. if you are using the sample YAML files, then the same webapp service is “redtail-webapp”. So the RedTail data source URL should be ("https://redtail-webapp:9680"). Save and test the configuration



Access the Swagger page for Hawk Console:

https://<HKConsole_LB_IP>:9687/hawkconsole/v1/docs

Access the Swagger page for Query Node:

https://<QueryNode_LB_IP>:9681/docs

 

Attachments

AttachmentSize
Package icon redtail_7.0.1_yamls.zip17.37 KB