As a transitional step, this site will temporarily be made Read-Only from July 8th until the new community launch. During this time, you can still search and read articles and discussions.

While the community is read-only, if you have questions or issues requiring TIBCO review/response, please access the new TIBCO Community and select "Ask A Question."

You will need to register or log in or register to engage in the new community.

How to Develop and Deploy TIBCO FTL® applications with TIBCO BusinessWorks™ Container Edition on Docker containers

Last updated:
7:34am Apr 14, 2020

This guide details out steps to deploy and TIBCO FTL® with TIBCO BusinessWorks™ Container Edition on docker containers. In order to demonstrate this, we take a simple use case where a file containing employee information is read and the information published on a FTL endpoint.

There are two types of subscribers to these messages and they filter the incoming stream using a application specific content matchers. Since it is possible that the subscribing applications might not be running always we implement a persistent store and durables so that the messages are available across subscriber restarts.

Introduction

We carry out the setup in two phases, viz. development and deployment

Development:

In this phase we setup and test FTL/BWCE on a local machine through the following steps:

  1. install FTL
  2. install BWCE
  3. install BWCE Plugin for FTL
  4. start and configure FTL Realm Server
  5. develop the BWCE application and make a deployable image.

Deployment:

  1. install docker
  2. install FTL
  3. create docker images for the realm server, tibstore and tibagent and start them
  4. configure the FTL Realm Server
  5. deploy BWCE image

Software used for this tutorial:

  • TIBCO FTL® 5.2.1 to 5.3.2
  • TIBCO BusinessWorks™ Container Edition 2.3.2

Setting up the Development Environment

Download and install the BWCE and the FTL.

  • Extract the installer and run the above products by running the TIBCOUniversalInstaller* command form the respective directory where the above products are installed and choose the default installation.
  • setup the following environment variables:
    #======================
    FTL_HOME=<FTL_INSTALLATION_PATH>
    LD_LIBRARY_PATH=$FTL_HOME/lib:$LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH for mac)
    PATH=$FTL_HOME/bin:$PATH
    
    export FTL_HOME LD_LIBRARY_PATH PATH
    #======================
    
  • start BWCE Studio under <TIBCO_HOME>/studio/4.0/eclipse directory using a terminal with the above variables set

  • Install the BWCE FTL plugin.

Download the corresponding eclipse installer (*.p2installer.zip) or the one matching the OS and carry out the installation as per the installation guide. For Example for the ecplise plugin choose Help Menu > Install New Software> click Add…

On the new Dialog, click the “Archive” Button and select the downloaded .p2Installer file.

Clicking next a new times will get you to a dialog where you could select the FTL installation location of your choice.

 

Setup the FTL Realm

It is best that you decide upon the basic messaging infrastructure before you embark upon application development as you could pick and choose realm objects from BW/BWCE studio once the realm connectivity has been established.

Start the Realm Server

We start by defining a basic authentication file as follows, which is in the form of user: password, role1,role2,..

% cat auth.list 
admin: admin123, ftl-admin,ftl
rs: rs123, ftl-primary,ftl-satellite,ftl-backup,ftl-admin,ftl,auth
PUB: PUB123, ftl
SUB1: SUB1123, ftl
SUB2: SUB2123, ftl
S1: S1123, ftl
S2: S2123, ftl
S3: S3123, ftl

Start up the realm server: From <Tibco home>/ftl/5.2/bin start realm server

./tibrealmserver -ht 0.0.0.0:8080 --auth.url file://auth.list

Where, auth.list is the file with usernames and passwords in cleartext (you can set up External JAAS authentication as well).

Configure basic messaging infrastructure

Once the realm server is up running, create applications and endpoints used by the applications, by clicking on the (+) sign on the applications page

NOTE: please remember to change the Edit mode to “On”.

Realm Configuration - Applications

Next, create a transport by clicking on the transports link and (+) sign on the transports link. Since we are developing for cloud environment, you should use Dynamic TCP transport, and for simplicity, we will use the “mesh” type. This step is optional as you could use the default transport itself to link your endpoints.

Realm Configuration - Transports

We now can go back to the application page and specify the new transport for our endpoints.

Realm Configuration - Application Endpoints

Configure Message Formats

FTL allows us to predefine message formats that the applications intend to exchange. This also helps BW/BWCE to use the message format schema while defining input or output of the FTL subscriber or publisher respectively.

Realm Configuration - Message Format

create the message format by clicking the (+) sign on the Formats page and add fields to the format by clicking (…) under the format column and selecting add field.

We now go back to the Applications page and for both the applications we created, select (…) and view details. On the details page associate the predefined format with the application and click the save button.

Realm Configuration - Application's Message Format

Adding Persistence

To add persistence to our application we define a Persistent cluster from the “Clusters” link, and add total of 3 servers to achieve a quorum for added assurance.

NOTE: This entire section could be skipped for now and resumed after the development task from BWCE is over.

Realm Configuration - Persistent Cluster

Next we define the durable stores that will use this cluster for persistence. Create a store by clicking on the (+) sign, and add two durables - one for each consumer by clicking (…) icon under the Store Name column.

Realm Configuration - Persistent Cluster

We set the publishing Mode to Store —> Confirm —> Send so that the persistent store guarantees that the message is safely stored if the publish call succeeds.

Set the Durable Type to Static/Standard (Choosing Static/Shared will load balance the incoming messages across all matching subscribers.)

Set the Match field, type, value as above similar to the durable subscribers that we will be developing in the next section.

At this stage we see that the status of the cluster and stores is “offline”

Realm Configuration - Store and Durables

Realm Configuration - Mapping - Application and Stores

To get the cluster and the store up and running we need to start the tibstore instances S1,S2 and S3 from the command-line in three separate windows.

./tibstore -pw ./S1/pass   -n S1 -rs  http://localhost:8080
./tibstore -pw ./S2/pass   -n S2 -rs  http://localhost:8080
./tibstore -pw ./S3/pass   -n S3 -rs  http://localhost:8080

where the S?/pass file contains the user and the password (either plaintext or in a mangled form (use <ftl_home>/bin/tibrealmadmin —mangle)) matching the ones in the auth.list file.

e.g:

% cat S1/pass 
S1
_mng_BmRkZ2Y=

You should now see that the cluster and stores should be online.

Developing the BWCE FTL Application

Set BWCE container mode to docker:

set the “ContainerTarget” variable to Docker as shown below in the <TIBCO_HOME>/studio/4.0/eclipse/configuration/config.ini file

#######################
osgi.bundles.defaultStartLevel=4
eclipse.p2.data.area=file\:/home/sp/tibbwce/eclipse-platform/bundlepool/1.0/
eclipse.application=org.eclipse.ui.ide.workbench
ContainerTarget=Docker
#######################

Alternatively, from the BWCE>Preferences… menu, open the preferences dialog, and search for “Container”. Click on Container Platform, and select platform as ‘Docker’ form the BWCE Studio.

For this tutorial, we use a simple FTL publisher and two FTL Subscribers, however to be able to deploy them in three separate containers, we create three separate applications

We start by creating a BWCW Studio Project by selecting new > Project > BusinessWorks >

BusinessWorks Application Module in the BWCE studio.

We then add two more application modules to have three separate applications in the BWCE project

BWCE Project Setup1 BWCE Project Setup2

We then create a FTLRealmServerConnection resources as above for all process such that there are two subscriber resources and one publisher resource. The difference being that the FTL application name of the publisher resource is set to “tradepublisher” whereas that of the subscriber resources, be set to “tradesubscriber” matching the applications we created in realm server. The recommended method is to supply a Realm URL first as a module property and set the username and password if any. During development, you could use the admin username and password so that BW is able to fetch the configuration from the realm server.

BWCE RealmServer Shared Resource Configuration

You could preset the following module properties for all three applications, so that we could use a common setting while deploying these application as containers.

BWCE Module Properties

With the module properties and FTL Connection resources completed for all three projects we set up the FTLpublishers and subscribers to use the correct resources created in the realm. We can simply browse or select from the drop down for most of the fields other than content matchers

Publisher:

BWCE Publisher Process

  • Here we look for the correct realm server connection resource. There should be just one for each application
  • Browse for the correct endpoint, again there should be just one for a given application
  • Set the format to “Predefined” and browse for the correct format name - there should be just one format in this case.

Subscribers:

BWCE Subscriber Process

  • Here too we look for the correct realm server connection resource. There should be just one for each application.
  • Browse for the correct endpoint, again there should be just one for a given application- both subscribers share the same endpoint.
  • Set the format to “Predefined” and browse for the correct format name - there should be just one format in this case.
  • Content matchers are in the format -> {“name”:value}. We set different content matchers for each subscriber, where we use Buy=1 for BuyOrderSubcriber and Buy=0 for SellOrderSubscriber. NOTE: The Name of the field needs to be surrounded with double quotes “”, since the Buy field is defined as long we don’t surround it with quotes in this case. If the field type was string, we would have to define the format as {“name”:“value”}.
  • Finally in the advanced tab for the FTL Subscribers, add the module property for setting the durable name. For SellTradeSubscriber set the value to the FTL_SELL_DURABLENAME, where as for the BuyTradeSubscriber set it to FTL_BUY_DURABLENAME.

BWCE Subscriber Process - Durable configuration

Generate traffic and get ready for deployment:

We map the timer’s second value to create a finite set of message fields at the publisher

BWCE Publisher Process - Generate randomised traffic

When we run all apps in the BWCE Debugger you should be able to see the logs similar to this:

BWCE Debugger output

The module properties we exported earlier, are not directly accessible from the BWCE’s runtime when you actually deploy the application into a docker container. You need to create an additional profile on the Property Configurations page at <applicationname>.application -> Package Unit -> Properties link. Click on the “New Profile” button and name the profile “Docker”. You will see a column created by that name. Click twice on the field adjacent to the property you want to set and it will become editable. When you click the icon set Container Propertiesto the right of the editable field, you the correct value of the property for Docker profile would be automatically filled. So if the module property is FTLREALMSERVERURL the Docker profile equivalent would be %FTLREALMSERVERURL%

BWCE - Container Properties

When you deploy this application in the docker container, you can supply the value for the FTLREALMSERVERURL through an environment variable inside the container with the same name. You could have multiple Property profiles, however which one you use at the application’s container is governed by the environment variable BW_PROFILE, in our case we need to set it to BW_PROFILE=“Docker”. The value is case sensitive and needs to match the name of the property profile you created earlier.

Finally create deployable artefacts by clicking on the “Export Application for Deployment” link on the application—>Package Unit —> Overview Page and specify the location on the next dialog.

BWCE -Export Application for Deployment

Preparing Docker environment for Deploying the application and FTL components

Install Docker on the host machines where you intend to deploy the applications as well as your client machine e.g. your development machine.

Preparing docker images

On your development machine run the carry out the following steps to create docker images:

FTL images:

Download and extract (run the Linux installer for FTL 5.2) in a temp directory

set environment variable FTL_EXTRACT to the path where you extracted FTL.

Download FTL 5.2 HF1 and extract it to another temp location and move the deb/tibco-ftl-5.2.0-Linux-servers.deb to the ftl directory created as shown below:

% mkdir /tmp/build
% cd /tmp/build
% mkdir ftl

Now, copy the FTL debian installation files to the ftl directory as well.

% cp ${FTL_EXTRACT}/deb/*.deb ftl/

Now copy the samples from the FTL 5.2 installation on the development machine to the ftl directory as well.

% cp -r ${FTL_INSTALL}/samples ftl/

We need to modify the Docker files for containers for FTL components to add the hot fix as follows.

open each Dockerfile.* from ftl/samples/docker/ and modify the line starting with “RUN dpkg -i …” and append “ /tmp/*hotfix*.deb” to this line. This will install the hot fix after installing the FTL Server package for the container in question.

Since we intend to start the realm server with authentication we need to change Dockerfile.tibrealmserver file and modify Entrypoint comand to look like-

ENTRYPOINT ["tibrealmserver", "--http", "*:13131", "--ftl", "*:13134", "--gui", "*:13136", "--auth.url", "file://auth.list"]

At this stage you should have the environment ready to build the FTL images, to build run the following commands.

% docker build -f ftl/samples/docker/Dockerfile.tibrealmserver -t ftl-tibrealmserver .
% docker build -f ftl/samples/docker/Dockerfile.monitor -t ftl-monitor .
% docker build -f ftl/samples/docker/Dockerfile.tibstore -t ftl-tibstore .
% docker build -f ftl/samples/docker/Dockerfile.tibagent -t ftl-tibagent .

You should now see the following images:

$ docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
ftl-realmserver        latest              fbff762f022e        2 hours ago        694 MB
ftl-monitor            latest              9cb6e75217fa        2 hours ago        566 MB
ftl-tibstore           latest              3dbe22fd12ad        2 hours ago        363 MB
ftl-tibagent           latest              10f652d9115b        2 hours ago        363 MB
ftl-tibrealmserver     latest              3d71f8332ce5        2 hours ago        694 MB
ubuntu                 16.04               0ef2e08ed3fa        2 hours ago        130 MB

BW Application images:

Now that we have the FTL setup ready, we need to get the BWCE images ready and to achieve this we carry out the following steps:

Create a BWCE base image with FTL libraries

copy contents of <BWCE_HOME>/docker/ directory to /tmp/build (where we built the ftl images from). Also create a directory named “bw” and move the bwce base image docker /tmp/build/Dockerfile file to “bw” directory.

cd /tmp/build; mkdir bw; mv Dockerfile bw/Dockerfile.bwce

cd to the build directory and copy bwce-runtime-2.3.2.zip file (downloaded from edelivery.tibco.com) to the resources/bwce-runtime directory under the build directory (/tmp/build).

ls <FTL_HOME>/lib/*.so* | zip -j <BUILD_HOME>/resources/addons/lib/ftl_linux_client_libs -@

Next we provision the FTL jars and libs from the  OSGi Component from FTL to BWCE images adding directory.

#extract the OSGi Component files
cd ${FTL_EXTRACT}/tar/
tar zxf tibco-ftl-<FTL Version>-Linux-osgi_bundle.tar.gz

ls opt/tibco/ftl/<FTL_VERSION>/components/shared/1.0.0/plugins/com.tibco.ftl*/lib*.so* | zip -j <BUILD_HOME>/resources/addons/lib/ftl_linux_client_libs -@

cp opt/tibco/ftl/<FTL_VERSION>/components/shared/1.0.0/plugins <BUILD_HOME>/resources/addons/jars

Note: Currently BWCE supports 5.2.x, which is quite old. To run the BWCE applications with FTL 5.2.x, you need to provision the OSGi component from the appropriate FTL version.

Next, download TIB_bwpluginftl_*_bwce-runtime.zip from edelivery and copy it to <BUILD_HOME>/resources/addons/plugins directory and build the BWCE/FTL Base image

To build the bwce/ftl base image run the following command from the build directory:

docker build -f bw/Dockerfile.bwce -t tibco/bwce-ftl .

You should see something like:

$ docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
tibco/bwce-ftl         latest              68e59e1c4f32        2 minutes ago      371 MB
debian                 jessie              3e83c23dba6a        2 minutes ago      124 MB
ftl-realmserver        latest              fbff762f022e        6 hours ago        694 MB
ftl-monitor            latest              9cb6e75217fa        6 hours ago        566 MB
ftl-tibstore           latest              3dbe22fd12ad        6 hours ago        363 MB
ftl-tibagent           latest              10f652d9115b        6 hours ago        363 MB
ftl-tibrealmserver     latest              3d71f8332ce5        6 hours ago        694 MB
ubuntu                 16.04               0ef2e08ed3fa        6 hours ago        130 MB

Check to see if the image has the file bwce-runtime-2.3.2.zip in the correct location.

docker run --rm -it --name bwcebase --entrypoint=/bin/bash tibco/bwce-ftl
root@eda75beadd34:/# ls -la /resources/bwce-runtime/
drwxr-xr-x 2 root root      4096 Jan 23 22:28 .
drwxr-xr-x 4 root root      4096 Jan 22 19:29 ..
-rw-r--r-- 1 root root 200047776 Jan 23 22:28 bwce-runtime-2.3.2.zip

Add application archives from our BW CE application - one for each container type.

Create a DockerFile for each application image we have with the following contents in the <BUILD_HOME>/bw directory:

Note: The ENV directive in the BWCE applications dockerfile does two things:

FROM tibco/bwce-ftl:latest
# ADD <Application Archive (EAR)> /
ADD TradeStockBuyOrderSubscriber.application_1.0.0.ear / 
ENV BW_PROFILE="Docker"
ENV FTL_REALMSERVER_URL="discover://"
ENV FTL_PUBLISHER_ENDPOINT="tradepublisher-ep"
ENV FTL_SUBSCRIBER_ENDPOINT="tradesubscriber-ep"
ENV FTL_BUYDURABLE_NAME="BuyDurable"
ENV FTL_SELLDURABLE_NAME="SellDurable"
# Pass relative credentials to the container at build time, or set them at run time later on
ENV FTL_BUYAPP_USER=S1
ENV FTL_BUYAPP_PASS=S1123

And run the command below from the build directory to create separate images for each application to be run inside a container:

docker build -f bw/Dockerfile-publisher -t tibco/bwceftlpub .
docker build -f bw/Dockerfile-BuySubscriber -t tibco/bwceftlbuysub .
docker build -f bw/Dockerfile-SellSubscriber -t tibco/bwceftlsellsub .

You should now see the following images in addition to the FTL ones:

REPOSITORY             TAG                 IMAGE ID            CREATED            SIZE
tibco/bwceftlsellsub   latest              8c6e6e174830        4 seconds ago      389 MB
tibco/bwceftlbuysub    latest              17f9d48d092c        15 seconds ago     389 MB
tibco/bwceftlpub       latest              608b86011982        33 seconds ago     389 MB
tibco/bwce-ftl         latest              624f8ffeacbe        9 minutes ago      389 MB

NOTE: You can run the following command to see if everything is in order

docker run --rm -it --name bwceftlsellsub  --entrypoint=/bin/bash tibco/bwceftlsellsub

You could check if the environment variables you specified are correct.

Deployment Strategy

run the application manually by running “/scripts/start.sh”Or run the application container in detached mode as:

docker run --rm -d -p 13131:13131 -v `pwd`/auth.list:/data/auth.list -v `pwd`/:/data/ -p 13134:13134 -p 13136:13136 --name ftl-tibrealmserver ftl-tibrealmserver
docker run --rm -d -v `pwd`/S1:/data/ --name ftl-tibstore-S1 -v `pwd`/S1/pass:/data/pass ftl-tibstore -pf ./s1/pass -rs  discover:// -d data/ -n S1 
docker run --rm -d -v `pwd`/S2:/data/ --name ftl-tibstore-S2 -v `pwd`/S2/pass:/data/pass ftl-tibstore -pf ./s2/pass -rs  discover:// -d data/ -n S2 
docker run --rm -d -v `pwd`/S3:/data/ --name ftl-tibstore-S3 -v `pwd`/S3/pass:/data/pass ftl-tibstore -pf ./s3/pass -rs  discover:// -d data/ -n S3 
docker run -d --restart=unless-stopped -v /var/run/docker.sock:/var/run/docker.sock -v `pwd`/A1/pass:/root/pass  -p 13139:13139 -p 13140:13140 --name ftl-tibagent ftl-tibagent -rs localhost:13131
docker run --rm -it --name bwceftlbuysub tibco/bwceftlbuysub
docker run --rm -it --name bwceftlsellsub tibco/bwceftlsellsub
docker run --rm -it --name bwceftlpub tibco/bwceftlpub

Note: You would want to run the realm server with the same configuration you used in dev earlier, to do this there are a few options:

copy ‘rs_data.db’ file from the realmserver’s current directory in dev to the build directory so that the -v `pwd`/:/data/ option will map this file and make it available to the /data directory where the realm server would be running.

Alternatively, you could save the JSON config from the dev environment by clicking Actions menu on the right hand corner from the realm server GUI. You can either upload this JSON config to the new realm server from its GUI or use tibrealmadmin -ur <JSON file> to upload the config.

Realm Configuration - upload/download realm configuration

Alternate Deployment (docker-compose)

create a yaml file (refer to the attached file Docker-Compose.yml).

we also need to create a .env file as follows:

% cat .env
PRIMARY_RS_HOST=<Realm server IP>
SECONDARY_RS_HOST=<secondary Realm sever - Optional >
RS_HTTP_PORT=13131
FTL_APP_NAME="bwce-docker"
FTL_APP_INSTANCEID="tcp"
FTL_APP_ENDPOINT="docker-ep"
BW_PROFILE= "Docker"
FTL_REALM_SERVER_URL="discover://"
FTL_REALMSERVER_URL="discover://"
FTL_PUBLISHER_ENDPOINT="tradepublisher-ep"
FTL_SUBSCRIBER_ENDPOINT="tradesubscriber-ep"
FTL_BUYDURABLE_NAME="BuyDurable"
FTL_SELLDURABLE_NAME=“SellDurable"

You can simply deploy as :

docker-compose -f docker-compose.yml up -d

once all services have started you should see the following logs in the subscribers:

docker-compose -f docker-compose.yml logs

docker Logs showing the subscribers receiving data

Once you are done with the tests you can bring the entire setup down by running:

docker-compose -f docker-compose.yml down

Deploying to a realistic setup:

The deployment model we use is as follows: We install the various components on three Ubuntu machines as follows.

Realistic Deployment

We start by creating deployable tar archives for each of the FTL/BWCE components shown above and transferring them to their respective locations.

docker save -o tibco-ftlserver.tar ftl-realmserver
docker save -o tibco-ftlagent.tar ftl-tibagent
docker save -o tibco-ftlstore.tar ftl-tibstore
docker save -o tibco-bwceftlbuysub.tar tibco/bwceftlbuysub
docker save -o tibco-bwceftlsellsub.tar tibco/bwceftlsellsub
docker save -o tibco-bwceftlpub.tar tibco/bwceftlpub

You can load these tar files as docker images on the target compter after transferring the archives there, using the command:

docker load -i <tar file>

Split the docker compose file into 3 individual docker-compose files such that only services that meant to be run on that machine are in the file. e.g. tibagent should exist on all hosts.

Also copy .env file and modify the PRIMARY_RS_HOST and SECONDARY_RS_HOST(optional) to match this environment.

to start the services, issue the following command:

docker-compose -f docker-compose.yml up -d —-no-build

Attachments

Feedback (2)

hi,

Most likely the bw ftl plugin was not installed correctly. please review the following step.

================

download TIB_bwpluginftl_*_bwce-runtime.zip from edelivery and copy it to <BUILD_HOME>/resources/addons/plugins directory and build the BWCE/FTL Base image

================

Alternatively, 

change the entrypoint of the application container

docker run --rm -it --name bwcebase --entrypoint=/bin/bash tibco/bwce-ftl

and try debugging your setup:

e.g. you should see something like:

ls -al resources/addons/plugins/
total 168
drwxr-xr-x 2 root root   4096 Jun  1 15:13 .
drwxr-xr-x 8 root root   4096 Jun  1 13:14 ..
-rw-r--r-- 1 root root 162808 Jun  1 15:10 TIB_bwpluginftl_6.4.1_v2.1_bwce-runtime.zip

or by manually running scripts/start.h in debug

 

 

 

Saurabh Prabhu 12:18pm Jun. 18, 2018

bkallipa,

Please ask this as a question.  You will more likely get an answer.

TIBCO Community Wiki page Feedback Policy.

Don Johnson 11:28am Jun. 18, 2018