Delphix Developer Edition, Bonita and Docker

Delphix Developer Edition, Bonita and Docker

Introduction 

What if you want to spin up quickly new Bonita BPM application? Delphix DE and Docker are great choice to do this!

Delphix Developer Edition is a part of Lanshark demo environment or can be requested seperately.

More information here:

• https://community.delphix.com/delphix/topics/what-is-landshark-de-lsde 
• http://datavirtualizer.com/delphix-express-free-version-of-delphix-available/
• http://www.delphix.com/free-trial-request
• https://docs.delphix.com/display/LS22/Landshark+2.2+Documentation 

Delphix Express GUI

Delphix Express vs Delphix Enterprise

Landshark environment together with Delphix Express engine contains two VMs. Linuxsource VM and Linuxtarget VM have preinstalled Postgresql, Oracle XE and more...

Wikipedia: https://en.wikipedia.org/wiki/Bonita_BPM

"Bonita BPM is an open-source business process management and workflow suite created in 2001. It was started in France National Institute for Research in Computer Science, and then had incubated
years inside of the French computer science company Groupe Bull. Since 2009, the development of Bonita is supported by a company dedicated to this activity: Bonitasoft. ..."

Bonita BPM

Show Time

We are using:

• Delphix Express 4.3.1.0
• Landshark source VM (linuxsource)
• Landshark target VM (linuxtarget)
• Official Docker Bonita image (https://hub.docker.com/_/bonita/)  

Steps

1. Register Postgresql Data Source (dSource) - 1 time action
2. Provision Postgresql cloned database
3. Spin up Bonita docker image

Let's do it!

1. Data Source

Menu Environments

Open "Environments"
Choose "Databases"
Scroll down to postgresql part.
Click "Add dSource"

After clicking "Add dSource"

Enter user "postgres" password "delphix". Next.
Change name to "usda".
Next.Next.Next.Finish.

Click on Delphix logo. Left upper corner.

dSource "usda"

2. VDB (Virtual DB)

Click on "usda".

"Provision" in right lower corner.

Click on "Provision".

Provisioning. First Tab.

Choose LINUXTARGET. Enter port number 5433. Click advanced.

Enter new rule in pg_hba tab.

Enter rule "host   all   all 0.0.0.0/0   trust" in pg_hba tab. Next. Next. Next. Finish.

VDB Ready!

3. Docker part.

• Login to linuxsource host. User root password delphix.
• Enter following commands staying in /root

chkconfig docker on
service docker start

export PGUSER=postgres
export PGPASSWORD=delphix
export PGHOST=`hostname --ip-address`
export PGDATABASE=usda
export PGPORT=5433
psql -U $PGUSER  -h $PGHOST -d $PGDATABASE --command "DROP DATABASE bonita;"
psql -U $PGUSER  -h $PGHOST -d $PGDATABASE --command "DROP USER bonita;"
psql -U $PGUSER  -h $PGHOST -d $PGDATABASE --command "DROP DATABASE businessdb;"
psql -U $PGUSER  -h $PGHOST -d $PGDATABASE --command "DROP USER businessuser;"
psql -U $PGUSER  -h $PGHOST -d $PGDATABASE --command "CREATE USER bonita WITH SUPERUSER PASSWORD 'bpm';"
psql -U $PGUSER  -h $PGHOST -d $PGDATABASE --command "CREATE DATABASE bonita OWNER bonita;"

rm -rf bonita
mkdir -p bonita

docker run -e DB_VENDOR=postgres -e DB_HOST=$PGHOST -e DB_NAME=bonita -e DB_USER=bonita -e DB_PASS=bpm -e DB_ADMIN_PASS=delphix -e DB_PORT=5433 -e TENANT_LOGIN=install -e TENANT_PASSWORD=install --name bonita_custom -v /root/bonita:/opt/bonita -d -p 8080:8080 bonita

•  Open url "http://linuxtarget:8080/bonita" in your browser.

Bonita BPM using VDB ready to go!

Conlusion

• You can use Delphix VDBs together with Docker containerized application without much effort. It's very easy.

• You can deliver data to containerized applications in agile way. No expensive data duplication. No storage waste. Fast!

 

MemSQL & Delphix - Win Win Option

MemSQL & Delphix Express - Win Win Option

Actors

• Delphix Express 4.3.2.1

 

Delphix: "Delphix Positioned as a Leader in the Gartner Magic Quadrant for Structured Data Archiving and Application Retirement" 

Source: twitter.com/delphix

Source: http://www.delphix.com/products/daas-overview/

About Delphix: http://kolibero.eu/delphix
 

• MemSQL 4.1.9 Community Edition

According to Wikipedia (https://en.wikipedia.org/wiki/MemSQL)

"MemSQL is a distributed, in-memory database that is part of the NewSQL movement.[2] It is an ACID-compliant RDBMS that most notably converts SQL into C++ through code generation.[3] It is being developed by MemSQL Inc., that was founded in 2011 and is a graduate of the Y Combinator startup program. MemSQL Inc. has raised more than $40 million to date from a variety of investors including First Round Capital, IA Ventures, NEA, and several prominent angels including Paul Buchheit, Max Levchin, Aaron Levie, and Ashton Kutcher.[4] MemSQL Inc. launched its database to the public on June 18, 2012."

MemSQL & Gartner:
"Gartner Positions MemSQL in the Visionaries Quadrant of the Magic Quadrant for Operational Database Management Systems
MemSQL scores high for its completeness of vision, technology innovation and ability to execute against its R&D model"

http://www.memsql.com/releases/gartner-magic-quadrant-for-operational-database-management-systems/

Source: memsql.com Home Page

Plan

I'm trying to test 2 things

1. Is it possible to use Delphix vFiles functionality to handle persistence in MemSQL?
2. What is its the impact on MemSQL performance?

Test Environment

• Delphix Express VM

Delphix Express in VMWare Player 7

Delphix Express Console

• Centos VM for MemSQL

MemSQL Linux VM

MemSQL Linux VM Box Parameters

Test Steps Breakdown

1. Install MemSQL
2. Create a table and fill with some data
3. Test performance (using MemSQL Ops)
4. Shutdown MemSQL
5. Replace Data directory with resource provisioned with Delphix
6. Copy existing data to new provisioned resource
7. Startup MemSQL
8. Check if everything works
9. Check performance again (using MemSQL Ops)
10. Conclusions

Test Steps

1. Install MemSQL

[memsql@volt1 ~]$ tar -xvzf memsql-ops-4.1.9.tar.gz
[memsql@volt1 ~]$ cd memsql-ops-4.1.9
[memsql@volt1 memsql-ops-4.1.9]$ ls
install.sh  memsql-ops  README
[memsql@volt1 memsql-ops-4.1.9]$ sudo install.sh

MemSQL Ops

2. New Table and Some Data

[memsql@volt1 memsql]$ mysql -u root -h volt1 -P 3306 --prompt="memsql> "
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.5.8 MemSQL source distribution (compatible; MySQL Enterprise & MySQL Commercial)

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

memsql> -- create a new user
memsql> grant all on *.* to 'myuser'@'%' identified by 'mypassword';
Query OK, 0 rows affected (0.38 sec)

memsql> -- show all memsql leaf nodes
memsql> show leaves;
+-----------------+------+--------------------+-----------+-----------+--------+--------------------+------------------------------+
| Host            | Port | Availability_Group | Pair_Host | Pair_Port | State  | Opened_Connections | Average_Roundtrip_Latency_ms |
+-----------------+------+--------------------+-----------+-----------+--------+--------------------+------------------------------+
| 192.168.111.141 | 3307 |                  1 | NULL      |      NULL | online |                  1 |                        0.255 |
+-----------------+------+--------------------+-----------+-----------+--------+--------------------+------------------------------+
1 row in set (0.00 sec)

memsql> -- create a new database
memsql> create database mymemsqldb;
Query OK, 1 row affected (3.25 sec)

memsql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| memsql             |
| mymemsqldb         |
| sharding           |
+--------------------+
4 rows in set (0.01 sec)

memsql> use mymemsqldb;
Database changed
memsql> -- create a new table
memsql> create table mytable (id INT PRIMARY KEY, name varchar(10));
Query OK, 0 rows affected (10.03 sec)

memsql> show tables;
+----------------------+
| Tables_in_mymemsqldb |
+----------------------+
| mytable              |
+----------------------+
1 row in set (0.00 sec)

memsql> describe mytable;
+-------+-------------+------+------+---------+-------+
| Field | Type        | Null | Key  | Default | Extra |
+-------+-------------+------+------+---------+-------+
| id    | int(11)     | NO   | PRI  | NULL    |       |
| name  | varchar(10) | YES  |      | NULL    |       |
+-------+-------------+------+------+---------+-------+
2 rows in set (0.01 sec)

memsql> -- perform data query operations
memsql> insert into mytable values (1, 'hello');
Query OK, 1 row affected (1.66 sec)

memsql> select * from mytable;
+----+-------+
| id | name  |
+----+-------+
|  1 | hello |
+----+-------+
1 row in set (3.10 sec)

3. Test performance (using MemSQL Ops)

 

Reference Performance Score

 

4. Shutdown MemSQL

 [memsql@volt1 memsql]$ memsql-ops memsql-stop
 Index  ID       Agent Id  Process State  Cluster State  Role    Host             Port
 1      DB6DBFC  Ad92cbe   RUNNING        CONNECTED      MASTER  192.168.111.141  3306
 2      F0031A1  Ad92cbe   RUNNING        CONNECTED      LEAF    192.168.111.141  3307
Select a MemSQL node: 2
2015-11-05 19:24:33: J60c2bf [INFO] Stopping MemSQL node F0031A1E49417690DC51BD4F718863DCD901DE6B on Agent Ad92cbe2bc08845b8b5958b69673e26ec
2015-11-05 19:24:38: J60c2bf [INFO] Successfully stopped MemSQL node F0031A1E49417690DC51BD4F718863DCD901DE6B
[memsql@volt1 memsql]$ memsql-ops memsql-stop
 Index  ID       Agent Id  Process State  Cluster State  Role    Host             Port
 1      DB6DBFC  Ad92cbe   RUNNING        CONNECTED      MASTER  192.168.111.141  3306
 2      F0031A1  Ad92cbe   NOT RUNNING    OFFLINE        LEAF    192.168.111.141  3307
Select a MemSQL node: 1
2015-11-05 19:24:44: J476b0d [INFO] Stopping MemSQL node DB6DBFCC07F087C826D73597A7F2488A5E7F51DD on Agent Ad92cbe2bc08845b8b5958b69673e26ec
2015-11-05 19:24:47: J476b0d [INFO] Successfully stopped MemSQL node DB6DBFCC07F087C826D73597A7F2488A5E7F51DD

 5. Replace Data directory with resource provisioned with Delphix

[memsql@volt1 leaf-3307-1]$ pwd
/var/lib/memsql/leaf-3307-1

We have to find data folder and replace it with Delphix provisioned folder

[memsql@volt1 data]$ pwd
/var/lib/memsql/leaf-3307-1/data

Data folder renamed.

Data folder of the MemSQL leaf renamed and new empty folder created.

Environment Registered in Delphix

We have to register host with MemSQL in Delphix.

Provisioning Delphix Storage to MemSQL Host

Next we have to provision Delphix storage to volt1 host and mount it in new empty data folder.

Provisioning to Path /var/lib/memsql/leaf-3307-1/data

Storage Provisioned to /var/lib/memsql/leaf-3307-1/data

We can check this using mount command.

Storage mounted from Delphix platform

6. Copy existing data to new provisioned resource

Now we are copying contents of old __data__ folder to provisioned one.

Copying Data to Delphix Storage

Now snapshot of data in Delphix.

Snapshot done

Now time for starting MemSQL nodes.

7. Startup MemSQL

Nodes started.

MemSQL Ops shows running nodes

Success! Nodes are up and running. Now SQL check.

8. Check if everything works

Database is running fine. Select statement returned correct response.

9. Check performance again (using MemSQL Ops)

Performance after switching to Delphix storage

 10. Conclusions

• It is possible to use Delphix storage as persistence store for MemSQL nodes.
• Performance is not affected. 
• Technically using Delphix provisioned storage has to be orchestrated using some automation tool. We have to replace storage for every leaf node.

Summary

Possibility of symbiosis between MemSQL and Delphix is really interesting.

• We get live backup of MemSQL.
• We get amazing refresh/rewind/clone features of Delphix.
• We optimize storage required for persistence purposes.
• What more?

Great Tandem: VoltDB & Delphix

Great Tandem: VoltDB & Delphix

VoltDB is gaining its market share as a great real time analytics database. The buzz word is "Smart Data Fast". Gartner Magic Quadrant for Operational Databases makes it clear.

What if we merge VoltDB real time capabilities with power of Data as a Service? Below I'm testing if it is possible to make VoltDB's persistent store reside within Delphix vFiles.

Moving Parts

• Delphix Engine: I am using Delphix Express 4.2.3.1

 

• Test Host: Linux CentOS 6.6

• VoltDB Trial 5.6 Enterprise Edition (https://voltdb.com/products/editions)

Test Steps

• Prepare volt user

[root@volt1 home]# groupadd volt

[root@volt1 home]# useradd -m -g volt volt

[root@volt1 home]# passwd volt

• Add entry with volt to /etc/sudoers

Defaults:volt !requiretty

volt ALL=NOPASSWD: /bin/mkdir, /bin/rmdir, /bin/ps, /bin/mount, /bin/umount, /sbin/mount, /sbin/umount

• Unpack VoltDB binaries

[volt@volt1 ~]$ tar -xvzf LINUX-voltdb-ent-5.6.tar.gz

• Turn off Transparent Huge Pages

[root@volt1 ~]# echo never >/sys/kernel/mm/transparent_hugepage/enabled

[root@volt1 ~]# echo never >/sys/kernel/mm/transparent_hugepage/defrag

• Create vFiles for persistence space

[volt@volt1 ~]$ mkdir voltdbroot2

[volt@volt1 ~]$ cd voltdbroot2/

[volt@volt1 voltdbroot2]$ pwd

/home/volt/voltdbroot2

 

• Prepare config file voltdbconfig_2.xml

<?xml version="1.0"?>

<deployment>

  <cluster hostcount="1"

           sitesperhost="1"

           kfactor="0"

  />

  <paths>

     <voltdbroot path="/home/volt/voltdbroot2" />

  </paths>

</deployment>

• Create VoltDB instance using provisioned Delphix storage

[volt@volt1 ~]$ voltdb create --deployment=voltdbconfig_2.xml
Initializing VoltDB...

 _    __      ____  ____  ____
| |  / /___  / / /_/ __ \/ __ )
| | / / __ \/ / __/ / / / __  |
| |/ / /_/ / / /_/ /_/ / /_/ /
|___/\____/_/\__/_____/_____/

--------------------------------

Build: 5.6 voltdb-5.6-0-g3743684-local Enterprise Edition
Connecting to VoltDB cluster as the leader...
Host id of this node is: 0
Starting VoltDB with trial license. License expires on Oct 26, 2015.
Initializing the database and command logs. This may take a moment...
WARN: This is not a highly available cluster. K-Safety is set to 0.
Server completed initialization.

• Check Delphix storage contents

[volt@volt1 voltdbroot2]$ ls

catalog-report.html  command_log  command_log_snapshot  config_log  dr_overflow  export_overflow  snapshots

[volt@volt1 voltdbroot2]$ pwd

/home/volt/voltdbroot2

•  Create snapshot of "voltdbroot2" object in Delphix. We have ingested and fixed current database into Delphix.

• Let us play a little bit with VoltDB instance

[volt@volt1 voltdbroot2]$ sqlcmd

SQL Command :: localhost:21212

1> CREATE TABLE towns (

2>     town VARCHAR(64),

3>     county VARCHAR(64),

4>     state VARCHAR(2)

5>  );

Command succeeded.

6> insert into towns values ('Billerica','Middlesex','MA');

(Returned 1 rows in 0.05s)

7> insert into towns values ('Buffalo','Erie','NY');

(Returned 1 rows in 0.01s)

8> insert into towns values ('Bay View','Erie','OH');

(Returned 1 rows in 0.00s)

9> select count(*) as total from towns;

TOTAL

------

     3

(Returned 1 rows in 0.02s)

10> select town, state from towns ORDER BY town;

TOWN       STATE

---------- ------

Bay View   OH

Billerica  MA

Buffalo    NY

(Returned 3 rows in 0.01s)

11>

• Now again Delphix snapshot
• Suddenly small disaster happens

[volt@volt1 ~]$ sqlcmd
SQL Command :: localhost:21212
1> select * from towns;
TOWN       COUNTY     STATE
---------- ---------- ------
Billerica  Middlesex  MA
Buffalo    Erie       NY
Bay View   Erie       OH

(Returned 3 rows in 0.02s)
2> delete from towns;
(Returned 3 rows in 0.01s)
3> select * from towns;
TOWN  COUNTY  STATE
----- ------- ------

(Returned 0 rows in 0.00s)
4> exit

• Fortunately we have Delphix snapshot done before our disaster. What we can do?

1. Shutdown VoltDB
2. Rewind to Delphix snapshot done before our unwanted operation
3. Restore VoltDB

[volt@volt1 ~]$ voltadmin shutdown
Status: -1
Information: Connection broken

Click "Rewind" button.

[volt@volt1 ~]$ voltdb recover --deployment=voltdbconfig_2.xml
Initializing VoltDB...

 _    __      ____  ____  ____
| |  / /___  / / /_/ __ \/ __ )
| | / / __ \/ / __/ / / / __  |
| |/ / /_/ / / /_/ /_/ / /_/ /
|___/\____/_/\__/_____/_____/

--------------------------------

Build: 5.6 voltdb-5.6-0-g3743684-local Enterprise Edition
Connecting to VoltDB cluster as the leader...
Host id of this node is: 0
Starting VoltDB with trial license. License expires on Oct 26, 2015.
Initializing the database and command logs. This may take a moment...
WARN: This is not a highly available cluster. K-Safety is set to 0.
Restoring from path: /home/volt/voltdbroot2/command_log_snapshot with nonce: 1442441721521
Finished restore of /home/volt/voltdbroot2/command_log_snapshot with nonce: 1442441721521 in 0.13 seconds
Server completed initialization.
 

• Now we can again select our table. Just after few seconds. In case of big databases this will be no more than 15 minutes.

[volt@volt1 ~]$ sqlcmd
SQL Command :: localhost:21212
1> select * from towns;
TOWN       COUNTY     STATE
---------- ---------- ------
Billerica  Middlesex  MA
Buffalo    Erie       NY
Bay View   Erie       OH

Conclusion

VoltDB and Delphix are great tandem. If we store our persistence data in Delphix vFiles we can have:

• Live backup
• Easy way to multiply environments. We can provision new instances of VoltDB instantly and without administrative overhead. It is easy to imagine that we can store in Delphix not only persistence space but also VoltDB binaries.
• Easy way to recover from human errors just by rewinding Delphix vFiles
• Extreme agility for test and development purposes