Plugins

0

LittleCMDB (An Orchestrator and WaveMaker project) – Part 1

Table of Content

In Part1 we start with the SQL DB Plugin and create the required database for our need.

In Part2 we start with the development of our Workflow. We will start with a few elements.

In Part3 we  finish the  collection of the VM information.

In Part4 we insert our data into the database and test our created workflow

In Part5 we create our webview to get a look on our Data in the SQL Database

In Part6 we will make our Workflow smarter to update the DB with actual VM information

In Part7 problems with vAPP located virtual machines are fixed

Part1

Today I want to start with a series of posts for an example how to to create an Orchestrator and Wavemaker project. This project came from a “real-world” situation. The customer wants his VM configuration saved outside the virtual environment into a SQL Database. During the talks with the customer, the idea for a little CMDB was born. The actual customer project is much bigger than the example in this article series, but you will become a good insight view on all relevant topics and themes.

When we think about a CMDB we have to choose which data we want to save in our database. Here are some things I included in my database:

  • Unified Identifier (VMID) (The primary key for the VM)

  • VM UUID (the second key for the VM)

  • VM name

  • CPU Configuration

  • Memory Configuration

  • Cluster

  • Host

  • Folder

  • Resource Pools

  • Network

  • IP Address

  • Datastore(s)

  • Disk Size

Surely there are more things which could be integrated but in this post I will focus these values. These identified values bring us to a database schema. Also there are not so many values; I prefer to save the data in different tables. On every table I choose the Unified Identifier with the name VMID as primary key. As second key I select the VMUUID with should be unique for every virtual machine within a vCenter Server. I also split the different information for the VMs in different tables for maybe future grow.

I create the following tables with these values:

VM_Info

Name

Data Type

IsNull

Comments

VMID

varchar(100)

False

PrimaryKey

VMUUID

varchar(100)

False

VMName

varchar(100)

False

CPUConfig

Numeric(18)

False

MemConfig

Numeric(18)

False

VM_Host

Name

Data Type

Is Null Allowed

Comments

VMID

varchar(100)

False

PrimaryKey

VMUUID

varchar(100)

False

Cluster

varchar(100)

True

Host

varchar(100)

False

ResourcePool

varchar(100)

True

Folder

varchar(100)

True

VM_Network

Name

Data Type

Is Null Allowed

Information

VMID

varchar(100)

False

PrimaryKey

VMUUID

varchar(100)

False

Network

varchar(100)

True

Multiple Values Possible

IPAddress

varchar(500)

true

Multiple Values Possible

VM_Datastore

Name

Data Type

Is Null Allowed

Information

VMID

varchar(100)

False

PrimaryKey

VMUUID

varchar(100)

False

DiskSize

varchar(100)

True

Multiple Values Possible

Datastore

varchar(500)

true

Multiple Values Possible

I am not a database Admin and I cannot say if this is a good DB schema, but it fits my needs and I can grow with additional tables.

As Database a use a MS-SQL Server. My DB is named LittleCMDB. Here is the Script to create the DB on an MS-SQL or MS-SQL Express DB.

USE [LittleCMDB]
GO
/****** Object:  Table [dbo].[VM_Network]    Script Date: 07/13/2012 15:33:09 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[VM_Network](
[VMID] [varchar](100) NOT NULL,
[VMUUID] [varchar](100) NOT NULL,
[Network] [varchar](100) NULL,
[IPAddress] [varchar](500) NULL,
CONSTRAINT [PK_VM_Network] PRIMARY KEY CLUSTERED
(
[VMID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
/****** Object:  Table [dbo].[VM_Info]    Script Date: 07/13/2012 15:33:09 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[VM_Info](
[VMID] [varchar](100) NOT NULL,
[VMUUID] [varchar](100) NOT NULL,
[VMName] [varchar](100) NOT NULL,
[CPUConfig] [numeric](18, 0) NOT NULL,
[MemConfig] [numeric](18, 0) NOT NULL,
CONSTRAINT [PK_VM_Info_1] PRIMARY KEY CLUSTERED
(
[VMID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
/****** Object:  Table [dbo].[VM_Host]    Script Date: 07/13/2012 15:33:09 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[VM_Host](
[VMID] [varchar](100) NOT NULL,
[VMUUID] [varchar](100) NOT NULL,
[Cluster] [varchar](100) NULL,
[Host] [varchar](100) NULL,
[ResourcePool] [varchar](100) NULL,
[Folder] [varchar](100) NULL,
CONSTRAINT [PK_VM_Host] PRIMARY KEY CLUSTERED
(
[VMID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
/****** Object:  Table [dbo].[VM_Datastore]    Script Date: 07/13/2012 15:33:09 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[VM_Datastore](
[VMID] [varchar](100) NOT NULL,
[VMUUID] [varchar](100) NOT NULL,
[DiskSize] [varchar](100) NULL,
[Datastore] [varchar](500) NULL,
CONSTRAINT [PK_VM_Datastore] PRIMARY KEY CLUSTERED
(
[VMID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO

For the usage of a DB in Orchestrator, you can use the SQL-Plugin. The Plug in could be get from the VMware Website (https://my.vmware.com/web/vmware/details?downloadGroup=VCO_SQL_PLUGIN_10&productId=229)

The Installation is done over the Orchestrator Configuration site (http://ORCHESTRATOR_IP:8282). I will not describe the installation. If you are not familiar with the plug in installation then RTFM in the official documentation here http://pubs.vmware.com/orchestrator-plugins/index.jsp?topic=/com.vmware.using.sql.plugin.doc_10/GUID-66110FFD-EB0F-484A-937B-8C131C8DFFB2.html

After you finished the installation of the SQL Plug in, it is time to integrate the SQL Server and the required Database tables.

You can add your database with the vCO Client Workflows Library SQL “Add a database”

There you have to define your SQL Connection Details. In my case, that are the required parameter for the SQL connection. In my case that are:

  • A name for the connection

  • The SQL Server IP with Port and database name (for details look in the screen shot..)

  • A username with permissions on the DB

  • And the password for the user

After the database is successfully added, we have to integrate the preconfigured tables. This is done with the workflow “Add tables to a database”

 

After the start of the workflow we first have to choose the database

In the opening window we take our DB (LittleCMDB in my case…)

 

Then we have to integrate our tables

 

You can add tables by clicking in the “Tables” field

 

there you have to insert all fields beginning with VM_*. Add the end the your Array of String has to look like this:

 

After you have „Accept“ your choice you can „Submit“ the Workflow.

 

Before I start to generate the SQL Statements, I create a Folder in which I will place my SQL-Statements. For that I have created a folder “vcoportal.de” there a sub folder “LittleCMDB”. In the “LittleCMDB” Folder a also created a Subfolder with the name “SQL-Statements”.

Feel free to change your names and folders, i prefer this structure.

To generate your SQL-Statements we have to browse to the Workflow “Generate CRUD workflows for a table”

This Workflow must be executed for every table we want to use.

For that, you have to provide the following inputs:

  • The table

  • The destination directory

  • If you want to overwrite existing workflows

  • And the read-only columns

 

Here is a example for one of my tables:

 

When you are ready for your tables, take a look into your destination folder. For every table, there must be a record to “Insert, Update, Read and Delete” values in your tables.

The preparation of the SQL Server statements is done. Know we can start to build up our workflow to feed the SQL Database…..

So, that’s all for Part1. Stay tuned for Part2. Then we will start to create the Workflow…..

SQL Script
SQL Script
LittleCMDB_SQL.zip
622.0 B
Details...
0

The Dream of a self-healing Datacenter: Integrate vCenter Operations (vCOPS) with Orchestrator

Workflows in vCenter Orchestrator allow you to automate tasks in vCenter. That’s “Kindergarten“.

Workflows also allow you to orchestrate IT-Services all over the infrasctructure, leveraging all that generic or specific plugins (Wanna see the list? Go to VMware’s Solution Exchange: https://solutionexchange.vmware.com/store/category_groups/15/categories/21).
That’s the “Advanced to Master“-Level.

Kicking-off “healing” workflows based on “unhealthy” conditions in your datacenter fully-automated, using vCO as a headless orchestration platform? That sounds like a job for Wizards!
Well, let’s see……

Orchestrator Wizard (bearded)

The Basics

What is VMware vCenter Operations?
Quoting http://www.vmware.com/products/datacenter-virtualization/vcenter-operations-management/overview.html:
“… Automate performance, capacity, and configuration management with patented analytics and an integrated approach to management. Eliminate the finger pointing, improve team collaboration and reduce manual problem solving efforts by as much as 40% with automated root cause analysis… blah blah Marketing blah 🙄 ”

In short: It’s great! 

VMware vCenter Operations (vCOPS)


What do we need in vCenter Orchestrator?
Just some basic facts
: Orchestrator Workflows are triggered mostly in one of these three ways:

  • Manually: By an administrator (using the vCO-Client), or an Helpdesk-Agent (using the weboperator Webview, or the Perspectives-Plugin) or  a customer/end-user (via a self-service portal (built in Wavemaker of course  ;-))
  • As Scheduled Task in vCO: The workflow will be started by the vCO server on a regular base, e.g. for reporting, checking for exhausting snapshots, …… (It’s even possible to schedule a workflow programmatically, see a nice example here: http://communities.vmware.com/thread/318791)
  • By an external System: Another component of your Infrastructure kicks-off the workflow via the API. This could be an high-level Business-Process engine (like VMware Service Manager), a system to manage classroom-environments, a release management system, an I/S/P/X/Y/Z-aaS Manager, whatever…

And there is a fourth, not so well known way to start a workflow:

Policies in Orchestrator to start Workflows on signals

  • Event based: The workflow is started when a specific event occurs . You can either use a “Waiting event”-Element in the workflow, or you can create a Policy in vCO. For both vCO-Plugins can provide Triggers.
    Examples:  The AMQP-Plugin (which can be triggered e.g by a Blocking Task in vCloud Director), or the SNMP-Plugin (listens to SNMP-Traps, sent e.g. from vCenter or other systems…)
    Waiting Event in a workflow

Subtotal:

vCenter Operations (vCOPS) can send SNMP-Traps when unwanted situations occur.

vCO can start workflows when an SNMP-Trap is received.

So: Integrate (again)!

Step 1:  Install the SNMP-Plugin on your vCO-Server. RTFM on https://www.vmware.com/support/pubs/vco_plugins_pubs.html

Step 2: Run the Workflows Library / Trap Host Management / Set the SNMP trap port and Start the trap host to make vCO (exactly: the SNMP-Plugin) listen to SNMP traps.

Step 3: Run the Workflow Library / Device Management / Register an SNMP device and make vCO listen to SNMP traps exactly from vCOPS. Make sure you use the hostname / IP of the vCOPs Analytics VM, not the UI VM!

Run Workflows to setup the environment

Step 4: Configure vCOPS to send SNMP-Traps to the Orchestrator server. Open the vCOPS Manager Administration, and define your vCO-Server as (receiving) Host.

Enable SNMP notifications in vCenter Operations

Step 5: Configure vCOPS  to “activate” alerting: Open the menu “Notifications” in vCOPs usual webinterface (not the Administration anymore), and create a “New Rule” and enable all the Conditions you want to be notified on.
(I’m not sure if this step is really necessary,  I didn’t find any information if these rules are for email only, or if they are also used for SNMP notifications 😕 )

Create Rules to enable alerting

Step 6: For a first test: Run the Workflow Library / SNMP Wait for a trap on an SNMP device. Select the vCOPS Analytics VM as SNMP Device, you registered it in Step 3!
The Workflow should stall at the “Waiting Element” for an SNMP trap from vCOPS. Now break something in your datacenter 😈 , so that vCOPS sends a trap…

Waiting for something bad...

Once the trap is received by vCO, the workflow should continue and finish successfully. In the Logs-Tab of the Workflow-Token you can find some details about the trap.

Step 7To be the wizard: Create a new Policy based on the Policy Template which comes with the SNMP Plugin.
Apply a new  Orchestrator Policy

Then edit the created Policy to select the Workflow to be started when a trap is received:

Step 8: Start the Policy! (That’s something you will forget only once, after hours of senseless troubleshooting…)

DON'T FORGET TO START THE POLICY!

Finito!

Now every condition in your datacenter which leads to an alerting in vCenter Operations will trigger a Workflow in Orchestrator, which can deal with the problem. And the best: All that without writing a single line of code!

From there I leave it to you…

<site note>
Integrating a Systems Monitoring tool with vCenter Orchestrator is not a new:
up.time software provides a vCO plugin for years:
http://support.uptimesoftware.com/orchestrator.php
http://www.uptimesoftware.com/uptimeblog/uncategorized/cloud-computing-and-popular-culture/
</site note>

The Dream of a self-healing Datacenter

Proofing that it’s possible, consider what you can do (or better: what you can let do fully-automated) with an integration of vCenter Operations and vCenter Orchestrator:

  • Run a workflow that automatically creates a Ticket or updates the CMDB on vCOPS alerts
  • A datastore runs out of capacity? A workflow will fix it automatically, increase the LUN on the storage system and the VMFS partition. Or just full-stack provision a new datastore and add it to the storage cluster.
  • YourMightyStorageVendor(tm) provides an additional workflow: If now your storage system runs out of capacity, new disks will automatically be ordered by the workflow 🙂
  • Response time for Outlook users to high? A workflow will deploy new Exchange CAS instances to scale-out (and leverage the Powershell-Plugin to adjust it settings)
  • Your web-app is getting “Slashdotted”? See how Radware already leverages vCO to scale it fully-automated: http://www.youtube.com/watch?v=4rkV3ebQens
  • “No worries, Captain! I already put 80% energy to the front shields to prepare for the Klingonian attack!”
  • With the brand-new vCenter Operations for VMware View and a View API/Plugin for vCO you could… … …

***RING – RING – YOUR WAKEUP CALL – RING – RING ***

SIGH! What a nice dream…  :mrgreen:

0

Let your vCO Workflow text you! Yes, via SMS!

Working on another blogpost on some design best-practices for workflows (stay tuned…) a promising idea came into my mind:

What about sending text messages to my mobile from workflows?

I guessed there are many SMS-Gateway-Services out there which provide any kind of SOAP or REST-API… (I want to leverage an existing plugin for vCO for quick results). So google came up with this nice discussion:

http://stackoverflow.com/questions/2383542/recommendations-for-sms-gateways-with-api-support

I decided to give https://www.clickatell.com a try, because they

  • provide interational service
  • provide multiple kinds of API
  • seem to be reliable and widely used
  • don’t force you in a contract/monthly plan for my limited test usage

What you need to reproduce this example:

  • a clickatell account (I think it will work with most of the other SMS-Gateway services in a comparable fashion)
  • the REST-Plugin for vCO installed in your Orchestrator environment (see details here: http://www.vmware.com/support/pubs/vco_plugins_pubs.html (Select HTTP-REST for the plugin-download and a bunch of examples)

Step-by-step…

I quickly signed-up for an account, used the web-form to send a test message to German) mobile number, and prepaid for some credits to be able to send via the API.

After that, I created a new access for the “HTTP”-API of clickatell (no special settings):

(You may open clickatell’s API Documentation now in background if you’re a read-first-try-then kind of guy.)
For myself I just started my vCO client and ran the “Add a REST host”-Workflow from the library, providing just a name and http://api.clickatell.com as URL:

The workflow finished sucessfully, I confirmed the new host in the Inventory:

Now I called the “Add a REST operation”-Workflow to invoke the API-Call to clickatell (based on their example given when creating the API account):

(instead of hardcoding my credentials I used the generic {parameter}-notation of the REST-plugin for all the necessary parameters like username, password, API-Id, …, so that I can re-use the  operation later flexibly). The full “Template URL” value is:


/http/sendmsg?user={username}&password={password}&api_id={apiid}&to={tonr}&text={message}

When the operation was created (again: Confirm in Inventory-Tree) I started the “Invoke a REST operation” workflow providing all the input parameters:

aaand voila,  😎 seconds later the text messgae arrived on my mobile!   :mrgreen:

Now, the real-world use-case:

Well, sending generic messages to a mobile from a vCO workflow is cool|geeky, but what for? For a realistic example I used the Workflow-Generation of the REST-Plugin to let vCO generate my very own “Invoke ‘send a text message'”-Workflow in my very own Workflow folder for easy re-usage:

So I was able to re-use the created workflow in my Snapshot-Hunter-Workflow… (you know the pain with backup software which accidentally forgets to cleanup the VM snapshots! :evil:)

(The first element to find these workflows leverages the xpath-filter of the VcPlugin.getAllSnapshots()-Method, see the discussion here: http://communities.vmware.com/thread/342686 and http://communities.vmware.com/message/1673575 )

At the end: After scheduling the workflow as Scheduled Task in vCO, starting from tomorrow I will get waked-up by my mobile receiving an text message from vCO when there are leftover snapshots from the nightly backups!

Summary

  • Don’t fear any tasks integrating vCenter Orchestrator! When the external system provides an API, chances are that you get in run in a few minutes…
  • Remember: less and simple workflow inputs make it easy to use, many and generic workflow inputs make it easy to re-use! (referencing the “Workflow Development Best Practices”-session as a shameless plug :mrgreen:)
  • Sometimes it IS good to be creative|distracted when drafting a blog-post (expect the post I actually was working on soon…)
  • Turn off your mobile when you want to sleep in!
  • …and thank god I’m too tired to think about other use-cases for this…
    (I will not make an Orchestrator & Ringtone joke!
    I do not plan to provide a monthly subscription for my content, just text a SMS with “vCO Rockz!” to …..
    And I better not search for an API for Asterisk (“If you want to execute workflow XYZ, press 1; for all other requests press 2” 8-O)

Download the example

SnapshotHunter Package
SnapshotHunter Package
de.vcoportal.snapshothunter.package
23.3 KiB
Details...

Be aware that it will NOT run out-of-the-box, because you have to add the REST-Host and the Operation in your environment! And: Don’t forget to set the necessary values for username, password, mobilenr, … in the configuration elements.You know that blogpost whith a great description of Configuration Elements, don’t you! 😛

0

4 (in words: FOUR) new Plugins for vCenter Orchestrator released

Last Friday VMware released FOUR new Plugins for vCO (read from the source here: http://blogs.vmware.com/orchestrator/2011/12/vmware-releases-four-new-vmware-vcenter-orchestrator-plug-ins.html )

  • The SQL Plugin makes it possible to access external Databases within workflows (in a much more comfortable and reusable way than the already existing plain JDBC-calls)
  • The Auto Deploy Plugin automates the new vSphere 5 Auto Deploy mechanism, which absolutely makes sence to orchestrator the deployment of ESXi hosts.
  • The vCO Multi Site Plugin allows you synchronize, execute and monitor Workflows on other vCO-hosts (this also was possible in the past via the “Nested Workflows”-Element, but now its murch more powerful and flexible). There are several use-cases for this, expect a more detailed blogentry here soon 😀
  • The PowerShell Plugin allows you to call external PowerShell Scripts on other systems, via OpenSSH or WinRM. The plugin also implements a “piping” methodology in your workflows, so if you’re familiar with PowerShell, you can build workflows in a comparable way. Calling external PowerShell scripts is very powerful, and was also available in the past (see http://www.vcoportal.de/powersshell-plugin/ and http://www.vcoportal.de/2011/03/vco-powershell/ ).
    You may ask, what’s the difference and overlapping points between VMware’s PowerShell Plugin and my PowerSSHell-Plugin for vCO… expect more details about this soon as well :mrgreen:

So, the list of available plugins for vCO gets longer and longer, it seems VMware is on the right way to establish a good ecosystem around vCO…

0

VMware Labs releases CIM Plugin for vCO

Newest Fling from VMware Labs: An Orchestrator-Plugin to get CIM-Information from ESX-Hosts. At the first glance this enables great abilities for Monitoring and Reporting Workflows.

Download the Plugin from here: http://labs.vmware.com/flings/cim-plugin