Get Started!

2

Application Onboarding with vCO and WaveMaker

Last month I had a customer, who wanted an onboarding process for his virtual machines. This onboarding should work as follow:
– There is a standard Linux template for all virtual machines
– The users can choose from a web site what machine type is required
– From the website a vCO workflow is started which clones the template and installed the required packages into the VM
– At the end the application configuration is done
To be honest, the customer requirements were quite simple so the usage of Puppet or Chef was not an option.
After everything the customer wanted was clear, I thought about the options and came to the idea to choose the vCO and WaveMaker for the solution.
I don’t like passwords in Workflows so the first think I did, was creating a vCO SSH Key. This could be done with a predefined workflow in vCO.

After I had my SSH public key, I created the Linux template with the public key in the file /root/.ssh/authorized_keys
You can do this manually or also use a predefined vCO Workflow

This allows me, to administer the Linux VM without the need for a password.
The next thing I did, was to create a folder on the vCO Appliance. I created the folder under the root directory with the name ConfigFiles. The vCO has a secure configuration so to allow the vCO Server to access this folder you must allow this explicit. For that, you have to edit the file: js-io-rights.conf on the vCO appliance this file is located under the path:

/opt/vmo/app-server/server/vmo/conf/js-io-rights.conf

I added my created path and after that, the file was looking like this:


-rwx /
+rwx /var/run/orchestrator
+rx ../../configuration/jetty/logs/
+rx ../server/vmo/log/
+rx ../bin/
+rx ./boot.properties
+rx ../server/vmo/conf/
+rx ../server/vmo/conf/plugins/
+rx ../server/vmo/deploy/vmo-server/vmo-ds.xml
+rx ../../apps/
+r ../../version.txt
+rw /ConfigFiles

After you have done this editing, you must restart the vCO Service to take the changes in effect.
For later use, I placed a file with the name “named.conf” into the folder. This file contains the configuration for the bind installation which I will use later in this Blog post.
For our Onboarding, we need the SCP workflow. With the predefined workflow, we have a little problem…..it doesn’t work with ssh Keys. So we have to modify the predefined workflow. Therefore we copy the existing workflow.

I called my workflow “SCP Put command with SSH Key”. After we have copied the workflow, we have to modify it.

We have to change the content in the SCP put file Scriptable task.

try{
var session = new SSHSession(hostName,username);

if(passwordAuthentication){
System.log("Connecting with password");
} else {
if(path == null || path == ""){
System.log("using default");
path = defaultKeyPairPath;
}
System.log("Connecting with key pair ("+path+")");
password = passphrase;
}

session.connectWithPasswordOrIdentity(passwordAuthentication,password,path);
System.log("Connected!");

session.putFile(localFile,remoteFile) ;
output = session.getOutput();
error = session.getError();
exitCode = session.exitCode;

System.log("Output: '"+output+"'");
System.log("Error: '"+error+"'");
System.log("Exit code: '"+exitCode+"'");

session.disconnect();

} catch (e) {
throw "Unable to execute command " + e;
}

After we have made the scripting changes, we need to add an Attribute. I choose defaultKeyPairPath as name. It is from type string and added as value ../server/vmo/conf/vco_key (the path to the vCO SSH Key).

On the input site of the workflow we have to add three Inputs. The first we have to create is passwordAuthentication from type boolean. As default value we choose no. The second Input variable is path from type path. The last one we need is passphrase from type SecureString. The last one is required, if we protect our SSH Key with a passphrase.

After we have created the SCP Put command with SSH Key it is time to build our workflow.
In this blog, we create a simple onboarding workflow to configure a Name Server with bind. The Template I use is a Cent-OS Minimal installation with installed VMwareTools and the SSH Public Key from the vCO Server.
I created a new Workflow with the name “Blog_Configure_DNS”.

In this workflow we go to the schema and add the following Workflows.
The first one is a Workflow with the name “Clone, Linux with single NIC” to clone the template. The second workflow we use is “Run SSH command”. The next one is our “SCP put command with SSH Key” and as last one we also use “Run SSH Command”.

Now we rename the commands. I rename the first “Run SSH….” to “Install named” the “SCP put command….” to “Put configuration named.conf” and the last Workflow to “Restart named”.

After the workflow design, you have to create the needed Attributes, In- and Outputs for every element. I did some examples in my “Little CMDB” series so if you are not familiar to create the needed parameter take a look here:

http://www.vcoportal.de/2012/07/introducing-the-littlecmdb-a-vcenter-orchestrator-wavemaker-demo-project/
and following.

When you’re finished with this, we have to build up the WaveMaker interface.
There are also a lot of good examples available to do this. You will find lot information here:
Using WaveMaker as Web-Fontend for vCO

http://www.vcoportal.de/2011/11/using-wavemaker-as-web-frontend-for-vco/

Off-topic(?): Lessons learned with WaveMaker
http://www.vcoportal.de/2012/02/lessons-learned-with-wavemaker/

Howto setup LDAP-Authentication for Wavemaker (Part 1 & Part2)
http://www.vcoportal.de/2012/05/howto-setup-ldap-authentication-for-wavemaker-part-1/

http://www.vcoportal.de/2012/07/introducing-the-littlecmdb-a-vcenter-orchestrator-wavemaker-demo-project/

To choose the right configuration, you can create a Drop Down field with different options. Here a Screen Shot for the DNS Server.

And here a Screen Shot for the DHCP Server with definition of the Scope.

Beside the option to use WaveMaker, you could also use an automatic provisioning depending on the actual load of a resource pool. I made an example video to show how it could be done with the vCO. You can find the video here:

http://www.vcoportal.de/2012/11/generate-vms-based-on-actual-load-in-a-resource-pool/
So have fun and orchestrate your virtual environment 😉

1

vCO and Veeam Backup&Replication a powerful combination

Last week I did a Webinar for Veeam in Germany. My topic in this webinar was Automation and Orchestration. Due the circumstance that the Webinar was in German, I decided to make this post to share the information for the rest of the non-German speaking world.

For those who understand German the Webinar was recorded and can be found here:

http://www.veeam.com/de/videos.html?ad=de-topmenu

Before you start some really important notes on the combination of vCO and Veeam B&R.

Some Veeam B&R commands need a connection to the vCenter Server. When you invoke your commands in a PowerShell Window on your Backup host, the PowerShell uses CredSSP to provide the vCenter Server Login Information’s from Veeam B&R to the vCenter Server. If you do the same in a vCO Workflow, this does not work! The reason why it not work is because the vCO PowerShell plugin only Supports Basic- and Kerberos Authentication. In every Environment I used so far, the Servers where in a Windows AD. This allowed me to use the Kerberos Authentication in the vCO PowerShell Plugin. In the last time, I did many tests with the Basic Authentication and had a lot of problems and errors with that type of Authentication. So my recommendation for the Authentication is “use the Kerberos Authentication to avoid a lot of trouble and problems!”

Prepare the Backup Server

At the moment Veeam Backup&Replication has no SOAP or REST API Interface. The only available interface is PowerShell.  To use the Power Shell from vCO some necessary preparations has to be done.

First of all Veeam Backup&Replication must be installed with the PowerShell Extension. This is done during Installation or if you already installed it without PowerShell  to just start the Installation again and add the PowerShell feature.

After you have installed the PowerShell Extension, you can start it from the Management Console.

This Button starts a PowerShell shell with an already loaded Veeam Extension. The Files for this Veeam PowerShell Extensions reside here:  “C:\Program Files\Veeam\Backup and Replication” in this path the file ”Install-VeeamToolkit.ps1” is important to load the extension automatically. We will use this file later in our vCO Workflows.

The next we have to do is to check if the Veeam Backup Server has the PowerShell in the Version 3.

For the first workflows and test I recommend to change the Host execution Policy to unrestricted. When everything goes fine, you can change the execution Policy to remote-signed

Set ExecutionPolicy (RemoteSigned / Unrestricted )

After that, we need a command Window on the Backup server. Here we have to insert the following commands:

Run the following command to set the default WinRM configuration values.


c:\> winrm quickconfig

(Optional) Run the following command on the WinRM service to check whether a listener is running, and verify the default ports.

c:\> winrm e winrm/config/listener The default ports are 5985 for HTTP, and 5986 for HTTPS.

Enable basic authentication on the WinRM service.

Run the following command to check whether basic authentication is allowed.

c:\> winrm get winrm/config

Run the following command to enable basic authentication.

c:\> winrm set winrm/config/service/auth @{Basic="true"}

Run the following command to allow transfer of unencrypted data on the WinRM service.

c:\> winrm set winrm/config/service @{AllowUnencrypted="true"}

Enable basic authentication on the WinRM client.

Run the following command to check whether basic authentication is allowed.

c:\> winrm get winrm/config

Run the following command to enable basic authentication.

c:\> winrm set winrm/config/service/auth @{Basic="true"}

Run the following command to allow transfer of unencrypted data on the WinRM client.

c:\> winrm set winrm/config/client @{AllowUnencrypted="true"}

Run the following command to enable winrm connections from vCO host.

c:\> winrm set winrm/config/client @{TrustedHosts ="vco_host"}

After we have executed the commands, we are ready with the Backup Server. Let’s now switch to the vCO Server.

Prepare the vCO

From the view of the vCO the first and important thing is, that the PowerShell Plugin is installed and activated in the vCO Server. If you are not familiar with this, the documentation can be found here:

http://pubs.vmware.com/orchestrator-plugins/index.jsp?topic=/com.vmware.using.powershell.plugin.doc_10/GUID-8AE1CFF2-F6F0-4233-BDD9-F318E461AB2F.html

When the PowerShell Plugin is ready, we can start to add the Backup Server to our repository. This could be done be starting the PowerShell Workflow to “Add a new Server”. The needed information’s are self-explaining.

On the second site we have to choose as PowerShell remote host type “WinRM”. As Protocol we use “HTTP” or “HTTPS”. The last point is Authentication. Here we choose “Kerberos”.

On the last page we have to choose if we use a “Shared session” or a “User Session”. When you chose the shared session you have to insert User credentials. When you decide to use “User Session” you have to insert the Authentication Details in every PowerShell call.

After we are finished with the pre requirements we can start with our first Workflow. Let’s us a simple one…..

Develop the vCO Workflows

If we want to figure out which Veeam Jobs exist on our Backup Server we need the command Get-VBRJob.

The easiest way to start is to copy the Workflow “Invoke a PowerShell Script” into a folder of your choice.

There you have to insert a second scripting element and move the host and script Inputs as Attributes.

In this scripting element we put our script which includes the PowerShell code.

To use a Veeam PowerShell command in a vCO Workflow we need somewhat more input then just the command. We have to load the Veeam Extension into our PowerShell Session which we invoke from the vCO Server. Here is the complete code for the call:

script = "# Load Veeam Powershell Extension into the actual session \n"
+ "'C:/\Program Files/\Veeam/\Backup and Replication/\Install-VeeamToolkit.ps1' \n"
+ "add-pssnapin VeeamPSSnapin \n"
+ "# Veeam is loaded \n"
+ "Get-VBRJob";

For us the full example looks like this.

Now you can use this command in your own workflows. Now that command isn’t really useful by now. Let insert a virtual machine into a Backup Job after creation. For that we have to use the Veeam Command Add-VBRJobObject. For this command we need some information which we can collect during the Session. A full command to insert a VM into a workflow looks like this:

Add-VBRJobObject -Job $(get-VBRjob -Name "+ JOBNAME+ ") -Server $(get-VBRServer| Where {$_.Type -eq 'VC'}) -Objects " + VMNAME + " }"

The Values JOBNAME and VMNAME must be specified as vCO Attributes or Inputs.

When you now try to execute this like the command before:

You will get an error like this one:

Failed to login to “vcenter.example.com” by SOAP, port 443, user „root”, proxy srv: port:0 +   CategoryInfo : InvalidOperation: (Veeam.Backup.Po…FindVBRViEntity:FindVBRViEntity)         [Find-VBRViEntity], Exception + FullyQualifiedErrorId :   Backup,Veeam.Backup.PowerShell.Command.FindVBRViEntity

Why this happens?  Here we get into trouble with the Authentication against the vCenter Server. If everything was fine before and you can execute the command from a PowerShell shell the problem is in your workflow. Like described before we have to Authenticate against the vCenter Server from our Workflow.  vCO has no option to do this automatically . We have to change our Workflow to this:

 

script = "invoke-command -session $(New-PSSession <strong>BACKUPSERVER</strong> -Authentication Kerberos -Credential $(new-object -typename System.Management.Automation.PSCredential -argumentlist<strong> USER@DOMAIN</strong>, $(convertto-securestring -string '<strong>PASSWORD</strong>' -asplaintext -force))) -scriptblock{ set-item wsman:localhost\Shell\MaxMemoryPerShellMB 1024"
      + "\n Add-PSSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue"
      + "\n Add-VBRJobObject -Job $(get-VBRjob -Name "+ <strong>JOBNAME</strong> + ") -Server $(get-VBRServer| Where {$_.Type -eq 'VC'}) -Objects " + VMNAME + " }"

This script looks really different then the script before. What do we do here? We generate a new Powershell session on the Backup Server (New-PSSession).  For this session, we define a Username (USER@Domain) and a Passwort (PASSWORD). For the Username it is very important that the user is written as user@domain. Otherwise the Kerberos Authentication will not work and the Workflow will fail! At last we set the Memory for the new Shell to 1024 MB (set-item wsman:localhost\Shell\MaxMemoryPerShellMB 1024) If we doesn’t exceed the Memory the workflow will also fail! At last we load the Veeam Snapin and execute the Script job…..

That’s easy or?

With this Background Knowledge you can start to implement your own Automation Workflows with included Backup of your virtual machines with Veeam. It is also possible to integrate the Replication….you have just to implement the replication command and start your Automation….

In the Veeam Community there is a good PowerShell forum. So if you have trouble with your Veeam PowerShell commands, get a look there:

http://forums.veeam.com/viewforum.php?f=26

Have fun with the Power of vCO 😉

10

Generate VMs based on actual load in a Resource Pool

During the last VMworld in Barcelona I was in a session (INF-VSP2033 – Auto Scaling and Cloud Bursting in the Hybrid IaaS Cloud) from Christophe Decanini and Chris Knowles who showed a solution with two sites located in Europe and the USA with vSphere and vCO environments on both sites. The sites were connected with F5 Loadbalancer and the load could be migrated from one Country to the other Country……sincerely the session is not online in the VMworld online session folder so you cannot see what the showed…..

To give you a hint what is possible with the vCO, I created my “own” solution based on the load in a given resource pool, see following video:

Some background notes what I did with my Workflow:

  • I monitor a resource pool for the load in the RP. It is also possible to do so in a folder
  • I can define a minimum amount of VMs with must be run in the RP. A deletion below this minimum point is not possible from the Workflow
  • I can define a maximum amount of running virtual machines in the RP. The workflow cannot provision VMs above the maximum amount.
  • My virtual machines witch were provisioned are all Linux based (I use a CentOS). The data for the machines (for example a Website) must be located outside the VMs. I connect them via NFS or iSCSI. The machines are an ideal example for vApps
  • The load is captured over a time interval of 5 minutes. Short peaks will not result in any action (create new VMs or delete existing VMs)

So have fun with the Video and get a look what is possible with the vCO. If you have questions feel free to ask (cs@vcoportal.de).

Load based generation of virtual machines from Christian Strijbos on Vimeo.

0

vCenter Orchestrator QuickStart Series for VMware Partners

VMware Partner University provides a free two-session QuickStart series for vCenter Orchestrator, running on September 19th and 20th 2012.
If you are a VMware Partner (AFAIK no matter what Partner Program/Level): Login to PartnerCentral, browse to Partner University and select “Programs/QuickStart Series”.

Regardless if you are a Consulting Partner or an Software vendor or an hardware Vendor:
vCenter Orchestrator becomes a more and more important piece in VMware’s portfolio, so it’s about time to get started!
NOW!

0

vCO, SNMP Traps and vCOPS

Today I had the request, to generate a vCOPS Demo with an vCO integration. For that, I had a first look on Jörg’s post about is “Self-Healing datacenter” which can be found here: http://www.vcoportal.de/2012/05/integrate-vcops-and-vco/

For me, that was a good starting point but I want to integrate the SNMP Trap with an existing workflow. The creation of a Workflow to deal with snmp traps is not a big problem. The bigger problem is the “automatic” processing via Policy’s because there is no documentation (or I didn’t find it) . So I started to talk with Jörg and made some research in the web.  There I found this excellent post from William Lam about “Automatically Securing Virtual Machines Using vCenter Orchestrator”  http://blogs.vmware.com/vsphere/2012/07/automatically-securing-virtual-machines-using-vcenter-orchestrator.html

This post from William (and his package) has everything inside we need.

First we have to create the preconditions like the SNMP installation in the vCO and the configuration in the vCOPS and the vCO. Jörg has this already documented here: http://www.vcoportal.de/2012/05/integrate-vcops-and-vco/ so I will not repeat this steps here.

Let’s start with the Workflow development. First, we have to create a new workflow. I will call it “Execute Host Maintenance after trap” and I insert the following description “Waits to receive an SNMP trap from a vCenter Server instance, then set the host in maintenance mode or exist the maintenance mode “

After we created the Workflow, we go to the “Schema” tab

Here we insert some Elements. We need:

  • One Scriptable Task
  • One Decsion
  • The Workflow Enter Maintenance Mode
  • The Workflow Exit Maintenance Mode
  • And a Workflow End.

When you are ready your order should look like mine.

Now, let’s start with the Scriptable Task. I name it “Retrieve Host”. We want to extract the information out of the SNMP Message, for that we need some variables and SNMP Trap information. Let’s start with the SNMP Trap Information. The SNMP Messages comes with OID Numbers. Each OID Number has a different meaning and stands for a other component or message. The values of this OID Number can be found in the SNMP MIB Files which can be downloaded from the VMware Website. Every trap data has more than one OID Number

=============

oid: 1.3.6.1.2.1.1.3.0

type: Number

snmp type: Timeticks

value: 1743301911

Element 2:

=============

oid: 1.3.6.1.6.3.1.1.4.1.0

type: String

snmp type: OID

value: 1.3.6.1.4.1.19004.0.25

Element 3:

=============

oid: 1.3.6.1.4.1.19004.2.1

type: String

snmp type: Octet String

value: localhost

Element 4:

=============

oid: 1.3.6.1.4.1.19004.2.2

type: String

snmp type: Octet String

value: 10.10.120.183

Element 5:

=============

oid: 1.3.6.1.4.1.19004.2.3

type: String

snmp type: Octet String

value: Resource

Element 6:

=============

oid: 1.3.6.1.4.1.19004.2.4

type: String

snmp type: Octet String

value: 1346068003943

Element 7:

=============

oid: 1.3.6.1.4.1.19004.2.5

type: String

snmp type: Octet String

value: Critical

Element 8:

=============

oid: 1.3.6.1.4.1.19004.2.6

type: String

snmp type: Octet String

value: New alert by id 36 is generated at Mon Aug 27 11:46:43 UTC 2012; Root Cause : (2 SYMPTOMS)

1. MESSAGE EVENT    (1 OF 3)

33% - FAULT -

33% - CHANGE EVENT -

Element 9:

=============

oid: 1.3.6.1.4.1.19004.2.7

type: String

snmp type: Octet String

value: https://10.10.120.166/vcops-vsphere/?alert=36

Element 10:

=============

oid: 1.3.6.1.4.1.19004.2.8

type: Number

snmp type: Integer

value: 36

Element 11:

=============

oid: 1.3.6.1.4.1.19004.2.9

type: String

snmp type: Octet String

value: Verbindungsstatus der physischen Netzwerkkarte vmnic1 ist nicht bereit.

Element 12:

=============

oid: 1.3.6.1.4.1.19004.2.10

type: String

snmp type: Octet String

value: Health

Element 13:

=============

oid: 1.3.6.1.4.1.19004.2.11

type: String

snmp type: Octet String

value: Faults

For us, we want to search for the OID Number of the hostname and the error Message.

With this background information, we can create our needed In- and Outputs for our Workflow.

Local Parameter Variable Name Module Direction Type Value
trapData trapData Retrieve Host in Array/Properties
HostOID HostOID Retrieve Host in String 1.3.6.1.4.1.19004.2.2
MessageOID MessageOID Retrieve Host in String 1.3.6.1.4.1.19004.2.6
Host Host Retrieve Host out VC:HostSystem
SNMPMessage SNMPMessage Retrieve Host out String
NewAlert NewAlert Retrieve Host out Boolean

One important thing here: The trapData must be defined as Input Parameter not as attribute!

After we have created our variables, we can start with our scripting. I made a lot comments in the script, so everybody should understand what there happens….


// We extract the host name out of the SNMP TrapData
var HostName;
for (var x = 0; x < trapData.length; x++) {
var prop = trapData[x];
if (prop.get("oid") == HostOID) {
HostName = prop.get("value");
break;
}
}

// We compare the extracted Hostname with the Hosts registred in the vCenter Server.
// when we hava a match, we have the managed object ID of the Host
var hosts = VcPlugin.getAllHostSystems();
for (var i = 0; i < trapData.length; i++) {
var tempHost = hosts[i];
if (tempHost.name == HostName) {
Host = tempHost;
break;
}
}

// We search for the OID in the Message to get the Field with the recieved error Message
var SNMPTrap;
for (var y = 0; y < trapData.length; y++) {
var prop = trapData[y];
if (prop.get("oid") == MessageOID) {
SNMPTrap = prop.get("value");
break;
}
}

// Our search values in the Value Field of the Message
// When the Field contains the search string we beome the position. Otherwise be become a -1 value back.
var SNMPAlert = SNMPTrap.indexOf("New alert");
var SNMPCancel = SNMPTrap.indexOf("is cancelled");

// We check if the SNMP Trap has "New Alert" in the value field. Is so, we have a new alert.
if ( SNMPAlert != -1) {
NewAlert = true}
else {
NewAlert = false
};

Then we go to the Decsion. I name it “New Alert”. As Input we only need “NewAlert”. In the Decision field itself we set “NewAlert is true”.

After that, we go to the Enter Maintenance Mode. Here we only need to input variables:

Local Parameter Variable Name Module Direction Type Value
Host Host Enter Maintenance in VC:HostSystem
timeout timeout Enter Maintenance in number 0

Our Visual Binding has to look like this:

The same Variables and the same Visual Binding is required for Exit Maintenance Mode Workflow.

Local Parameter Variable Name Module Direction Type Value
Host Host Exit Maintenance in VC:HostSystem
timeout timeout Exit Maintenance in number 0