vCO and defensive scripting – practical experience

In the last couple of weeks I did a lot of Customer vCO Workshops to enable the Customers to create their own solutions. During the workshops I noticed some recurrent errors which were made from the  teams.

On some point of our Workflow, we will work with Userinputs. These Inputs can be direct Inputs from the users or maybe also Lines which are read from a file (CSV as example). Mostly these Inputs are compared to other Information’s . In this example I will use the ESX Hostname.

As you can see I have a VC with an cluster and one ESX Host with the name vmware01.example.com

Now we want to check if a Host with a specific name. So we use a really simple workflow.

In the scripting element I use this code:

for (i = 0; i < allHostsofVC.length; i++) {

if (HosttoCheck == allHostsofVC[i].name)


System.log("Yeah, we found the Host. Input is " + HosttoCheck + " Value from allHostsofVC[i].name is: " +  allHostsofVC[i].name);




System.log("oh no......no host found.Input is " + HosttoCheck + " Value from allHostsofVC[i].name is: " +  allHostsofVC[i].name);



So, now let’s make a first run

As we can see the search string and the hostname are identical.

Now let’s see what happens when the user write the hostname in Uppercase.

As we can see the name isn’t resolved. This leads to wrong results. The easiest way to avoid this is to change everything into LowerCase. So let’s modify our scripting element:

for (i = 0; i < allHostsofVC.length; i++) {

if (HosttoCheck.toLowerCase() == allHostsofVC[i].name.toLowerCase())


System.log("Yeah, we found the Host. Input is " + HosttoCheck + " Value from allHostsofVC[i].name is: " +  allHostsofVC[i].name);




System.log("oh no......no host found.Input is " + HosttoCheck + " Value from allHostsofVC[i].name is: " +  allHostsofVC[i].name);



Let’s start the last run again:

Also the User Input and the return string are different, we got the correct match.

This example leads us to another problem when we have to catch a user input regarding the hostnames. Most users tend to write only the name as shortname.

We can check it with the code we used before.

As we can see we didn’t catch the server. Why can we see in the vCenter Server MOB Browser. The Server is registered with the FQDN and from VMware API sight the shortname isn’t available.

How can we solve this problem? It is quite easy……. We just split the name at the dot and compare it with the name.

for (i = 0; i < allHostsofVC.length; i++){

var HostShortName = new Array()

HostShortName = allHostsofVC[i].name.toLowerCase().split(".")

if (HosttoCheck.toLowerCase() == allHostsofVC[i].name.toLowerCase() || HosttoCheck.toLowerCase() == HostShortName[0])


System.log("Yeah, we found the Host. Input is " + HosttoCheck + " Value from allHostsofVC[i].name is: " +  allHostsofVC[i].name);




System.log("oh no......no host found.Input is " + HosttoCheck + " Value from allHostsofVC[i].name is: " +  allHostsofVC[i].name);



Then we get our Hostname also when it is insert as shortname.

So, hope this help some to get their workflow created without to big problems. Have fun and Orchestrate the world ;-)


vCO and the parseInt JavaScript “Bug”

Today I was by a customer to develop a Custom workflow to evacuate one of his DCs with vCO. The evacuation should be done by only invoking an String into the Workflow. The Hosts were stretched over both DCs and also the Datastores. The ESX Hosts in his environment were named with vmwareXX were the XX represent a number. For the DC1 the Hosts had an even number. The Hosts in the DC2 the numbers are even.

There were hosts with names from

vmware01 till vmware24

So we developed some task to get the Datastores and the Hosts to divide them to their correct DCs. Therefore we created two Arrays were the hosts were placed for future actions. The names of the hosts were split and the Number was taken into a variable as integer. We made this with the command:

substring = parseInt(Hostname[i].substring(6,8))

After we had the number, we created an loop with if-else and placed the Hosts into the correct array.

var Odd_array = new Array ()

var Even_array = new Array()

var Nametemp

var Substring

for (i = 0; i < Hostname.length; i++) {

substring = parseInt(Hostname[i].substring(6,8))

System.log("Substring = " + substring + " Hostname = " + Hostname[i])

if (substring % 2)


//it is odd


System.log( Hostname[i] + " is in Odd Array")




//it is even


System.log( Hostname[i] + " is in even Array")



The above script generates this output.

Take a deeper look on the part with the hostnames vmware07 to vmware09. As you can see we get wrong results. The Host with the Name vmware07 was in the correct Array. The Hosts with the name vmware09 was in the even array were it was wrong.

It took some time to figure out that this is a JavaScript parseInt “Bug”. A good explanation about this behavior can be found here:


So for us the solution was to use

substring = parseInt(Hostname[i].substring(6,8), 10)

as command and everything worked like expected.

Maybe everyone expect me did not this before but this information could be useful for others so that is the reason why I created this post.

Have fun and orchestrate the World ;-)


Import workflow package to your vCO server via command line

vCenter Orchestrator’s REST API allows not only to start and monitor workflows, but also manage the content of your vCO server. With the Packages manager of the API it’s for example possible to import workflow packages via  the API. Example to do exactly this via command line using curl:

curl --form "name=file" --form "file=@de.vcoportal.someTest.package"  -X POST -u vcoadmin:vcoadmin -k

…with de.vcoportal.someTestPackage is the full filename of the package to upload.

See more details in the API reference documentation on https://your-vco-server:8281/vco/api/docs


Automate your Zerto Disaster Recovery solution with vCO

Today I show you how to automate your Zerto installation with the vCO. Or those which are not familiar with Zerto here a quote from the Zerto Website:

What is Zerto?

Zerto provides enterprise-class business continuity and disaster recovery (BCDR) solutions for virtualized infrastructure and cloud. [……]  Zerto Virtual Replication, is the industry’s first hypervisor-based replication solution for tier-one applications. Zerto Disaster Recovery solutions replace traditional array-based BCDR that was not built to deal with virtual environments.”

So for those which know the VMware Site Recovery Manager, Zerto is an alternative Product to the SRM.

For a customer Self Provisoning Portal I had to  automate the deployment of the virtual Machines within the Disaster Recovery Solution.  In my case this Automation must be archived with the VMware Orchestrator.

Zerto offers a limited documentation of the API which can be accessed with a REST API. The documentation can be found here: http://downloads.zerto.com/documentation_4acccec86a1e7cbf77f8e6bd3831c31c/3.1/Zerto%20Virtual%20Replication%20REST%20APIs%20Online%20Help/index.html

With this documentation I started my integration.

First thing what we need to do to integrate Zerto is to add the Zerto Hosts as REST Clients in the vCO Server. This integration is done over a vCO Workflow.

If have documented this before in another post (http://www.vcoportal.de/2014/02/automating-veeam-with-vco-and-the-restful-api/) so I will only point to the parameter which are needed.

For the configuration, you give a name, have to provide the URL to the ZVM Machine with the configured Port (the default port is 9669) and a Connection and Operation timeout. Please use for the connection a HTTPS connection. HTTP is possible but I had a lot of problems that the vCO run into timeouts.

We do not need a proxy Server.

As Authentication we use the “Basic” Authentication Type.

I used a “Shared Session” for the configuration but Single Sessions are also possible.

When you “Submit” the Workflow you will get a Certificate error.

Install the Certificate and go ahead.

After we have connected your Zerto REST Connection, we have to create our API connection. The initial request we must made is to make a call to the API. This call is done with a username and password for permissions to the vCenter Server. Here it is important, that the Username and the Password are for a User with permissions vCenter Server. The Zerto REST connection response includes an “x-zerto-session” header. The String in this Header is used for any future communication with the ZVM (Zerto Virtual Manager) and the vCenter Server. All other ways doesn’t function!

First thing we have to do is to create the initial request. For the request this URL is used: “https://zvm_ip:port/v1/“. I place this initial request in an own workflow. For the workflow we need some Attributes, Input and Output Variables.

Variable Name Type Value
Attribute sessionMode String Shared Session
Attribute RestAuthType String Basic
Inputs RestHost REST:RESTHost
Inputs authUserName String
Inputs authPassword SecureString
Output xZertoSessionKey String

In the Workflow we create a Script Element.

In this scripting element we include all Attributes and Inputs as Input and the Output element as Output for the Workflow.

Then we create our Script:

// we must use "dedicated" User Credentials. With the Provided Credentials we need Access to the VC!

var authParams = [sessionMode, authUserName, authPassword];

var authenticationObject = RESTAuthenticationManager.createAuthentication(RestAuthType, authParams);

// Create the Host Authentication

RestHost.authentication = authenticationObject;

//Create the Session

var CreateSession = RestHost.createRequest("POST", "/v1/session/Add", null).execute();

// Place the Response in a separate Variable to Access the Content

var SessionResponse = new Properties();

SessionResponse.put("statusCode", CreateSession.statusCode);

SessionResponse.put("headers", CreateSession.getAllHeaders());

if ( SessionResponse.statusCode == "200") {

System.log("Session successful created")

xZertoSessionKey = SessionResponse.headers.get("x-zerto-session")


else {

System.log("Session could not be established")


With this small script we establish our connection to the ZVM and extract the x-zerto-session key.

Know we create the next workflow. In my case the next Workflow I use the command “get peersites” to access the connections between both locations.  The command to do this is: “https://zvm_ip:port/v1/peersites” .  Also for this workflow we need some Inputs and Outputs.

Variable Name Type Value
Inputs RestHost REST:RESTHost
Inputs xZertoSessionKey String
Output getPeersitesOutput String

Also here we create a scripting element within the workflow.

We use the created Input as Inputs and the Output as output with the Scriptable Task.  This Script is used with the workflow:


//Authenticate the request with the Zerto x-session-header
var CreateSession = RestHost.createRequest("GET", "/v1/peersites", null)

// Build up the Session Header information

CreateSession.setHeader("x-zerto-session", xZertoSessionKey);

// The Output of the Command is catch in a variable

getPeersitesOutput = CreateSession.execute();

System.log ("ContentString " + getPeersitesOutput.contentAsString);

As last point we have to build up a third Workflow witch put both before created workflow together.

The Input and Outputs are created from the need within the Workflow created before.

Variable Name Type Value
Attribute xZertoSessionKey String
Attribute getPeersitesOutput String
Inputs RestHost REST:RESTHost
Inputs authUserName String
Inputs authPassword SecureString
Output xZertoSessionKey String

After you have finished everything you can execute the Workflow and see the Output of the command.

There are many more commands witch you can use to automate your Zerto installation. All commands are documented in the API Documents to which I pointed before.

So have fun and orchestrate the World ;-)


SRM and the planned migrations…..

Be aware, this post has nothing to do with Automation or Orchestration. This post is only related to the VMware Site Recovery Manager and a solution for a Problem during a planned migration. Maybe the post is useful for someone else which encounters the same problems…..

Last weekend I supported a customer witch had to power down one of his both datacenter. For this, we had to migrate the virtual Machines from one DC in the other. At the end all machines must be migrated back. From my site a view, this should be an easy thing because the customer had an SRM implementation. The storage is served via an IBM V7000 and the LUNs are replicated over both datacenter….. The customer had built the recovery Plans and tested them before the Migration should occur…

So from my point of view I expected an easy migration……

After everything was cleared the users were at home we started with a “Planned Migration” from the Datacenter 1 (DC1) to Datacenter 2 (DC2). This was quite easy and at the end we created our “Failback Plan” with the SRM.

No for us it was time to take a drink and wait for the Power and Air-conditions Guys to finish their jobs.

After a few hours it was time for us to migrate the VMs back to DC1…….

The customer created a recovery plan for his two different clusters. In the first cluster only the normal VMs were placed. In the second cluster the Database VMs were located…..

So we started the planned and the VMs out of the first Cluster fail back without any problems…..sincerely the VMs from the Database Cluster could be relocated……

We got the error:  No host with hardware version ’9′ and datastore ‘snap-ef732565ae’ which are powered on and not in maintenance mode are available….

So we checked the vSphere client…..the Hosts were online (Host Version 5.5) and the Datastore with the name ‘snap-ef732565ae’ was also present…..

Really strange……a quick search in the Web leads to this VMware Documentation (http://pubs.vmware.com/srm-55/index.jsp#com.vmware.srm.admin.doc/GUID-FE6A85EC-B44E-415A-9C5F-1E17BC846119.html) were the problem was described with the solution to wait 15 Minutes for the next try because the SRM had cached some old information. So we took a coffee break  and after 20 Minutes we started the next try……unfortunately we had the same problems……

I tried to figure out the problem in the logs but I could found anything what pointed to the error…..

So we tried a lot of things to finish our “Planned Migration”…..every try needed a lot of time……one of the last things we did was to restart all ESX Hosts of the DB Cluster…..after all Hosts were Online we did the next try and “voila” I worked……

In the last week I did a lot of research about this behavior.  I figured out that I could reproduce the error when I power-up the ESXi Server quickly without any delay.  So from my point of view it seems to be a “communication” problem between the ESX Hosts of a cluster.


So if anyone has the same problem….try a reboot of the ESX Hosts……

Page 1 of 2212345...1020...Last »