Automating Veeam with vCO and the Restful API

Do some of you remember my Blog Post about automating Veeam with vCO and Powershell (http://www.vcoportal.de/2013/04/vco-and-veeam-backupreplication-a-powerful-combination/)? The problem at this time was that Veeam only offers a PowerShell API. Now, some time is gone and Veeam released the Version 7 of Backup&Replication.  In this Version, Veeam offers an Enterprise Manager which offers a Restful API for automating.  Please be aware that the Restful API is only available in the Enterprise Plus Version of Veeam B&R.  The Veeam Restful API can be accessed over this URLs and ports:

Veeam Restful API HTTP URL  http://<Enterprise-Manager>:9399/api/

Veeam Restful API HTTPS URL https://<Enterprise-Manager>:9398/api/  

A comparison can be found here: http://www.veeam.com/backup-version-standard-enterprise-editions-comparison.html So, let’s start with the integration. First think what we have to do is to connect the vCO and the Veeam B&R server. Therefore we need the vCO Rest Plugin which must be installed over the vCO Plugin page. Details how to do install a plugin can be found in the vCO Documentation for the Plugin (http://pubs.vmware.com/orchestrator-plugins/index.jsp?topic=/com.vmware.using.http_rest.plugin.doc_10/GUID-D303E545-0BFE-4EEB-BAC3-7776EFCDB516.html)

After we have installed the Plugin, we can connect the vCO and the Veeam Server. Therefore we have a build in Workflow “Add a Rest Host”

In the Workflow we have to give our Restful Host a Name and have to define the URL. The settings for the Connection and Operation timeout doesn’t have to be changed.

The Authentication for the B&R Server has to set to Basic

The next thing is to chose a Session mode. You can chose between a “Shared Session” and a “per User Session”. Which you chose depends on your security settings. I will take a shared Session in the Screen-Shots. For the shared session you have to define a user and a password.

If you use a HTTPS connection, you have to accept the SSL Certificate for the B&R Server.

At the end you start the Workflow and if everything goes well, your Workflow finished without any problems.


The next thing you can do, is to import the Veeam Rest Schema. The Schema is located in the Path “C:\Program Files\Veeam\Backup and Replication\Enterprise Manager\schemas\RestAPI.xsd” on you B&R Server. To import the file, you have to transfer it to a Webserver so that the vCO Server can import the schema. To start the import we start the Workflow “Add schema to REST Host”. 

In the Workflow, we have to change the REST Host to which we want import the schema.

In the next screen we have to define the Web URL were the schema file is located.

If everything goes well, your workflow finished without any error and the Schema is available under the Rest Host.  

During my test, I had some problems with the Schema. So I decided to implement my task manual…… To implement the Veeam Restful API functions manually, the Veeam documentation is required. The documentation can be found here: http://helpcenter.veeam.com/backup/70/rest/overview.html

First thing which we have to do is to open a Restful API Connection from the vCO Server to the B&R Server. To connect the API two commands are used.

The first command is:

GET http://EnterpriseManager:9399/api/

and the second is

POST http://EnterpriseManager:9399/api/sessionMngr/

To send the commands to the Enterprise Manager you need authentication. Do the facts, that we configured our vCO Server for authentication, vCO handle this Authentication for us. Now, let’s post this request into a vCO Workflow. Therefore we create a new Workflow. For the Workflow we need one Input Parameter from Type REST:RestHost

In the Workflow itself we place a “Scriptable task” to insert our Code.

In this “Scriptable Task” we include this code:

var PostResponse = BRHost.createRequest("POST", "/api/sessionMngr/", null).execute();

System.log("Connection Successful:  " + PostResponse.contentAsString);

So, what does this code do? In the first line, we open the connection to the Enterprise Manager. The Authentication is done from the vCO Server (the provided Credentials during the Rest Host connection). The Second line just gives us the response of the Connections and the available things we can make. Now let’s expand this Workflow and let us see which Backup Jobs are implemented.

var PostResponse = BRHost.createRequest("POST", "/api/sessionMngr/", null).execute();

var BackupResponse = BRHost.createRequest("GET", "api/jobs?type=job", null).execute();

System.log("Backup Jobs " + BackupResponse.contentAsString);

The interesting Part here is the second line witch gets the Backups Jobs from the Server:

Backup Jobs <?xml version=”1.0″ encoding=”utf-8″?><EntityReferences xmlns=”http://www.veeam.com/ent/v1.0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”><Ref UID=”urn:veeam:Job:a8f5b929-c5c8-4901-b749-d8c7ea9b8462″ Name=”Job1″ Href=”″ Type=”JobReference”><Links><Link Href=”″ Name=”″ Type=”BackupServerReference” Rel=”Up”/><Link Href=”” Name=”Job1″ Type=”Job” Rel=”Alternate”/><Link Href=”” Type=”BackupJobSessionReferenceList” Rel=”Down”/></Links></Ref></EntityReferences>

As we can see we have a Backup Job with the name “Job1”. Now let’s have a look with VMs are included into the Job. First of all, we need a Input parameter for the Jobname (we will use this to filter the results…)

After we are finished we will extend our code:

var PostResponse = BRHost.createRequest("POST", "/api/sessionMngr/", null).execute();
var BackupResponse = BRHost.createRequest("GET", "api/jobs?type=job", null).execute();
var XMLFile = XMLManager.fromString(BackupResponse.contentAsString);
var XMLelement = XMLFile.documentElement.getElementsByTagName("Ref");

 var BackupJob = XMLelement.item(i).getAttribute("Name")
if ( BackupJob == BackupJobName){

 var UIDBackup = XMLelement.item(i).getAttribute("UID")
 var UIDBackupJob = UIDBackup.substring(14,50)
 var VMResponse = BRHost.createRequest("GET", "/api/jobs/" + UIDBackupJob + "/includes", null).execute();

System.log("VMs included: " + VMResponse.contentAsString);

When we make a call to the Restful API we become a XML Response. We call the XMLManager and place the content into the variable XMLFile. On the next row we filter the XML response for the Word “Ref. Within the “Ref” response we search with a loop for the Attribute “Name” and place the response in  a variable. Then we check this name against the Backup Job name an extract the unique identifier for the Job. With this unique identifier we mal a job request and get the list of the included VMs.

The answer from the call is this:

[2014-02-09 23:15:06.281] [I] VMs included: <?xml version=”1.0″ encoding=”utf-8″?><ObjectsInJob xmlns=”http://www.veeam.com/ent/v1.0″ xmlns:xsd=”http://www.w3.org/2001/XMLSchema” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”><ObjectInJob Href=”″ Type=”ObjectInJob”><Links><Link Href=”″ Name=”WinXP2″ Type=”ObjectInJob” Rel=”Delete”/><Link Href=”” Name=”Job1″ Type=”Job” Rel=”Up”/></Links><ObjectInJobId>b39bdbf3-8333-4525-b393-110ef3ea9332</ObjectInJobId><HierarchyObjRef>urn:VMware:Vm:ea294886-2e12-419b-934f-70b371e0f746.vm-15</HierarchyObjRef><Name>WinXP2</Name><DisplayName>WinXP2</DisplayName><Order>0</Order><GuestProcessingOptions><AppAwareProcessingMode>RequireSuccess</AppAwareProcessingMode><FileSystemIndexingMode>ExceptSpecifiedFolders</FileSystemIndexingMode><IncludedIndexingFolders/><ExcludedIndexingFolders><Path>%windir%</Path><Path>%ProgramFiles%</Path><Path>%TEMP%</Path></ExcludedIndexingFolders><CredentialsId/></GuestProcessingOptions></ObjectInJob></ObjectsInJob>

So with the techniques and methods shows in the post you are able to do a lot more interesting stuff with vCO and Veeam Enterprise Manager. A complete list about all the Enterprise Manager Restful API commands can be found here: http://helpcenter.veeam.com/backup/70/rest/em_web_api_reference.html Have fun with our automation and orchestrate the World ;-)


Introduction to vCenter Orchestrator #vBrownbag podcast

Last night (literally) I had the honor to present an introduction to vCO as part of the #vBrownbag “Automate all the Things” community podcast series. See the recording below, more information and future agenda on http://professionalvmware.com/vbrownbag-automate-all-the-things-training-schedule/


VMware vCenter Server Heartbeat – Restore on a second node….a journey….

Warning: The following post has absolutely nothing to do with automation or Orchestration! Be aware of this while reading the post ;-)

In the last days I had a job for a customer who wanted to implement vCenter Server Heartbeat. For those of you, which are not familiar with vCSHB here is the description from the VMware Website:

“VMware® vCenter™ Server Heartbeat™ protects VMware® vCenter Server™ virtual infrastructure from problems related to applications, configurations, operating systems, networks and hardware.”

If I had to say it I would bring it to the words: vCSHB is a cluster service for different VMware products like the vCenter Server (including the SQL Server if installed on the same Server), VMware Composer and a lot more…..

As picture vCSHB looks like this:

The vCSHB is based on Neverfail with tuning for VMware…… With vCSHB you can build different “models” to protect your vCenter Server. This could be a physical to virtual model or also a physical to physical model. There are many amore options to build your application HA with vCSHB….. Most implementations that I have done so far, were physical to virtual implementations….this model is relative simple to implement and you get the vCHSB easily up and running…..

Now let’s come back to the customer project…. The customer wanted to implement a physical to physical implementation. The Servers where located on a remote site so that’s mean for as that we didn’t have a chance to get a hand on the server. The only possibility for us to work with the server is to log in to the Windows system which was installed on the server via RDP or use the remote connection board.

For the customer we started with the primary server. First thing you have to do, beside the installation of Windows, is the installation of the VMware vCenter Server, a SQL Server and the other VMware Products you want to protect via vCSHB. After you are finished with the installation, you can start with the installation of vCHB. We also did it in that way and everything did it fine. During the vCSHB installation, a Backup of the Primary system ( the system were you installed the vCSHB first) is taken. After the first node is installed, it is necessary to install vCSHB on the second note. The only “prerequisite” on the second node is that you need an installed Windows with the Windows Backup installed. The wording installation is properly wrong in this context because you start the installation and then you use the Backup of the first node.

Here began the Journey with the installation…….

On our first try, the installation failed with an error. The Log provided these information:

Log of files for which recovery failed:


wbadmin 1.0 – Backup command-line tool

(C) Copyright 2012 Microsoft Corporation. All rights reserved.

Starting a system state recovery operation [08.11.2013 14:17].

Processing files for recovery. This might take a few minutes…

Processed (176) files.

Processed (1657) files.

Processed (19373) files.

Processed (27841) files.

Processed (47849) files.

Processed (53873) files.

Processed (74813) files.

Processed (97831) files.

Processed (120041) files.

Processed (120041) files.

Processed (120041) files.

Summary of the recovery operation:


The recovery of the system state failed [29.11.2013 14:19].

Log of files successfully recovered:


Log of files for which recovery failed: C:\Windows\Logs\WindowsServerBackup\SystemStateRestore_Error-29-11-2013_13-17-40.log

Access is denied.

An exception occurred:

Message: Execution failed with return code -3. Restore was aborted.

See vCSHB-Ref-2273 for resolution procedure.



wbadmin start systemstaterecovery -backupTarget:”\\Server\share” -version:29/08/2013-11:51 -quiet

The installation cannot continue.

The most interesting Part of the Log was the pointing to the vCSHB reference 2273 for resolution procedure. The Links to an VMware KB Article which can be found here:


Depending on the Information provided in the KB article and the Batch file, we tried a second restore. Also this second restore ends up in the same error message……after a lot of searching we didn’t find any suitable answer for our problem……so we still had to try to solve the problem….

The next we tried was to reinstall the second node and started the Batch file before we tried a next restore…..again we didn’t end in a successful story…..

So we decided to open a VMware SR. We explained our problem and the things we did before…..sadly the VMware Support could help us so much because his only possible solution was the KB article. These was the answer from the support…..

If the steps in the KB do not work then there is no other workaround that we have the problem here is heartbeat using the underlining MS technology to do the backup and clone so if this is failing there is nothing I can do from the VMware side Also he couldn’t help us directly, we could use the information “underlining MS technology”……

So again we started to search but this time we focused on the Microsoft topics for Backup and restore. In the Microsoft forums we found some information from other users which had problems to restore a backup on a physical node witch was made with MS Backup technology. The could restore there nodes if they:

- Deactivated the public (primary) Network Connection

- Didn’t use RDP to connect to the server

- Use the private (second) network connection to access the Backup file

We gave these tips a chance and could install the Backup on the second node……yes :-) ….. After that restore was successfully the rest was easy going and we could implement, configure and test the heartbeat installation.

From a discussion with a other vExpert, Mike Schubert, I know that it is possible to restore the vCSHB also with “RAID” Copy (take a Disk from the first node and place it in a second node….)with some manual work…. Hope you can use this information if you get in trouble during your Heartbeat installation….so habe fun a Orchestrate the World ;-)


Update 09.02.1014

For the German speaking people there is another way provided by Mike Schubert to restore the Heartbeat installation on the second node. Mike’s article can be found here: http://www.die-schubis.de/doku.php/vmware:heartbeat#physical_to_physical_-_ein_anderer_weg

Mike way is to “copy” the Installation via a raid copy with some additional manual task.


What’s the future of WaveMaker? And: WaveMaker book available

Earlier this year WaveMaker, our preferred tool to built custom web frontends for vCenter Orchestrator, moved from VMware to Cloudjee Inc, a spin-off of Pramati Technologies (http://dev.wavemaker.com/blog/2013/05/30/cloudjee-inc/). In September the new owner hold a nice webex about their future plans with WaveMaker. See for yourself:

These are really exciting plans, looking forward to give the new mobile features a try!

And more good news: Ed Callahan has published a book about “Easy Web-Development with WaveMaker”, which I absolutely recommend you to have a look at! http://www.packtpub.com/easy-web-development-with-wavemaker/book 1783OS Easy Web Development with WaveMaker 0


New vCO video available!

Brian Watrous, one of VMware’s most senior technical trainers (you know him from the videos on www.vmwarelearning.com/orchestrator, don’t you…), has published a new video about how to customize workflow icons in vCenter Orchestrator. For your convenience, I embed the youtube video below. Make sure you subscribe to Brian’s blog on www.vvork.info and his youtube channel, there is more to come…

Great work, Brian! Keep them coming! :-)

Page 1 of 2112345...1020...Last »