Workflow Design

0

Nutanix Calm – a deeper look from an automation consultant perspective (Part 2)

This post will be the second post on Nutanix Calm and will build up on my first post which can be found here: http://www.vcoportal.de/2018/09/nutanix-calm-a-deeper-look-from-an-automation-consultant-perspective-part-1/ if you missed it.

 

In the last post we only created a virtual machine without any customization. In this post we will install additional packages and add an SSH Key for login to the VM.

Let’s go back to our previously created Blueprint. On the left site we have the tab “Package”. Choose this, give the Package a Name and use “Configure install”

 

What you will see is that beside the LinuxServer Icon a new Icon will open up. Here we choose “+ Task” to create some modifications during the VM build process.

 

On the left site you must configure some values. You provide a name, choose what you want to do (I want to execute a Shell Script), and take the Credentials you need to access the VM. In the script section you past your script (Linux Bash programming in my case) which you want to execute.

My script for example looks like this:


#!/bin/bash

cat <<'EOF' > /etc/resolv.conf

# Generated by NetworkManager

search vcoportal.de

nameserver 192.168.1.10

nameserver 192.168.1.20

EOF

yum -y update

yum -y install net-tools traceroute bind-utils

First, I declare my script as bash script (line 1). Then I change the DNS Server configuration (line 2 to 7) and update everything on my VM and then install some tools like dig, traceroute and so on for my virtual machine.

When you are finished, save the Blueprint and give it a try for a run.

In the Audit Section of the Application you can take a look at your installation and configuration. As you will see the new generated section InstallSoftware will also be there as point. You can see what the Output of the section is. When you click on the Small window in the market red rectangle you can pop up a bigger window with the output.

On the button you can also have a look at the created script.

In the next step we will insert an SSH Key. This key can be insert from the user wo execute the Blueprint. So, let’s go back to our Blueprint. Here we go the the “Service” Section from our Service. There we insert a Name for the Variable (in my case SSHKey) and a default value. It is important that you activate the “Runtime” checkbox. We want the SSHKey auf our user, not the default and so the user can insert “his” key.

 

When we are finished with the variable we add a “Task”. This task we use to insert the SSH Key into the authorized_keys field for a specific account (I am lacy so I use the root account. For security reason I would not recommend this in production….)

 

The script itself is also a Linux Bash script and easy.


#!/bin/bash

mkdir /root/.ssh

touch /root/.ssh/authorized_keys

echo "@@{SSHKey}@@" >> /root/.ssh/authorized_keys


 

In the first line we declare the script as a Linux Bash script. The second line create the necessary directory and the third line creates the file. In the last line we insert something in the create file……

Did you recommend the “@@{SSHKey}@@ in the last line? What’s that for a strange value?
In Calm variables which we use in script are market with “@@{VALUE}@@” so in our case that means we take the value of the before created Variable and replace the “@@{SSHKey}@@” field with the content.

One note why I used a new task and not placed the script in the first task we already did. From my point of view, I would recommend to create a new task for every goal which I want to reach. I personally thing that debugging is much easier in this case because we can see what went wrong (in which script). The next thing for me is, that even on script maybe doesn’t function like expected we still have the chance to access the VM and fix it during life time.

After we are done with our script, save the Blueprint and Launch the VM. When we are on the Blueprint just open the “Configuration” part and insert the SSH Key which you want to access the VM without password.

 

When everything works like expected you can login in your VM via SSH. If not, take a look in the Audit section.

As you can see the start with calm is very easy and there are a lot of things we can be done with calm.

During my “play around” with Calm I found some things which must be improved within Calm or limit Calm at the moment:

  • Calm is limited to Nutanix, vSphere-Cluster and Cloud Solutions. So, some customers still have to use a second solution for their automation need.
  • You cannot export created Blueprint and import them with all needed scripts, variables and images. With our easy Blueprint that is not a big issue but when you create a Blueprint with plenty of variables there is a “room” for mistakes and typos.
  • At the moment Calm has some limitations to work with for example ServiceNow and so on.

Some Ideas for Calm for future functions / extension:

  1. RFE: Blueprint Full Backup with images, scripts, cloudinit etc. à Provide a central repository in PC to make things re-usable: cloudinit, sysprep, scripts, images etc.
  1. RFE: DNS-Update Feature when VXLAN-DHCP from PE assigns IP-address à Autofill connection URL after application deployment (aka openshift) in the app description
  2. RFE: Integration of 3rd-Party config-mgmt tools like git, puppet, chef etc. in the GUI (dropdown menu, not via scripts) à Make existing scripts / tools / 3rd-party development re-useable
  3. RFE: Must-have: Native container integration to push CALM! No customer with an existing automation or configuration solution will switch to CALM without that feature.
  4. RFE: VM customization options with cloudinit for linux and sysprep for windows delivered as one-click/dropdown options in CALM (forgot that in the call)

 

If you can life with this limitations Calm is a good choice to start automation in your Nutanix environment.

0

Automate your Zerto Disaster Recovery solution with vCO

Today I show you how to automate your Zerto installation with the vCO. Or those which are not familiar with Zerto here a quote from the Zerto Website:

What is Zerto?

Zerto provides enterprise-class business continuity and disaster recovery (BCDR) solutions for virtualized infrastructure and cloud. [……]  Zerto Virtual Replication, is the industry’s first hypervisor-based replication solution for tier-one applications. Zerto Disaster Recovery solutions replace traditional array-based BCDR that was not built to deal with virtual environments.”

So for those which know the VMware Site Recovery Manager, Zerto is an alternative Product to the SRM.

For a customer Self Provisoning Portal I had to  automate the deployment of the virtual Machines within the Disaster Recovery Solution.  In my case this Automation must be archived with the VMware Orchestrator.

Zerto offers a limited documentation of the API which can be accessed with a REST API. The documentation can be found here: http://downloads.zerto.com/documentation_4acccec86a1e7cbf77f8e6bd3831c31c/3.1/Zerto%20Virtual%20Replication%20REST%20APIs%20Online%20Help/index.html

With this documentation I started my integration.

First thing what we need to do to integrate Zerto is to add the Zerto Hosts as REST Clients in the vCO Server. This integration is done over a vCO Workflow.

If have documented this before in another post (http://www.vcoportal.de/2014/02/automating-veeam-with-vco-and-the-restful-api/) so I will only point to the parameter which are needed.

For the configuration, you give a name, have to provide the URL to the ZVM Machine with the configured Port (the default port is 9669) and a Connection and Operation timeout. Please use for the connection a HTTPS connection. HTTP is possible but I had a lot of problems that the vCO run into timeouts.

We do not need a proxy Server.

As Authentication we use the “Basic” Authentication Type.

I used a “Shared Session” for the configuration but Single Sessions are also possible.

When you “Submit” the Workflow you will get a Certificate error.

Install the Certificate and go ahead.

After we have connected your Zerto REST Connection, we have to create our API connection. The initial request we must made is to make a call to the API. This call is done with a username and password for permissions to the vCenter Server. Here it is important, that the Username and the Password are for a User with permissions vCenter Server. The Zerto REST connection response includes an “x-zerto-session” header. The String in this Header is used for any future communication with the ZVM (Zerto Virtual Manager) and the vCenter Server. All other ways doesn’t function!


First thing we have to do is to create the initial request. For the request this URL is used: “https://zvm_ip:port/v1/“. I place this initial request in an own workflow. For the workflow we need some Attributes, Input and Output Variables.

Variable Name Type Value
Attribute sessionMode String Shared Session
Attribute RestAuthType String Basic
Inputs RestHost REST:RESTHost
Inputs authUserName String
Inputs authPassword SecureString
Output xZertoSessionKey String

In the Workflow we create a Script Element.

In this scripting element we include all Attributes and Inputs as Input and the Output element as Output for the Workflow.

Then we create our Script:


// we must use "dedicated" User Credentials. With the Provided Credentials we need Access to the VC!

var authParams = [sessionMode, authUserName, authPassword];

var authenticationObject = RESTAuthenticationManager.createAuthentication(RestAuthType, authParams);

// Create the Host Authentication

RestHost.authentication = authenticationObject;

//Create the Session

var CreateSession = RestHost.createRequest("POST", "/v1/session/Add", null).execute();

// Place the Response in a separate Variable to Access the Content

var SessionResponse = new Properties();

SessionResponse.put("statusCode", CreateSession.statusCode);

SessionResponse.put("headers", CreateSession.getAllHeaders());

if ( SessionResponse.statusCode == "200") {

System.log("Session successful created")

xZertoSessionKey = SessionResponse.headers.get("x-zerto-session")

}

else {

System.log("Session could not be established")

}

With this small script we establish our connection to the ZVM and extract the x-zerto-session key.

Know we create the next workflow. In my case the next Workflow I use the command “get peersites” to access the connections between both locations.  The command to do this is: “https://zvm_ip:port/v1/peersites” .  Also for this workflow we need some Inputs and Outputs.

Variable Name Type Value
Inputs RestHost REST:RESTHost
Inputs xZertoSessionKey String
Output getPeersitesOutput String

Also here we create a scripting element within the workflow.

We use the created Input as Inputs and the Output as output with the Scriptable Task.  This Script is used with the workflow:

 


//Authenticate the request with the Zerto x-session-header
var CreateSession = RestHost.createRequest("GET", "/v1/peersites", null)

// Build up the Session Header information

CreateSession.setHeader("x-zerto-session", xZertoSessionKey);

// The Output of the Command is catch in a variable

getPeersitesOutput = CreateSession.execute();

System.log ("ContentString " + getPeersitesOutput.contentAsString);

As last point we have to build up a third Workflow witch put both before created workflow together.

The Input and Outputs are created from the need within the Workflow created before.

Variable Name Type Value
Attribute xZertoSessionKey String
Attribute getPeersitesOutput String
Inputs RestHost REST:RESTHost
Inputs authUserName String
Inputs authPassword SecureString
Output xZertoSessionKey String

After you have finished everything you can execute the Workflow and see the Output of the command.

There are many more commands witch you can use to automate your Zerto installation. All commands are documented in the API Documents to which I pointed before.

So have fun and orchestrate the World 😉

0

New vCO video available!

Brian Watrous, one of VMware’s most senior technical trainers (you know him from the videos on www.vmwarelearning.com/orchestrator, don’t you…), has published a new video about how to customize workflow icons in vCenter Orchestrator. For your convenience, I embed the youtube video below. Make sure you subscribe to Brian’s blog on www.vvork.info and his youtube channel, there is more to come…

Great work, Brian! Keep them coming! 🙂

0

vCO and Veeam Backup&Replication a powerful combination

Last week I did a Webinar for Veeam in Germany. My topic in this webinar was Automation and Orchestration. Due the circumstance that the Webinar was in German, I decided to make this post to share the information for the rest of the non-German speaking world.

For those who understand German the Webinar was recorded and can be found here:

http://www.veeam.com/de/videos.html?ad=de-topmenu

Before you start some really important notes on the combination of vCO and Veeam B&R.

Some Veeam B&R commands need a connection to the vCenter Server. When you invoke your commands in a PowerShell Window on your Backup host, the PowerShell uses CredSSP to provide the vCenter Server Login Information’s from Veeam B&R to the vCenter Server. If you do the same in a vCO Workflow, this does not work! The reason why it not work is because the vCO PowerShell plugin only Supports Basic- and Kerberos Authentication. In every Environment I used so far, the Servers where in a Windows AD. This allowed me to use the Kerberos Authentication in the vCO PowerShell Plugin. In the last time, I did many tests with the Basic Authentication and had a lot of problems and errors with that type of Authentication. So my recommendation for the Authentication is “use the Kerberos Authentication to avoid a lot of trouble and problems!”

Prepare the Backup Server

At the moment Veeam Backup&Replication has no SOAP or REST API Interface. The only available interface is PowerShell.  To use the Power Shell from vCO some necessary preparations has to be done.

First of all Veeam Backup&Replication must be installed with the PowerShell Extension. This is done during Installation or if you already installed it without PowerShell  to just start the Installation again and add the PowerShell feature.

After you have installed the PowerShell Extension, you can start it from the Management Console.

This Button starts a PowerShell shell with an already loaded Veeam Extension. The Files for this Veeam PowerShell Extensions reside here:  “C:\Program Files\Veeam\Backup and Replication” in this path the file ”Install-VeeamToolkit.ps1” is important to load the extension automatically. We will use this file later in our vCO Workflows.

The next we have to do is to check if the Veeam Backup Server has the PowerShell in the Version 3.

For the first workflows and test I recommend to change the Host execution Policy to unrestricted. When everything goes fine, you can change the execution Policy to remote-signed

Set ExecutionPolicy (RemoteSigned / Unrestricted )

After that, we need a command Window on the Backup server. Here we have to insert the following commands:

Run the following command to set the default WinRM configuration values.


c:\> winrm quickconfig

(Optional) Run the following command on the WinRM service to check whether a listener is running, and verify the default ports.

c:\> winrm e winrm/config/listener The default ports are 5985 for HTTP, and 5986 for HTTPS.

Enable basic authentication on the WinRM service.

Run the following command to check whether basic authentication is allowed.

c:\> winrm get winrm/config

Run the following command to enable basic authentication.

c:\> winrm set winrm/config/service/auth @{Basic="true"}

Run the following command to allow transfer of unencrypted data on the WinRM service.

c:\> winrm set winrm/config/service @{AllowUnencrypted="true"}

Enable basic authentication on the WinRM client.

Run the following command to check whether basic authentication is allowed.

c:\> winrm get winrm/config

Run the following command to enable basic authentication.

c:\> winrm set winrm/config/service/auth @{Basic="true"}

Run the following command to allow transfer of unencrypted data on the WinRM client.

c:\> winrm set winrm/config/client @{AllowUnencrypted="true"}

Run the following command to enable winrm connections from vCO host.

c:\> winrm set winrm/config/client @{TrustedHosts ="vco_host"}

After we have executed the commands, we are ready with the Backup Server. Let’s now switch to the vCO Server.

Prepare the vCO

From the view of the vCO the first and important thing is, that the PowerShell Plugin is installed and activated in the vCO Server. If you are not familiar with this, the documentation can be found here:

http://pubs.vmware.com/orchestrator-plugins/index.jsp?topic=/com.vmware.using.powershell.plugin.doc_10/GUID-8AE1CFF2-F6F0-4233-BDD9-F318E461AB2F.html

When the PowerShell Plugin is ready, we can start to add the Backup Server to our repository. This could be done be starting the PowerShell Workflow to “Add a new Server”. The needed information’s are self-explaining.

On the second site we have to choose as PowerShell remote host type “WinRM”. As Protocol we use “HTTP” or “HTTPS”. The last point is Authentication. Here we choose “Kerberos”.

On the last page we have to choose if we use a “Shared session” or a “User Session”. When you chose the shared session you have to insert User credentials. When you decide to use “User Session” you have to insert the Authentication Details in every PowerShell call.

After we are finished with the pre requirements we can start with our first Workflow. Let’s us a simple one…..

Develop the vCO Workflows

If we want to figure out which Veeam Jobs exist on our Backup Server we need the command Get-VBRJob.

The easiest way to start is to copy the Workflow “Invoke a PowerShell Script” into a folder of your choice.

There you have to insert a second scripting element and move the host and script Inputs as Attributes.

In this scripting element we put our script which includes the PowerShell code.

To use a Veeam PowerShell command in a vCO Workflow we need somewhat more input then just the command. We have to load the Veeam Extension into our PowerShell Session which we invoke from the vCO Server. Here is the complete code for the call:

script = "# Load Veeam Powershell Extension into the actual session \n"
+ "'C:/\Program Files/\Veeam/\Backup and Replication/\Install-VeeamToolkit.ps1' \n"
+ "add-pssnapin VeeamPSSnapin \n"
+ "# Veeam is loaded \n"
+ "Get-VBRJob";

For us the full example looks like this.

Now you can use this command in your own workflows. Now that command isn’t really useful by now. Let insert a virtual machine into a Backup Job after creation. For that we have to use the Veeam Command Add-VBRJobObject. For this command we need some information which we can collect during the Session. A full command to insert a VM into a workflow looks like this:

Add-VBRJobObject -Job $(get-VBRjob -Name "+ JOBNAME+ ") -Server $(get-VBRServer| Where {$_.Type -eq 'VC'}) -Objects " + VMNAME + " }"

The Values JOBNAME and VMNAME must be specified as vCO Attributes or Inputs.

When you now try to execute this like the command before:

You will get an error like this one:

Failed to login to “vcenter.example.com” by SOAP, port 443, user „root”, proxy srv: port:0 +   CategoryInfo : InvalidOperation: (Veeam.Backup.Po…FindVBRViEntity:FindVBRViEntity)         [Find-VBRViEntity], Exception + FullyQualifiedErrorId :   Backup,Veeam.Backup.PowerShell.Command.FindVBRViEntity

Why this happens?  Here we get into trouble with the Authentication against the vCenter Server. If everything was fine before and you can execute the command from a PowerShell shell the problem is in your workflow. Like described before we have to Authenticate against the vCenter Server from our Workflow.  vCO has no option to do this automatically . We have to change our Workflow to this:

 

script = "invoke-command -session $(New-PSSession <strong>BACKUPSERVER</strong> -Authentication Kerberos -Credential $(new-object -typename System.Management.Automation.PSCredential -argumentlist<strong> USER@DOMAIN</strong>, $(convertto-securestring -string '<strong>PASSWORD</strong>' -asplaintext -force))) -scriptblock{ set-item wsman:localhost\Shell\MaxMemoryPerShellMB 1024"
      + "\n Add-PSSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue"
      + "\n Add-VBRJobObject -Job $(get-VBRjob -Name "+ <strong>JOBNAME</strong> + ") -Server $(get-VBRServer| Where {$_.Type -eq 'VC'}) -Objects " + VMNAME + " }"

This script looks really different then the script before. What do we do here? We generate a new Powershell session on the Backup Server (New-PSSession).  For this session, we define a Username (USER@Domain) and a Passwort (PASSWORD). For the Username it is very important that the user is written as user@domain. Otherwise the Kerberos Authentication will not work and the Workflow will fail! At last we set the Memory for the new Shell to 1024 MB (set-item wsman:localhost\Shell\MaxMemoryPerShellMB 1024) If we doesn’t exceed the Memory the workflow will also fail! At last we load the Veeam Snapin and execute the Script job…..

That’s easy or?

With this Background Knowledge you can start to implement your own Automation Workflows with included Backup of your virtual machines with Veeam. It is also possible to integrate the Replication….you have just to implement the replication command and start your Automation….

In the Veeam Community there is a good PowerShell forum. So if you have trouble with your Veeam PowerShell commands, get a look there:

http://forums.veeam.com/viewforum.php?f=26

Have fun with the Power of vCO 😉

0

Safe your job, create unmaintainable workflows!

The need for unmaintainable workflows

As an external consultant as well as permanent employee it’s always a good idea to make yourself indispensable for your customer/company. This ensures ongoing engagement/a safe job.

In IT in general you can achieve this by devloping unique solutions nobody else except yourself can understand (and maintain). I’m sure: You all can instantly give at least one example for an IT-component where youself/your teammates/everybody else follow the policy: “Never touch a running system“.

 <buzzword-for-the-robots>The modern term for this is NoOps.</buzzword-for-the-robots>

 As a workflow developer you also can follow some simple rules to make sure: You, and only you, will be able to maintain, re-use or even exand your workflows!

Here we go…:

Documentation, Labeling, Description Fields

Avoid Documentation, and use the default labels for workflow elements. So one has really click into each element, read the code, see the bindings to figure out what happens here.


When naming workflows and actions, and their parameters and attributes: Don’t be too obvious! A simple “name” or “vm” or “theArray” is enough.

Be aware of the Description fields! They are shown in so  many different places (when you re-use the elements, mouse-over in the schema, for documentation, …), that your really want to avoid this disclosure of “what this element is for”.

Logging

For the usage of System.log(), System.debug() and co. you have the choice:

  • Don’t log at all! Your NoOps-Team will thank you for that.
  • Log everything! It’s always a good challenge for the NoOps-Team to find the really important messages in hundrets of lines with traced-out variables (Eastern is coming!)…
  • Just log values! The value of a variable is what’s interesting. Nobody wants to know what variable it is… System.log(myVariable) does the trick.
  • Pro-Tip: You can even improve confusion by logging WRONG things, e.g. when you copy&paste scripts to re-use it in another context.

Workflow Design

Especially important for Library-Workflows which are intended to be called by other Workflows:
Hardcoding values in the script, not exposing important parameters as INPUT-parameters and  creative usage of  the data types ANY and Properties (of course whithout proper documentation) will ensure that it’s you, and only you, who can re-use the workflow.

In General:
Use the default namesactionResult” and “errorCode” when binding Action elements and Exception paths…

Forget about the “Add note…”-Feature! Structuring workflow part with coloured background just creates a better visual structure for the workflow.

Workflow Organization

Duplicate Workflows for small adjustments! This helps to keep your work big if something has to change in there for the next version.

Don’t use too many Workflow Folders, or clear labes for them! Leave it to the user to figure out which workflows are intended to be called manually, and which ones are just the “drivers” for them.

Create hidden dependencies! Calling Actions or nested Workflows not via the corresponding Elements, but within scriptable tasks creates nice surprises if one is changing the called.

Optimize your JavaScript coding style

You can find a long list of tips for that here: http://thc.org/root/phun/unmaintain.html

Forget vCO, just do scripting!

Who needs the visual, self-explaning graphical flowcharts of vCenter Orchestrator when you can to something like this??:

Pro-Tip: Use the PowerShell-Plugin or the PowerSSHell-Plugin to combine the best of both worlds. This will garantuee your job forever!

:mrgreen: HAPPY APRIL FOOLS’ DAY!!! :mrgreen:

Special Greetings to my mates in Sofia 😎 : Please do not take that criticism of some parts of the vCenter-Plugin-Library personal! I know it’s an hard challenge to maintain this inherited aged content 🙄 . You do a great job, especially with all the new “written-from-scratch” Plugins! Keep them coming!!! 😀

Got more?

Comment on the post and share your finest ideas  how to create poor workflows! 😈