Best-Practices

0

VMware vCloud Director and Cassandra DB

By
Christian Strijbos
– September 10, 2018Posted in: Best-Practices, Examples, vCloud Director

In the last week I had a customer appointment where we made an update of the VMware vCloud Director installation to the vCD Version 9.1. For this update we had to make a couple of steps and also a migration of the used MS SQL Database. Everything went smooth to the new versions and then des customer decided that he wanted to use the VM Performance monitoring for the vCD Customers.
The installation and configuration is documented in the VMware vCD Docs (https://docs.vmware.com/en/vCloud-Director/9.1/com.vmware.vcloud.admin.doc/GUID-55D8360A-B4F9-4CDC-8BDA-972C4F387D74.html). So, we started with the installation of the Cassandra cluster and the configuration of the vCD. After everything was configured the customers chose the option to not provide all performance metrics to the customers. Therefore you can limit the metric collection (https://docs.vmware.com/en/vCloud-Director/9.1/com.vmware.vcloud.admin.doc/GUID-983C566A-7DE5-4F06-9044-C979F8AE7C99.html). We created the /tmp/metrics.groovy file with the required input:


configuration {
metric("cpu.usage.average")
metric("cpu.usagemhz.average")
metric("cpu.usage.maximum")
metric("disk.used.latest") {
currentInterval=300
historicInterval=300
entity="VM"
instance=""
minReportingInterval=1800
aggregator="AVERAGE"
}
}

And after that we imported everything and everything looked good.
After a last restart of a cell we ran into an error on the cell:


2018-08-23 15:00:05,027 | INFO | Cell Application | AbstractCellApplication | Application startup event: Application startup complete. |
2018-08-23 15:00:05,027 | DEBUG | Cell Application | LifecycleStateManager | Transition complete: com.vmware.vcloud.common.main.LegacyCellApplication@7bac686b transition from STOPPED to STARTED via STARTING_FROM_STOPPED |
2018-08-23 15:00:05,132 | INFO | OSGI Delegator - listener dispatcher | BootstrapApplication | Cell startup completed in 1m 35s |
2018-08-23 15:00:05,381 | DEBUG | Spring Context: com.vmware.vcloud.ui-vcloud-webapp | OsgiBundleSpringContextFactory | Finished constructing ApplicationContext for Bundle: com.vmware.vcloud.ui-vcloud-webapp in 3 seconds |
2018-08-23 15:00:58,666 | DEBUG | ell DiscoveryAgent listener: 484a81d5-738e-4c08-9f17-e013e8f98aab:Thread-94 | CellDiscoveryAgent | ADDING Cell to Broker Network. Cell UUID 810fe296-828e-4ead-8354-e9209f1ef044, Broker URI: tcp://172.16.6.105:61616 |
2018-08-23 15:00:58,802 | DEBUG | .105:61616@48618, localBroker= vm://484a81d5-738e-4c08-9f17-e013e8f98aab#16 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:01:53,538 | DEBUG | ActiveMQ Transport: tcp:///172.16.6.105:36820@61616 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:01:58,672 | DEBUG | ell DiscoveryAgent listener: 484a81d5-738e-4c08-9f17-e013e8f98aab:Thread-94 | CellDiscoveryAgent | ADDING Cell to Broker Network. Cell UUID 6bc3326d-df7b-4f37-8d59-cd76f7394d47, Broker URI: tcp://172.16.6.106:61616 |
2018-08-23 15:01:58,704 | DEBUG | .106:61616@34410, localBroker= vm://484a81d5-738e-4c08-9f17-e013e8f98aab#18 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 15:02:01,647 | DEBUG | ActiveMQ Transport: tcp:///172.16.6.106:59120@61616 | CellAuthenticatedBrokerFactory | Successfully authenticated user |
2018-08-23 17:53:02,838 | WARN | processor-NetworkingWeb | DatabaseConfiguration | Internal error |
java.sql.SQLException: SHUTDOWN ist in Bearbeitung.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:477)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.apache.commons.configuration.DatabaseConfiguration.getProperty(DatabaseConfiguration.java:177)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:264)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:282)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimWithCheckForExpectedExceptions(ActivityQueueServiceImpl.java:106)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.lambda$claimNextRunnableActivity$5(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl$WorkItem.run(HeartbeatAwareRetryingExecutorServiceImpl.java:240)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submitInner(HeartbeatAwareRetryingExecutorServiceImpl.java:517)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submit(HeartbeatAwareRetryingExecutorServiceImpl.java:455)
at com.vmware.vcloud.common.activity.dao.impl.AbstractActivityService.execute(AbstractActivityService.java:31)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimNextRunnableActivity(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueue.dequeue(DefaultActivityQueue.java:96)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor.dequeueElement(DefaultActivityQueueProcessor.java:313)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor.access$100(DefaultActivityQueueProcessor.java:54)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueueProcessor$1.run(DefaultActivityQueueProcessor.java:192)
2018-08-23 17:53:02,839 | WARN | processor-ValFabric | DatabaseConfiguration | Internal error |
java.sql.SQLException: SHUTDOWN ist in Bearbeitung.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:477)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.apache.commons.configuration.DatabaseConfiguration.getProperty(DatabaseConfiguration.java:177)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:264)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:282)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimWithCheckForExpectedExceptions(ActivityQueueServiceImpl.java:106)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.lambda$claimNextRunnableActivity$5(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl$WorkItem.run(HeartbeatAwareRetryingExecutorServiceImpl.java:240)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submitInner(HeartbeatAwareRetryingExecutorServiceImpl.java:517)
at com.vmware.vcloud.common.retry.impl.HeartbeatAwareRetryingExecutorServiceImpl.submit(HeartbeatAwareRetryingExecutorServiceImpl.java:455)
at com.vmware.vcloud.common.activity.dao.impl.AbstractActivityService.execute(AbstractActivityService.java:31)
at com.vmware.vcloud.common.activity.dao.impl.ActivityQueueServiceImpl.claimNextRunnableActivity(ActivityQueueServiceImpl.java:87)
at com.vmware.vcloud.activity.toolkit.queueing.DefaultActivityQueue.dequeue(DefaultActivityQueue.java:96)
[.......]

The Cell wasn’t come up and also after a restart of the other Cell we hade the same issue on every Cell which was restarted. After a log of searching I figured out that the created configuration for the performance metrics produced this error. Problems / errors in the vCD Database stopped the start of the vCD Services.
After some search I found a Blog Post on the VMware Blogs (https://blogs.vmware.com/vcat/2017/11/virtual-machine-performance-metrics-vmware-vcloud-director-9-0.html) which didn’t indicate this problem but the solution from there also worked in my case.
The relevant Information’s are these lines:
2. Edit the content of the /tmp/metrics.groovy file to:

configuration {
}

3. Run the following command:

# cell-management-tool configure-metrics –metrics-config /tmp/metrics.groovy

After that I was able to restart my cells and everything went smooth again.

0

Import workflow package to your vCO server via command line

By
Joerg Lew
– July 10, 2014Posted in: Best-Practices, Get Started!, Tools

vCenter Orchestrator’s REST API allows not only to start and monitor workflows, but also manage the content of your vCO server. With the Packages manager of the API it’s for example possible to import workflow packages via  the API. Example to do exactly this via command line using curl:


curl --form "name=file" --form "[email protected]"  -X POST -u vcoadmin:vcoadmin -k https://172.16.96.130:8281/vco/api/packages

…with de.vcoportal.someTestPackage is the full filename of the package to upload.

See more details in the API reference documentation on https://your-vco-server:8281/vco/api/docs

0

Safe your job, create unmaintainable workflows!

By
Joerg Lew
– April 1, 2012Posted in: Best-Practices, Get Started!, Workflow Design

The need for unmaintainable workflows

As an external consultant as well as permanent employee it’s always a good idea to make yourself indispensable for your customer/company. This ensures ongoing engagement/a safe job.

In IT in general you can achieve this by devloping unique solutions nobody else except yourself can understand (and maintain). I’m sure: You all can instantly give at least one example for an IT-component where youself/your teammates/everybody else follow the policy: “Never touch a running system“.

 <buzzword-for-the-robots>The modern term for this is NoOps.</buzzword-for-the-robots>

 As a workflow developer you also can follow some simple rules to make sure: You, and only you, will be able to maintain, re-use or even exand your workflows!

Here we go…:

Documentation, Labeling, Description Fields

Avoid Documentation, and use the default labels for workflow elements. So one has really click into each element, read the code, see the bindings to figure out what happens here.


When naming workflows and actions, and their parameters and attributes: Don’t be too obvious! A simple “name” or “vm” or “theArray” is enough.

Be aware of the Description fields! They are shown in so  many different places (when you re-use the elements, mouse-over in the schema, for documentation, …), that your really want to avoid this disclosure of “what this element is for”.

Logging

For the usage of System.log(), System.debug() and co. you have the choice:

  • Don’t log at all! Your NoOps-Team will thank you for that.
  • Log everything! It’s always a good challenge for the NoOps-Team to find the really important messages in hundrets of lines with traced-out variables (Eastern is coming!)…
  • Just log values! The value of a variable is what’s interesting. Nobody wants to know what variable it is… System.log(myVariable) does the trick.
  • Pro-Tip: You can even improve confusion by logging WRONG things, e.g. when you copy&paste scripts to re-use it in another context.

Workflow Design

Especially important for Library-Workflows which are intended to be called by other Workflows:
Hardcoding values in the script, not exposing important parameters as INPUT-parameters and  creative usage of  the data types ANY and Properties (of course whithout proper documentation) will ensure that it’s you, and only you, who can re-use the workflow.

In General:
Use the default names “actionResult” and “errorCode” when binding Action elements and Exception paths…

Forget about the “Add note…”-Feature! Structuring workflow part with coloured background just creates a better visual structure for the workflow.

Workflow Organization

Duplicate Workflows for small adjustments! This helps to keep your work big if something has to change in there for the next version.

Don’t use too many Workflow Folders, or clear labes for them! Leave it to the user to figure out which workflows are intended to be called manually, and which ones are just the “drivers” for them.

Create hidden dependencies! Calling Actions or nested Workflows not via the corresponding Elements, but within scriptable tasks creates nice surprises if one is changing the called.

Optimize your JavaScript coding style

You can find a long list of tips for that here: http://thc.org/root/phun/unmaintain.html

Forget vCO, just do scripting!

Who needs the visual, self-explaning graphical flowcharts of vCenter Orchestrator when you can to something like this??:

Pro-Tip: Use the PowerShell-Plugin or the PowerSSHell-Plugin to combine the best of both worlds. This will garantuee your job forever!

:mrgreen: HAPPY APRIL FOOLS’ DAY!!! :mrgreen:

Special Greetings to my mates in Sofia 😎 : Please do not take that criticism of some parts of the vCenter-Plugin-Library personal! I know it’s an hard challenge to maintain this inherited aged content 🙄 . You do a great job, especially with all the new “written-from-scratch” Plugins! Keep them coming!!! 😀

Got more?

Comment on the post and share your finest ideas  how to create poor workflows! 😈

0

The Philosopher’s Stone of vCO: Workflow Reflection

By
Joerg Lew
– March 8, 2012Posted in: Best-Practices, Examples

Don’t fear anything, it’s not necessary to live in a barrel to understand this topic! You just have to go a bit beyond the obvious…

“In computer science, reflection is the process by which a computer program can observe (do type introspection) and modify its own structure and behavior at runtime.” says Wikipedia.

The Orchestrator also provides a kind of reflection for workflows: Using the keyword “workflow” in a JavaScript Element a workflow can access its current runtime state. This article explains the basic understanding about Workflows and WorkflowToken, script examples and common use-cases for that.

The Basics

An important conecpt of vCO is to create a so called Workflow Token for every single time a workflow gets executed. Only so it’s possible to keep track of multiple in-parallel runs of the same workflow, what’s absolutely possible. (Currently supported in vCO 4.2: 150 concurrent running workflows).

In the vCO Smart Client you can see all the workflow token (=past executions) in the Workflow library, as leafs under the workflow itself:

If you are familiar with object-oriented programming, consider the Workflow as a class (description), the WorkflowToken is an actual instance of this class.

When you select a workflow token in the tree, you can see its details in the main window, most important:

  • the state (e.g. running, failed, completed, waiting)
  • the current vlaues of the workflow attributes
  • the log output (done via System.log() or System.debug())
  • (if there is one:) the error message at the bottom of the Variables-Tab

<Additional Technical Background>: All information about the state of a workflow token is stored in vCO’s database (the state, the attributes, the current element,…). So it’s possible for vCO to continue the workflow even after a crash of the server (Checkpointing).
For a quick & (really!) dirty ( & completly unsupported!) monitoring you could do SQL-cram against the vCO database like that:

ALTER trigger [dbo].[monitor] on [dbo].[VMO_WorkflowToken]
AFTER UPDATE
AS
SET NOCOUNT ON
IF EXISTS ( SELECT * FROM Inserted I WHERE I.globalState = 'failed')
BEGIN
/* ***DEMO ONLY:** */ Insert into dbo.fails VALUES ('adsf'); # send a mail, wake somebody up...
END

</Additional Technical Background>

How to…

Back to topic: You can also access the runtime information about the current workflow execution “from inside”, by using the keyword “workflow” (small-caps, remember: JavaScript is case-sensitive!) in a scriptable element.
This keyword returns an object of type WorkflowToken.  This object now represents the current instance of the workflow. This object provides some attributes, see the script below for examples…

Important: If you want to access the Workflow-object (the (class/description/definition), you have to use the rootWorkflow-Attribute of the WorkflowToken (eg. “var theWorkflow = workflow.rootWorkflow”; theWorkflow will be of type Workflow). (Yeah, that’s what I meant with “beyond the obvious”)

The following script examples shows all that stuff: First we get some information about the current instance (you got it, the WorkflowToken). Starting in line 10 we access the Workflow, the last part from line 23 uses the WorkflowToken again.

//get the current workflow token /"instance"
 var token = workflow;
 System.debug(token);

//get some details about the current workflow token/execution/instance
 System.debug("startDate: " + token.startDate);
 System.debug("businessState: " + token.businessState);

//get the Workflow-"Class" of the current token
 var rootWorkflow = workflow.rootWorkflow;
 System.debug(rootWorkflow);

// get "declaraction": Array of Input Parameters of the workflow in general
 var inParams = rootWorkflow.inParameters;
 System.debug("inParams: " + inParams);
 //loop through array and print out details for each parameter
 for (var i in inParams) {
 loopParam = inParams[i];
 //print out in type:::name:::description
 System.debug(loopParam.type + ":::"+ loopParam.name + ":::" + loopParam.description);
 }

// get the current token's inParameters (they include the values, and are returned as Properties
 //(key : values tuples)
 var curInParams = token.getInputParameters();
 System.debug("curInParams: " + curInParams);
 //crawl through the Properties
 var keys = curInParams.keys;
 for (var i in keys) {
 var curKey = keys[i];
 //print out properties in format key :::: value
 var curValue = curInParams.get(curKey)
 System.debug(curKey + " :::: " + curValue );
 }

All that happens just in the script, there are no workflow attributes bound to this scriptable element. The Workflow you can download below has one INPUT Parameter, just to see something in the logs :-). It’s not used anywhere.

The script produces output like this:

Use-Cases

Now, what to use all that stuff for?

  • Obviously, if you need some of the information, like the startDate, later in the workflow
  • If your workflow has an User Interaction element, you can send an URL to answer it via email (there is a complete example in Library / Mail)
  • It’s possible to change the User context the workflow runs, using the changeCredential()-Method
  • If you are working with the Business State field of workflow elements, you eventually want to read this in script.

Not really related to the “Reflection”, but useful anyway:
The same objects are used when you access past workflow tokens not of the current workflow, but of other workflows. This can be used for instance to create Healthcheck Workflows which monitor the vCO Server for failed or canceled workflow runs (a much cleaner & more reliable ( & supported) way to replace the SQL-cram mentioned above!).
Insider Tip: See the Actions provided by the module “com.vmware.web.webview” for a lot of useful scripts! They have many more use-cases than only within webviews…

Now, finally:

Download the example!

It should just run out of the box…

Workflow Reflection example
Workflow Reflection example
WorkflowReflection.workflow
1.8 KiB
Details...
0

Use the Workflow Validation tool in vCO!

By
Joerg Lew
– January 13, 2012Posted in: Best-Practices, Get Started!

One of the first things I got teached during my studies was: Always deliver code that compiles with

“0 Warnings, 0 Errors“!

We had to follow that principle while trying to let LEDs blink (I miss a vCO-Plugin for MPC555 :-D) , but I still remember my Prof.’s words when I develop workflows.

The vCenter Orchestrator provides a Workflow validation mechanism, which checks the workflow for

  • open ends
  • unreachable workflow elements
  • unused workflow Attributes
  • unbound IN- and OUT parameters of workflow elements
  • formal errors in scriptable tasks (JavaScript)
  • unbound Exceptions (if you use an Exception connector)
  • Unset parameters in Configuration Elements
  • …

You can run the validation by pressing the “Validate”-Button in the Workflow editor, or in the context menu of a workflow in the workflowlList.

The results of the validation is shown in a popup window. There are two kinds of validation issues:
Errors are hard formal violations which prevent the workflow from running sucessful.
Warnings are “soft” violations: Your workflow can be started, but it likely has some bugs.

For some of the validation results “Quick Fix actions” are recommended. They might fix the issue, but do not trust them blindly!

When you ran the validation at least once, errors and warnings are also shown as small red or yellow symbols directly at the related workflow element in the schema. This is very helpful for larger workflows!

By default a workflow is also validated before you start it (only when you try to start the workflow in the vCO Client, not via weboperator nor the SOAP API!). If your workflow has an error (warnings don’t matter), you cannot start the workflow, and you’ll get this error message:

You can change this behavior in the menue “Tools / User Preferences… ” of the vCO client, there in the “Workflows”-section (the checkbox is labeled with “Validate workflow before running it”). I recommend to only do this in your development environment, when you want to start workflows with errors (e.g. while they’re not developed completely yet)!

Also be aware:
The Workflow Validation only can check for formal errors. Semantic errors in your workflow and its JavaScript parts can not be detected by the validation.
A valid workflow is not necessarily error-free!

In very rare caes you also might get some false positives in the validation results. The only situation I remember is, when you need a workflow attribute only for input presentation processing, but not in the workflow: You’ll get a “Attribute XYZ is never used!” warning (, and the quick-fix would delete it and break the processing of input presentation!).

In sum: Always end up with a workflow that shows “0 Warnings, 0 Errors“! :mrgreen: