Get Started!

0

Nutanix Calm – a deeper look from an automation consultant perspective (Part 2)

This post will be the second post on Nutanix Calm and will build up on my first post which can be found here: http://www.vcoportal.de/2018/09/nutanix-calm-a-deeper-look-from-an-automation-consultant-perspective-part-1/ if you missed it.

 

In the last post we only created a virtual machine without any customization. In this post we will install additional packages and add an SSH Key for login to the VM.

Let’s go back to our previously created Blueprint. On the left site we have the tab “Package”. Choose this, give the Package a Name and use “Configure install”

 

What you will see is that beside the LinuxServer Icon a new Icon will open up. Here we choose “+ Task” to create some modifications during the VM build process.

 

On the left site you must configure some values. You provide a name, choose what you want to do (I want to execute a Shell Script), and take the Credentials you need to access the VM. In the script section you past your script (Linux Bash programming in my case) which you want to execute.

My script for example looks like this:


#!/bin/bash

cat <<'EOF' > /etc/resolv.conf

# Generated by NetworkManager

search vcoportal.de

nameserver 192.168.1.10

nameserver 192.168.1.20

EOF

yum -y update

yum -y install net-tools traceroute bind-utils

First, I declare my script as bash script (line 1). Then I change the DNS Server configuration (line 2 to 7) and update everything on my VM and then install some tools like dig, traceroute and so on for my virtual machine.

When you are finished, save the Blueprint and give it a try for a run.

In the Audit Section of the Application you can take a look at your installation and configuration. As you will see the new generated section InstallSoftware will also be there as point. You can see what the Output of the section is. When you click on the Small window in the market red rectangle you can pop up a bigger window with the output.

On the button you can also have a look at the created script.

In the next step we will insert an SSH Key. This key can be insert from the user wo execute the Blueprint. So, let’s go back to our Blueprint. Here we go the the “Service” Section from our Service. There we insert a Name for the Variable (in my case SSHKey) and a default value. It is important that you activate the “Runtime” checkbox. We want the SSHKey auf our user, not the default and so the user can insert “his” key.

 

When we are finished with the variable we add a “Task”. This task we use to insert the SSH Key into the authorized_keys field for a specific account (I am lacy so I use the root account. For security reason I would not recommend this in production….)

 

The script itself is also a Linux Bash script and easy.


#!/bin/bash

mkdir /root/.ssh

touch /root/.ssh/authorized_keys

echo "@@{SSHKey}@@" >> /root/.ssh/authorized_keys


 

In the first line we declare the script as a Linux Bash script. The second line create the necessary directory and the third line creates the file. In the last line we insert something in the create file……

Did you recommend the “@@{SSHKey}@@ in the last line? What’s that for a strange value?
In Calm variables which we use in script are market with “@@{VALUE}@@” so in our case that means we take the value of the before created Variable and replace the “@@{SSHKey}@@” field with the content.

One note why I used a new task and not placed the script in the first task we already did. From my point of view, I would recommend to create a new task for every goal which I want to reach. I personally thing that debugging is much easier in this case because we can see what went wrong (in which script). The next thing for me is, that even on script maybe doesn’t function like expected we still have the chance to access the VM and fix it during life time.

After we are done with our script, save the Blueprint and Launch the VM. When we are on the Blueprint just open the “Configuration” part and insert the SSH Key which you want to access the VM without password.

 

When everything works like expected you can login in your VM via SSH. If not, take a look in the Audit section.

As you can see the start with calm is very easy and there are a lot of things we can be done with calm.

During my “play around” with Calm I found some things which must be improved within Calm or limit Calm at the moment:

  • Calm is limited to Nutanix, vSphere-Cluster and Cloud Solutions. So, some customers still have to use a second solution for their automation need.
  • You cannot export created Blueprint and import them with all needed scripts, variables and images. With our easy Blueprint that is not a big issue but when you create a Blueprint with plenty of variables there is a “room” for mistakes and typos.
  • At the moment Calm has some limitations to work with for example ServiceNow and so on.

Some Ideas for Calm for future functions / extension:

  1. RFE: Blueprint Full Backup with images, scripts, cloudinit etc. à Provide a central repository in PC to make things re-usable: cloudinit, sysprep, scripts, images etc.
  1. RFE: DNS-Update Feature when VXLAN-DHCP from PE assigns IP-address à Autofill connection URL after application deployment (aka openshift) in the app description
  2. RFE: Integration of 3rd-Party config-mgmt tools like git, puppet, chef etc. in the GUI (dropdown menu, not via scripts) à Make existing scripts / tools / 3rd-party development re-useable
  3. RFE: Must-have: Native container integration to push CALM! No customer with an existing automation or configuration solution will switch to CALM without that feature.
  4. RFE: VM customization options with cloudinit for linux and sysprep for windows delivered as one-click/dropdown options in CALM (forgot that in the call)

 

If you can life with this limitations Calm is a good choice to start automation in your Nutanix environment.

0

Nutanix Calm – a deeper look from an automation consultant perspective (Part 1)

The last couple of days I played somewhat around with Nutanix Calm. Calm is the automation Solution from Nutanix based on the Acropolis. The official Calm statement from Nutanix can be found here:

What is Nutanix Calm?

Calm provides application automation and lifecycle management for the Nutanix and public clouds, as part of the Nutanix Enterprise Cloud Platform. Calm orchestrates the provisioning, scaling and management of applications across multiple environments to make the entire IT infrastructure more agile and application centric.

From Organization perspective calm works with Blueprint to organize the layers of the automation.

Also, here an official description:

Nutanix Calm models applications as Blueprints giving you a full end-to-end view of the application, its dependencies, and its requirements. From here you can quickly spin up multiple copies of an application and ensure that each one is done right each time.

Operations can be orchestrated, delegated, and repeated knowing each time that it’s done perfectly no matter who runs it. At the same you gain an impressive amount of visibility into who did what, when, and what the result was because Calm logs all actions and all logs from each action no matter where the application is deployed.

Blueprints can be published to end users based on their group or role, giving them the ability to instantly provision and use applications across clouds while ensuring policy is properly applied and logged. You can attach budgets and quotas to each team to track their unique spend or customize the cost of a particular application to control user behavior based on what applications you want them using.

 

A lot of customers are asking about Nutanix Acropolis and also for an automation solution. So, I decided to play a little bit around with calm to see how easy it is to archive first results with calm.

Bevor you can start with the development of a first Application you have to understand the different between a Blueprint and an Application.

In the Blueprint you declare the Number of virtual Machines, the “Automation” Part of the VMs and all other necessary things.

The Application is the running configuration of a Blueprint.

So, when you start with calm you have to create a Blueprint bevor you can create an Application. Let’s start with our first Blueprint which will be an easy on. The goal for this Blueprint is to create a running virtual Machine (Linux).

 

A Windows Open were you must provide a name for the Blueprint. A description is optional, but I would recommend this as a best practice.

 

After you are done with your Blueprint setup the Main Window of the Blueprint is opened.

 

Make yourself comfortable with the buttons and screen and then let’s create our first service.

The Services “hold” the virtual machine which offers a service for other virtual machines or user. A service could be a Load Balancer for example or a Webserver. When you create more advanced Blueprint (then this one we are doing now) you can build dependency between different services.

For now we only create one service.

On the left site of the screen we configure the necessary configuration. Create a Service Name, a Name for the Sub (Name is not the VM Name!), choose your Cloud (Nutanix or AWS for example) and your Operating System. I use Linux so that’s my choice.

 

On the lower part we provide a Name for the VM in our environment. Calm automatically adds some variables to provide unique names in our environment. And the Images we want to use as base for our deployment. If necessary you can change the Device Type and Device Bus. Note the “running men” nearby some of the fields. Here we are able to provide the user who execute the Workflow to provide his configurations variables for example a special VM Name. It is a good practice to provide default values for all parameter.

 

 

On the bottom of the field you configure CPU, Memory, additional Disks (VDISK) and the Network. If most network configurations are taken from the Nutanix default configuration so in my case I only need to insert an IP or use DHCP.

 

The last things to configure is the “Connection” Section. Here you must provide credentials to login to the base image. These credentials are used to configure the VM.

After that Save the Blueprint at give it a first try with “Lunch”

Now we must enter an Application Name and then let’s build the VM.

The build process starts automatically. Under the “Audit” log you can see what happens within your Blueprint and if it is successful.

Till now no rocket science and easy to go……

That’s it for now, in the next post we start to customize our VM…..

 

0

Honeypot as a Service (HaaS) Part 3

This is the third Post of the Haas Series…..

After we have everything up and running would should take a deeper look at the vRealize Log Insight.  There we get our notifications in the interactive analytics. In normal cases nobody will monitor the whole time the interactive analytics therefore we create a Dashboard. The create dashboard is only for your custom view. The log Insight documentation state it like this “You can add, modify, and delete dashboards in your Custom Dashboards space.”

So, in the first step we create a dashboard for your personal view. The creating is quite easy. So, if you have an alert in the interactive analytics you can create the Dashboard from there.

 

 

Just click in the New Dashboard Icon on the right site. In the Wizard you create a new dashboard. Provide a Name and if you want share the Dashboard with other Users in the environment.

I use the chart graph for my Dashboard. After you are finished you will find your Dashboard in on the Dashboard View under Shared Dashboards.

Quite Easy or? But would It be better to get an Alert Notification from Log Insight?

This is also very easy. When you go back in the Interactive Analytics just take the Alerts Button.

In the Windows we choose “Create Alert from Query”.

There you provide the required Details for the Alert. Be sure that you configured your SMTP Server before when you use E-Mail as alert notification.

That was cool so far, or?

In the last months I made a couple of NSX Implementations. NSX provides us some really cool feature like the possibility to move VMs into Quarantine to isolate them from the communication with other VMs. So why wo should not use these feature, to migrate the VM were the Access violation was made, into a quarantine location to research what is happening on this VM? When the access was done from a physical machine, we can also create a firewall rule which deny the access to the virtual environment? From my point of view this is security in an automated way.

In this blog Post I will not show how to install and configure the NSX Part of this series. There are a lot of useful information available.

To archive these goal, we can use a REST API Call to the vRO Server to call a workflow which we develop to migrate the VMs or create the firewall rule. One of the first things which we need, is the possibility to interact with NSX from the VMware Orchestrator. Therefore, will install the VMware Orchestrator NSX Plugin. The plugin can be found on the VMware Page:

https://my.vmware.com/web/vmware/details?downloadGroup=NSXV_VROPLUGIN_120&productId=417

We also need two additional Plugins The first plugin is needed to decrypt the BASE64 encrypt string from the Shim in vRO. The Plugin can be found here: https://communities.vmware.com/docs/DOC-24991

The next Plugin is not really necessary but from my point of view the JSON Implementation in vRO is not the best in the world…. therefore, I use the jsonPath for Orchestrator Plugin frim Soeldner-Consulting. The documentation and the Link to the VMware Solution Exchange can be found here:

jsonPath for Orchestrator

After we download the all plugins we can install them via the vRO Control center.

After we are finished with that, we have to create our Workflow that must start when an alert is triggered. The triggering can be done via email or REST API Call. I prefer the REST-API Call du the circumstance that this more flexible, but wait…….here we have a problem. The VMware vRealize Log Insight doesn’t offer any option to create a REST API Call. In the Web GUI only Webhook Notification is available. Therefore, we need a translation between the Webhook and the REST API Call which we need for the vRealize Orchestrator. As always there is an solution available…..Steve Flanders create a Shim which can act as Proxy for the Webhook notification in direction of the VMware vRealize Orchestrator. Steve has a GIT Repository with a good documentation about the different available versions and the installation. I will not go further on the installation as already everything is explained here: https://github.com/vmw-loginsight/webhook-shims

Another useful post can you find here:

https://blogs.vmware.com/management/2017/03/webhook-shims-now-available-on-docker-hub.html

After we installed and configured the Shim, we are ready to make our next steps.

I created a vRO Workflow which used as Input parameter the name of the Attacker VM. The workflow checks the Attacker VM against the configured Datacenter. When it is a virtual machine, then an NSX Security Profile is attached to the virtual machine and all communication is denied. When it is not a virtual machine, then an IPSet is created and this IPSet is added to a Security Group

For the workflow it is necessary, that the NSX Manager is configured within vRO. Also, that a security Policy exist which we can use in the workflow. Also, the Datacenter which host our virtual machines in our environment must be provided.

Now let’s have a look at the Workflow:

The Workflow consists from different areas. In the First Part of the Workflow (the yellow box) we have to decode the Base64 string and parse the VM Name from the Log Insight Message.

When we are finished with that, we check if the Attacker Machine is a virtual machine (the red box).  When the machine is a VM, we apply the security Group to the VM (the light brown box) which denies all communication via the Security Policy.

When the Attacker VM is not a virtual machine then the machine will be checked against DNS (the green box) to gather the IP. The DNS checking is done via vRO DNS query so the Orchestrator Appliance must be configured correctly with DNS Server. The DNS Check is done via Name and FQDN. When the name could be resolved and we have the IP Address, then an IPSet is created and the Attacker machine is applied to the security Policy. This only protects the virtual environment!

In all cases a email is created (the blue box) and send to an Administrator.

The Workflow itself has some Attributes which must be filled:

  1. Datacenter –> The Datacenter were to check against the Attacke Machine if it is a VMs
  2. SecurityGroupID –> The Securitygroup ID. The ID is not the name but should look like this: securitygroup-98765. If you don’t know the SecurityGroupID, you can Browse within vRO in the NSX Manager Connection. There you can find the correct name
  3. NSXManager –> Your corresponding NSX Manager (which must be configured before in vRO)
  4. smtpHost –> Your Mail Server
  5. fromName –>  Email Sender Name
  6. fromAddress –>  Email Sender Address
  7. toAddress –> Were the Mail should arrive
  8. DNSDomain –> Your DNS Name to check against FQDN
  9. Optional: SMTPUsername –> The Mail Server Username, if required
  10. Optional: SMTPPassword –> Password for the Mail Server user

For the Workflow itself there is room for optimizing. A loop to check against different Datacenter, or a loop to check against different Domain names……For me and this Demo the Workflow makes his job….

The Workflow can be found here: http://www.vcoportal.de/download/workflow/de.vcoportal.HaaS_.package

 

Honeypot as a Service (HaaS) Part 1 Link: http://wp.me/p7tsEp-C1
Honeypot as a Service (HaaS) Part 2 Link: http://wp.me/p7tsEp-Cb

0

Honeypot as a Service (HaaS) Part 2

In the second Post of this series, we create the Log File Reporting which we will send to vRealize Log Insight.

 

After our samba server is configured (in Part 1) we have to deal with our logfiles and report them to our vRealize Log Insight.

In our Samba Server we already configured the logging which is done in the path /var/log/samba. There files are created for every client which tries to access our server. As long as no client tried to access the file the director should log like this:

When you already accessed your server, you will have a log entry like log.192.168.157.10 or similar.

I will create an event every time a client is accessing the share. Therefore, I will use the inotify-tools. unfortunately, these program is not included the standard CentOS Repository so we have to add an additional repo. I use the fedoraproject repo where we download the Repo configuration and install it.


wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-10.noarch.rpm

rpm -ivh epel-release-7-10.noarch.rpm

After that we can install the inotify-tools with yum.


yum -y install inotify-tools-devel.x86_64

After that we create the string to monitor the samba server for client access.


inotifywait -mrq -e create -e modify -e delete /var/log/samba

In this monitoring we ignore changes to the log.smbd file, due the circumstance that this a a standard file form the samba service. The output of this monitoring locks like this:


/var/log/samba/ MODIFY log log.client1-vcoportal

As we want the information on a remote server, we have to extend our logging string with logger to send the information to a remote server.


inotifywait -mrq -e create -e modify -e delete /var/log/samba  | while read file; do ( logger -n loginsight.vcoportal.de-t securitybreak $file ) done

I send my string to a vRealize Log Insight server and tag the log entry with the word “securitybreak”. In Log Insight we can filter for that word……

So, if you use the interactive analyses you can search for your string.

For me the is still room for improvement. The only relevant information for me is the client name. So, let’s modify the monitoring string.


inotifywait -mrq -e create -e modify -e delete /var/log/samba/host | while read file; do hostname=$(echo $file | cut -d '.' -f2,3,4,5 ); (logger -n loginisght.vcoportal.de -t securitybreak $hostname) done

 

We strip done the string we send to our Log Insight Server to just the Client name. I guess there is a better way to create the string but for me this one work fine.

Now we get only the Client name reported in our Log Insight

How cool is that?

The last thing we have to do is to create a startup script for our monitoring string.

In our case, we will want to run inotifywait as a service and create an init script. First we will create our configuration file:


# specify log file

LOGFILE=/var/log/inotify.log

# specify target directory for monitoring

MONITOR=/var/log/samba

# specify target events for monitoring ( comma separated )

# refer ro man inotifywait for kinds of events

EVENT=create,delete,modify

# Log Insight or SNMP Server

SNMPSERVER=loginsight.vcoportal.de

# Security tag

SECURITYTAG=securitybreak

 

Next, we will create the init script where we use the config file we created before as input for the variables.


#!/bin/bash

# inotifywait: Start/Stop inotifywait

#

# chkconfig: 2345 10 90

# description: inotifywait waits for changes to files using inotify.

#

# processname: inotifywait

. /etc/rc.d/init.d/functions

. /etc/sysconfig/network

. /etc/inotifywait.conf

LOCK=/var/lock/subsys/inotifywait

RETVAL=0

start() {

echo -n $Starting inotifywait:

# Clear Log File

if [ -e $LOGFILE ];then /usr/bin/truncate -s0 $LOGFILE;fi

# Start the Monitoring

/usr/bin/inotifywait -e $EVENT -dmrq $MONITOR --exclude ‘\(log.smbd\)*’ -o $LOGFILE | cut -d '.' -f2,3,4,5 $LOGFILE | while read file; do (/usr/bin/logger -n $SNMPSERVER -t $SECURITYTAG $file) done

RETVAL=$?

echo [ $RETVAL -eq 0 ] && touch $LOCK return $RETVAL

}

stop() {

echo -n $Stopping inotifywait:

killproc inotifywait

RETVAL=$?

echo [ $RETVAL -eq 0 ] && rm -f $LOCK return $RETVAL

}

case $1 in start)

start;;

stop)

stop;;

status)

status inotifywait;;

restart)

stop

start;;

*)

echo $Usage: $0 {start|stop|status|restart}

exit 1

esac

exit $?

 

The permissions need to be 755:


chmod 755 /etc/rc.d/init.d/inotifywait

 

We can start the service now:


/etc/rc.d/init.d/inotifywait start

 

The result will be like this:


[root@backup ~]# /etc/rc.d/init.d/inotifywait start

Starting inotifywait (via systemctl):                      [  OK  ]

 

We need to ensure that the service starts when the server boots, so we need to add it to chkconfig


chkconfig --add inotifywait

chkconfig inotifywait on

 

The log entry is written in the /var/log/inotify.log file and from there they were cut down to post the to the log insight server. We have to use this loop due the circumstance, that in our start script inotiywait is used in daemon mode. There a logfile is mandatory.

That’s it for Part 2

 

Honeypot as a Service (HaaS) Part 1 Link: http://wp.me/p7tsEp-C1
Honeypot as a Service (HaaS) Part 3 Link: http://wp.me/p7tsEp-Cl

0

Honeypot as a Service (HaaS) Part 1

Disclaimer:

This series will show some Ideas which include “homemade” Security Functions. For professional environments, I recommend to talk to the security Vendor of your choice to build up a secure environment. This series will show examples which include different VMware products and their Interaction and teach things which can be done with them.

Now, let’s start.

Today I will start a series which I call Honeypot as a Service (HaaS). What is the Idea behind the HaaS?

In most environments hackers are already a long time present bevor the get localized. When the hacker has access on one of the server, he will start to search for other systems where he can get access and grab sensitive information (credit card data, intellectual property). So, which are system / functions a hacker is locking for?

So, most Hacker are looking for routing information and will start a “slow and silence network scan” for systems which he can attack and get into his hands. When we now start to place virtual machines in our network, which doesn’t have any functions for our network nobody should access them in normal case. From the standpoint of an attacker, he doesn’t know if a system is relevant or not. That’s will be the point for us, to provide the hacker a system with an old infectible SMB Server which he can attack. So, if somebody / somewhat tries to access such an “no functional VM”, we generate a log entry which is forwarded to an Log system. There we can parse the logs and generated “automated” rules to isolate VMs or VM communications.

In this example, I will use a lot of different VMWare products to archive these HaaS environment and the actions we will use.

This series will include:

  • VMware vRealize Log Insight as log host
  • VMware vRealize Orchestrator as “workhorse for a lot of task”
  • VMware NSX for VM isolation and Firewall Rules
  • A small Linux System as Honeypot with an SAMBA Service

So, let start to build up a Linux System which we can use in our vRA Blueprint as Honeypot. I prefer a CentOS Distribution. I choose a 7.x Release of CentOS. The ISO can be downloaded here http://ftp.hosteurope.de/mirror/centos.org/7/isos/x86_64/

I will use the minimal Installation ISO and Install Samba and SSH afterwards.

We will change the Hostname and IP Settings for a later time in vRA so during installation we keep the hostname and network on default settings. We have one important things which we take care about during installation. By default, the NICs are not automatically connected during installation, so that should be changed.

After our Installation has finished we will install the following packages:

  • Perl
  • Samba
  • DNS tools (just for Troubleshooting)
  • Network Manager (Text Version if you need to reconfigure your Config)

This can be done via this command:

yum -y install perl samba system-config-network-tui.noarch dns-utils

When you don’t have a DHCP Server in place, then you need to configure your network settings on Command line: $ sudo ifconfig eth0 192.168.1.50 netmask 255.255.255.0

Also, you need to set your default route:

$ sudo route add default gw 192.168.1.1 eth0

As last step you need some DNS Server entries:

$ sudo vi /etc/resolv.conf

Modify or enter nameserver as follows:

nameserver 8.8.8.8

After we have installed the need packages we can start with the installation from the VMware tools. This is documented well in the VMware documentation so I will not write it down here.

The next Step is to configure our Samba Server for a SMB Share which is available for the attacker.

Now, let us create a fully accessed anonymous share for the users. Everybody can read/write in this share.

We create a directory called /samba/backup and set full permission. The reason (for me) to set the name to backup is, to mark the share as important and a useful target which information or potential interesting thinks in it. I also will create some additional folders to have a interesting target. I use a linux bash script to create the folder and set the required permissions.

#!/bin/bash
echo "This script creates a bunch of folder which were used in a samba share example on vcoportal.de"

dirname=/samba/backup

echo "$dirname"

if [ ! -d "/$dirname" ]

then

echo "Folder doesn't exist. Creating now"

mkdir --p "/$dirname"

echo "Create some other Folders ....."

mkdir --p "/$dirname/HR"

mkdir --p "/$dirname/IT"

mkdir --p "/$dirname/Dev"

mkdir --p "/$dirname/Public"

mkdir --p "/$dirname/Files"

mkdir --p "/$dirname/Userhome"

echo "Set some permissions for the parent Folder"

chmod -R 0775 /samba/backup

echo "Set nobody as Owner for the Folder so that everybody can read see and read it"

chown  nobody:nobody /samba/backup

echo "Change SE Linux Permissions on the Folder"

chcon -t samba_share_t /samba/backup

else

echo "Parent Directory exists"

fi

Just create a file and make it executable with the script content in it. You can change the directory names to whatever you want / need.

After we created the directory we must edit the Samba configuration file.

vi /etc/samba/smb.conf
# See smb.conf.example for a more detailed config file or
# read the smb.conf manpage.
# Run 'testparm' to verify the config is correct after
# you modified it.
 
[global]
        workgroup = vcoportal
        security = user
 
        passdb backend = tdbsam
 
        unix charset = UTF-8
        dos charset = CP932
 
        hosts allow = 127. 192. 172. 10.
 
        max protocol = SMB2
 
 
        map to guest = Bad User
 
        # log files split per-machine (the host logs are stored on a non-default path):
        log file = /var/log/samba/log.%m
Log level = 2
        # maximum size of 50KB per log file, then rotate:
        max log size = 50
 
 
[homes]
        comment = Home Directories
        valid users = %S, %D%w%S
        browseable = No
        read only = No
        inherit acls = Yes
 
 
[Backup]
comment = Backup Share
path = /samba/backup
browsable =yes
writable = yes
guest ok = yes
read only = no
force user = nobody

Now we can test the Samba server configuration

We can test the Samba server configuration syntax errors using the command ‘testparm’.

testparm

When everything looks good you can browse to your samba share.

 

 

 

 

That’s it for Part 1

 

Honeypot as a Service (HaaS) Part 2 Link: http://wp.me/p7tsEp-Cb
Honeypot as a Service (HaaS) Part 3 Link: http://wp.me/p7tsEp-Cl