Tag Archives: Distaster recovery

Zerto 4.5 – Installing DR between vCenters

Weeks ago I reviewed Zerto in its previous version.
The new one brings many new features that I’ll explain during this post.
I’ll begin it installing in my vCenter Lab, and connecting it to another vCenter. In a later post, I’ll consider connecting to my Org in Cloud (it’s a vCloud Director Org).

First of all, I’ll prepare a Windows Server: you can use 2008R2 or 2012, important is the language – some bugs were discovered with other languages. Remember: it CANNOT be the same server where vCenter is installed.

I’ll download the bundle, made of the application plus the .NET 4.5. If not already installed, setup will ask for it.


Here is the folder containing the 2 files:


and this is the warning you’ll receive if .NET 4.5 isn’t present – offering you to install it automatically:


In some cases (like mine) you could face the following error during .NET installation:


it means that you need to update Windows via WSUS. It could take much, much time, so maybe you have time for a beer in the while 😉

After udates, I proceed again to install:


Since .NET updated, the system will require a reboot:


and it will continue automatically the install process, with related warnings:

After accepting the usual EUL Agreement, it will check for the API version:

During installation you’ll be asked to install the Zerto Storage Controller too – it’s safe, you can allow it:

And now you can open the browser at URL: kttps://<zvmIPorName>:9669/zvm

Maybe you’ll have to wait for a couple of minutes to start services. You’ll be promped for the license number, and you’ll get the login page:


This is the dashboard:


Now, it’s time to install the VRAs on every ESXì. The Virtual Replication Appliances are small VMs that will execute the replication, attaching disks for every VM protected, and using them when a failover (test or live) will be performed.

In the following images I already installed 3 out of 4 VRAs, so it will be shown the last one. In the section “Setup” you’ll find the list of installed VRAs and the ESXis. You’ll see that the last one is missing. Clicking on “New VRA” a pop up will appear, asking for some VRA’s detail.

progress is shown below, by “Running tasks”, successfully completed:


Well, the system is ready to be paired to another vCenter, or to an org in vCloud Director, or to an Hyper-V system or, lastly, to an AWS environment. In this case, we’ll pair it to another vCenter. From tab “Sites”, let’s choose “Pair”, inserting, in the pop up, the address of the remote ZVM – if a FW in the middle, you should permit 9081/TCP (at least).

Bingo! Sites are paired. From now ahead, you’ll create all the needed VPGs and so on.

In a new post I’ll show how to connect to vCloud Director org.



VSPHERE 6 – vSphere Replication Upgrade

VSPHERE 6 – vSphere Replication Upgrade


Things are looking good after upgrading vCenter at both of my home labs to version 6.0. The process I followed for this was outlined in my previous post here.

However since upgrading vCenter to 6.0, I have lost my ability to monitor and configure my vSphere Replication workflows. Per the VMware product compatibility matrix athttps://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2109760, I know that my current build of vSphere Replication is going to need to also be upgraded in order to return functionality.

vSphere Replication between my two home labs is a primary concern for me. The site hosting this blog is actually a VM running in one of my home labs, front ended by NSX, running in a Linux guest OS via an NGINX instance. For disaster recovery, the bits are kept in sync with a remote lab via vSphere Replication. I want to ensure replication is up to date, so vSphere Replication will be the next component to receive the upgrade.


As a reminder, my lab components are in the below matrix. Items in RED remain to be upgraded.

PRODUCT Current Version Current Build Future Version Future Build
vCenter Server 5.5u2d 2442329 6.0u1b 3343019
Update Manager 5.5u2d 2061929 6.0u1 2945804
ESXi 5.5u2 2638301 6.0 Express Patch 5 3568940
vRealize Orchestrator Appliance 2945833 7.0 3310032
vSphere Replication Appliance 2613527 6.1 3051487
NSX 6.2.0 2986609 6.2.0 2986609


This should be a pretty short post. As is the case with most of the self-contained appliances from VMware, the upgrade procedure is stupid simple.

There is no external database to back up in my lab. The appliance is running on the embedded database, as I only have a handful of replications configured for the solution.

In the past I have pulled minor version updates to the Replication appliance straight from the appliance’s management web UI. To access this feature, simply log in to the appliance at http://vsphere-replication-appliance:5480 as root, leveraging the credentials that were setup for the original installation.

Once authenticated, navigate to the Update -> Status tab and run “Check Updates”.

Unfortunately, the jump from major version 5 to 6 is not available via the system web update utility.

Per the upgrade documentation at http://pubs.vmware.com/vsphere-replication-61/topic/com.vmware.vsphere.replication-admin.doc/GUID-30083484-FB13-485E-AEC9-0695EADB7B3D.html, we need to download and mount the ISO to the appliance and run the upgrade from there. Log into my.vmware.com and pull the ISO down to a system from which the file can be mapped as a bootable CDROM to the appliance. For my lab, I am going to the latest 6.1 build at the time of this writing, which is build 3051487.

Once downloaded, map the ISO to the CDROM of the vSphere Replication Appliance. Then log into the web management page of the appliance at https://vsphere-replication-appliance:5480 and navigate to the Settings area of the Update tab. Change the update repository to the local CDROM, and save the settings.

Navigate back to the Status area, and select “Check Updates”. Apply the update to version 6.1.

The update procedure will run through its installation – there is no progress bar for this procedure. You may end up staring at the web page and application console wondering what is going on, like I did.

A little bit of patience is necessary – it took about 15 minutes on my appliance before the web page finally told me to reboot to complete.

Go ahead and reboot the appliance from the System tab.

I always prefer to watch the console of the appliance during these reboots, as often times issues or failed service starts are indicated during while the init scripts run.

The reboot should take place without issue. Once the system is sitting at login prompt, log back into the web management UI and verify the current version number.

Fantastic. Next we need to navigate over to the VR tab and re-register the appliance with the appropriate vCenter. Select the Configuration section of the VR tab.

Re-enter SSO admin credentials, and save the settings in order to re-register the appliance.

The LookUp Service SSL cert will need to be accepted.

The appliance should then show a running state with the existing configuration in place.


Following the re-registration of the appliance to vCenter, I found that a restart of the Web Client service on the vCenter server was necessary in order to get the appliance plugin to show up again in the web client.

Once the restart was performed, the vSphere Replication icon was finally available again. However I now encountered the following issue in the Web Client:

There weren’t many details given for what the configuration item could be. Hovering the mouse over the error in the web client actually revealed a mouse over informing me that the NTP service was not running in the vSphere Replication appliance.

This was an odd issue that took me a lot of time to find a solution to. Simply starting/restarting NTPD on the SUSE appliance would not suffice. I ended up locating the following KBhttps://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2126965 on the issue which is what fixed the configuration. After following the steps in this KB, my appliance was finally healthy per the Web Client!!

Now, complete the above steps on the partner site.


Once the other side is also upgraded to 6.1, re-configure the pairing of sites within the web client.

Awesome! Navigating over to the Monitor tab for vSphere Replication shows my existing replications are healthy.

(Courtesy of Mistwire)

VMware Social Media Advocacy

VMware Validated Designs Business Continuity and Disaster Recovery

VMware Validated Designs Business Continuity and Disaster Recovery

VMware Validated Designs Business Continuity and Disaster Recovery

In this video, Ryan Johnson demonstrates the failover of the Software-Defined Data Center management, automation and operations solutions – distributed deployments of vRealize Automation, vRealize Orchestrator and vRealize Operations – between regions in the IT Automation Cloud validated design. Follow Ryan Johnson on Twitter as @tenthirtyam on on our podcast at vmware.com/go/podcast. Learn more at vmware.com/go/vvd or follow updates on Twitter @VMwareSDDC – – – – – – – – – – – – – – – – – -…Read More

VMware Social Media Advocacy

Zerto: DRaaS in cloud

This is my second post regarding Zerto, but I’d like to focus on the opportunity that it gives to take advantage of the cloud, in particular, VMware vCloud Director or even AWS. We’ll talk about vCD here.

This is the case when a customer has his own infrastructure on premise, but wants to have a parachute in the cloud. Or the opposite: his production is in the cloud, but he wants to keep his investment using the on premise infrastructure as DR.

First of all I’d like to thanks to Vladan Seget, who did a post about Zerto Lab setup here. I’m adding the 3rd case, recovery to/from cloud.

The architecture in this case is slightly different: there’s one more piece, ZCM (Zerto Cloud Manager) that is upon the ZVMs and vCloud Director (s). The installation is simple as ZVM, a light Windows Server and a package to be installed, very few options to set.


We’ll access the GUI using the server’s credential.


And this is what we get. I apologize for the black boxes, there will be many of them since it’s a production environment. This page summarizes all our customers, name (ZORG: Zerto Organization), a CRM ID for internal purposes (billing, among other) and then the numner of organization they have in vCloud Director and the number of sites on premise that they connect.

Here we should distinguish: there’s one more solution, called by Zerto In-Cloud. The customer can have all in cloud, an organization for production, and another one, usually in another DC, for replication. And nothing at home. Or a combination of this. The only limit is that a single VM can’t be replicated in 2 places. But the same organization can have vApps replicated in cloud and others on premise, for example.

When we’ll create a new ZORG, these is are info requested:


Now, let’s take an existing ZORG to explain what’s inside.


As before, the ZORG, the ID and, most important, the path where to upload VMDK to preseed big VMs.


on permissions tab, we can choose what the customer can do. And, the credentials to access the ZSSP (Zerto Self Service Portal) that we’ll see later. This portal is needed in In-Cloud cases, and when the customer experiences a disaster in his vCenter, so he can access Zerto panel via browser.


The organization (s) belonging to the customer are displayed here, with the cloud site (physical one, everyone with its own vCD), nam eof the organization as appears in vCD, and among other info, note the Max limits: we can limit either in number of VM either in storage GB for every customer.

To connect the infrastructure on premise, we need a connector:


ZCC, Zerto Cloud Connector – in this case the customer has 2 vCenter connected to his organizations. The connector is a simple VM, deployed automatically by ZCM, that has 2 NICs: one connected to the customer’s organization network, the other one to the main management network. To better understand the role of the connector, it’s like a proxy: in this case Zerto is multitenant, the customer can manage and see only his own VMs.

The last 2 tabs display the groups of VMs replicated (VPG, as Vladan will cover in his next post), and the errors FOR that customer:



And, lastly, the ZSSP: the portal for the customer to manage his replication. This is the access, where insert the credential previously seen:


after which we land here:


edit, delete, peer site (vCenter or vCD), status and RPO.

All you need after a catastrophic event (or even much more less…)

Thank you again Vladan.

Disaster recovery: ZeRT0 vision for a virtual world

Several times in my career I had to face a system issue that needed a “time machine” to solve. Usually a restore from backup was enough, whereas the backup was scheduled. In other situations, if and when the company I worked for could afford a DR traditional solution, I was lucky to rebuild the system from its mirror. Many times I had to rebuild the whole system, loosing data, usually not so critical to be backed up.


Then virtualization came. Snapshots were a godsend, the main thing was to schedule them, no matter if they could have an impact on performances, the pros were far superior than cons.

After some time, when virtualization became important and not only an exception,  snapshots started to be annoying. In the last years, having a backup was no more enough for a critical system in production – too many hours could be lost. And snapshots weren’t anymore adequate because or they were catched frequently – but with performances impact – or occasionally, between a backup and another – which in this case lots of data could be lost.

At 2012 VMworld I visited ZeRT0‘s booth. I was amazed by their solution, because it was almost syncronous, no snapshot based, no need of an alternative datacenter (either hot or warm) but, overall, hypervisor based.


Through these years the application evolved ’till covering, today, even hypervisor-hybrid environment. So nomore just HW agnostic, but even Hypervisor agnostic.

We latey upgraded to the last version, 4.0, it’s a major release.

Before reviewing the last version, I’ve to say that the upgrade process took very few time and, overall, so few operations – it was a real easy walk even in a complex environment like ours.

First of all, the architecture. We’ll talk of VRA (Virtual Replication Appliance), ZVM (Zert0 Virtual Manager), ZCM (Zert0 Cloud Manager) and VPG (Virtual Protection Group), just to name some acronyms.

At the very low layer are the VRAs: little linux appliances rooted one for a hypervisor that take care of every single VM, all its pointers, all its network data, where its storage is, etc.

Just upper lay the ZVM: it manages all the VRAs, transposes user’s setting via an intuitive (and nice) GUI, and connect to the recovery site pair ZVM.


In the case of Cloud providers, or simply in environments where more than a ZVM is needed, you’ll have to install a ZCM: an orchestrator for the ZVMs, but, especially, a tool that offers cloud multitenancy, exposing a Self Service interface for the end user (ZSSP) and that deals with Connector installation – a proxy that acts as a ZVM pair for the customer, but that allows a complete privacy showing only its organization and separating all the other customers’ ones.


Some of the benefit experienced in these years, and not only “marketing words”:

  • RTO of minutes (since the name of ZeRT0 – Zero RTO), RPO of secons
  • Native multitenant architecture
  • HW agnostic
  • Full vCloud Director compatibility and integration
  • Zero impact on production VMs
  • Full automation of failover, failback and test
  • Partial recovery

The main difference I fond with the other solutions in the market was the RTO/RPO timing (very low to call it not only DR but BC too), and no usage of snapshots thanks to a journaling file where all the modifications written on a protected vDisk are reported to the replicated one and consolidated one by one after the chosen history time.

Compared to a traditional solution, well, in this case the difference is even wider, from the possibility to recover a single VM and not all the storage volume, no need of mirrored DC, and the consequent cost.

Now, what this solution misses from my point of view.

  • From version 3.5 ZeRT0 implemented the extended recovery: it’s a middle way between DR and backup. I’d like to reduce the number of vendors for many reasons, so if backup could be covered by ZeRT0 I’ll be satisfied. But today we can’t use it for backup for 2 main reasons: it doesn’t dedupe, and restore from backup isn’t integrated in vCloud Director like, instead, recovery from DR is. So customer won’t be autonomous.
  • From a CSP point of view, another missing service is the possibility to connect to customers residing on other CSP.
  • It is a only-virtual solution. But this is not a limit for our needs.
  • It does not replicate IDE disks – some virtual network devices are based on IDE disks – Watchguard is one of them.
  • It doesn’t replicate vShield Edge rules, either org ones and vApp ones. The first case could be solved replicating by hand that rules before any disaster should occur, but the vApp ones cannot since the vApp is created only when a failover starts.

My opinion, anyway, is that this is a real enterprise solution, where the benefits are far more than the disadvantages. A particular mention goes to the Support guys. They’re highly professional, responsibles at any hour of the day (and the night) and they won’t leave you, if a WebEx is needed, ’till the issue is solved, no sentences like “I’ll call you after my senior engineer will evaluate the issue”, that senior engineer will be ready online for your issue.

Be aware, anyway, that ZeRT0 is a great tool, but it’s a TOOL, not a DR Plan. You need a plan to decide in an objective way when to declare a disaster and which are the procedures to be taken. Inside the plan you’ll insert ZeRT0 as part of it. Don’t make this mistake as some of our customers did.

I’m sure I forgot some important feature, I hope ZeRT0 will forgive me. Below the comments section will welcome any suggestion and correction.