Building an enterprise datacenter is easy. When building multiple datacenters, and trying to share workloads, that is an animal of a totally different colour.
The goal is to have a separation of services into availability zones (which in itself, has multiple definitions).
From a very high level perspective, I looked at VMware NSX, Microsoft SDN, and OpenStack to see how they function.
First up is VMware NSX. NSX comes in two flavours: a) NSX-V, better known as NSX-vSphere, which uses VxLAN as the encapsulation layer, and b) NSX-T, better known as NSX Transformers, which uses Geneve as the encapsulation layer. VMware appears to be no longer adding services to NSX-V, and is only performing bug fixes. VMware is instead putting the development effort into NSX-T, which has ties into a broader range of platforms such as KVM, as well as ESXi/vSphere. Since NSX-V is, for all intents and purposes, being deprecated, I will focus on NSX-T. The key technology document supplied by VMware appears to be DOC-39405 - NSX-T 2.4 Multisite.
Based upon that document, it is difficult to design a two-semi-autonomous-datacenter solution. Basically, the lack of ‘local egress’ on stretched vlans without massive, all-or-nothing scripted gateway migration reduces perceived flexibility of their solution. That is, if you want to have a controller instance at each site, which in itself, drives up licensing costs, by about double. And there are limitations to WAN round trip times if you want to split the controller across sites, which gets into split brain problems and resiliency. VMware seems to be going to more of the MS PowerShell for automation. (and leaving Python in the dust?)
One more item is Brief History of VMware NSX saying "Active-active multi-site deployment is a joke and works almost as well as stretched data center fabric control plane - when you lose the inter-site link in an active-active setup, one of the sites shuts down".
With VMware no longer allowing kernel level insertions, there are very few effective avenues for providing third party routing/encapsulation engines. Cisco's ACI is said to have a plug-in for vSphere, and for insertion into each virtualization guest (which is a bit grotesque, I think). - 20191116 - Cisco programs directly against the VMware API now for the things they need to do.
So... moving on to Microsoft Hyper-V with the SDN add-on. Most of the docs, as of this writing, appear to reference Windows 2016, when Windows 2019 is 'out there'. And in those docs, the underlay/overlay does not appear to handle IPv6!! And in a similar manner to VMware, if two separate datacenters are required, a lot of SCCM and controller licensing is probably going to have to be doubled up. And I'm not really sure how two sites will connect or inter-operate. Another interesting fact is that the Microsoft SDN solution appears to use the Open vSwitch engine under the hood. But it appears as though they have disabled or hidden all the interesting command line and troubleshooting tools. More embrace, extend, and extinguish philosophy. And as a final observation, Microsoft SDN does use the VxLAN encapsulation capability. Cisco is said to have an ACI shim to fit in there to help with the network side provisioning aspects.
One more possibility I've been examining, and have been examining this one in more depth: openstack. No licensing fees. Automated via Python. Runs VxLAN as the encapsulation layer. Knows how to work with Open vSwitch and keep the related engine diagnostic commands visible. It can also integrate with Free Range Routing for the EVPN side of things. [TODO: find the Cisco link which talks about using LLDP and Cisco Data Center Network Manager (DCNM) to communicate with OpenStack to provision VxLAN end points]. With a standard multi-site deployment, OpenStack works with a single instance of Horizon and Keystone. I will need to experiment with that to understand the ramifications for Availability Zones.
