As it turns out, this can not be done with a cable modem that's connected to a Cisco CMTS. There is MAC Address security at the CMTS to prevent spoofing and it just won't work. I'm migrating to new physical firewall hardware and looking into a fiber connection.

Hello Brian

Let me first not get your hopes up, I have never configured a vSphere Distributed Switch I use the free portion of ESXI. I have waited to see if anyone more qualified would answer your posts.

I am going to make an assumption that you do not have a need to physically separate your lan’s you only need to do this virtually. I believe the first step would be to establish communication via one vlan to the ClearOS vm and the rest should hopefully fall into place. I do not know how to configure your physical switch and regrettably in the world where everyone making a Layer 2 and above switch seems to run their own show in the hardware world. Now you may ask where vDS fits in all this, well it does not for now best to get a handle on you equipment. I am going to describe the steps that are required to have proper communication and you will be required to find the right configuration parameters for your equipment. Let say we attempted to connect a PC to your ClearOS vm Lan adapter:
PC < -- > stream frames not tagged < -- > Physical Switch (i.e. Access Port 1) configure the port to initiate tagging on the stream frames to (i.e. vlan 100), note configuration must be set to remove tags on the way back to the pc < -- > Configure the Physical Switch to create a Trunk, may or may not be required to configure a group? Must accept frames that are tagged for vlan 100 from Port 1, exit via (i.e. Access via Port 2), note trunk configuration should not remove tags on exiting the switch < -- > ESXI Physical Adapter (i.e. vmnic0) frames are still tagged < -- > ESXI Standard vSwitch0 frames are tagged, configure new VMKernel Group Name vlan100 to use VLAN ID: 100, tags will automatically be removed to frames exiting to the vm adapter card and tagged on the way back < -- > ClearOS vm adapter card configured to use VMKernel Group vlan100, stream frames are not tagged.

I made another assumption that we are going to use a Trunk between the Physical Switch and the ESXI Server. You could use a port without trunking but keep the logic consistent to a Trunk you must configure the port to not remove the tags when you exit the Switch with the frames on port 2. I noticed that you have a Layer 3 physical switch do not use the routing capabilities of your switch in other words stick to the Layer 2 capabilities for your configuration. I do not recall changing the network policy on my ESXI Switch, I recently built an ESXI 6 server connected the Trunk and everything worked out of the box. In regards to which lan adapter to use from your CLearOS vm, if you are not sure let me know we will have to track down the mac address from your vm adapters in the CLearOS server.

If you ping the ClearOS box after all this we are half there.

For anyone who cares to hear an update, I'm really struggling with this job and finally restored to undoing everything I've tried and posting in the VMware vNetwork forums with hopes of getting help from someone with deeper knowledge of networking and vDS (Virtual Distributed Switches) than me. If you're interested in reading, it's over at: https://communities.vmware.com/message/2551058#2551058

thanks,

-brian