Following some documentation I found, it seems to me that I have to setup FW rules between the Security Server and each 
VM. If I have a lot of VM's, that seems overkill to me.
See https://pubs.vmware.com/horizon-61-view/topic/com.vmware.ICbase/PDF/view-61-architecture-planning.pdf Table 5-2, 5-3, or
VMware View PCoIP Remote Access in View 4.6 - YouTube time 16:17 onwards.
And if I'd go directly, without Security and Connection Server, the setup is again, per VM.
Page 13, Section Using Network Address Translation and Port Mapping to page 16. Table 2-2 explains nicely.
(this document just explains it much better than any documentation I found with Security Server involved)
Do I really have to add that bunch of rules to the FW for each VM??
Dan