My Lync 2013 Lab

As of today my lab consists of the following:
1 x Dell 690 Workstation (2 x Xeon 3.2Ghz Dual Core CPU)
16GB RAM  (2 x 8 PC2 5300)
4 x SATA Drives ~250GB each
2 x NIC (1 = LAN, 1 = WAN)
OS = ESXi 5.5

This server is currently hosting 20 VM's although generally ~14 can be powered on at a time before resources become an issue - mostly RAM and then everything becomes very slow. 

I will focus this post on just the Lync 2013 environment for now, excluding the Asterisk PBX integration and Exchange 2013 integration.

The domain is mtest.local. I also have subdomain, sub.mtest.local which was used to simulate a client site however generally I try to put everything into the root domain.

The network configuration
Both NIC's are plugged into my ISP router. One NIC is on a vSwitch labelled LAN, the other on a vSwitch labelled WAN.

The DMZ has two guests on it:
PFSense (Firewall, Reverse Proxy)
WAN IP 192.168.1.199
LAN IP 192.168.0.205

LyncEdge.mtest.local
WAN IP 192.168.1.99
LAN IP 192.168.0.99

The ISP Router DMZ configuration is pointed to 192.168.1.199. PFSense is then able to manage all external connections as per the firewall configuration.

The Lync Environment (all 1 CPU, 4GB unless otherwise specified)

PROD
DCRoot.mtest.local (2008 R2 Domain Controller, 1CPU, 1GB)
PFSense (FreeBSD based Firewall/Router/Reverse Proxy, 1 CPU, 1GB)
LyncEdge.mtest.local (Lync Edge Server in DMZ)
LyncFE.mtest.local (PROD FE Server)
LyncFE-a.mtest.local (PROD FE Server)
LyncFE-b.mtest.local (PROD FE Server)
SQL.mtest.local (PROD SQL Server, mirrored with SQL-A)
SQL-a.mtest.local (PROD SQL Server, mirrored with SQL)

DR
LyncFE2.mtest.local
SQL2.mtest.local

*Note to self: Plan a naming scheme next time :)


External DNS
Access/SIP/LyncDiscover/LyncWeb/LyncWebCoq.domain.com all point to my ISP IP.

There is also a _sip._tls.domain.com SRV record pointing to access.domain.com on port 4433.


PFSense Firewall Config
Rules:
TCP * * WAN Address 5555 * none <-- Goes to lyncwebcoq.domain.com (DR Lync Pool)
TCP * * WAN Address 4333 * none <-- Goes to lyncweb.domain.com (PROD Lync Pool)
TCP * * 192.168.1.99 5061 * none (NAT Rule, refer below)
TCP * * 192.168.1.99 4433 * none (NAT Rule, refer below)
UDP * * 192.168.1.99 3478 * none (NAT Rule, refer below)
TCP/UDP * * 192.168.1.99 50000-59999 * none (NAT Rule, refer below)
TCP * * 192.168.1.99 4444 * none (NAT Rule, refer below)


NAT
WAN TCP * * WAN address 5061 192.168.1.99 5061 <-- Goes to Edge Server (Only 1 for both pools atm)
WAN TCP/UDP * * WAN address 50000-59999 192.168.1.99 50000-59999 (Goes to Edge for RTP Ports)
WAN UDP * * WAN address 3478 192.168.1.99 3478 (STUN Protocol to Edge)
WAN TCP * * WAN address 4433 192.168.1.99 4433 (A/V port on Edge)
WAN TCP * * WAN address 4444 192.168.1.99 4444 (Web Conf Port)

Reverse Proxy
The Reverse Proxy is within PFSense and is the Squid3 package.

It is configured to for the WAN interface on IP 192.168.1.199 with external FQDN domain.com. Everything is default other than what I've specified. The Reverse Proxy port is either 4333 or 5555 depending on whether I want the requests going to PROD or DR.

The Reverse SSL Certificate also must be configured via PFSense Cert Manager. Under Certificates, Add a new certificate and input the external facing lyncweb.domain.com certificate. This cert should have your external facing Lync Web and Simple URLs (lyncweb, meet, dialin, lyncdiscover). To add the certificate it must be extracted from PFX format with the private key. A couple of notes here:

1. When requesting this certificate via Lync Deployment Wizard, Mark as Exportable MUST be checked otherwise you cannot export the private key. 

2. The PFX can be extracted into a PEM and KEY file with OpenSSL (download the Windows Binary, extract to C:\openssl). Copy the PFX file there and run the following:

openssl pkcs12 -in certname.pfx -nocerts -out key.pem -nodes
openssl pkcs12 -in certname.pfx -nokeys -out cert.pem

You can then copy these contents into the certificate manager accordingly and the certificate will be available in the Reverse Proxy configuration General tab. Also note that you will need to have two certificates in here, one for the PROD environment and one for DR so that when you switch between the two both certificates are available.

The Web Servers tab needs to have an entry for LyncWeb pointing to the IP of the FE (again make sure to change this when switching PROD/TR). The Peer port is 4443, this is the port the FE is listening on by default for external Web connections. 

The Mappings tab needs to have a mapping to LyncWeb as well. The URIs can be added here for both PROD and DR like this:

^https://lyncweb.domain.ca:4333/.*

^https://lyncdiscover.domain.ca:4333/.*

^https://lyncwebDR.domain.ca:5555/.*

^https://lyncdiscover.domain.ca:5555/.*

Note that Lyncdiscover has to remain the same URL however it can be in here twice because of the different ports being used. I had to do this because my ISP blocks 80/443 so standard ports cannot be used.

Few issues I've run into so far when configuring this:

1. Do not use spaces in names in PFSense. Anywhere. It wouldn't start anymore because I put a space in a peer name. This really goes for Lync too, spaces are just troublesome and should be avoided along with any other non alphanumeric characters.

2. LyncDiscover can be fickle. For some reason one of my FE's although running fine did not have the "Lync Web App" started - I'm not sure how to start this manually at the moment but suspect Get-CSWindowsService may have the answer. 

Anyway, when I would access lyncdiscoverinternal.mtest.local in a browser (DNS entry being a cname to lyncpool.mtest.local) it would point me to LyncFE-A and say page cannot load. After accessing the server via https://lyncpool.mtest.local suddenly in event viewer the Lync Web App started and then accessing LyncDiscoverInternal also works. This is an issue because the Lync 2013 clients are relying on this and when it wasn't working, the EndpointConfigurationCache was blank and the client would not connect. As soon as the "Lync Web App Started" entry appeared, the client was properly able to discover the server and populate that file.

3. External Lync Mobile Client. I was testing using my Android and for some reason it refused to connect although everything checked out. It turns out that on my network, 4G seems to have an issue connecting whereas LTE works fine. I stumbled upon this by chance when I went for a drive and suddenly the network switched and my Lync client signed in. Odd. 

Also with this client if you need to setup manual configuration for any reason, the path is this:

https://lyncweb.domain.ca:4333/AutoDiscover/autodiscoverservice.svc/Root

And if for some reason it still can't connect, use the browser and make sure that URL loads a XML file with the URL's the client will use. If it doesn't, you have an issue likely at the Reverse Proxy config.

 4. Edge <-> Internal calls were failing to connect. I used the CLS Logging Tool and found in the logs that connections from the FE's to Edge on 5062 were failing. This is the Edge A/V Auth port. I ran Netstat -a and could see the port was not listening. I then restarted the service and ran Netstat -a again and could see 5062 was now established to the FE's. The calling issue was also resolved.

Packet Capture
I've also taken a few packet captures to see the ports being utilized during an audio call from Edge to Internal.

Mobile Client is performing all activity via the Lyncweb 4333 port.

Lync 2013 client via Edge, utilized ports 3478 (STUN), 5061 (SIP), 4433 (AV Edge)