WSUS: An error occurred trying to connect the WSUS server

Ran into this error message when configuring a new WSUS server:

Restarted the WSUS, WID, and IIS services to no avail. I even rebooted the server. The WSUS console would work for a short period of time, and then would randomly stop working.

I found that the WSUS app pool in IIS was being disabled anytime new clients connected to the server. I believe this was because the app pool was running out of usable memory. 

You can manually start the app pool in IIS, but it will continue to crash. 

The solution for me was to increase the memory limit available for the application pool:

By default it is only configured to use just under 2 GBs. I reconfigured it to use up to 4 GB and the WSUS console has not crashed since. After re configuring the memory for the application pool, run an IIS reset or reboot the server.

UPDATE: Setting the Private Memory Limit to "0" will allow the application pool to use whatever amount of memory it needs.