Friday, October 12, 2012

A quick way to bring down SharePoint 2010

Yesterday I was working through the steps to configure incoming email for my SharePoint 2010 farm.  After some testing, internal mail sent to SharePoint was being dropped into the correct document libraries, as intended.

But somewhere along the way, when configuring the SMTP service on the SharePoint machine to pick up files from the drop folder, I had hit a snag trying to change the SMTP domain from the default <servername> (which was not user friendly) to a more friendly @spmail.company.com.  The emails were being received in the default drop folder in the /mailroot and not in the custom drop folder I had configured on a data volume.

While experimenting with different solutions I had the (un)brilliant idea to add another http binding to the SharePoint Web Services IIS site for the host header spmail.company.com.  I must have been low on coffee at the time because I don't recall even making this change.  This apparently was a ticking timebomb because although SharePoint appeared to work just fine the rest of the day, I was greeted the next morning (no doubt after some nightly timer job bombed) by a nice correlation error on the SharePoint home page. Digging through log files on our WFE and App servers I found a myriad of errors:

WFE log:
system.nullreferenceexception: Object reference not set to an instance of an object.    at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()

App log:
Request for security token failed with exception: System.ServiceModel.ServiceActivationException: The requested service, 'http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc/actas' could not be activated.

Viewing User Profile Service Application in Central Admin:
This User Profile Application's connection is currently not available. The Application Pool or User Profile Service may not have been started.

Now having had NUMEROUS issues with User Profile Service App in the past, my default is always to run IISRESET on the App and WFE machines. This time I refreshed my homepage and discovered the magic bullet didn't work this time.

Troubleshooting the User Service profile app led me down the same paths as many others, checking to make sure the farm account had to proper credentials to run the timer service, FIM services, and the app pool.  Everything checked out, except the profile synchronization service was hung at 'starting'.

I turned my attention to the STS app.  App pool checked out, IIS site checked out.  Then someone suggested opening the STS web service url on the App server to look for clues.  I was presented with this very helpful message:

This collection already contains an address with scheme http. There can be at most one address per scheme in this collection.
Checking the bindings for SharePoint Web Services site revealed the culprit. I removed the binding and checked SharePoint again and voila, immediately sites were working again.  I returned to Central Admin and the User Profile Synchronization service was no longer stuck at Starting and was not Started.

Lesson learned, don't tinker with SharePoint if your coffee level is running low, there is no telling what will happen during your caffeine blackout!