A quick way to bring down SharePoint 2010

Yesterday I was working through the steps to configure incoming email for my SharePoint 2010 farm.  After some testing, internal mail sent to SharePoint was being dropped into the correct document libraries, as intended.

But somewhere along the way, when configuring the SMTP service on the SharePoint machine to pick up files from the drop folder, I had hit a snag trying to change the SMTP domain from the default <servername> (which was not user friendly) to a more friendly @spmail.company.com.  The emails were being received in the default drop folder in the /mailroot and not in the custom drop folder I had configured on a data volume.

While experimenting with different solutions I had the (un)brilliant idea to add another http binding to the SharePoint Web Services IIS site for the host header spmail.company.com.  I must have been low on coffee at the time because I don't recall even making this change.  This apparently was a ticking timebomb because although SharePoint appeared to work just fine the rest of the day, I was greeted the next morning (no doubt after some nightly timer job bombed) by a nice correlation error on the SharePoint home page. Digging through log files on our WFE and App servers I found a myriad of errors:

WFE log:
system.nullreferenceexception: Object reference not set to an instance of an object.    at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()

App log:
Request for security token failed with exception: System.ServiceModel.ServiceActivationException: The requested service, 'http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc/actas' could not be activated.

Viewing User Profile Service Application in Central Admin:
This User Profile Application's connection is currently not available. The Application Pool or User Profile Service may not have been started.

Now having had NUMEROUS issues with User Profile Service App in the past, my default is always to run IISRESET on the App and WFE machines. This time I refreshed my homepage and discovered the magic bullet didn't work this time.

Troubleshooting the User Service profile app led me down the same paths as many others, checking to make sure the farm account had to proper credentials to run the timer service, FIM services, and the app pool.  Everything checked out, except the profile synchronization service was hung at 'starting'.

I turned my attention to the STS app.  App pool checked out, IIS site checked out.  Then someone suggested opening the STS web service url on the App server to look for clues.  I was presented with this very helpful message:

This collection already contains an address with scheme http. There can be at most one address per scheme in this collection.
Checking the bindings for SharePoint Web Services site revealed the culprit. I removed the binding and checked SharePoint again and voila, immediately sites were working again.  I returned to Central Admin and the User Profile Synchronization service was no longer stuck at Starting and was not Started.

Lesson learned, don't tinker with SharePoint if your coffee level is running low, there is no telling what will happen during your caffeine blackout!

SharePoint 2010 - Sign in as a different user causes immediate 401 UNAUTHORIZED

While working among half a dozen SharePoint 2010 sites one day, spread through Internet Explorer 9 tabs, I was testing some security trimming on some new groups for a particular site. When I tried to sign in as another user, I was immediately given a blank white page that simply said 401 UNAUTHORIZED.

The only related information I could find on the web was a post referring to the error but without a suitable solution: http://www.3guysonsharepoint.com/?p=979

I fired up a session of Chrome and tried the same thing. The Login prompt was displayed and everything worked just fine. I returned to IE9 and tried to sign in as a different user on the other tabs I had open. All returned the 401 UNAUTHORIZED error.

I closed down all instances of IE9, cleared my cache, and fired it back up. This time all the sign in operations worked as expected. I'm not exactly sure which part of the IE session got out of whack, but starting fresh seemed to clear things up. Hopefully it's not a problem that will resurface.

Error with upgraded publishing site

Today I tried creating a new Welcome page on a site that has the publishing feature activated. The site was upgraded from 2007 to 2010 long ago with no hiccups. Suddenly just trying to create a new page threw a runtime error that required digging through the SharePoint Log with a correlation ID to find:

The site is not valid. The 'Pages' document library is missing.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: Microsoft.SharePoint.Publishing.InvalidPublishingWebException: The site is not valid. The 'Pages' document library is missing.

The Pages library is certainly on the site. Deactivating and re-activating the publishing features from the site collection and site features lists didn't seem to have any impact on the error. SharePoint engineer to the rescue: Stefan Goßner has your fix!

http://blogs.technet.com/b/stefan_gossner/archive/2011/09/15/common-error-the-site-is-not-valid-the-pages-document-library-is-missing-error.aspx

Digging into ReportServer Database

I was looking for some information about my ReportServer database and ran across a great script by Scott Hebert that displays a lot of useful information:

http://sqlninja.blogspot.com/2009/01/querying-reportserver-database.html

USE ReportServer

SELECT
         CatalogParent.Name ParentName,
         Catalog.Name ReportName,
         ReportCreatedByUsers.UserName ReportCreatedByUserName,
         Catalog.CreationDate ReportCreationDate,
         ReportModifiedByUsers.UserName ReportModifiedByUserName,
         Catalog.ModifiedDate ReportModifiedDate,
         CountExecution.CountStart TotalExecutions,
         ExecutionLog.InstanceName LastExecutedInstanceName,
         ExecutionLog.UserName LastExecutedUserName,
         ExecutionLog.Format LastExecutedFormat,
         ExecutionLog.TimeStart LastExecutedTimeStart,
         ExecutionLog.TimeEnd LastExecutedTimeEnd,
         ExecutionLog.TimeDataRetrieval LastExecutedTimeDataRetrieval,
         ExecutionLog.TimeProcessing LastExecutedTimeProcessing,
         ExecutionLog.TimeRendering LastExecutedTimeRendering,
         ExecutionLog.Status LastExecutedStatus,
         ExecutionLog.ByteCount LastExecutedByteCount,
         ExecutionLog.[RowCount] LastExecutedRowCount,
         SubscriptionOwner.UserName SubscriptionOwnerUserName,
         SubscriptionModifiedByUsers.UserName SubscriptionModifiedByUserName,
         Subscriptions.ModifiedDate SubscriptionModifiedDate,
         Subscriptions.Description SubscriptionDescription,
         Subscriptions.LastStatus SubscriptionLastStatus,
         Subscriptions.LastRunTime SubscriptionLastRunTime
   FROM
         dbo.Catalog
         JOIN         dbo.Catalog CatalogParent
         ON Catalog.ParentID = CatalogParent.ItemID
         JOIN
         dbo.Users ReportCreatedByUsers
         ON Catalog.CreatedByID = ReportCreatedByUsers.UserID
         JOIN         dbo.Users ReportModifiedByUsers
         ON Catalog.ModifiedByID = ReportModifiedByUsers.UserID
         LEFT JOIN
   (
   SELECT
            ReportID,
            MAX(TimeStart) LastTimeStart
      FROM
            dbo.ExecutionLog
      GROUP BY
            ReportID
   ) LatestExecution
         ON Catalog.ItemID = LatestExecution.ReportID
         LEFT JOIN   (
   SELECT
            ReportID,
            COUNT(TimeStart) CountStart
      FROM
            dbo.ExecutionLog
      GROUP BY
            ReportID
   ) CountExecution
         ON Catalog.ItemID = CountExecution.ReportID
         LEFT JOIN         dbo.ExecutionLog
         ON LatestExecution.ReportID = ExecutionLog.ReportID
      AND LatestExecution.LastTimeStart = ExecutionLog.TimeStart
         LEFT JOIN         dbo.Subscriptions
         ON Catalog.ItemID = Subscriptions.Report_OID
         LEFT JOIN         dbo.Users SubscriptionOwner
         ON Subscriptions.OwnerID = SubscriptionOwner.UserID
         LEFT JOIN         dbo.Users SubscriptionModifiedByUsers
         ON Subscriptions.ModifiedByID = SubscriptionModifiedByUsers.UserID
   ORDER BY
         CatalogParent.Name,
         Catalog.Name

Clean up your SharePoint Features

If you noticed my previous two posts, you'll see that I've run across some feature upgrade issues when moving from SharePoint 2007 to 2010, and then again when applying service pack 1 to SharePoint 2010.

Today I ran across a really nice tool to help remove references to the missing features that may cause issues during your upgrades: http://featureadmin.codeplex.com/

This tool is extremely fast, easy to use, and effective at removing missing/faulty features from SharePoint farms, web applications, site collections, and sites.

SharePoint SP1 aftereffects

If you are running the Enterprise version of SharePoint 2010, you may notice something missing after the upgrade to Service Pack 1.

The Customize Form button may be missing from the List Ribbon's Customize List section.  This button is only available in the Enterprise version of SharePoint and not in Foundation.

To get your button back, navigate to the Top Level Site Settings. Choose Site Collection Features and scroll down to find SharePoint Server Enterprise Site Collection features.  This feature enables InfoPath Forms Services which will enable the Customize Form button.

If you aren't at the Top Level Site Settings and instead find yourself at the Manage Site Features page for your site,  you will see a very similar feature named SharePoint Server Enterprise Site features, but this will not affect the Customize Form button.

SharePoint 2010 SP1 troubleshooting

So our upgrade from 2007 to 2010 a couple of years back wasn't without errors in the log, but most of these referenced outdated, unused features or web templates that had been decommisioned from any production site. They didn't prevent the upgrade from succeeding.

But when I applied SP1 recently, I realized some of these lingering references were again surfacing to give me trouble.  The service pack installed successfully, but the PSConfig wizard failed. Log files contained 3 errors:

• A missing site feature
• A missing web template
• A missing web.config file

The result of this service pack install and failed PSConfig was a sharepoint farm that was completely down. A request to any page generated an error and referenced a correlation ID. Tracking this error down in the log was a looong dead end because it was just referencing services that were down as a result of the failed upgrade. I cursed under my breathe about testing this install on a development system that wasn't identical to production, after all, the development system upgraded to SP1 without a hitch.  Prior to implementing the rollback strategy (yay VM snapshots!) I decided to dive in and see if the upgrade was salvageable.

Our farm consists of an App server and a WFE. Although PSConfig failed on the WFE, the upgrade status in Central Administration said it was fully up to date, so I focused on the App server.

The web template was installed on the WFE, but not the App server. So I copied over the folder from the C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\SiteTemplates folder.

Since the WFE was responsible for Central Admin and all sites, I was surprised to find a site reference in IIS on the App server. Checking the site and folder revealed it was empty, and the site status was stopped.  After ensuring this site was indeed running on the WFE, I deleted the site from IIS on the App server and removed the folder from the wss\Virtual Directories in inetpub.

I performed a powershell search for the missing site feature:
Get-SPFeature | Sort -Property Id

The feature was a FBA Management package developed by Visigo to manage FBA user accounts from the Site Settings page in SharePoint.  This had been deployed to the WFE but not the App server.

It was then time to run PSConfig again. Success. All sites functional again.

Next steps:

1. June 2011 Cumulative Update
2. Office Web Apps with SP1