During a recent project I had to migrate two Microsoft Office SharePoint Server (MOSS) 2007 farms between data centres located in separate countries. Due to the complexity of these farms, it was decided to perform a Physical-To-Virtual process to create virtual machine clones of these servers.
The new data centres required that all the SharePoint and SQL servers have their hostnames changed to reflect the new naming convention.
This change of hostname within SharePoint results in the Search service not being able to communicate with the various servers in the farm that host the crawl role, typically the web front ends.
An error would be shown in the Search Administration homepage for the status of each crawl server. Also the below is the error would be registered in the application event log every one minute:
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (35a1c76d-514f-48e2-b2c7-e95daebe5ee9). Reason: The underlying connection was closed: An unexpected error occurred on a send. Techinal Support Details: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream. at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult) at System.Threading.ExecutionContext.runTryCode(Object userData) at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result) at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.ConnectStream.WriteHeaders(Boolean async) --- End of inner exception stack trace --- at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn) at Microsoft.Office.Server.Search.Administration.SearchApi..ctor(WellKnownSearchCatalogs catalog, SearchSharedApplication application) at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize() at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob) For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
The reason we see this error is due to the use of an SSL certificate used for intra-farm communications between servers. The SSL certificate is used to secure communications between the Office Server Web Services service installed on each server. Normally this error is produced when the SSL certificate becomes corrupt. In our case the SSL certificate was created with the original hostname of the SharePoint server, as the hostname has now changed the SSL certificate is no longer valid and needs to be recreated.
Microsoft have kb support article on how to recreate the SSL certificate should it become corrupt: http://support.microsoft.com/kb/962928
When I followed these steps I was able to resolve the error appearing in the search administration homepage, however the event log error was continuing to appear every minute resulting in an event log full of errors and not much else!
The Resolution
A slight tweak to the Microsoft steps listed in the article above resolved this error:
- Open the Services from the control panel and disable and stop the Office SharePoint Services Search service. I made sure I disabled the service before stopping it.
- Download and install the SelfSSL tool, the tool comes as part of the IIS 6.0 Resource Kit.
- On each server in the farm open the SelfSSL tool.
- At the command prompt type selfssl /s:951338967 /v:1000
- Enable and start the Office SharePoint Services Search service again.
I believe the cause of the error in the event log not disappearing was due to the fact the Search service was instantly restarting once it had been stopped. By disabling the service before stopping it, the service was unable to restart allowing me to recreate the SSL certificate and register it in the farm properly.
After following these steps the error in the event log was no longer appearing.
- James
