Okay, so I am back with another rendition of things I should have learned in Kindergarten, but must have been absent that day. J
Thursday of last week , we had a team site owner cleaning up a her team site getting rid of things that were no longer needed and lists that were no longer used. Well, it just so happens she executed a click path that sent about 23 of her 78 sites in to the ether and whenever she or her teammates tried to access the sites they would get a plain SharePoint "500 – Internal Server Error" page.
I checked the logs and could find nothing that was being documented as for the server error.
After some looking around in the Webs table of the content database for the SharePoint web application, I noticed that the sites were not deleted or missing. So this pointed out to me either we had an IIS problem or an internal SharePoint problem. Since much of the configuration is stored in the database for SharePoint I was betting on it being a SharePoint problem. Especially since we were getting a plain 500 Error page from SharePoint, not like the one that IIS would put out if it had run in to a problem.
Finally I had resolved to call Microsoft Support after working through all of my known debugging techniques. The call started with us redoing much of the checks I had already performed, but then the support tech mentioned there is a forthcoming KB article that would mention a Hotfix for SharePoint that would not be publicly available, but you would have to get in touch with Microsoft Support to get it.
The KB article that he was talking about was KB936867. It is not posted yet, but according to him it is in process of being published. The hotfix is supposed to fix about 11 different issues that Microsoft has been seeing with SharePoint that has been the highest on the radar of issues.
Before we started the Hotfix process, I had mentioned to the support technician that the farm we were working with was a B2 installation that was migrated to B2TR migrated to RTM. This was not a concern to him. Nevertheless we started the Hotfix install. Hotfix installed fine, then we needed to run the Configuration Wizard to apply the hotfix to the SharePoint farm. Half way through the applying of the hotfix, the configuration wizard bombs and tells us that half of the SharePoint Database is now upgraded and the other half is not. The 2 databases the blew up was the Shared Services Provider database and the Configuration database.
So now we have ½ of the farm database upgraded and the other ½ not.
Needless to say after some major panic and working through a few different scenarios, the support tech finally gets a product guy in from the Hotfix or SharePoint team (not sure which), but from what I gained from talking to him is that we shouldn't have used this hotfix if this installation was migrated from the Beta 2 bits. Oooops.
From that moment on, it was pretty apparent that I would be rebuilding my SharePoint farm and reconfiguring my web applications, reconfiguring my Shared Service Provider and going through general hell.
For the rest of the evening we recreated the farm in such a way that it would have the Hotfix installed, but we ended up having to recreate the Configuration database and the Shared Service Database.
Lesson to be learned from this. Never ever only ask once if your farm being migrated from beta bits will be a problem. Make sure that who you are talking to knows for sure that this has been tried and proved.
So, if you run across someone that is mentioning to you that you need to install this hotfix be VERY sure that you are not running the hotfix on a Beta 2 migrated installation. Or be sure that it mentions that this is an approved use of the hotfix. If not, you might be running in to a situation like I did and have to go through hell to get your farm backup and running.
Hopefully this will keep you from running in to the same situation.
Michael