AddressBook Error after CU applied

Problem
This one came up a few months ago...

One thing that has been omitted from the automatic update procedure is the PS command to update the SQL database  Don't get me wrong, I am really happy about the simplicity introduced by running the LyncServerUpdateInstaller.exe that figures out what updates to apply to which servers on its own. 

In my case, after running the updates I launched the PS Command as instructed

Install-CsDatabase -Update -ConfiguredDatabases -SqlServerFqdn <EE.BE.SQL_FQDN> -UseDefaultSqlPaths

Only to find that I no longer had full access to the SQL backend (thanks SQL Admin). Once this was corrected I attempted the PS Command once more and to my disappointment was greeted by this error below:-

Running script: C:\Windows\system32\cscript.exe //Nologo "C:\Program Files\Common Files\Microsoft Lync Server 2010\DbSetup\RtcAbDBSetup.wsf" /sqlserver:<ServerName> /serveracct:EMSC\RTCComponentUniversalServices /verbose
---------------
Installed SQL Server 2005 Backward Compatibility version is 8.05.2312
Connecting to SQL Server on <ServerName>
SqlMajorVersion : 10
SqlMinorVersion : 50
SqlBuildNo : 1600
SQL version is acceptable: 10.50.1600.1
Default database data file path is F:\SQL_Data
Default database log file path is G:\SQL_Logs
Opened database rtcab

Db version unknown. Clean install required.
(Major upgrade of database required.)
Due to schema changes this database cannot be re-used. It must be dropped and a new one created.
To preserve data, you must use this product's backup/export restore/import solution. Examine the product documentation for instructions.
---------------
Exit code: ERROR_NEED_MAJOR_UPGRADE_USE_IMP_EXP (-50)
---------------
What! The ABSStore database was not getting created, and now the RTCAB and RTCAB1 databases are poked. They were working fine just before.

Effect

I had run out of time and needed to get off the systems so I did a quick check and found that all seemed to work fine. I then noticed that my Lync Event log on the FE was reporting that the ABServer and all things related to AB was stuffed. The ABServer.exe was attempting to start and then failing in a cycle that quickly filled the logs. Shortly after this users started getting the old ! to indicate that the client was unable to download the GAL. Makes sense since the AB wasn't running.


Interestingly, a Lync Trace of the ABServer reported this:

Connection string "Data Source=<EEBESQL_FQDN>;Initial Catalog=RtcAb;Integrated Security=True;Enlist=False;Connection Reset=False;Connect Timeout=10"

Followed by connection errors. Taking a closer look at the myriad of errors in the event log I also saw this:

Process: 'C:\Program Files\Microsoft Lync Server 2010\Server\Core\ABServer.exe' Exit Code: C3E8302D!_HRX! (The worker process failed to initialize itself in the maximum allowable time.!_HRM!).
Cause: This could happen due to low resource conditions or insufficient privileges.
Resolution: Try restarting the server. If the problem persists contact Product Support Services.

Restarting the server, of course...that didn't work.

I know it isn't resourcing as I been trying this at different times of the day and the SQL server wasn't low at all. Ahh privileges ..wasn't that how this all started? Running the update PS Command with insufficient rights? That has been remedied. Talking to the inhouse DBA I was informed that the RTCAB and RTCAB1 look rather unusual. RTCAB1 had been locked and the Schema was odd. Some rights didn't look right (all DBA talk, right).

Solution 

First up let me remind you that I get rather anxious simply by using the SQL acronym in speech. After trying a new CU update (which failed with the same error) from advice here and here.. 

I decided to "drop" the RTCAB and RTCAB1 databases (more SQL talk). Had the SQL guy back them up as a precaution (they were broken already but somehow this made me feel a little more comfortable).

Stopped all things Lync from the FE Server (stop-cswindowsservice), dropped the databases and then launched the PS Command (above) that started all this trouble. 

SUCCESS!

What I noticed was that the PS Command Install-CsDatabase -Update simply installed the RTCAB and RTCAB1 databses when it found that they had been dropped...phew..gulp! 

I then ran update-csaddressbook, replicated to other Lync servers. Manually deleted the GAL and fired up the client, GAL was downloaded. Finally!

The SQL DBA tells me that the RTCAB and RTCAB1 now look very different as far as permissions and shema are concerned. 

I'll ask them to explain this to me once more but don't hold your breath for a better take on this from me, as I said before touching S-Q-L makes me very uncomfortable, talking about it is almost forbidden so it may simply be a whisper.