TL;WR Probably the SG90 acting up again. Those things are weird. Rebooting the SG90 and the Director server fixed it for me. YMMV.
While moving around some VM’s we had a Shoretel Director server running without a network connection for 4 hours during a maintenance window. Afterwards users couldn’t change their Call Handling Modes or change their agent logged in/out status. It failed from both the phones and from communicator. At first I thought it was a CAS problem; however the phone directory, history, options, and speed dial features were all working correctly.
I popped up the IPDSCASCfgTool (see bottom) to set the log levels for the CAS to include all the DB and CAS flags for a start. After that I used powershell to stream the logs with Get-Content. I use Measure-Object first to grab the line count of the file so that we can skip the first 393,000 lines straight to the live output. That’ll work like tail -f in linux and just continuously stream the logs to the console.
Note: You SHOULD be able to use Get-Content -Wait -Tail to skip to the end, but that wasn’t working on this particular server. Gremlins…
PS C:\\Shoreline Data\\Logs>** Get-Content .\\ipds-190225.000000.Log | measure Count : 393693 PS C:\\Shoreline Data\\Logs>** Get-Content .\\ipds-190225.000000.Log -Wait | where -Property ReadCount -gt 393700 17:49:28.837 ( 3264: 3512) >SetUserCHM. User: 123. CHM: 2 17:49:28.888 ( 3264: 3512) 15:52:01.574 ( 7508: 5168) >CDBWriter::SetUserCHM::CDBUpdateTable::Update() failed. Error: 0xc1200db5.
SetUserCHM was me (unsuccessfully) changing from CHM 1 (Standard) to CHM 2 (In a Meeting) from communicator (testing from a phone will also log here but it’s noisier). That error sent me off looking for database issues, communications problems, etc. No dice. The evt log showed some interesting output though:
15:55:17.013 ( 4080: 4476) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.029 ( 7508: 4036) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.183 ( 4080: 4436) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.183 ( 2992: 6552) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.183 ( 1764: 1856) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.187 ( 4972: 5232) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
After digging around for named pipe issues and doing traces I tried the same on the voice switch and didn’t see any interesting errors. Theoretically the phone’s button and control traffic hits the voice switch and is making it to the director but I couldn’t verify that because I couldn’t get the switch to start a packet cap. Supposedly that’s because of a cipher mismatch: the director server tries to ssh into the voice switch to start the packet cap but it fails to login using it’s certificate.
Anyway I ran out of ideas and just waited until I could restart the switch and it worked. Everything was fine. Once again the SG90 blew up in my face was behaving unexpectedly and sent me one step closer to trying the vswitch.
Another one of those 1/1 google searches: