Windows Update Refused to Work, So I Spoon-Fed it.

TL;WR: Windows update never got past checking for updates. Nothing worked. Used my private WSUS server and spoonfed it 10 updates at a time. Suggested Pairing: Rye.


Warning: The following was a holiday experiment turning into a WSUS bastardization rabbit hole. Real research would’ve involved some debugging. All this was probably related to a terminal WID or something else that real work would’ve resolved.


I have a laptop to fix. The poor thing was worked over by power Luser. The belligerently ignorant bastard left his mark on everything he could. He installed every browser, installed Avast (but left defender running… somehow), pretty much everything on ninite, pirated keys (even though he had the oem keys for it), rosetta stone with no lang packs, etc. etc.

The worst though was killing windows update. Something about the government using it to spy on people. Ironically he left telemetry on.

I digress. I tried everything on this computer. Windows update would stick on checking for updates and never return. It hadn’t been updated in two and a half years so I expected a delay, but not 20 hours.

A packet cap was showing that wuau was reaching the M$ servers and after 20 or so packets it received a 200 OK then just stopped responding. Then all of the tcp connections would timeout and close. Not. A. Peep. All it would do is poke a few reg values for the wuau gpo settings.

First up the update agent  (Here for win 8.1/ Server 2012 R2, watch for the prerequisites). Which in this case happened to be pretty much the last update it received before the Luserpocalypse.

I ran the Windows update diagnostic cab (Here) and each time it did its thing but the problem never went away. Stopping the windows update service (wuauserv) and deleting the \Windows\SoftwareDistribution\ folder (data store that catalogues the updates and stores update info) would get it started again but the same problem kept popping up.

After checking AV, other network apps, other file apps, running a procmon trace, sfc, dism, and little Christmas Drinking I decided to try one last thing before a clean boot.

I popped onto my home WSUS server, added a computer group for this poor laptop, and added the windows 8.1 updates to the catalogue.

  • I set up two empty computer groups.
  • Went through and picked out the updates I wanted, approved them for the first empty group to start the download
  • Blindly made the tweaks Here as per cargo-cult administration standards.
  • On the laptop I updated the GPO to point at my wsus server and added it to the second empty computer group.
  • As updates came in (starting with top of tree/cumulative monthlies) I added approvals for the second group in batches of 5 (8-10 at a time later).
  • Popped on the laptop and started the updates. THEY WORKED.

After all that I got cocky and decided to try approving 20 updates. No dice, I had to wipe the SoftwareDistribution cache again just to get it working. Finally I got it up to date, 8 updates at a time. Never did figure out why, it’s on the slate for a reinstall soonish. If you’ve got any ideas before I do the reinstall I’m open to taking a look, let me know. She’s long gone, few caps popped shortly after one of the LCD ffc’s gave out and it was relegated to the boneyard.

Console Access to Shoretel Switches and Phones

SG series switches, T series, and probably others:

Serial:

  • Connect with a standard serial cable (dte to dce)
  • 19200 N,8,1
  • User: anonymous
  • Password: ShoreTel

Telnet: (Skip this and use SSH for a V switch)

  • Login to your Shoreware Director (it has to be the Shoreware Director, as of version 13.something you could only connect from the SWD. May have changed in 14 but I haven’t bothered to check)
  • command prompt at %PROGRAMFILES(x86)%\Shoreline Communications\ShoreWare Server\
  • ipbxctl.exe -pw ShoreTel [-telneton | -telnetoff] x.x.x.x
  • (Putty: go to the Terminal -> Keyboard settings and change the backspace key to “Control-?”)
  • User: anonymous
  • Password:ShoreTel

On switches with long uptimes (or if you left a debug flag on for a long time, shame!) sometimes this just plain won’t work. Only console will until you reboot.

You should see the console at the bottom.

Entering “gotoshell” will drop you to their VxWorks shell. From here you can set kernel flags. Do so with caution. The most useful one I’ve used is “sip_debug_level = 2”. There are other debug flags but at that point may god have mercy on your soul.

For the brave souls out there, once you realize that it’s bastardized VxWorks you might be tempted to see what commands shoretel implemented or left on their image. This works on the phones too but others have explored that in much more detail.

https://danielkuchenski.wordpress.com/2014/03/23/telnet-commands-for-shoretel-switches-part-2/

 

 

Shoretel for Sadists (1xx-rel sip session ending on a SG-90)

TL;WR: SIP Provider changed their 1xx-rel implementation and removed an SDP from the final invite (Closer to RFC now). Shoregear switches can’t deal with that apparently (pfft-RFC? hah!). Suggested Matching: Earl-Grey & Bourbon.

Update: The Allstream tech’s enabled repeat SDP and outbound calls started working again, I changed our Ingate sip rules to set the CID in From instead of leaving it in P-Asserted-Ident. Shannon gave more details on the specifics of the upgrade in the comment section.


Is this relevant? If your packet cap shows invite, 100 trying, then a 183 session progress with “Require: 100rel” in the header. If your voice switch is sending ACK sip:… BYE sip:… after a 200 OK with an invite and no sdp.

Shoretel Voice Switch (SG-90, Director 14.2)
Ingate Siparator SBC (5.0.11)


Whelp, I’ve finally cracked, I got tired of digging through an endless stream of bookmarks and old tickets. What better way to document a descent into madness than starting with a SIP issue!

My current employers SIP provider decided that there was no way upgrading their SBC would affect their customers… so of course at 00:00 UTC January 18th 2017 all of our outbound calls started failing.

Outbound calls will fail after the far end answers. Sometimes a few second of media will get through before the call ends. This can happen because of a delay between the SG-90 sending bye and the media stream ending. Occasionally a call will work. We’ll go into why in a second.

What’s happening:
In our case the provider’s upgrade actually brought them closer to the RFC (https://tools.ietf.org/html/rfc6337#section-3.1.1).

You’ll see on their ascii flow diagram the following at packet F12:

SDP should not be included in the final response once offer/answer has been completed.

Now our provider was sending an SDP in packet F12. RFC says you shouldn’t without good reason but they were anyway. Not that I mind. After their upgrade they removed the redundant SDP. In my case the useful SDP’s were being sent in packet F1 and F6.

The SG-90 behind my Ingate was flowing as follows.

F1: Outbound Invite (With SDP)
F2: Inbound 100 Trying
F6: Inbound 183 Session Progress (with SDP & Require 100rel)
F7: Outbound PRACK
F8: Inbound 200 PRACK
F12: Inbound 200 Invite (now with no SDP)
F13: Outbound ACK

On packet F12 the far end has answered and the provider is asking me to join the media session. Unfortunately Shoretel isn’t a fan of not having the SDP on the invite. The SG-90 will dutifully Ack the Invite, then promptly send Bye and end the session.

If you console onto the SG-90 (here) you shouldn’t even need to do a sip trace (although you can here). At the end of the call you’ll see the following two errors:

ERROR : SIPIntf_sip.c:1828 : errorCode:(-13)Not_Found (sipintf_sip_dvc_call_answer)
WARNING : SIPIntf_RvCallLeg.c:781 : errorCode:(-13)Not_Found (SIPIntf_CallLegStateChangedEvHandler)

At the time of this writing (Jan 19th 2017) I can’t help you much further. I’m currently engaging Shoretel and the SIP provider for some further answers but I can’t Interop-magic my way out of this one without breaking some features. This isn’t a case where I can tell the SBC to just swallow a message or handle it on its own, the SG90 apparently needs an SDP on packet F12 and I can’t just generate one with an ingate.

Here’s where it stands: Allstream got bought by Zayo, Upgraded their SBC (which now conveniently no longer honors P-Asserted-Identity), and decided to change their SIP implementation. I doubt either of those is going to change soon. It’s not great when the alternative is to wait for Shoretel to make a firmware

I am not alone though, my lovely friend at Allstream tells me it’s quietly going to hell over there, so who knows.

Update 2019-02-21: After some recent changes I was speaking to an engineer and they now honor PAI again.