All posts by ThingBreaker

Office 365/Apps for enterprise sign-in page blank

  • Are you having fun with blank sign-in screens for basically any Microsoft app that uses the web authentication pop up?
  • Do you see “©yyyy Microsoft Privacy & Cookies” at the bottom of a white/blank sign in popup?
  • Is it annoying the hell out of you?
  • Did you look at every @#!$ page in the O365 Administration Troubleshooting Site?
  • Did you reset activation and use the sign in troubleshooter/SaRA?
  • Did you then do the inetcpl reset?
  • Are you aware of (and annoyed by) the wrong, outdated, and or irrelevant advice to disable ADAL, WAM, etc.?
  • Did you DISM and SFC? Just kidding.

If the answer to all of the above questions is yes, then you’re as desperate as I was and there’s a small chance I have the answer for you! I’m supposed to advise you to do the ‘supported thing’ which is reinstall windows… but if you’re reading this then you probably don’t care!

Fortunately for you this isn’t Microsoft “Answers” and I have two ways to get the page working: skip to Option A or B if you’re feeling bold.

Background

At first I thought that maybe it was a JavaScript issue, maybe bad zone security settings, etc. I used fiddler to scrape the URL that office was trying to load for the sign in and found something along the lines of “https://odc.officeapps.live.com/odc/v2.1/hrd” (plus a bunch of queries) and bingo, that was the page that was trying to load. Works in every browser *except* IE… which happens to be the engine that O365 apps use to render the login page.

Digging further I saw a bunch of strange issues in the JS console and on a whim changed the emulation settings. You can test this yourself pretty easily:

  1. iexplore.exe https://odc.officeapps.live.com/odc/v2.1/hrd
  2. The page should be nothing but a color gradient and a little footer
  3. Open devtools (F12)
  4. Go to the emulation tab and change Document Mode to 11

If the prompt suddenly works then I think we’re on the same page😉! If you look in the source you can see that Microsoft specifies that it wants IE to use IE10 mode with the x-ua-compatible tag. I’ll be damned if I know why they set that, but I *do* know that IE11 mode is the one that actually works.

So now we’re sophomores: We’ve learned that IE is trying (and failing) to render this in IE10 mode and that it works correctly in Edge mode. We are still foolish as to the reason this is only happening to us and not a few million other companies (my first guess would probably be something vague like “Borked IE registry”). Fortunately we’re now wise to a potential workaround: force the embedded IE window to render the page in Edge mode.

Option A) FeatureControl Registry “Fix”

We’re going to use a registry flag as per MSDN to tell IE that it should ignore all hints and render everything that your target application requests in IE11 mode.

  1. Pick an app you’re going to use for sign in (OneNote, Word, Excel, etc.)
  2. Go to HKLM\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BROWSER_EMULATION
  3. Create a new REG_DWORD with the name of your chosen app (e.g. OneNote.exe), set the value to 0x00002af9 (11001).
  4. Relaunch your app and sign in! Congrats!
  5. GO BACK AND DELETE THE KEY! Don’t be lazy, do it now, otherwise you’re going to tear your hair out when something dumb breaks in the future and you’ve forgotten all about this.

Option B) Ugly Fiddler Hack“Fix”

Well, just use fiddler and edit the contents of the page before it lands in the target application. We’re going to change the hint from content="ie=10" to content="EdgeHTMLwhich will tell IE to render the page in IE11/Edge mode.

  1. Open fiddler with HTTPS interception enabled.
  2. Open the FiddlerScript tab, then go to the OnBeforeResponse function.
  3. Add this line to the function oSession.utilReplaceInResponse("ie=10","EdgeHTML");
  4. Relaunch your app and sign in! More congrats!
  5. GO BACK AND REMOVE THAT LINE FROM THE FIDDLERSCRIPT TAB! We both know that you’ll forget and the next time you’re using fiddler you’ll waste 3 hours of troubleshooting before you remember that this line is running. It has no conditions or anything, it’s what you might generously call enthusiastic programming.

One one hand this doesn’t fix the root cause because I don’t know what the root cause ACTUALLY IS. On the other hand it does finally let you sign into your 365/Visual Studio apps. If you have any ideas what the heck is going on please do share!

VBA Dumb Problem of the Day

Sure, many of us loathe VBA, but we still have to support it. I was trying to update an Excel macro that configured a print layout and suddenly I was unable to insert page breaks. “It was working a second ago”.

Run-time error '1004': unable to set the PageBreak property of the Range class

Turns out that when applied to my test data the PrintArea of the print layout was set so that the first row I was trying to add a page break to was also the first row of the PrintArea. I guess it makes sense but that isn’t documented anywhere.

So if you’ve been trying a dozen different ways of adding a page break, do yourself a favour and double check the area selected for printing.

Scheduled Powershell Scripts without storing credentials

Sometimes I want to schedule a script to run with specific domain credentials without storing anything blatantly risky. For example here I wanted to schedule maintenance notifications to users that have been logged in for so long that their hosts are up for replacement…

Easiest way is to set up a GMSA and use it for the scheduled tasks, the only caveat is that you can’t select a G/MSA account for those tasks in the task scheduler UI. Hmm.

The workaround is to set the task account from either SC or PowerShell; however I also wanted to script the rest of the task setup. Nobody likes instructions that involve “Do a dozen things by hand in the UI, then write some lines to modify it”.

The next step to low footprint bliss would be to say goodbye to all the files and ACLs, let’s just inline the scripts (if they’re short enough)! So here’s my script to create a file-less, credential-less (kind of) PowerShell scheduled task that in this case schedules desktop messages for Citrix sessions. Ironically this specific example drops transcript copies but you get the point.

#Encode script as Base64, send a Citrix message in this example
function EncodeMessageTaskScript ($MessageText, $AdminAddress)
{
    $TaskScript = @"
    Start-Transcript "C:\ScriptLogs\ScheduledPS.log" -Append
    & { Add-Pssnapin @('Citrix.Host.Admin.V2′,'Citrix.Broker.Admin.V2′)}

    `$CurrentSessions = Get-BrokerSession -AdminAddress "$AdminAddress" -MaxRecordCount 1000 | ? DesktopGroupName -eq 'Nope'
    `$CurrentSessions | % { Send-BrokerSessionMessage -AdminAddress "$AdminAddress" -InputObject `$_ -MessageStyle Critical -Text "$MessageText"`nMessage Sent [`$(Get-Date)]" -Title "Maintenance Warning"}

    Stop-Transcript

"@#Let's pretend that this is indented and that you can't end a herestring with whitespace...

    [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($TaskScript))
}

function CreateScheduledGMSATask ($EncodedPsScript, [datetime]$TriggerDateTime, $TaskName)
{

    $Action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-EncodedCommand `"$EncodedPsScript`" -NoLogo -NoProfile -ExecutionPolicy Bypass"
    $Principal = New-ScheduledTaskPrincipal -LogonType Password -RunLevel Limited -UserId 'DOMAIN\[G]MSA$'
    $Settings = New-ScheduledTaskSettingsSet -Compatibility Win8
    $Trigger = New-ScheduledTaskTrigger -Once -At $TriggerDateTime

    $TaskObj = New-ScheduledTask -Action $Action -Principal $Principal -Trigger $Trigger -Settings $Settings
    Register-ScheduledTask -TaskName $TaskName -InputObject $TaskObj

}

$EncodedReminderTask = EncodeMessageTaskScript -MessageText "Your Message Text" -AdminAddress $CitrixAdminAddress
CreateScheduledGMSATask -EncodedPsScript $EncodedReminderTask -TriggerDateTime $MaintReminder -TaskName "MessageTask $($MaintStartTime.ToString('yyyyMMdd.HHmmss'))"

DPM Scripted VM Recovery Fails (Error 104 0x80041002, 3111)

-RecoveryType

Specifies the recovery type. If you specify the HyperVDatasource parameter, the only valid value is Recover. The acceptable values for this parameter are: Recover or Restore.

https://docs.microsoft.com/en-us/powershell/module/dataprotectionmanager/new-dpmrecoveryoption?view=systemcenter-ps-2019

The Microsoft documentation is flat out wrong. It very explicitly states that the only valid RecoveryType for HyperVDatasource is Recover. When trying to recover to an alternate disk location their example does not work. Based on the example script you would expect the code below to work. Instead if you try it you’ll get a powershell error stating “The recovery point location that you have passed is invalid. Please try again with a different value (ID:31050).”

$BadOption = New-DPMRecoveryOption -HyperVDatasource -TargetServer "target.contoso.com" -RecoveryLocation CopyToFolder -RecoveryType Recover -TargetLocation "D:\DestinationFolder"

So maybe instead of that you’d google around then try the following, and that appears to work… At first.

$BadOption = New-DPMRecoveryOption -HyperVDatasource -TargetServer "target.contoso.com" -RecoveryLocation AlternateHyperVServer -RecoveryType Recover -TargetLocation "D:\DestinationFolder"

However at some point that job will fail with “An unexpected error occurred while the job was running. (ID 104 Details: Unknown error (0x80041002) (0x80041002))” which is entirely unhelpful. If you go to the job details you’ll get an equally unhelpful error 3111. Making some assumptions around that error code (WMI object not found error) I’m thinking that it’s trying to import the VM to a hyper-v instance running on that server. That doesn’t work if there’s no valid hypervisor running. Instead you need to user the parameters -RecoveryLocation CopyToFolder and -RecoveryType Restore.

$WorkingOption = New-DPMRecoveryOption -HyperVDatasource -TargetServer "target.contoso.com" -RecoveryLocation CopyToFolder -RecoveryType Restore -TargetLocation "D:\DestinationFolder"

Miscellanea

Archived Ten Laws of Security 2.0 (https://web.archive.org/web/20180529154650/https://technet.microsoft.com/en-us/library/hh278941.aspx)

Archived Ten Laws response (https://web.archive.org/web/20190928204316/http://www.edgeblog.net/2006/10-new-immutable-laws-of-it-security/)

Archived Ten Laws Re-Review (https://web.archive.org/web/20190710001511/https://docs.microsoft.com/en-us/previous-versions/technet-magazine/cc895640(v=msdn.10))

Krebs on Security (https://krebsonsecurity.com/)

Raymond Chen’s Blog (https://devblogs.microsoft.com/oldnewthing/)

Barracuda Spam Firewall Rooting (http://blog.shiraj.com/2009/09/barracuda-spam-firewall-root-password/)

Group Policy team blog (https://blogs.technet.microsoft.com/grouppolicy/)

Aaron Stebner’s Weblog (notes on .Net) (https://docs.microsoft.com/en-ca/archive/blogs/astebner/)

AskPerf Ask The Performance Team (https://aka.ms/AskPerf)

AskDS Ask the Directory Services Team (https://aka.ms/AskDS) (Archive: https://docs.microsoft.com/en-ca/archive/blogs/askds/) (A lot of interesting deep dives on ESE)

Thomas Maurer’s Blog (Azure Advocate) (https://www.thomasmaurer.ch/)

Carl Stalhood’s EUC Blog (https://www.carlstalhood.com/)

Robin Hobo (https://www.robinhobo.com/)

Helge Klein’s Blog (https://helgeklein.com/)

Brent Ozar’s Corp Blog (https://www.brentozar.com/)

DBA Reactions (Lighthearted fun) (https://dbareactions.com/)

VMM Migration Error 20413 (Hyper-V-VMMS 20770)

I was trying to migrate a VM from one of our less-used staging hosts when I started getting an exception at the Live Migration step.

----------------------------------------------------
--------------- Bucketing Parameters ---------------
----------------------------------------------------
EventType=VMM20
P1(appName)=vmmservice.exe
P2(appVersion)=4.0.2413.0
P3(assemblyName)=ImgLibEngine.dll
P4(assemblyVer)=4.0.2413.0
P5(methodName)=Microsoft.VirtualManager.Engine.ImageLibrary.HyperVHAVM.AddDiskResourceToVMFromFilePath
P6(exceptionType)=System.NullReferenceException
P7(callstackHash)=a724

SCVMM Version=4.0.2413.0

...

Base Exception Assembly name=ImgLibEngine.dll
Base Exception Method Name=Microsoft.VirtualManager.Engine.ImageLibrary.HyperVHAVM.AddDiskResourceToVMFromFilePath
Exception Message=Object reference not set to an instance of an object.

...

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VirtualManager.Engine.ImageLibrary.HyperVHAVM.AddDiskResourceToVMFromFilePath(String path, IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.Engine.VmOperations.DeployVmBase.MigrateVM(IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.Engine.VmOperations.DeployHost2Host.RunSubtask(IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.Engine.TaskRepository.SubtaskBase.Run(IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.DB.SqlContext.Connect(Action`1 action)
   at Microsoft.VirtualManager.Engine.TaskRepository.Task`1.SubtaskRun(Object state)

On the source host I saw a number of SMBClient errors with ID 30905:

The client cannot connect to the server due to a multichannel constraint registry setting.

Server name: \<TARGETHOST>

Guidance:
The client attempted to use SMB Multichannel, but an administrator has configured multichannel support to prevent multichannel on the client. You can configure SMB Multichannel on the client using the Windows PowerShell cmdlets: New-SmbMultichannelConstraint and Remove-SmbMultichannelConstraint.

Short answer was that the source server had some constraints (Get-SmbMultichannelConstraint) and I was in a position where I could just temporarily disable multichannel (Set-SmbClientConfiguration -EnableMultiChannel $false). Realistically the right answer would have been to validate the configuration and get it working correctly, but this host was up for decommissioning so we let it slide.

Error 0x8009030E Trying to Migrate VM in System Center VMM

Working with VMM 2016

Error (23008)
The VM BlahBlahBlah cannot be migrated to Host BlahHost.contoso.ads due to incompatibility issues. The Virtual Machine Management Service failed to establish a connection for a Virtual Machine migration with host 'BlahHost.contoso.ads': No credentials are available in the security package (0x8009030E).
  1. Double checked that hosts were setup with the correct Kerberos delegation settings (and set to Kerberos only, others say this doesn’t work but I *think* you just have to wait a few minutes after doing klist purge -li 0x3E7 to clear the computer account tickets on each host and it will start working)
  2. Double checked that our VMM management account was setup under Host Access > Host management credentials > Run As Account
  3. Double check that hosts are configured to use Kerberos as their Live Migration method

Zabbix HTTP Agent LLD Rule Example

UPDATE: The DPM part of this whole ordeal was partially invalidated by the recent addition of event publishing for DPM. If you can, get the update and just setup windows event monitors for backup actions. Although in all honesty I don’t think I’d trust DPM’s events for critical monitoring.

Jump to Zabbix Item Examples

TL;DR: Built an asp.net API to query a DPM view and spit out JSON that Zabbix could handle for both discovery and data. Put this here because there weren’t many resources on the whole HTTP LLD deal.

Rough Draft, I built this whole project in about 5 hours. I imagine you’re here for the Zabbix HTTP Agent LLD stuff so I left the API part out. If you want the whole shebang (API, Code, Setup) let me know with a comment. I don’t want to clean up a whole project if it’s just going to rot in my corner of the internet.

We’ve been using DPM for our backups only to be thwarted in our monitoring attempts. We could have used Operations Manager but the problem was that we weren’t using OM for anything else. The only thing worse than an incomplete dashboard is two incomplete dashboards. So I bit the bullet and now we can finally monitor DPM with Zabbix.

DPM’s built in reporting was a royal pain and took too much manual review time. The email alerts were pretty much all or nothing and I’m loathe to contribute to alert blindness so I hammered this… thing… out.

I built a quick web API with two controllers, one to provide discovery data, and another for the details. The discovery URL (/api/DpmDiscovery/{HOST.NAME}) would hand back the LLD formatted JSON and the other URL (/api/DpmStatus/{#RECPOINT.BACKUPPATH}) would spit out details.

A call for https://dpmapi.contoso.ca/api/DpmDiscovery/file01 would return the following json.

{
    "data": [
        {
            "{#RECPOINT.STATUS}": 2,
            "{#RECPOINT.IDSN}": "D:\\",
            "{#RECPOINT.SERVERNAME}": "file01.contoso.ca",
            "{#RECPOINT.BACKUPPATH}": "file01.contoso.ca..D:..",
            "{#RECPOINT.CREATIONTIME}": "2019-04-25T00:05:28-06:00",
            "{#RECPOINT.UNIXTIME}": "1556172328"
        },
        {
            "{#RECPOINT.STATUS}": 2,
            "{#RECPOINT.IDSN}": "E:\\",
            "{#RECPOINT.SERVERNAME}": "file01.contoso.ca",
            "{#RECPOINT.BACKUPPATH}": "file01.contoso.ca..E:..",
            "{#RECPOINT.CREATIONTIME}": "2019-04-25T00:05:42-06:00",
            "{#RECPOINT.UNIXTIME}": "1556172342"
        },
        {
            "{#RECPOINT.STATUS}": 2,
            "{#RECPOINT.IDSN}": "System State",
            "{#RECPOINT.SERVERNAME}": "file01.contoso.ca",
            "{#RECPOINT.BACKUPPATH}": "file01.contoso.ca..System_State",
            "{#RECPOINT.CREATIONTIME}": "2019-04-25T02:10:59-06:00",
            "{#RECPOINT.UNIXTIME}": "1556179859"
        }
    ]
}

Then the LLD rule creates an HTTP Agent item to call https://dpmapi.contoso.ca/api/DpmStatus/file01.contoso.ca..System_State

{
    "status": 2,
    "interpretedDsn": "System State",
    "serverName": "file01.contoso.ads",
    "backupPath": "file01.contoso.ads..System_State",
    "creationTime": "2019-04-25T02:10:59-06:00",
    "unixCreationTime": "1556179859"
}

Technically my API returned the data as application/json; however I had accidentally checked “Convert To JSON” so you’ll see a body element in the JSON path below (e.g. $.body.status). In theory I could uncheck that and remove the body element. In practice it works as-is so it’ll stay that way for now.

Example Screenshots

Zabbix Discovery Rule

Zabbix Data Item

Dependent Item



These dependent items use JSON Path processing to extract the actual data out of my details response.

(body element was inserted because I had checked “Convert to JSON”)

Citations Nonsense:
DPM SQL View Documentation: https://docs.microsoft.com/en-us/previous-versions/system-center/data-protection-manager-2010/ff399120(v=technet.10)
Handy JSON validator: https://jsonformatter.curiousconcept.com/
Go-To JSON Browser: http://jsonviewer.stack.hu/

Dynamics CRM Plugin Mistake

Quick one (i.e. not the prettiest article): I was building another CRM plugin and kept getting a really annoying exception. Followed by an uncatchable exception.

System.NullReferenceException: Microsoft Dynamics CRM has experienced an error.

Useful, I know. If I turned on profiling and tried to replay the plugin it would execute as expected. Turing to the CRM server event log I saw this:

ASP.NET event 1309
Exception information: 
    Exception type: NullReferenceException 
    Exception message: Object reference not set to an instance of an object.
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.IsLocalizedAttribute(AttributeMetadata attributeMetadata)
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.AppendDataValueJson(StringBuilder dataValues, String attributeLogicalName, Entity entity, FormMediator formMediator, Boolean encodeValues, IOrganizationContext context)
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.GetEntityAttributeJsonContent(Entity entity, FormMediator formMediator, Boolean encodeValues, IOrganizationContext context)
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.<EntityPropertiesToJsonInternal>d__3.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at Microsoft.Crm.Application.InlineEdit.InlineEditExtensionMethods.WriteSeparatedValues(TextWriter writer, IEnumerable`1 values, Char separator)
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.WriteEntityProperties(TextWriter writer, Entity entity, FormMediator formMediator, NotificationCollection notifications, PrivilegeCheck privilegeChecks, Boolean appendEntriesForFirstTimeLoad, Dictionary`2 parameters, Boolean encodeValues)
   at Microsoft.Crm.Application.InlineEdit.ReadFormDataBuilder.WriteFormDataJson(TextWriter writer)
   at Microsoft.Crm.Application.InlineEdit.ReadFormDataBuilder.WriteFormattedEntityData(TextWriter writer, Boolean isTurboForm)
   at Microsoft.Crm.Application.Pages.Form.FormDataPage.Render(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

I figured I was sending data that couldn’t be rendered. After going back and forth trying to debug I noticed that my attribute keys had a capital letter in the middle (i.e. “contoso_entity_customBlah”). That’s was an hour of my life because of a capital letter.

p.s. I noticed that sometimes when debugging the profiler would throw an uncatchable exception, but only if a debugger was attached. The debugger couldn’t detach once the exception was thrown.

I’d replay the plugin: no exception.

Attach the debugger and replay: see a caught exception! Then the plugin tool would crash due to an uncaught win32 exception. Of course I couldn’t debug the plugin tool because I already had a debugger attached, and I couldn’t detach the debugger because yadda yadda yadda. Turns out if you try to debug a sandboxed plugin in some circumstances the debugger in traceinternal tries to get fileiopermission and fails (because sandbox). So yeah, it was the debugger throwing an exception that it didn’t catch.

I ended up attaching the debugger, hitting a breakpoint, detaching the debugger, then reattaching the debugger after the plugin tool threw an exception. Of course the solution was to debug outside the sandbox.

Remote bulk fix for VSS LLDP CAPI 513 error.

I’m a stickler for keeping error logs clean where possible. I wanted to fix the VSS CAPI 513 error (https://support.microsoft.com/en-ca/help/3209092) on my DPM protected servers; however, I’m also lazy efficient and didn’t want to do it manually. Here’s my quick and dirty powershell function to apply the fix to all of the appropriate servers.

Automation is a fantastic way to break things with unprecedented speed. Scripts should be understood before running. Also all the error decorations aren’t necessary, but who’s to say I can’t have fun with a blog post?
Caveat Emptor.

function Repair-mslldpPermissions {

    param (

        [string]$TargetComputer

    )

 

    $mslldpSDDL = Invoke-Command -ComputerName $TargetComputer -ScriptBlock {sc.exe sdshow mslldp}

    $ntserviceSecString = ‘(A;;CCLCSWLOCRRC;;;SU)’

 

    if ($mslldpSDDL -match $ntserviceSecString) {

        Write-Warning “mslldp service already has NT Service permission fix applied on $TargetComputer!”

        return;

    }

 

    if ($mslldpSDDL -match “[OGS]:”) {

        Write-Error “I’m not smart enough to understand the SDDL on $TargetComputer.

        I expect the SDDL for this service to match the default, which only contains dacl flags.

        Make me smarter if you want to continue!” -Category InvalidOperation

    }

 

    $newSDDL = $mslldpSDDL$ntserviceSecString

    $output = Invoke-Command -ComputerName $TargetComputer -ScriptBlock {$sddl = $args[0]; sc.exe sdset mslldp $sddl} -ArgumentList $newSDDL

 

    switch -Wildcard ($output) {

        “*5*” {

            Write-Error “Insufficient permissions to alter SDDL of mslldp service. Failed to set SDDL” -Category PermissionDenied

            return;

        }

        “*SetServiceObjectSecurity SUCCESS*” {

            Write-Host “Successfully updated mslldp service SDDL”

            return;

        }

        Default {

            Write-Error “sc returned unexpected result:`n$output -RecommendedAction “RTError” -Category InvalidResult

            return;

        }

    }

 

}