Category Archives: Uncategorized

VMM Migration Error 20413 (Hyper-V-VMMS 20770)

I was trying to migrate a VM from one of our less-used staging hosts when I started getting an exception at the Live Migration step.

----------------------------------------------------
--------------- Bucketing Parameters ---------------
----------------------------------------------------
EventType=VMM20
P1(appName)=vmmservice.exe
P2(appVersion)=4.0.2413.0
P3(assemblyName)=ImgLibEngine.dll
P4(assemblyVer)=4.0.2413.0
P5(methodName)=Microsoft.VirtualManager.Engine.ImageLibrary.HyperVHAVM.AddDiskResourceToVMFromFilePath
P6(exceptionType)=System.NullReferenceException
P7(callstackHash)=a724

SCVMM Version=4.0.2413.0

...

Base Exception Assembly name=ImgLibEngine.dll
Base Exception Method Name=Microsoft.VirtualManager.Engine.ImageLibrary.HyperVHAVM.AddDiskResourceToVMFromFilePath
Exception Message=Object reference not set to an instance of an object.

...

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VirtualManager.Engine.ImageLibrary.HyperVHAVM.AddDiskResourceToVMFromFilePath(String path, IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.Engine.VmOperations.DeployVmBase.MigrateVM(IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.Engine.VmOperations.DeployHost2Host.RunSubtask(IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.Engine.TaskRepository.SubtaskBase.Run(IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.DB.SqlContext.Connect(Action`1 action)
   at Microsoft.VirtualManager.Engine.TaskRepository.Task`1.SubtaskRun(Object state)

On the source host I saw a number of SMBClient errors with ID 30905:

The client cannot connect to the server due to a multichannel constraint registry setting.

Server name: \<TARGETHOST>

Guidance:
The client attempted to use SMB Multichannel, but an administrator has configured multichannel support to prevent multichannel on the client. You can configure SMB Multichannel on the client using the Windows PowerShell cmdlets: New-SmbMultichannelConstraint and Remove-SmbMultichannelConstraint.

Short answer was that the source server had some constraints (Get-SmbMultichannelConstraint) and I was in a position where I could just temporarily disable multichannel (Set-SmbClientConfiguration -EnableMultiChannel $false). Realistically the right answer would have been to validate the configuration and get it working correctly, but this host was up for decommissioning so we let it slide.

Error 0x8009030E Trying to Migrate VM in System Center VMM

Working with VMM 2016

Error (23008)
The VM BlahBlahBlah cannot be migrated to Host BlahHost.contoso.ads due to incompatibility issues. The Virtual Machine Management Service failed to establish a connection for a Virtual Machine migration with host 'BlahHost.contoso.ads': No credentials are available in the security package (0x8009030E).
  1. Double checked that hosts were setup with the correct Kerberos delegation settings (and set to Kerberos only, others say this doesn’t work but you just have to wait a few minutes after doing klist purge -li 0x3E7 to clear the computer account tickets on each host and it will start working)
  2. Double checked that our VMM management account was setup under Host Access > Host management credentials > Run As Account
  3. Double check that hosts are configured to use Kerberos as their Live Migration method

Zabbix HTTP Agent LLD Rule Example

UPDATE: The DPM part of this whole ordeal was invalidated by the recent addition of event publishing for DPM. If you can, get the update and just setup windows event monitors for backup actions.

Jump to Zabbix Item Examples

TL;DR: Built an asp.net API to query a DPM view and spit out JSON that Zabbix could handle for both discovery and data. Put this here because there weren’t many resources on the whole HTTP LLD deal.

Rough Draft, I built this whole project in about 5 hours, sue me. I imagine you’re here for the Zabbix HTTP Agent LLD stuff so I left the API part out. If you want the whole shebang (API, Code, Setup) let me know with a comment. I don’t want to a whole project if it’s just going to rot in my little corner of the internet.

We’ve been using DPM for our backups only to be thwarted in our monitoring attempts. We could have used Operations Manager but the problem was that we weren’t using OM for anything else. The only thing worse than an incomplete dashboard is two incomplete dashboards. So I bit the bullet and now we can finally monitor DPM with Zabbix.

DPM’s built in reporting was a royal pain and took too much manual review time. The email alerts were pretty much all or nothing and I’m loathe to contribute to alert blindness so I hammered this… thing… out.

I built a quick web API with two controllers, one to provide discovery data, and another for the details. The discovery URL (/api/DpmDiscovery/{HOST.NAME}) would hand back the LLD formatted JSON and the other URL (/api/DpmStatus/{#RECPOINT.BACKUPPATH}) would spit out details.

A call for https://dpmapi.contoso.ca/api/DpmDiscovery/file01 would return the following json.

{
    "data": [
        {
            "{#RECPOINT.STATUS}": 2,
            "{#RECPOINT.IDSN}": "D:\\",
            "{#RECPOINT.SERVERNAME}": "file01.contoso.ca",
            "{#RECPOINT.BACKUPPATH}": "file01.contoso.ca..D:..",
            "{#RECPOINT.CREATIONTIME}": "2019-04-25T00:05:28-06:00",
            "{#RECPOINT.UNIXTIME}": "1556172328"
        },
        {
            "{#RECPOINT.STATUS}": 2,
            "{#RECPOINT.IDSN}": "E:\\",
            "{#RECPOINT.SERVERNAME}": "file01.contoso.ca",
            "{#RECPOINT.BACKUPPATH}": "file01.contoso.ca..E:..",
            "{#RECPOINT.CREATIONTIME}": "2019-04-25T00:05:42-06:00",
            "{#RECPOINT.UNIXTIME}": "1556172342"
        },
        {
            "{#RECPOINT.STATUS}": 2,
            "{#RECPOINT.IDSN}": "System State",
            "{#RECPOINT.SERVERNAME}": "file01.contoso.ca",
            "{#RECPOINT.BACKUPPATH}": "file01.contoso.ca..System_State",
            "{#RECPOINT.CREATIONTIME}": "2019-04-25T02:10:59-06:00",
            "{#RECPOINT.UNIXTIME}": "1556179859"
        }
    ]
}

Then the LLD rule creates an HTTP Agent item to call https://dpmapi.contoso.ca/api/DpmStatus/file01.contoso.ca..System_State

{
    "status": 2,
    "interpretedDsn": "System State",
    "serverName": "file01.fireball.ads",
    "backupPath": "file01.fireball.ads..System_State",
    "creationTime": "2019-04-25T02:10:59-06:00",
    "unixCreationTime": "1556179859"
}

Technically my API returned the data as application/json; however I had accidentally checked “Convert To JSON” so you’ll see a body element in the JSON path below (e.g. $.body.status). In theory I could uncheck that and remove the body element. In practice it works as-is so it’ll stay that way for now.

Example Screenshots

Zabbix Discovery Rule

Zabbix Data Item

Dependent Item



These dependent items use JSON Path processing to extract the actual data out of my details response.

(body element was inserted because I had checked “Convert to JSON”)

Citations Nonsense:
DPM SQL View Documentation: https://docs.microsoft.com/en-us/previous-versions/system-center/data-protection-manager-2010/ff399120(v=technet.10)
Handy JSON validator: https://jsonformatter.curiousconcept.com/
Go-To JSON Browser: http://jsonviewer.stack.hu/

Dynamics CRM Plugin Mistake

Quick one (i.e. not the prettiest article): I was building another CRM plugin and kept getting a really annoying exception.

System.NullReferenceException: Microsoft Dynamics CRM has experienced an error.

But if I turned on profiling and tried to replay the plugin it would execute as expected. Turing to the CRM server event log I saw this:

ASP.NET event 1309
Exception information: 
    Exception type: NullReferenceException 
    Exception message: Object reference not set to an instance of an object.
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.IsLocalizedAttribute(AttributeMetadata attributeMetadata)
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.AppendDataValueJson(StringBuilder dataValues, String attributeLogicalName, Entity entity, FormMediator formMediator, Boolean encodeValues, IOrganizationContext context)
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.GetEntityAttributeJsonContent(Entity entity, FormMediator formMediator, Boolean encodeValues, IOrganizationContext context)
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.<EntityPropertiesToJsonInternal>d__3.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at Microsoft.Crm.Application.InlineEdit.InlineEditExtensionMethods.WriteSeparatedValues(TextWriter writer, IEnumerable`1 values, Char separator)
   at Microsoft.Crm.Application.InlineEdit.InlineEditJsonConverter.WriteEntityProperties(TextWriter writer, Entity entity, FormMediator formMediator, NotificationCollection notifications, PrivilegeCheck privilegeChecks, Boolean appendEntriesForFirstTimeLoad, Dictionary`2 parameters, Boolean encodeValues)
   at Microsoft.Crm.Application.InlineEdit.ReadFormDataBuilder.WriteFormDataJson(TextWriter writer)
   at Microsoft.Crm.Application.InlineEdit.ReadFormDataBuilder.WriteFormattedEntityData(TextWriter writer, Boolean isTurboForm)
   at Microsoft.Crm.Application.Pages.Form.FormDataPage.Render(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

I figured I was sending data that couldn’t be rendered. After going back and forth trying to debug I noticed that my attribute keys had a capital letter in the middle (i.e. “contoso_entity_customBlah”). That’s was an hour of my life because of a capital letter.

p.s. I noticed that sometimes when debugging the profiler would throw an uncatchable exception, but only if a debugger was attached. The debugger couldn’t detach once the exception was thrown.

I’d replay the plugin: no exception.

Attach the debugger and replay: see a caught exception! Then the plugin tool would crash due to an uncaught win32 exception. Of course I couldn’t debug the plugin tool because I already had a debugger attached, and I couldn’t detach the debugger because yadda yadda yadda. Turns out if you try to debug a sandboxed plugin in some circumstances the debugger in traceinternal tries to get fileiopermission and fails (because sandbox). So yeah, it was the debugger throwing an exception that it didn’t catch.

I ended up attaching the debugger, hitting a breakpoint, detaching the debugger, then reattaching the debugger after the plugin tool threw an exception. Of course the solution was to debug outside the sandbox.

Remote bulk fix for VSS LLDP CAPI 513 error.

I’m a stickler for keeping error logs clean where possible. I wanted to fix the VSS CAPI 513 error (https://support.microsoft.com/en-ca/help/3209092) on my DPM protected servers; however, I’m also lazy and didn’t want to do it manually. Here’s my quick and dirty powershell function to apply the fix to all of the appropriate servers.

Automation is a fantastic way to break things with unprecedented speed. Scripts should be understood before running.
Caveat Emptor.

function Repair-mslldpPermissions {

    param (

        [string]$TargetComputer

    )

 

    $mslldpSDDL = Invoke-Command -ComputerName $TargetComputer -ScriptBlock {sc.exe sdshow mslldp}

    $ntserviceSecString = ‘(A;;CCLCSWLOCRRC;;;SU)’

 

    if ($mslldpSDDL -match $ntserviceSecString) {

        Write-Warning “mslldp service already has NT Service permission fix applied on $TargetComputer!”

        return;

    }

 

    if ($mslldpSDDL -match “[OGS]:”) {

        Write-Error “I’m not smart enough to understand the SDDL on $TargetComputer.

        I expect the SDDL for this service to match the default, which only contains dacl flags.

        Make me smarter if you want to continue!” -Category InvalidOperation

    }

 

    $newSDDL = $mslldpSDDL$ntserviceSecString

    $output = Invoke-Command -ComputerName $TargetComputer -ScriptBlock {$sddl = $args[0]; sc.exe sdset mslldp $sddl} -ArgumentList $newSDDL

 

    switch -Wildcard ($output) {

        “*5*” {

            Write-Error “Insufficient permissions to alter SDDL of mslldp service. Failed to set SDDL” -Category PermissionDenied

            return;

        }

        “*SetServiceObjectSecurity SUCCESS*” {

            Write-Host “Successfully updated mslldp service SDDL”

            return;

        }

        Default {

            Write-Error “sc returned unexpected result:`n$output -RecommendedAction “RTFError” -Category InvalidResult

            return;

        }

    }

 

}

 

DPM Azure Recovery Services Agent Crashing

DPM 2016 deployments have been filling up my error logs with crash reports for the Microsoft Azure Recovery Services Management Agent. Turns out that’s the statistics agent for the Azure dashboards that don’t work on the LTSC releases of DPM (http://blog.teknikcs.ca/2019/02/21/dpm-protected-items-dont-appear-in-azure-vault/).

System Event ID: 7031 
The Microsoft Azure Recovery Services Management Agent service terminated unexpectedly
Application Event ID: 1000
Faulting application name: OBRecoveryServicesManagementAgent.exe
Application Event ID: 1026
Application: OBRecoveryServicesManagementAgent.exe
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
at .CTraceProvider.TraceToErrorFile(CTraceProvider, DLS_TRACE_EVENT)

Disable it if you’re on DPM 2016 or DPM 2012. No impact that we’ve seen.

Update: We did start having dependency issues after updating the MARS agent. It appears that the agent now depends on the management service. Not getting errors anymore though so we reset things back to normal.

Shoretel Users Can’t Change Call Handling Mode or Agent Status

TL;WR Probably the SG90 acting up again. Those things are weird. Rebooting the SG90 and the Director server fixed it for me. YMMV.

While moving around some VM’s we had a Shoretel Director server running without a network connection for 4 hours during a maintenance window. Afterwards users couldn’t change their Call Handling Modes or change their agent logged in/out status. It failed from both the phones and from communicator. At first I thought it was a CAS problem; however the phone directory, history, options, and speed dial features were all working correctly.

I popped up the IPDSCASCfgTool (see bottom) to set the log levels for the CAS to include all the DB and CAS flags for a start. After that I used powershell to stream the logs with Get-Content. I use Measure-Object first to grab the line count of the file so that we can skip the first 393,000 lines straight to the live output. That’ll work like tail -f in linux and just continuously stream the logs to the console.

Note: You SHOULD be able to use Get-Content -Wait -Tail <Number of Lines> to skip to the end, but that wasn’t working on this particular server.

PS C:\Shoreline Data\Logs> Get-Content .\ipds-190225.000000.Log | measure
Count : 393693
Average :
Sum :
Maximum :
Minimum :
Property :
PS C:\Shoreline Data\Logs> Get-Content .\ipds-190225.000000.Log -Wait | where -Property ReadCount -gt 393700
17:49:28.837 ( 3264: 3512) >SetUserCHM. User: 123. CHM: 2
17:49:28.888 ( 3264: 3512)
15:52:01.574 ( 7508: 5168) >CDBWriter::SetUserCHM::CDBUpdateTable::Update() failed. Error: 0xc1200db5.

SetUserCHM was me (unsuccessfully) changing from CHM 1 (Standard) to CHM 2 (In a Meeting) from communicator (testing from a phone will also log here but it’s noisier). That error sent me off looking for database issues, communications problems, etc. No dice. The evt log showed some interesting output though:

 
15:55:17.013 ( 4080: 4476) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.029 ( 7508: 4036) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.183 ( 4080: 4436) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.183 ( 2992: 6552) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.183 ( 1764: 1856) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C
15:55:17.187 ( 4972: 5232) [evtl] (Error) CEventLibImpl::sendReceiveIPC failed - 0xC126100C

After digging around for named pipe issues and doing traces I tried the same on the voice switch and didn’t see any interesting errors. Theoretically the phone’s button and control traffic hits the voice switch and is making it to the director but I couldn’t verify that because I couldn’t get the switch to start a packet cap. Supposedly that’s because of a cipher mismatch: the director server tries to ssh into the voice switch to start the packet cap but it fails to login using it’s certificate.

Anyway I ran out of ideas and just waited until I could restart the switch and it worked. Everything was fine. Once again the SG90 blew up in my face was behaving unexpectedly and sent me one step closer to trying the vswitch.

Another one of those 1/1 google searches:
IPDSCASCfgTool (https://oneview.mitel.com/s/article/How-to-set-the-Log-Levels-for-IPDS-CAS-on-Shoreware-Servers)

DPM Protected Items Don’t Appear in Azure Vault

TL;WR: None of the current Long Term (i.e. 2012 R2/2016) DPM releases actually send updates to azure. So unless you’re on the bleeding edge DPM stream you don’t get to see this stuff. Suggested Pairing: Two drawn-out eye rolls.

Simple enough problem, I was looking at the Recovery Services vault page for a DPM installation and noticed all these nice dashboards for alerts, items, jobs, etc. All of them were empty.

  • 0 Backup Items
  • 0 Backup Jobs
  • 1 Azure Backup Agent… with item count 0

After a blazing fast support call I learned that this feature currently doesn’t work with any released version of DPM (Microsoft’s flagship backup software in case you forgot). According to support the feature won’t be implemented until DPM 2019 is released (due March 2019).

Semi-Related: On our last deployment it really annoyed me to find out that no 2016 version of DPM supported server 2019. Again… Msft’s Flagship Backup Software… and it doesn’t support the latest version of the server OS. I get that delaying 5 months between GA launch and integrating with system center doesn’t seem that crazy, but it certainly was annoying to find out that the brownfield deployment simply wouldn’t be able to use server 2019 without buying a new set of DPM licenses.

Yeah yeah “DPM 2016 is more than 8 days old so obviously it won’t protect new versions of windows.” Whatever.

Cannot disconnect windows server iSCSI sessions when you ignore your own advice

TL;WR: If you can’t eject a disk and you have apps open, try closing them! Duh. Suggested Pairing: A third of your remaining tea/coffee vessel.

Wasted 15 minutes of my life today trying to disconnect two iSCSI sessions on a windows server 2012 R2 hyper-v host. Kept getting “This session cannot be logged out since a device on that session is currently being used”. Pulled up Process Explorer looking for handles on the disks (searching MPIO in my case because we were using MPIO). Lo and behold our task manager had open handles on the disks.

I immediately realized that this particular test server had disk perf monitoring running (DISKPERF -Y). That puts basic disk performance counters into task manager and of course I had task manager open in the background. While that’s handy for a test server, it’s not recommended in production for performance reasons and things like this. Solution was to follow my own advice and close background apps when troubleshooting access problems. Do as I say etc.

Powershell Notes to Self

Inline text updating:
Lazy way of having a progress indicator that doesn’t fill the whole screen history and doesn’t require Write-Progress. Use NoNewline to avoid implicit newline/carriage return, then manually add a carriage return (`r) at the start of your line to overwrite existing text.

Write-Host "`r$TimestampOrWhatever: $ValueOrWhatever" -NoNewline