Category Archives: System Center

DPM Scripted VM Recovery Fails (Error 104 0x80041002, 3111)

-RecoveryType

Specifies the recovery type. If you specify the HyperVDatasource parameter, the only valid value is Recover. The acceptable values for this parameter are: Recover or Restore.

https://docs.microsoft.com/en-us/powershell/module/dataprotectionmanager/new-dpmrecoveryoption?view=systemcenter-ps-2019

The Microsoft documentation is flat out wrong. It very explicitly states that the only valid RecoveryType for HyperVDatasource is Recover. When trying to recover to an alternate disk location their example does not work. Based on the example script you would expect the code below to work. Instead if you try it you’ll get a powershell error stating “The recovery point location that you have passed is invalid. Please try again with a different value (ID:31050).”

$BadOption = New-DPMRecoveryOption -HyperVDatasource -TargetServer "target.contoso.com" -RecoveryLocation CopyToFolder -RecoveryType Recover -TargetLocation "D:\DestinationFolder"

So maybe instead of that you’d google around then try the following, and that appears to work… At first.

$BadOption = New-DPMRecoveryOption -HyperVDatasource -TargetServer "target.contoso.com" -RecoveryLocation AlternateHyperVServer -RecoveryType Recover -TargetLocation "D:\DestinationFolder"

However at some point that job will fail with “An unexpected error occurred while the job was running. (ID 104 Details: Unknown error (0x80041002) (0x80041002))” which is entirely unhelpful. If you go to the job details you’ll get an equally unhelpful error 3111. Making some assumptions around that error code (WMI object not found error) I’m thinking that it’s trying to import the VM to a hyper-v instance running on that server. That doesn’t work if there’s no valid hypervisor running. Instead you need to user the parameters -RecoveryLocation CopyToFolder and -RecoveryType Restore.

$WorkingOption = New-DPMRecoveryOption -HyperVDatasource -TargetServer "target.contoso.com" -RecoveryLocation CopyToFolder -RecoveryType Restore -TargetLocation "D:\DestinationFolder"

VMM Migration Error 20413 (Hyper-V-VMMS 20770)

I was trying to migrate a VM from one of our less-used staging hosts when I started getting an exception at the Live Migration step.

----------------------------------------------------
--------------- Bucketing Parameters ---------------
----------------------------------------------------
EventType=VMM20
P1(appName)=vmmservice.exe
P2(appVersion)=4.0.2413.0
P3(assemblyName)=ImgLibEngine.dll
P4(assemblyVer)=4.0.2413.0
P5(methodName)=Microsoft.VirtualManager.Engine.ImageLibrary.HyperVHAVM.AddDiskResourceToVMFromFilePath
P6(exceptionType)=System.NullReferenceException
P7(callstackHash)=a724

SCVMM Version=4.0.2413.0

...

Base Exception Assembly name=ImgLibEngine.dll
Base Exception Method Name=Microsoft.VirtualManager.Engine.ImageLibrary.HyperVHAVM.AddDiskResourceToVMFromFilePath
Exception Message=Object reference not set to an instance of an object.

...

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VirtualManager.Engine.ImageLibrary.HyperVHAVM.AddDiskResourceToVMFromFilePath(String path, IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.Engine.VmOperations.DeployVmBase.MigrateVM(IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.Engine.VmOperations.DeployHost2Host.RunSubtask(IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.Engine.TaskRepository.SubtaskBase.Run(IVmmDbConnection dbConnection)
   at Microsoft.VirtualManager.DB.SqlContext.Connect(Action`1 action)
   at Microsoft.VirtualManager.Engine.TaskRepository.Task`1.SubtaskRun(Object state)

On the source host I saw a number of SMBClient errors with ID 30905:

The client cannot connect to the server due to a multichannel constraint registry setting.

Server name: \<TARGETHOST>

Guidance:
The client attempted to use SMB Multichannel, but an administrator has configured multichannel support to prevent multichannel on the client. You can configure SMB Multichannel on the client using the Windows PowerShell cmdlets: New-SmbMultichannelConstraint and Remove-SmbMultichannelConstraint.

Short answer was that the source server had some constraints (Get-SmbMultichannelConstraint) and I was in a position where I could just temporarily disable multichannel (Set-SmbClientConfiguration -EnableMultiChannel $false). Realistically the right answer would have been to validate the configuration and get it working correctly, but this host was up for decommissioning so we let it slide.

Error 0x8009030E Trying to Migrate VM in System Center VMM

Working with VMM 2016

Error (23008)
The VM BlahBlahBlah cannot be migrated to Host BlahHost.contoso.ads due to incompatibility issues. The Virtual Machine Management Service failed to establish a connection for a Virtual Machine migration with host 'BlahHost.contoso.ads': No credentials are available in the security package (0x8009030E).
  1. Double checked that hosts were setup with the correct Kerberos delegation settings (and set to Kerberos only, others say this doesn’t work but I *think* you just have to wait a few minutes after doing klist purge -li 0x3E7 to clear the computer account tickets on each host and it will start working)
  2. Double checked that our VMM management account was setup under Host Access > Host management credentials > Run As Account
  3. Double check that hosts are configured to use Kerberos as their Live Migration method

Zabbix HTTP Agent LLD Rule Example

UPDATE: The DPM part of this whole ordeal was partially invalidated by the recent addition of event publishing for DPM. If you can, get the update and just setup windows event monitors for backup actions. Although in all honesty I don’t think I’d trust DPM’s events for critical monitoring.

Jump to Zabbix Item Examples

TL;DR: Built an asp.net API to query a DPM view and spit out JSON that Zabbix could handle for both discovery and data. Put this here because there weren’t many resources on the whole HTTP LLD deal.

Rough Draft, I built this whole project in about 5 hours. I imagine you’re here for the Zabbix HTTP Agent LLD stuff so I left the API part out. If you want the whole shebang (API, Code, Setup) let me know with a comment. I don’t want to clean up a whole project if it’s just going to rot in my corner of the internet.

We’ve been using DPM for our backups only to be thwarted in our monitoring attempts. We could have used Operations Manager but the problem was that we weren’t using OM for anything else. The only thing worse than an incomplete dashboard is two incomplete dashboards. So I bit the bullet and now we can finally monitor DPM with Zabbix.

DPM’s built in reporting was a royal pain and took too much manual review time. The email alerts were pretty much all or nothing and I’m loathe to contribute to alert blindness so I hammered this… thing… out.

I built a quick web API with two controllers, one to provide discovery data, and another for the details. The discovery URL (/api/DpmDiscovery/{HOST.NAME}) would hand back the LLD formatted JSON and the other URL (/api/DpmStatus/{#RECPOINT.BACKUPPATH}) would spit out details.

A call for https://dpmapi.contoso.ca/api/DpmDiscovery/file01 would return the following json.

{
    "data": [
        {
            "{#RECPOINT.STATUS}": 2,
            "{#RECPOINT.IDSN}": "D:\\",
            "{#RECPOINT.SERVERNAME}": "file01.contoso.ca",
            "{#RECPOINT.BACKUPPATH}": "file01.contoso.ca..D:..",
            "{#RECPOINT.CREATIONTIME}": "2019-04-25T00:05:28-06:00",
            "{#RECPOINT.UNIXTIME}": "1556172328"
        },
        {
            "{#RECPOINT.STATUS}": 2,
            "{#RECPOINT.IDSN}": "E:\\",
            "{#RECPOINT.SERVERNAME}": "file01.contoso.ca",
            "{#RECPOINT.BACKUPPATH}": "file01.contoso.ca..E:..",
            "{#RECPOINT.CREATIONTIME}": "2019-04-25T00:05:42-06:00",
            "{#RECPOINT.UNIXTIME}": "1556172342"
        },
        {
            "{#RECPOINT.STATUS}": 2,
            "{#RECPOINT.IDSN}": "System State",
            "{#RECPOINT.SERVERNAME}": "file01.contoso.ca",
            "{#RECPOINT.BACKUPPATH}": "file01.contoso.ca..System_State",
            "{#RECPOINT.CREATIONTIME}": "2019-04-25T02:10:59-06:00",
            "{#RECPOINT.UNIXTIME}": "1556179859"
        }
    ]
}

Then the LLD rule creates an HTTP Agent item to call https://dpmapi.contoso.ca/api/DpmStatus/file01.contoso.ca..System_State

{
    "status": 2,
    "interpretedDsn": "System State",
    "serverName": "file01.contoso.ads",
    "backupPath": "file01.contoso.ads..System_State",
    "creationTime": "2019-04-25T02:10:59-06:00",
    "unixCreationTime": "1556179859"
}

Technically my API returned the data as application/json; however I had accidentally checked “Convert To JSON” so you’ll see a body element in the JSON path below (e.g. $.body.status). In theory I could uncheck that and remove the body element. In practice it works as-is so it’ll stay that way for now.

Example Screenshots

Zabbix Discovery Rule

Zabbix Data Item

Dependent Item



These dependent items use JSON Path processing to extract the actual data out of my details response.

(body element was inserted because I had checked “Convert to JSON”)

Citations Nonsense:
DPM SQL View Documentation: https://docs.microsoft.com/en-us/previous-versions/system-center/data-protection-manager-2010/ff399120(v=technet.10)
Handy JSON validator: https://jsonformatter.curiousconcept.com/
Go-To JSON Browser: http://jsonviewer.stack.hu/

DPM Azure Recovery Services Agent Crashing

Update: We did start having dependency issues after updating the MARS agent. It appears that the agent now depends on the management service. Not getting errors anymore though so we reset things back to normal. Stuff below is just for posterity.

DPM 2016 deployments have been filling up my error logs with crash reports for the Microsoft Azure Recovery Services Management Agent. Turns out that’s the statistics agent for the Azure dashboards that don’t work on the LTSC releases of DPM (http://blog.teknikcs.ca/2019/02/21/dpm-protected-items-dont-appear-in-azure-vault/).

System Event ID: 7031 
The Microsoft Azure Recovery Services Management Agent service terminated unexpectedly
Application Event ID: 1000
Faulting application name: OBRecoveryServicesManagementAgent.exe
Application Event ID: 1026
Application: OBRecoveryServicesManagementAgent.exe
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
at .CTraceProvider.TraceToErrorFile(CTraceProvider, DLS_TRACE_EVENT)

Disable it if you’re on DPM 2016 or DPM 2012. No impact that we’ve seen.

DPM Protected Items Don’t Appear in Azure Vault

TL;WR: None of the current Long Term (i.e. 2012 R2/2016) DPM releases actually send updates to azure. So unless you’re on the bleeding edge DPM stream you don’t get to see this stuff. Suggested Pairing: Two drawn-out eye rolls.

Simple enough problem, I was looking at the Recovery Services vault page for a DPM installation and noticed all these nice dashboards for alerts, items, jobs, etc. All of them were empty.

  • 0 Backup Items
  • 0 Backup Jobs
  • 1 Azure Backup Agent… with item count 0

After a blazing fast support call I learned that this feature currently doesn’t work with any released version of DPM (Microsoft’s flagship backup software in case you forgot). According to support the feature won’t be implemented until DPM 2019 is released (due March 2019).

Semi-Related: On our last deployment it really annoyed me to find out that no 2016 version of DPM supported server 2019. Again… Msft’s Flagship Backup Software… and it doesn’t support the latest version of the server OS AND their snazzy dashboards don’t integrate. I get that delaying 5 months between GA launch and integrating with system center doesn’t seem that crazy, but it certainly was annoying to find out that the brownfield deployment simply wouldn’t be able to use server 2019 without buying a new set of DPM licenses.

Yeah yeah “DPM 2016 is old so obviously it won’t protect new versions of windows.” I’d argue that sentiment isn’t appropriate in the context of a backup application with non-trivial behaviour meant for long term use.

S2D Cluster Creation From VMM 2016

TL;WR: If you’re trying to setup an S2D Hyper-Converged Cluster before adding it to your VMM Fabric: Don’t. It will work, but you won’t be able to see or manage the storage from VMM. Let VMM build it all. Suggested Pairing: Half a dozen hours of diagnostics and ███████ [REDACTED].

Quick Trials and Tribulations Lessons:

  1. I couldn’t for the life of me get VMM 2016 to manage the S2D portions of a cluster created outside of VMM. Did it manually, everything worked fine, validated RDMA etc. Then I realized that VMM wouldn’t let me manage the S2D tiers when creating volumes. Finally after some googling I read in a couple places that VMM won’t fully cooperate with an S2D Cluster created externally. Womp Womp.
  2. Most online documentation regarding externally created clusters just says (paraphrasing Microsoft docs): “Just add the cluster to VMM! Magic!”. That’s not how that works, adding an existing cluster to VMM is easy, adding a hyper-converged S2D cluster to VMM was definitely not easy.
  3. The fairly unhelpful message “Error (25325): The cluster creation failed because of the following error: An error occurred while performing the operation.” For me was because of permissions, somewhere. Even though I applied all of the CNO OU rights, local admin rights, DNS rights, AD rights, and every cluster validation passed without warning; I kept getting this error. Eventually I gave up and used a highly privileged AD account and voilà.

In the end after creating several S2D clusters trying to get VMM to cooperate I finally got it to show me the magic “Configure advanced storage and tiering settings” box.