Deployment – airdesk

Autopilot and Intune Faults

Posted on July 25, 2022 | by Airdesk | Leave a Comment

“When sorrows come, they come not single spies, but in battalions.” We are deploying thousands of devices with Autopilot and Intune, and the service faults come in battalions.

We have been tracking these faults for a while. There are two types:

Microsoft identifies a fault with a service announcement
We raise a ticket, there is no cause found for the fault. No service announcement.

In mid-May, account setup failed to complete on pre-provisioned devices. The setup just hung. No cause found.

There was a service incident at the same time (now rolled over in the logs). Users unable to use Autopilot. Different problem, but possibly related.

Application failed to unzip after downloading. No cause found.

Application failed to download from Intune, with “endpoint failed to respond.” No cause found.

Late June, Autopilot failed at the beginning, before entering ESP. Error is 80072ee2. DNS query failed for “enterpriseregistration.microsoft.com”. Network timeout trying to register the device at DRS. No cause found.

From 21 June to 7 July, incident IT396955 “Users’ devices may have incorrectly appeared as non-compliant after Autopilot pre-provisioning in Microsoft Intune”. We don’t allow non-compliant devices to connect, so this caused a complete failure. Root cause: “A recent fix for an unrelated issue.” Although the incident dates from 21 June, it was only identified as an incident on 4 July.

On 21 July, incident IT402961 “Users and admins may have been unable to access the Microsoft Intune service or see limited functionality.” Root cause: “a network gateway outage.”

The facts show that the Autopilot service, with Intune, is fundamentally unreliable. If it were Intune alone, users would experience a failure of policy updates, or application deployments. But, during Autopilot, the result is a failed deployment.

At present, I recommend not using Autopilot to deploy devices, for the next year or so. It is too unreliable. My guess is that an internal service agreement has the wrong incentives.

Intune, WDAC and Managed Installer

Posted on May 14, 2022 | by Airdesk | Leave a Comment

WDAC has an option (Option 13) to allow apps installed by a Managed Installer. This sounds great! Everything you install using your preferred installer would be allowed, without going to the trouble of creating rules. But there’s a snag. There is no Configuration Service Provider (CSP) to deliver this policy in Intune.

The Managed Installer option actually uses the same method to allow executables to run as the Intelligent Security Graph option (Option 14). When a file is authorised by one of these methods, an extended attribute is written to the file. You can see this attribute with the fsutil utility. The method is documented here: Automatically allow apps deployed by a managed installer with Windows Defender Application Control.

The documentation on Managed Installer is a little confusing. The main documentation shows a policy that allows the Intune Management Extension, as well as the SCCM extension.

So, looking at that, we would obviously be able to allow Intune apps in Intune, right? But we cannot. The reason is that the documentation also describes implementing this policy in a GPO. But in Intune we cannot use GPO’s and, instead, we use Configuration Service Providers (CSP). The Managed Installer option is implemented as an AppLocker policy, and the AppLocker CSP does not contain a section for the Managed Installer rule collection type.

Although we cannot implement this as an Intune policy (because there is no CSP), we could theoretically implement it another way. With a registry key, for example, even if there were no CSP to configure the registry key, we could simply add, change or delete it in script. With AppLocker policies, we can use PowerShell to create a policy from an XML file, using Set-AppLockerPolicy. So the solution is to deliver a custom AppLocker policy with PowerShell, to enable the Intune agent as a Managed Installer in WDAC.

There are three significant drawbacks:

The effort and constraints in managing the policies manually through PowerShell. For example, there is no Remove cmdlet for a policy in PowerShell
Managed Installer tags the installed files, but not automatic updates. To allow the updates, you would either have to reinstall, or apply rules to allow the updated files, which would defeat the purpose.

Autopilot Faults and Logs

Posted on April 26, 2022 | by Airdesk | Leave a Comment

This is a post about where to look to find the cause when Autopilot fails.

By “Autopilot”, I am referring to the whole process of deploying, enrolling and setting up a Windows device. The process really contains several distinct parts:

The Out of Box Experience (OOBE) like selecting language, region and keyboard
Enrolment in Intune and joining the Azure AD domain (or hybrid)
Implementing all the policies and apps assigned to the user or device by Intune
The Enrollment Status Page (ESP) which monitors and controls the process nearly from the beginning to the end.

But I am using the term “Autopilot” to refer to all these, for convenience.

We can distinguish two types of failure. One, when setting the process up and testing it to see if it works. Another, during deployments when everything is supposed to be working. This post is about the second. For the first, you can generally follow the guides for setting up Autopilot and ESP, and search the documentation if it is not working. For the second, you need a good understanding of how the process works, what happens when it goes wrong, and where to look to find the cause.

Here is the best end-to-end diagram of the process: Windows Autopilot deployment process. And here is the page that best describes what happens in each phase of EPS: Enrollment Status Page tracking information. It is worth studying these in detail.

Getting the logs for Autopilot is straightforward. From a command prompt, run:

“mdmdiagnosticstool.exe -area DeviceEnrollment;DeviceProvisioning;Autopilot;Tpm –cab C:\Temp\Autopilot.cab”.

You will need to run elevated to get the TPM diagnostics. You will also need to make sure that whoever runs the command is able to save in the location specified. If you are asking a standard user to run the diagnostics, you can use Settings > Accounts > Access work or school > Export your management log files.

Michael Niehaus has written scripts to provide a quick interpretation of the diagnostics logs: Get-AutopilotDiagnostics. Running this script against the cab file is the first place to start.

There are a few points to note about the diagnostics:

You need to run it as soon as possible after the problem occurs. Several of the event logs contained in the diagnostics collection roll over, so events will be lost if you only run it later.
If you are reproducing a reproducible fault, you can add other tools like netsh, by breaking in with Shift+Fn10 before the fault occurs.
The diagnostic logging only captures events relating to Autopilot and enrolment. If you want a wider selection of logs, you may want to run One Data Collector at the same time. It only takes a few minutes to run.
Running Get-AutopilotDiagnostics with the –Online parameter will fetch the application name to match the ID, saving you a lot of time in trawling logs to find the name of a failed app ID.

If the failure occurred before the ESP started, then this is Autopilot proper. The place to look is in the event log: Applications and Services logs > Microsoft > Windows > ModernDeployment-Diagnostics-Provider > Autopilot.

When the ESP starts, we can find the place that it fails in the registry. This is obtained from: HKLM\Software\Microsoft\Provisioning\AutopilotSettings. Each sub-category has a status of: notStarted; inProgress, succeeded; or failed. It would be great if these could be surfaced somewhere more accessible. As it is, they can be found in the diagnostics log file: MdmDiagReport_RegistryDump.reg.

For example, see this obscure fault: Error code 0x80180014 when re-enrolling using self-deployment or pre-provisioning mode. The documentation is not correct. Normally, when you do a “Fresh Start” or a “Wipe”, the object in Intune is soft-deleted. But, occasionally, this back-end process fails. As a result, when you do a reset, it will fail in Device Preparation, at the “Registering your device for mobile management” stage i.e. enrolling in Intune. The status is recorded at: HKLM\Software\Microsoft\Provisioning\AutopilotSettings\DevicePreparationStatus.Category.

This happens because the Intune object has not been deleted as it should be. The solution is to find the object and delete it manually. Knowing the stage it failed at enables you to investigate why this happened.

It can be useful to know exactly when the fault occurred. This helps us to correlate different logs. Bizarrely, the time is shown in the event log: Applications and Services logs > Microsoft > Windows > Shell-Core > Operational. Use the “Find” action to search for the word “failed”. This will show the CloudExperienceHost Web App Event with “Subcategory ID = DevicePreparation.MdmEnrollmentSubcategory; state = failed.” When you know the time, you can search other logs to see if anything distinctive happened at exactly that time.

As I said at the beginning, the purpose of this post is to describe how to investigate unexpected faults. Once you know where and when in the process the fault occurred, you can follow the trail to diagnose it. If you know the type of fault, you may well need to reproduce it with additional logging to find the cause.

Autopilot Faults and the Network

Posted on April 9, 2022 | by Airdesk | Leave a Comment

The list of network requirements for Intune and Autopilot is extensive. This post is about finding out if the client cannot connect to one or more of the required endpoints during Autopilot.

Microsoft publishes the list of required endpoints for Autopilot and Intune. There is no point in repeating the information here. The important points to note are:

The documents also contain references to secondary services. For example, it requires access to Windows Activation, Azure AD (for authentication), NTP time service and others
In an enterprise, HTTP and HTTPS traffic will be intercepted by a proxy. Proxy servers may inspect the traffic, and require authentication. All this traffic needs to bypass the proxy; or bypass inspection and authentication. This is not easy to do, and it is easy to make mistakes.
The lists often require URL’s and wildcards, for example *.microsoftaik.azure.net for TPM attestation.

As a result, you may need to troubleshoot things that should be working, but are not. You need to be able to find out what is blocked, or not connecting.

The simplest way, which everyone will be familiar with, is to use Wireshark. You can install Wireshark at the beginning of Autopilot, and use Alt+Tab to get back to it after the problem has occurred (see Autopilot and Shift+Fn10).

You may think you are looking for a dropped connection. But the place to look for that is on the firewall. In reality the packet capture will require a lot more work to analyse than that. Almost all the traffic is encrypted HTTPS. You cannot see the contents of the transactions. What you will see (roughly) is:

A DNS query to resolve one of the required endpoints
A DNS response resolving the query to an IP address corresponding to a traffic manager, load balancer, CDN
TCP SYN
TCP ACK
Client Hello
Server Hello, Certificate, Server Key Exchange, Server Hello Done
Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
Application Data

Azure Traffic Manager is a DNS service that distributes client traffic to the closest available Azure endpoint. If the client makes a DNS query for (for example) fe2cr.update.microsoft.com, it will obtain the IP address of an endpoint corresponding to fe2cr.update.msft.com.trafficmanager.net.

This means you can’t expect to see traffic to and from the endpoints contained in the lists of endpoints. A query for account.live.com will result in traffic to and from l-0013.l-msedge.net.

If you use a display filter of “dns”, you will see all the queries made. You can check on the proxy that all those names bypass inspection and authentication. If they do not, you may expect to see:

No application data after a handshake
A TCP FIN followed by another attempt to connect with a new TCP SYN.

As the Autopilot traffic is nearly all HTTPS, you could also use Fiddler to capture the exchange between client and server. When you install Fiddler, you choose to accept the Fiddler certificate. This means that the traffic can be proxied and decrypted by Fiddler. As a result, you should be able to see the content of the transactions.

I recently had a case where Autopilot failed at the start of the Account Setup phase of Autopilot. As it happens, I could see in the Microsoft-Windows-Shell-Core/Operational event log that a “AAD user token request failed” but not why it failed. I wanted to do a packet capture during the time this request failed. But you cannot break into Autopilot during the account setup phase and, even if you could, it would be too late to start a capture. Ideally you want to start a capture at a stopped point in the process, so that you have time to set it up before continuing.

This was a case where the Network Shell utility (netsh.exe) can help. Netsh can record a trace file (*.etl) using different providers. Most importantly, it is not limited to packet capture. It records the activity of each provider specified in the command. Providers are bundled up in “scenarios” for different types of network problem. The “InternetClient” scenario, for example, contains the following providers that we may be interested in:

Web-Http
WinInet
WebIO
WinHttp
DNS-Client
TCPIP.

A unique advantage of Netsh is the “Persistent” parameter. This enables the trace to continue through a restart. By making it persistent, we can start a trace in the Device Setup phase, continue through a restart into the Account Setup phase, and then stop the trace when the fault has occurred.

A difficulty to overcome is that the trace needs to be started and stopped in the same security context. In Device Setup, the console user is DefaultUser0 with admin rights. But this account does not persist into Account Setup.

To get around this, we need to use PsExec. This is the SysInternals utility that enables me (among many other things) to run a command in the system context. If I start the trace in the system context, then I can do the same to stop it. So the full plan of attack is:

Shift+Fn10.
In the console, download PsExec or copy it from a USB stick.
Run: “psexec.exe –s –i cmd.exe”. This brings up a new console running in the System context.
In the new console, run: “netsh trace start capture=yes report=yes persistent=yes traceFile=[path to trace.etl] scenario=InternetClient”. Depending on the circumstances, you can use maxSize to limit the size of the trace file. By default, the logging is circular.
Continue through to when the fault occurs.
In this case, because Autopilot was failing in Account Setup, I needed to use the “Continue” option in the Enrollment Status Page (ESP) profile, so that I could finish and get back to a user session, to be able to run PsExec again and stop the trace with “Netsh trace stop”.

A significant disadvantage of Netsh is that the best way to read the ETL file is with Microsoft Message Analyzer. But this is now obsolete and no longer available to download. There are utilities to convert the ETL to pcap, so it can be read in Wireshark, but that loses the benefit of the network provider logging, like DNS-Client and WinInet. I keep a copy of Microsoft Message Analyzer just for this.

So, this post has given three tools and methods to investigate a network fault causing Autopilot to fail. You will need to adapt the methods to the particular circumstances of the case.

Autopilot Faults and Shift+Fn10

Posted on April 9, 2022 | by Airdesk | Leave a Comment

Most people probably know that you can break into Autopilot by pressing Shift+Fn10 to bring up a command prompt. Here are some aspects that people may be less aware of.

The Command Prompt runs in the security context of DefaultUser0. This is a local admin account used for deployment. The account and the profile are supposed to be deleted at the end of Autopilot, but they are not. You can still see the profile in C:\Users.

Obviously, you can run any command line actions. But you can also bring up GUI applications:

“explorer” to open Windows Explorer. From there, you can navigate to run other GUI applications.
“control panel” to open the Control Panel (or “appwiz.cpl” to bring up Programs and Features)
“compmgmt” to bring up Computer Management, for Services, Device Manager, Disk Manager, Event Viewer, Scheduled Tasks, Users and Group
“eventvwr” for Event Viewer
“taskmgr” for Task Manager
“start ms-settings:” to bring up the Settings app.

You can run “powershell.exe” to run PowerShell cmdlets in the Cmd console. Or you can use Explorer to navigate to PowerShell and run it from there.

You can open the Edge Chromium browser to download or upload stuff.

You can use Shift+Fn10 at the beginning, to create a configuration before the main Enrollment Status Page (ESP) phases begin. You can use it when Autopilot fails in Device Preparation or Device Setup. You can use it any time in between. But you cannot use it in the Account Setup phase because, in this phase, the session runs in the user security context and not DefaultUser0.

Some of the things you may want to do at the beginning:

Increase the size of the Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Operational event log, from the default 1028 KB
Install Wireshark (but, if you do this, you need to make sure the VC++ dependency does not interfere with any VC++ app deployment)
Create a C:\Temp folder to save logs and exports.

After you open the Command Prompt, if you navigate back to the ESP, the console will be hidden again. But it is still running. Click Alt+Tab to open it again, or to select any of the GUI applications that are still running. For example, if you started Wireshark at the beginning, then you can go back to the same Wireshark at any time with Alt+Tab.

Troubleshooting Faults in Autopilot

Posted on April 9, 2022 | by Airdesk | Leave a Comment

I have experienced a lot of faults in the Autopilot service recently. Autopilot is the Microsoft service that configures a Windows desktop out of the box (OOBE), when it first boots up.

If you google for Autopilot Failure, or Enrollment Status Page (ESP) Failure, you will find a lot of Microsoft documentation, and secondary documentation expanding on similar faults. There’s a very long list of resolved issues, and a list of continuing known issues. To be fair, there are many variables, and many possible scenarios leading to different types of failure: virtual machine; TPM version; user-driven vs. self-deploying; hybrid AAD join. The most obvious cause of a failure is that you might actually have something wrong with the network, or with an application deployment.

But this post is not about these failures. It is about what to do if the service is failing and it does not seem to be caused by one of these documented faults. We have experienced quite a number of these. The fact is, the troubleshooting steps documented by Microsoft are based on known and expected faults, and not on unexpected faults; and certainly not on faults in the Microsoft infrastructure supporting the Autopilot service.

I think I will probably break this post out into separate snippets on how to perform various troubleshooting tasks. Otherwise it will be too cluttered. But let me first illustrate what I mean. If you go to the Intune Tenant Administration page, you will see a tab for the “Service health and message center”. The incidents here roll over very fast, but you will see a continuous flow of faults. We’ve found that only a percentage of faults are acknowledged in this way. But you can get an idea of what is happening from the explanations.

“Admins may have seen provisioning failures when attempting to create new Cloud PCs within Windows 365.”

Start time: Wednesday, January 26, 2022, at 12:00 PM UTC End time: Friday, April 1, 2022, at 11:59 PM UTC [over two months later]
Root cause: An Azure Host Operating System change resulted in provisioning failures when admins attempted to create new Cloud PCs within Windows 365.

If you are lucky, when a fault occurs, you will find a known service health incident. But how many faults in infrastructure and code are actually acknowledged? Based on my experience it is less than half. We had a total outage of the Autopilot service from 14 Feb to 28 Feb. There was no service health message. But we received an explanation from an internal MS channel to tell us that a code update had gone wrong. I think the reason for this disparity is that Intune and Autopilot are not really homogenous services. They are supposed to be. But it seems clear that they are really a collection of instances of services. The incidents often refer to “customers on the affected infrastructure”. How do you know if you are on the affected infrastructure? You don’t. How does Microsoft know if the affected infrastructure is working or not? It doesn’t. Microsoft incidents refer to telemetry. But telemetry will only tell you what the metrics are measuring. This does not seem to include actual successful completion of transactions. We have often asked Microsoft Premier Support to reproduce faults for themselves, instead of asking us to repeatedly collect logs for them. They can’t.

So, let’s get into the elements of how to find out what is going on.

Basic Flaws in Intune Package for Microsoft 365 Apps

Posted on March 10, 2022 | by Airdesk | Leave a Comment

Intune (or Microsoft Endpoint Manager) has a built in package for Microsoft 365 Apps. This is the obvious package to use to deploy Office 365. But there are several fatal flaws.

Intune deploys the Microsoft 365 Apps as a policy. This sounds strange, but it is deployed by the Office CSP. What happens is that this policy installs two small Click to Run MSI’s, which then pull down the Microsoft 365 Apps like Word, Excel, Outlook etc.

Sometime back in about 2018 Microsoft chained the Teams installation to Office 365. Consequently, you can select Teams as one of the suite of apps to install. What happens is that sometime after the other apps have been installed, the Teams MSI runs to install the Teams Installer. The Teams Installer is the component that detects a user logon and installs Teams itself for the user, in the user’s %AppData% folder. Teams is not actually installed as part of the Office Click to Run. It is a separate MSI chained to the Office setup.

Because the Microsoft 365 Apps are installed by a policy, the installation is performed by the OMA-DM policy agent, and not by the Intune Management Extension (IME). The IME is the agent that is used to install all Win32 apps. Policies are applied as soon as the Device Setup starts, so before even the IME agent (itself an MSI) has been installed, and before the Win32 apps to be installed have been evaluated. Because the Office Click to Run MSI’s are small, they are installed before other app processing has started.

Microsoft warns that you should not mix Win32 apps and Line of Business (LOB) apps in an Autopilot deployment. The wording in Microsoft docs is particularly poor, because “LOB” can mean different things. I have spoken to several Premier Support engineers who clearly don’t know what it is supposed to mean. In this case it refers to the single MSI packages that were used before Microsoft introduced the IME agent. Before the IME agent, you could install applications but only if they used a single MSI file (no folders, no transform). This installation of a LOB MSI is performed by the OMA-DM agent.

As an example of poor documentation, this page Add a Windows line-of-business app to Microsoft Intune says the following: ”When deploying Win32 apps using an installation file with the .msi extension (packaged in an .intunewin file using the Content Prep Tool), consider using Intune Management Extension. If you mix the installation of Win32 apps and line-of-business apps during AutoPilot enrollment, the app installation may fail.” But it fails to say what is meant by a “line-of-business” app (and it is Autopilot, not AutoPilot).

In contrast, this page Set up the Enrollment Status Page says the following: “It’s preferable to deploy the offline-licensed Microsoft Store for Business apps. Don’t mix LOB and Win32 apps. Both LOB (MSI) and Win32 installers use TrustedInstaller, which doesn’t allow simultaneous installations. If the OMA DM agent starts an MSI installation, the Intune Management Extension plugin starts a Win32 app installation by using the same TrustedInstaller. In this situation, Win32 app installation fails and returns an ‘Another installation is in progress, please try again later’ error message. In this situation, ESP fails. Therefore, don’t mix LOB and Win32 apps in any type of Autopilot enrollment.”

The reason not to mix LOB and Win32 apps is obvious. Any MSI is actually executed by msiexec.exe, which calls the Windows Installer service. If you have ever tried to run an MSI when one is already running, you will have seen a warning that another installation is in progress, please wait.

If you have two different agents running msiexec and calling Windows Installer, they will conflict. This is exactly what happens with the Teams MSI in the Microsoft 365 Apps package. In other words, the Microsoft 365 Apps package is guilty of exactly the fault that Microsoft warns you to avoid.

In the verbose Microsoft 365 Apps log, you will see this error:

01/01/2021 01:01:01.010 OFFICECL (0x2264) 0xdc4 Click-To-Run Non Task Error co7l2 Monitorable C2R::TeamsAddon::Install {“MachineId”: “a3b97b64ce6fff4980eaef7d16bed3bd”, “SessionID”: “a7e42957-b65a-45f2-93d7-fdb88842597a”, “GeoID”: 242, “Ver”: “16.0.14430.20342”, “C2RClientVer”: “16.0.14430.20314”, “ContextData”: “{\”message\”:\”InstallTeams: MsiInstallProduct failed due to install already in progress.\”,\”Status\”:\”1618\”}”}

You get the verbose log by breaking into Autopilot and setting this registry key: HKLM\SOFTWARE\Microsoft\ClickToRun\OverRide: LogLevel = DWORD 3.

Amazingly, the Microsoft 365 Apps setup process does not detect that the Teams installation has failed. It carries on and exits with a success code. So it will never try again. In practice, what happens is that sometimes Teams wins the race for Windows Installer, and gets installed, and sometimes it doesn’t. If it wins, then the standard retry in the IME will ensure that whichever app lost to Teams will try again and get installed. So the result is that, apparently randomly, some devices will complete Autopilot without Teams, and with no way to get it back.

But there is more. The Intune package for Microsoft 365 Apps automatically includes OneDrive. You can’t select or unselect it. OneDrive is already natively installed on Windows, but as a per-user application in the user’s %AppData% folder. The Microsoft 365 Apps package installs the 32-bit per-device version of OneDrive, and removes the per-user version. But Microsoft recommends using the 64-bit version of OneDrive on standard Windows devices. So, if you create a package to install the 64-bit version, Intune will first install the 32-bit version, then (hopefully) uninstall it and install the 64-bit version.

And there is more. Visio and Project are both available in the Microsoft 365 Apps package. But, if you select (say) Visio without the other apps (Word, Excel, Outlook etc), then all those other apps will be uninstalled. One Microsoft 365 Apps packages entirely replaces, rather than adds to, another. If you select Visio with the other apps, then the existing installation of Office will be removed, and replaced with the new one. Obviously, there are different combinations: with Visio but not Project; with Project but not Visio; with Visio and Project. It is completely unworkable. In addition, the installation of Visio or Project will force-close other Office apps, which is not nice.

And there’s more! Microsoft Surface devices come pre-installed with Microsoft 365 Apps in 13 languages. If you run the built-in Intune package, it will not uninstall the pre-installed languages. You might say this doesn’t matter. But, if you do a Fresh Start on a device, then it will not have the 13 languages, and so you will have different devices with different configurations.

Taking all this into account, we decided to build our own packages for all Office apps. Once you have a basic script to run the setup, and after you generate the required XML, it is fairly easy to adapt to the different packages you need for Office 365, Teams, OneDrive, Visio and Project. The good thing is that you are still using exactly the same core method: setup.exe with an XML file. You can do some of this by using the XML option instead of the Configuration Designer option in the Intune package, but we needed to do more, including making the Visio and Project installations interactive, so we chose to go entirely custom.

Autopilot Failure

Posted on March 9, 2022 | by Airdesk | Leave a Comment

A few weeks ago, Autopilot stopped working. Autopilot is the service that builds a Windows desktop from scratch when it first boots up, a bit like MDT. If the device hardware ID is registered in the Autopilot service then, when it starts up, it contacts Autopilot and runs the Out of Box Experience (OOBE) according to the settings in the profile.

On 14 Feb 2022, routine setup of new devices stopped working. We did not at first know why, so we spent a couple of days testing and trying different things. Then we raised a ticket with Microsoft Premier Support. The symptoms were that Store Apps were failing to install (they said “blocked by AppLocker”) and Win32 apps were failing with random errors.

Microsoft Premier Support clearly failed to understand the basics of the problem. They ran through known issues, and asked us to check basic things, and to exclude things that were failing. But Autopilot was working before 14 Feb. Why would applications that were working before 14 Feb start not working?

We spent two weeks in pointless dialog. Then on 25 Feb Autopilot started working again. We made no changes. It just started to work, as it had before 14 Feb. Premier Support offered no analysis and no explanation.

If you were in the middle of a rollout, that could be hundreds or thousands of devices that would fail. How is it even possible that Microsoft had no clue that the service had failed, no clue about why, and no clue about how to fix it?

Some time later, from a different source, we received an explanation. The issue was caused by an error in the code, where devices would sometimes randomly hit a web exception when trying to fetch the content or content info of some Win32 apps from the service. Microsoft created a hotfix and, as a result, error 0x81036502 no longer occurs.

But Premier Support did not know about the fault, did not know about the hotfix, and did not even connect our problem with the fault. This is a failure on a grand scale.

Fault with Company Portal

Posted on March 2, 2022 | by Airdesk | Leave a Comment

This is a story about the complete failure of Microsoft Premier Support to diagnose and resolve a fault in the Company Portal.

It is difficult to put into words how complete the failure is. But it includes a failure to define the problem; a failure to capture it or reproduce it; and a failure to provide any diagnosis of the cause.

The Company Portal is the Modern App, or Store App, that enables a user to install applications that have been published and made available to them from Intune. It is an essential part of the Mobile Device Management (MDM) platform. Without the Company Portal, a user can only have the applications that are “Required”. So, after Autopilot, Company Portal will often be the first place a user goes, to obtain the rest of the applications that they need to work with. An example is specialist finance applications. These might be made available to a community of finance users, but each person with install the ones they need individually.

The problem we have had for several months is that the Company Portal will suddenly disappear from a user’s desktop. It is gone. The user can search for “Company Portal” and it is not there. Where has it gone? No idea. How do you get it back? Well, you obviously can’t use the Company Portal to get it!

The facts of the problem are simple and clear, though you would not believe it from the number of times we have been asked to explain and provide logs:

After Autopilot completes, Company Portal is present and working.
Some short time later, it has disappeared from the user’s Start menu.
If you run Get-AppXPackage as the user, the Company Portal is not listed. However, if you log on as an admin, and run Get-AppXPackage –AllUsers, then the portal is shown as installed and available but only for the admin user.
The AppX event log does not show any obvious relevant events.
It seems to happen in episodes. And it seems to happen sometimes and not others.

We have been asked repeatedly to provide logs: Autopilot logs and One Data Collector logs. But, obviously, if you gather logs before it has disappeared, then there is nothing to see. If you gather logs after it has disappeared, then there is also nothing to see.

After a while, we asked Microsoft Premier Support to try to reproduce the fault themselves instead of continuously asking us for logs. Amazingly, they are unable to do this. Microsoft Premier Support does not have access to virtual machine, or physical machines, that can be used to reproduce faults in Intune and Autopilot. Just let that sink in. Premier Support is unable to attempt to reproduce a fault in Autopilot. It depends on the customer to reproduce it.

We had a long discussion with Premier Support about Offline versus Online apps. The Microsoft documentation for Autopilot recommends in several places that you should use the Offline version of Company Portal. This is counter-intuitive. Offline apps are designed, intended, to be used offline. The scenario given in Microsoft documentation is a kiosk or shared device that is not connected to the internet. The Offline app is installed by DISM in a task sequence, and is used offline. Company Portal, by definition, is of no use offline. It is used to install applications from Intune. If the device were offline, it would not connect to Intune. So why install the Offline version?

We eventually established, at least we think, that an Offline app is in some way cached by Intune; whereas an Online app is obtained directly from a Microsoft Store repository. This seems relevant to the case of the disappearing portal, but we never discovered more about the true difference.

In an early occurrence, we found an AppX event to say that the Company Portal was not installed because of a missing dependency. The missing dependency was the Microsoft Services Store Engagement app. This is the app that enables users to provide feedback. But this app is (apparently) an embedded part of Windows 10 and cannot be missing. We heard no more about this.

The Company Portal stopped disappearing for a while, and we deduced that the fault was in some way related to version updates. It occurred frequently when the version changed from 111.1.107.0 to 11.1.146.0. It has started to occur frequently again now the version is 11.1.177.0. Of course, we have no idea how it is related to the update. We don’t even really know how an update of an Offline app happens.

Finally Microsoft Premier Support has asked us to gather a SysInternals Procmon log, together with a UXTrace log. I have done a lot of troubleshooting with Procmon. It generates huge log files, of every file and registry operation, as well as some TCP operations. To use Procmon effectively, you need a way to stop it when the fault occurs. Microsoft Premier Support simply asked us to run it and stop it when the fault occurred. There are several problems with this. The first is that the user needs to run UXTrace and Procmon with elevated rights. In our environment (as in almost any production environment) the user does not have admin rights and cannot elevate. The second is that Procmon creates huge logs. You can’t just run it for an unspecified length of time, then stop it and save the log. Microsoft Premier Support were clearly unable to understand the problem of gathering the logs, let alone provide a solution. This is dismal. I would expect a competent second-line engineer to have the skills to work out a strategy for collecting logs. It is part of the basic skill of troubleshooting.

So, three months on, Microsoft Premier Support has no clue, and no practical problem-solving approach.

The thing we have found is that Premier Support seems to have no access to the people who develop and run Intune, Autopilot and Company Portal. They are just as much in the dark as you or I.

Windows 7 Deployment Part 7

Posted on November 10, 2009 | by Airdesk | Leave a Comment

There are several different tools for installing drivers in Windows 7. This blog aims to describe them and show how they differ.

Driver installation tools for Windows 7:

DISM
DPInst
DrvInst
PnpEnum
PnPUnattend
PnPUtil

DISM

Deployment Image Servicing and Management (dism.exe) is the new tool for modifying Windows images. It replaces the individual tools that were introduced for Vista images. There is plenty of documentation about DISM.

DISM is a "framework" tool that gives access to different "providers". The DISM host itself controls things like logging and rebooting. The different providers do the work with their own command line options, called by the DISM host.

CBSProvider (Component Based Servicing)
OSProvider (OS updates)
WIMProvider (handling the WIM file)
SMIProvider (Settings Management)
DMIProvider (Driver Management)

DISM is a tool for servicing both Online and Offline images. Dism /image:[path] refers to a mounted offline image. Dism /online refers to the current running image. However, you can not use DISM to add or remove drivers from an online Windows 7 image. The facility to do this does not exist. The commands /add-driver and /remove-driver apply only to offline images. When the image is Online:

you can add and remove updates
you can enable and disable features
you can not add and remove drivers.

Servicing in Audit mode, with the image online, uses PnPUnattend, not DISM.

DPInst

Driver Package Installer (DPInst.exe) is part of the Driver Installer Framework (DIFx) provided to vendors to enable them to distribute drivers. DIFx gives vendors three tools of varying complexity to install drivers. The intention is that the user just clicks Run and is insulated from the method used.

DPInst
DIFxApp
DIFxAPI

DPInst is a very simple installer. Put the inf and sys files together in a folder with a copy of DPInst.exe. Execute DPInst and the driver is imported into the Driver Store. By default, the tool searches the current directory and tries to install all driver packages found.
DIFxApp is a plug-in or extension to Windows Installer or InstallShield. It provides the actions that enable, for example, a Windows Installer msi to install drivers.
DIFxApi enables the vendor to write custom installers.

So by using DPInst you would effectively be authoring a driver installation package on behalf of the vendor. Why does Microsoft provide DPInst at all? Because it has simple features to enable a vendor to distribute a driver, including:

localisation
customisation of text, icons and bitmaps
an option to add an EULA

It might be fun to use this to add drivers during deployment, but it is not what it is for.

DrvInst

DrvInst.exe is the Driver Installation Module of Windows 7. When Windows detects new hardware, DrvInst is the module that selects drivers from the Driver Store and sets up the driver for the hardware.

PnpEnum

PnPEnum.exe is a Microsoft utility that enumerates the Plug and Play hardware ID’s in a system. It is not part of the operating system. It is supplied as part of the Microsoft Platform Support Reporting tools. It is also part of Microsoft Deployment Toolkit (MDT).

In MDT 2010 it is used by the ZTIDrivers.wsf script as part of the task of importing drivers:

PnpEnum.exe outputs the hardware ID’s in pnpenum.xml
this is matched against the list of MDT Out-of-Box Drivers in drivers.xml
matching drivers are copied to c:drivers
c:drivers is defined as the Driver Path in the PnPCustomization component of WinPE or auditSystem.

PnPUnattend

PnPUnattend.exe is part of the operating system. During the auditSystem pass of setup (if there is one configured to run) it automatically imports drivers in the path defined in the unattend.xml PnpCustomizationsNonWinPE component.

This is specifically to install drivers from a path, unattended, as part of the Audit pass of setup. The command line options are: /s to search the driver path without importing; and /l to show logging information.

PnPUtil

PnPUtil.exe is part of the operating system. PnPUtil is a command-line tool to add and remove third party plug and play drivers. After Windows 7 is deployed, you can use PnPUtil to add or update specific drivers.

PnPUtil -a imports the specified driver or drivers
PnPutil -e lists the third party drivers that have already been added (but it does not provide a facility to pipe the output to a file)
PnPUtil -d deletes the specified third party driver.