ConfigMgr Everyday

Just got to work this morning with several computers that failed our Windows Updates deployment with the exact same issue as the one I most recently blogged about, the “updates handler job was cancelled.”

I checked which ones were failing, and it was the “Cumulative Update” for whichever version of Windows was on the computer.

The run-times for the Cumulative Updates had been reduced to an even shorter time of FIVE MINUTES?! We absolutely have computers that will not be able to complete this in a five minute time frame. I switched it back to 60 minutes.

Why did these move to 5 minutes? No idea. Maybe Microsoft knows…

Coming into a new environment it’s fun to see all kinds of new errors and new ways that SCCM has difficulty with things. Moving from a small (~2000 computers) to a larger (~15000 computers) seems to make this even more interesting.

Here’s an example of something that we didn’t see in my previous environment.

Last night was the night that our first large test group had reboots forced upon them for this month’s Windows patches. The deployment went fairly well ~500 installed correctly, but there were still a few machines that need remediation ~100. Okay, from what I’m understanding it went as normal per their expectations, there were more failures than I expected.

A large number of the computers that failed ended up failing for the same reason. The error code was “0x87D00664 Updated handler job was cancelled.”

A quick web search comes up with a few good descriptions of what the error actually means (updates handler job was cancelled is just not that descriptive,) and how to fix it.

What actually happened

 What the error is actually telling us happened is that SCCM timed out when trying to install the update, and because it hit the preset time out limit, it stopped installing. At that point it failed the process. So what can we do about it

Possibly unbeknownst to many people, each update that comes into SCCM has a maximum run time associated with it. That means if the update isn’t finished installing within that time it is considered failed and will stop installing. Microsoft assigns those runtimes and they are different for different types of updates. The two that were failing for us, cumulative updates for Windows 10 1607 and Windows 1703 were assigned a run time of 20 minutes.

To find this information go to the update in question in Software updates -> All software updates.

Right click on it and click properties. One of the tabs up above will be “Maximum Run Time”. Change it to something higher than it is currently set.

We let our installs run overnight, so giving these updates a runtime of 60 minutes should be fine. Your mileage may vary.

This seems to have taken care of the issues that we have had with updates. Just to be safe for the next couple of Cumulative Updates I might change the run times just to make sure this doesn’t happen on such a large scale again.

So why didn’t I see this in my earlier environment?

This is an interesting question. It doesn’t seem like this should be something that only happens in large vs small environments since it is the actual install running on the computers that it having the problem. I’m going to narrow it down to one of two things.

1. This error is potentially having this problem everywhere. It’s possible that this update just takes a bit longer, and that’s why I’m seeing it now and haven’t seen it before.
2. It could also be compounded by the fact that these are slower computers than I am used to having Windows 10 installed on. All of the computers that had Windows 10 installed in my previous environment were newer with fast SSD’s. It would be expected that those should install faster. Maybe it’s just slower hardware.

Anyway, things are working better now. It’s always fun to see what ConfigMgr throws our way.

Here are the resources I used to help with the remediation of this issue:

http://ccmblogs.co.uk/?p=132

https://blogs.msdn.microsoft.com/minfangl/2012/08/17/configuration-manager-may-fail-to-install-exchange-rollup-kb2706990-kb2734323-because-of-timeout/

So the preface of this article is that I have moved to a new company! I’m still working in ConfigMgr everyday, but it’s a slightly different environment. I’ve moved from being the primary person in a 2,500 Windows client environment, to a member of a team of 6 managing 18,000 computers in a much more distributed environment. That means my new employer has many more locations.

I ran into an issue that I haven’t had to deal with previously yesterday, and if this helps someone then great! I’m glad my embarrassment can be useful for you.

Just starting I’m jumping into some menial tasks, one of which was working with the Windows Updates that came out this patch Tuesday. No problem I can go through those and make sure that everything looks good. They mentioned that the current deployments, for January – April had not been condensed into one deployment yet. Cool I can do that too.

So I got the patch Tuesday deployment ready with the help of a co-worker, and then started moving the group membership of the January – March patches into the April patch collection. These groups were still deployed at the time.

Do you see the issue I just created? I didn’t until some of our smaller sites (with slower network connections) started complaining of network slowness. The network guys also noted that the links were becoming saturated.

Investigation determined that it was computers connecting to our WSUS servers to determine if they needed the patches in the April patch collection or not. Lots of computers were checking. All the computers were checking! They were doing it very slowly (because we have BITS throttling in place) but so many were doing it that the computers were still clogging the slower links and causing the problem.

The interim solution was to delete the deployment that was causing the computers to check in, and to stop the WSUS service on the WSUS servers. The effect was immediate and networks recovered almost immediately.

There was a mitigating circumstance that the person that I was working with was not aware of. Sometime in the last few months the WSUS servers were “right-sized” for the number of machines that were connecting. So, in the past, when changes like the ones we made were done, the bottleneck was on the WSUS servers themselves. They would be pegged at 100% CPU for “days” when this type of change was made. Computers would be denied access, and they would check back later to see what updates they needed. When the WSUS servers were expanded, they were able to process all the requests that were coming to them. They were able to process so well that the new bottleneck became slower network connections.

Moral of the story: changing the content of the deployment made computers in the environment re-evaluate themselves against the whole group of patches in the collection, not just new ones. In our situation the traffic between the client and the server was able to significantly slow down network traffic at slow locations.

Second Moral of the Story: When you fix one thing, sometimes you shift the bottleneck and break something else, make sure you are ready for that.

There is also an article on Microsoft’s website about a known issue that causes WSUS to cause high network bandwidth that is located here https://support.microsoft.com/en-us/help/4163525/high-bandwidth-use-when-clients-scan-for-updates-from-local-wsus-serve it’s possible that this caused some additional problems with our collection, but it certainly wasn’t our only issue.

I try to learn something new every day. This was my learning for yesterday.

 

 

Or “Pending updates, the bane of my existence”

Depending on who you talk to, we are either in the middle of or beginning the process to update all of our user workstations from Windows 7 to Windows 10. During testing I have come up with some fun errors, most of which I have been able to search quickly on Google and come up with the solution. I did have one that was a little bit trickier.

After I updated one of my test Virtual Desktops, which was one of the oldest images in the company, to Windows 10 1709 Software Center would no longer open. One of two things would happen, and they often alternated with each other. The first is the standard “Software Center could not open” error.

We have protocols to follow when this message shows up. Often the problem is that the ConfigMgr client is left in something called “Provisioning Mode” which makes Software Center give the above error. Microsoft recommends running the below command on the machine when this happens:

Powershell.exe Invoke-WmiMethod -Namespace root\CCM -Class SMS_Client -Name SetClientProvisioningMode -ArgumentList $false

This command can usually can clear the error and make things work correctly again. But it didn’t in this case. In fact, looking at the registry, I could tell that the client was not stuck in provisioning mode. So it had to be something else.

The alternate error that I was getting was a .net error that had a whole bunch of stuff that didn’t mean much to me. The error seemed to insinuate that .net was not installed correctly. With this knowledge I tried the following steps.

1. Uninstalled and reinstalled the ConfigMgr Client through the console– no change
2. Removed and re-added .net 4.7 – no change
3. Uninstalled and reinstalled the ConfigMgr client manually -no change
4. Ran the Microsoft .net 4.7 repair tool – no change

Finally, I went back to searching the Internet to see if there was anything else I could find with all my new knowledge (which didn’t change anything at all). I finally found a Reddit post from 8 months ago that had a little tidbit in it.

At first I didn’t take this too seriously. We use ConfigMgr for all of our software updates so I didn’t think there could possibly something waiting in Windows Update. But after doing a little more searching I finally got back to checking Windows Update. Here’s what was waiting for me:

I took this screenshot after I had started the update installing, but there was definitely a Windows Update that had snuck onto the machine at some point. After completing the update and rebooting the machine Software Center opened correctly and is installing software correctly.

So here’s a new reason why Software Center won’t open… Make sure to check your pending updates, even if you use ConfigMgr for updating.

So, a couple of posts ago I mentioned that I was happy about finally figuring out how Windows 10 should be updated, and I tried to decode how all the Windows 10 versions were named. Microsoft must have heard that I figured it out because they decided to change it again.

Bye Bye CB and CBB…

So they did away with the “Current Branch” and “Current Branch for Business” nomenclature, and are instead calling the new Windows 10 servicing model the “Semi Annual” Channel. To add additional confusion, they still are separating the new Semi Annual Channel into two “releases”.

Preview Release

There will still be preview releases of Windows 10 as the software is being developed

Hello Semi Annual Channel

When a version of Windows 10 is widely released, it will be known as “Semi Annual (Pilot).” During this time Microsoft recommends that you have some appropriate people testing the release so you can fix any issues with how it works in your environment. After a couple of months, it will become “Semi Annual (broad).” At this time Microsoft is assuming that you have tested sufficiently and are ready to deploy the software to the masses.

What’s With The Long Term Servicing Channel?

The Long Term Servicing Channel (formerly Long Term Servicing Branch) still exists in this model, but it is still should only really be used in static environments. It still will not support new technologies when they are released, and exists so that environments that absolutely can not update (the example that is always given is Hospitals, who are installing on computers that control large diagnostic equipment that can’t be taken offline to update, or might cost a ton of money to certify with a new OS).

The User Environment

Here is a run down of how our environment stacks up with each of the releases. (now that Microsoft has changed the names.)

1. I think one of our network guys runs the Preview Releaseof Windows 10 on one of his machines. Maybe. We largely ignore Preview Releases of Windows 10. Sorry Microsoft.
2. For our company we allow the infrastructure team to install Semi Annual (Pilot) on their primary machine. They do not have access to the Service Desk for support (though they also generally do not need it.)
3. We are currently deploying Semi Annual (Broad) to all of our machines that get Windows 10. That number is not very large right now. We moved to Windows 10 on all of our Surface Pros because of the ability to work with multiple resolutions was much more advanced than Windows 8.1. We are also testing a few users with Windows 10 so that we can try to get ahead of the 2020 deadline for the removal of Windows 7.
4. Long Term Servicing channel doesn’t support our needs at all. We don’t use it. Unless you have mission critical machines that can’t/don’t ever need to change over the next 7 years, you shouldn’t be using it either.

Here’s an outline of how we are doing Windows 10 servicing in our user environment.

Servicing Plan for Windows 10

1. During Operating System Deployment (OSD), the newest version of Semi Annual (Broad) will always be installed. (currently this is the 1703 version.)
2. Monthly patches will be provided through our “normal” monthly patching process.
3. When support expiration is announced for a specific version, two branches of Semi Annual (Broad) will become available in Software Center for users of the expiring version. (Example, when 1703 expiration is announced, the two active CBB versions (1709 and 1803 will be made available)
4. When support expiration is reached, if the user has not updated their computer they will be required to install the newest of the two CBB Branches. (Example, when 1703 expiration is reached, users will be required to update to 1803 rather than 1709, which is the older version.) Jumping them ahead a version should make it so the user will need to update the computer less frequently.

How we are making it happen

1. We will create device collections in ConfigMgr that include machines with each of the Windows versions. (Example: machines with 1607 Pro and Ent, 1703 Pro and Ent, 1709 Pro and Ent, 1803 Pro and Ent)
2. We will create upgrade task sequences in ConfigMgr for each Windows 10 version as they are released by Microsoft.
3. Those Upgrade task sequences will become available in Software Center for members of the Windows 10 device collections as versions become Semi Annual (Broad). Example: when 1709 becomes Semi Annual (Broad), it will be an available update for members of the 1703 and 1607 device collections.
4. Task sequences will become required as branches reach End Of Life so that computers will not be unpatched in the environment. Example: When 1607 becomes EOL, the 1709 Upgrade Task Sequence will become a required install for computers in the 1607 device collection.

So I think this is all correct. With the Semi Annual channels, Microsoft is promising two releases a year in March and September. It will be interesting to see if they can keep up with the pace, and what happens if they run into a big bug right before the release of one of the semi annual updates. Do they keep the name the same and just release it a bit late, or will they feel obligated to call it 1811 if the release comes out in November rather than September as promised. Time will tell.

 

 

One of the most useful tools that exists within SCCM is the Task Sequence. Often when we first use ConfigMgr we think of Task Sequences as being used in Operating System Deployment, and don’t get much further than that. At least I didn’t. However, the truth is that Task Sequences can go way beyond Operating System Deployments, can be deployed to any computer and can do some great things that can’t be done other ways.

So Why would I want to do that?

1. So that you can link things together that normally can’t be linked together. Think about all of the things the Task Sequence is doing in an OSD scenario. Even the most basic OSD Task Sequences format the hard drive of your machine, install an OS, load drivers, and install applications. That is a huge variety of things that can normally not be linked together in ConfigMgr. Sure, It is possible to have an application install be dependent on another install. For example you can tell SCCM to install Salesforce for Outlook, but before it does that, make sure that the required C++ versions are installed. However that will only work if both your Salesforce for Outlook and C++ installs have been created as Applications (or Packages.) If for some reason one of your required apps installs better as a package, and your other installer is an application, they can not be linked together. However, if you create a task sequence, you can have it first install the required package, and then after that install the application you needed in the first place. It’s as simple as creating a two step task sequence one step for the package, and one step for the application.The above screen shot uninstalls Java using a Package, and then installs Java using an Application.
2. Do you have two applications (or an application and a package) that need to be run, but you need a reboot between the applications? Task Sequences can reboot a machine, and then continue on with the task sequence and with the steps that are following. We often use this when we are removing an older version of software (like say Office 2010) before installing the new version (Office 365) when the installer doesn’t clean up the old version. In this example our task sequence looks like this. We run the uninstaller app and restart, and when the computer comes back up it will run the installer application. 
3. Other more complex Task Sequence “things”. I will describe this in more detail in an upcoming blog post, but we also use task sequences to update the BIOS/UEFI on our machines quarterly. In this scenario, we run a task sequence step to disable Bitlocker on our laptops and Surface Pros, run the bios update software, reboot, and then reenable Bitlocker after the machine comes back up. 

There are a couple of things to note about using Task Sequences in this way.

1. Task Sequences, just like pretty much everything else that can be deployed in ConfigMgr, can be made to be available, or to be required. You can have them show up in Software Center to be run, or you can hide them from software center. They will show up as they are running with the “running task sequence” dialog box.
2. Task sequences have a cool feature that allows you to download all of the task sequence steps before beginning. That means it won’t download the software from each step as needs it, it will cache it beforehand. This will be extremely helpful if the machines that are being installed to are on slower connections.
3. As of ConfigMgr 1703, Task Sequences can only be assigned to computers or collections of computers. They can not be assigned to users or user groups. So remember it may take a little while for the Task Sequence to show up in the Software Center window.

That’s it for now, I will get into some of the nitty gritty of how we have done things like enabling Bitlocker remotely on laptops using a Task Sequence, and how we uninstalled Office 2010 and installed Office 365 on all of our user machines using a task sequence but that will have to be in an upcoming post.

How Windows 10 versions should be updated was something I had a ton of trouble understanding from an enterprise perspective. On one level as master of my own machine, I could install every new version when it was released. That would have been the original install in July 2015 (version number 1507), and upgrades in November 2015 (1511) , July 2016 (1607), and March 2017 (1703).

But then we started installing Windows 10 in our user environment. We gave those machines to our Executives, on Surface Pros. Suddenly it became more important to figure out how we were going to upgrade machines when versions of Windows 10 became end of life. Oh yes, versions of Windows 10 become end of life, and after that they won’t be getting any further security updates. No pressure right?

Check with Microsoft?

Microsoft has a lot of information out there on how the different “rings” and “branches” work on their website. Feel free to search for it and take a look. However, when I was done I was still confused how it all worked together and what the different branches of Windows 10 even meant.

Eureka!

The real answer of what to do with Windows 10 servicing came when I officially comprehended (at the Midwest Management Summit of all places) that each of these “Branches” of Windows 10, is really the same version, just at a different place in its support lifespan! No wonder I couldn’t find the Current Branch for Business release anywhere on the Volume Licensing site. Suddenly thinking about how to do Windows 10 Servicing with ConfigMgr became much easier

Background on Windows 10 Branches

Each Windows 10 release has 18 months from street date to end of life. During that time the releases go through different “branches” or their life.

As of today, Windows 10 has four different branches. They are:

1. Preview Release: This is not part of the 18 months of Street Date to EOL. Microsoft would love for you to install this on one or two machines so they can be assured that it’s working correctly. Only ever put it on a machine that is not critical, and can be completely reimaged if necessary.
2. Current Branch: This is what Windows 10 is considered immediately after it is released, and it remains until the day Microsoft removes it from support.
3. Current Branch for Business: After Current Branch has been released for approximately three months or so, Microsoft considers that it is ready for Enterprise customers, and adds the version to what is called Current Branch for Business. There will always be two releases of Windows 10 that are considered Current Branch for Business, and it is possible that there will be three at some times.
4. Long Term Servicing Branch: This branch is for very specific machines, and Microsoft recommends that it is only used for very specific needs. In this branch, the actual features of the OS will never change. Microsoft will provide security updates, but that is the extent of the support. Don’t use this in a real world environment unless you know you need it. Don’t do it. Someone will want to bring a new machine into your environment in a couple of years and it won’t work, because it won’t support new features.

So the real question is how do all of these fit into an actual working user environment?

The User Environment

Here is a run down of how our environment stacks up with each of the releases.

1. I think one of our network guys runs the Preview Release of Windows 10 on one of his machines. Maybe. We largely ignore Preview Releases of Windows 10. Sorry Microsoft.
2. For our company we allow the infrastructure team to install Current Branch on their primary machine. They do not have access to the Service Desk for support (though they also generally do not need it.)
3. We are currently deploying Current Branch for Business to all of our machines that get Windows 10. That number is not very large right now. We moved to Windows 10 on all of our Surface Pros because of the ability to work with multiple resolutions was much more advanced than Windows 8.1. We are also testing a few users with Windows 10 so that we can try to get ahead of the 2020 deadline for the removal of Windows 7.
4. Long Term Servicing Branch doesn’t support our needs at all. We don’t use it. Unless you have mission critical machines that can’t/don’t ever need to change over the next 7 years, you shouldn’t be using it either.

Here’s an outline of how we are doing Windows 10 servicing in our user environment.

Servicing Plan for Windows 10

1. During Operating System Deployment (OSD), the newest version of Current Branch for Business will always be installed. (currently this is the 1607 version.)
2. Monthly patches will be provided through our “normal” monthly patching process.
3. When support expiration is announced for a specific version, two branches of CBB will become available in Software Center for users of the expiring version. (Example, when 1507 expiration is announced, the two active CBB versions (1511 and 1607) will be made available)
4. When support expiration is reached, if the user has not updated their computer they will be required to install the newest of the two CBB Branches. (Example, when 1507 expiration is reached, users will be required to update to 1607 rather than 1511, which is the older version.) Jumping them ahead a version should make it so the user will need to update the computer less frequently.

How we are making it happen

1. We will create device collections in ConfigMgr that include machines with each of the Windows versions. (Example: machines with 1507 Pro and Ent, 1511 Pro and Ent, 1607 Pro and Ent, 1703 Pro and Ent)
2. We will create upgrade task sequences in ConfigMgr for each Windows 10 version as they are released by Microsoft.
3. Those Upgrade task sequences will become available in Software Center for members of the Windows 10 device collections as versions become Current Branch for Business. Example: when 1703 becomes CBB, it will be an available update for members of the 1511 and 1607 device collections.
4. Task sequences will become required as branches reach End Of Life so that computers will not be unpatched in the environment. Example: When 1507 becomes EOL, the 1607 Upgrade Task Sequence will become a required install for computers in the 1507 device collection.

So that’s how I was able to get my head around the servicing of Windows 10 with ConfigMgr… hopefully this helps you think about it as well.