Saying the Right Thing the Right Way

Saying things the right way can make all the difference

Working with the security team in our organization can be challenging. Often they come into meetings with preconceived ideas about how things work or how things are supposed to work. The default answer to a new technology or new request from the business or a user is “no” rather than the “how can we do this securely” that I would prefer.

That changed on a very important technology yesterday, and that was because of someone being able to say the right thing the right way.

The story starts two years ago, which makes it over a year before I got here. Security was working on a requirement that drives on all laptops and tablets be encrypted. Microsoft was brought in for a POC to get MBAM up and running in the environment. Everything was working great until someone from the Security Team said “well now we need to turn on FIPS compliance.” The Security Team wanted to be compliant to the Federal Information Processing Standards. This is fine, until you turn FIPS on for the Desktop. When you do that, every certificate and every connection then becomes FIPS compliant. When that happens it changes how everything is encrypted and it basically breaks EVERYTHING. Additionally the MBAM clients stop talking to the MBAM server, and you have no way of reporting on what is encrypted and what is not. Because of this issue it was decided to use Symantec Endpoint Encryption instead of Bitlocker.

Recently the Security team was given a presentation on security by Microsoft and the question of Bitlocker came up yet again. The response from the security team was “have you fixed FIPS compliance yet?” To Microsoft’s credit, they asked some important questions that found some interesting answers. The way that FIPS had been enabled earlier is well known to break everything. In fact Microsoft recommends against doing it that way. In addition, they noted that it is not required to have FIPS “turned on” on each endpoint to be FIPS compliant. They provided resources to the security team that showed why this works and how other organizations are using Bitlocker in FIPS compliant environments.

The bottom line was, in order to be FIPS compliant an organization does not need to turn on the FIPS switch on the computers and break everything. FIPS compliance can also be achieved by using FIPS compliant algorithms in the encryption. News flash, the default algorithms for Bitlocker are already FIPS compliant.

Once this was explained and accepted, we are now moving down the road to removing SEE from the environment and installing Bitlocker. And we are right back where we should have been two years ago.

An (old) ugly problem rears its ugly head

We ran into an issue yesterday that continued into today that is an oldie but a goody… Windows 7 computers started showing up with a black background and claiming that “This Copy of Windows is not Genuine.” It’s one of those things that when you see it you immediately start to look at the KMS server to make sure that it’s accepting connections and is working correctly. It was.

We contacted Microsoft, and before we even had an engineer assigned our TAM had responded with a solution. The problem is a KB (KB971033) installed on Enterprise Windows 7 systems that was meant for retail versions of Windows 7.

the KB is from 2010 and doesn’t even exist in our SCCM updates, so the computers that have it must have gotten it from some lawless time when they were being updated in a different way. Apparently it started causing problems again yesterday, but looking through the web I was able to find several others that have had the problem surface at random times through the years.

Anyway, here are the instructions on how to fix the problem.

Here is the procedure that you need to follow –

  1. Uninstall KB971033 (if it is still installed)
  2. Reboot the machine
  3. Run this commands manually, or through a .bat script on ONLY problematic Windows 7 Enterprise machines.
net stop sppuinotify
sc config sppuinotify start= disabled
net stop sppsvc 
del %windir%\system32\7B296FB0-376B-497e-B012-9C450E1B7327-5P-0.C7483456-A289-439d-8115-601632D005A0 /ah 
del %windir%\system32\7B296FB0-376B-497e-B012-9C450E1B7327-5P-1.C7483456-A289-439d-8115-601632D005A0 /ah 
cd %windir%\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\SoftwareProtectionPlatform
ren tokens.dat tokens.oldbad
cd cache
ren cache.dat cache.oldbad
net start sppsvc 
cscript c:\windows\system32\slmgr.vbs /ipk 33PXH-7Y6KF-2VJC9-XBBR8-HVTHH 
cscript c:\windows\system32\slmgr.vbs /ato
sc config sppuinotify start= demand

The key above is for Windows 7 Enterprise specifically, if you have other editions please refer to the following article and change the product key in the above command.

KMS Client setup keys – https://docs.microsoft.com/en-us/windows-server/get-started/kmsclientkeys

 

Windows Updates released out of band 11/28/2018

I always love when Microsoft releases out-of-band patches for Windows.

It seems that these updates kind of get swept under the rug (unless it’s a huge vulnerability) because everyone already has patch processes defined that are centered around Patch Tuesday.

Our testing and partial roll-out process basically takes the whole month before the updates are finally installed. Because of this, updates like the ones released yesterday aren’t part of our updates until next month.

Link to info about the updates

When your day isn’t going well…

At least you didn’t have to send an email to the whole Health System with the following text:

“It has come to our attention that the flyer for the career fair sent out yesterday indicated the incorrect date.  The flyer has been modified to now include the correct date and location of the event.  Thank you.”

It was sent to everyone. In the whole Health System. Once with the wrong information, and once with the correct information…

Microsoft Windows 10 Servicing Changes, Windows Virtual Desktop (The beginning of the end of locally installed Windows?)

At this point this blog should probably be called “Microsoft servicing. When it changes I’ll write a new blog.”

So it’s official, Microsoft has changed the servicing model for Windows 10 yet again. This time the changes are quite large, including a “tick-tock” schedule for Windows 10 Enterprise and Education versions. The full details can be read here:

https://www.microsoft.com/en-us/microsoft-365/blog/2018/09/06/helping-customers-shift-to-a-modern-desktop/

There are some important things to note from the document above. First of all, the support schedule for Windows 10 Enterprise and Education has changed to 30 months for the September Update. “Tick-tock” comes into place because the alternate update (the March update) will still have 18 months of support.

Changing the support time for Windows to 30 months is great news for those of us who are unable to move to a new version of Windows 10 for several months because of software needing to be certified for the new OS. For example we have a piece of software that is installed on every computer in the environment. The time between Windows 10 release and when the software is certified to use in the OS can be as long as 6 months. Because of this we were facing only 12 months of use on a specific Windows 10 version before we would need to upgrade it again. If you add in the time to deploy to our 16,000 computers, upgrades would be happening constantly. This was not going to be a great solution for our users, and the amount of support IT was going to have to provide for this model was going to increase greatly.

Microsoft listened to us as corporate administrators. Let me say that again. We talked, and Microsoft listened to us. Microsoft clearly wanted to be able to move Windows 10 versions out of support quickly so they could move development time to newer versions. They also clearly didn’t want a repeat of Windows 7 where they will be providing updates until 2020 for software that was originally released in 2009. However we made enough noise, and Servicing was changed!

With 30 months of support for the September Windows 10 version, even if the certification of our software takes 6 months, we still should have 24 months of support for each version of Windows 10! The real question is who is going to be installing the Enterprise or Education March versions? I would wager there won’t be many organizations.

Windows 7 Extended Security Updates

The second announcement in the above article is about the extension of servicing for Windows 7. Microsoft is calling it Windows 7 Extended Security Updates (ESU). If you need to keep computers on Windows 7 for some reason past 2020 (even though you’ve known about Windows 7 retirement for several years at this point) Microsoft will let you purchase extra support until 2023. But it’s going to cost you. Microsoft isn’t saying how much at this point, but it is saying that you can expect the cost to go up each time you renew the support (up to January of 2023).

Windows Virtual Desktop

At Microsoft Ignite this week Microsoft has introduced something else that is desktop related. It’s called Windows Virtual Desktop. To read the announcement check out the website below:

https://azure.microsoft.com/en-us/blog/microsoft-365-adds-modern-desktop-on-azure/

Windows Virtual Desktop is much what it sounds like. It’s a full blown Windows 7 or 10 desktop that exists in Azure. You connect through the internet to your Virtual Desktop which will come pre-loaded with Office 365 applications. They were very clear to mention that there is a browser plug in that can be installed so that you can connect to your virtual desktop right through your web browser. The demos were also quick to note that connection speeds were great, the demos were run from Orlando, FL and were using an Azure data center in the Seattle area. No word on if the issue with Azure a couple of weeks ago would have affected your WVD in a negative way (I bet it would have).

Another thing that is important to note about the WVD is that you can create Windows 7 desktops. When you do this, they will automatically have the Extended Security Updates, which means this might be a good way to get updates until 2023 for users that need it.

It’s hard to say how much of an impact WVD’s will have. Certainly I can run a VMware virtual machine in a web browser if that is something that I would want to do, but of course that would require a separate license of both Windows and Office. If Microsoft can come in at a cheaper price than someone would be paying otherwise, it is definitely something to look at.

Here’s another thing to think about; Is this our first glimpse at what our security team has been worrying about for some time now? Is this the beginning of the end of a legacy Windows operating system running on a local computer? Is this the beginning of a future “desktop” where all you need to connect to it is a web browser? This presents a lot of benefits for those of us who are managing computers, it also adds a lot of challenges.

 

But Seriously, check the run times of your Updates…

Just got to work this morning with several computers that failed our Windows Updates deployment with the exact same issue as the one I most recently blogged about, the “updates handler job was cancelled.”

I checked which ones were failing, and it was the “Cumulative Update” for whichever version of Windows was on the computer.

The run-times for the Cumulative Updates had been reduced to an even shorter time of FIVE MINUTES?! We absolutely have computers that will not be able to complete this in a five minute time frame. I switched it back to 60 minutes.

Why did these move to 5 minutes? No idea. Maybe Microsoft knows…

New Environment, New Caveats

Coming into a new environment it’s fun to see all kinds of new errors and new ways that SCCM has difficulty with things. Moving from a small (~2000 computers) to a larger (~15000 computers) seems to make this even more interesting.

Here’s an example of something that we didn’t see in my previous environment.

Last night was the night that our first large test group had reboots forced upon them for this month’s Windows patches. The deployment went fairly well ~500 installed correctly, but there were still a few machines that need remediation ~100. Okay, from what I’m understanding it went as normal per their expectations, there were more failures than I expected.

A large number of the computers that failed ended up failing for the same reason. The error code was “0x87D00664 Updated handler job was cancelled.”

failed to install

A quick web search comes up with a few good descriptions of what the error actually means (updates handler job was cancelled is just not that descriptive,) and how to fix it.

What actually happened

 What the error is actually telling us happened is that SCCM timed out when trying to install the update, and because it hit the preset time out limit, it stopped installing. At that point it failed the process. So what can we do about it

Possibly unbeknownst to many people, each update that comes into SCCM has a maximum run time associated with it. That means if the update isn’t finished installing within that time it is considered failed and will stop installing. Microsoft assigns those runtimes and they are different for different types of updates. The two that were failing for us, cumulative updates for Windows 10 1607 and Windows 1703 were assigned a run time of 20 minutes.

To find this information go to the update in question in Software updates -> All software updates.

allsoftwareupdates

Right click on it and click properties. One of the tabs up above will be “Maximum Run Time”. Change it to something higher than it is currently set.

cuinfo

We let our installs run overnight, so giving these updates a runtime of 60 minutes should be fine. Your mileage may vary.

This seems to have taken care of the issues that we have had with updates. Just to be safe for the next couple of Cumulative Updates I might change the run times just to make sure this doesn’t happen on such a large scale again.

So why didn’t I see this in my earlier environment?

This is an interesting question. It doesn’t seem like this should be something that only happens in large vs small environments since it is the actual install running on the computers that it having the problem. I’m going to narrow it down to one of two things.

  1. This error is potentially having this problem everywhere. It’s possible that this update just takes a bit longer, and that’s why I’m seeing it now and haven’t seen it before.
  2. It could also be compounded by the fact that these are slower computers than I am used to having Windows 10 installed on. All of the computers that had Windows 10 installed in my previous environment were newer with fast SSD’s. It would be expected that those should install faster. Maybe it’s just slower hardware.

Anyway, things are working better now. It’s always fun to see what ConfigMgr throws our way.

Here are the resources I used to help with the remediation of this issue:

http://ccmblogs.co.uk/?p=132

https://blogs.msdn.microsoft.com/minfangl/2012/08/17/configuration-manager-may-fail-to-install-exchange-rollup-kb2706990-kb2734323-because-of-timeout/

What did I learn today…

So the preface of this article is that I have moved to a new company! I’m still working in ConfigMgr everyday, but it’s a slightly different environment. I’ve moved from being the primary person in a 2,500 Windows client environment, to a member of a team of 6 managing 18,000 computers in a much more distributed environment. That means my new employer has many more locations.

I ran into an issue that I haven’t had to deal with previously yesterday, and if this helps someone then great! I’m glad my embarrassment can be useful for you.

Just starting I’m jumping into some menial tasks, one of which was working with the Windows Updates that came out this patch Tuesday. No problem I can go through those and make sure that everything looks good. They mentioned that the current deployments, for January – April had not been condensed into one deployment yet. Cool I can do that too.

So I got the patch Tuesday deployment ready with the help of a co-worker, and then started moving the group membership of the January – March patches into the April patch collection. These groups were still deployed at the time.

Do you see the issue I just created? I didn’t until some of our smaller sites (with slower network connections) started complaining of network slowness. The network guys also noted that the links were becoming saturated.

Investigation determined that it was computers connecting to our WSUS servers to determine if they needed the patches in the April patch collection or not. Lots of computers were checking. All the computers were checking! They were doing it very slowly (because we have BITS throttling in place) but so many were doing it that the computers were still clogging the slower links and causing the problem.

The interim solution was to delete the deployment that was causing the computers to check in, and to stop the WSUS service on the WSUS servers. The effect was immediate and networks recovered almost immediately.

There was a mitigating circumstance that the person that I was working with was not aware of. Sometime in the last few months the WSUS servers were “right-sized” for the number of machines that were connecting. So, in the past, when changes like the ones we made were done, the bottleneck was on the WSUS servers themselves. They would be pegged at 100% CPU for “days” when this type of change was made. Computers would be denied access, and they would check back later to see what updates they needed. When the WSUS servers were expanded, they were able to process all the requests that were coming to them. They were able to process so well that the new bottleneck became slower network connections.

Moral of the story: changing the content of the deployment made computers in the environment re-evaluate themselves against the whole group of patches in the collection, not just new ones. In our situation the traffic between the client and the server was able to significantly slow down network traffic at slow locations.

Second Moral of the Story: When you fix one thing, sometimes you shift the bottleneck and break something else, make sure you are ready for that.

There is also an article on Microsoft’s website about a known issue that causes WSUS to cause high network bandwidth that is located here https://support.microsoft.com/en-us/help/4163525/high-bandwidth-use-when-clients-scan-for-updates-from-local-wsus-serve it’s possible that this caused some additional problems with our collection, but it certainly wasn’t our only issue.

I try to learn something new every day. This was my learning for yesterday.

 

 

Software Center not working after update to Windows 10 from Windows 7

Or “Pending updates, the bane of my existence”

Depending on who you talk to, we are either in the middle of or beginning the process to update all of our user workstations from Windows 7 to Windows 10. During testing I have come up with some fun errors, most of which I have been able to search quickly on Google and come up with the solution. I did have one that was a little bit trickier.

After I updated one of my test Virtual Desktops, which was one of the oldest images in the company, to Windows 10 1709 Software Center would no longer open. One of two things would happen, and they often alternated with each other. The first is the standard “Software Center could not open” error.

software center won't open

We have protocols to follow when this message shows up. Often the problem is that the ConfigMgr client is left in something called “Provisioning Mode” which makes Software Center give the above error. Microsoft recommends running the below command on the machine when this happens:

Powershell.exe Invoke-WmiMethod -Namespace root\CCM -Class SMS_Client -Name SetClientProvisioningMode -ArgumentList $false

This command can usually can clear the error and make things work correctly again. But it didn’t in this case. In fact, looking at the registry, I could tell that the client was not stuck in provisioning mode. So it had to be something else.

The alternate error that I was getting was a .net error that had a whole bunch of stuff that didn’t mean much to me. The error seemed to insinuate that .net was not installed correctly. With this knowledge I tried the following steps.

  1. Uninstalled and reinstalled the ConfigMgr Client through the console– no change
  2. Removed and re-added .net 4.7 – no change
  3. Uninstalled and reinstalled the ConfigMgr client manually -no change
  4. Ran the Microsoft .net 4.7 repair tool – no change

Finally, I went back to searching the Internet to see if there was anything else I could find with all my new knowledge (which didn’t change anything at all). I finally found a Reddit post from 8 months ago that had a little tidbit in it.

Screen Shot 2018-02-21 at 10.09.00 AM

At first I didn’t take this too seriously. We use ConfigMgr for all of our software updates so I didn’t think there could possibly something waiting in Windows Update. But after doing a little more searching I finally got back to checking Windows Update. Here’s what was waiting for me:

update pending

I took this screenshot after I had started the update installing, but there was definitely a Windows Update that had snuck onto the machine at some point. After completing the update and rebooting the machine Software Center opened correctly and is installing software correctly.

So here’s a new reason why Software Center won’t open… Make sure to check your pending updates, even if you use ConfigMgr for updating.