Smart Check – Monitor Your Citrix Sites

Citrix Smart Check is a software and a service that installs on a Citrix Delivery Controller and collects diagnostic data, sends it to the Citrix Cloud account, where it gets analyzed and presented on the Citrix Cloud website. The information helps Citrix administrators to prevent and resolve issues before they happen or impact the users, give recommendations on fixes and to keep the Citrix environment stable.

The Smart service helps Citrix administrators that do not have their own monitoring setup or are unable to monitor their sites for other reasons and presents it on a webpage overview. The administrators can also get scheduled summarized mail reports regarding errors, warnings and information regarding the state of the different sites.

Citrix Cloud Smart Tools

Smart Check – Sites Overview

What Smart Check provides

  • Overview of the Citrix sites and products used, site-by-site
  • An extensive diagnostic and health checks for the different sites and services
  • Scheduled health controls of Delivery Groups, StoreFronts, Delivery Controllers, Machine Catalogs, Provisioning and License Servers
  • Give recommendations what administrators should do with the site to keep it up-to-date and stable
  • Help with simplified troubleshooting and pin down where the issue may be impacting users
  • Upload diagnostic data to Citrix Insight Services (CIS)
Smart Check - Overview

Smart Check – Overview

How to get started

First, you need a Citrix Cloud account. Register an account at https://smart.cloud.com. After you have created an account you can login, click Add Site and download the Smart Check software. The software should be installed on a Delivery Controller on the site and comes with a one-time signed JSON Web Token (JWT) that is used to connect your site to the Citrix Cloud – Smart Tools service.

Smart Tools - Add Site

Smart Check – Steps to take

Add Site - CitrixSmartToolsagent.exe

Add Site – CitrixSmartToolsagent.exe

Once the Smart Check agent is installed it will show up on the Citrix Cloud – Smart Check webpage as Site Discovered. You will need to click on Complete Setup and provide a domain user account that is a member of the local Administrator group of the Delivery Controller and full administrator role in Citrix Studio. PowerShell 3.0 or greater needs to be installed on the Delivery Controllers and outbound internet access on port 443 enabled to be able to upload to Citrix Cloud.

Smart Check - Site Discovered

Smart Check – Site Discovered

Smart Check - Enter Credentials

Smart Check – Enter Credentials

For VDA the following must be enabled:

  • File and Printer Sharing
  • Windows Remote Management (WinRM)
  • Windows Management Instrumentation (WMI)

For a full list of requirements and supported site components, visit Citrix Product Documentation – Smart Check requirements.

Smart Checks

Below is a list of the checks that are available as of this post. There are probably more to come:

  • Site Health
  • Citrix Optimizer
  • Citrix Provisioning
  • Delivery Controller Configuration
  • License Server
  • LTSR Compliance
  • Product LifeCycle
  • StoreFront
  • VDA Health

Each category contains several checks. You can read an excerpt of the different checks performed below.

Site Health Checks

Site Health Checks provide a comprehensive evaluation of all the FMA services including their database connectivity on your Delivery Controllers. Citrix recommends you run these checks at least once daily. Site Health Checks verify the following conditions:

  • A recent site database backup exists
  • Citrix broker client is running for environment test
  • Citrix Monitor Service can access its historical database
  • Database connection of each FMA service is configured
  • Database can be reached by each FMA service
  • Database is compatible and working properly for each FMA service
  • Endpoints for each FMA service are registered in the Central Configuration service
  • Configuration Service instances match for each FMA service
  • Configuration Service instances are not missing for each FMA service
  • No extra Configuration Services instance exists for each FMA service
  • Service instance published by each FMA Service matches the service instance registered with the Configuration service
  • Database version matches the expected version for each FMA service
  • Each FMA service can connect to Configuration Logging Service
  • Each FMA service can connect to Configuration Service

Citrix Provisioning Checks

Citrix Provisioning Checks verifies Citrix Provisioning status and configuration.The following checks are performed:

  • Installation of Provisioning Server and Console
  • Inventory executable is running
  • Notifier executable is running
  • MgmtDaemon executable is running
  • StreamProcess executable is running
  • Stream service is running
  • Soap Server service is running
  • TFTP Service is running
  • PowerShell minimum version check
  • Database and Provisioning server availability
  • License Server connectivity
  • Provisioning Update Check
  • PXE service is running
  • TSB service is running

StoreFront Checks

StoreFront Check validates the services status, connectivity to Active Directory, Base URL setting, IIS Application Pool version and the SSL certificates for Storefront, and verifies the following conditions:

  • Citrix Default Domain Services is running
  • Citrix Credential Wallet services is running
  • The connectivity from the StoreFront server to port 88 of AD
  • The connectivity from the StoreFront server to port 389 of AD
  • Base URL has a valid FQDN
  • Can retrieve the correct IP address from the Base URL
  • IIS application pool is using .NET 4.0
  • Certificate is bound to the SSL port for the host URL
  • Whether or not the certificate chain is incomplete
  • Whether or not certificates have expired
  • Whether or not certificate(s) will expire within one month

VDA Health Checks

VDA Health Checks help Citrix administrators troubleshoot VDA configuration issues. This check automates a series of health checks to identify possible root causes for common VDA registration and session launch issues.

  • VDA software installation
  • VDA machine domain membership
  • VDA communication ports availability
  • VDA services status
  • VDA Windows firewall configuration
  • VDA communication with each Controller
  • VDA registration status

For Session Launch:

  • Session launch communication ports availability
  • Session launch services status
  • Session launch Windows firewall configuration
  • Validity of Remote Desktop Server Client Access License

Closing words

You can run checks manually, but it is also possible to schedule (recommended) the different health checks and get a summarized report daily or every week at designated time of day. The summary gets mailed to the registered Citrix Cloud account and to view more information you need to logon to the Smart Cloud website.

It is possible to view previous reports of the Smart Check runs and hide alerts that has been previously acknowledged:

Smart Check Health Alerts

Smart Check – Health Check Runs History

Under Site Details you can view components or add new ones. If needed it is also possible to Edit Site Credentials, Sync Site Data or Delete the Site:

Smart Check - Site Details

Smart Check – Site Details

Smart Check is supported both on-prem and in the Citrix Cloud environment.
It is easy to setup and brings a great deal of value. You should try it out! Let me know how it went in the comments down below.

Smart Tools contains Smart Checks and Smart Scale. Smart scale helps reduce your XenApp and XenDesktop on Azure Cloud resource costs. But this will be in covered another post.

Source: https://docs.citrix.com/en-us/smart-tools/whats-new.html



Netscaler: Resolve large http POST packets against AAA protected LB vServers

Recently while working with two customers and setting up load balancing and external access to some of their applications, I’ve encountered issues with large POST packets (either large form submits or file uploads in the web app) stop working when AAA is activated on the load balancing (LB) vServer. In these cases the backend web applications have all been quite old and not exactly well-written coding-wise.

By default when AAA is activated on Netscaler, any large POST packet sent to the LB vServer, and subsequently to the backend, will be split up into two packets with the first one having a Content-Length header value of 0 and the secondary packet containing the correct Content-Length header value and the actual POST body. The backend code in this case is unable to handle the initial 0 length POST packet, and therefore crashes or bugs out before processing the second, real packet. See picture below of the WireShark capture showing the two packets.

IP adress 10.200.40.159 is a SNIP and 10.200.41.131 is the backend server hosting the web application.

 

The solution is to either have the backend software rewritten to handle above case, or to run a special Netscaler command which will disable the split-post-packet-into-two-packets behaviour. Article https://support.citrix.com/article/CTX225681 contains the relevant info. To resolve this through Netscaler, run the following command in shell on Netscaler, nsapimgr_wr.sh -ys arg1=0 -ys arg2=1 -ys arg3=16 -ys call=”set_sso_post_data_handler”. You have to run the command on all Netscaler nodes (if HA/Cluster) and you also have to put this command line in the file /nsconfig/rc.netscaler, otherwise the change will be lost on the next Netscaler reboot. See https://support.citrix.com/article/CTX122271 for more info regarding the rc.netscaler file.

After the change, you can see how from below picture that only one packet is sent.

One interesting thing to note is that the article https://support.citrix.com/article/CTX225681 doesn’t mention Netscaler version 12.1 as affected, but my guess is that nothing has changes between 12.0 and 12.1 regarding above (ie you still need to run above command to resolve it).

Feel free to email me at rasmus.kindberg@xenit.se, or leave a message here on this blog post, if you have any questions or comments.



Automating delimiter selection when working with Csv cmdlets

Recently I walked passed a collegue with an error on his Powershell console which peaked my interest. He quickly noted that he had chosen the wrong delimiter for the Csv he imported which resulted in errors in the code, I then jokingly said ”Why don’t you just use the ’Get-CsvDelimiter’ cmdlet?” and we had a quick laugh.
30 minutes later the ”Get-CsvDelimiter” function was born, but first,

Let’s dive into how Import-Csv, ConvertFrom-Csv & delimiters really work

When used correctly Import-Csv & ConvertFrom-Csv creates an array object with each row as a PSCustomObject with named content based on headers.

Import-Csv has two constructors, one with Path as a required and the other with Path & Delimiter as required.
If you omit the delimiter parameter the Path constructor will be used, now this constructor has a required parameter (according to the documentation) that is auto filled. Instead of using Delimiter we can use ”UseCulture” which takes the current culture delimiter as input,

”To find the list separator for a culture, use the following command: (Get-Culture).TextInfo.ListSeparator.” –docs.microsoft

If we interpret the documentation this is a required parameter, but it is not really required, as we are able to pass only the -Path parameter and delimiter will autofill

Assume we’re using the following CSV as input ($File)

the command ”Import-Csv $File” will output a valid object. Now in my opinion it should fail. As my culture specifies semicolon (;) as the default listseparator, while forcing ”-UseCulture” can’t produce a correct csv object. This basically means that UseCulture is not a required parameter and the default unless specified is always a comma (,)

The exact same goes for ConvertFrom-Csv.

Presenting Get-CsvDelimiter

The below function will search and find the most probable delimiter used in a Csv file.

Let’s see how it works.

$File is a csv file with the below data.

If we run Import-Csv $File we’ll get a incorrect table object, every row will have a single property containing the row data.

If we instead specify a delimiter as shown below, we’ll calculate the most probable delimiter and use that to produce a correct CSV table object without having to inspect the csv culture or assume anything.

And to prove it works, the below code outputs only the Name column in the Csv

If we have a stringobject and need to convert it that is also possible.

With this method we can use virtually any delimiter we want, assuming that the Csv is formatted correctly. Again we’re using the same csv input but we change the delimiter to a ”greater than” (>) symbol and see how the function performs.

Now when we run the function standalone we’ll get a Greater than symbol as the return

And finally, when running the previous code we get the same output as when we were using a semicolon as the delimiter.

 

Get-CsvDelimiter



VDA stuck when updating

Sometimes (no idea why, seems random to me) the VDA update is stuck after prompted to reboot in the middle of the update like below.

VDA Update Stuck

After going through the logs (Get-content ”$env:LOCALAPPDATA\Temp\Citrix\XenDesktop Installer\XenDesktop Installation.log” -wait) I noticed that the VDA setup creates a RunOnce key just before the restart.

VDA Update Powershell Log

After checking the registry path ”HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce” there was indeed a key created with the name ”!XenDesktopSetup”.

VDA Update Stuck Registry

After deleting this key the update proceeded immediately.

VDA Update Proceed

I guess the registry key should delete it self after a reboot, but for whatever reason it doesn’t. This update was from 7.18 to 1808, but have happened to me in all different versions from 7.15 and later.



Mapped network printers unavailable due to SMB1 being obsolete

INTRODUCTION

As we all might be familiar with, printers are one of those little peculiar matters within IT. Implementing these in an IT-environment is self-explanatory oftentimes, but when they do not cooperate the issue itself can stem from one single obscure root cause, if not a string of these having to be checked upon.

Recently, I encountered a particular printer issue which I found interesting enough to share. The root cause here, in summary, was due to the network protocol SMB1 (Server Message Block) being obsolete in recent Windows releases.



Run Windows Defender inside a sandbox

Last week Microsoft released the news that they have added a new feature to Windows Defender Antivirus. The new feature allows Windows Defender Antivirus to run within a sandbox.

Other antivirus providers have been offering the possibility to open files in a sandbox-environment before but what Windows Defender now offers is the feature that all virus scans are done inside a virtual sandbox. The biggest benefit of running the virus scans in a virtual sandbox is when the antivirus engine is scanning a malicious file. The malicious code that usually would be executed to exploit a vulnerability will now only affect the virtual sandbox and not the actual computer resources.

This new feature proves that Microsoft are really making an effort to increase the reputation of Defender. There is a debate whether this is the best way to go but it is good sign that Microsoft is really making an effort to develop Defender Antivirus with features like this.

Microsoft describes the feature in the following way on their blog:

Running Windows Defender Antivirus in a sandbox ensures that in the unlikely event of a compromise, malicious actions are limited to the isolated environment, protecting the rest of the system from harm. This is part of Microsoft’s continued investment to stay ahead of attackers through security innovations. Windows Defender Antivirus and the rest of the Windows Defender ATP stack now integrate with other security components of Microsoft 365 to form Microsoft Threat Protection. It’s more important than ever to elevate security across the board, so this new enhancement in Windows Defender Antivirus couldn’t come at a better time.

It sure sounds interesting, so why don’t we try it out?

Requirements:

  • Windows 10, version 1703 and above.

How to activate the sandbox feature:

  • Open an elevated ’Command Prompt’
  • Type: ’setx /M MP_FORCE_USE_SANDBOX 1’ and then press enter
  • Then restart your computer and that should be it. Just make sure that you actually restart it since it won’t be activated if you just do a regular shutdown/start.

How to inactivate the sandbox feature:

  • Open the ’Control Panel’
  • Navigate to ’System’ and click on ’Advanced System Settings’
  • Click on ’Environment Variables’, navigate to ’System Variables’ and remove ’MP_FORCE_USE_SANDBOX 1’.
  • Restart the computer.

The feature is not enabled by default yet so it might not be a good idea to use it in a production environment, but it might be a good time to give Windows Defender a new chance? Have you had time to try it out yet? What do you think about it?



Teams in your Multi-user environment done right!

Microsoft Teams is on the rise, more and more businesses is seeing the potential of Teams and want a piece of the action.

Unfortunately Microsoft Teams is not ideally designed to work on a Multi-user environment like Citrix Xenapp or Microsoft Remote Desktop services. It is entirely installed in the users profile, and its quite big. A clean installation of teams is roughly 600 MB and will quickly grow, and you know what that means… You guessed it: Super long logon time, since logging on to the Multi-user environment often means the profile would be downloaded to Session Host before you are properly logged on, the users will not be happy! And on top of that, the latest recommendation in size per Teams installation is 3 GB…

There is however some rumors indicating there will be releasing a business version soon addressing this very issue! But if you are anything like me, and cant simply wait, there is a solution if you are willing to pay a small price, and you will at the same time have access to tons of other great stuff.

FSLogix Profile Container

FSLogix Profile Container is a great product that basically removes the profile size entirely, is an little agent you install on your Session Hosts and configure with an ADMX, you also need a file share with enough space for some big profiles. FSLogix is in the business of so called filter-drivers, what it does is simply put, lying to Windows. For example, when you install a 32-bit application to your 64-bit Windows System, Windows will use its own filter-driver to get it to work, its the same technology, its efficient and simple. In FSLogix case it is lying to the windows about the profiles, Windows thinks its a local profile, it does not know that in fact, the entire profile is contained in a vhd-file, mounted to the server. Because its a virtual disk that attaches to the server, there is only one SMB handle. It will therefor not be a huge load on the network, which you often sees when you for example roam your profiles.

Install Teams

When you have FSLogix Profile Container in place you can now install teams on your environment.  In early October Microsoft released a new version of Teams with some new features when deploying Teams to all the users in an organization, we are going to use parts of that to install Teams on to our environment!

 

  1. Download the latest version of Teams MSI-file (x64) file here!
  2. If you like to disable Auto-start of Teams use the following install string (otherwise just install without the option):
    This will put an Install file under ”C:\Program Files”, and when a user logon it will automatically install Teams to this user.
  3. You do not need to update the MSI to the latest version, Teams will automatically download and install pending updates on the next logon of the user.

There you go, now your users can benefit from the full experience of Teams in your Multi-user environment, with one exception: if you are using Citrix, you have ”Skype for Business Optimization Pack” to utilize local client resources for best quality of Skype meetings and calls. There is no support for Teams as of for now. It will soon be available though. With that said, I wouldn’t uninstall Skype for business just yet.

Other Great stuff

As mentioned above, there is a lot of benefits using FSLogix Profile Container. For a great period of time, Citrix User Profile Manager has been the best way to reduce the size of the profiles while still have the most important settings saved in your profile. But this is still just a trade-off, you trade off your caches and settings that impact your profile logon, but at the same time still trying to get the best experience for the user, this will sometimes collide and you have to choose between longer logon time or full functionality of a certain application.

With FSLogix Profile Container you no longer need to worry about large profiles, you don´t need to trade off! There are a lot of applications that saves a ton of settings and files in your profile that you now can install without impacting the user experience, this opens up a great deal of opportunities. You can for example install OneNote with it´s (potentially)  gigantic cache, CAD applications with thousands of files in the user profile and so much more.

 

If you find this interesting and would like a trial of FSLogix Profile Container to see if this fits your organizations needs, please contact us. It is easily installed and does not require additional servers or infrastructure!

 



Migrate home directory to a new location

Recently, I have been involved in some larger XenApp projects where one of the objects have been to migrate home directories when users change environment. People tend to lock themselves to the idea that IT must help migrate home directory when the user is given access to the new environment. A better way to approach this is to publish a script to the start menu and inform users to run the script when first logging in to the new environment. In one of the projects it were a bit more complicated because the home directory structure was different from the structure in the new XenApp environment (see below).

 

Old structure

  • Links
  • Favorites
  • Downloads
  • Documents
    • My Music
    • My Videos
    • My Pictures
  • Desktop
  • Contacts
  • AppData

 

New structure

  • Downloads
  • Documents
  • Pictures
  • Music
  • Videos
  • Desktop

 

To solve this we had to create a PowerShell script with a bit more logic. Basically what we do is the following;

  1. Copy everything from old home directory to the new home directory, except the excluded folders and files
  2. Copy ”\Documents\Music” to ”\Music”
  3. Copy ”\Documents\Videos” to ”\Videos”
  4. Copy ”\Documents\Pictures” to ”\Pictures”
  5. Copy ”\Favorites” to ”\%USERPROFILE%\Favorites” (It’s bad practice to redirect Favorites. Read more here)
  6. Copy necessary items from ”\AppData\~” to ”\%USERPROFILE%\AppData\~” (We use Citrix Profile Management instead of redirecting AppData to the home share. Redirecting AppData to the home share is also bad practice. Same reason as why it’s bad practice to redirect Favorites.)

 

 

The end result looks like this;.

User runs ”Copy Home Directory” from Start Menu as instructed

Copy Home Directory

A box pops up

Popup - Copy the entire home directory from the old environment

Another box pops up when the script is finished copying the old home directory

Popup - Finished



Update Workspace Environment Management from 4.5 to 1808

The other day I tried to update Workspace Environment Management from 4.5 to 1808. I followed the guidelines provided from Citrix here. Everything went fine with the update of ”Infrastructure Services”, ”Database” and ”Administration Console”, but when I tried to connect to the ”Infrastructure Services” with the ”Administration Console” I faced the error ”Specified Infrastructure Server seems to be offline or have a wrong database configuration. Please check configuration and try again.”.

User-added image

I saw that the connection started to initialize to the database and everything went fine until WEM tried to read ”StorefrontSettings”, then the error came up. I started digging by enabling ”debug mode” in ”WEM Infrastructure Service Configuration”. This saves a log to ”C:\Program Files (x86)\Norskale\Norskale Infrastructure Services\Citrix WEM Infrastructure Service Debug.log” with information and errors connecting to the ”Infrastructure Services”. Unfortunately I did not save the exact error message but it was something like ”Error reading dbo.VUEMStorefrontSettings”.

WEM Debug

I remembered that Citrix added the functionality to point to a StoreFront store in version 4.6.

WEM Storefront

To resolve the issue I restored the database and server to 4.5 and upgraded all the components and database to 4.6, then 4.7 and then finally 1808. After this everything worked as expected.

Seems to me that Citrix forgot to add to create ”dbo.VUEMStorefrontSettings” if not previously existing in 1808…



Create Azure Policy’s based on Resource Graph querys

If you have used Resource graph to query resources you might realized it comes very handy when creating Azure Policy’s, for example you might check the SKU of virtual machines before you create the policy to audit specific sizes of virtual machines or even prevent creation of them. (If you haven’t yet used Azure Resource Graph you can check my previous post out – https://tech.xenit.se/azure-resource-graph/)

Let’s take it further and actually create a Policy based on our Resource Graph query.

In my example below i query all storage accounts that allows connection from all Virtual Networks and the where environment is set to Prod.

Iam running all commands in Cloud Shell and CLI, but you could just aswell use Powershell.

CLI

The query is looking for below setting, it can be found under Firewalls and virtual networks under your storage accounts.

Creating the policy

To create the Policy, I am using the tool GraphToPolicy. The tool and instructions can be found here http://aka.ms/graph2policy

Follow the instructions for the tool and when you have the tool imported to your cloud shell environment you are ready to go.

Iam using the same query as before and creates a Policy to Audit all storage accounts that allows connections from all Virtual Networks and have the environment tag set to Prod.

CLI

Output:

CLI

Same policy as above but query in variable

After creation the policy is ready for assignment. I assigned it to my test subscription and as you can see in my example it shows that one of my storage accounts are non-compliant.

Summary

Resource Graph is a handy tool and as you might have understood its very useful when looking for specific properties or anomalies in your resources. Together with the GraphToPolicy it’s easy to create Azure Policys based on your Resource Graph Querys.

Credit for the tool goes to robinchapas https://github.com/robinchapas/ConvertToPolicy

If you have any questions you can reach me at tobias.vuorenmaa@xenit.se