Citrix – Xenit Technical

Category: Citrix

Black screen at session logon with VDA newer than 7.15 CU1

The black screen of death

Introduction

The logon process for users accessing a XenApp/Virtual App-environments is not completely simple to explain or understand in its entirety. There are several processes and services that need to work together, to let a user log on and begin to work in a virtual session. An issue that is not especially uncommon with regards to the logon process is what I would like to call the Black screen of death, BSOD. This should not be confused by the other BSOD! 🙂 When an environment has black screen issues I know that the troubleshooting and eventually finding a solution could most likely be long and challenging.

There have been several discussions regarding black screens at logon lately, especially when looking at Virtual Apps (i.e. XenApp) and published desktops. There are some obvious, and quite straight forward reasons why users get a black screen at logon. I’m not going to get into those in this blog post, apart from mentioning two really good articles from Citrix on the subject; XenApp/XenDesktop : Black Screen Is Displayed While Launching A Published Applications From Windows Server 2016 VDA [1] and XA/XD – Black or Blue Screen Connecting to Published Desktop [2].

I would also like to shed some light on a second ”Black Screen-issue” also currently discussed, the Windows-service AppReadiness and black screen at logon. Funnily enough, it seems like that issue is also introduced with VDAs newer than 7.15 CU1. If there’s an interest in diving into that issue too, I’m happy to do so in another blog post. My explanation of that issue can be found on the Citrix Discussion forum [3].

Last but not least, the latest of all ”Black Screen-issues” I have encountered, and the topic of today’s blog post.

Scenario

Users log on to a published desktop where the VDA is newer than 7.15 CU1, in my case i tried them all, 7.16, 7.17, and the newly released version 7.18. The session went black at logon and explorer.exe did not start. Even after waiting for more than 30 minutes. It did not matter if it was new profile or existing, in this case Citrix User Profile Mgmt, nor did it matter if the VDA was newly installed or updated from 7.15 CU1. Sending CTRL+ALT+DEL did not do a thing.

Everything worked fine on VDA 7.15 CU1 and previous versions, the only change I did to the MCS image when this occured was updating the VDA.

BSOD when initiating a new user session

Troubleshooting

I did some initial trial and error without any luck, so I decided to use my favorite troubleshooting tool, Process Monitor (aka Procmon). Within a couple of minutes I noticed that there was a process stuck in some kind of never-ending loop when a user tried to log on to the VDA. The process stuck was the ”Citrix Profile management message utility”upmEvent.exe [4].

What I also could see was that the process upmEvent.exe was the last process during the logon before the login process got stuck, and the user got the BSOD. I could not at the time identify exactly why, other than I knew which process broke the attempted login. It didn’t matter if it was a new or existing profile.

After having identified the culprit process I forcefully terminated it, and boom, the login process progressed as we are used to. Explorer.exe and all the other processes eventually started like nothing was wrong. From a user perspective, everything began to work and the desktop was shown as soon as the process upmEvent.exe was terminated.

From experience I knew that this was not the first time that specific process have have had different kind of issues. If you do a quick Google search on “upmEvent.exe” you will see that there have been some interesting issues with it over the past. The last change I know of, were when customers needed help because Citrix made a change in how it should be configured to upload data to Citrix Director. In short that change was needed because we hade to change from using UpmUserMsg.exe to upmEvent.exe. I also knew that the startup of the process had been changed previously, from the Run-key to the Userinit-key. From this I had reason to believe that this scenario might not be very different from last time [5] [6].

To summarize

I knew that upmEvent.exe by default has moved from the legacy Run-key to Userinit starting the process in user context. I also knew that the way the process needs to be configured has historically changed depending on what VDA-version is used. What I finally knew was that the configuration of the process is usually controlled in one way or another, for example with a scheduled task, GPP, GPO, registry, or something completely else.

I did a quick check to verify that the Key changed between my two VDA-versions.

Citrix VDA 7.15 CU1 is not using the Userinit registry key

Citrix VDA 7.18 is using the Userinit registry key

Indeed, there’s a difference! Closer to the solution, great!

In this specific environment I found out that the user-context startup of the upmEvent.exe-process was made with a GPO. When looking at the configuration I could see that it was configured in the old way of using upmEvent.exe. Not the new way of doing it!

The GPO configuration

Solution

When the VDA was updated to a newer version than 7.15 CU1 the GPO was reconfigured at the same time. In this case we removed the logon script and let the VDA configure the Userinit registry value. When the MCS machine was rolled out everything worked as it should, even though the VDA was updated!

I didn’t do more digging than needed, as I could see that everything started to work after the reconfiguration. It seems like newer versions of the VDA, and the move to Userinit, collide with the GPO configuration. Because of the collide the users gets a black screen at logon. A deadlock occurs when the script and Userinit is configured to run the process at the same time.

Hope this helps someone out there!

References

[1] https://support.citrix.com/article/CTX135782

[2] https://support.citrix.com/article/CTX235681

[3] https://discussions.citrix.com/topic/394538-continued-problems-with-black-screen-at-session-start-with-windows-10/?do=findComment&comment=2006811

[4] C:\Program Files\Citrix\Virtual Desktop Agent\upmEvent.exe

[5] https://www.jgspiers.com/reduce-citrix-director-interactive-session-time/

[6] https://tech.xenit.se/oregelbunden-loggning-av-inloggningar-citrix-director/



Double-hop configured with Citrix Receiver inside a published desktop

 

We started a new project with one of our clients creating a new MCS master with Windows Server 2016. One of their most critical business applications do not support Windows Server 2016 with their current version of the application. In the best of world it woulds, we would just update the application, but sometimes this is just not possible or an option. One of our ideas to solve this was to create a second master with Windows Server 2008R2 and publish the application in the Windows Server 2016 start menu with Citrix Receiver. I will guide you below how we managed to get this to work very smoothly.

 

The first thing you will want to do is to install Citrix Receiver on to the Windows Server 2016 Master. If you installed Citrix Receiver with the VDA-agent, you may skip this step. If not, you must install Citrix Receiver using below parameters:

 

After this you will want to create a new GPO, apply it to all your Windows Server 2016 Targets and configure the following settings:

When the user logs in we want Citrix Receiver to start immediately and connect to the StoreFront. Since we are using redirected start menus for all users we published the following icon in the ”startup” folder.

Starting Citrix Receiver with the tag ”ipoll” will contact the server to refresh application details, but if no authentication context is available, prompt the user for credentials. You may read more about the Citrix Receiver tags here.

 

The next thing you will want to do is to log in with a test user. Citrix Receiver should now start for the user in the background and connect to the StoreFront. Log in as an administrator to the session host and browse to ”HKEY_USERS\{SID_FOR_TEST_USER}\Software\Microsoft\Windows\CurrentVersion\Uninstall”. You may now see all published applications as keys.

You’ll want to focus on the registry setting ”LaunchString”. Copy the value in ”LaunchString”. It should look something like below.

 

This string is unique for every application. This string is not unique for every user. We will want to use this string with Citrix Receiver.

 

Optimal would of course be to publish a shortcut in the redirected start menu, but since the string is to long the shortcut is capped with max characters. We must therefore create a script like below.

 

Browse to the redirected start menu and create a shortcut with target like below.

When the user logs in to the session and launches the application it should start from the Windows Server 2008R2 machines like below.

 

Hope this works as well for you as it does for me. Give me a comment below if you have any problems.



Choosing ”HTML5 Receiver” vs ”Native Receiver” dynamically through Netscaler Rewrite Policies

After a user has authenticated on a NSGW vServer, the user will either be prompted to select which Receiver Type (HTML5 vs Native) he/she wants to use, or a choice will be made automatically depending on how well the user’s web browser manages to detect a local Citrix Receiver install. See below picture for an example of the prompt I’m referring to.

You can however get rid of below prompt, and at the same also have a mechanism that selects which Receiver Type that should be for a particular user or scenario. This is achieved through Netscaler Rewrite policies.

How does it work?

In a normal scenario, after the Receiver Type has been selected (either automatically or by user), then the cookie ‘CtxsClientDetectionDone=true’ will be created in the user’s web browser. If Native Receiver has been chosen, then the cookie ‘CtxsUserPreferredClient=Native’ will also be created. By using Rewrite Policies we can create these two cookies by ourselves for the user, and therefore suppress the prompt for the user and automatically choose which Receiver Type to use.

If HTML5 should be used, then we only want to apply the Rerwite policy ”RWP-RES-DISABLE-RECEIVER-CHECK” to suppress the prompt. When Netscaler sees that the cookie ‘CtxsUserPreferredClient’ Cookie is missing, it will default to HTML5 Receiver (this is dependent on your Storefront configuration – see further down). If we want to force the Native Receiver, we also apply the rewrite policy RWP-RES-SET-NATIVE-RECEIVER” to create the cookie ‘CtxsUserPreferredClient=Native’.

In below scenario, I have defined an Expression for my Rewrite Policy ‘RWP-RES-SET-NATIVE-RECEIVER’ to only apply if the user is connecting from IP subnet 10.240.5.0/24. You can also use ”HTTP.REQ.HEADER(\”User-Agent\”).CONTAINS(\”Chrome\”)” to only apply it to Chrome Users, or use most other type of Expressions. I tried to use HTTP.REQ.USER.ATTRIBUTE(1) and HTTP.REQ.USER.IS_MEMBEROF(\”GroupName\”) expressions, but it seems that these expressions will always evaluate to false for a Rewrite Policy bound to a VPN vServer, so they don’t work, which is a shame.

 

 

For the choice between Native Receiver and HTML5 Receiver to work, you will need to configure your Storefront so that both HTML5 and Native Receivers are possible, like below picture. If you configure ”Always use Receiver for HTML5” instead of ”Use Receiver for HTML5 if local Receiver is unavailable”, then it doesn’t matter that the cookie ‘CtxsUserPreferredClient=Native’ exists. Similarly, if you configure ”Install locally” instead of ”Use Receiver for HTML5 if local Receiver is unavailable”, then Native Receiver will always be used.

If you want want the dynamic choice between HTML5 and Native Receiver, then don’t use ”Use Receiver for HTML5 if local Receiver is unavailable” and only create the ‘CtxsClientDetectionDone’ cookie to suppress the unnecessary prompt for the user.

Feel free to email me at rasmus.kindberg@xenit.se if you have any suggestions or questions related to this blog post.



Automate tasks with use of XenServer Powershell Module

Working with backups of your virtual machines is obviously essential. Working with exports in XenServer can some times be time consuming, particularly with bigger virtual disks attached to your virtual machine. In this scenario I will show you an alternative to manually export via XenCenter, by doing it with Powershell to an remote server using XenServer Powershell module.



Flickering Desktop Icons and re-directed folders

This blog post will only cover a scenario with Microsoft Windows Server 2016 Remote Desktop Services (RDS) and re-directed folders where flickering icons appear. Other solutions may apply to different scenarios.
Since the release of Windows 10 / Server 2016 and their different releases 1607, 1703, 1709 and 1803 there has been several issues regarding flickering icons on the Start-menu, in File Explorer and taskbar.

SCENARIO

During the deployment of Citrix Virtual Apps and Desktops 7.15 on Windows Server 2016 with published Desktops and re-directed Desktop folder, users could experience that the desktop icons kept flickering continuously. The more shortcuts, folders or files on the Desktop the more prevalent the issue was. Constantly blinking icons on the desktop looked like refreshing the desktop with F5 or Ctrl+R and would also flash when browsing network shares.

My first thought was to activate ”Always show icons, never thumbnails” in Folder Options since there seemed to be a constant query to network shares where the re-directed Desktop folder resided.

File Explorer - Options

File Explorer – Options

File Explorer - Always show icons

File Explorer – Always show icons

INVESTIGATION

The moment I clicked on View in Folder Options the desktop icons ceased flashing in my session. Dwelling deeper with Procmon investigating what actually happens when opening View tab in Folder Options I found out that explorer.exe queries a registry key in the users HKEY_CURRENT_USER registry. If the registry entry does not exist it will be created.

  • HKEY_CURRENT_USER\SOFTWARE\Classes\CLSID\{031E4825-7B94-4dc3-B131-E946B44C8DD5}
Explorer query and creation of registry key

ProcMon – Explorer.exe query and creation of registry key

SOLUTION

With the knowledge that the registry key was missing and creating they key would stop the icons from flashing for users on Windows Server 2016 RDS, the appropriate solution was to use Group Policy Preferences (GPP) that created the registry key for users during logon (run in logged-on users’s security context) and apply it to Windows 2016 RDS servers.
Gorup Policy Preferences - User Configuration - Registry

Gorup Policy Preferences – User Configuration – Registry

Apply to Current User

Apply to HKEY_CURRENT_USER and set Key Path

Run in logged-on users security context

Run in logged-on users security context

Step 1: Create a USER GPP that will be applied to affected targets

Step 2: Create a Registry Item

Step 3: Add registry key

  • Hive: HKEY_CURRENT_USER
  • Key Path: SOFTWARE\Classes\CLSID\{031E4825-7B94-4dc3-B131-E946B44C8DD5}
  • Tab Common: [v] Run in logged-on user’s security context (user policy option)

If you have any questions regarding above solution, or ideas on how to handle above in a better way, please contact me at viktor.glinski@xenit.se or post a comment below.



Netscaler: ADFS protected by AAA – How to handle SAML POST requests

A limitation with Netscaler AAA is that it cannot handle FormData sent in a POST request to a Netscaler LB vServer that is protected by a AAA vServer. What happens is that the Form data in the POST will not be included when the user is redirected back to the LB vServer after AAA authentication. This becomes relevant in scenarios where you have a SAML ServiceProvider (SP) that is configured to do a login POST to an SAML IdentityProvider (IDP) and that IDP is protected by Netscaler AAA.

Below is the process flow:
1. User browses to the SAML SP address https://app1.somedomain.com/saml/login, which in this scenario is the URL that initiates the SAML logon process
2. The SP gives the user a SAML request and the user’s browser performs a POST against the IDP URL https://adfs.mycompany.com/adfs/ls/ with this SAML Request as the Form data.
3. The address https://adfs.mycompany.com points to a Netscaler LB vServer which is protected by AAA, so when Netscaler sees the incoming GET request above it will redirect the user to https://aaa.mycompany.com for AAA authentication (we assume the user has not authenticated against this AAA vServer this web session).
4. User performs AAA authentication, and is afterwards redirected back to the original URL https://adfs.mycompany.com/adfs/ls. HOWEVER, the SAML Request Form data is now missing.
5. User will land on https://adfs.mycompany.com/adfs/ls and receive an error message, because the ADFS server doesn’t know how to handle a request that doesn’t have any SAML form data.

 

Important notes:

  • Form Data passed along with a POST to a LB vServer, such as ADFS, that is protected by AAA will be ‘dropped’ when the user is redirected back to the LB vServer after successful AAA authentication. This only applies if the user has not authenticated against the AAA in the current web session (ie the user does not have a NSC_TMAS cookie). We will make use of this later on.
  • Query values included in a POST are not ‘dropped’, so this flaw is limited to Form data only.

 

Solution/work-around:
The easiest solution is to simply ask the SAML SP to use Redirect instead of POST for the SAML authentication process, but if that is not an option (the SAML SP’s backend code or configuration doesn’t support SAML Redirect) then below is a work-around I’ve been using. Basically what you do is that you store the original SP URL, https://app1.somedomain.com/saml/login, in a cookie in the user’s browser and in step 5 the user will be redirected back to this URL again.

Below is the process flow with a work-around implemented for POST:
1. User browses to https://app1.somedomain.com/saml/login, which in this scenario is the url that initiates the SAML logon process
2. The SP gives the user a SAML request and the user’s browser performs a POST against the IDP URL https://adfs.mycompany.com/adfs/ls/ with this SAML Request as the Form data.
3. The address adfs.mycompany.com points to a Netscaler LB vServer which is protected by AAA, so when Netscaler sees the incoming GET request above it will redirect the user to https://aaa.mycompany.com for AAA authentication.
3b. NEW: When the user is redirected to https://aaa.mycompany.com now, a Rewrite policy will trigger that will create a cookie ”ADFSPostCookieURL” for the user, and this cookie will contain the value ”https://app1.somedomain.com/saml/login”.
4. User performs AAA authentication, and is afterwards redirected back to the original URL https://adfs.mycompany.com/adfs/ls.
5. NEW: We have a Responder policy on our ADFS LB vServer that checks if the path is ”/adfs/ls” and if the cookie ”ADFSPostCookieURL” exists, and if both are true then we read the value in cookie ”ADFSPostCookieURL” and Redirects the user to that URL.
6. User is redirected back to https://app1.somedomain.com/saml/login, which will restart the SAML logon process
7. The SP gives the user a new SAML request and the user’s browser again performs a POST against the IDP URL https://adfs.mycompany.com/adfs/ls/ with this SAML Request as the Form data.
8. A key difference now is that the user already has done AAA authentication this web session and thus has a valid AAA cookie, and won’t be redirected to https://aaa.mycompany.com for authentication. The POST against https://adfs.mycompany.com/adfs/ls/ will therefore happen successfully and the ADFS backend server will see the SAML Form data since that has not been dropped by AAA redirect.
9. Assuming the SAML Request ticket is valid, the ADFS server will give the user a SAML Response ticket and redirect the user to https://app1.somedomain.com/myApp and the user is now logged on to this 3rd party site successfully.

 

Takeaways:

  • Our workaround revolves around storing the original url (https://app1.somedomain.com/saml/login) in some way so we can access it later, and requesting a SAML Request ticket twice from our SAML SP because in the second round we will not be bothered by AAA authentication.
  • Above solution is a bit hacky and involves requesting double SAML tickets from the SP, and there are a lot of Redirects involved, but it works well from an end-user perspective and it enables us to support SAML Post in conjunction with AAA.

 

If you have any questions regarding above solution, or ideas on how to handle above scenario in a better way, please contact me at rasmus.kindberg@xenit.se.

 

 

Below is the Netscaler configuration:



Citrix Synergy 2018 highlights

Synergy is Citrix main event and Xenit are of course on site to try out new features and solutions. The conference includes both a business-oriented and technical track for customers and partners. Måns Hurtigh, Simon Gottschlag (CTP), Adam Clark and Linus Lindström from Xenit are on site to test the latest Citrix products and features. Citrix CEO, David Henshall opened Synergy by talking about their key strategic priorities for 2018. Mainly there are three key areas that Citrix is talking about:

 

  • Unify Citrix portfolio to simplify user and IT experiences
  • Accelerate to the cloud to help companies work the way they want
  • Expand to new areas to meet the demands of the future

 

David Henshall continued by speaking about Citrix’s goals and strategies for 2018. Citrix announced several items this year that focuses on expanding Citrix Cloud with more features and unifying their already market leading products. A lot of new product and features were introduced:

 

  • Citrix Workspace App
  • Workspace Self-service with ServiceNow
  • Autoscale for Google Cloud
  • Citrix endpoint management capabilities
  • Citrix Cloud for Azure Government
  • Citrix Cloud App Control
  • SD-WAN Server for managed service providers (MSP)
  • Citrix Intelligent Traffic Management

 

Citrix Synergy 2018, David Henshall

On Summit 2017 Citrix talked about unifying there whole product series. It sure seems that Citrix is continuing down this path. Citrix slogan this year is ”The future is now”. Be sure to check out all new cool features in from this years Citrix Synergy.

 



Using NetScaler as OpenID Connect SP with ADFS as IDP

How do you configure Citrix NetScaler OpenID Connect Service Provider with Microsoft ADFS as OpenID Connect Identity Provider? I’ve tried making it easy to understand and how you do it using CLI (NetScaler CLI and powershell).

Read this post for doing this with SAML.



Using NetScaler as SAML SP with ADFS as IDP

How do you configure Citrix NetScaler SAML Service Provider with Microsoft ADFS as SAML Identity Provider? I’ve tried making it easy to understand and how you do it using CLI (NetScaler CLI and powershell).

Before we begin, let us look at what we need to establish the federation:

  • NetScaler (with at least Enterprise license)
  • Active Directory domain and ADFS (read this post if you want to load balance and use NetScaler as ADFS Proxy)
  • Website (lb vserver) we want to protect with AAA (will be referred to as the service provider)
  • AAA vserver to bind SAML Service Provider policy

In my case, the following FQDNs are used:

  • LB vserver: webapp-test.domain.com / LB-WEBAPP-TEST
  • AAA vserver: sp.domain.com / AAA-SP-DOMAIN.COM (note: it will actually not be access by the web browser)
  • ADFS: adfs.domain.com

When installing ADFS two self signed certificates are issued for Token-signing and Token-decryption. When it comes to the NetScaler, we could always use whatever certificate for the signing and decryption – but I recommend using a certificate that isn’t used for web site communication. That’s why I create a self signed certificate that I use: (note: I do this on my computer, modify the variables to match your environment – and even though this certificate and key is self signed – keep them secure)

The certificate (not the key) needs to be copied to the ADFS server for when we create the Relying Party Trust, and we also need to copy the ADFS Token-signing certificate to the NetScaler (below called adfs.domain.com-signing).

Copy the newly created certificate and key to the NetScaler, as well as the ADFS Token-signing certificate:

Now we need to create the SAML Service Provider action and profile, as well as bind it to the AAA vserver:

(Note: As I stated before, this policy is bound to the AAA vserver but the expression is matching the hostname of the LB vserver – since the web browser actually never is redirected to the AAA vserver in this scenario)

As a last step, create (if it isn’t already) an authentication profile and bind it to the LB vserver:

Now configure ADFS (modify the variables to match your need):

 



Outlook Search index med FSLogix

Något som upptäckts snabbt efter uppsättningen av sin ”FSlogix Office 365 Containers”-lösning i en fleranvändarmiljö är att sök-indexeringen för Outlook i vissa miljöer görs om vid varje ny inloggning, det gäller miljöer där man har flera Session Hostar användarna kan logga in på.

Sök-funktionen i Outlook använder sig av ”Windows Search” vilket är en databas över indexeringarna på hela Operativsystemet, det är alltså inget som lagras för varje enskild användare. Det innebär t.ex.  att en Citrix miljö med flera servrar kommer en användares Outlook indexera om hela Outlook vid varje ny server man loggar in på. Detta medför en långsam sökning (tills indexeringen är klar) och en onödigt belastning på CPU som i sin tur kan påverka hela miljön negativt. Det kan bli ännu värre i de fall man använder Citrix Provisioning Services (PVS) då den uppdaterade indexeringen försvinner vid varje omstart av servern.

FSLogix to the rescue

För att komma runt detta problem finns en funktion i FSLogix som tar med din Outlook indexering i VHD-filen, på så vis har du alltid din uppdaterade indexeringsdata med dig på vilken server du än hamnar på. Du behöver ändra på två stycken registervärden för att aktivera detta, jag själv föredrar att skapa/editera en GPO för detta.

Följande två registervärden ska justeras:

HKLM\Software\FSLogix\Apps

Type:                      DWORD

Value Name:          RoamSearch

Value Data:            2

 

HKLM\Software\Policies\FSLogix\ODFC

Type:                      DWORD

Value Name:          RoamSearch

Value Data:            2

 

Hör gärna av er om ni skulle vara intresserade av eller vill veta mer om produkter från FSLogix, se gärna våra tidigare blogginlägg om FSLogix nedan:

FSLogix Profile Containers – Enkel och snabb Profilhantering

Office365 med FSLogix i en fleranvändarmiljö

OneDrive with simulated Single Sign-On