Posted in : Azure, NetScaler By Simon Gottschlag Translate with Google ⟶

6 years ago

I wrote a blog post for NetScaler active/passive HA in Azure with multiple NICs two days ago, and I’ve been trying to figure out if this was the best way to do it. In the other post, I was using IPPattern in NetScaler to set the vServers to a /31 – which does work but that’s just because of how the underlying Azure infrastrucuture works (where machines outside of the VM – for example Azure LB – can only access the IP that has been assigned to the VM).
There is another way of doing this, which doesn’t require you to use a /31. The key is in configuring DSR (Direct Server Return) in Azure LB (also known as Floating IP). This will make it possible to use the same VIP on the NetScalers as the Frontend IP of the Azure LB – which saves IP-addresses and is easier to configure. This is the way Citrix has documented it and this is how their HA template does it.

What are the requirements before following my instructions below?

  • Create a vnet and three subnets, as well as a resource group and availability set
  • Configure INC on the NetScalers / each get a unique SNIP on each subnet
  • Place the Azure LBs on the same subnets as they are load balancing (the same subnets we have the VIPs)  and place the NetScalers on different subnets than resources using the Azure LBs. This may not be a requirement, but is how I’ve done it.

In my case, i have the following subnets:

  • management = 10.99.0.0/24
  • inside = 10.99.1.0/24
  • outside = 10.99.2.0/24
  • Azure LB Frontend IP & VIP on inside: 10.99.1.204
  • Azure LB Frontend IP & VIP on outside: 10.99.2.204

First of, create NetScaler #1:

$AVSetName = "avset-weu-prod-netscaler"
$RGName = "rg-weu-prod"
$VMName = "vm-weu-ns1"
$VMSize = "Standard_DS3_v2"
$OSDiskName = "osdisk-$($VMName)"
$VHDName = "$($OSDiskName).vhd"
$Location = "West Europe"
$VNetName = "vnet-weu-prod"
$NICName1 = "nic-$($VMName)-mgmt"
$NIC1PrivateIP1 = "10.99.0.200"
$NIC1PrivateIP2 = "10.99.0.202"
$Subnet1Name = "management"
$NICName2 = "nic-$($VMName)-inside"
$NIC2PrivateIP1 = "10.99.1.200"
$NIC2PrivateIP2 = "10.99.1.202"
$Subnet2Name = "inside"
$NICName3 = "nic-$($VMName)-outside"
$NIC3PrivateIP1 = "10.99.2.200"
$NIC3PrivateIP2 = "10.99.2.202"
$Subnet3Name = "outside"
$PublisherName = "citrix"
$OfferName = "netscalervpx-120"
$SKUName = "netscalerbyol"
$VMCredentials = Get-Credential
$AvailabilitySet = Get-AzureRmAvailabilitySet -ResourceGroupName $RGName -Name $AVSetName
$VM = New-AzureRmVMConfig -VMName $VMName -VMSize $VMSize -AvailabilitySetID $AvailabilitySet.Id
Set-AzureRmVMPlan -VM $VM -Publisher $PublisherName -Product $OfferName -Name $SKUName
Get-AzureRmMarketplaceTerms -Publisher $PublisherName -Product $OfferName -Name $SKUName | Set-AzureRmMarketplaceTerms -Accept
Set-AzureRmVMOSDisk -VM $VM -Name $OSDiskName -VhdUri $VHDName -Caching ReadWrite -CreateOption FromImage
Set-AzureRmVMOperatingSystem -VM $VM -Linux -ComputerName $VMName -Credential $VMCredentials
Set-AzureRmVMSourceImage -VM $VM -PublisherName $PublisherName -Offer $OfferName -Skus $SKUName -Version "latest"
Set-AzureRmVMOSDisk -VM $VM -Name $OSDiskName -CreateOption FromImage
$VNet = Get-AzureRMVirtualNetwork -Name $VNetName -ResourceGroupName $RGName
$Subnet1 = Get-AzureRmVirtualNetworkSubnetConfig -Name $Subnet1Name -VirtualNetwork $VNet
$NIC1IPConfig1 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName1)-IPConfig1" -Subnet $Subnet1 -PrivateIpAddress $NIC1PrivateIP1 -Primary
$NIC1IPConfig2 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName1)-IPConfig2" -Subnet $Subnet1 -PrivateIpAddress $NIC1PrivateIP2
$NIC1 = New-AzureRmNetworkInterface -Name $NICName1 -ResourceGroupName $RGName -Location $Location -IpConfiguration $NIC1IPConfig1,$NIC1IPConfig2
$Subnet2 = Get-AzureRmVirtualNetworkSubnetConfig -Name $Subnet2Name -VirtualNetwork $VNet
$NIC2IPConfig1 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName2)-IPConfig1" -Subnet $Subnet2 -PrivateIpAddress $NIC2PrivateIP1 -Primary
$NIC2IPConfig2 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName2)-IPConfig2" -Subnet $Subnet2 -PrivateIpAddress $NIC2PrivateIP2
$NIC2 = New-AzureRmNetworkInterface -Name $NICName2 -ResourceGroupName $RGName -Location $Location -IpConfiguration $NIC2IPConfig1,$NIC2IPConfig2
$Subnet3 = Get-AzureRmVirtualNetworkSubnetConfig -Name $Subnet3Name -VirtualNetwork $VNet
$NIC3IPConfig1 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName3)-IPConfig1" -Subnet $Subnet3 -PrivateIpAddress $NIC3PrivateIP1 -Primary
$NIC3IPConfig2 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName3)-IPConfig2" -Subnet $Subnet3 -PrivateIpAddress $NIC3PrivateIP2
$NIC3 = New-AzureRmNetworkInterface -Name $NICName3 -ResourceGroupName $RGName -Location $Location -IpConfiguration $NIC3IPConfig1,$NIC3IPConfig2
Add-AzureRmVMNetworkInterface -VM $VM -Id $NIC1.Id -Primary
Add-AzureRmVMNetworkInterface -VM $VM -Id $NIC2.Id
Add-AzureRmVMNetworkInterface -VM $VM -Id $NIC3.Id
New-AzureRmVM -VM $VM -ResourceGroupName $RGName -Location $Location

Create NetScaler #2:

$AVSetName = "avset-weu-prod-netscaler"
$RGName = "rg-weu-prod"
$VMName = "vm-weu-ns2"
$VMSize = "Standard_DS3_v2"
$OSDiskName = "osdisk-$($VMName)"
$VHDName = "$($OSDiskName).vhd"
$Location = "West Europe"
$VNetName = "vnet-weu-prod"
$NICName1 = "nic-$($VMName)-mgmt"
$NIC1PrivateIP1 = "10.99.0.201"
$NIC1PrivateIP2 = "10.99.0.203"
$Subnet1Name = "management"
$NICName2 = "nic-$($VMName)-inside"
$NIC2PrivateIP1 = "10.99.1.201"
$Subnet2Name = "inside"
$NICName3 = "nic-$($VMName)-outside"
$NIC3PrivateIP1 = "10.99.2.201"
$Subnet3Name = "outside"
$PublisherName = "citrix"
$OfferName = "netscalervpx-120"
$SKUName = "netscalerbyol"
$VMCredentials = Get-Credential
$AvailabilitySet = Get-AzureRmAvailabilitySet -ResourceGroupName $RGName -Name $AVSetName
$VM = New-AzureRmVMConfig -VMName $VMName -VMSize $VMSize -AvailabilitySetID $AvailabilitySet.Id
Set-AzureRmVMPlan -VM $VM -Publisher $PublisherName -Product $OfferName -Name $SKUName
Get-AzureRmMarketplaceTerms -Publisher $PublisherName -Product $OfferName -Name $SKUName | Set-AzureRmMarketplaceTerms -Accept
Set-AzureRmVMOSDisk -VM $VM -Name $OSDiskName -VhdUri $VHDName -Caching ReadWrite -CreateOption FromImage
Set-AzureRmVMOperatingSystem -VM $VM -Linux -ComputerName $VMName -Credential $VMCredentials
Set-AzureRmVMSourceImage -VM $VM -PublisherName $PublisherName -Offer $OfferName -Skus $SKUName -Version "latest"
Set-AzureRmVMOSDisk -VM $VM -Name $OSDiskName -CreateOption FromImage
$VNet = Get-AzureRMVirtualNetwork -Name $VNetName -ResourceGroupName $RGName
$Subnet1 = Get-AzureRmVirtualNetworkSubnetConfig -Name $Subnet1Name -VirtualNetwork $VNet
$NIC1IPConfig1 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName1)-IPConfig1" -Subnet $Subnet1 -PrivateIpAddress $NIC1PrivateIP1 -Primary
$NIC1IPConfig2 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName1)-IPConfig2" -Subnet $Subnet1 -PrivateIpAddress $NIC1PrivateIP2
$NIC1 = New-AzureRmNetworkInterface -Name $NICName1 -ResourceGroupName $RGName -Location $Location -IpConfiguration $NIC1IPConfig1,$NIC1IPConfig2
$Subnet2 = Get-AzureRmVirtualNetworkSubnetConfig -Name $Subnet2Name -VirtualNetwork $VNet
$NIC2IPConfig1 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName2)-IPConfig1" -Subnet $Subnet2 -PrivateIpAddress $NIC2PrivateIP1 -Primary
$NIC2IPConfig2 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName2)-IPConfig2" -Subnet $Subnet2 -PrivateIpAddress $NIC2PrivateIP2
$NIC2 = New-AzureRmNetworkInterface -Name $NICName2 -ResourceGroupName $RGName -Location $Location -IpConfiguration $NIC2IPConfig1
$Subnet3 = Get-AzureRmVirtualNetworkSubnetConfig -Name $Subnet3Name -VirtualNetwork $VNet
$NIC3IPConfig1 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName3)-IPConfig1" -Subnet $Subnet3 -PrivateIpAddress $NIC3PrivateIP1 -Primary
$NIC3IPConfig2 = New-AzureRmNetworkInterfaceIpConfig -Name "$($NICName3)-IPConfig2" -Subnet $Subnet3 -PrivateIpAddress $NIC3PrivateIP2
$NIC3 = New-AzureRmNetworkInterface -Name $NICName3 -ResourceGroupName $RGName -Location $Location -IpConfiguration $NIC3IPConfig1
Add-AzureRmVMNetworkInterface -VM $VM -Id $NIC1.Id -Primary
Add-AzureRmVMNetworkInterface -VM $VM -Id $NIC2.Id
Add-AzureRmVMNetworkInterface -VM $VM -Id $NIC3.Id
New-AzureRmVM -VM $VM -ResourceGroupName $RGName -Location $Location

Configure Azure LB:

$RGName = "rg-weu-prod"
$VNetName = "vnet-weu-prod"
$Location = "West Europe"
$ALBName = "alb-netscaler"
$ALBHealthProbeName = "healthprobe-alb-netscaler"
$Subnet1Name = "inside"
$Subnet1FEIP = "10.99.1.204"
$Subnet1FEName = "fe-alb-netscaler-inside"
$Subnet1BEPoolName = "be-alb-netscaler-inside"
$Subnet1LBRule1Name = "lb-alb-netscaler-inside-http"
$Subnet1LBRule2Name = "lb-alb-netscaler-inside-ssl"
$Subnet2Name = "outside"
$Subnet2FEIP = "10.99.2.204"
$Subnet2FEName = "fe-alb-netscaler-outside"
$Subnet2BEPoolName = "be-alb-netscaler-outside"
$Subnet2LBRule1Name = "lb-alb-netscaler-outside-http"
$Subnet2LBRule2Name = "lb-alb-netscaler-outside-ssl"
$NS1NICInsideName = "nic-vm-weu-ns1-inside"
$NS1NICInsideVIPConfigName = "nic-vm-weu-ns1-inside-IPConfig1"
$NS1NICOutsideName = "nic-vm-weu-ns1-outside"
$NS1NICOutsideVIPConfigName = "nic-vm-weu-ns1-outside-IPConfig1"
$NS2NICInsideName = "nic-vm-weu-ns2-inside"
$NS2NICInsideVIPConfigName = "nic-vm-weu-ns2-inside-IPConfig1"
$NS2NICOutsideName = "nic-vm-weu-ns2-outside"
$NS2NICOutsideVIPConfigName = "nic-vm-weu-ns2-outside-IPConfig1"
$VNet = Get-AzureRMVirtualNetwork -Name $VNetName -ResourceGroupName $RGName
$Subnet1 = Get-AzureRmVirtualNetworkSubnetConfig -Name $Subnet1Name -VirtualNetwork $VNet
$Subnet2 = Get-AzureRmVirtualNetworkSubnetConfig -Name $Subnet2Name -VirtualNetwork $VNet
$FEIPSubnet1 = New-AzureRmLoadBalancerFrontendIpConfig -Name $Subnet1FEName -PrivateIpAddress $Subnet1FEIP -SubnetId $Subnet1.ID
$FEIPSubnet2 = New-AzureRmLoadBalancerFrontendIpConfig -Name $Subnet2FEName -PrivateIpAddress $Subnet2FEIP -SubnetId $Subnet2.ID
$BEPoolSubnet1 = New-AzureRmLoadBalancerBackendAddressPoolConfig -Name $Subnet1BEPoolName
$BEPoolSubnet2 = New-AzureRmLoadBalancerBackendAddressPoolConfig -Name $Subnet2BEPoolName
$ALBHealthProbe = New-AzureRmLoadBalancerProbeConfig -Name $ALBHealthProbeName -Protocol Tcp -Port 9000 -IntervalInSeconds 5 -ProbeCount 2
$LBRule1Subnet1 = New-AzureRmLoadBalancerRuleConfig -Name $Subnet1LBRule1Name -FrontendIpConfiguration $FEIPSubnet1 -BackendAddressPool $BEPoolSubnet1 -Probe $ALBHealthProbe -Protocol Tcp -FrontendPort 80 -BackendPort 80 -EnableFloatingIP
$LBRule2Subnet1 = New-AzureRmLoadBalancerRuleConfig -Name $Subnet1LBRule2Name -FrontendIpConfiguration $FEIPSubnet1 -BackendAddressPool $BEPoolSubnet1 -Probe $ALBHealthProbe -Protocol Tcp -FrontendPort 443 -BackendPort 443 -EnableFloatingIP
$LBRule1Subnet2 = New-AzureRmLoadBalancerRuleConfig -Name $Subnet2LBRule1Name -FrontendIpConfiguration $FEIPSubnet2 -BackendAddressPool $BEPoolSubnet2 -Probe $ALBHealthProbe -Protocol Tcp -FrontendPort 80 -BackendPort 80 -EnableFloatingIP
$LBRule2Subnet2 = New-AzureRmLoadBalancerRuleConfig -Name $Subnet2LBRule2Name -FrontendIpConfiguration $FEIPSubnet2 -BackendAddressPool $BEPoolSubnet2 -Probe $ALBHealthProbe -Protocol Tcp -FrontendPort 443 -BackendPort 443 -EnableFloatingIP
$ALB = New-AzureRmLoadBalancer -ResourceGroupName $RGName -Name $ALBName -Location $Location -FrontendIpConfiguration $FEIPSubnet1,$FEIPSubnet2 -LoadBalancingRule $LBRule1Subnet1,$LBRule2Subnet1,$LBRule1Subnet2,$LBRule2Subnet2 -BackendAddressPool $BEPoolSubnet1,$BEPoolSubnet2 -Probe $ALBHealthProbe
$NS1NICInside = Get-AzureRmNetworkInterface -ResourceGroupName $RGName -Name $NS1NICInsideName
$NS1NICOutside = Get-AzureRmNetworkInterface -ResourceGroupName $RGName -Name $NS1NICOutsideName
$NS2NICInside = Get-AzureRmNetworkInterface -ResourceGroupName $RGName -Name $NS2NICInsideName
$NS2NICOutside = Get-AzureRmNetworkInterface -ResourceGroupName $RGName -Name $NS2NICOutsideName
($NS1NICInside.IPConfigurations | ?{$_.Name -eq $NS1NICInsideVIPConfigName}).LoadBalancerBackendAddressPools.Add($BEPoolSubnet1)
($NS1NICOutside.IPConfigurations | ?{$_.Name -eq $NS1NICOutsideVIPConfigName}).LoadBalancerBackendAddressPools.Add($BEPoolSubnet2)
($NS2NICInside.IPConfigurations | ?{$_.Name -eq $NS2NICInsideVIPConfigName}).LoadBalancerBackendAddressPools.Add($BEPoolSubnet1)
($NS2NICOutside.IPConfigurations | ?{$_.Name -eq $NS2NICOutsideVIPConfigName}).LoadBalancerBackendAddressPools.Add($BEPoolSubnet2)
Set-AzureRmNetworkInterface -NetworkInterface $NS1NICInside
Set-AzureRmNetworkInterface -NetworkInterface $NS1NICOutside
Set-AzureRmNetworkInterface -NetworkInterface $NS2NICInside
Set-AzureRmNetworkInterface -NetworkInterface $NS2NICOutside

Please note: I’m only using internal LBs here, you need to modify the configuration to create a Public IP.
Now, configure IPs and HA (with INC) and disable MBF/configure PBR (not required).

# NetScaler #1
set ns config -IPAddress 10.99.0.200 -netmask 255.255.255.0
add ns ip 10.99.1.200 255.255.255.0 -vServer DISABLED
add ns ip 10.99.2.200 255.255.255.0 -vServer DISABLED
add ns ip 10.99.0.202 255.255.255.0 -vServer DISABLED
add HA node 1 10.99.0.201 -inc ENABLED
add ns pbr PBR-MANAGEMENT ALLOW -srcIP = 10.99.0.0-10.99.0.255 -destIP "!=" 10.99.0.0-10.99.0.255 -nextHop 10.99.0.1 -priority 10
add ns pbr PBR-INSIDE ALLOW -srcIP = 10.99.1.0-10.99.1.255 -destIP "!=" 10.99.1.0-10.99.1.255 -nextHop 10.99.1.1 -priority 20
add ns pbr PBR-OUTSIDE ALLOW -srcIP = 10.99.2.0-10.99.2.255 -destIP "!=" 10.99.2.0-10.99.2.255 -nextHop 10.99.2.1 -priority 30
# NetScaler #2
set ns config -IPAddress 10.99.0.201 -netmask 255.255.255.0
add ns ip 10.99.1.201 255.255.255.0 -vServer DISABLED
add ns ip 10.99.2.201 255.255.255.0 -vServer DISABLED
add ns ip 10.99.0.203 255.255.255.0 -vServer DISABLED
add HA node 1 10.99.0.200 -inc ENABLED
add ns pbr PBR-MANAGEMENT ALLOW -srcIP = 10.99.0.0-10.99.0.255 -destIP "!=" 10.99.0.0-10.99.0.255 -nextHop 10.99.0.1 -priority 10
add ns pbr PBR-INSIDE ALLOW -srcIP = 10.99.1.0-10.99.1.255 -destIP "!=" 10.99.1.0-10.99.1.255 -nextHop 10.99.1.1 -priority 20
add ns pbr PBR-OUTSIDE ALLOW -srcIP = 10.99.2.0-10.99.2.255 -destIP "!=" 10.99.2.0-10.99.2.255 -nextHop 10.99.2.1 -priority 30

Please note: I’m not configuring HA encryption or chaning the rpcNode password. Should always be done. Only showing what I think is the bare minimum to get it working.
Configure the most basic content switches for inside and outside:

# Creating responder action/policy to show something on the CS
add responder action REA-NETSCALER_IP respondwith "\"NetScaler IP: \" + SYS.NSIP + \" | VIP: \" + CLIENT.IP.DST"
add responder policy REP-NETSCALER_IP true REA-NETSCALER_IP
# Create CS for inside (HP = healthprobe, should match Azure LB configuration)
add cs vserver CS-INSIDE_HTTP HTTP 10.99.1.204 80 -cltTimeout 180
bind cs vserver CS-INSIDE_HTTP -policyName REP-NETSCALER_IP -priority 100 -gotoPriorityExpression END -type REQUEST
# Create CS for outside (HP = healthprobe, should match Azure LB configuration)
add cs vserver CS-OUTSIDE_HTTP HTTP 10.99.2.204 80 -cltTimeout 180
bind cs vserver CS-OUTSIDE_HTTP -policyName REP-NETSCALER_IP -priority 100 -gotoPriorityExpression END -type REQUEST<br>

Now you should be able to failover between the NetScalers. From a VM on the same vnet, you should be presented with the following when NetScaler #1 is active:
 
http://10.99.1.204 = NetScaler IP: 10.99.0.200 | VIP: 10.99.1.204
And the following when NetScaler #2 is active:
http://10.99.1.204 = NetScaler IP: 10.99.0.201 | VIP: 10.99.1.204
Good luck with the configuration and feel free to drop a comment if you have any feedback or questions!
Update: If you are experiencing issues with failover where heartbeats are only seen on one interface – see the following post.

Tags :

Personlig rådgivning

Vi erbjuder personlig rådgivning med författaren för 1400 SEK per timme. Anmäl ditt intresse i här så återkommer vi så snart vi kan.

Comments

Ross says

Hello,
Thank you for the great post! I was following the Citrix guide (https://docs.citrix.com/en-us/netscaler/12/deploying-vpx/deploy-vpx-on-azure/configure-ha-pair-using-powershell.html) to configure my NetScalers in Azure with a public ALB when I came across your article and it did help but unfortunately I'm still stuck and I was wondering if you might be kind enough to point out where I went wrong. I am only using a VM size which supports 2 NICs so I figured I would attach 1 NIC to my "Server" subnet and the other NIC to the "Public/DMZ" subnet. My NSIP is also on the server subnet. So it looks like this:
NetScaler 1:
NIC0 = Public Subnet - 10.9.0.4 (SNIP)
NIC1 = Server Subnet - 10.9.1.20 (NSIP), 10.9.1.21 (SNIP)
NetScaler 2:
NIC0 = Public Subnet - 10.9.0.5 (SNIP)
NIC1 = Server Subnet - 10.9.1.22 (NSIP), 10.9.1.23 (SNIP)
I setup the ALB with a public IP, the TCP 9000 health check, the 2 NS 'Public Subnet' IPs (10.9.0.4 & 10.9.0.5) as the backend, and a rule for port 80 with floating enabled. I also setup the responder/content switch on the NS as per your guide for testing. When I hit the public IP on the ALB unfortunately it just times out.
I created a route on my 'Server subnet' in Azure to route to my ALB public IP via the primary NS (10.9.1.21), then tested from another server in my 'Server Subnet' and confirmed the content switch was working. I also ran a port scan to confirm port 9000 was open on the primary NS.
The final testing I did was to setup an internal ALB, all the configuration is identical except I put the front end IP on my server subnet (10.9.1.40). I updated the content switch VIP to use the .40 address and it works perfectly.
I suspect the issue is the return route, just not too sure what I'm missing... Sorry to bother you with such a long question, maybe it will be more obvious to you.
Cheers
Ross

Simon Gottschlag says

Hi,
I haven't actually used the ALB with an external IP in this case, since we have network appliances that we use for security, routing and nat. I'm not sure DSR works when using the external IP, maybe this can be solved by using an internal IP and NAT in the ALB - but not sure.
You may have to create the VIP on the public subnet with the actual external IP (maybe).
I usually setup the environment in a way that internal resources accesses the NetScaler on IPs on the internal subnet, never the external. If external access works and the only issue is related to internal servers trying to access external IP, use split-dns instead.

Ross says

Hi Simon
Thanks for your response. Just wanted to let you know I managed to figure out where I went wrong and it was pretty simple, If you're using Public IPs just make sure you include a rule in the NSG on your 'DMZ' subnet that has the public IP as the destination. Typically the NSG rule would have the private IP of the NIC as the destination (ie. if you had a PIP assigned to a vNIC), but since we are using ALBs and the VIP on the NetScaler is the public IP, this is the IP that needs to be in a rule. For example my rule allows traffic from any source/port to destination PIP on ports 80 and 443. This allows for ICA proxying etc.
Also I can confirm this setup works if you are using a server that only has 2 NICs, as was my case where my NSIP was on the same subnet as my backend 'server' subnet.
Hopefully this saves someone else some time.

Add comment

Your comment will be revised by the site if needed.