FME Flow: 2024.2

Deploying to Azure Kubernetes Service Using Azure Files

  1. Launch the Cluster
    2. Note  For Microsoft Azure documentation on deploying and connecting to an AKS cluster, click here.
    1. Navigate to the Microsoft Azure portal in a web browser.
    2. Navigate to the Kubernetes services section.
    3. Click Click Create > Create a Kubernetes Cluster.
    4. On the Basics tab, specify the parameters:
      • Subscription: Specify your subscription and resource group that Kubernetes will run in.
      • Cluster Details: Specify a name for the cluster and the region to launch it. The Kubernetes version can remain as the default.
      • Nodes:For each FME Flow Core you wish to run, you need approximately 8GB of memory. For a single-Core deployment, we recommend 2 vCPUS, 8GB RAM, and 2 nodes. For a two-Core deployment, we recommend 4 vCPUs, 16GB RAM, and 3 nodes.
        • Note  More than one Core container requires a minimum of two nodes due to a constraint in the Helm chart that prevents Core containers residing on the same node. To ensure fault tolerance, we recommend three nodes because if a node is lost, two Cores can still run on the remaining two nodes.
      • For a test deployment, leave all other parameters at their default settings.
    5. Click Review + Create.
    6. After validation checks complete, review the settings, and click Create.
  2. Install Helm and an NGINX Ingress Controller
    1. Using Azure Cloud Shell, connect to the Kubernetes cluster.
    2. Check that Helm is installed. Run:

      helm version.

      If not installed, install Helm.

    3. Give Helm deploy permissions.

      4. Run:

      kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

    4. Install the NGINX ingress controller:

      5. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-<version>/deploy/static/provider/cloud/deploy.yaml

      Replace <version> with the most recent version in the corresponding script in the NGINX Ingress Controller Installation Guide. As well, replace other syntax elements with any changes in the updated script.

  3. Setup Shared Storage using Azure Files

    4. To deploy FME Flow across multiple nodes, create a storage class that defines a shared storage volume backed by Azure Files.

    1. Using Azure Cloud Shell, run:

      2. az aks show --resource-group <resource_group> --name <cluster_name> --query nodeResourceGroup -o tsv

      Replacing <resource_group> and <cluster_name> with what you defined when you deployed your cluster. This command provides the AKS node resource group for your cluster.

    2. Create a general-purpose storage account. Using the Azure CLI, run:

      3. az storage account create --resource-group <node_resource_group> --name <storageaccountname> --sku Premium_LRS --kind FileStorage

      Replacing <node_resource_group> with the result of the command in the previous step, and <storageaccountname> with a unique name for the storage account.

    3. Create a storage class in Kubernetes using the new Azure Files storage account. Use vi or nano to create the following file in Azure Cloud Shell:

      4. kind: StorageClass

      apiVersion: storage.k8s.io/v1

      metadata:

      name: azurefiles-fmeflow

      provisioner: kubernetes.io/azure-file

      parameters:

      skuName: Premium_LRS

      storageAccount: <storageaccountname>

      mountOptions:

      - uid=1363

      - gid=1363

      Replacing <storageaccountname> with the name you specified previously when creating the account.

    4. Run:

      5. kubectl apply -f <path_to_file>

      Where <path_to_file> is the YAML file created above.

      To see azurefiles-fmeflow in the list, run:

      kubectl get sc

    5. If RBAC is enabled, grant permissions to Azure Files:

      6. kubectl create clusterrolebinding add-on-cluster-vol-binder --clusterrole=cluster-admin --serviceaccount=kube-system:persistent-volume-binder

  4. Deploy FME Flow

    5. Perform the following in the Azure Cloud Shell:

    1. Add the Safe Software Helm repository:

      2. helm repo add safesoftware https://safesoftware.github.io/helm-charts/

    2. Get the IP address of the ingress controller:

      3. kubectl get services --namespace ingress-nginx

      Note the "External IP" of the nginx-ingress-controller. This is the ingress into the cluster. In a production environment, set up a DNS that points to this IP address and use that as the hostname when deploying FME Flow. In this procedure, we access FME Flow directly using the IP.

    3. Create a namespace in which to deploy. Run:

      4. kubectl create namespace <namespace_name>

      For example:

      kubectl create namespace fmeflow

    4. Create a values.yaml file that will pass user-supplied parameter values into the Helm chart.

      5. To get the default Helm chart values file, run:

      helm show values <chart>

      For example:

      helm show values safesoftware/fmeflow

      To write the values to a file:

      helm show values safesoftware/fmeflow >> values.yaml

      For more information on value files, see the Helm Docs.

      A current list of supported parameters for FME Flow can be found on GitHub.

    5. (Optional) Update parameters in values.yaml.

      6. The following are parameters you may want to change in the values.yaml file:

      Parameter

      Example or Possible Value

      Description
      fmeflow.image.tag 2024.0 Set to the FME Flow major release version you want to deploy, such as 2024.0 or 2024.1.
      fmeflow.engines[].name “standard-group” The name of the engine group. This can be changed, particularly if creating multiple engine groups.
      fmeflow.engines[].engines 2 Controls the number of engine pods to start.
      fmeflow.engines[].type STANDARD | DYNAMIC Controls the type of engine to start.
      deployment.hostname   If a DNS has been configured to point to the external IP address, set to that value. If using the external IP address only, leave as localhost and set deployment.useHostnameIngress to false.
      deployment.numCores 2 Starts two FME Flow Core pods for fault tolerance and load balancing.
      deployment.useHostnameIngress true | false If DNS is set up for the ingress controller, specify the DNS name for deployment.hostname and leave this parameter to true. If not, specify false.
      storage.fmeflow.class azurefiles-fmeflow Uses the Azure Files storage account set up previously.
    6. Install FME Flow.

      7. helm install <name> <chart> -f values.yaml -n <namespace>

      For example:

      helm install fmeflow safesoftware/fmeflow -f values.yaml -n fmeflow

      If you make changes to the values.yaml file after FME Flow is installed (such as to scale FME Flow engines), use the helm upgrade command:

      helm upgrade <name> <chart> -f values.yaml -n <namespace>

      For example:

      helm upgrade fmeflow safesoftware/fmeflow -f values.yaml -n fmeflow

    7. To access FME Flow after it deploys, invoke the external IP in your browser.
      Note  If you are using the external IP address, and not a DNS, to access the deployment, you must update the URLs for each of the FME Flow Web Services to that address. To update, see Services.