FME Flow: 2024.2

Deploying to Amazon Elastic Kubernetes Service Using Elastic File System

  1. Launch the Cluster and Connect

    2. To launch and connect to a cluster in Amazon Elastic Kubernetes Service (EKS), follow the instructions in the Getting Started with Amazon EKS documentation. You must configure some prerequisite resources such as an Amazon Virtual Private Cloud (VPC) and Identity and Access Management (IAM) role for the cluster.

    To deploy an FME Flow to the cluster for test purposes, we recommend the following settings:

    • NodeAutoScalingGroupMinSize: 3
    • NodeAutoScalingGroupDesiredSize: 3
    • NodeAutoScalingGroupMaxSize: 4
    • NodeInstanceType: t3.large
    • Disk size: 40 GB

    You can skip the step to "Launch a Guest Book Application."

  2. Install Helm and an NGINX Ingress Controller
    1. Check if Helm is already installed. Run:

      2. helm version

      If not installed, install Helm.

    2. Give Helm deploy permissions.

      3. Run:

      kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

    3. Install the NGINX ingress controller for AWS:

      4. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-<version>/deploy/static/provider/aws/deploy.yaml

      Replace <version> with the most recent version in the corresponding script in the NGINX Ingress Controller Installation Guide. As well, replace other syntax elements with any changes in the updated script.

  3. Configure Shared Storage Using Amazon Elastic File System

    4. To deploy FME Flow across multiple nodes, configure storage for the FME Flow System Share using Amazon Elastic File System (EFS).

    Note  The default Amazon EFS performance options may not be adequate for how your FME Flow is used. If you experience performance issues, see the Amazon EFS Performance documentation.
    1. From the AWS Management Console, navigate to EFS.
      1. Click Create file system.
      2. Select the new VPC created for the Kubernetes cluster.
      3. Click Customize.
      4. Add any tags as desired. Everything can be left as default. Click Next.
    2. The subnets should change to the EKS subnets. Add the EKS Node security group to the list of security groups so that our nodes have permissions to mount them.
      1. Click Next and then Create.
      2. Create Access Point. We recommend the following settings:
        • Root directory path: /data/fmeflowdata
        • POSIX user
          • User ID: 1363
          • Group ID: 1363
        • Root directory creation permissions
          • Owner user ID: 1363
          • Owner group ID: 1363
          • Permissions: 0755
    3. Install the Amazon EFS CSI Driver for your region.
      4. Note  You can use either dynamic or static provisioning for your EFS drive. The example in the next step shows how to use dynamic provisioning for the FME Flow System Share on EFS.

    4. To create the EFS storage class, save the following .yaml snippet and apply with kubectl:

      5. kind: StorageClass

      apiVersion: storage.k8s.io/v1

      metadata:

      name: efs-sc

      provisioner: efs.csi.aws.com

      parameters:

      provisioningMode: efs-ap

      fileSystemId: fs-<file-system-id>

      directoryPerms: "700"

       

      Replace <file-system-id> with the File System ID for your EFS.

    5. Create the persistent volume using the following yaml file as an example. You will need to replace the volumeHandle with your File system ID and Access point ID from the EFS.

      6. apiVersion: v1

      kind: PersistentVolume

      metadata:

      name: fmeflow-data

      spec:

      capacity:

      storage: 10Gi

      volumeMode: Filesystem

      accessModes:

      - ReadWriteMany

      persistentVolumeReclaimPolicy: Retain

      storageClassName: efs-sc

      csi:

      driver: efs.csi.aws.com

      volumeHandle: fs-<file-system-id>::fsap-<access-point-id>

       

    6. Add the Amazon EBS CSI driver add-on to your cluster.
  4. Deploy FME Flow
    1. Add the Safe Software Helm repository:

      2. helm repo add safesoftware https://safesoftware.github.io/helm-charts/

    2. Get the IP address of the Ingress Controller:

      3. kubectl describe service ingress-nginx-controller --namespace ingress-nginx

      Note the "LoadBalancer Ingress:" of the nginx-ingress-controller. This is the ingress into the cluster.

    3. Create a namespace in which to deploy. Run:

      4. kubectl create namespace <namespace_name>

      For example:

      kubectl create namespace fmeflow

    4. Create a values.yaml file that will pass user-supplied parameter values into the Helm chart.

      5. To get the default Helm chart values file, run:

      helm show values <chart>

      For example:

      helm show values safesoftware/fmeflow

      To write the values to a file:

      helm show values safesoftware/fmeflow >> values.yaml

      For more information on value files, see the Helm Docs.

      A current list of supported parameters for FME Flow can be found on GitHub.

    5. (Optional) Update parameters in values.yaml.

      6. The following are parameters you may want to change in the values.yaml file:

      Parameter

      Example or Possible Value

      Description
      fmeflow.image.tag 2024.0 Set to the FME Flow major release version you want to deploy, such as 2024.0 or 2024.1.
      fmeflow.engines[].name “standard-group” The name of the engine group. This can be changed, particularly if creating multiple engine groups.
      fmeflow.engines[].engines 2 Controls the number of engine pods to start.
      fmeflow.engines[].type STANDARD | DYNAMIC Controls the type of engine to start.
      deployment.hostname   If a DNS has been configured to point to the external IP address, set to that value. If using the external IP address only, leave as localhost and set deployment.useHostnameIngress to false.
      deployment.numCores 2 Starts two FME Flow Core pods for fault tolerance and load balancing.
      deployment.useHostnameIngress true | false If DNS is set up for the ingress controller, specify the DNS name for deployment.hostname and leave this parameter to true. If not, specify false.
      storage.fmeflow.class efs-sc Uses the EFS storage class set up previously. To verify the name, run kubectl get sc
    6. Install FME Flow.

      7. helm install <name> <chart> -f values.yaml -n <namespace>

      For example:

      helm install fmeflow safesoftware/fmeflow -f values.yaml -n fmeflow

      If you make changes to the values.yaml file after FME Flow is installed (such as to scale FME Flow engines), use the helm upgrade command:

      helm upgrade <name> <chart> -f values.yaml -n <namespace>

      For example:

      helm upgrade fmeflow safesoftware/fmeflow -f values.yaml -n fmeflow

    7. To access FME Flow after it deploys, invoke the external IP in your browser.
      Note  If you are using the external IP address, and not a DNS, to access the deployment, you must update the URLs for each of the FME Flow Web Services to that address. To update, see Services.