skip to Main Content

I have an AWS Batch compute environment with a Job Definition.

I created all this using Cloud Formation.

Now I want to add an EFS Volume (Name: EFS-000, File system ID: fs-9999999) and a MountPoint to this Job Definition.

I read

In the first link we have an example of a Task Definition ( AWS ECS instead AWS Batch concept)

{
    "containerDefinitions": [
        {
            "memory": 128,
            "portMappings": [
                {
                    "hostPort": 80,
                    "containerPort": 80,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "mountPoints": [
                {
                    "containerPath": "/usr/share/nginx/html",
                    "sourceVolume": "efs-html"
                }
            ],
            "name": "nginx",
            "image": "nginx"
        }
    ],
    "volumes": [
        {
            "name": "efs-html",
            "efsVolumeConfiguration": {
                "fileSystemId": "fs-1324abcd",
                "transitEncryption": "ENABLED"
            }
        }
    ],
    "family": "efs-tutorial"
}

Seems easy add the right code to my Cloud Formation recipe (I choosed yaml syntax). Into my ContainerDefinition I added…

Volumes:
  - Name: SRV 
    EfsVolumeConfiguration:
      FileSystemId: fs-9999999
      TransitEncryption: ENABLED

But when I run the Cloud Formation recipe I get….

The following resource(s) failed to update: [ContentJob1].
Property validation failure: [Encountered unsupported properties in {/ContainerProperties/Volumes/0}: [EfsVolumeConfiguration]]

If EfsVolumeConfiguration is not a valid property…

how I have to do to add an EFS volume to a AWS Batch Job Definition using Cloud Formation?

2

Answers


  1. Chosen as BEST ANSWER

    Until CloudFormation supported EFS, I used this solution. But its still usefull.

    I create a Launch Template

      LaunchTemplate:
        Type: AWS::EC2::LaunchTemplate
        Properties: 
          LaunchTemplateName: !Join ['', [!Ref ServiceName, '-EC2-', LaunchTemplate]]
          LaunchTemplateData: 
            UserData:
              Fn::Base64: !Sub |
                MIME-Version: 1.0 
                Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
    
                --==MYBOUNDARY==
                Content-Type: text/cloud-config; charset="us-ascii"
    
                runcmd:
                - mkdir /mnt/efs-misc
                - mkdir /mnt/efs-jobs
                - echo "${EfsJobs}:/ /mnt/efs-jobs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
                - echo "${EfsMisc}:/ /mnt/efs-misc nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
                - mount -a
                --==MYBOUNDARY==--
            InstanceMarketOptions:
              MarketType: spot
              SpotOptions:
                MaxPrice: 0.096
    

    Then I created a Compute Environment which use this template.

      BatchCompute:
        Type: 'AWS::Batch::ComputeEnvironment'
        DependsOn: LaunchTemplate
        Properties:
          ComputeEnvironmentName: !Join ['', [!Ref ServiceName, '-EC2']]
          ComputeResources:
            AllocationStrategy: SPOT_CAPACITY_OPTIMIZED
            DesiredvCpus: 0
            Ec2KeyPair: !Join ['-', [MyComputeEnv, !Ref EnvironmentLow]]
            InstanceRole: !GetAtt BatchInstanceProfile.Arn
            InstanceTypes:
              - c4.large
              - c5.large
            LaunchTemplate: 
              LaunchTemplateId: !Ref LaunchTemplate
              Version: $Latest
            MaxvCpus: 256
            MinvCpus: 0
            DesiredvCpus: 0
            SecurityGroupIds:
              - !Ref DefaultSecurityGroup
            Subnets: !Split [',', !Join [',', [!Ref SubnetA, !Ref SubnetB, !Ref SubnetC ]]]
            Type: SPOT
            SpotIamFleetRole: SpotFleetTagServiceRole
          ServiceRole: !Ref BatchServiceRole
          State: ENABLED
          Type: Managed
    

    Finaly, I used ContainerProperties/[Volumes|MountPoints] of AWS::Batch::JobDefinition to add my EFS volumes.

      ContentJob0:
        Type: 'AWS::Batch::JobDefinition'
        Properties:
          Type: Container
          ContainerProperties:
            Command: 
              - ls
            Environment:
              - Name: AAA
                Value: AAA
            ExecutionRoleArn: !GetAtt BatchJobRole.Arn
            Image: !Join ['', [!Ref 'AWS::AccountId','.dkr.ecr.', !Ref 'AWS::Region', '.amazonaws.com/', !Ref Image ]]
            LogConfiguration:
              LogDriver: awslogs
            JobRoleArn: !Ref BatchContainerRole
            Memory: 2048
            Vcpus: 2
            Volumes:
              - Name: MISC
                Host:
                  SourcePath: '/mnt/efs-0'
              - Name: JOBS
                Host:
                  SourcePath: '/mnt/efs-1'
            MountPoints:
              - SourceVolume: MISC
                ContainerPath: '/mnt/efs-0'
              - SourceVolume: JOBS
                ContainerPath: '/mnt/efs-1'
          JobDefinitionName: !Join ['-', ['MyJob', !Ref ServiceName, 'EC2', '0']]
          PlatformCapabilities:
            - EC2
          PropagateTags: true
          RetryStrategy: 
            Attempts: 3
          Timeout: 
            AttemptDurationSeconds: 300
    

    Since I need to set the price of SPOT instances and other thigs into the LaunchTemplate, it had been a good solution to me.

            InstanceMarketOptions:
              MarketType: spot
              SpotOptions:
                MaxPrice: 0.096
    

Please signup or login to give your own answer.
Back To Top
Search