This document explains how to set up Bitmovin Encoding on AWS infrastructure so that the Bitmovin platform can run encoders using the AWS EC2 API.

The instructions in this document for the REST API Services apply to live encoding and file-based encoding. For a complete list of formats and input types, see the Bitmovin website.

Prerequisites

🚧

Activation required

This feature requires a commercial agreement and needs to be specifically activated for a Bitmovin account, it is not available by default. You will not be able to complete the configuration below without this activation.

• A Bitmovin account enabled for usage of Cloud Connect
  • If you want to use Cloud Connect with a sub organization, this sub organization must be enabled for usage of Cloud Connect
• An AWS account

Configure your AWS account

In this section, you will create a separate user that will be used by the Bitmovin platform when interacting with your AWS infrastructure. Also, we will create the appropriate infrastructure setup to enable Encoding jobs.

📘

Optional: Terraform

If you prefer infrastructure-as-code, Bitmovin AWS Cloud Connect is also configurable using Terraform scripts: Bitmovin Terraform Cloud Connect

Create an Amazon IAM user

🚧

Do not use multi-factor authentication (MFA)

Please make sure not to use any IAM policies on this account that require the user to have a virtual multi-factor authentication (MFA) device set up.

This user is used by the Bitmovin platform to get access to your AWS account and create compute resources to run parts of the the encoding service within it, including starting and stopping EC2 instances.

1. Open the Identity and Access Management (IAM) console: https://console.aws.amazon.com/iam/
2. Go to "Access management > Users", click the "Create user" button
3. In "Specify user details" (Step 1), provide a "User name" of your choice

Note: Since this user is only used for programmatic access, you do not need to grant AWS Management Console access here.

4. In "Set permissions" (Step 2), ...
   1. select "Permissions options > Attach policies directly"
   2. search for and select the managed "Permissions policies > AmazonEC2FullAccess" policy (recommended for testing/evaluation purposes only)

5. In "Review and create" (Step 3) check if everything is configured as shown before, then click the "Create user" button

📘

Tighten permissions for production

Since the managed AmazonEC2FullAccess policy might not be suitable for security-aware production configurations, we recommend to use a tailored user policy. See "Appendix: Limited-access IAM user policy".

📘

Considerations about the used AWS Account

Bitmovin Cloud Connect requires the IAM User to be created in the same AWS Account that is also used for Encoding Jobs. A role-based login across accounts is at the moment not possible. Its not a requirement but a suggestion to properly separate workloads and easily identify and allocate Bitmovin Cloud Connect related cloud costs to create a dedicated AWS Account for Bitmovin Cloud Connect only.

Get the Amazon IAM user credentials

This step will collect the access credentials to allow the Bitmovin platform to access your infrastructure.

1. Under "IAM > Access management > Users", click on the name of your newly created user to open the details view
2. At the top, click "Summary > Create access key"

3. In "Access key best practices & alternatives" (Step 1), select "Third-party service" and check "I understand the above recommendation and want to proceed to create an access key.", then click the "Next" button
4. In "Set description tag" (Step 2), provide a "Description tag value" of your choice, then click the "Create access key" button
5. In "Retrieve access keys" (Step 3), copy the "Access key" and "Secret access key", or click the "Download .csv file" button
6. Then click the "Done" button

Configure your Amazon VPC resources (optional)

We recommend that you use the default Virtual Private Cloud (VPC) for the respective region. The default VPC is the only VPC in which you can avoid specifying a subnet, which gives you the advantage that if one availability zone has reached its capacity, your instance will automatically be launched in another (free) zone. If you select a specific subnet on instance launch (which you are required to do if a non-default VPC is used), the instance can only be launched in the Availability Zone of the selected subnet, and if there are no more resources available in that zone, then no instance can be launched.

Recreate a default VPC

If you have deleted the default VPC in your AWS account, you can follow the AWS "Create a default VPC" docs.

Create a custom VPC configuration

Follow the AWS "Create a VPC" docs for your custom configuration.

• The subnet must have Auto-assign public IPv4 address enabled
• Custom VPCs are supported from Encoder v2.51.0 and above

Create a Security Group

A Security Group defines firewall rules that enable network access into your VPC and to the EC2 instances in particular.

1. Open the EC2 Dashboard console: https://console.aws.amazon.com/ec2/home
2. Make sure to have your desired region selected at the top right, e.g. "Europe (Frankfurt) eu-central-1"

3. Go to "Network & Security > Security Groups", click the "Create security group" button
4. Under "Basic details", provide a "Security group name" and a "Description" of your choice
5. Click the "Create security group" button (the security group needs to be created to be available for selection in the next step)
6. On the security group detail view under the "Outbound rules" tab, make sure that the default rule still exists (otherwise, create it according to the table)

7. On the security group detail view under the "Inbound rules" tab, click the "Edit inbound rules" button
8. Under the "Inbound rules" tab, create new rules for each of the following tables (3 tables will result in 5 rules) by clicking the "Add rule" button:

Note: Firewall rules are checked to be limited to the correct source ip addresses, and therefore allowing all ip addresses in source would fail the configuration when adding region settings.

📘

Live encodings

Additional inbound rules are required if you are encoding live streams transported over RTMP, SRT or Zixi.

Additional inbound rules for RTMP live streams

Additional inbound rules for SRT live streams

Additional inbound rules for Zixi live streams

Configure your Bitmovin account

Before you continue, make sure you have collected the following information from AWS:

• From your AWS account
  • Account ID
• From your IAM user
  • Access key
  • Secret access key
• From your security group
  • Security group ID
• (if applicable) From your custom VPC
  • Subnet ID

Link your AWS account

To enable your Bitmovin account to run encodings in your AWS account, you need to link it with Infrastructure and Region Settings objects.

1. Open the Bitmovin Dashboard: https://dashboard.bitmovin.com/
2. Go to "VOD/LIVE Encoding > Configurations", select "Cloud Connect"
3. Click the "Add infrastructure account" button, select the "Add AWS account" tab
4. Provide a "Name" of your choice, fill in the Access key, Secret (access) key and Account ID/Number, click the "Add" button
5. Click on the name of your newly linked infrastructure account to open the details view
6. Click the "Add region settings" button
7. Select the appropriate "Cloud Region" (where the security group was created), fill in the Security group ID (and if applicable, the Subnet ID), click the "Add" button

Request access to AMIs

The Bitmovin platform uses Amazon Machine Images (AMIs) from which to create encoder instances in AWS.

After you have created all infrastructure objects in the account (or accounts) that you need, ask your Bitmovin technical contact to whitelist access to the AMIs for your specific AWS service account. You will need to provide us with your AWS Account ID/Number to do so.

Run encoding jobs in AWS

After configuration has been completed, you will be able to run encoding jobs in your own AWS account. To do so, use the Bitmovin API client SDKs to submit encoding jobs, in the same way as you would do for encodings running in the Bitmovin Managed Cloud service. The only difference is that you need to specify the new infrastructure instead of public cloud regions.

Here is a Python snippet demonstrating how to link your encoding to your infrastructure.

    # ID of the Infrastructure object created in step 5.
    infra_id = ‘<infrastructure_id>’ 
    
    # AWS region of the AWS-connect setup
    infra_region = CloudRegion.AWS_EU_WEST_1
    infrastructure = InfrastructureSettings(infrastructure_id=infra_id, 
                            cloud_region=infra_region)
    
    encoding = Encoding(name='aws connect encoding',
        cloud_region=CloudRegion.EXTERNAL,
        infrastructure=infrastructure,
        encoder_version='STABLE')

Sub Organizations

If you have set up your infrastructure in a sub organization, you must tell the Bitmovin API that you want to run the encoding in that sub organization. Thus, in addition to the code snippet above, make sure to set the tenant_org_id alongside the api_key in the bitmovin_api object:

    # ID of the sub organisation you added the infrastructure to
    organisation_id = '<sub_organisation_id>'

    bitmovin_api = BitmovinApi(api_key=config_provider.get_bitmovin_api_key(), 
                       tenant_org_id=organisation_id,
                       logger=BitmovinApiLogger())

Resource Quotas

If you want to run several encodings in parallel, the default limits may not be sufficient. In that case, you will have to request limit increases for the following resource in your Region(s), through the Service Quotas page:

For the limits to request we will be using these variables:

Using the variables above, please request the following limits:

The values above assume 8-core instances. If you believe that your use case requires instances with a different number of cores, this number may need to be increased after discussion with your Bitmovin team.

We recommend requesting a limit increase for spot instance requests of at least 20,000-30,000 to ensure error-free and fast encodings.

Generally, it is a good idea to multiply the expected limit calculated for your current situation by 2, to have some margin in case you need to ramp up.

📘

New Resource Quota requests for encoder versions >= 2.179.0

Starting with the Bitmovin Encoder Version 2.179.0, we use the new more performant IO2 SSD storage type for our encodings starting with certain cloud regions. We are going to rollout this change to further regions.

When you're using an encoder version greater or equal to 2.179.0 or BETA as your configured encoder version in your encoding workflow in combination with Bitmovin Cloud Connect AWS, you need to ensure to request new quota requests for the IO2 storage type in order to ensure a smooth encoding process:

• IOPS for Provisioned IOPS SSD (io2) volumes
• Storage for Provisioned IOPS SSD (io2) volumes, in TiB

In addition to this change, we've also doubled the amount of IOPS used per disk to increase encoding performance. In order to keep your workflow running smoothly with the abovementioned newer encoder versions, you need to request quota increases for double the amount of IO1 (and IO2) than you had configured beforehand. We've already adjusted the values in the table above accordingly.

Note: there is no form for Spot Instance Request Limit increases, so you have to request this in the Use Case description of the Instance Limit increase form.

Appendix

Troubleshooting Guide

When configuring your Cloud Connect account, several things might go wrong, especially when you are doing it for the first time. Below is a concise list of common issues along with corresponding error messages or behaviors.

Wrong account number, access key, secret key

You will encounter a failure when attempting to create a new region setting for that infrastructure.

Error message: “Got AmazonEC2Exception on retrieval of security group 'sg-12345': AWS was not able to validate the provided access credentials.”

Wrong security group ID

You will encounter a failure when attempting to create a new region setting.

Error message: “Got AmazonEC2Exception on retrieval of security group 'sg-12345': Invalid id: "sg-12345".”

Wrong region for the security group ID

You will encounter a failure when attempting to create a new region setting.

Error message: “Got AmazonEC2Exception on retrieval of security group 'sg-12345': The security group 'sg-12345' does not exist.”

Wrong inbound rule "All traffic" - security group internal

You will encounter a failure when attempting to create a new region setting.

Error message: “No 'All' IpPermission found for Group sg-12345 .”

Wrong inbound rules: "SSH port / Port 22"

The encoding process is stuck in the 'Queue' state, but resolving the inbound rule issue allows the encoding to complete.

Wrong inbound rules: "Port 999"

The encoding process is stuck in the 'Queue' state, but resolving the inbound rule issue allows the encoding to complete.

Wrong outbound rule

The encoding process is stuck in the 'Queue' state, but resolving the inbound rule issue allows the encoding to complete.

Limited-access IAM user policy

The following JSON snippet contains the set of permissions required to perform encodings on EC2 instances. You can assign it to the IAM User, if you prefer not to use the AmazonEC2FullAccess policy.

1. Under "IAM > Access management > Users", click on the name of your (newly created) user to open the details view
2. Under "Permissions > Permissions policies", click on the "Add permissions > Create inline policy" button
3. Select "Policy editor > JSON"
4. Paste the following JSON policy into the "Policy editor" window:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:TerminateInstances",
        "ec2:StartInstances",
        "ec2:CreateTags",
        "ec2:RunInstances",
        "ec2:StopInstances"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:subnet/*",
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:volume/*",
        "arn:aws:ec2:*:*:security-group/*",
        "arn:aws:ec2:*:*:network-interface/*",
        "arn:aws:ec2:*::image/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:RequestSpotInstances",
        "ec2:DescribeTags",
        "ec2:DescribeVpnConnections",
        "ec2:DescribeVolumesModifications",
        "ec2:DescribeSpotInstanceRequests",
        "ec2:DescribeSecurityGroups",
        "ec2:GetConsoleOutput",
        "ec2:DescribeSpotPriceHistory",
        "ec2:CancelSpotInstanceRequests",
        "ec2:GetPasswordData",
        "ec2:GetLaunchTemplateData",
        "ec2:DescribeScheduledInstances",
        "ec2:DescribeVpcs",
        "ec2:DescribeScheduledInstanceAvailability",
        "ec2:DescribeElasticGpus",
        "ec2:DescribeInstanceStatus"
      ],
      "Resource": "*"
    }
  ]
}

5. Click the "Next" button
6. In "Review and create" (Step 2), provide a "Policy name" of your choice, then click the "Create policy" button