Using Bitmovin Cloud Connect with GCP

This document explains how to set up Bitmovin Encoding on Google Cloud Platform (GCP) infrastructure so that the Bitmovin platform can run encoders using the Google Compute Engine (GCE) API.

The instructions in this document for the REST API Services apply to live encoding and file-based encoding. For a complete list of formats and input types, see the Bitmovin website.

Prerequisites

🚧

Activation required

This feature requires a commercial agreement and needs to be specifically activated for a Bitmovin account, it is not available by default. You will not be able to complete the configuration below without this activation.

  • A Bitmovin account enabled for usage of Cloud Connect
    • If you want to use Cloud Connect with a sub organization, this sub organization must be enabled for usage of Cloud Connect
  • A Google Cloud account
  • A Google Cloud project with enabled Compute Engine API
    • It is recommended to have a dedicated project for encoding with Bitmovin

Configure your GCP account

In this section, you will create a separate resource group that will be used by the Bitmovin platform when interacting with your GCP infrastructure. Also, we will create the appropriate infrastructure setup to enable Encoding jobs.

📘

Optional: Configure with Terraform

If you prefer infrastructure-as-code, Bitmovin AWS Cloud Connect is also configurable using Terraform scripts: Bitmovin Terraform Cloud Connect

Create a service account

The service account is used by the Bitmovin platform to get access to your GCP account and create compute resources to run parts of the the encoding service within it, including starting and stopping GCE instances.

  1. Open the "IAM & Admin > Service Accounts" settings in the Google Cloud console: https://console.cloud.google.com/iam-admin/serviceaccounts
  2. Select the project you want to use for encoding with Bitmovin, then click the "Create service account" button
  3. In "Service account details" (Step 1), provide a "Service account ID", then click the "Create and continue" button
  1. In "Grant this service account access to project" (Step 2), select the "Compute Instance Admin (v1)" role, then click the "Continue" button
  1. In "Grant users access to this service account" (Step 3), click the "Done" button
  2. Click on the "Email" or your newly created service account
  3. In the "Keys" tab, select "Add key" and click "Create new key"
  4. Select "JSON" and click the "Create" button
    1. This will download a JSON service account key containing your private key and other information, required in later steps

Configure a VPC (optional)

By default, the Bitmovin platform assumes that you are running instances in an auto mode VPC named "default".

If you prefer creating a specific custom mode VPC for your project, make a note of the VPC name and subnet(s) that you create. Follow the Google "Create and manage VPC networks" docs for your custom configuration.

  • Custom VPCs are supported from Encoder v2.51.0 and above

Configure and create firewall rules

Firewall rules enable network access into your VPC and to the GCP infrastructure in particular.

  1. Open the "Network Security > Firewall policies" settings in the Google Cloud console: https://console.cloud.google.com/networking/firewalls/list
  2. Under the "VPC firewall rules" section, make sure the existing firewall rules are configured according to the following tables by clicking on the respective role, then click the "Edit" button:
KeyValue
Namedefault-allow-internal
DescriptionFor communication between the session manager VM instance and its instance manager VM instances
Source filterIPv4 ranges: 10.0.0.0/8 (or add all VPC Subnets)
Protocols and portsTCP: 0-65535
UDP: 0-65535
Other: icmp
KeyValue
Namedefault-allow-ssh
DescriptionFor incoming commands from the Bitmovin API to control the encoding
Source filterIPv4 ranges: 104.199.97.13/32, 35.205.157.162/32
Protocols and portsTCP: 22
  1. Additionally, create new rules for each of the following tables by clicking the "Create firewall rule" button:
KeyValue
Nameencoder-service
DescriptionFor communication with the service that manages the encoding
Source filterIPv4 ranges: 104.199.97.13/32, 35.205.157.162/32
Protocols and portsTCP: 9999
KeyValue
Nameencoder-service-https
DescriptionFor HTTPS communication with the service that manages the encoding
Source filterIPv4 ranges: 104.199.97.13/32, 35.205.157.162/32
Protocols and portsTCP: 9443

📘

Live encodings

Additional firewall rules are required if you are encoding live streams transported over RTMP, SRT or Zixi.

Additional inbound rules for RTMP live streams

KeyValue
Namertmp-listener
DescriptionFor RTMP live streams
Source filterIPv4 ranges: 0.0.0.0/0 (or the specific set of addresses where streams will originate from)
Protocols and portsTCP: 1935
KeyValue
Namertmps-listener
DescriptionFor RTMPS live streams
Source filterIPv4 ranges: 0.0.0.0/0 (or the specific set of addresses where streams will originate from)
Protocols and portsTCP: 443

Additional inbound rules for SRT live streams

KeyValue
Namesrt-listener
DescriptionFor SRT live streams
Source filterIPv4 ranges: 0.0.0.0/0 (or the specific set of addresses where streams will originate from)
Protocols and portsTCP: 2088
UDP: 2088, 2089, 2090, 2091

Additional inbound rules for Zixi live streams

KeyValue
Namezixi-listener
DescriptionFor Zixi live streams
Source filterIPv4 ranges: 0.0.0.0/0 (or the specific set of addresses where streams will originate from)
Protocols and portsTCP: 4444

Configure your Bitmovin account

Before you continue, make sure you have collected the following information from your GCP account:

Link your GCP project

To enable your Bitmovin account to run encodings in your GCP project, you need to link it with Infrastructure and Region Settings objects.

  1. Open the Bitmovin Dashboard: https://dashboard.bitmovin.com/
  2. Go to "VOD/LIVE Encoding > Cloud Connect"
  3. Click the "Add infrastructure account" button, select the "Google Cloud Platform" option, click the "Next" button
  4. Provide a "Name" of your choice, fill in the Service Account Email, Project ID and Private Key, click the "Next" button
  5. Select the appropriate "Cloud Region", fill in the Network ID and Subnet ID, click the "Next" button

Getting access to MIs

The Bitmovin platform uses private machine images (MIs) from which to create encoder instances in GCE. After you have created all infrastructure objects in the account (or accounts) that you need, ask your Bitmovin technical contact to whitelist access to the MIs for your specific GCP service account.

You will need to provide:

  • the Client email of your GCP service account, and
  • the Organization ID of your Bitmovin account (or email address)

Run encoding jobs in GCP

After configuration has been completed, you will be able to run encoding jobs in your own GCP project. To do so, use the Bitmovin API client SDKs to submit encoding jobs, in the same way as you would do for encodings running in the Bitmovin Managed Cloud service. The only difference is that you need to specify the new infrastructure instead of public cloud regions.

Here is a Python snippet demonstrating how to link your encoding to your infrastructure.

    # ID of the Infrastructure object
    infra_id = ‘<infrastructure_id>’ 
    
    # GCP region of the GCP-connect setup
    infra_region = CloudRegion.GOOGLE_EUROPE_WEST_1
    infrastructure = InfrastructureSettings(infrastructure_id=infra_id, 
                            cloud_region=infra_region)
    
    encoding = Encoding(name='gcp connect encoding',
        cloud_region=CloudRegion.EXTERNAL,
        infrastructure=infrastructure,
        encoder_version='STABLE')

Sub Organizations

If you have set up your infrastructure in a sub organization, you must tell the Bitmovin API that you want to run the encoding in that sub organization. Thus, in addition to the code snippet above, make sure to set the tenant_org_id alongside the api_key in the bitmovin_api object:

    # ID of the sub organisation you added the infrastructure to
    organisation_id = '<sub_organisation_id>'

    bitmovin_api = BitmovinApi(api_key=config_provider.get_bitmovin_api_key(), 
                       tenant_org_id=organisation_id,
                       logger=BitmovinApiLogger())

Resource Quotas

If you want to run several encodings in parallel, then the default quota limits may not be sufficient. In that case, you will have to request limit increases for the following quotas in your regions, through the Quotas page in the Cloud Console.

For the limits to request we will be using these variables:

Variable nameExplanation
(maximum number of encodings)The maximum number of parallel encodings the infrastructure must be able to run. Typically this is the number of encoding slots assigned to the Bitmovin account or sub-org associated with the infrastructure.
(maximum number of instances per encoding)The number of instances used by one encoding.
This number varies depending on the input file size and the number and data rate of the encoder representations. However, we recommend to use 60 as the maximum number of instances per encoding when getting started and to increase this limit if it proves insufficient.
For Dolby Vision encodings and conversions, starting from Encoder v2.208.0 , the maximum number of instances have been increased to
- 100 instances if the encodingMode configured is STANDARD (which currently maps to TWO_PASS), SINGLE_PASSor TWO_PASS
- 200 instances if the encodingMode configured is THREE_PASS

Using the variables above, please request the following limits:

Quota nameLimit to request
In-use IP addresses(maximum number of encodings) * (maximum number of instances per encoding)
CPUs(maximum number of encodings) * 8
Preemptible CPUs(maximum number of encodings) * (maximum number of instances per encoding) * 8
Persistent Disk SSD (TB)(maximum number of encodings) * 0.5 + ((number of instances) * (number of encodings)) * 0.05
VM instances(maximum number of encodings) * (maximum number of instances per encoding)

The values above assume 8-core instances. If you believe that your use case requires instances with a different number of cores, this number may need to be increased after discussion with your Bitmovin team.

The maximum number of instances needed depends on the maximum number of parallel encodings multiplied by the maximum number of instances for one encoding. The number of instances used by one encoding varies depending on the input file size and the number and data rate of the encoder representations.

Generally, it is a good idea to multiply the expected limit calculated for your current situation by 2, to have some margin in case you need to ramp up.