Using Bitmovin Cloud Connect with GCP
This document explains how to set up Bitmovin Encoding on Google Cloud Platform (GCP) infrastructure so that the Bitmovin platform can run encoders using the Google Compute Engine (GCE) API.
The instructions in this document for the REST API Services apply to live encoding and file-based encoding. For a complete list of formats and input types, see the Bitmovin website.
Prerequisites
Activation required
This feature requires a commercial agreement and needs to be specifically activated for a Bitmovin account, it is not available by default. You will not be able to complete the configuration below without this activation.
- A Bitmovin account enabled for usage of Cloud Connect
- If you want to use Cloud Connect with a sub organization, this sub organization must be enabled for usage of Cloud Connect
- A Google Cloud account
- A Google Cloud project with enabled Compute Engine API
- It is recommended to have a dedicated project for encoding with Bitmovin
Configure your GCP account
In this section, you will create a separate resource group that will be used by the Bitmovin platform when interacting with your GCP infrastructure. Also, we will create the appropriate infrastructure setup to enable Encoding jobs.
Optional: Configure with Terraform
If you prefer infrastructure-as-code, Bitmovin AWS Cloud Connect is also configurable using Terraform scripts: Bitmovin Terraform Cloud Connect
Create a service account
The service account is used by the Bitmovin platform to get access to your GCP account and create compute resources to run parts of the the encoding service within it, including starting and stopping GCE instances.
- Open the "IAM & Admin > Service Accounts" settings in the Google Cloud console: https://console.cloud.google.com/iam-admin/serviceaccounts
- Select the project you want to use for encoding with Bitmovin, then click the "Create service account" button
- In "Service account details" (Step 1), provide a "Service account ID", then click the "Create and continue" button
- In "Grant this service account access to project" (Step 2), select the "Compute Instance Admin (v1)" role, then click the "Continue" button
- In "Grant users access to this service account" (Step 3), click the "Done" button
- Click on the "Email" or your newly created service account
- In the "Keys" tab, select "Add key" and click "Create new key"
- Select "JSON" and click the "Create" button
- This will download a JSON service account key containing your private key and other information, required in later steps
Configure a VPC (optional)
By default, the Bitmovin platform assumes that you are running instances in an auto mode VPC named "default".
If you prefer creating a specific custom mode VPC for your project, make a note of the VPC name and subnet(s) that you create. Follow the Google "Create and manage VPC networks" docs for your custom configuration.
- Custom VPCs are supported from Encoder v2.51.0 and above
Configure and create firewall rules
Firewall rules enable network access into your VPC and to the GCP infrastructure in particular.
- Open the "Network Security > Firewall policies" settings in the Google Cloud console: https://console.cloud.google.com/networking/firewalls/list
- Under the "VPC firewall rules" section, make sure the existing firewall rules are configured according to the following tables by clicking on the respective role, then click the "Edit" button:
Key | Value |
---|---|
Name | default-allow-internal |
Description | For communication between the session manager VM instance and its instance manager VM instances |
Source filter | IPv4 ranges: 10.0.0.0/8 (or add all VPC Subnets) |
Protocols and ports | TCP: 0-65535 UDP: 0-65535 Other: icmp |
Key | Value |
---|---|
Name | default-allow-ssh |
Description | For incoming commands from the Bitmovin API to control the encoding |
Source filter | IPv4 ranges: 104.199.97.13/32, 35.205.157.162/32 |
Protocols and ports | TCP: 22 |
- Additionally, create new rules for each of the following tables by clicking the "Create firewall rule" button:
Key | Value |
---|---|
Name | encoder-service |
Description | For communication with the service that manages the encoding |
Source filter | IPv4 ranges: 104.199.97.13/32, 35.205.157.162/32 |
Protocols and ports | TCP: 9999 |
Key | Value |
---|---|
Name | encoder-service-https |
Description | For HTTPS communication with the service that manages the encoding |
Source filter | IPv4 ranges: 104.199.97.13/32, 35.205.157.162/32 |
Protocols and ports | TCP: 9443 |
Live encodings
Additional firewall rules are required if you are encoding live streams transported over RTMP, SRT or Zixi.
Additional inbound rules for RTMP live streams
Key | Value |
---|---|
Name | rtmp-listener |
Description | For RTMP live streams |
Source filter | IPv4 ranges: 0.0.0.0/0 (or the specific set of addresses where streams will originate from) |
Protocols and ports | TCP: 1935 |
Key | Value |
---|---|
Name | rtmps-listener |
Description | For RTMPS live streams |
Source filter | IPv4 ranges: 0.0.0.0/0 (or the specific set of addresses where streams will originate from) |
Protocols and ports | TCP: 443 |
Additional inbound rules for SRT live streams
Key | Value |
---|---|
Name | srt-listener |
Description | For SRT live streams |
Source filter | IPv4 ranges: 0.0.0.0/0 (or the specific set of addresses where streams will originate from) |
Protocols and ports | TCP: 2088 UDP: 2088, 2089, 2090, 2091 |
Additional inbound rules for Zixi live streams
Key | Value |
---|---|
Name | zixi-listener |
Description | For Zixi live streams |
Source filter | IPv4 ranges: 0.0.0.0/0 (or the specific set of addresses where streams will originate from) |
Protocols and ports | TCP: 4444 |
Configure your Bitmovin account
Before you continue, make sure you have collected the following information from your GCP account:
- From the JSON service account key
- Project ID
- Client email
- Private key
- From your (custom) VPC: https://console.cloud.google.com/networking/networks/list
- Network ID
- Subnet ID
Link your GCP project
To enable your Bitmovin account to run encodings in your GCP project, you need to link it with Infrastructure and Region Settings objects.
- Open the Bitmovin Dashboard: https://dashboard.bitmovin.com/
- Go to "VOD/LIVE Encoding > Cloud Connect"
- Click the "Add infrastructure account" button, select the "Google Cloud Platform" option, click the "Next" button
- Provide a "Name" of your choice, fill in the Service Account Email, Project ID and Private Key, click the "Next" button
- Select the appropriate "Cloud Region", fill in the Network ID and Subnet ID, click the "Next" button
Getting access to MIs
The Bitmovin platform uses private machine images (MIs) from which to create encoder instances in GCE. After you have created all infrastructure objects in the account (or accounts) that you need, ask your Bitmovin technical contact to whitelist access to the MIs for your specific GCP service account.
You will need to provide:
- the Client email of your GCP service account, and
- the Organization ID of your Bitmovin account (or email address)
Run encoding jobs in GCP
After configuration has been completed, you will be able to run encoding jobs in your own GCP project. To do so, use the Bitmovin API client SDKs to submit encoding jobs, in the same way as you would do for encodings running in the Bitmovin Managed Cloud service. The only difference is that you need to specify the new infrastructure instead of public cloud regions.
Here is a Python snippet demonstrating how to link your encoding to your infrastructure.
# ID of the Infrastructure object
infra_id = ‘<infrastructure_id>’
# GCP region of the GCP-connect setup
infra_region = CloudRegion.GOOGLE_EUROPE_WEST_1
infrastructure = InfrastructureSettings(infrastructure_id=infra_id,
cloud_region=infra_region)
encoding = Encoding(name='gcp connect encoding',
cloud_region=CloudRegion.EXTERNAL,
infrastructure=infrastructure,
encoder_version='STABLE')
Sub Organizations
If you have set up your infrastructure in a sub organization, you must tell the Bitmovin API that you want to run the encoding in that sub organization. Thus, in addition to the code snippet above, make sure to set the tenant_org_id
alongside the api_key
in the bitmovin_api
object:
# ID of the sub organisation you added the infrastructure to
organisation_id = '<sub_organisation_id>'
bitmovin_api = BitmovinApi(api_key=config_provider.get_bitmovin_api_key(),
tenant_org_id=organisation_id,
logger=BitmovinApiLogger())
Resource Quotas
If you want to run several encodings in parallel, then the default quota limits may not be sufficient. In that case, you will have to request limit increases for the following quotas in your regions, through the Quotas page in the Cloud Console.
For the limits to request we will be using these variables:
Variable name | Explanation |
---|---|
(maximum number of encodings) | The maximum number of parallel encodings the infrastructure must be able to run. Typically this is the number of encoding slots assigned to the Bitmovin account or sub-org associated with the infrastructure. |
(maximum number of instances per encoding) | The number of instances used by one encoding. This number varies depending on the input file size and the number and data rate of the encoder representations. However, we recommend to use 60 as the maximum number of instances per encoding when getting started and to increase this limit if it proves insufficient. For Dolby Vision encodings and conversions, starting from Encoder v2.208.0 , the maximum number of instances have been increased to - 100 instances if the encodingMode configured is STANDARD (which currently maps to TWO_PASS ), SINGLE_PASS or TWO_PASS - 200 instances if the encodingMode configured is THREE_PASS |
Using the variables above, please request the following limits:
Quota name | Limit to request |
---|---|
In-use IP addresses | (maximum number of encodings) * (maximum number of instances per encoding) |
CPUs | (maximum number of encodings) * 8 |
Preemptible CPUs | (maximum number of encodings) * (maximum number of instances per encoding) * 8 |
Persistent Disk SSD (TB) | (maximum number of encodings) * 0.5 + ((number of instances) * (number of encodings)) * 0.05 |
VM instances | (maximum number of encodings) * (maximum number of instances per encoding) |
The values above assume 8-core instances. If you believe that your use case requires instances with a different number of cores, this number may need to be increased after discussion with your Bitmovin team.
The maximum number of instances needed depends on the maximum number of parallel encodings multiplied by the maximum number of instances for one encoding. The number of instances used by one encoding varies depending on the input file size and the number and data rate of the encoder representations.
Generally, it is a good idea to multiply the expected limit calculated for your current situation by 2, to have some margin in case you need to ramp up.
Updated about 1 month ago