Documentation
StreamsPlaybackEncodingBitmovin Dashboard

Getting Started with AI Scene Analysis

This guide shows you how to enable and use AI Scene Analysis with your Bitmovin VOD encoding workflow.

Suggest Edits

Quick Start

AI Scene Analysis is enabled by adding a configuration object to your encoding configuration and runs automatically during the encoding process with no separate processing steps required.

Requires Bitmovin VOD Encoder v2.232.0 or later

Configuration Structure

Add the aiSceneAnalysis object to your encoding Start call or into an Encoding Template:

aiSceneAnalysis:
  features:
    assetDescription:
      filename: asset-description.json

  #the following are optional to output the analysis as a JSON file
      outputs:
      	- outputId: <output-id>
      	  outputPath: ai_content_analysis/{encodingId}
      			acl:
      			- permission: PUBLIC_READ


{
  "aiSceneAnalysis": {
    "features": {
      "assetDescription": {
        "filename": "asset-description.json",
        "outputs": [
          {
            "outputId": "<your-output-id>",
            "outputPath": "ai_content_analysis/{encodingId}",
            "acl": [
              {
                "permission": "PUBLIC_READ"
              }
            ]
          }
        ]
      }
    }
  }
}

Automatic Ad Placement

Enable automatic SCTE marker and keyframe insertion at scene boundaries:

aiSceneAnalysis:
	features:
		assetDescription:
      filename: asset-description.json
            automaticAdPlacement:
              schedule:
                - position: 300
              maxDeviation: 30
                duration: 45
                - position: 600
                  maxDeviation: 30
                  duration: 30

{
  "aiSceneAnalysis": {
    "features": {
            "assetDescription":{
              "filename": "asset-description.json"
            },
						"automaticAdPlacement":{
              "schedule":
              [
							{
              	"position": 300,
                "maxDeviation": 30,
                "duration": 45
              },
							{
                "position": 600,
                "maxDeviation": 30,
                "duration": 30
              }
              ]
            }
    }
  }
}

How it works:

  1. You specify ideal ad positions (e.g., every 5 minutes)
  2. AI Scene Analysis finds the nearest scene boundary within maxDeviation
  3. SCTE markers and key frames are automatically inserted at optimal points
  4. Result: Less disruptive ad breaks at natural content transitions

Example: For position: 300, maxDeviation: 30:

  • Search window: 270-330 seconds
  • SCTE marker placed at nearest scene boundary in this range
  • If no scene boundary found, marker placed at exact position

Retrieving Your Metadata

Once encoding completes, retrieve metadata in three ways:

1. Direct from Storage

If you configured output to your cloud storage:

# Example S3 path
s3://your-bucket/ai_content_analysis/<encoding-id>/asset-description.json

# Example GCS path
gs://your-bucket/ai_content_analysis/<encoding-id>/asset-description.json

2. Via Bitmovin API

curl --request GET \
     --url https://api.bitmovin.com/v1/ai-scene-analysis/analyses/by-encoding-id/{encoding_id}/details \
     --header 'X-Api-Key: <YOUR_API_KEY>' \
     --header 'accept: application/json'

Understanding Your Results

The metadata output is a JSON file with this structure:

{
  "scenes": [
    {
      "title": "Scene Title",
      "startInSeconds": 0,
      "endInSeconds": 198.133,
      "id": "uuid",
      "type": "MAIN_CONTENT",
      "content": {
        "characters": [...],
        "objects": [...],
        "settings": [...]
      },
      "summary": "Brief scene description",
      "verboseSummary": "Detailed scene description",
      "keywords": [...],
      "sensitiveTopics": [...],
      "iab": {
        "version": "3.0",
        "contentTaxonomies": [...],
        "adOpportunityTaxonomies": [...],
        "sensitiveTopicTaxonomies": [...]
      }
    }
  ],
  "description": "Overall asset description",
  "keywords": [...],
  "ratings": [...],
  "sensitiveTopics": [...],
  "iabSensitiveTopicTaxonomies": [...],
  "metadata": {
    "disclaimer": "Generated by Bitmovin AI",
    "version": "0.0.0"
  }
}

Common Use Cases

Contextual Advertising

  1. Enable AI Scene Analysis with IAB taxonomies
  2. Pass IAB codes to ad server for targeting
  3. Display contextually relevant ads

Auto Ad Placement

  1. Define ideal ad schedule (e.g., every 5 minutes)
  2. Configure automatic ad placement with reasonable deviation
  3. SCTE markers inserted automatically during encoding
  4. Feed output to AWS MediaTailor or similar SSAI solution

Content Discovery

  1. Extract scene metadata for entire library
  2. Ingest into search engine or recommendation system
  3. Use keywords and IAB taxonomies for matching
  4. Enable scene-level search and recommendations

Additional Resources

Updated about 8 hours ago