Motivation

On 2.186.0 we changed our internal rate change logic to always produce output segments of constant length. This means that in any configuration, every segment of an output video stream will have the exact same length in frames e.g. 24 fps & 4 seconds --> 96 frames segment length. This behaviour is required by some downstream workflows, e.g. packetizers.

But with this new rate change behaviour we have some new requirements for the frame rate values on multiple video rendition output configuration.

New Restrictions and how to resolve them

There are 3 cases that will lead to a new fail fast error by our encoder which are described below and how to avoid them:

1. Output fps compatibility on different video renditions

It is not possible anymore to configure combinations of fractional and non fractional output fps on different output renditions. The solution is to either change the non fractional ones to fractional output fps values or the fractional ones to non fractional fps values.

--> all videoConfigurations must have fractional fps values or all videoConfigurations must have non-fractional fps values

The most common fractional fps values are 23.976 fps (24000 / 1001), 29.97 fps (30000 / 1001) or 59.94 fps (60000 / 1001). Non-fractional fps are those without a decimal point like 24 fps, 25 fps or 30 fps.

Example (java)

var videoConfig1 = new H264VideoConfiguration();
var videoConfig2 = new H264VideoConfiguration();
var videoConfig3 = new H264VideoConfiguration();

videoConfig1.setRate(24.0);❌
videoConfig2.setRate(24000.0 / 1001.0);❌
videoConfig3.setRate(30000.0 / 1001.0);❌

videoConfig1.setRate(24.0);✅
videoConfig2.setRate(24.0);✅
videoConfig3.setRate(30.0);✅

videoConfig1.setRate(24000.0 / 1001.0);✅
videoConfig2.setRate(24000.0 / 1001.0);✅
videoConfig3.setRate(30000.0 / 1001.0);✅

If it is required for you to have both a fractional and a non fractional output fps. It is necessary since 2.186.0 to do two separate encodings to achieve all the outputs you need. It is still possible to fallback to a version older than 2.185.0 but then at least one of the output renditions will show a non-constant segment length behavior (for more details on this check "Background" below).

2. Fps values are not set on some videoConfigurations

When the fps values are set only on some of your videoConfigurations our encoder will take over the input fps for the videoConfigurations that don't have an fps value set. Depending on your values for the configured fps and for the value of the input fps a configuration like this might work or just run into Case 1.) (compare above). To prevent this behaviour it is recommended to:

--> for all videoConfigurations a value for fps should be set.

Example (java) with input fps 23.976:

videoConfig1.setRate(24.0);❌
videoConfig2.setRate(24.0);❌
videoConfig3.setRate(null);❌ (not set)

this config for an input of 23.976 fps would automatically lead to the following setup:

videoConfig1.setRate(24.0);❌
videoConfig2.setRate(24.0);❌
videoConfig3.setRate(23.976);❌

which is invalid by case 1.) and should be resolved by just setting a value to all videoConfigurations e.g.:

videoConfig1.setRate(24.0);✅
videoConfig2.setRate(24.0);✅
videoConfig3.setRate(24.0);✅

3. Fractional segment length

with fractional segment length values, e.g. 1.8 seconds segment legnth. It might be that even different non-fractional values are incompatible to each other. It is not possible to reach exactly 1.8 seconds with some fps values like e.g. 24 fps but with e.g. 25 fps it is possible to reach exactly 1.8 seconds. This case of two different segment length configurations is not supported.

--> all fps values need to be chosen to fit the configured segment duration.

assuming the segment duration is configured to 1.8 seconds:

videoConfig1.setRate(24.0);❌
videoConfig2.setRate(25.0);❌

videoConfig1.setRate(24.0);✅
videoConfig2.setRate(48.0);✅ (all streams would result in 1.8333 seconds output segment length)

videoConfig1.setRate(25.0);✅
videoConfig2.setRate(50.0);✅
videoConfig3.setRate(30.0);✅ (all streams result in 1.8 seconds segment length)

4. Wrong rounding

In case when fractional frames rates are rounded wrongly, the new fail fast will trigger.

Fractional frame rates (besides the common ones described above) are constructed by an integer value multiplied by 1000 and devided by 1001. E.g.: 24000.0 / 1001.0 = 23.976023976. We configured the new fail fast to let rounded values pass that are similar to 23.976 in case of 24000.0 / 1001. But roundings like 23.975 or 23.98 for the value of 24000.0 / 1001.0 will not pass.

--> fractional frame rate configurations need to have some precision. It is recommended to just configure them with the division:

Example (java)

videoConfig1.setRate(24000.0 / 1001.0);✅
videoConfig2.setRate(23.975);❌
videoConfig3.setRate(29.97);✅

videoConfig1.setRate(24000.0 / 1001.0);✅
videoConfig2.setRate(23.976);✅
videoConfig3.setRate(29.97);✅

better

videoConfig1.setRate(24000.0 / 1001.0);✅
videoConfig2.setRate(24000.0 / 1001.0);✅
videoConfig3.setRate(30000.0 / 1001.0);✅

Background

On 2.186.0 we changed our internal rate change logic to always produce output segments of constant length. This means that in any configuration of (e.g. 4 seconds segment length and 24 fps output), every segment will have a length of 96 frames. This is different from the previous behavior (before version 2.186.0) of the rate change logic. In cases where a rate change of 23.976 fps (fractional) to 24 fps was done in the former implementation, some of the segments would be 97 frames in length to correct the drift error between the framerates.

We are now correcting this error in a different way which enables us to consistently output 96 frames for 24 fps outputs no matter the input fps provided to our encoder. But with this new way of error correction comes a new requirement for it to work correctly for the configured output fps of multiple output streams:

With the new rate change behavior in 2.186.0 the encoder will produce segments with a fixed number of frames. However, this comes with a restriction to the timing drift correction between the input and all outputs. For example, for a rate change from 24 fps to 23.976 fps, the rate change logic will take the input frames and modify their presentation times to the new framerate of 23.976 fps. Every repositioned frame adds a small drift error to the original input. This error will accumulate over time until it reaches a value of exactly 1 frame in duration (1 frame duration = 1 / fps). At this point in the video stream, the error will be corrected by dropping the corresponding input frame resulting in the expected output frame rate of 23.976 fps.

Since only one error correction from the input to all outputs is supported, it is no longer possible to e.g. change the frame rate from 24 to 23.976 and on a second output stream convert the input frame rate from 24 to 25. The conversion from 23.976 to 24 requires a drift error correction but the conversion from 24 to 25 requires no drift error correction. The conversion from 24 to 25 requires only the duplication of 4 frames from the 96 input frames to reach the 100 output frames required for 25 fps and this doesn't require a drift error correction.

The most common use case of this error correction vs. no error correction is the usage of fraction fps and non-fractional fps.