Live Slicer Failover

Requires Live Slicer version 20081700 or higher

Live Slicer failover minimizes the impact to your viewer's playback experience when a Live Slicer's performance is sub-optimal by automatically switching the live stream's source to a different Live Slicer. It is able to do so because each Live Slicer in a failover group provides status information to our system at regular and frequent intervals. If the primary Live Slicer is unhealthy, then our system will switch the source of the live stream to a different Live Slicer. This may cause viewers to experience a few seconds of discontinuity.

Key information:

• Live Slicer failover is only supported when streaming a live channel.
• Setting up Live Slicer failover requires creating a failover group and defining the failover_id and enable_remote_config settings within each Live Slicer's configuration file.
• You may only assign a Live Slicer to a failover group if it has been previously been started.

• You may only assign a Live Slicer to a single failover group.

  Reassign a Live Slicer to a different failover group by first removing it from its current failover group.

• You may only assign a live channel to a single failover group.

  Reassign a live channel to a different failover group by first removing it from its current failover group.

• Live Slicer health status is assessed through one or more metric(s). A Live Slicer is considered unhealthy if at least one metric falls below a custom threshold for a given duration.

  Restart the Live Slicers associated with a failover group to apply threshold changes.

  Learn more about metrics.

  Each metric is described below.

  Metric	Description
  Audio loss	Determines the length of time during which audio is not detected before triggering failover. Periods of silence may be normal. Consider the source content when setting thresholds.
  Black screen	Determines the length of time that black frames may be sent by a Live Slicer before triggering failover. Periods of black video may be normal. Consider the source content when setting thresholds. This metric measures the duration of black video by averaging the video's luminosity percentage over the last few seconds.
  CC last seen	Determines the amount of time that a Live Slicer may not receive closed captioning data before triggering failover. Key information: This metric is reported in 10 second intervals. This metric is inapplicable when closed captioning data has not been received from the Live Slicer. Consider the content when setting the threshold for this metric. Some content may contain an extended time period without closed captioning data.
  Dropped frames	Determines how many frames may be dropped within the current reporting period before triggering failover.
  Input loss	Determines the length of time that the system may not receive the Live Slicer's signal before triggering failover.
  Nielsen last seen	Determines the length of time that may elapse since the Live Slicer last received a Nielsen watermark before triggering failover. Key information: Nielsen watermarks may be inserted into the audio stream sent to the Live Slicer. They are typically inserted at 10 second intervals. However, this interval may vary according to your implementation. Nielsen watermarks are leveraged by the Live Slicer to generate ID3 tags through which a media player reports viewership data.
  Processing queue	Determines how many packets may be queued to be read by the Live Slicer before triggering failover. Key information: The recommended levels for this metric varies according to the signal type. For example, a UDP Unicast stream should have a much lower threshold (e.g., 1,000) than a UDP Multicast stream (e.g., 10,000). A high value may be indicative of insufficient CPU resources on the computer hosting the Live Slicer. A transient spike in this metric may not be cause for concern.
  SCTE last seen	Determines the length of time that may elapse since the Live Slicer last received a SCTE 35/104 signal before triggering failover.
  Static audio	Determines the length of time during which static audio is detected before triggering failover. Static audio is detected by analyzing the audio's loudness percentage over the last few seconds. This metric ignores periods of silence.
  Static video	Determines the length of time during which static video (e.g., green screen, color bars, or a frozen frame) is detected before triggering failover. Static video is detected by analyzing the video's average luminosity percentage over the last few seconds.
  TR 101 290 P1 errors	Requires a UDP source and Live Slicer version 22083100 or higher Determines the number of first priority errors that may occur before triggering failover. A first priority error occurs when a Digital Video Broadcasting (DVB) measurement test identifies an issue that may prevent the MPEG-2 Transport Stream (TS) from being decoded. The parameters that are evaluated for this test are defined within an ETSI technical report (ETSI TR 101 290).
  TR 101 290 P2 errors	Requires a UDP source and Live Slicer version 22083100 or higher Determines the number of second priority errors that may occur before triggering failover. A second priority error occurs when a Digital Video Broadcasting (DVB) measurement test identifies an issue with a parameter that should be continuously monitored. The parameters that are evaluated for this test are defined within an ETSI technical report (ETSI TR 101 290).
  Upload queue	Determines how many slices may be queued for upload before triggering failover. A value higher than 2 may be indicative of Live Slicer connectivity issues.
  Video loss	Determines the length of time during which video is not detected before triggering failover. Loss of video is assessed according to whether a video packet identifier (PID) is detected in the live stream.

• Minimize slate upon Live Slicer failover by setting at least one backup Live Slicer to hot mode. The available modes are described below.

  • Hot-Warm Mode: This mode prioritizes cost saving over slate by only allowing the primary Live Slicer to encode and store content. Although backup Live Slicers may slice content (i.e, Slicing state), they will not upload that content to our system. As a result, this mode may cause a few seconds of slate upon Live Slicer failover.

  • # Hot Backup(s): This mode balances the viewer's experience with cost saving by allowing one or more backup Live Slicers within a failover group to encode and store content. Using this mode allows near instantaneous failover to the designated number of hot backup Live Slicers. However, a viewer may experience a slight amount of discontinuity since those Live Slicers are not synched.

    If your failover group contains additional backup Live Slicers, then they will operate in warm mode. This means that they are allowed to slice content (i.e, Slicing state). However, they are not allowed to upload that content to our system while in this mode.

    Failover workflow:

    1. Upon detecting an issue with the current Live Slicer, our service switches to a hot backup Live Slicer.
    2. If it detects at least one warm backup Live Slicer, then it will instruct it to turn hot. This process may take up to 45 seconds.
  • Hot-Hot Mode: This mode prioritizes the viewer's experience over cost saving by allowing all Live Slicers within a failover group to encode and store content. Using this mode allows near instantaneous failover. However, a viewer may experience a slight amount of discontinuity since those Live Slicers are not synched.

• Failover mode determines how our system chooses which Live Slicer will serve as a source for your live stream. Each failover mode is described below.

  • Prioritized: This mode allows you to define a failover order for your Live Slicers. The source for your live stream is the Live Slicer with the highest priority, provided that it is healthy. If it is unhealthy, then our system will use the Live Slicer with the next highest priority.

    The Auto Failback option determines whether to switch back to a higher priority Live Slicer when it resumes a healthy state.

  • Flat: The source for your live stream is a healthy Live Slicer that is randomly selected from your failover group.

• You may temporarily prevent our system from failing over to a specific Live Slicer by disabling its availability status.

  A Live Slicer's availability status only affects failover. It does not prevent the Live Slicer from slicing and encoding your content.

• View a failover group's audit log by opening it and then clicking the Change Log tab.
• Review a failover group's events by opening it and then clicking on the Event Log tab.

To set up Live Slicer failover for one or more live channel(s)

1. Navigate to the Slicers page. How?From the main menu, navigate to Slicers.
2. Click + Create Failover Group.
3. From the Failover Group Name option, type the name that will be assigned to the failover group.
4. Click Create & Edit.

5. From the Hot-Warm Mode option, determine whether backup live slicers will run in hot or warm mode.

   The recommended mode is 1 Hot Backups. This mode ensures an optimal viewing experience by allowing our service to quickly switch your video feed to a hot backup Live Slicer when your primary Live Slicer experiences issues.

6. From the Failover Mode option, select whether Live Slicer selection on failover will be determined by priority or randomly.

7. Identify the set of Live Slicers that will be added to this failover group. Segregate them into the following categories:

   • Assigned: From the list of Live Slicers that will be added to this failover group, identify each Live Slicer that is already associated with a live channel.
   • Unassigned: All other Live Slicers are considered unassigned.

8. Add the desired unassigned Live Slicers to this failover group.

   It is important to add Live Slicers to a failover group in 2 stages. Add unassigned Live Slicers in the first stage. Once the live channel has failed over to a Live Slicer within your failover group, you should add the Live Slicer that was previously assigned to the live channel. The purpose of this 2-step approach is to avoid streaming slate to your live viewers.

   1. If one or more of the desired Live Slicer(s) have never been previously used, then you should start those Live Slicer(s) now.
   2. Click Add Slicers to Group.
   3. From the Slicers option, select the desired Live Slicers and then click .
   4. Click Update Slicer Mapping(s). The selected Live Slicers will be listed within the Slicers in Group section.
   5. From the Available column, enable each desired Live Slicer by toggling its availability status to enabled ().

   6. Prioritized Failover Mode

      Order the Live Slicers in this failover group from highest to lowest priority. Drag a Live Slicer to the desired position by hovering over it and then dragging its icon.

9. Add the desired live channel(s) to this failover group.

   1. Click Add Channel Mapping.
   2. From the Channels option, select the desired live channel(s) and then click .
   3. Click Update Channel Mapping(s). The selected live channels will be listed within the Channels using Group section.

10. Determine the conditions under which a Live Slicer is considered healthy by performing the following steps:

    1. Click the Thresholds tab.

    2. Determine which metrics will trigger failover and set their failover and recovery thresholds.

       1. Determine whether a metric will trigger failover by toggling its status between enabled () and disabled ().

       2. For each enabled metric, define the conditions that will trigger failover. Metric configuration varies according to whether a metric toggles between two states or if it performs measurements.

          • States

            1. Configure the Set Duration option to the number of seconds that the sub-optimal state (e.g., no signal, no closed captions detected, or black video was detected) must persist before triggering failover.
            2. Configure the Clear Duration option to the number of seconds that the optimal state must persist before the Live Slicer will be considered healthy.

          • Measurements

            1. Configure the Set Threshold option to a threshold value that identifies when a Live Slicer is performing sub-optimally.
            2. Configure the Set Duration option to the number of seconds that a metric must fall below its threshold before triggering failover.
            3. Configure the Clear Threshold option to the number of seconds that a metric must meet or exceed its threshold before the Live Slicer will be considered healthy.
11. From the Configuration tab, copy the failover group's system-defined ID which is displayed under the Failover Group ID label. You will use this ID when defining a Live Slicer's failover_id setting.
12. Click Save.

13. Perform the following steps for each Live Slicer associated with this failover group:

    1. Open its Live Slicer configuration file.

    2. Set the failover_id setting to the failover group ID copied in step 11.

       failover_id: 1232b8646dea4cd0a48f5e0ffaa4f8c7

    3. Add one of the following settings:

       • Live Slicer version 21071400 or higher

         Insert the enable_remote_config setting after the failover_id setting.

         enable_remote_config: 1

       • Live Slicer version 21070801 or lower

         Insert the remote setting after the failover_id setting.

         remote:

    4. Review your Live Slicer configuration file. It should now look similar to the following example:

       description: Live Capture

       username: joe@example.com

       apikey: abcDEFghiJKLmnoPQRtuvWXYz123ABCdefGHIJKL

       slicerID: slicer1234

       failover_id: 1232b8646dea4cd0a48f5e0ffaa4f8c7

       enable_remote_config: 1

       ...

    5. Save the Live Slicer configuration file.
    6. Restart the Live Slicer.

14. Wait until the live channel fails over to a Live Slicer in your failover group. This may take a few minutes.

    Applying a failover group to a live channel allows our system to determine which Live Slicer should be assigned to it. Upon failover, our system will disassociate the Live Slicer currently assigned to the live channel. You should add this Live Slicer to your failover group as indicated in the next step.

15. Add the Live Slicer that was previously assigned to your live channel to the failover group. Repeat this step for each live channel associated with the failover group.

    If the failover group uses prioritized failover mode, then you should ensure that these Live Slicer(s) are prioritized over other Live Slicers in your failover group.

16. Perform step 13 for the Live Slicer(s) added in the previous step.

    If the failover group uses prioritized failover mode and these Live Slicers have been prioritized, then the system will failover to the Live Slicer with the highest priority within a few minutes.

To modify a failover group

1. Navigate to the Slicers page. How?From the main menu, navigate to Slicers.
2. Click on the desired failover group.

3. Perform one or more of the following changes:

   • From the Failover Group Name option, set the name that will be assigned to the failover group.
   • From the Hot-Warm Mode option, determine whether backup live slicers will run in hot or warm mode.
   • From the Failover Mode option, select whether Live Slicer selection on failover will be determined by priority or randomly.
   • Add Live Slicers to this failover group. After which, for each Live Slicer added to the failover group, you will need to add the failover_id and the enable_remote_config settings to its Live Slicer configuration file and then restart it.

   • Determine whether the system may failover to a Live Slicer by toggling its availability status between enabled () and disabled ().

     It may take a minute or two before newly enabled Live Slicers are available for failover.

   • Remove Live Slicers from this failover group.

     1. From the Slicers in Group section, mark each Live Slicer that will be removed from the failover group.
     2. Click Remove.

   • Add live channel(s) to this failover group.

     Reassign a Live Slicer to a different failover group by first removing it from its current failover group.

     1. Click Add Channel Mapping.
     2. From the Channels option, select the desired live channel(s) and then click .
     3. Click Update Channel Mapping(s). The selected live channels will be listed within the Channels using Group section.

   • Remove live channels from this failover group.

     1. From the Channels using Group section, mark each live channel that will be removed from the failover group.
     2. Click Remove.
   • Determine the conditions under which a Live Slicer is considered healthy.
4. Click Save.
5. Restart all of the Live Slicers associated with this failover group if you have updated the conditions (i.e., thresholds) under which a Live Slicer is considered healthy.

To delete a failover group

1. Navigate to the Slicers page. How?From the main menu, navigate to Slicers.
2. Mark each failover group that will be deleted.
3. Click Delete # Failover Group(s).
4. When prompted, confirm the deletion by clicking Yes, Delete.

Failover Notifications

Publish failover events through the following workflow:

1. Our service pushes data to Amazon SNS whenever we fail over to another Live Slicer.
2. Amazon SNS broadcasts data to one or more destination(s) (e.g., mobile device, web server, or Slack).

Get started with Amazon SNS for free through its SNS free tier.
Learn more.

Our service formats data using JSON. This data may then be filtered via custom code. This article explains how to strip out additional data generated by Amazon SNS via a custom function in Amazon Lambda.

Getting Started with Failover Notifications

Perform the following steps to set up notifications:

1. Set up an Amazon SNS topic.

   Our service pushes Live Slicer health and failover notifications to the same Amazon SNS topic. You may skip this step if you have already created an Amazon SNS topic for Live Slicer health notifications.

2. Configure communication between our service and Amazon SNS.

   Our service pushes Live Slicer health and failover notifications to the same Amazon SNS topic. This means that updating the SNS topic for either Live Slicer health or failover notifications will affect both types of notifications.

   1. Navigate to the Failover page. How?From the main menu, navigate to Slicers and then select Failover from the side navigation bar.

   2. Click Update SNS Topic from the right-hand pane.
   3. Set the Update your SNS Topic ARN option to the ARN for the topic created above.
   4. Click Save Topic ARN.

3. Configure Amazon SNS to broadcast notifications to the desired destination(s).

   Learn how to set up Amazon SNS and Lambda to broadcast notifications to a Slack channel.

Key Failover Notification Fields

Our service sends information that describes a failover event in JSON format. Key parameters in this notification are described below.