Netflix Open Connect

Fill, Updates, and Maintenance

Each Open Connect Appliance (OCA) stores a portion of the Netflix catalog, which in general is less than the complete content library for a given region. Popularity changes, new titles that are added to the service, re-encoded movies, and routine software enhancements are all part of the nightly updates, or fill, that each appliance must download to remain current.

Fill windows

By design, OCAs follow a “push fill” methodology. They fill every day during a window of time that corresponds to your off-peak hours. The timing of the fill window is defined in partnership with your network planning team.

The goal is to set the fill window such that:

  • It occurs during the trough of your Netflix traffic
  • It does not disrupt your inbound traffic peaks

Setting the fill window this way allows the system to continue to download updates and serve clients without negatively impacting playback performance.

Requirements for the fill window:

SettingValue
Length of time12 hours
Timeframe2AM to 2PM (local time) (can be shifted +/- 2 hours)
BandwidthMaximum rate of 1.2 Gbps per system
Total trafficUp to 6.5 TB

The system might complete filling before the end of the fill window, and it might fill more than once if new content becomes available before the fill window ends.

Fill patterns

In general, appliances determine where to receive fill using selection criteria that is similar to the steering criteria that is used by Netflix client devices. The following scenarios provide more detail.

If you have specific fill requirements, discuss them with your partner engagement manager during the initial site design process. If you require changes to your appliance's fill behavior post-deployment, contact the Open Connect operations team.

Peer filling

OCAs will attempt to peer fill from each other in either of these two cases:

  • If they are clustered
  • If they are on the same subnet

For example, assume that in the following diagram, OCA #1 and OCA #2 are in a cluster. If OCA #1 downloads a movie via a regular cache fill, OCA #2 will then seek to download the movie from OCA #1 instead of via transit or Open Connect peering. Routes to the OCAs must be advertised over Open Connect peering to ensure that the peering session is used for fill traffic.

image showing one OCA doing a cache fill and another OCA on the same cluster filling from the first one

If OCAs on multiple sites are configured within the same cluster, there is a strong chance that they will not be numbered within the same subnet. However, even if the cluster spans multiple sites, the “same-cluster” rule still applies - therefore the OCAs will attempt to peer fill as described.

Tier filling

Peer filling is the most efficient method, however as a second-best option tier filling is considered. OCAs that can see each other’s IP address in their BGP feed (but are not in the same cluster or subnet) will consider filling from each other rather than via a regular cache fill. To enable tier filling if you have separate clusters that are located in two different sites, ensure that the appliances within one cluster can hear the subnets from the other cluster via the BGP connection that is established with your router.

image showing OCAs not on the same cluster tier filling from each other

Out-of-cycle fills

There will be cases that cause an urgent need to deploy or update a small subset of titles on the OCAs outside of the normal fill window. This need can arise for various reasons, for example:

  • If re-encoding of a popular title is required to repair viewing issues
  • To make a new, “day-of-broadcast”, or “day-after-broadcast” title available

When these out-of-cycle fills occur, you will see a small portion of fill traffic for a limited period of time outside of the configured fill window. Doing an out-of-cycle fill instead of waiting until the normal fill window helps to ensure that popular titles can continue to be served from your OCAs (the preferred method) and not via peering or transit. This approach improves the overall efficiency and offload potential of the system. Because out-of-cycle fills are limited to very specific popular titles, they do not negatively impact overall server efficiency - rather, they increase the local offload of popular content.

Automatic software updates

Netflix manages the OCA software, and we will periodically update or enhance the software. All non-emergency software updates are performed during the configured fill window, and you will be notified before and after they occur. All appliance software, including the operating system, web server, and control code, is updated as a single image. If a new image fails to properly load, the appliance will automatically fall back to the previous image. In either case, you will receive an email to let you know the status of the upgrade after it is attempted.

Health monitoring

As part of the Open Connect program, Netflix continuously monitors all deployed Netflix OCAs, including all aspects of the performance and availability of the appliance. In addition, you can monitor the basic health of the OCAs that are deployed on your network by using the Netflix Partner Portal at https://my.oc.netflix.com. You will be invited to create an account in the Partner Portal as part of the OCA installation and turn-up process.

Unplanned Outages or Service-Impacting Events

For service-wide outages with broad impact to the service of multiple OCA partners, partners might receive an initial notification, status updates on a regular basis if the outage is prolonged, and a final resolution notification. If you want to filter these announcements, the sender address is: cdn-noc@mailer.netflix.com.

For per-ISP partner incidents, including routing and connectivity anomalies that persist over a significant period of time, the Netflix operations team will be alerted by our internal monitoring tools. We will resolve any issues under our control as soon as reasonably possible. Depending on the severity and impact to traffic, we might ask relevant ISP partners for additional comments on the incident.

Potential Service-impacting Maintenance

Whenever possible, for any planned maintenance or activity that might adversely impact an OCA or the services that it provides, Netflix will communicate according to the following schedule:

  1. Netflix will send a “Scheduled Maintenance” email notification to the partner NOC contact e-mail address in our system two (2) business days prior to the activity.
  2. Netflix will send a “Confirm Maintenance” email two hours (02:00) before the start of the planned maintenance activity, to confirm that the maintenance will be performed as expected.
  3. Netflix will send a “Maintenance Completed” email when the maintenance activity is completed, to signal the return to a normal operational state.

For emergency service activities that do not permit our standard advance period, we will notify as early as possible.

During scheduled and emergency Netflix maintenance windows, ISP partners might see their BGP session flap, notice a link-state change with their cache-connected interfaces, and see traffic served from the appliance decrease or go to zero. All of these events are expected.

Under normal conditions, software updates that are not service-impacting will occur during the fill window at a time of low traffic, thereby minimizing potential user impact.

Partner Maintenance

The only OCA components that Netflix will request partners to replace in the field are:

  • Power supplies
  • SFP+ optics

We will ship these components to you in case of failure. Netflix will not request any ISP partner to install or replace any internal OCA hardware components such as memory, hard drives, or fans. Note that OCAs are designed to operate in data center environments with high-capacity fans that are required to cool the dense disk and memory installed. You might notice loud fan noise from OCAs, which is normal.

For ISP partners who are engaging in maintenance that might impact Netflix OCA availability, please observe the following guidelines:

  • Plan your maintenance or downtime to occur during the cache fill window in order to minimize service disruption to our shared subscribers.
  • Plan to drain appliances of traffic 2 to 3 hours in advance of the maintenance. This will serve to direct any new Netflix viewing activity to an alternate source, and sessions that are currently associated with the appliance should finish by the start of the maintenance period. You can disable and drain appliances from the Open Connect Partner Portal following these instructions: Draining and disabling appliances
  • After completion of maintenance, please ensure that the OCAs are re-enabled to serve traffic. This will redirect new session starts away from the alternate sources.

Appliance RMA guidelines

OCAs are designed for resiliency, including the ability to have multiple hard drives fail without a noticeable impact on delivery. At times, however, hardware performance might degrade to a point where replacement of the entire OCA is necessary to maintain the ability to offload traffic in a meaningful manner.

In general, RMA requests are initiated by the Netflix NOC team, who will reach out to ISP partners via email and request for them to fill out a short online form to start the RMA process.

Our standard process is to ship a replacement pre-loaded OCA within three business days. The partner returns the impacted hardware in the same shipping box, and Netflix pays shipping both ways, at no shipping cost to the partner. In some instances, we might need additional time in order to fully pre-load content for specific geographic regions. The total time until shipment from our US facility is likely to be 10 US business days at most, however customs clearance can vary according to region and country and might delay final receipt.

See the Operating Level Agreement for a PDF version of the maintenance information in this section.


Back to top

© 2016, 2017 Netflix, Inc. All rights reserved.