Each Open Connect Appliance (OCA) stores a portion of the Netflix catalog, which in general is less than the complete content library for a given region. Popularity changes, new titles that are added to the service, re-encoded movies, and routine software enhancements are all part of the nightly updates, or fill, that each appliance must download to remain current.
By design, OCAs follow a “push fill” methodology. They fill every day during a window of time that corresponds to your off-peak hours. The timing of the fill window is defined in partnership with your network planning team.
The goal is to set the fill window such that:
- It occurs during the trough of your Netflix traffic
- It does not disrupt your inbound traffic peaks
Setting the fill window this way allows the system to continue to download updates and serve clients without negatively impacting playback performance.
Requirements for the fill window, by hardware type:
|Length of time||12 hours||12 hours||3 hours|
|Timeframe||2AM to 2PM (local time) (can be shifted +/- 2 hours)||2AM to 2PM (local time) (can be shifted +/- 2 hours)||3AM to 6AM (local time) or 6AM to 9AM (local time)|
|Bandwidth||Maximum rate of 1.2 Gbps per system||Maximum rate of 0.4 Gbps per system||Maximum rate of 1.2 Gbps per system|
|Total traffic during fill window||Up to 6.5 TB||Up to 2.1 TB||Up to 1.5 TB|
The system might complete filling before the end of the fill window, and it might fill more than once if new content becomes available before the fill window ends.
If flash-based appliances are part of your network, they will be configured to fill as much as possible from the storage appliances that are on site. Because they are SSD-based, they are configured with a shorter staggered fill window, and they will not serve while they are filling. Because the flash-based appliances do not serve while they are filling, all Netflix traffic is served by the storage appliances on site during their fill period.
In general, appliances determine where to receive fill using selection criteria that is similar to the steering criteria that is used by Netflix client devices. The following scenarios provide more detail.
If you have specific fill requirements, discuss them with your partner engagement manager during the initial site design process. If you require changes to your appliance's fill behavior post-deployment, contact the Open Connect operations team.
OCAs will attempt to peer fill from each other in either of these two cases:
- If they are clustered
- If they are on the same subnet
For example, assume that in the following diagram, OCA #1 and OCA #2 are in a cluster. If OCA #1 downloads a movie via a regular cache fill, OCA #2 will then seek to download the movie from OCA #1 instead of via transit or Open Connect peering. Routes to the OCAs must be advertised over Open Connect peering to ensure that the peering session is used for fill traffic.
If OCAs on multiple sites are configured within the same cluster, there is a strong chance that they will not be numbered within the same subnet. However, even if the cluster spans multiple sites, the “same-cluster” rule still applies - therefore the OCAs will attempt to peer fill as described.
Peer filling is the most efficient method, however as a second-best option tier filling is considered. OCAs that can see each other’s IP address in their BGP feed (but are not in the same cluster or subnet) will consider filling from each other rather than via a regular cache fill. To enable tier filling if you have separate clusters that are located in two different sites, ensure that the appliances within one cluster can hear the subnets from the other cluster via the BGP connection that is established with your router.
There will be cases that cause an urgent need to deploy or update a small subset of titles on the OCAs outside of the normal fill window. This need can arise for various reasons, for example:
- If re-encoding of a popular title is required to repair viewing issues
- To make a new, “day-of-broadcast”, or “day-after-broadcast” title available
When these out-of-cycle fills occur, you will see a small portion of fill traffic for a limited period of time outside of the configured fill window. Doing an out-of-cycle fill instead of waiting until the normal fill window helps to ensure that popular titles can continue to be served from your OCAs (the preferred method) and not via peering or transit. This approach improves the overall efficiency and offload potential of the system. Because out-of-cycle fills are limited to very specific popular titles, they do not negatively impact overall server efficiency - rather, they increase the local offload of popular content.
Automatic software updates
Netflix manages the OCA software, and we will periodically update or enhance the software. All non-emergency software updates are performed during the configured fill window, and you will be notified before and after they occur. All appliance software, including the operating system, web server, and control code, is updated as a single image. If a new image fails to properly load, the appliance will automatically fall back to the previous image. In either case, you will receive an email to let you know the status of the upgrade after it is attempted.
As part of the Open Connect program, Netflix continuously monitors all deployed Netflix OCAs, including all aspects of the performance and availability of the appliance. In addition, you can monitor the basic health of the OCAs that are deployed on your network by using the Netflix Partner Portal at https://my.oc.netflix.com. You will be invited to create an account in the Partner Portal as part of the OCA installation and turn-up process.
Unplanned Outages or Service-Impacting Events
For service-wide outages with broad impact to the service of multiple OCA partners, partners might receive an initial notification, status updates on a regular basis if the outage is prolonged, and a final resolution notification. If you want to filter these announcements, the sender address is: firstname.lastname@example.org.
For per-ISP partner incidents, including routing and connectivity anomalies that persist over a significant period of time, the Netflix operations team will be alerted by our internal monitoring tools. We will resolve any issues under our control as soon as reasonably possible. Depending on the severity and impact to traffic, we might ask relevant ISP partners for additional comments on the incident.
Potential Service-impacting Maintenance
Whenever possible, for any planned maintenance or activity that might adversely impact an OCA or the services that it provides, Netflix will communicate according to the following schedule:
- Netflix will send a “Scheduled Maintenance” email notification to the partner NOC contact e-mail address in our system two (2) business days prior to the activity.
- Netflix will send a “Confirm Maintenance” email two hours (02:00) before the start of the planned maintenance activity, to confirm that the maintenance will be performed as expected.
- Netflix will send a “Maintenance Completed” email when the maintenance activity is completed, to signal the return to a normal operational state.
For emergency service activities that do not permit our standard advance period, we will notify as early as possible.
During scheduled and emergency Netflix maintenance windows, ISP partners might see their BGP session flap, notice a link-state change with their cache-connected interfaces, and see traffic served from the appliance decrease or go to zero. All of these events are expected.
Under normal conditions, software updates that are not service-impacting will occur during the fill window at a time of low traffic, thereby minimizing potential user impact.
The only OCA components that Netflix will request partners to replace in the field are:
- Power supplies
- SFP+ optics
We will ship these components to you in case of failure. Netflix will not request any ISP partner to install or replace any internal OCA hardware components such as memory, hard drives, or fans. Note that OCAs are designed to operate in data center environments with high-capacity fans that are required to cool the dense disk and memory installed. You might notice loud fan noise from OCAs, which is normal.
For ISP partners who are engaging in maintenance that might impact Netflix OCA availability, please observe the following guidelines:
- Plan your maintenance or downtime to occur during the cache fill window in order to minimize service disruption to our shared subscribers.
- If your maintenance has the potential to impact Netflix customer service, notify the Netflix team by opening a maintenance ticket via the Partner Portal.
- Plan to drain appliances of traffic 2 to 3 hours in advance of the maintenance. This will serve to direct any new Netflix viewing activity to an alternate source, and sessions that are currently associated with the appliance should finish by the start of the maintenance period. You can disable and drain appliances from the Open Connect Partner Portal following these instructions: Draining and disabling appliances
- After completion of maintenance, please ensure that the OCAs are re-enabled to serve traffic. This will redirect new session starts away from the alternate sources.
- If you intend to make any changes to a site that contains multiple OCAs in a cluster, you must notify the Open Connect team so that the cluster can be reconfigured properly. For more details, see the information about clustering architectures.
See the Operating Level Agreement for a PDF version of the maintenance information in this section.
Back to top
© 2016 - 2019 Netflix, Inc. All rights reserved.