Netflix Open Connect

Network Configuration

This section describes how Open Connect Appliances are typically configured in a network. If you are an Open Connect ISP partner, Netflix works closely with you to determine the optimal configuration for your particular needs.

For more information, see the FAQs

Overview

OCAs are directed cache appliances, meaning that the manner in which traffic is directed to the appliance is determined explicitly by you and by Netflix, not by the appliance itself.

An OCA only serves clients at IP addresses that you advertise to the OCA via a BGP session. In other words, traffic is only delivered from your embedded OCAs to the customer prefixes that you explicitly announce to them, as described in the following sections. Therefore, you as the ISP partner have full control over the networks that the appliances will serve. BGP sessions are established between appliance(s) and the closest connected router.

If content is requested that is not contained on an embedded OCA, the client request is directed to the closest Netflix content site via peering (if present) or via transit.

Reconfiguring the IP address of an OCA

Each appliance comes fully configured based on the IP address details that you provided to Netflix in your site survey before it was shipped.

For all appliances, the IP address can be updated via a keyboard and monitor. Interfaces are on the front of the chassis, but might be hidden behind a panel. The updated IP address will only take effect after a reboot, so it is import to drain the appliance two hours before the change by shutting down its BGP session (or sessions) to ensure that there is no traffic being served by the appliance.

Router interface configuration

When you are connecting the appliances to your router, follow these guidelines:

  • Only one usable IPv4 address per appliance is required. IPv6 is also supported, but optional.
  • You can assign an address to the appliance from an IPv4 subnet of /31 and larger, or an IPv6 subnet of /126 and larger.
  • It is acceptable to assign an address to the appliance from a larger subnet (for example, a /24). However, because only one IPv4 address is required per appliance, a smaller subnet is typically used.
  • The router interfaces must be configured for Link Aggregation Group (LAG) with LACP. Even if you are connecting only one port to the router, that single router interface must be configured for LAG.
  • A standard maximum transmission unit (MTU) must be configured on each router interface. Do not use jumbo frames.
  • If there are multiple routers available that can provide redundancy in a site, it is recommended to stagger appliances between routers. Appliances on the same router should be in the same subnet to optimize filling. Appliances on separate routers should be in separate subnets. Appliances are not designed to be connected to two separate routers.
  • Each OCA is hardened against network attack and is designed to be directly connected to the internet. Filtering inbound or outbound traffic can cause operational issues, so we strongly recommend that you allow all traffic on all ports, do not use ACLs, and ensure that your router has a default route or full routing table. If you absolutely must filter, the current list of inbound and outbound usage follows. Please note that these can change at any time without prior notification.
    • Outbound: Allow all destination addresses and ports.
    • Inbound: Allow TCP 22, 80, 179, 443, UDP 123 (source and destination), ICMP types 0, 3, 8, 11, and all ICMPv6 from any public IP/port. Allow all return traffic from any appliance-initiated connection.
  • Each network interface must be receiving between 0 dBm and -10 dBm of light to ensure good data throughput. The LCD panel on the front of the appliance displays the current light levels for each interface. If your appliance does not have an LCD panel, access the console with a keyboard and mouse, then follow the console instructions to check light levels. If light levels are out of the acceptable range, clean the optics. If cleaning the optics does not bring them into the acceptable range, contact Netflix to have new optics shipped to you.

Routing and content steering via BGP

We steer clients to our OCAs based on an ISP’s BGP advertisements, coupled with the routing and steering algorithms in the Open Connect control plane. ISP partners can control some aspects of content steering via the BGP routes that are announced to each OCA.

The control plane steers requests from end user clients to the best available appliance based on multiple factors. Assuming that the appliance has the requested title and available serving capacity, the control plane provides clients with a ranked list of appliances (typically 3 or more reliable sources) to stream from.

Appliance selection criteria

The following appliance selection criteria are considered, in order, by the Open Connect control plane services. If there is a tie for a given criterion, then the next criterion is considered. If there is a tie on all criteria, traffic is balanced between appliances.

  1. The appliance that receives the most-specific route to the client’s prefix.
  2. The appliance that receives the route to the client’s netblock with the shortest AS path.
  3. The appliance that receives the route to the client’s netblock with the lowest multi-exit discriminator (MED). (See the notes on MEDs below).
  4. The geographically closest appliance. We geolocate based on client IPs, whose location is then compared to the latitude and longitude of nearby OCAs to determine the closest available system.

BGP notes
  • Prefixes for Open Connect embedded appliances:
    • IPv4 prefixes between /8 and /31 (inclusive) are accepted.
    • IPv6 prefixes between /19 and /64 (inclusive) are accepted.
  • Prefixes for Open Connect peering sessions:
    • IPv4 prefixes between /8 and /24 (inclusive) are accepted.
    • IPv6 prefixes between /19 and /48 (inclusive) are accepted.
  • As an implicit requirement, all appliances must have a BGP session configured in order to correctly participate in Netflix content steering and delivery.
  • Advertised routes that are received by an OCA are synchronized with Open Connect control plane services approximately every five minutes.
  • To localize traffic, the best practice is to advertise the most specific routes to the appliance. For example, if you are announcing a /22 to the OCA, but a /24 is received from the same block over settlement-free interconnection (SFI) peering or transit, the /24 will be preferred, delivering content traffic from the remote source instead of the OCA.
  • If you are deploying only one OCA in your network, you should advertise the most specific (shortest) prefix for that OCA over the peering session that you want the OCA to use for nightly filling purposes.
  • If you are deploying multiple OCAs in your network, see the additional information about clustering architectures.
  • MEDs can be injected and will be respected. You can use MEDs to de-preference routes between appliances or otherwise distribute traffic to meet your requirements. See additional notes below.
  • Netflix does not use any BGP community information that is advertised by partners to OCAs or via Open Connect peering.
Additional notes on MEDs
  • Important: Marking MEDs on already installed and working Open Connect Appliances can be hazardous, because it must be done on all BGP sessions for all appliances at the same time.
  • There is no cap on the maximum MED value.
  • A missing MED is treated the same as a MED of 0, and indicates that the appliance should receive all servable traffic for the associated prefixes (also often referred to as MED-missing-as-best). Remember, if multiple appliances receive the same prefix with the same metric, traffic is load-balanced across those appliances. Because a missing MED will be equivalent to 0, it is preferred over any >0 MED on other appliances.
  • For information about MEDs with respect to peering only, see Peering Locations.

Clustering architectures

Two or more OCAs that are intended to serve the same set of customers can be configured by the Open Connect operations team as a manifest cluster. OCAs in a manifest cluster share content storage and function together as one logical server/storage unit. The Netflix team collaborates with you to determine whether clustering is warranted and how to set up optimal clusters, depending on your particular site and network configurations.

Clustering has the following potential benefits:

  • Greater offload for unique content

    In a typical two-OCA cluster, both appliances will use approximately 40% of their storage for the same popular content. This popular content typically represents roughly 60% of the OCA’s total offload. The remaining 60% of storage space on each OCA is used to store less frequently-accessed content. The collection of less frequently-accessed content is unique on each OCA, therefore a cluster of OCAs provides greater total offload than an unclustered group of OCAs.

  • Better resiliency

    Redundancy is generally acceptable in a two-OCA cluster. In the event of a single OCA failure, the healthy appliance will take over the majority of the traffic that the failed unit was serving. See the failover scenarios in the sample architectures.

Notes:

  • Appliances intended to serve the same set of customers can be clustered if they are located at the same site, or if they are in close geographical or network proximity.
  • Appliances in a cluster must receive the exact same BGP route advertisements.
  • Appliances cannot be clustered if they are not intended to serve the same set of customers.
  • To enable efficient nightly fill: If you have separate clusters that are located in two different sites, ensure that the appliances within one cluster can hear the subnets for the appliances in the other cluster via the BGP connection that is established with your router. See the Fill and updates information for more details.

Embedded OCAs combined with peering sessions

The ideal Open Connect implementation is a mixture of both SFI peering and deployed embedded OCAs. Netflix uses two separate autonomous systems for peering:

  • AS2906 is the AS number that Netflix uses for peering at its PoPs
  • AS40027 is the AS number that embedded OCAs use to peer with ISP networks

See BGP notes for prefix announcements that are accepted on peering sessions.

The same prefix announced both to a private or public peering session (using AS2906) and to an OCA (using AS40027) will always be preferred on the appliance over peering, because the Open Connect control plane will have two BGP entries for that prefix:

  • one with an AS PATH LENGTH of 1 (<AS_NUMBER>) from the appliance itself
  • one with an AS PATH LENGTH of 2 (2906 <AS_NUMBER>) from an IX location

When OCAs and Open Connect SFI peering is combined, OCAs are nominal and peering is used primarily for backup, for filling, and for serving long-tail titles.

Offload appliances

If you are an ISP with very large amounts of Netflix traffic, we will likely include offload appliances in your OCA deployment architecture. Offload appliances are 1U flash storage-based servers that are deployed when you reach a threshold number of OCAs, to augment the delivery capability of the main (storage) appliances.

If offload appliances are part of your network, they will be configured to fill as much as possible from the storage appliances that are on site. Because they are SSD-based, they will be configured such that their fill window is 3 Hours, and they will not serve while they are filling. Because the offload appliances do not serve while they are filling, all Netflix traffic is served by the storage appliances on site during their fill period.

Offload appliances are not clustered unless they are in the same site, and they should not be set up in the same manifest cluster as the main storage appliances.

Rail kit instructions for offload appliances can be accessed online:

http://openconnect.netflix.com/railkit-1.pdf

Note: Offload appliances, unlike storage appliances, have their ports on the back of the device.


Back to top

© 2016, 2017 Netflix, Inc. All rights reserved.