When launching a container that
requested a routable IP address, Titus
attaches an AWS ENI (Elastic Network
Interface) 2 to the agent VM that is running the container. Attaching an ENI
creates a new network interface on the
VM from which multiple IP addresses
can be assigned. These addresses are
allocated from the same classless in-ter-domain routing (CIDR) range as
VMs in the VPC, meaning containers
and VMs can directly address each
other’s IPs. Allocating IPs via ENIs
lets Titus avoid managing the available VPC IPs or directly modifying
VPC routing tables.
When Titus is preparing to launch
a new container, it creates a network
namespace for it, assigns it a specific
IP address from an ENI, and connects
the container’s network namespace
to the host’s using a veth (virtual Eth-ernet) interface. Routing rules on the
host route all traffic for that IP to the
veth interface, and routing rules within
the network namespace configure the
allocated IP for the container.
Another benefit of containers sharing the same VPC network as VMs is
that they use common network security
policies, such as AWS Security Groups
that provide virtual firewalls. 1 Each ENI
can be configured to use a set of SG
firewall rules that apply to any traffic
coming in or out of the interface. Titus
applies a container’s requested SGs to
the ENI with which that container is associated, allowing enforcement of SG
firewall rules to its traffic.
The number of ENIs that can be
attached to a VM is limited and potentially less than the number of containers that Titus could assign to it.
To use ENIs more efficiently, Titus allows containers to share the same ENI.
This sharing, however, is possible only
when containers use the same security
group configurations, as SGs can be
configured only for the entire ENI. In
this case, each container would have
a unique IP address, with common
firewall rules being applied to their
traffic. An example of sharing an ENI
is shown in Figure 2. In this example, each of the three containers has
a unique IP address, allocated from
ENIs attached to the host. Containers 1 and 2, however, can route traffic
through the same ENI because they
both use only Security Group X.
IAM role (in this case, the host’s) to
assume the identity capabilities of another principal temporarily (in this
case, the container’s). This approach
provides containers with only their
IAM credentials using the same IAM
roles that are used in EC2 VMs. In addition to IAM roles, the proxy provides
containers with Titus instance identity
information instead of EC2 identity information. Client libraries such as the
Eureka client use this information.
A common network is key. An important enabler for many of the integrations was a common networking
infrastructure. Seamless network communication between containerized applications and existing infrastructure
removed many integration and adoption hurdles.
A common solution to container
networking is to provide an overlay network that creates a separate network on
top of an existing one. This approach is
appealing because it decouples the two
networks and does not require changing the underlying one. However, an
overlay segregates the containers’ networking space from the existing network and requires gateways or proxies
to connect them.
Another common approach is to
allocate specific ports from the host’s
IP to containers. While this allows the
container’s IP to be part of the existing
network IP space, it allows containers to use only specific ports that they
must know upfront, limits colocating
containers that use the same ports,
and exposes the host’s networking
policies to the container. Additionally,
applications and infrastructure must
handle container networking (ports)
differently from how they handle VM
networking (IPs). Since many Netflix systems are IP aware, but not port
aware, retrofitting additional port data
into the necessary systems would have
Titus provides a unique IP address
to each container by connecting containers to the same AWS Virtual Private
Cloud (VPC) network to which VMs
are connected. Using a common VPC
allows containers to share the same
IP address space as VMs and use the
same networking policies and features
such as AWS SGs (Security Groups).
This approach avoids the need to manage ports, as each container gets its
own IP and full port range, and network gateways.
Figure 2. Titus container IP configuration.
Linux policy based routing
+ traffic control