This app-note describes a mechanism to migrate a legacy cloud-based infrastructure which could use a L2 based network segmentation using technologies such as Linux Bridge or OVS to a Contrail SDN based cloud infrastructure. Both Legacy and Contrail clusters can co-exist side by side such that a single L3 subnet can span across both legacy and contrail clusters. This facilitates the cloud administrators to perform a phase wise migration of virtual workloads from legacy to contrail cluster. In addition to this both Legacy and Contrail workloads also have a public internet access.

Proposed solutions:

Idea is to use separate logical L2 & L3 SDN gateways to interconnect the Legacy cloud and the Contrail cloud and provide public internet access from both the clouds. L2 gateway PE will be peered to Contrail Controller for EVPN address family and will hence exchange the MAC address routes of the VMs. Also the the PE-CE link of the EVPN routing-instance is connected to the legacy cloud L2 fabric. This will make the Contrail Virtual Network to be stretched to the legacy cloud. 2 EVPN PE’s are installed for redundancy and the PE Ethernet interface on both the PE’s are configured as single homed Ethernet segment (ESI 0). Spanning tree will be used to break the Layer 2 loop formed by the L2 Network and the redundant EVPN PE’s. In this case, only one L3-SDN GW is shown. Adding the second L3-GW for the redundancy doesn’t add much complexity and is just a matter of configuring BGP between Contrail and the second L3-GW.

Note: The L2 & L3 gateway functionalities can be easily collapsed 2 physical MX routers. This can be done either by using 2 separate routing instances for EVPN and L3VPN and separate PE-CE link towards Legacy cloud (EVPN L2 ifl) and the public network (L3VPN L3 ifl). In this example, separate routers are used for ease of illustration.

The public access from the legacy cloud VM is through the L3-GW configured on the IRB (VLAN) interface aggregation switch (192.168.1.250 in this example) whereas for the VMs running on the Contrail cloud the traffic from the VM towards the public internet is through the default route originated from MX L3GW VRF. This default route actually will redirect the traffic to inet.0 on the MX L3-GW for internet access.

Return traffic from the internet always will be routed to the legacy and the contrail cloud via the legacy L3 GW configured on the aggregation switch. The interface between qfx5100-sw1 and mx-l3gw is configured as a L3 interface and provides the return path from the public network to the legacy cloud and the contrail cloud.

Example setup: Verification:

Contrail Virtual Network configuration and vrouter routing table entries

Verify traffic flow from Legacy VM to Contrail VM and Internet:

root@legacy-vm:~# ip route
default via 192.168.1.250 dev p514p1 
10.84.0.0/16 via 10.87.65.126 dev p514p2
10.87.0.0/16 via 10.87.65.126 dev p514p2
10.87.65.0/25 dev p514p2  proto kernel  scope link  src 10.87.65.2
192.168.1.0/24 dev p514p1  proto kernel  scope link  src 192.168.1.111
root@legacy-vm:~#
root@legacy-vm:~# ping 8.8.8.8 -c 2
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=63 time=0.398 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=63 time=0.412 ms
 

--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.398/0.405/0.412/0.007 ms
root@legacy-vm:~# ping 192.168.1.4 -c 2
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=62 time=1.34 ms
64 bytes from 192.168.1.4: icmp_seq=2 ttl=62 time=0.384 ms
 

--- 192.168.1.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.384/0.863/1.342/0.479 ms
root@legacy-vm:~# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.4              ether   02:9a:66:37:06:5c   C                     p514p1
192.168.1.250            ether   54:4b:8c:a8:8c:00   C                     p514p1
10.87.65.126             ether   30:7c:5e:0f:8f:c0   C                     p514p2
root@legacy-vm:~#

On the MX L2 & L3 gateways:

mx-l2-gw1 (connected to qfx5100-sw1 whose interface is in spanning blocked state)

root@mx-l2-gw1# run show bridge mac-table

 

MAC flags       (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC)

 

Routing instance : contrail_l2_4_VN-1
Bridging domain : bd-4, VLAN : none
MAC                 MAC      Logical                Active
address             flags    interface              source
02:9a:66:37:06:5c   D        vtep.32769             10.87.65.1
54:4b:8c:a0:cc:82   D        vtep.32770             172.16.101.3
90:e2:ba:aa:81:20   D        vtep.32770             172.16.101.3
 

[edit]

root@mx-l2-gw1#

mx-l2-gw1 (connected to qfx5100-sw2 whose interface is in spanning forwarding state)

root@mx-l2-gw2# run show bridge mac-table

 

MAC flags       (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC)

 

Routing instance : contrail_l2_4_VN-1
Bridging domain : bd-4, VLAN : none
MAC                 MAC      Logical                Active
address             flags    interface              source
02:9a:66:37:06:5c   D        vtep.32769             10.87.65.1
54:4b:8c:a0:cc:82   D        xe-0/0/1.0
90:e2:ba:aa:81:20   D        xe-0/0/1.0

[edit]

root@mx-l2-gw2#

mx-l3-gw

root@mx-l3-gw# run show route table contrail-l3_4_VN-1.inet.0

 

contrail-l3_4_VN-1.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

 
0.0.0.0/0          *[Static/5] 02:48:06
to table inet.0
192.168.1.0/24     *[Static/5] 1d 08:30:30
Discard
192.168.1.4/32     *[BGP/170] 03:03:33, MED 100, localpref 200, from 10.87.65.1
AS path: ?, validation-state: unverified
> via gr-0/0/0.32769, Push 16

[edit]

root@mx-l3-gw# run show route table inet.0 192.168.1.0/24
inet.0: 10 destinations, 11 routes (10 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

 
192.168.1.0/24     *[Static/5] 07:59:03
> to 192.168.50.1 via xe-0/0/1.0
 

[edit]

root@mx-l3-gw#

qfx5100-sw1

root@qfx5100-sw1# run show spanning-tree interface

 

Spanning tree interface parameters for instance 0
 
Interface                  Port ID    Designated         Designated         Port    State  Role
port ID           bridge ID          Cost
xe-0/0/42                 128:1095     128:1097  16384.544b8ca0cc82         2000    BLK    ALT
xe-0/0/45                 128:1101     128:1101   8192.544b8ca88c02         2000    FWD    ROOT
 

{master:0}[edit]

root@qfx5100-sw1#

qfx5100-sw2

root@qfx5100-sw2# run show spanning-tree interface

 
Spanning tree interface parameters for instance 0
 
Interface                  Port ID    Designated         Designated         Port    State  Role
port ID           bridge ID          Cost
xe-0/0/43                 128:1097     128:1097  16384.544b8ca0cc82         2000    FWD    DESG
xe-0/0/44                 128:1099     128:1099   8192.544b8ca88c02         2000    FWD    ROOT
 

{master:0}[edit]

root@qfx5100-sw2#

This is a guest blog by Johnny Chen from AT&T with contributions by Qasim Arham, Henon Huang, Vijay Kamisetty from Juniper Networks.

Overview

The purpose of this is to lay out a method of tracing flows through OpenStack & Contrail environment with a scaled-out Service Instance (virtual firewall) in the path.

Environment Versions:
OpenStack == Juno (2014.2.4)
Contrail == 3.0.0

Table of Contents

Overview

Table of Contents

Background & Drawings

Tracing: Outside => In

Step 1: On the SDNGW, find the NEXT-HOP (NH) to the vFW VM in the routing table
Step 2: Verify Traffic to vFW Service Instance VM Compute
Step 3: Trace Flow into vFW Service Instance VM
Step 4: Verify Traffic is leaving vFW towards DST VM
Step 5: Verify Traffic is getting to DST VM TAP Interface
Step 6: On DST VM, Verify Traffic Inbound

Tracing: Inside => Out

Step 1: On DST VM Compute, Trace Return Flow
Step 2: Verify Return Traffic hitting same vFW as Inbound Path

References

Background & Drawings

Access to some useful elements for this exercise:

Caveat: There is often more than one way to get to the end result so while this is an attempt to document one method, there could be other ways of achieving to a diagnosis. For this example, we will use a lab network depicted in the drawing below as a reference and trace the packet flow between 12.12.0.102 (SRC) and 10.10.0.100 (DST) where the destination is a VM in the Overlay Network.

The flow path is as follows:
12.12.0.102 <> Physical Network <> SDN-GW <> VN_A <> FW Service Instance <> VN_B <> 10.10.0.100

Tracing: Outside => In

STEP 1: On the SDNGW, find the NEXT-HOP (NH) to the vFW VM in the routing table.

1a. Single VM or Compute (Single Next-Hop)

On SDNGW:

SDNGW> show route 10.10.0.100

vntest.inet.0: 50 destinations, 150 routes (50 active, 0 holddown, 0 hidden)
 @ = Routing Use Only, # = Forwarding Use Only
 + = Active Route, - = Last Active, * = Both

10.10.0.0/24 *[BGP/170] 1w1d 21:30:43, MED 100, localpref 200, from 172.20.0.5
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.19283, Push 31
 [BGP/170] 1w1d 21:30:43, MED 100, localpref 200, from 172.20.0.6
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.19283, Push 31

SDNGW> show interfaces gr-0/1/0.19283
 Logical interface gr-0/1/0.19283 (Index 345) (SNMP ifIndex 526)
 Flags: Up Point-To-Point SNMP-Traps 0x4000 IP-Header 172.20.0.23:10.10.10.10:47:df:64:0000000800000000 Encapsulation: GRE-NULL
 Gre keepalives configured: Off, Gre keepalives adjacency state: down
 Input packets : 4332961
 Output packets: 7699862
 Protocol inet, MTU: 9062
 Flags: None
 Protocol mpls, MTU: 9050, Maximum labels: 3
 Flags: None

Useful Information from Output:
– 172.20.0.23: NEXT-HOP Compute
– 31: Label of NEXT-HOP on Compute

What we find is the GRE interface (Label 31) for the NEXT-HOP points to the COMPUTE at 172.20.0.23. You can find which COMPUTE this is via the Contrail WebGUI (Monitor => Virtual Routers => [Search Field]172.20.0.23 or search for the VHOST0 IP’s of your computes. In this case, the COMPUTE “Hostname” matches with the NEXT-HOP IP 172.20.0.23 matches with COMPUTE cmpt001. cmpt001 hosts the VM of the vFW Service Instance that is the next element in the path to the DST VM. From here, go to Step 2.

1b. Multiple VM / Compute (Composite Next-Hop)

On SDNGW:

SDNGW> show route 10.10.0.100

vntest.inet.0: 70 destinations, 150 routes (70 active, 0 holddown, 0 hidden)
 @ = Routing Use Only, # = Forwarding Use Only
 + = Active Route, - = Last Active, * = Both

10.10.0.0/24 @[BGP/170] 1w1d 21:11:22, MED 100, localpref 200, from 172.20.0.4
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.74659, Push 62
 [BGP/170] 1w1d 21:14:54, MED 100, localpref 200, from 172.20.0.5
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.19283, Push 31
 [BGP/170] 1w1d 21:11:22, MED 100, localpref 200, from 172.20.0.6
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.74659, Push 62
 [BGP/170] 1w1d 21:14:54, MED 100, localpref 200, from 172.20.0.6
 AS path: ?, validation-state: unverified
 > via gr-0/1/0.19283, Push 31
 #[Multipath/255] 1w1d 21:13:23, metric 100, metric2 0
 via gr-0/1/0.74659, Push 62
 > via gr-0/1/0.19283, Push 31

SDNGW> show interfaces gr-0/1/0.19283
 Logical interface gr-0/1/0.19283 (Index 345) (SNMP ifIndex 526)
 Flags: Up Point-To-Point SNMP-Traps 0x4000 IP-Header 172.20.0.23:10.10.10.10:47:df:64:0000000800000000 Encapsulation: GRE-NULL
 Gre keepalives configured: Off, Gre keepalives adjacency state: down
 Input packets : 4332961
 Output packets: 7699862
 Protocol inet, MTU: 9062
 Flags: None
 Protocol mpls, MTU: 9050, Maximum labels: 3
 Flags: None

If your environment utilizes logging correlation software like the Splunk or Juniper JSA/STRM, that tool would be useful to figure out which “next-hop” scaled-out service instance vFW. Otherwise, we will have to log into each vFW and look at the flow or session table entries to figure out which compute is next in path.

Based on the above information, we can see the traffic transit a particular vFW in the scaled-out Service Instance and we can go directly to Compute where that vFW VM resides. If you know which vFW is taking the traffic (we will assume VFW001 in this case), you can look up with COMPUTE that VM is instantiated on and go from there.

Figure out which COMPUTE the ServiceInstance vFW is on:

toolsvm:~$ nova list | grep -i vfw
 | a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q | VFW001 | ACTIVE | - | Running | ADMIN=192.168.0.194; VN_A=11.11.0.64; LOGNET=192.168.11.65; VN_B=10.10.0.64 |
 | z1y2x3w4v5u6t-7s8r-9q0p-1o2n3m4l5k6j | VFW002 | ACTIVE | - | Running | ADMIN=192.168.0.196; VN_A=11.11.0.66; LOGNET=192.168.11.67; VN_B=10.10.0.65 |

toolsvm:~$ nova show a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q
 +------------------------------------------+------------------------------------------------------------+
 | Property | Value |
 +------------------------------------------+------------------------------------------------------------+
 | VN_A network | 11.11.0.64 |
 | LOGNET network | 192.168.11.65 |
 | ADMIN network | 192.168.0.194 |
 | VN_B network | 10.10.0.64 |
 | OS-DCF:diskConfig | MANUAL |
 | OS-EXT-AZ:availability_zone | default |
 | OS-EXT-SRV-ATTR:host | cmpt001 |
 | OS-EXT-SRV-ATTR:hypervisor_hostname | cmpt001.test.net |
 | OS-EXT-SRV-ATTR:instance_name | instance-00000eef |
 | OS-EXT-STS:power_state | 1 |
 | OS-EXT-STS:task_state | - |
 | OS-EXT-STS:vm_state | active |
 | OS-SRV-USG:launched_at | 2016-06-01T00:35:36.000000 |
 | OS-SRV-USG:terminated_at | - |
 | accessIPv4 | |
 | accessIPv6 | |
 | config_drive | |
 | created | 2016-06-01T00:35:36Z |
 | flavor | flv_vsrx (12345678-012a-3bcd-e4f5-6789g01h2345) |
 | hostId | asdflkijhbalk1h23k132427896r98a7asfhio1u241234iuhewnoafs |
 | id | a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q |
 | image | vsrx15.1x49d30.3 (6baishdu-ewiu-9238-sibh-02394isfoayx) |
 | key_name | - |
 | name | VFW001 |
 | os-extended-volumes:volumes_attached | [] |
 | progress | 0 |
 | security_groups | default |
 | status | ACTIVE |
 | tenant_id | 9asdf2983riahdf9987sdf9ya9sya9sy |
 | updated | 2016-06-01T00:35:36Z |
 | user_id | d8923923hisuadfhf893923h2hfdasfh |
 +------------------------------------------+------------------------------------------------------------+

toolsvm:~$ for x in $(nova list | grep -i foam | awk {'print $4'}); do nova show $x | sed -n '10p;24p;28p' | awk {'print $4'} | xargs | sed 's/ / || /g'; done
 cmpt001 || a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q || VFW001
 cmpt003 || z1y2x3w4v5u6t-7s8r-9q0p-1o2n3m4l5k6j || VFW002

We see that VFW001 with ID a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q is on COMPUTE cmpt001.

TOP

STEP 2: Verify Traffic is making it into vFW ServiceInstance VM Compute

2a. Go to COMPUTE

toolsvm ~]$ ssh user@cmpt001
 password for [user]:
 Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-61-generic x86_64)

2b. Find COMPUTE interfaces

cmpt001:~# cat /etc/contrail/contrail-vrouter-agent.conf | grep -A13 -i virtual-host-interface
 [VIRTUAL-HOST-INTERFACE]
 # Everything in this section is mandatory

# name of virtual host interface
 name=vhost0

# IP address and prefix in ip/prefix_len format
 ip=172.20.0.23/32

# Gateway IP address for virtual host
 gateway=172.20.0.1

# Physical interface name to which virtual host interface maps to
 physical_interface=p1p1

cmpt001:~# ifconfig vhost0
 vhost0 Link encap:Ethernet HWaddr b0:ob:ab:ba:0a:a0
 inet addr:172.20.0.23 Bcast:172.20.0.31 Mask:255.255.255.240
 UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
 RX packets:84487487 errors:0 dropped:182627 overruns:0 frame:0
 TX packets:82063519 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:253984497954 (253.9 GB) TX bytes:67502412941 (67.5 GB)

cmpt001:~# ifconfig p1p1
 p1p1 Link encap:Ethernet HWaddr b0:ob:ab:ba:0a:a0
 UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
 RX packets:194126327 errors:0 dropped:0 overruns:0 frame:0
 TX packets:125130748 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:286638778868 (286.6 GB) TX bytes:94956347917 (94.9 GB)
 Interrupt:40 Memory:f3000000-f37fffff

2c. Verify vRouter is exchanging XMPP information with the CONTRAIL CONTROLLER(s)

cmpt001:~# tcpdump -D | grep -i vhost0
 1.vhost0

cmpt001:~# tcpdump -nei 1 port xmpp-server
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on vhost0, link-type EN10MB (Ethernet), capture size 65535 bytes
 12:04:23.420090 c9:36:dd:24:po:p0 > b0:ob:ab:ba:0a:a0, ethertype IPv4 (0x0800), length 66: 172.20.0.5.5269 > 172.20.0.23.58699: Flags [.], ack 3731677826, win 8748, options [nop,nop,TS val 1268013680 ecr 1267113680], length 0
 12:04:23.420120 b0:ob:ab:ba:0a:a0 > c9:36:dd:24:po:p0, ethertype IPv4 (0x0800), length 66: 172.20.0.23.58699 > 172.20.0.5.5269: Flags [.], ack 1, win 6792, options [nop,nop,TS val 1267114931 ecr 1268012429], length 0
 ^C
 2 packets captured
 4 packets received by filter
 0 packets dropped by kernel

2d. Verify ICMP packets from example (12.12.0.102 => 10.10.0.100) is making it into the COMPUTE cmpt001

cmpt001:~# tcpdump -D | grep p1p1
 7.p1p1

cmpt001:~# tcpdump -nei 7 proto 47 | grep 11.11.0.60
 tcpdump: WARNING: bond0.2004: no IPv4 address assigned
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on bond0.2004, link-type EN10MB (Ethernet), capture size 65535 bytes
 12:07:02.093007 00:1d:c2:8f:22:db > 0a:36:cc:61:aa:l0, ethertype IPv4 (0x0800), length 126: 172.20.0.25 > 172.20.0.23: GREv0, proto MPLS unicast (0x8847), length 92: MPLS (label 31, exp 0, [S], ttl 255) 12.12.0.102 > 10.10.0.100: ICMP echo request, id 15119, seq 20030, length 64
 12:07:03.094702 00:1d:c2:8f:22:db > 0a:36:cc:61:aa:l0, ethertype IPv4 (0x0800), length 126: 172.20.0.25 > 172.20.0.23: GREv0, proto MPLS unicast (0x8847), length 92: MPLS (label 31, exp 0, [S], ttl 255) 12.12.0.102 > 10.10.0.100: ICMP echo request, id 15119, seq 20031, length 64
 ^C
 2 packets captured
 4 packets received by filter
 0 packets dropped by kernel

TOP

STEP 3: Trace the flow into vFW Service Instance VM

3a. Find NEXT-HOP (mpls –dump will output the entire Label => NextHop table) – we found the MPLS Label 31 from the Next-Hop from the SDNGW

cmpt001:~# mpls --get 31
 MPLS Input Label Map

Label NextHop
 -------------------
 31 49

3b. Find NEXT-HOP Interface and VRF that the FW VM interface that is created on for this flow on VN_A.

cmpt001:~# nh --get 49
 Id:49 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:8 Vrf:11
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:24 Len:14
 Encap Data: 02 95 3d 1d c9 69 00 00 5e 00 01 00 08 00

cmpt001:~# vif --get 24
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/24 OS: tap93d2dja8-22
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:11 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:728634 bytes:71113479 errors:0
 TX packets:756736 bytes:89113687 errors:0

3c. [IGNORE FOR NOW] On Vrf:11 (VN_A), we see the routing-table point to the FW’s “RIGHT” interface as the NEXT-HOP to 10.10.0.0/24 Network (VN_B)

cmpt001:~# rt --dump 11 | grep 10.10.0
 Vrouter inet4 routing table 0/13/unicast
 Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP

Destination PPL Flags Label Nexthop Stitched MAC(Index)
 10.10.0.0/24 24 P - 164 -

cmpt001:~# nh --get 164
 Id:164 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:11
 Flags:Valid, Policy, Ecmp,
 Sub NH(label): 95(31) -1 14(62)

root@cmpt001:~# nh --get 95
 Id:95 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:25 Vrf:11
 Flags:Valid,
 EncapFmly:0806 Oif:24 Len:14
 Encap Data: 02 95 3d 1d c9 69 00 00 5e 00 01 00 08 00

cmpt001:~# vif --get 24
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/24 OS: tap93d2dja8-22
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:11 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:733959 bytes:71635018 errors:0
 TX packets:762144 bytes:89646655 errors:0

cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
 <target dev="tapc218da9d-32"/>
 <target dev="tap13a2d42c-5d"/>
 <target dev="tap93d2dja8-22"/>
 <target dev="tapa8d2gf3a-k4"/>

(Interfaces are enumerated in the order MGMT, LEFT, RIGHT, OTHER… so the TAP interface above it the “RIGHT” interface in VN_A)

3d. Check for “Discards” on Vrf:11

cmpt001:~# watch vrfstats --get 11

Every 2.0s: vrfstats --get 11 Fri Jun 03 19:13:13 2016

Vrf: 11
 Discards 0, Resolves 0, Receives 0, L2 Receives 7783, Vrf Translates 0, Unknown Unicast Floods 0
 Ecmp Composites 0, L2 Mcast Composites 26818, Fabric Composites 24593, Encap Composites 24593, Evpn Composites 0
 Udp Tunnels 0, Udp Mpls Tunnels 4000360, Gre Mpls Tunnels 7780, Vxlan Tunnels 0
 L2 Encaps 3321946, Encaps 32480
 GROs 2662900, Diags 0
 Arp Virtual Proxys 654, Arp Virtual Stitchs 1539, Arp Virtual Floods 600, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Flo
 ods 0

Based on the above output, it does not appear that there are any discards on Vrf:11 that would be causing the connectivity failure (Counter at “0” and not incrementing).

3e. Look for “Discards” via dropstats and check relevant counters (“Flow Action Drop” & “Discards” in this case) to see if they increment

cmpt001:~# watch dropstats

Every 2.0s: dropstats Fri Jun 03 19:16:59 2016

GARP 0
 ARP no where to go 0
 Invalid ARPs 0

Invalid IF 0
 Trap No IF 0
 IF TX Discard 0
 IF Drop 0
 IF RX Discard 0

Flow Unusable 0
 Flow No Memory 0
 Flow Table Full 0
 Flow NAT no rflow 0
 Flow Action Drop 12040
 Flow Action Invalid 0
 Flow Invalid Protocol 0
 Flow Queue Limit Exceeded 132

Discards 8372
 TTL Exceeded 0
 Mcast Clone Fail 0
 Cloned Original 348762341

Invalid NH 20637113
 Invalid Label 0
 Invalid Protocol 0
 Rewrite Fail 0
 Invalid Mcast Source 0

Push Fails 0
 Pull Fails 0
 Duplicated 0
 Head Alloc Fails 0
 Head Space Reserve Fails 0
 PCOW fails 0
 Invalid Packets 0

Misc 764
 Nowhere to go 0
 Checksum errors 0
 No Fmd 0
 Invalid VNID 0
 Fragment errors 0
 Invalid Source 12
 Jumbo Mcast Pkt with DF Bit 0
 ARP No Route 0
 ARP Reply No Route 0
 No L2 Route 136423

“Discards” and “Flow Action Drop” counters stayed steady at “12040” and “8372” respectively and did not increment.

3f. Find which VM is associated with TAP Interface tap93d2dja8-22 (there is more than one way to get this information)

CLI Method:

– From previous discovery, we know that the ID of the VM VFW001 on cmpt001 is a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q. Map the TAP Interface to the VM:

cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
 <target dev="tapc218da9d-32"/>
 <target dev="tap13a2d42c-5d"/>
 <target dev="tap93d2dja8-22"/>
 <target dev="tapa8d2gf3a-k4"/>

Contrail GUI Method: Find the Compute where the VM resides (cmpt001) via Monitor => Virtual Routers = cmpt001 // Interfaces. From there, do a search for the interface tap93d2dja8-22. From there, you should be able to find which VM the TAP Interface is mapped to.

3g. From the above information, we know that this is mapped to the vFW”RIGHT” interface which is on the VN_A Network. We can do a TCPDUMP to verify if the traffic is making it to the FW’s Ingress/VN_A interface:

cmpt001:~# tcpdump -nei tap93d2dja8-22
 tcpdump: WARNING: tap93d2dja8-22: no IPv4 address assigned
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on tap93d2dja8-22, link-type EN10MB (Ethernet), capture size 65535 bytes
 12:09:41.995714 bk:2a:n6:00:02:00 > 02:95:3d:1d:c9:69, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 10575, length 40
 12:09:42.995769 bk:2a:n6:00:02:00 > 02:95:3d:1d:c9:69, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 10593, length 40
^C
2 packets captured
4 packets received by filter
 0 packets dropped by kernel

3h. From the above, we see ICMP echo requests going into the FW, but no replies. Let’s check for what the vFW sees:

toolsvm:~$ ssh 192.168.0.194

Password:
 --- JUNOS 15.1X49-D30.3 built 2015-12-17 04:39:24 UTC
 VFW001>

VFW001> show security flow session destination-prefix 10.10.0.100
 Session ID: 206346, Policy name: allow_restricted/6, Timeout: 2, Valid
 In: 12.12.0.102/34054 --> 10.10.0.100/61;icmp, If: ge-0/0/1.0, Pkts: 1, Bytes: 60,
 Out: 10.10.0.100/61 --> 12.12.0.102/34054;icmp, If: ge-0/0/0.0, Pkts: 1, Bytes: 0,

Session ID: 206347, Policy name: allow_restricted/6, Timeout: 4, Valid
 In: 12.12.0.102/34058 --> 10.10.0.100/61;icmp, If: ge-0/0/1.0, Pkts: 1, Bytes: 60,
 Out: 10.10.0.100/61 --> 12.12.0.102/34058;icmp, If: ge-0/0/0.0, Pkts: 1, Bytes: 0,

Session ID: 206350, Policy name: allow_restricted/6, Timeout: 2, Valid
 In: 12.12.0.102/34057 --> 10.10.0.100/61;icmp, If: ge-0/0/1.0, Pkts: 1, Bytes: 60,
 Out: 10.10.0.100/61 --> 12.12.0.102/34057;icmp, If: ge-0/0/0.0, Pkts: 1, Bytes: 0,
 Total sessions: 3

The vFW sees packets inbound through it to the VM 10.10.0.100 but no return traffic (which matches what we see on the TCPDUMP on the FW’s GE-0/0/1 or “RIGHT” interface). At this point, we need to look at the traffic leaving the FW on the “LEFT”/VN_B interface.

3i. Check the COMPUTE vRouter Flow Table to see if the traffic is being forwarded.

cmpt001:~# flow --match 10.10.0.100
 Flow table(size 68157440, entries 542386)

Entries: Created 898146 Added 898144 Processed 898146 Used Overflow entries 0
 (Created Flows/CPU: 368496 140472 62462 39956 28374 22527 25634 19742 20169 15999 6767 6693 6514 6322 7041 6249 6200 6272 6324 6235 7985 9035 7329 9146 6741 6608 5981 7374 10025 8246 663 941 721 734 700 671 771 837 1120 4070)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
 TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([10.10.0.100]:*)

Index Source:Port/Destination:Port Proto(V)
 -----------------------------------------------------------------------------------

168624<=>23140 12.12.0.102:1393 1 (2->3)
 10.10.0.100:0
 (Gen: 5, K(nh):33, Action:F, Flags:, S(nh):94, Stats:257760/25260480, SPort 53824)
 --
 430164<=>119628 12.12.0.102:1393 1 (13->5)
 10.10.0.100:0
 (Gen: 11, K(nh):49, Action:F, Flags:, S(nh):88, Stats:257760/21651840, SPort 60610)
 --

TOP

STEP 4: Verify traffic is leaving vFW Service Instance VM towards 10.10.0.100

4a. Verify next TAP Interface for FW (“LEFT” / VN_B) sees traffic exit the vFW destined for the VM at 10.10.0.100

cmpt001:~# nh --get 33
 Id:93 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:12
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:12 Len:14
 Encap Data: 02 31 e6 d2 4c 7d 00 00 5e 00 01 00 08 00

cmpt001:~# vif --get 12
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/12 OS: tap13a2d42c-5d
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:12 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:1236404 bytes:135113415 errors:0
 TX packets:1231952 bytes:120251878 errors:0

cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
 <target dev="tapc218da9d-32"/>
 <target dev="tap13a2d42c-5d"/>
 <target dev="tap93d2dja8-22"/>
 <target dev="tapa8d2gf3a-k4"/>

cmpt001:~# tcpdump -nei tap13a2d42c-5d
 tcpdump: WARNING: tap13a2d42c-5d: no IPv4 address assigned
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on tap13a2d42c-5d, link-type EN10MB (Ethernet), capture size 65535 bytes
 12:24:31.175890 02:dd:26:2b:24:p2 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 11410, length 40
 12:24:32.176227 02:dd:26:2b:24:p2 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 11413, length 40
 ^C
 2 packets captured
 4 packets received by filter
 0 packets dropped by kernel

4b. Traffic (ICMP echo requests) is leaving vFW egress interface (GE-0/0/0) “LEFT” or VN_B interface destined towards the VM 10.10.0.100. Check to see if there are any discards on the VRF.

cmpt001:~# watch vrfstats --get 12

Every 2.0s: vrfstats --get 12 Fri Jun 03 19:13:13 2016

Vrf: 12
 Discards 0, Resolves 0, Receives 0, L2 Receives 12369, Vrf Translates 0, Unknown Unicast Floods 0
 Ecmp Composites 0, L2 Mcast Composites 18742, Fabric Composites 12190, Encap Composites 12168, Evpn Composites 0
 Udp Tunnels 0, Udp Mpls Tunnels 17, Gre Mpls Tunnels 4010, Vxlan Tunnels 0
 L2 Encaps 71982, Encaps 1302371
 GROs 2687, Diags 0
 Arp Virtual Proxys 8658, Arp Virtual Stitchs 4858, Arp Virtual Floods 3748, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 30

No discards.

4c. Verify routing on Vrf:10 (VN_B) FIB table for 10.10.0.100

cmpt001:~# rt --dump 12 | grep 10.10.0.100
 Vrouter inet4 routing table 0/2/unicast
 Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP

Destination PPL Flags Label Nexthop Stitched MAC(Index)
 10.10.0.100/32 32 LP 31 88 2:38:c8:ea:9a:21(211664)

4d. Find the NEXT-HOP COMPUTE to get to the VM on VN_B at 10.10.0.100

cmpt001:~# nh --get 88
 Id:88 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:144 Vrf:0
 Flags:Valid, MPLSoUDP,
 Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:8c dc d4 10 74 c0 8c dc d4 10 75 a0 08 00
 Vrf:0 Sip:172.20.0.23 Dip:172.20.0.24

Next-Hop 88 points to a destination IP of 172.20.0.24 (vhost0 Interface IP of the destination Compute on Default Vrf0) with MPLS Label 31

TOP

STEP 5: Verify traffic is getting to Destination VM at 10.10.0.100 on TAP Interface

5a. Find NEXT-HOP COMPUTE at 172.29.2.82

CLI Method (From toolsvm – use the IP of the Destination VM to find Compute):

toolsvm:~$ for x in $(nova list | grep -i 10.10.0.100 | awk {'print $4'}); do nova show $x | sed -n '7p;21p;25p' | awk {'print $4'}; done
 cmpt002
 98asdfhjkn213-sdai-ncxv-3421987kjlsm
 testvm-VN_B_1

toolsvm:~$ nova show 98asdfhjkn213-sdai-ncxv-3421987kjlsm
 +------------------------------------------+-------------------------------------------------------------+
 | Property | Value |
 +------------------------------------------+-------------------------------------------------------------+
 | MNS_shared_OAM_PROTECTED_NET_1_1 network | 10.10.0.100 |
 | OS-DCF:diskConfig | AUTO |
 | OS-EXT-AZ:availability_zone | nova |
 | OS-EXT-SRV-ATTR:host | cmpt002 |
 | OS-EXT-SRV-ATTR:hypervisor_hostname | cmpt002.test.net |
 | OS-EXT-SRV-ATTR:instance_name | instance-00000f28 |
 | OS-EXT-STS:power_state | 1 |
 | OS-EXT-STS:task_state | - |
 | OS-EXT-STS:vm_state | active |
 | OS-SRV-USG:launched_at | 2016-06-01T00:35:36.000000 |
 | OS-SRV-USG:terminated_at | - |
 | accessIPv4 | |
 | accessIPv6 | |
 | config_drive | |
 | created | 2016-06-01T00:35:36Z |
 | flavor | m1.tiny (2) |
 | hostId | a0s9dfuwqehrlkqwerl89ydfkj1394874329y2nlkadfasygas98ydfs |
 | id | 98asdfhjkn213-sdai-ncxv-3421987kjlsm |
 | image | ubuntu-14 (1f062a45-4a90-437d-a2b8-b2b5a565d95a) |
 | key_name | - |
 | metadata | {} |
 | name | testvm-VN_B_1 |
 | os-extended-volumes:volumes_attached | [] |
 | progress | 0 |
 | security_groups | default |
 | status | ACTIVE |
 | tenant_id | 9asdf2983riahdf9987sdf9ya9sya9sy |
 | updated | 2016-06-01T00:35:36Z |
 | user_id | d8923923hisuadfhf893923h2hfdasfh |
 +------------------------------------------+-------------------------------------------------------------+

GUI Method (From Contrail GUI)

Monitor => Infrastructure => Virtual Routers => Search

5b. Go to COMPUTE cmpt002. Based on Label 31 for NEXT-HOP on this COMPUTE, we can find where the traffic is destined to.

cmpt002:~# mpls --get 31
 MPLS Input Label Map

Label NextHop
 -------------------
 31 73
 root@cmpt002:~# nh --get 73
 Id:73 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:6 Vrf:13
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:14 Len:14
 Encap Data: 02 38 c8 ea 9a 21 00 00 5e 00 01 00 08 00

cmpt002:~# vif --get 14
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/14 OS: tap83f2ki4w-72
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:13 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:2272968 bytes:214007093 errors:0
 TX packets:2294863 bytes:287437540 errors:0

5c. Verify the above output matches the VM libvert.xml

cmpt002:~# cat /var/lib/nova/instances/98asdfhjkn213-sdai-ncxv-3421987kjlsm/libvirt.xml | grep -i tap83f2ki4w-72
 <target dev="tap83f2ki4w-72"/>

5d. Check for Vrf:13 discards

cmpt002:~# watch vrfstats --get 13

Every 2.0s: vrfstats --get 13 Fri Jun 03 19:33:33 2016

Vrf: 13
 Discards 0, Resolves 0, Receives 0, L2 Receives 3299, Vrf Translates 0, Unknown Unicast Floods 0
 Ecmp Composites 0, L2 Mcast Composites 3364, Fabric Composites 330, Encap Composites 739, Evpn Composites 0
 Udp Tunnels 0, Udp Mpls Tunnels 4, Gre Mpls Tunnels 0, Vxlan Tunnels 0
 L2 Encaps 3338, Encaps 2421364
 GROs 40513, Diags 0
 Arp Virtual Proxys 2906, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0

5e. Check vRouter Flow Table for “Action:F”

cmpt002:~# flow --match 10.10.0.100
 Flow table(size 68157440, entries 532480)

Entries: Created 2101736 Added 2100485 Processed 2101736 Used Overflow entries 0
 (Created Flows/CPU: 367906 175642 111664 82241 69167 55275 51234 46499 43885 43268 16593 19378 19266 18389 18108 17846 17739 18159 17939 18510 31225 119065 99542 54060 37515 32004 30342 31058 41341 28021 7123 86430 156010 50283 20184 11714 9180 8086 7844 12001)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
 TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([10.10.0.100]:*)

Index Source:Port/Destination:Port Proto(V)
 -----------------------------------------------------------------------------------
 14325<=>438965 12.12.0.102:61 1 (6->5)
 10.10.0.100:0
 (Gen: 10, K(nh):73, Action:F, Flags:, S(nh):22, Stats:1166/69960, SPort 61800)
 --

5f. Verify Traffic is being forwarded on TAP Interface on the VM

cmpt002:~# tcpdump -nei tap83f2ki4w-72
 tcpdump: WARNING: tap83f2ki4w-72: no IPv4 address assigned
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on tap83f2ki4w-72, link-type EN10MB (Ethernet), capture size 65535 bytes
 13:02:32.939608 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59313, length 40
 13:02:33.935583 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59317, length 40
 13:02:34.952637 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59321, length 40
 13:02:35.960168 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59322, length 40
 ^C
 4 packets captured
 6 packets received by filter
 0 packets dropped by kernel

The traffic flow packets are making it all the to the TAP Interface on the vRouter of the Compute that is assigned to the VM 10.10.0.100. We need to get the DST VM owner to verify that they can see the traffic and at this point it could be any number of problems:
– VM Host Routing Table
– VM IPTables or similar service filtering inbound/outbound traffic
– OpenStack Security Group applied to VM
– Controller/Compute Routing Issue
– Etc.,

TOP

STEP 6: On VM, run TCPDUMP to inspect inbound traffic

6a. Access VM (this example will utilize Link-Local Address from Compute)
Find Link-Local Address of TAP interface of VM via Contrail Web GUI

On Compute, SSH to Link-Local Address 169.254.0.14

cmpt002:/etc# ssh 169.254.0.14
 169.254.0.14's password:
 Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-32-generic x86_64)

testvm-vn_b:~$

6b. Verify Traffic is making it to the VM

testvm-vn_b:~$ ifconfig
 eth0 Link encap:Ethernet HWaddr 7b:23:a9:tu:c3:p0
 inet addr:10.10.0.100 Bcast:172.20.50.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:2298122 errors:0 dropped:0 overruns:0 frame:0
 TX packets:2276178 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:287697225 (287.6 MB) TX bytes:214259840 (214.2 MB)

eth1 Link encap:Ethernet HWaddr 02:8a:db:64:fb:45
 inet addr:192.168.0.58 Bcast:192.168.0.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:2298122 errors:0 dropped:0 overruns:0 frame:0
 TX packets:2276178 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:287697225 (287.6 MB) TX bytes:214259840 (214.2 MB)

lo Link encap:Local Loopback
 inet addr:127.0.0.1 Mask:255.0.0.0
 UP LOOPBACK RUNNING MTU:65536 Metric:1
 RX packets:34 errors:0 dropped:0 overruns:0 frame:0
 TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:7130 (7.1 KB) TX bytes:7130 (7.1 KB)

testvm-vn_b:~# tcpdump -D
 1.eth0
 2.eth1
 3.any (Pseudo-device that captures on all interfaces)
 4.lo

testvm-vn_b:~$ tcpdump -nei 1 proto 1
 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
 listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
 13:16:15.056859 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 65, seq 109, length 40
 13:16:15.056963 7b:23:a9:tu:c3:p0 > bk:2a:n6:00:02:00, ethertype IPv4 (0x0800), length 74: 10.10.0.100 > 12.12.0.102: ICMP echo reply, id 65, seq 109, length 40
 ^C
 2 packets captured
 4 packets received by filter
 0 packets dropped by kernel

testvm-vn_b:~$

– VM Host Routing Table
– VM IPTables or similar service filtering inbound/outbound traffic
– OpenStack Security Group applied to VM
– Controller/Compute Routing Issue
– Etc.,

TOP

Tracing: Inside => Out

STEP 1: On Compute where VM is being hosted on, Trace flow back out to Source Host

>>*Assume traffic is fixed on the DST VM and now we are tracing the return flow back to the SRC IP

1a. Get vRouter Flow Table information for return traffic from 10.10.0.100 to 12.12.0.102

cmpt002:~# flow --match 12.12.0.102
 Flow table(size 68157440, entries 532480)

Entries: Created 2105344 Added 2104093 Processed 2105344 Used Overflow entries 0
 (Created Flows/CPU: 368319 176011 111801 82325 69226 55341 51282 46532 43914 43306 16597 19384 19280 18394 18114 17851 17745 18169 17948 18521 31269 119556 99662 54124 37564 32051 30401 31112 41397 28061 7134 86678 156797 50372 20220 11723 9190 8099 7854 12020)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
 TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([12.12.0.102]:*)

Index Source:Port/Destination:Port Proto(V)
 -----------------------------------------------------------------------------------
 463620<=>89580 10.10.0.100:65 1 (6->5)
 12.12.0.102:0
 (Gen: 14, K(nh):73, Action:F, Flags:, E:0, S(nh):73, Stats:3/222, SPort 51745)
 --

1b. Find Return Next-Hop Information

cmpt002:~# nh --get 73
 Id:73 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:6 Vrf:13
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:14 Len:14
 Encap Data: 02 38 c8 ea 9a 21 00 00 5e 00 01 00 08 00

cmpt002:/etc# rt --dump 13 | grep 172.20.212
 Vrouter inet4 routing table 0/6/unicast
 Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP

Destination PPL Flags Label Nexthop Stitched MAC(Index)
 172.20.212.0/24 0 P - 102 -

cmpt002:~# nh --get 102
 Id:102 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:6601 Vrf:13
 Flags:Valid, Policy, Ecmp,
 Sub NH(label): 22(27) 22(31)

cmpt002:/etc# nh --get 22
 Id:22 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:100 Vrf:0
 Flags:Valid, MPLSoUDP,
 Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:8c dc d4 10 75 a0 8c dc d4 10 74 c0 08 00
 Vrf:0 Sip:172.20.0.24 Dip:172.20.0.23

TOP

STEP 2: On Compute where Next-Hop vFW VM is being hosted on, verify return flow is hitting same egress vFW as ingress

2a. Trace path of return flow back (MPLS Label 27) to vFWand verify it is the same vFW that was utilized for the inbound traffic.

The COMPUTE with Vhost0 IP of 172.20.0.23 == cmpt001 and VFW001 has a ID of a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q from previous discovery.

cmpt001:~# mpls --get 27
 MPLS Input Label Map

Label NextHop
 -------------------
 27 65
 root@cmpt001:~# nh --get 65
 Id:65 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:12
 Flags:Valid, Policy,
 EncapFmly:0806 Oif:29 Len:14
 Encap Data: 02 31 e6 d2 4c 7d 00 00 5e 00 01 00 08 00

cmpt001:~# vif --get 29
 Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
 Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
 Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
 Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/29 OS: tap13a2d42c-5d
 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
 Vrf:12 Flags:PL3L2D MTU:9160 Ref:6
 RX packets:1311527 bytes:145819268 errors:0
 TX packets:1305396 bytes:125789289 errors:0

cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
 <target dev="tapc218da9d-32"/>
 <target dev="tap13a2d42c-5d"/>
 <target dev="tap93d2dja8-22"/>
 <target dev="tapa8d2gf3a-k4"/>

Continue the same process to trace the flows outbound from the vFW to the VM.

TOP

References

TOP

Note: This is a guest blog from tcpCloud, authored by Marek Celoud & Jakub Pavlik (tcp cloud engineers). To see the original post,click here.

It’s been a while since new version 3.X of OpenContrail was released and we have not got so much time to take a good look at new features of this most deployed SDN/NFV with OpenStack. This blog post therefore brings our perspective on specific use cases and how to use BGP as a Service in OpenStack private cloud.

BGPaaS together with configurable ECMP, Intel DPDK and SRIOV support are key features of the new release. All of these features show that OpenContrail became the number one SDN/NFV solution for telco and service providers. Simply because telcos as Deutsche Telekom, France Telekom and AT&T pick this as a solution for the SDN. The last named has significantly influenced features in the last OpenContrail release. To explain the reasons for these requirements and decisions you can watch Austin OC meetup videos, where AT&T has explained their use cases and why they like MPLS L3VPN approach.

tcp cloud tries to bring the real use cases not only for telco VNF for running virtual router appliances. Therefore we try to show you another interesting use case for the global community, where BGPaaS is very important part not only for VNF. We deployed Kubernetes with Calico on top of OpenStack with OpenContrail and redistributed routes through the BGPaaS.

#Peering with default GW/vRouter
    protocol bgp contrail1 {
            debug all;
            local as 64512;
            neighbor 172.16.10.1 as 64512;
            import all;
            export all;
            source address 172.16.10.115;
    }

    #Peering with default DNS server/vRouter
    protocol bgp contrail2 {
            debug all;
            local as 64512;
            neighbor 172.16.10.2 as 64512;
            import all;
            export all;
    }

    #Peering with calico nodes
    protocol bgp calico_master {
            local as 64512;
            neighbor 172.16.10.111 as 64512;
            rr client;
            import all;
            export all;
    }

    protocol bgp calico_node1 {
            local as 64512;
            neighbor 172.16.10.112 as 64512;
            rr client;
            import all;
            export all;
    }

    protocol bgp calico_node2 {
            local as 64512;
            neighbor 172.16.10.113 as 64512;
            rr client;
            import all;
            export all;
    }

    #Peering with second route reflector BIRD
    protocol bgp rtr1 {
            local as 64512;
            neighbor 172.16.10.114 as 64512;
            import all;
            export all;
    }

Then we can see Established BGP peerings (172.16.10.114 and .115) under peers in Control Nodes.

Calico uses by default bgp full mesh topology. We had to disable full mesh and configure only peerings with route reflectors (RTR01 and RTR02).

root@kubernetes-node01:~# calicoctl bgp node-mesh off
root@kubernetes-node01:~# calicoctl bgp peer add 172.16.10.114 as 64512
root@kubernetes-node01:~# calicoctl bgp peer add 172.16.10.115 as 64512

Calico status shows Established peerings with our RTR01 and RTR02.

root@kubernetes-node01:~# calicoctl status
calico-node container is running. Status: Up 44 hours
Running felix version 1.4.0rc2

IPv4 BGP status
IP: 172.16.10.111    AS Number: 64512 (inherited)
+---------------+-----------+-------+----------+-------------+
|  Peer address | Peer type | State |  Since   |     Info    |
+---------------+-----------+-------+----------+-------------+
| 172.16.10.114 |   global  |   up  | 13:14:54 | Established |
| 172.16.10.115 |   global  |   up  | 07:26:10 | Established |
+---------------+-----------+-------+----------+-------------+

Finally we can see part of VRF routing table for our virtual network on compute 01. It shows direct interface for RTR01 VM (172.16.10.114/32) and tunnel to RTR02 (172.16.10.115/32). Subnet 192.168.156.192/26 is for Kubernetes pods and it is dynamically propagated by Calico through BIRD route reflectors.

Conclusion

In this blog post we showed how easy it is to use BGPaaS in OpenContrail and how you can look at general use case of running Kubernetes on top of OpenStack. All OpenContrail installations can be automated via Heat templates, but contrail-heat resources for BGPaaS require some modifications to work properly.

Recently, OpenContrail Advisory Board (OCAB) members Edgar Magana, Cloud Operations Architect at Workday, Jakub Pavlik, CTO of tcpCloud, and Lachlan Evenson got together to discuss the state of the OpenContrail project as well as the fast-moving cloud industry. OCAB members guide the product now and into the future, and use it in their respective networks. In this 5-episode video series called Past, Present & Future: Bridging the Boundaries of Digital World & Beyond, Edgar, Jakub, and Lachlan share how important the OpenContrail community is, why they choose OpenContrail, how they solve ever-evolving networking problems, and what the future looks like in the SDN space.

When “SDN” became as hot as Pokemon Go five years ago, everyone knew it had great promise, but no one knew what to do with it. With the urge to create an open architecture that worked for all of their customers, Edgar, Jakub and Lachlan met in 2012 with the same desire to shift towards a software-centric network, but relied on each other and the community to test, implement, and troubleshoot their respective issues. With so many different technologies to test, and so many nuances in each unique implementation, OCAB members leveraged the community learn from each other and replicate success.

Find out in this first episode, how the community has flourished over the last few years, and what considerations OCAB members and their respective organizations took before adopting OpenContrail.

This video series will be published one episode at a time every Thursday. Subscribe to Juniper’s YouTube channel using the below button to get notified about the new episodes every week. Stay tuned for Episode 2!

To learn more:

Try out OpenContrail for yourself risk-free today!

Download Contrail source code: https://github.com/Juniper/contrail-controller
Download Juniper’s Contrail package: http://www.juniper.net/support/downloads/?p=contrail#sw
Don’t forget to follow us on our Twitter, Facebook and LinkedIn social media sites.

The goal that Edgar, Jakub, and Lachlan share, is to create for public cloud users, the same experience that private cloud users have: peace of mind that their information is secure. But first, they tackle the issue of helping their customers migrate their IT infrastructure to the cloud with continuous communication. They need technology that gives them granular visibility and control over security policies over every workload, whether those workloads are on bare-metal server, or in a virtualized environment. They need to have visibility into and control over everything down to the network level.

These providers want to provide a consistently secure experience for their users across cloud providers that’s fully automated (no manual configuration of static VPNs). They turned to OpenContrail to implement secure virtual overlay networks to logically separate departments, and integrate virtualized workloads in both public and private clouds. Using standards-based protocols, OpenContrail allows enterprises to use their existing physical infrastructure (supporting BGP) for their on-prem infrastructure.

The rest of this video series will be published weekly. Subscribe to Juniper’s YouTube channel using the below button to get notified about the new episodes every week. Stay tuned for Episode 3!

To learn more:

In case you missed it, here’s Episode 1.

Try out OpenContrail for yourself risk-free today!
Download Contrail source code: https://github.com/Juniper/contrail-controller
Download Contrail package: http://www.juniper.net/support/downloads/?p=contrail#sw

Don’t forget to follow us on our Twitter, Facebook and LinkedIn social media sites.

As companies move towards the cloud, the migration won’t be completely seamless, or happen overnight. For the foreseeable future, the cloud is made up of both physical and virtual elements, like bare-metal servers, virtual machines, containers residing in both public and private clouds. With so many elements, Edgar and Jakub need something that enables applications to communicate, no matter what computing environment they’re in. Jakub stresses that the most critical point when trying to connect these disparate elements, is usually networking.

Edgar, Jakub, and Lachlan turned to OpenContrail to connect the cloud’s different elements through a single pane of glass for provisioning across different compute vehicles. Then, they wanted to keep pushing the flexibility of OpenContrail. At OpenStack Summit Vancouver in May 2015, our OCAB members wanted to see if they could integrate OpenContrail with Kubernetes, a Container management tool. Weeks later, a demo was ready. This is what happens when you have a technology that’s flexible enough to adopt other technologies. Learn more about how this community drives innovation at lightning speed by watching the video.

The rest of this video series will be published weekly. Subscribe to Juniper’s YouTube channel using the below button to get notified about the new episodes every week. Stay tuned for Episode 4!

In case you missed it, here’s Episode 1, on the importance of the open source community for OpenContrail, and Episode 2, on how OpenContrail implements secure and automated virtual overlay networks.

To learn more,
Try out OpenContrail for yourself risk-free today!
Download Contrail source code: https://github.com/Juniper/contrail-controller
Download Contrail package: http://www.juniper.net/support/downloads/?p=contrail#sw

As computers become smarter, many new technologies have emerged. One technology which has taken a significant leap in the past few years is Artificial Intelligence & many new services have been built around this.

In this blog, we’ll look at one such service which is getting attention off late, called ChatOps (Chat Operations). ChatOps constitute a “ChatBot” or simply “Bot”, which are asynchronous conversational interfaces that help users to interact with applications. There are many bots out there, which can help you book flight tickets, order food, etc. “Bot as a Service (BaaS)” & “Artificial-Intelligence as a Service (AIaaS)” are the new buzzwords.

Taking it one step further, I introduce you to a bot which I call the “Contrail-Bot”. The bot interacts with the OpenContrail SDN controller & has a complete overview of the cloud (OpenContrail + OpenStack) infrastructure. By interfacing with OpenContrail, the bot offers the cloud-admin an interactive tool to build & manage the OpenStack infrastructure. The cloud-admin strikes a conversation by making requests to the Contrail-Bot & the bot will in turn translate these requests into a language that the OpenContrail SDN controller understands (Python/REST/CLI). The response from the controller is translated back & presented to the cloud-admin in a human readable format.

I started developing the Contrail-Bot with a couple of objectives in mind,

• I should be able to know if my cloud-services are all healthy & are running fine

• I should be able to execute all Linux commands on the target nodes (Power-User mode)

• I should be able to reimage & provision servers quickly with the OS & build of my choice (Contrail-Server-Manager integration)

• I should be able to monitor my cloud & minimize downtime (Get real-time text/call alerts)

As the Contrail-Bot evolves, I plan to introduce AI, which will allow the bot to learn continuously from user requests & take intelligent decisions (Eg. running diagnostic tests, healing the network, etc.) in the event of a failure.

Let the bot revolution begin!

Virtualization and Cloud Computing are accelerating the movement of System monitoring from manual, to reactive to proactive. Knowing when a host or service is down is certainly important, but the ready availability of additional metrics and events from both the application and the infrastructure enables us to get ahead of the curve, and manage performance and availability in a better way. One important aspect of being proactive is Anomaly Detection. Once an event notification infrastructure is place, we can raise alerts based on static thresholds. But, it’s often unclear what’s anomalous and what isn’t for a given deployment. Anomaly Detection via Machine learning techniques can help address this.

Contrail Networking provides Alerts based on any aspect of the system’s Operational State. (Operational State in the OpenContrail system: UVE – User Visible Entities through Analytics API).  We also provide an Anomaly Detection model based on time-series analysis of any metric. Based on past time series information, we learn what to expect in the future. If a given metric reports values that are far from this expectation, we raise an Alert. Of course, this must all be configurable – the parameters and algorithms used for learning the expectation, as well as the threshold that causes the alert.

In this example, we will use statistical process control, based on computing the running average and standard deviation and examining the current value using real-time stream processing. The metric being used is the number of flows being added on a Virtual Machine Interface on a host/vRouter. We run a multitier application with some clients – a redmine webserver with a separate mysql database. Then, we launch a TCP SYN attack on the webserver, which causes an unusual number of flows. This triggers an Alert.

Let’s dive in.

Setting up the Application

Our system has the “default-domain:demo” project with two Virtual Networks – “fe-front” and “be-vn”.
“fe-front” has the VM “fe-web” running a Redmine WebServer. It has a floating IP 10.84.53.83 for clients to access it.
“be-vn” has the VM “be-db”, running a Mysql DB being used by the Redmine Webserver.
This is the Network Monitoring page from the Contrail UI.

Configuring the Alert

Every vRouter reports per-VMI (virtual machine interface) flow statistics against the VMI UVE every 30 seconds:

From controller/src/vnsw/agent/uve/vrouter.sandesh:
struct VrouterFlowRate {
1: optional u32 added_flows;
2: optional u32 max_flow_adds_per_second;
3: optional u32 min_flow_adds_per_second;
4: optional u32 deleted_flows;
5: optional u32 max_flow_deletes_per_second;
6: optional u32 min_flow_deletes_per_second;
7: optional u32 active_flows
}

The vRouter also calculates the Exponential Weighted Mean and Standard Deviation for added_flows, deleted_flows and active_flows as per the standard formulas:
μ_i = (1−α)μ_i-1 + α x_i
σ²_i= S_i   = (1−α)(S_i−1+α(x_i − μ_i-1)²)
Here x_i is the observation in the i-th step, μ_i-1  is the estimated EWM (Exponentially Weighted Mean) , and S_i−1 is the previous estimate of the variance.

We will be using α (alpha) of 0.1 on added_flows in this example.

From controller/src/vnsw/agent/uve/interface.sandesh:
struct UveVMInterfaceAgent {
1: string                name (key=”ObjectVMITable”)
…
26: optional vrouter.VrouterFlowRate flow_rate
41: optional derived_stats_results.AnomalyResult added_flows_ewm
(stats=”flow_rate.added_flows:DSAnomaly:EWM:0.1″)
42: optional derived_stats_results.AnomalyResult deleted_flows_ewm
(stats=”flow_rate.deleted_flows:DSAnomaly:EWM:0.1″)
43: optional derived_stats_results.AnomalyResult active_flows_ewm
(stats=”flow_rate.active_flows:DSAnomaly:EWM:0.1″)
…
}

(It is possible to override alpha by changing the vRouter configuration file at /etc/contrail/contrail-vrouter-agent.conf. The EWM calculation can also be disabled completely)

The vRouter publishes the value of sigma () as it runs the calculation on each VMI. This is a normalized measure of how far the last sample was from the mean. We can configure an Alert based on this. Lets use 2.5.  (For a normal distribution, 98.8% of samples fall between μ ± 2.5σ)

Alerts can be configured at the global level for all objects, or at the project level for objects that are associated with a project, such as Virtual Networks or Virtual Machine Interfaces. Let’s configure this Alert under the default-domain:demo project. So, we can go to the Contrail UI under the Alarm Rules tab of Configure -> Alarms -> Project -> default-domain -> demo. This is what the Alert will look like:

We added an Alert named vmi-anomalous-added-flows. The EWM calculations for the flows added to a VMI every 30 seconds are available in the UVE as UveVMInterfaceAgent.added_flows_ewm. This is the VMI UVE, but in the Alert contents, it is also useful to have the UVE Key of the VM and the VN, so we add them as the variables UveVMInterfaceAgent.vm_name and UveVMInterfaceAgent.virtual_network .

The Application and the Anomaly

We will use a script to simulate clients accessing the Redmine Webserver via the floating IP.
# while sleep $[ ( $RANDOM % 6 )  + 1 ]s ; do wget 10.84.53.83 -O /dev/null -o /dev/null ; done
Client access results in flows with clients, and between the Redmine Webserver and Database. This is visible in the VMI UVE, looking at the Analytics API:

Now, we will use the hping3 utility to generate a TCP SYN attack to the Floating IP:

# hping3 10.84.53.83 -i u100000 -S -p 80

After 30 seconds, the vRouter sees a change in flow metrics:

The added_flows attribute of the VMI UVE has been seen as 268, as the exponential mean has moved from 16.22 to 42.49 and sigma is 2.997. This triggers an alert.

In Contrail UI:

In the Analytics streaming API:

The Alert is against the VMI object, but its contents also report the “fe-front” Virtual Network and the “fe-web” VM.

Anomaly Detection Capabilities

We just saw an example of Anomaly Detection against the Networking Flows of a Virtual Machine Interface. Contrail Networking offers anomaly detection on other vRouter traffic metrics as well – Physical Interface Bandwidth and per-Physical Interface flows. In addition, we also do this for some controller metrics, such as BGP Route Updates.

All Alerts are configurable via the Contrail Configuration API.  To achieve scalability, the Anomaly Detection Algorithms run on the individual Contrail processes (contrail-vrouter-agent or contrail-control) that report the metrics. They can be adjusted on a per-process basis as required by changing configuration files.

Requirements: OpenContrail with OpenStack

What if you have an environment which needs to scale up during periods of peak loads & scale down during normal operations (Eg. Black-Friday). Can you have an automated solution which can handle this kind of scenario to save OpEx/CapEx?

The answer is YES! OpenContrail as the SDN controller provides a robust solution which will intelligently scale up & scale down the cloud-stack based on user-defined metrics. With this solution, cloud administrators define trigger points & the decision to scale is left to the cloud, thus resulting in an intelligent & elastic cloud.

In the previous blog, you have seen how we deploy a service chain using OpenContrail. In this blog, we will see how to enhance the existing service chain to implement the autoscaling feature which will scale up/down service-instances (Eg. Load-Balancers, Firewalls, etc.) using OpenContrail. An OpenStack project called as “Ceilometer” is used to generate alarms which will in turn trigger the scale up/down policies that are defined in the Heat Orchestration Template (HOT).

NOTE: OpenStack Heat is an OpenStack project which is used to orchestrate composite cloud applications using a declarative template format written in YAML through an OpenStack REST API.

OpenStack Ceilometer is an OpenStack project which provides metering, monitoring & alarming features that can be consumed by other OpenStack projects such as Heat.

Below, the video demonstrates how the resources from OpenContrail, Heat & Ceilometer work together to scale up/down service-instances belonging to a service chain. In theory, the process works as described below,

• Once a service-instance is created, Ceilometer constantly polls the virtual-machine for key performance metrics that are defined in the template. In our case, we specify the metric “average CPU utilization (avg, cpu_util)” of the service-instance cluster. The template also defines thresholds (low & high) to implement the autoscaling feature.

• Whenever the CPU utilization (load) of the service-instance in the cluster is greater than the upper-threshold value, Ceilometer raises an alarm (Alarm-High) & indicates Heat to instantiate & add a new service-instance to the cluster. Therefore, we now have two service-instances in the chain deployed in Active-Active configuration. This reduces the CPU utilization of the first service-instance since the load is now distributed across TWO service-instances in the cluster. If the average CPU utilization of the cluster is still greater than the upper-threshold value, the autoscaling policy adds another service-instance to the cluster. This autoscaling feature is indicated in the template by the “scaleup_policy”.Alternatively, if the average CPU utilization of the entire cluster is less than the lower-threshold value, Ceilometer raises an alarm (Alarm-Low) & indicates Heat to remove the newly added instances from the service-instance cluster. This is indicated in the template by the “scaledown_policy”.

• Therefore, we now have an intelligent elastic service-chain that can scale up/down based on a certain user-defined metric. This provides a higher scalability index along with low OpEx.

1. Some of the key resources & their properties used in the HOT are listed below,OS::ContrailV2::VirtualNetwork – A resource to create a Virtual Network
   OS::ContrailV2::VirtualMachineInterface – A resource to create a Virtual Machine Interface
   OS::ContrailV2::InstanceIp – A resource to allocate an IP address to an instance
   OS::ContrailV2::ServiceTemplate – A resource to create a Service Template
   OS::ContrailV2::ServiceInstance – A resource to create a Service Instance
   OS::ContrailV2::PortTuple – A resource to create a Service-Instance port tuple
   OS::ContrailV2::NetworkPolicy – A resource to create a Virtual-Network Policy
   OS::Heat::AutoScalingGroup – A resource that allows to create a desired number of similar resources.

   Properties:

   1. Max_size: Maximum number of resources (Service-Instances) in the group
   2. Min_size: Minimum number of resources (Service-Instances) in the group
   3. Desired_capacity: Desired initial number of resources

   OS::Heat::ScalingPolicy – A resource to manage scaling of OS::Heat::AutoScalingGroup

   Properties:

   1. Adjustement_type: Type of adjustement (Percentage or Absolute)
   2. Auto_scaling_group_id: Group ID to which the autoscaling policy is applied
   3. Scaling_adjustment: Size of adjustment (+N for scale-up by N, -N for scale-down by N)

   OS::Ceilometer::Alarm – A resource used to raise an alarm

   Properties:

   1. Meter_name: The name of the meter. (cpu_util, image, memory)
   2. Statistic: Meter statistic to evaluate (avg, sum, min, max, count)
   3. Period: Period to evaluate over (in seconds)
   4. Evaluation_periods: Number of periods to evaluate over
   5. Threshold: Threshold to evaluate against
   6. Comparison_operator: Operator used to compare specified statistic with threshold (gt, lt, eq)

Last week, we had a blast attending OpenStack Summit, and we hope you did too! One theme that we saw, and resonated with, from day one of the summit was Interoperability. The basis of OpenStack, and its interoperability, is openness: open source, open community, open development, and open design. OpenContrail, a plug-in for OpenStack Neutron, shares this foundational element.

OCUG Barcelona 2016During OpenStack Summit, we hosted OpenContrail user group. Edgar Magana, member of OpenStack board of directors and OpenContrail Advisory Board member, kicked off the session for an open discussion with six OpenContrail users: Deutsche Telekom, Verizon Enterprise Solutions, Orange Business Services, Avi Networks, Netronome and AppOrbit. Each company talked about how OpenContrail enables their different use cases – from agile telco cloud for SaaS, to application transformation with containers. With over 200 attendees, the resonating lesson was that OpenContrail is incredibly flexible and mature enough to enable transformation, whether users are dealing with bare-metal servers, virtual machines, or containers, in public, private, or hybrid cloud environments.

In other words, OpenContrail is one networking controller that can work seamlessly across heterogeneous networking environments. In Rudra Rugge’s talk at the OpenStack Demo Theater, he gives a captivating explanation of the OpenContrail solution (which provides security in addition to seamless interoperability with any compute vehicle and underlay, analytics and troubleshooting capabilities, single-pane-of-glass management, etc. ).

We had several other demos at our booth, which we’ll be releasing shortly, in addition to the OpenContrail user group footage. We’ll post them to OpenContrail.com/videos. Don’t forget to follow us @OpenContrail for updates!

If you’re hoping to try OpenContrail for yourself after hearing from other users, play with it here risk-free!

The OpenStack Foundation has recorded and shared all breakout sessions and posted them on their YouTube channel. The list of our sessions can be found here.

Summary:

MPLS-OVER-UDP Tunnels are used on datacenter environment as overlays. Existing technologies (like MPLS-OVER-GRE) to encapsulate Multi-Protocol Label Switching(MPLS) over IP are not adequate for efficient load balancing of MPLS application traffic, such as Layer3 Virtual Private Network (L3VPN) traffic across IP networks. This document specifies IP-based encapsulation technology, referred to as MPLS-in-User Datagram Protocol (UDP), which can facilitate the load balancing of MPLS application traffic across IP networks. This document also gives details on how to enable MPLS-OVER-UDP encapsulation when MX router interop with contrail controller.

Description:

Figure 1 shows the frame formats of MPLS-OVER-GRE and MPLS-OVER-UDP.There no easy way to load share the traffic between 2 tunnel end points when MPLS-OVER-GRE encapsulation is used.The issue is GRE header is same for all the flows between the 2 Tunnel end points.

But when we use MPLS-OVER-UDP encapsulation the source port value(in the MPLS-OVER-UDP header) can be generated based on the certain fields in the customer packets(in our case based on the packets generated by VMs in the vRouter).With this the routers between the Tunnel end points can load share the packets using hash of five-tuple of UDP packets.This is one of reason why MPLS-OVER-UDP overlays preferred over MPLS-OVER-GRE overlays.

Figure 1: IP over MPLS over UDP Packet Format

Figure 2: IP over MPLS over GRE Packet Format

The picture given above shows the implementation of Contrail cloud solution. In this topology PE2 & PE3 will be acting as local Datacenter gateways.PE1 will be the remote Datacenter gateway.Between PE2/PE3 and compute nodes (or vRouters) MPLS-OVER-UDP overlays will be implemented.So all traffic between compute nodes to PE2/PE3 will be encapsulated with MPLS-OVER-UDP header.Contrail controllers will be managing the vRouters using XMPP.PE2 &PE3 will talk to Contrail controllers with BGP.The vRouters in contrail cloud will act similar to PE router in L3 VPN environment.When vRouter need to send traffic to vRouters in another datacenter it will encapulate packets with MPLS-OVER-UDP header and forward it to PE2 or PE3.PE2 or PE3 will decapsulate the MPLS-OVER-UDP header and then encapsulate it with MPLS headers and forward it to PE1.PE1 decapsulate MPLS header and forward the packets to the node in remote Datacenter.

When a VM in a vRouter need to send traffic to VM in another vRouter in the same datacenter, the source vRouter will encapsulate the packets with MPLS-OVER-UDP/MPLS-OVER-GRE header and then forward it to the IP fabric.After the destination vRouter receives the packet it will decapsulate the packets and forward it to the destination VM.

Configuration on MX gateway:

BGP Configuration :

BGP group contrail:

This section gives details on BGP configuration of PE2 & PE3.Both the PEs will have a similar configuration. BGP group “contrail” (given below) peers to contrail controller. In this configuration, Family route-target knob controls the advertisement of routes from PE2 & PE3 to contrail controller.It makes sure that PE2 & PE3 advertise only required routes based on the route-targets received from the contrail controller.

Contrail controller advertises the route targets(which is applied to the virtual Networks created in contrail controller) to PE2 & PE3.PE2 & PE3 advertise routes with matching route targets. Unmatched routes will not be advertised. This avoids unnecessary routes advertised to contrail controller. The knob external-paths 5 is required when you have 3 controllers in a HA environment.

Before advertising any routes to contrail controller the next-hop of the route (through policy from-remote-pe-vrf1) is changed to PE2 or PE3 loopback address.This avoids the need for the controller to learn remote PE1 loopback address.

This policy also adds encapsulation extended community to the remote datacenter route before advertising it to the contrail controller.This community tells the vRouter(compute nodes) to use MPLS-OVER-UDP encapsulation before it forwards traffic for this route.If this community is not advertised,  then vRouter will use the default encapsulation (which is MPLS-OVER-GRE).

The encapsulation community  is in the format of “members 0x030c:XXX:13”. This identifies it as opaque extended community (type 0x03) of sub-type encapsulation (0x0c).Administrator field is set to 0 since it’s reserved(But JUNOS do not allow 0 so you need to set it to some value recommended is AS NUMBER) and encap value is 13 (for MPLSoUDP).

The configuration given below is used in PE2 and PE3 routers.

{master}[edit]
regress@PE2# show protocols bgp group contrail     
type internal;
local-address 10.255.181.172;
family inet-vpn {
    any;
}
family inet6-vpn {
    any;
}
family route-target {
    external-paths 5;
    advertise-default;
}
export from-remote-pe-vrf1;
vpn-apply-export;
cluster 2.2.2.2;
neighbor 3.3.3.2;

{master}[edit]
regress@PE2# 

master}[edit]
regress@PE2# show policy-options policy-statement from-remote-pe-vrf1
term 1 {
    from {
          protocol bgp;
          route-filter 103.0.4.0/24 orlonger;
             }
    then {
        next-hop self;
        accept;
    }
}

{master}[edit]
regress@PE2# show policy-options community udp         
members 0x030c:64512:13;

{master}[edit]
regress@PE2# 

You can verify encapsulation status in vRouter(in compute node) as shown below.For this first find the tap interface.Then check the route using the vrf number.This output shows the type of encapsulation, source and destination IPs of the tunnel and other details.

root@vm6:~# vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/3      OS: tap6b9bb806-20
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:1 Flags:PL3L2D MTU:9160 Ref:5
            RX packets:15533540  bytes:4717521394 errors:0
            TX packets:15659512  bytes:4721801850 errors:0


root@vm6:~# rt --dump 1 |grep 103.0.8.0/24
103.0.8.0/24           24            P          -             24        -
root@vm6:~# nh --get 24
Id:24         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:6          Vrf:1
              Flags:Valid, Policy, Ecmp, 
              Sub NH(label): 16(17) 12(17)


Id:12         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:0
              Flags:Valid, MPLSoUDP, 
              Oif:0 Len:14 Flags Valid, MPLSoUDP,  Data:02 00 08 00 00 2b 52 54 00 23 13 39 08 00 
              Vrf:0  Sip:60.60.0.6  Dip:10.255.181.172

root@vm6:~# 

The picture given below shows the bgp configuration in Contrail controller.There are 2 nodes shown here.Router Type “BGP Router” shows the PE1 configuration details and Router Type “Control Node” details BGP configuration of Contrail Controller.

The output from MX router (given below) shows that bgp connection established between MX and Contrail Controller.

{master}[edit]
regress@PE2# run show bgp summary 
Groups: 4 Peers: 5 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.rtarget.0        
                      17         13          0          0          0          0
bgp.l3vpn.0          
                      58         54          0          0          0          0
bgp.l3vpn.2          
                       0          0          0          0          0          0
bgp.l3vpn-inet6.0    
                       0          0          0          0          0          0
bgp.l3vpn-inet6.2    
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
3.3.3.2               64512         40         56       0       0       11:15 Establ
  bgp.rtarget.0: 13/17/17/0
  bgp.l3vpn.0: 4/4/4/0
  bgp.l3vpn-inet6.0: 0/0/0/0
  vrf1.inet.0: 4/4/4/0

BGP Group core:

This BGP Group “core” peers to the remote PE(PE1) connected to remote data center. The policy “change-next” will change the next hop off routes advertised to remote PE1.

{master}[edit] 
regress@PE2# show protocols bgp group core 
type internal;
local-address 10.255.181.172;
family inet-vpn {
    any;
}
family inet6-vpn {
    any;
}
export change-next;
vpn-apply-export;
neighbor 10.255.178.174;

{master}[edit]
regress@PE2# 


{master}[edit]
regress@PE2# show policy-options policy-statement change-next 
Dec 07 14:27:33
term 1 {
    from protocol bgp;
  then {
        next-hop self;
        accept;
    }
}

{master}[edit]
regress@PE2#

Dynamic tunnel configuration:

This section explains Dynamic Tunnel Configuration between MX to Contrail Controller and MX to vrouter.

Dynamic Tunnel to contrail controller:

PE2 & PE3 needs to have dynamic tunnel (MPLS-OVER-UDP/MPLS-OVER-GRE) created to the contrail controller. This configuration will create route to the controller ip address in inet.3 table.Without this route MX will not advertise bgp.l3vpn.0 table routes to controller(when you have family route-target enabled on PE2 & PE3).This Tunnel status can be verified as shown below.This tunnel will be in Dn state as there are no routes received from controller with protocol next-hop of contrail controller IP address(in this case it is 3.3.3.2).This is expected as contrail controller will not be advertising any route with 3.3.3.2 as protocol next hop.

{master}[edit]
regress@PE2# show routing-options dynamic-tunnels to-controller       
source-address 10.255.181.172;
udp;
destination-networks {
    3.3.3.0/24;
}

{master}[edit]
regress@PE2# 


{master}[edit]
regress@PE2# run show dynamic-tunnels database terse 
Table: inet.3

Destination-network: 3.3.3.2/32
Destination         Source          Next-hop                  Type        Status
3.3.3.2/32          10.255.181.172  0x487e67c nhid 0          udp         Dn    nexthop not installed

master}[edit]
regress@PE2# run show route 3.3.3.0
Dec 07 13:53:43 

inet.0: 43 destinations, 44 routes (42 active, 0 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both

3.3.3.0/24         *[OSPF/150] 00:40:14, metric 0, tag 0
                    > to 1.0.1.1 via ae0.0

inet.3: 18 destinations, 31 routes (18 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

3.3.3.0/24         *[Tunnel/300] 00:37:18
                      Tunnel

{master}[edit]
regress@PE2# 

Dynamic Tunnels to vRouters:

PE2 & PE3 should create Dynamic TUNNELS to all the vROUTERs. In our case it is MPLS-OVER-UDP TUNNEL created between PE2 & PE3 to all vROUTERs.When PE2 or PE3 receives traffic from remote PE1 it uses this MPLS-OVER-UDP tunnel to forward the traffic to the respective vROUTER.The Tunnel status can be verified as given below.The Tunnel will come up only when a route received with the protocol nexthop in the segment 60.60.0.0/16(which is configured in destination-networks as shown below).In this case 60.60.0.0/16 is the IP segment used vrouters.

regress@PE2# show routing-options dynamic-tunnels 
contrail-udp {
    source-address 10.255.181.172;
    udp;
    destination-networks {
       60.60.0.0/16;
    }
}
{master}[edit]
regress@PE2# run show dynamic-tunnels database 
Table: inet.3
Destination-network: 60.60.0.6/32
Tunnel to: 60.60.0.6/32
  Reference count: 1
  Next-hop type: UDP
    Source address: 10.255.181.172 Tunnel Id: 1610612742
    Next hop: tunnel-composite, 0x4875634, nhid 710
      Reference count: 2
      State: Up


{master}[edit]
regress@PE2# run show dynamic-tunnels database terse 
Table: inet.3

Destination-network: 60.60.0.6/32
Destination         Source          Next-hop                  Type        Status
60.60.0.6/32        10.255.181.172  0x48793f4 nhid 741        udp         Up   

This is the Routing instance configuration is created on PE2 & PE3 to install the routes to vrf table.This vrf configuration is required as it is a L3VPN scenario.

{master}[edit]
regress@PE2# show routing-instances vrf1 
instance-type vrf;
interface lo0.1;
route-distinguisher 64512:1;
vrf-import test1-import;
vrf-export test1-export;
vrf-table-label;
}

{master}[edit]
regress@PE2# show policy-options policy-statement test1-export
term 1 {
    from protocol  direct ;
    then {
        community add testtarget1;
        accept;
    }
}

{master}[edit]
regress@PE2# 

regress@PE2# show policy-options policy-statement test1-import 
term 1 {
    from community testtarget1;
    then accept;
}

{master}[edit]
regress@PE2#

Remote datacenter PE configuration:

This configuration is applied on the remote PE router (in this case PE1).This is a simple L3VPN configuration.

[edit]
regress@PE1# show routing-instances vrfs1 
Dec 06 13:21:03
instance-type vrf;
interface xe-1/1/1.1;
interface xe-1/1/1.2;
interface xe-1/1/1.3;
interface xe-1/1/1.4;
route-distinguisher 64512:1;
vrf-import test1-import;
vrf-export test1-export;
vrf-table-label;

[edit]
regress@PE1# show policy-options policy-statement test1-export 
Dec 06 13:21:10
term 1 {
    from protocol direct;
    then {
        community add testtarget1;
        accept;
    }
}

[edit]
regress@PE1# show policy-options policy-statement test1-import    
term 1 {
    from community testtarget1;
    then accept;
}

[edit]
regress@PE1# 

[edit]
regress@PE1# show protocols bgp
precision-timers;
group contrail-1 {
    type internal;
    local-address 10.255.178.174;
    family inet {
        unicast;
    }
    family inet-vpn {
        any;
    }
    family inet6-vpn {
        any;
    }
    cluster 3.3.3.3;
    neighbor 10.255.181.172;
        
[edit]
regress@PE1# 

Conclusion

In this document we have shown the use case of MPLS-OVER-UDP overlays. In addition to this configuration required to integrate MPLS-OVER-UDP overlays between MX and contrail controller is explained in this document.

APPENDIX A

The tunnel details from the FPC can be checked as shown below.The nexthop id can be used to check the tunnel details from the FPC.The Tunnel id,tunnel destination,Tunnel source in the FPC should match with the CLI.

{master}[edit]
regress@PE2# run show dynamic-tunnels database 
Dec 06 14:06:45
Table: inet.3

Destination-network: 60.60.0.6/32
Tunnel to: 60.60.0.6/32
  Reference count: 1
  Next-hop type: UDP
    Source address: 10.255.181.172 Tunnel Id: 1610612742
    Next hop: tunnel-composite, 0x4875634, nhid 710
      Reference count: 2
      State: Up

regress@PE2# run request pfe execute target fpc3 command "show nhdb id 710 extensive"       
Dec 07 13:58:26 
================ fpc3 ================
SENT: Ukern command: show nhdb id 710 extensive

   ID      Type      Interface    Next Hop Addr    Protocol       Encap     MTU               Flags  PFE internal Flags
-----  --------  -------------  ---------------  ----------  ------------  ----  ------------------  ------------------
  710    Compst  -              -                      MPLS             -     0  0x0000000000000000  0x0000000000000000

BFD Session Id: 0

Composite NH:
  Function: Tunnel Function
  Hardware Index: 0x0
  Composite flag: 0x0
  Composite pfe flag: 0xe
  Lower-level NH Ids:
  Derived NH Ids:
  Tunnel Data:
      Type     : UDP-V4
      Tunnel ID: 1610612742
      Encap VRF: 0
      Decap VRF: 0
      MTU      : 0
      Flags    : 0x2
      Encap Len: 28
      Encap    : 0x45 0x00 0x00 0x00 0x00 0x00 0x40 0x00
                 0x40 0x2f 0x00 0x00 0xac 0xb5 0xff 0x0a
                 0x06 0x00 0x3c 0x3c 0x00 0x00 0x00 0x00
                 0x00 0x00 0x00 0x00
      Data Len : 8
      Data     : 0x3c 0x3c 0x00 0x06 0x0a 0xff 0xb5 0xac
        Feature List: NH
           [pfe-0]: 0x08ce6d4c00100000;
           [pfe-1]: 0x08c12e3000100000;
          f_mask:0xe000000000000000; c_mask:0xe000000000000000; f_num:3; c_num:3,  inst:0xffffffff
        Idx#0          -:               
           [pfe-0]: 0x2bfffffd5e00a500  
           [pfe-1]: 0x2bfffffd5e00a500  
                                        
        Idx#1          -:               
           [pfe-0]: 0x23fffffc0000000c  
           [pfe-1]: 0x23fffffc0000000c  
                                        
        Idx#2          -:               
           [pfe-0]: 0x08c045d800080000  
           [pfe-1]: 0x08c03d8800080000  
                                        
Tunnel ID 1610612742                    
==============                          
         Ref-count 1                    
TunnelModel:                            
Dynamic Tunnel Model:                   
         Name = MPLSoUDP                
         MTU = 0                        
         VRF = default.0(0)             
         Source Entropy = 1             
         Packets = 0 Bytes = 0          
                                        
Source IP     : 10.255.181.172          
Destination IP: 60.60.0.6     


Ingress:                                
Index:0                                 
  PFE(0): 0x2bfffffd5e006500            
  PFE(1): 0x2bfffffd5e006500            
                                        
Index:1                                 
  PFE(0): 0x8ce6c8c00100000             
  PFE(1): 0x8c12d7000100000             
                                        
Handle JNH
0x8c045d800080000
0x8c03d8800080000
Egress:
Index:0
  PFE(0): 0x2bfffffd5e008500
  PFE(1): 0x2bfffffd5e008500

Index:1
  PFE(0): 0x23fffffc0000020a
  PFE(1): 0x23fffffc0000020a

Index:2
  PFE(0): 0x878b15400100000
  PFE(1): 0x878b2b000100000

Handle JNH
0x8ce6cec00100000
0x8c12dd000100000


  Routing-table id: 0

Full configuration:

PE2:

{master}[edit]
regress@PE2# show protocols ospf
area 0.0.0.0 {
    interface all {
        bfd-liveness-detection {
            minimum-interval 300;
            multiplier 3;
        }
    }
    interface fxp0.0 {
        disable;
    }
}

{master}[edit]
regress@PE2# 

regress@PE2# show protocols mpls
ipv6-tunneling;
interface all;
{master}[edit]
regress@PE2# 

egress@PE2# show protocols ldp;
interface ae0.0;
interface ae1.0;
interface ae2.0;
interface lo0.0;
{master}[edit]
regress@PE2#
 
{master}[edit]
regress@PE2# show protocols bgp group contrail   
type internal;
local-address 10.255.181.172;
family inet-vpn {
    any;
}
family inet6-vpn {
    any;
}
family route-target {
 external-paths 5;
 advertise-default;
}
export from-remote-pe-vrf1;
cluster 2.2.2.2;
neighbor 3.3.3.2;
{master}[edit]
regress@PE2# 

{master}[edit]
regress@PE2# show protocols bgp group core        
type internal;
local-address 10.255.181.172;
family inet-vpn {
    any;
}
family inet6-vpn {
    any;
}
export change-next;
vpn-apply-export;
neighbor 10.255.178.174;
{master}[edit]
regress@PE2# 

regress@leopard# show routing-options              
Dec 13 12:19:18
ppm {
    redistribution-timer 120;
}
nonstop-routing;
autonomous-system 64512;
dynamic-tunnels {
    gre next-hop-based-tunnel;
    controller {
        source-address 10.255.181.172;
        udp;
        destination-networks {
            3.3.3.2/32;
        }
    }
contrail-udp {
        source-address 10.255.181.172;
        udp;
        destination-networks {
            60.60.0.0/16;
  }
}
}
regress@PE2# show policy-options policy-statement from-remote-pe-vrf1
term 1 {
    from {
          protocol bgp;
          route-filter 103.0.4.0/24 orlonger;
             }
    then {
        next-hop self;
        accept;
    }
}

{master}[edit]
regress@PE2# show policy-options community udp         
members 0x030c:64512:13;

{master}[edit]
regress@PE2# 
{master}[edit]
regress@PE2# show policy-options policy-statement change-next 
Dec 07 14:27:33
term 1 {
    from protocol bgp;
  then {
        next-hop self;
        accept;
    }
}

{master}[edit]
regress@PE2#

Executive summary:

Contrail Cloud solution is an open cloud network automation product that uses software-defined networking (SDN) technology to orchestrate the creation of virtual networks with high scalability. It exposes a set of REST APIs for northbound interaction with cloud orchestration tools, as well as other applications. Contrail Cloud solution has multiple components like Contrail controller, Openstack controller and vRouters.

This product creates Virtual Machines on the compute nodes, attach Virtual Machine to the virtual network and connect the Virtual Machine to the storage.

This Document gives details on REST APIs to automate the following,

• Verifying hypervisor status
• Create Virtual Machine or delete Virtual Machine
• Create virtual Networks
• Attach Virtual Machine to virtual network
• Verifying Virtual Machine status
• Get information about existing Virtual Machines

Description:

In contrail cloud solution Openstack server is responsible for creation of Virtual Machines.Openstack

server has inventory of all the Hypervisors.When a VM creation is requested through openstack Horizon GUI or through RESTAPI commands,NOVA component in openstack controller will connect to the Hypervisor (which has sufficient resources to create VM) and request for VM creation.

After VM is created, NEUTRON component in Openstack controller will connect to the Contrail controller and request for attaching the VM to the specefic Virtual Network.Contrail controller will then connect to the vrouter in the  Hypervisor (through XMPP protocol) and request for routing instance creation for the specefic virtual network.The same sequence operation repeated for each VM creation.

When the VMs are created on multiple hypervisors.They exchange routes between them.Then MPLS-OVER-GRE/MPLS-OVER-UDP Tunnels created between them.

RESTAPIs can be used for automating VM creation, VM deletion and other operations described.

This picture shows the implementation details of Contrail cloud solution.In this topology PE2 & PE3 will be acting as local Datacenter gateway.PE1 will be the remote Datacenter gateway.

All compute nodes, controllers are attached to IP fabric.The IP Fabric is not aware of the virtual Networks to which the virtual machines are attached.There is no need for the IP fabric to know virtual network details.Between PE2 (&PE3) and compute nodes and between the compute nodes TUNNEL overlays is implemented.So all traffic between compute nodes to PE2/PE3 or between compute nodes will be encapsulated with TUNNEL header.IP fabric is aware of the IP addresses present in the Tunnel header(as these IP addresses are part of underlay/IP fabric network.

Contrail controllers will be managing the compute nodes (vRouters) using XMPP.PE2 &PE3  will talk to Contrail controllers with BGP.The vRouters in contrail cloud will act similar to PE router in L3 VPN environment.

When a VM in a vRouter need to send traffic to VM in another vRouter in the same datacenter,

the source vRouter will encapsulate the packets with Tunnel header and then forward it to the IP fabric.After the destination vRouter receives the packet it will decapsulate the packets and forward it to the VM.

When vRouter need to send traffic to vRouters in another datacenter it will encapulate packets with Tunnel header and forward it to PE2 or PE3.PE2 or PE3 will decapsulate the MPLS-OVER-UDP header and then encapsulate it with MPLS headers and forward it to PE1.PE1 decapsulate MPLS header and forward the packets to the node in remote Datacenter.

Automating Contrail/Openstack Controller:

This section gives details on Contrail/Openstack controller Automation.A sample perl code with RESTAPI details given for each operations like virtual network creation,VM creation,VM deletion etc..Perl module LWP::UserAgent and use JSON qw( decode_json encode_json) should be sourced for this.

Virtual Machine creation:

OpenStack Token-id and Tenant id:

This section gives details on automating VM creation.Before starting VM creation we need to get the token-id and tenant id from the openstack controller.This token id is required to execute any operational commands in the openstack controller.Tenant id is required to find the right openstack project.The sample code given below gives details on this . The Openstack component keystone provides this token-id.

Keystone is the identity service used by OpenStack for authentication (authN) and high-level authorization (authZ). It currently supports token-based authN and user-service authorization.

In this sample code “$OS_tenant_name” is the project name.This name can be found on openstack Horizon GUI(identity->Projects).Project name is required to find the correct tenant id.The “$CMDpath” shows  the syntax of RESTAPI command to get token-id from openstack controller.The tenant id and token id collected here will be used in next steps.

my openstack_ip = ‘1.1.1.1’;
      	     	my  $KEYSTONEport     =  "5000";               
        		my $OS_tenant_name = "PROJECTNAME";
        		my $OS_user = "admin";
        		my $OS_password = ‘xxxxxx’;
                 	my $CMDpath= "/v2.0/tokens";
        	my $APIport=$KEYSTONEport;
        	my $url= "http://".$openstack_ip.":".$APIport.$CMDpath;
             my $body =
        	{
               	"auth"   =>
                      	   {
                         	       "tenantName"  => "$OS_tenant_name",
                               	 "passwordCredentials" =>
                                                {
                                              	  "username"  => "$OS_user",
                                              	  "password"  => "$OS_password"
                                                }
                          }                     
         };               
        	my $request  = HTTP::Request->new( POST => $url );
        	my $bjson = encode_json $body;
        	$request->content_type("application/json");
        	$request->authorization_basic( "$OS_user", "$OS_password" );
        	$request->content($bjson);
        	my $http = LWP::UserAgent->new->request($request);
        	my @y = $http->{_content};
        	my $mes         = JSON::decode_json($y[0]);
         
        	my $token_id = $mes->{'access'}{'token'}{'id'};
        	my $token_exp = $mes->{'access'}{'token'}{'expires'};
        	my $tenant_id = $mes->{'access'}{'token'}{'tenant'}{'id'};
        	print("\n\n *******  token_id is $token_id and token_exp is $token_exp and tenant_id is $tenant_id ******* \n\n");

Verifying Hypervisor status:

Before going to next step we need to verify the status of compute nodes(Hypervisors).Before creating the VM,the Hypervisors should be Up and running.The RESTAPI command “/v1.1/”.$tenant_id.”/os-services” provides the status of the hypervisors.This is equivalent to “nova service-list command(the output given below).

my $NOVAport         =  "8774";
       $CMDpath= "/v1.1/".$tenant_id."/os-services";
        $APIport=$NOVAport;
        $url= "http://".$cont_ip.":".$APIport.$CMDpath;
        $request  = HTTP::Request->new( GET => $url );
        $request->header('content-type' => 'application/json');
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);
        @y = $http->{_content};
        print Dumper($http);    
        $mes         = JSON::decode_json($y[0]);
        my $len = @{$mes->{"services"}};          
        my %server_to_netaddr_hash;
        my %test;
        my $servername;
        my $state;
        my $status; 
        my $hostcount=0;
        for (my $x = 0; $x< $len ; ++$x) { $servername = $mes->{"services"}[$x]{"host"};
                        if( ($servername =~/vmg\d+/) || ($servername =~/vm\d+/) ) {
	        $state = $mes->{"services"}[$x]{"state"};
                         my $status = $mes->{"services"}[$x]{"status"};
                         if ($state ne "up" || $status ne "enabled") {
        print("\n\n *******  status or state of host $servername is not correct.The state is $state and status is $status ******* \n\n");
         return JT::FALSE;
                        }
        print ("\n\n *******  status or state of the compute node $servername is correct The state is $state and status is $status ******* \n\n");
                    $hostcount++;

                   }
	}

root@server01:~# nova service-list
+----+------------------+--------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host         | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+--------------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-scheduler   | server1 | internal | enabled | up    | 2016-12-09T21:32:39.000000 | -               |
| 2  | nova-console     |  server1 | internal | enabled | up    | 2016-12-09T21:32:42.000000 | -               |
| 3  | nova-consoleauth | server1 | internal | enabled | up    | 2016-12-09T21:32:42.000000 | -               |
| 4  | nova-conductor   | server1 | internal | enabled | up    | 2016-12-09T21:32:38.000000 | -               |
| 6  | nova-compute     | vm6          | nova     | enabled | up    | 2016-12-09T21:32:38.000000 | -               |
| 7  | nova-compute     | vm7          | nova     | enabled | up    | 2016-12-09T21:32:35.000000 | -               |
| 8  | nova-compute     | vm8          | nova     | enabled | up    | 2016-12-09T21:32:38.000000 | -               |
| 9  | nova-compute     | vm9          | nova     | enabled | up    | 2016-12-09T21:32:33.000000 | -               |
| 10 | nova-compute     | vm10         | nova     | enabled | up    | 2016-12-09T21:32:41.000000 | -               |
| 11 | nova-compute     | vm11         | nova     | enabled | up    | 2016-12-09T21:32:41.000000 | -               |
| 12 | nova-compute     | vm12         | nova     | enabled | up    | 2016-12-09T21:32:33.000000 | -               |
| 13 | nova-compute     | vm13         | nova     | enabled | up    | 2016-12-09T21:32:37.000000 | -               |
| 14 | nova-compute     | vm14         | nova     | enabled | up    | 2016-12-09T21:32:41.000000 | -               |
| 15 | nova-compute     | vm15         | nova     | enabled | up    | 2016-12-09T21:32:39.000000 | -               |
| 16 | nova-compute     | vm16         | nova     | enabled | up    | 2016-12-09T21:32:42.000000 | -               |
| 17 | nova-compute     | vm17         | nova     | enabled | up    | 2016-12-09T21:32:33.000000 | -               |
| 18 | nova-compute     | vm18         | nova     | enabled | up    | 2016-12-09T21:32:40.000000 | -               |
+----+------------------+--------------+----------+---------+-------+----------------------------+-----------------+
root@ server01:~# 

Creating Virtual Network:

The next step is to create virtual Network. Virtual network has to be created before creating VM.

To create virtual Network the following information is required.

• Network segment address and netmask
• Gateway address
• Network name
• Route-target attached to this network
• Project name (collected from openstack horizon GUI)
• Token-id (as explained in previous step)

To create virtual network ,the RESAPI command “/virtual-networks” is executed on the contrail controller.Port number 8082 is used for this.The correct project name should be added here where it says “PROJECTNAME”.”nova net-list” shows the network created as shown below.

$netname="net$i";
		$cont_ip=”1.1.1.1”;
		$target="target:64512:$value";
		$gateway="100.$n.$m.1";
		$pool = "100.$n.$m.0";
       		 $CMDpath = "/virtual-networks";
        		$APIport="8082";
       		 $url= "http://".$cont_ip.":".$APIport.$CMDpath;
          		$body =
              		{
               			"virtual-network"=>
                 			  {
                  				 "parent_type"=> "project",
                 				  "fq_name"=> [
                             			 "default-domain",
                              			 "PROJECTNAME",
                              			  	$netname ],

                    			"route_target_list"=>
                               		 {
                                			 "route_target"=> [
								$target
							      ]
				  },

				"network_ipam_refs"=> [{
                                         					  "attr"=> {
                                                 				   	  "ipam_subnets"=> [{
                                                                 				     "subnet"=> {
                                                                                	  	 "gateway_ip" => $gateway,
                                                                                 		 "ip_prefix"=> $pool,
                                                                                 		"ip_prefix_len"=> 24}}]},
                                          					  "to"=> [
                                                				     	"default-domain",
                                                    				     	"default-project",
                                                      				 "default-network-ipam"]}]}
            			   };

		$request  = HTTP::Request->new( POST => $url );
        		$bjson = encode_json $body;
       	 	$request->content_type("application/json");
       	 	$request->header('x-auth-token' => $token_id);
       	 	$request->content($bjson);
        		$http = LWP::UserAgent->new->request($request);

root@server1:~# nova net-list
+--------------------------------------+-------------------------+------+
| ID                                   | Label                   | CIDR |
+--------------------------------------+-------------------------+------+
| 363b4323-69d3-404a-b7b2-6587b4507479 | net6                    | None |
| 274b0b81-52be-48f9-8888-0c6be1d910c2 | net9                    | None |
| 72f26b86-f0bf-44c0-aadb-5f94c34b5eeb | net4                    | None |
| 72be70e4-c0d8-46ce-be8c-33d80f095c5a | default-virtual-network | None |
| aa0f528e-f276-4d5a-a02f-bb02f7289c9c | net10                   | None |
| ce1e9035-9c4d-49de-a867-2f4a27bd5691 | net13                   | None |
| b8599366-f1c0-4c2f-bab2-34533902466d | net12                   | None |
| b8543c48-872d-4e25-9de7-658f304c032a | net7                    | None |
| 34168c65-a4c8-439a-beda-3c7a1f380176 | net3                    | None |
| 7a244326-fc1e-416c-98fb-cc073f8762a0 | test                    | None |
| e7b4e0b9-6fc9-497f-9899-4e2aa344a97b | net11                   | None |
| d669a490-53f4-4fac-bed2-b79fe8538eae | __link_local__          | None |
| f5307953-9490-4da6-b365-7173035d69d3 | net5                    | None |
| a9eac604-a634-43df-8b9f-ea7258941313 | net8                    | None |
| a8363373-548d-4f78-8366-bd0277be8b80 | net1                    | None |
| c4d757cb-d2ab-4334-8e80-029cb0c31610 | ip-fabric               | None |
| fac53b08-59e7-45c6-9ef6-6007160979bb | net2                    | None |
+--------------------------------------+-------------------------+------+
root@ server1:~# 

VM Flavor id/VM Image id/virtual Network id Details:

Before creating the VM we need the following information.There are separate RESTAPI commands available to collect each of this information.

• VM name
• VM flavor id
• VM image id
• Virtual network id
• VM security group
• Tenant name( or project name)

VM Flavor id:

To get the VM flavor id the following code is used.The RESTAPI command “/v1.1/”.$tenant_id.”/flavors” returns the VM flavor id.This RESTAPI command is handled by NOVA component of OpenStack controller.

My $NOVAport         =  "8774";
           My $openstack_ip        =”1.1.1.1”;
	my $headers = "{ 'Content-Type' : 'application/json', 'Accept'  :  'application/json' , 'ser-Agent'  : 'python-novaclient', 'X-Auth-Token'   :    $token_id}";
        $CMDpath= "/v1.1/".$tenant_id."/flavors";
        $APIport=$NOVAport;
        $url= "http://". $openstack_ip.":".$APIport.$CMDpath;
        $request  = HTTP::Request->new( GET => $url );
        $request->header('content-type' => 'application/json');
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);
        @y = $http->{_content};
        $mes         = JSON::decode_json($y[0]);
        my $len = @{$mes->{'flavors'}};
        my $flavor_id;
        for (my $x = 0; $x< $len ; ++$x) { if ($mes->{'flavors'}[$x]{'name'} eq $add_vm_flavor) {
                        $flavor_id = $mes->{'flavors'}[$x]{'id'};
                        last;
                }
        }
        print ( "\n\n *******  image id for $add_vm_flavor is $flavor_id ******* \n\n");

VM image id:

To get the image id this code is used.The RESTAPI command “/v1.1/”.$tenant_id.”/images”returns the image id.This RESTAPI command is handled by NOVA component of openstack controller.

my $headers = "{ 'Content-Type' : 'application/json', 'Accept'  :  'application/json' , 'ser-Agent'  : 'python-novaclient', 'X-Auth-Token'   :    $token_id}";
        $CMDpath= "/v1.1/".$tenant_id."/images";
        $APIport=$NOVAport;
        $url= "http://".$cont_ip.":".$APIport.$CMDpath;
        $request  = HTTP::Request->new( GET => $url );
        $request->header('content-type' => 'application/json');
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);
        @y = $http->{_content};
        $mes         = JSON::decode_json($y[0]);
        my $len = @{$mes->{'images'}};
        my $image_id;
        for (my $x = 0; $x< $len ; ++$x) { if ($mes->{'images'}[$x]{'name'} eq $add_vm_image ) {
                        $image_id = $mes->{'images'}[$x]{'id'};
                        last;
                }
        }
        print ("\n\n *******  image id for $add_vm_image is $image_id ******* \n\n");

Virtual Network ID:

The RESTAPI command “/v1.1/”.$tenant_id.”/os-tenant-networks” returns the virtual network id.This RESTAPI command is handled by NOVA component of openstack controller.The network id of the networkname “$add_vm_net” can be identified by this code.

my $headers = "{ 'Content-Type' : 'application/json', 'Accept'  :  'application/json' , 'ser-Agent'  : 'python-novaclient', 'X-Auth-Token'   :    $token_id}";
        $CMDpath= "/v1.1/".$tenant_id."/os-tenant-networks";
        $APIport=$NOVAport;
        $url= "http://".$cont_ip.":".$APIport.$CMDpath;
        $request  = HTTP::Request->new( GET => $url );
        $request->header('content-type' => 'application/json');
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);
        @y = $http->{_content};
        $mes         = JSON::decode_json($y[0]);
        my $len = @{$mes->{'networks'}};
        my $network_id;
        for (my $x = 0; $x< $len ; ++$x) { if ($mes->{'networks'}[$x]{'label'} eq $add_vm_net ) {
                        $network_id = $mes->{'networks'}[$x]{'id'};
                        last;
                }
        }
        print ("\n\n *******  networks id for $add_vm_net is $network_id ******* \n\n");

VM creation:

Now we reached the final step to create VM.The RESTAPI “/v1.1/”.$tenant_id.”/servers” creates the VM.

The VM name,image id ,flavor id and security group name are required to run this RESTAPI command.

$headers = "{ 'Content-Type' : 'application/json', 'Accept' : 'application/json', 'ser-Agent' : 'python-novaclient', 'X-OpenStack-Nova-API-Version': '2.11', 'X-Auth-Token' : $token_id}";
        $CMDpath = "/v1.1/".$tenant_id."/servers";
        $APIport=$NOVAport;
        $url= "http://".$cont_ip.":".$APIport.$CMDpath;
        $body =
        {       "server" =>
                         {
                                "name" => "$add_vm_name",
                                "imageRef" => "$image_id",
                                "flavorRef" => "$flavor_id",
                                "max_count" => 1,
                                "min_count" => 1,
                                "networks" => [{
                                                "uuid" => "$network_id"
                                }],
                                "security_groups" => [{
                                                "name" => "$add_vm_secgroup"
                                }]
                        }
        };
        $request  = HTTP::Request->new( POST => $url );
        $bjson = encode_json $body;
        $request->content_type("application/json");
        $request->header('x-auth-token' => $token_id);
        $request->content($bjson);
        $http = LWP::UserAgent->new->request($request);

VM status verification:

After creating VM the status of the VM can be verified by the RESTAPI command “/v1.1/”.$tenant_id.”/servers/detail”.This sample code gives details on how to verify status of a VM.It checks for vm_state to be active and power_state to be 1.This RESTAPI command is equivalent to nova list command shown below.

$CMDpath= "/v1.1/".$tenant_id."/servers/detail";
        $APIport=$NOVAport;
        $url= "http://".$cont_ip.":".$APIport.$CMDpath;
        $request  = HTTP::Request->new( GET => $url );
        $request->header('content-type' => 'application/json');
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);
        @y = $http->{_content};
        print Dumper($http);
        $mes         = JSON::decode_json($y[0]);
        my $len = @{$mes->{"servers"}};
        my %server_to_netaddr_hash;
        my %test;
        my $servername;
        my $state;
        my $status;
        my $test;
        my $vmcount=0;

	$status = $mes->{"servers"}[$x]{"OS-EXT-STS:vm_state"};
          $state = $mes->{"servers"}[$x]{"OS-EXT-STS:power_state"};
          if ( $status ne "active" || $state != 1) {
        print ("\n\n *******  status/state  of the vm $test is not correct.The current  status is $status and state is $state ******* \n\n");
         return JT::FALSE;
                        }
        print ("\n\n *******  status and state  of the vm $test is correct The current status is $status and state is $state******* \n\n");

                     }

root@ server1:~# nova list
+--------------------------------------+--------+--------+------------+-------------+--------------------+
| ID                                   | Name   | Status | Task State | Power State | Networks           |
+--------------------------------------+--------+--------+------------+-------------+--------------------+
| 935bb371-9fb6-4d1e-92b6-239db2d28a2a | nvmg1  | ACTIVE | -          | Running     | net1=100.0.4.252   |
| 69b9dca5-a4e0-4168-8363-b87caa65e741 | nvmg10 | ACTIVE | -          | Running     | net10=100.0.13.252 |
| 176f9b62-1561-41e9-9c2f-de86f76afffd | nvmg11 | ACTIVE | -          | Running     | net11=100.0.14.252 |
| 15eff465-7251-4de8-b371-cc85790ca9a6 | nvmg12 | ACTIVE | -          | Running     | net12=100.0.15.252 |
| c1be1e20-4969-4911-8fdf-81e09c0023e9 | nvmg13 | ACTIVE | -          | Running     | net13=100.0.16.252 |
| 449becae-69b7-4f57-947a-74d78875ef39 | nvmg2  | ACTIVE | -          | Running     | net2=100.0.5.252   |
| db8c1d5b-5990-458f-bfcb-13a7d2bdbbda | nvmg3  | ACTIVE | -          | Running     | net3=100.0.6.252   |
| a7ceee7e-539a-4398-a768-209db065a71c | nvmg4  | ACTIVE | -          | Running     | net4=100.0.7.252   |
| 43713c5b-c262-4e56-9a77-ffde968f448e | nvmg5  | ACTIVE | -          | Running     | net5=100.0.8.252   |
| 8010f8a0-56d8-4904-a747-8e5dfa1f8ac8 | nvmg6  | ACTIVE | -          | Running     | net6=100.0.9.252   |
| 0c57cb78-922b-4f76-98d5-f962f08a85a6 | nvmg7  | ACTIVE | -          | Running     | net7=100.0.10.252  |
| 47ca6427-c898-4d71-9c2d-61ffa533e31e | nvmg8  | ACTIVE | -          | Running     | net8=100.0.11.252  |
| a08bbaac-3ed9-4cf6-ad6c-def7484a35e4 | nvmg9  | ACTIVE | -          | Running     | net9=100.0.12.252  |
+--------------------------------------+--------+--------+------------+-------------+--------------------+

The status of the VMs can also be checked in OPENSTACK horizon GUI as shown below.

VM Deletion:

Before start deleting the VMs we need to get the token-id from the Openstack server using the procedure given earlier.In addition to this we need to know the VM id.To get the VM id we can use the RESTAPI “/v2/”.$tenant_id.”/servers?name=”.$del_vm_name as shown below.

my $headers = "{ 'Content-Type' : 'application/json', 'Accept'  :  'application/json' , 'ser-Agent'  : 'python-novaclient', 'X-Auth-Token'   :    $token_id}";
        $CMDpath= "/v2/".$tenant_id."/servers?name=".$del_vm_name;
        $APIport=$NOVAport;
        $url= "http://".$cont_ip.":".$APIport.$CMDpath;
        $request  = HTTP::Request->new( GET => $url );
        $request->header('content-type' => 'application/json');
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);
        @y = $http->{_content};
        $mes         = JSON::decode_json($y[0]);
        #print "\n\n\n\n\n\---------------\n\n\n\n";
        #print Dumper($mes);
        my $del_vm_id = $mes->{'servers'}[0]{'id'};
        print ( "\n\n *******  delte vm instance id is $del_vm_id ******* \n\n");

After getting the VM ID we can use the RESTAPI “/v1.1/”.$tenant_id.”/servers/”.$del_vm_id.”/action” to delete the VM.This is equivlant to “nova delete <vm name/vm id>” command.

$headers = "{ 'Content-Type' : 'application/json', 'Accept'  :  'application/json' , 'ser-Agent'  : 'python-novaclient', 'X-Auth-Token'   :    $token_id}";
        $CMDpath = "/v1.1/".$tenant_id."/servers/".$del_vm_id."/action";
        $APIport=$NOVAport;
        $url= "http://".$cont_ip.":".$APIport.$CMDpath;
        $body =
        {
               "forceDelete"   =>  "null"
        };
        $request  = HTTP::Request->new( POST => $url );
        $bjson = encode_json $body;
        $request->content_type("application/json");
        $request->header('x-auth-token' => $token_id);
        $request->content($bjson);
        $http = LWP::UserAgent->new->request($request);

Conclusion

This document gives details on CONTRAIL cloud solution implementation. It also explains the RESTAPI details for  VM creation,VM deletion etc..

APPENDIX

The section give details on other RESTAPI commands.

Deleting virtual network:

This command is executed on the contrail controller. To run this command we need virtual network id.

$CMDpath = "/virtual-network/";
        $APIport="8082";
        $url= "http://".$cont_ip.":".$APIport.$CMDpath.$netid;
        $request  = HTTP::Request->new( DELETE => $url );
       $request->header('content-type' => 'application/json');
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);

To get the virtual network id this RESTAPI is used.The “uuid” returns the network id.

$CMDpath = "/virtual-networks";
        $APIport="8082";
        $url= "http://".$cont_ip.":".$APIport.$CMDpath;
        $request  = HTTP::Request->new( GET => $url );
        $request->content_type("application/json");
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);
         print Dumper $http;
          @y = $http->{_content};
        $mes         = JSON::decode_json($y[0]);
          $len = @{$mes->{"virtual-networks"}};
        my @network_list;
        my $i=1;
        for (my $x = 0; $x< $len ; ++$x) { if ($mes->{"virtual-networks"}[$x]{"fq_name"}[2]  =~ /net\d+/) {
                push (@network_list, $mes->{"virtual-networks"}[$x]{"uuid"});
                $i++;
            }
          }

To delete the ports which map virtual network with the VM we need to use the RESTAPI given below.

We need portid to run this command.

My $NEUTRONport      =  "9696";
         $CMDpath = "/v2.0/ports/";
        $APIport=$NEUTRONport;
        $url= "http://".$cont_ip.":".$APIport.$CMDpath.$portid.".json";
        $request  = HTTP::Request->new( DELETE => $url );
       $request->header('content-type' => 'application/json');
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);

The port id can be found from,

$CMDpath= "/v2.0/ports.json";
        $APIport=$NEUTRONport;
        $url= "http://".$cont_ip.":".$APIport.$CMDpath;
        $request  = HTTP::Request->new( GET => $url );
        $request->header('content-type' => 'application/json');
        $request->header('x-auth-token' => $token_id);
        $http = LWP::UserAgent->new->request($request);
        @y = $http->{_content};
        $mes         = JSON::decode_json($y[0]);
        my $len = @{$mes->{"ports"}};
        my @port_list;
        for (my $x = 0; $x< $len ; ++$x) { push (@port_list, $mes->{"ports"}[$x]{"id"});
        }

In between two weeks of events as I write this, I got to stop at home and in the office for only a day, but I was excited by what I saw on Friday afternoon… The best SDN and network automation solution out there, OpenContrail, is now set to rule the Kubernetes stage too. With title of “#1 SDN for OpenStack” secured for the last two years, the OpenContrail community is now ready for Kubernetes (K8s), the hottest stack out there, and a passion of mine for which I’ve been advocating for more and more of attention around the office at Juniper.

It couldn’t be better timing, heading into KubeCon / CloudNativeCon / OpenShift Commons in Berlin this week. For those that know me, you know I love the (b)leading edge of cloud and devops, and last year when I built an AWS cluster for OpenContrail + OpenShift / Kubernetes then shared it in my Getting to #GIFEE blog, github repo, and demo video, those were early days for OpenContrail in the K8s space. It was work pioneered by OpenContrail co-founder and a few star engineers to help me cut through the off-piste mank and uncharted territory. It was also inspired by my friends at tcp cloud (now Mirantis) who presented on Smart cities / IoT with K8s, OpenContrail, and Raspberry Pi at the last KubeCon EU. Today, OpenContrail, the all-capability open SDN is ready to kill it in the Kubernetes and OpenShift spaces, elevating it to precisely what I recently demo shopped for in skis: a true all-mountain top performer (BTW my vote goes to the Rosi Exp. HD 88). Now that you know what I was doing when not blogging recently… Skiing aside, our demos are ready for the KubeCon audience this week at our Juniper booth, and I’ll be there talking about bringing a ton of extra value to the Kubernetes v1.6 networking stack.

What sets OpenContrail apart from the other networking options for Kubernetes and OpenShift is that it brings an arsenal of networking and security features to bear on your stack, developed by years of the performance-obsessed folks at Juniper and many other engineers in the community. In this space OpenContrail often competes with smaller SDN solutions offered by startups. Their said advantage is positioned as nimbler and simpler solutions. This was particularly true in their easy, integrated installation with K8s (coming for OpenContrail very soon), but on the whole, their advantage, in general across the likes of Contiv, Calico, Flannel, Weave, etc. boils down to 2 things. Let’s have a look…

First, it is easy to be simpler when primitive. This isn’t a knock on the little guys. They are very good for some pointed use cases and indeed they are simple to use. They do have real limits however. Simple operation can always come from less knobs and fewer capabilities, but I believe an important goal of SDN, self-driving infrastructure, cognitive / AI in tech, is abstraction; in other words, simplicity needn’t come from striping away functionality. We only need start with a good model for excellent performance at scale and design for elegance and ease of use. It’s not easy to do, but have you noticed that Kubernetes itself is the perfect example of this? – super easy 2 years ago, still is, but at the same time there’s a lot more to it today. It’s built on solid concepts and architecture, from a lot of wisdom at Google. Abstraction is the new black. It’s open, layered and transparent when you need to peel it back, and it is the solution to manage complexity that arises from features and concepts that aren’t for everyone. General arguments aside, if you look at something like K8s ingress or services networking / balancing, none of the SDN solutions cover that except for very pointed solutions like GKE or nginx where you’d still need one or more smaller SDN tools. Furthermore, when you start to demand network performance on the control (protocol/API) and data (traffic feeds & speeds) planes with scale in many other K8s metrics, the benefits of OpenContrail that you get for free really stand apart from the defacto K8s networking components and these niche offerings. Better still, you get it all as one OpenContrail solution that is portable across public and private clouds.

Second, other solutions are often solely focused on K8s or CaaS stacks. OpenContrail is an SDN solution that isn’t just for Kubernetes nor containers. It’s one ring to rule them all. It works on adjacent CaaS stack integrations with OpenShift, Mesos, etc., but it’s equally if not more time-tested in VMware and OpenStack stacks and even bare-metal or DIY orchestration of mixed runtimes. You might have heard of publicized customer cases for CaaS, VMware and OpenStack across hybrid clouds and continuing with the same OpenContrail networking irrespective of stacking PaaS and CaaS upon pub/prvt IaaS or metal (here are a few 1, 2, 3). It’s one solution unifying your stack integration, network automation and policy needs across everything. In other words, the only-one-you-need all-mountain ski. The only competitor that comes close to this breadth is Nuage Networks, but they get knocked out quickly for not being open source (without tackling other important performance and scale detail here). If the right ski for the conditions sounds more fun to you, like the right tool for the right job, then wait for my next blog on hybrid cloud…and what I spoke about at IBM Interconnect (sorry no recording). In short, while having lots of skis may have an appeal to different conditions, having lots of networking solutions is a recipe for IT disaster because the network foundation is so pervasive and needs to connect everything.

With OpenContrail developers now making it is dead easy to deploy with K8s (think kubeadm, Helm, Ansible…) and seamless to operate with K8s, it’s going to do what it did for OpenStack networking, for Kubernetes networking very quickly. I’m really excited to be a part of it.

You can visit the Juniper booth to find me and hear more. As you may know, Juniper is the leading contributor to OpenContrail, but we’re also putting K8s to work in the Juniper customer service workbench and CSO product (NFV/CPE offering). Moreover, and extremely complementary to Juniper’s Contrail Networking (our OpenContrail offering), Juniper just acquired AppFormix 4 months ago. AppFormix is smart operations management software that will make your OpenStack or K8s cluster ops akin to a self-driving Tesla (a must if your ops experience is about as much fun as a traffic jam). Juniper’s AppFormix will give you visibility into the state of workloads both apps and software-defined infrastructure stack components alike, and it applies big data and cognitive machine learning to automate and adapt policies, alarms, chargebacks and more. AppFormix is my new jam (with Contrail PB)… it gathers and turns your cloud ops data into insight… The better and faster you can do that, the better your competitive advantage. As you can tell, KubeCon is going to be fun this week!

DEMOS DEMOS DEMOS

+update 3/28
Thanks to the Juniper marketing folks for a list of the demos that we’ll be sharing at KubeCon:

1. Contrail Networking integrated with a Kubernetes cluster (fully v1.6 compatible) –
   • Namespaces/RBAC/Network Policy improved security with OpenContrail virtual networks, VPC/projects and security groups (YouTube demo video)
   • Ingress and Services load balancing with OpenContrail HAProxy, vRouter ECMP, and floating IP addresses, improving performance (remove KubeProxy) and consolidating SDN implementation (YouTube demo video)
   • Connecting multiple hybrid clouds with simple network federation and unified policy
   • Consolidating stacked CaaS on IaaS SDN: optimized OpenContrail vRouter networking “passthru” for Kubernetes nodes on top of OpenStack VMs
2. Contrail Networking integrated with an OpenShift cluster (YouTube demo video)
3. Contrail Networking containerized and easy deployment in Kubernetes
4. AppFormix monitoring and analytics for Kubernetes and workload re-placement automation

I’ll light these up with links to YouTube videos as I get the URLs.

The private cloud market has changed in the past year, and our customers are no longer interested in just getting an amazing tool for installing OpenStack; instead, they are looking more at use cases. Because we see a lot of interest in NFV cloud use cases, Mirantis includes OpenContrail as the default SDN for its new Mirantis Cloud Platform. In fact, NFV has become a mantra for most service providers, and because Mirantis is a key player in this market, we work on a lot of testing and performance validation.

The most common value for performance comparison between solutions is bandwidth, which shows how much capacity a network connection has for supporting data transfer, as measured in bits per second. In this domain, the OpenContrail vRouter can reach near line speed (about 90%, in fact). However, performance also depends on other factors, such as latency, or packets-per-second (pps), which are as important as bandwidth. Packets per second rate is a key factor for VNF (firewalls, routers, etc.) instances running on top of NFV clouds. In this article, we’ll compare PPS rate for different OpenContrail setups so you can decide what will work best for your specific use case.

The simplest way to test PPS rate is to run a VM to VM test. We will provide a short overview of OpenContrail low-level techniques for NFV infrastructure, and perform a comparative analysis of different approaches using simple PPS benchmarking. To make testing fair, we will use only a 10GbE physical interface, and will limit resource consumption for data plane acceleration technologies, making the environment identical for all approaches.

OpenContrail vRouter modes

For different use cases, Mirantis supports several ways of running the OpenContrail vRouter as part of Mirantis Cloud Platform 1.0 (MCP). Let’s look at each of them before we go ahead and take measurements.

Kernel vRouter

OpenContrail has a module called vRouter that performs data forwarding in the kernel. The vRouter module is an alternative to Linux bridge or Open vSwitch (OVS) in the kernel, and one of its functionalities is encapsulating packets sent to the overlay network and decapsulating packets received from the overlay network. A simplified schematic of VM to VM connectivity for 2 compute nodes can be found in Figure 1:

Figure 1: A simplified schematic of VM to VM connectivity for 2 compute nodes

The problem with a kernel module is that packets-per-second is limited by various factors, such as memory copies, the number of VM exits, and the overhead of processing interrupts. Therefore vRouter can be integrated with the Intel DPDK to optimize PPS performance.

DPDK vRouter

Intel DPDK is an open source set of libraries and drivers that perform fast packet processing by enabling drivers to obtain direct control of the NIC address space and map packets directly into an application. The polling model of NIC drivers helps to avoid the overhead of interrupts from the NIC. To integrate with DPDK, the vRouter can now run in a user process instead of a kernel module. This process links with the DPDK libraries and communicates with the vrouter host agent, which runs as a separate process. The schematic for a simplified overview of vRouter-DPDK based nodes is shown in Figure 2:

Figure 2: The schematic for a simplified overview of vRouter-DPDK based nodes

vRouter-DPDK uses user-space packet processing and CPU affinity to dedicate poll mode drivers being served by a particular CPU. This approach enables packets to be processed in user-space during the complete life time – from physical NIC to vhost-user port.

Netronome Agilio Solution

Software and hardware components distributed by Netronome provide an OpenContrail-based platform to perform high-speed packet processing. It’s a scalable, easy to operate solution that includes all server-side networking features, such as overlay networking based on MPLS over UDP/GRE and VXLAN. The Agilio SmartNIC solution supports DPDK, SR-IOV and Express Virtio (XVIO) for data plane acceleration while running the OpenContrail control plane. Wide integration with OpenStack enables you to run VMs with Virtio devices or SR-IOV Passthrough vNICs, as in Figure 3:

Figure 3:  OpenContrail network schematic based on Netronome Agilio SmartNICs and software

A key feature of the Netronome Agilio solution is deep integration with OpenContrail and offloading of lookups and actions for vRouter tables.

Compute nodes based on Agilio SmartNICs and software can work in an OpenStack cluster based on OpenContrail without changes to orchestration. That means it’s scale-independent and can be plugged into existing OpenContrail environments with zero downtime.

Mirantis Cloud Platform can be used as an easy and fast delivery tool to set up Netronome Agilio-based compute nodes and provide orchestration and analysis of the cluster environment. Using Agilio and MCP, it is easily to setup a high-performance cluster with a ready-to-use NFV infrastructure.

Testing scenario

To make the test fair and clear, we will use an OpenStack cluster with two compute nodes. Each node will have a 10GbE NIC for the tenant network.

As we mentioned before, the simplest way to test the PPS rate is to run a VM to VM test. Each VM will have 2 Virtio interfaces to receive and transmit packets, 4 vCPU cores, 4096 MB of RAM and will run Pktgen-DPDK inside to generate and receive a high rate of traffic. For each VM a single Virtio interface will be used for generation, and another interface will be used for receiving incoming traffic from the other VM.

To make an analytic comparison of all technologies, we will not use more than 2 cores for the data plane acceleration engines. The results of the RX PPS rate for all VMs will be considered as a result for the VM to VM test.

First of all, we will try to measure kernel vRouter VM to VM performance. Nodes will be connected with Intel 82599 NICs. The following results were achieved for a UDP traffic performance test:

As you can see, the kernel vRouter is not suitable for providing a high packet per second rate, mostly because the interrupt-based model can’t handle a high rate of packets per second. With 64 byte packets we can only achieve 3% of line rate.

For the DPDK-based vRouter, we achieved the following results:

Based on these results, the DPDK based solution is better at handling high-rated traffic based on small UDP packets.

Lastly, we tested the Netronome Agilio SmartNIC-based compute nodes:

With only 2 forwarder cores, we are able to achieve line-rate speed on Netronome Agilio CX 10GbE SmartNICs on all size of packets.

You can also see a demonstration of the Netronome Agilio Solution here.

Since we have achieved line-rate speed on the 10GbE interface using Netronome Agilio SmartNICs we wanted to have the maximum possible PPS rate based on 2 CPUs. To determine the maximum performance result for this deployment, we will upgrade existing nodes with Netronome Agilio CX 40GbE SmartNIC and repeat the maximum PPS scenario one more time. We will use direct wire connection between 40GbE ports and will set up 64-bytes UDP traffic. Even with hard resources limitations, we achieved:

What we learned

Taking all of the results together, we can see a pattern:

Based on 64 byte UDP traffic, we can also see where each solution stands compared to 10GbE line rate:

OpenContrail remains the best production-ready SDN solution for OpenStack clusters, but to provide NFV-related infrastructure, OpenContrail can be used in different ways:

• The Kernel vRouter, based on interrupt model packet processing, works, but does not satisfy the high PPS rate requirement.
• The DPDK-based vRouter significantly improves the PPS rate, but due to high resource consumption and because of defined limitations, it can’t achieve the required performance. We also can assume that using a modern DPDK library will improve performance and optimise resource consumption.
• The Netronome Agilio SmartNIC solution significantly improves OpenContrail SDN performance, focusing on saving host resources and providing a stable high-performance infrastructure.

With Mirantis Cloud Platform tooling, it is possible to provision, orchestrate and destroy high performance clusters with various networking features, making networking intelligent and agile.

OpenContrail has solidified its position as the top SDN and network automation solution for OpenStack, and recently announced it’s integration into Kubernetes (K8s) and thus OpenShift (recent versions are powered by Kubernetes.) What does this mean for the organization mapping out their cloud infrastructure architecture?

VMware -> OpenStack -> Kubernetes

At this juncture in cloud adoption, service providers and enterprises alike are defining their ever evolving stacks. The emergence of containers has changed the game – changing the conversation from “OpenStack or Kubernetes?” to “OpenStack and Kubernetes.” In many cases VMware is still in the mix somewhere. The combination of these platforms allows an organization to select the best hosting platform for each workload, based on it’s own unique characteristics:

• Aging monolithic application that will be deprecated in the next few years? Keep it where it is (VMware.)
• Distributed multi-tier web app? Perhaps OpenStack is a good candidate.
• Highly variable, net new microservices based application(s) – Kubernetes will allow the individual microservices to scale up and down independently of one another.

While all of this choice provides flexibility, it introduces complexity – especially on the network with each of the three platforms with three different networking stacks.

How Contrail enables Successful Cloud Adoption

Introducing Contrail into a heterogeneous cloud/virtualization environment will help an organization move towards SDN adoption based on open standards. One SDN solution to manage them all! Furthermore, it isn’t realistic or likely for an organization to go from VMware to OpenStack to Kubernetes all at once – that is too much change to manage and it introduces risk. Because of it’s interoperability with all of these platforms, introducing Contrail will allow an organization to stitch together their virtualization/container platforms on their terms as their infrastructure evolves with the applications that sit atop it.

Check out how Contrail integrates with VMware, OpenStack, Kubernetes and OpenShift:

• Integrating VMware ESXi with OpenStack, OpenContrail
• Installing Contrail with VMware vCenter
• OpenContrail Kubernetes Integration
• Video Demo: OpenContrail integration with OpenShift, Kubernetes

3 ways to get Getting Started with OpenContrail

• Try it yourself! – OpenContrail Quick Start Guide (Software Installation guide for OpenContrail)
• Try OpenContrail Sandbox – The Contrail Sandbox is a cloud based testing environment that can help you evaluate Contrail Networking product on our infrastructure free of cost
• Let’s Chat – Shoot me an email!

One of the initiatives I was asked to lead when I joined Juniper Networks late last year was to map out the plan for building a diverse, engaged developer community around the OpenContrail project. This blog post is a quick update on our progress.

TL;DR: Juniper is engaged with key users to push OpenContrail even further ahead of other technologies while mapping out the community engagement plan.

The purpose of this effort is to address a major need: The OpenContrail community can be improved. We can increase the corporate diversity of upstream developers, we can amplify the voice of users in the roadmapping process, and we can broaden the application use cases for the core technology. We can do all these things while maintaining the wide technological and production-readiness lead that OpenContrail has created over alternative technologies.

Oddly, OpenContrail has been resource challenged largely due to its successes. Here’s what I mean: OpenContrail is found in the world’s largest SDN deployments at the largest carriers. It is the most battle-hardened SDN in use at scale, and that’s why OpenContrail is the #1 in SDN for OpenStack. All this means that an incredible amount of engineering work went into features, fixes and improvements. As with all technology endeavors, compromises were made, and one of those compromises was community management.

Another reality is that OpenContrail began its life as Contrail, the proprietary creation of a startup by the same name that was acquired by Juniper Networks in 2012. It was open sourced under the Apache 2.0 license in 2013. Since then, Juniper and a few, key contributors outside the organization have focused on building the project.

These factors are starting to change. We are working diligently towards leveling up our game in enabling the project’s community. Here’s a quick rundown of what else Juniper is working on to make the OpenContrail community a place that encourages and empowers both upstream contributions and the voice of the user:

• Hiring a dedicated community manager to drive community engagement and act as a dedicated conduit into Juniper
• Hired a highly visible open source advocate and expert to drive strategy (yours truly)
• Hired a dedicated team of open source developers to work on community-specific efforts, including:
  • 100% dedicated to open source with no responsibility for product feature delivery
  • Currently in the process of handing off packaging and container management from Contrail engineering
  • Re-factoring and cleaning up packages, containers and plugins to various other projects
  • Looking at contributing to and helping with acceptance of community-contributed code in OpenContrail
  • Committing code to other adjacent ecosystems (e.g., OpenStack-Helm, OpenStack, Kubernetes)
• Working through logistical issues around how code is upstreamed

Most importantly, I have been actively talking to members of the community, gathering information on what has worked and what hasn’t and helping the Contrail team to craft a long-term open source and community building strategy. As I’m thinking about Juniper’s open source community strategy, I’m bringing in lessons learned from OpenStack, Project CoprHD and elsewhere.

The reality is that not all open source communities are built the same way. Not all of them succeed due to following a cookie cutter approach. Some open source projects need no formal community or foundation to succeed. MySQL comes to mind, which never had a foundation, but there are many others. In fact, the default for most open source projects is an informal community. Other open source projects are largely dominated by a single contributor with an ecosystem of secondary developers, like Puppet, Chef, MongoDB, Ceph and many others. In Ceph, for example, all of the top contributors are from RedHat. The issue isn’t that one company currently dominates the OpenContrail community. The issue is that we haven’t discovered what kind of community we want to be yet.

Early reactions to this game plan have been positive. Last week, I spent time with a major OpenContrail user, syncing up with key executives and engineers who lead that company’s internal cloud efforts. The purpose of that meeting was to bring them up to speed on all of the changes in the pipeline for the OpenContrail community. Reactions were favorable and engaging—precisely what you want when kicking off a community strategy. Amongst a variety of joint efforts we’re working on with OpenContrail, users are co-presenting at upcoming conferences and co-development in both OpenContrail and OpenStack-Helm.

Finding the “OpenContrail Way” is a journey. We are taking the first steps toward shaping what the community will look like, and those who participate will be part of the journey and help shape the future of OpenContrail. We want everyone who wants to participate to engage fully. Together, we will make OpenContrail better, improving processes for governance and upstream contributions while improving tools for packaging, containers, build/release, CI/CD and more.

Join us today, and together we’ll make the best open source SDN for carrier-grade performance even better. I’m here to help and make certain this community is what we all want for the future.

With DockerCon in Austin happening this week, I’m reminded of last year’s DockerCon Seattle, and watching some announcements with utter fascination or utter disappointment. Let’s see if we can’t turn the disappointments into positives.

The first disappointment has a recent happy ending. It was a broadly shared observation in Seattle, the media, and discussion forums: Docker was overstepping when they bundled in Swarm with the core of Docker Engine in 1.12. This led to the trial balloon that forking Docker was a potential solution towards a lighter-weight Docker that could serve in Mesos and Kubernetes too. Last September I covered this in my blog sparing little disdain over the idea of forking Docker Engine simply because it had become too monolithic. There are other good options, and I’m happy to say Docker heeded the community’s outcry and cleanly broke out a component of Docker Engine called containerd which is the crux of the container runtime. This gets back to the elegant Unix-tool inspired modularization and composition, and I’m glad to see containerd and rkt have recently been accepted into the CNCF. Crisis #1 averted.

My next disappointment was not so widely shared, and in fact it is still a problem at large today: the viewpoint on container networking. Let’s have a look.

IS CONTAINER NETWORKING SPECIAL?

When it comes to containers, there’s a massive outpour of innovation from both mature vendors and startups alike. When it comes to SDN there’s no exception.

Many of these solutions you can discount because, as I said in my last blog, as shiny and interesting as they may be on the surface or in the community, their simplicity is quickly discovered as a double-edged sword. In other words, they may be easy to get going and wrap your head around, but they have serious performance issues, security issues, scale issues, and “experiential cliffs” to borrow a turn of phrase from the Kubernetes founders when they commented on the sometimes over-simplicity of many PaaS systems (iow. they hit a use case where the system just can’t do that experience/feature that is needed).

BACK TO DOCKERCON SEATTLE…

Let’s put aside the SDN startups that to various extents suffer from the over-simplicity or lack of soak and development time, leading to the issues above. The thing that really grinds my gears about last year’s DockerCon can be boiled down to Docker, a powerful voice in the community, really advocating that container networking was making serious strides, when at the same time they were using the most primitive of statements (and solution) possible, introducing “Multi-host networking”

You may recall my social post/poke at the photo of this slide with my sarcastic caption.

Of course, Docker was talking about their overlay-based approach to networking that was launched as the (then) new default mode to enable networking in Swarm clusters. The problem is that most of the community are not SDN experts, and so they really don’t know any better than to believe this is an aww!-worthy contribution. A few of us that have long-worked in networking were less impressed.

Because of the attention that container projects get, Docker being the biggest, these kind of SDN solutions are still seen today by the wider community of users as good networking solutions to go with because they easily work in the very basic CaaS use cases that most users start playing with. Just because they work for your cluster today, however, doesn’t make them a solid choice. In the future your netops team will ask about X, Y and Z (and yet more stuff down the road they won’t have the foresight to see today). Also in the future you’ll expand and mature your use cases and start to care about non-functional traits of the network which often happens too late in production or when problems arise. I totally get it. Networking isn’t the first thing you want to think about in the cool new world of container stacks. It’s down in the weeds. It’s more exciting to contemplate the orchestration layer, and things we understand like our applications.

On top of the fact that many of these new SDN players offer primitive solutions with hidden pitfalls down the road that you won’t see until it’s too late, another less pardonable nuisance is the fact that most of them are perpetrating the myth that container networking is somehow special. I’ve heard this a lot in various verbiage over the ~7 years that SDN has arisen for cloud use cases. Just this week, I read a meetup description that started, “Containers require a new approach to networking.” Because of all the commotion in the container community with plenty of new SDN projects and companies having popped up, you may be duped into believing that, but it’s completely false. These players have a vested interest, though, in making you see it that way.

THE TRUTH ABOUT NETWORKING CONTAINERS

The truth is that while workload connectivity to the network may change with containers (see CNM or CNI) or with the next new thing, the network itself doesn’t need to change to address the new endpoint type. Where networks did need some work, however, is on the side of plugging into the orchestration systems. This meant that networks needed better programmability and then integration to connect-up workloads in lock-step with how the orchestration system created, deleted and moved workloads. This meant plugging into systems like vSphere, OpenStack, Kubernetes, etc. In dealing with that challenge, there were again two mindsets to making the network more programmable, automated, and agile: one camp created totally net-new solutions with entirely new protocols (OpenFlow, STT, VxLAN, VPP, etc.), and the other camp used existing protocols to build new more dynamic solutions that met the new needs.

Today the first camp solutions are falling by the wayside, and the camp that built based on existing open standards and with interoperability in mind is clearly winning. OpenContrail is the most successful of these solutions.

The truth about networks is that they are pervasive and they connect everything. Interoperability is key. 1) Interoperability across networks: If you build a network that is an island of connectivity, it can’t be successful. If you build a network that requires special/new gateways, then it doesn’t connect quickly and easily to other networks using existing standards, and it won’t be successful. 2) Interoperability across endpoints connections: If you build a network that is brilliant at connecting only containers, even if it’s interoperable with other networks, then you’ve still created an island. It’s an island of operational context because the ops team needs a different solution for connecting bare-metal nodes and virtual machines. 3) Interoperability across infrastructure: If you have an SDN solution that requires a lot from the underlay/underlying infrastructure, it’s a failure. We’ve seen this with SDNs like NSX that required multicast in the underlay. We’ve seen this with ACI that requires Cisco switches to work. We’ve even seen great SDN solutions in the public cloud, but they’re specific to AWS or GCP. If your SDN solution isn’t portable anywhere, certainly to most places, then it’s still doomed.

IF YOU WANT ONE UNIFIED NETWORK, YOU NEED ONE SDN SOLUTION

This aspect of interoperability and portability actually applies to many IT tools if you’re really going to realize a hybrid cloud and streamline ops, but perhaps nowhere is it more important than in the network because of its inherently pervasive nature.

If you’re at DockerCon this week, you’ll be happy to know that the best solution for container networking, OpenContrail, is also the best SDN for Kubernetes, Mesos, OpenStack, NFV, bare-metal node network automation, and VMware. While this is one SDN to rule and connect them all, and very feature rich in its 4th year of development, it’s also never been more approachable, both commercially turn-key and in open source. You can deploy it on top of any public cloud or atop of private clouds with OpenStack or VMware, or equally easily on bare-metal CaaS, especially with Kubernetes, thanks to Helm.

Please drop by and ask for an OpenContrail demo and sticker! for your laptop or phone at the Juniper Networks booth, and booths of partners of Juniper’s that have Juniper Contrail Networking integrations: Red Hat, Mirantis, Canonical, and we’re now happy to welcome Platform9 to the party too. We at Juniper will be showcasing a joint demo with Platform9 that you can read more about on the Platform9 blog.

PS. If you’re running your CaaS atop of OpenStack, then even more reason that you’ll want to stop by and get a sneak peak of what you’ll also hear more about at the upcoming Red Hat and OpenStack Summits in Boston.

Red Hat OpenShift is the industry’s most secure and comprehensive enterprise-grade container platform. It is a Platform-as-a-Service (PaaS) based on Google’s Kubernetes, that allows developers to quickly develop, host, and scale applications in a cloud environment.

OpenContrail, is the leading SDN automation solution for the cloud. From network virtualization to integrated cloud management, OpenContrail delivers freedom of choice, intelligent automation, and always-on reliability for cloud, software-defined WAN (SD-WAN), and Network Functions Virtualization (NFV) environments.

Today, enterprises know that to be competitive, they must build, deploy, and scale applications faster. Slowly, the industry is moving away from monolithic architecture (which are difficult to scale, resource intensive and increase the time to market) to a microservice architecture, to speed up continuous deployment and delivery.

As the industry embraces the microservices model, cloud platforms (public and private) will start hosting the container platforms. With thousands of containers, VMs and bare-metals in an enterprise environment, managing workloads becomes a painful task and most importantly networking becomes complex. Since multiple overlay networks are involved, our customers want a single solution to stitch and manage their next-gen and legacy workloads. To solve this networking problem, we came up with a solution which supports all the leading Container orchestration platforms (Google’s Kubernetes, Red Hat OpenShift Container Platform and Apache Mesos). With the latest OpenContrail release, we can now manage a whole spectrum of workloads, be it containers, virtual-machines and bare-metal. Along with this, OpenContrail provides certain features which other Container networking solutions don’t (Eg. namespace isolation). Due to the rich feature-set and flexibility of OpenContrail, enterprises can now seamlessly migrate to a microservices model, and manage various workloads using a single solution.

The below demo shows OpenContrail’s integration with Red Hat OpenShift Container Platform and walk you through key features such as,

NAMESPACE ISOLATION

Isolate pods/services in different OpenShift projects from talking to each other

SOURCE NAT (EGRESS)

Allow pods/services to talk to the outside world (Internet)

By default, we restrict the pods/services from communicating with the outside world. This provides granular control over the OpenShift cluster

INGRESS

• Single Service Ingress

Expose a single service to the outside world

1:1 mapping between ingress and service

test.contrail.com -> 178.91.123.132 ->  / dev   service-dev:80

• Simple Fanout Ingress

Expose multiple services to the outside world

1:many mapping between ingress and services. Uses HAPROXY load-balancer in OpenContrail’s vRouter agent container to route the traffic to different backend services

test.contrail.com -> 178.91.123.132 -> / dev    service-dev:80

                                                                                        / qa      service-qa:80

• Name Based Virtual Hosting

Expose multiple services to the outside world through multiple hostnames

1:many mapping between ingress and services through hostnames. Uses the HTTP header information and HAPROXY load-balancer in OpenContrail’s vRouter agent container to route the traffic to different backend services

dev.contrail.com –|                       |-> service-dev:80

                                                                                     | 178.91.123.132  |

qa.contrail.com   –|                       |-> service-qa:80

 There’s also a couple of other interesting demos that are lined up, so keep watching this space for latest updates.