Note: This blog is co-authored by Aniket Daptari from Juniper Networks and Varun Lodaya from Symantec. Varun will be presenting his work at the upcoming OpenContrail User Group meeting during the OpenStack Summit (April 27th). RSVP to the event if you want to see his talk.
This blog highlights how OpenContrail has fostered the notion of User and Developer communities. Here, we highlight one example of a specific User and how they have contributed as a Developer to enhance the value they extract from OpenContrail. Symantec has now been a long time user and contributor to OpenContrail. In this particular blog we aim to highlight one of their most recent contributions that enhances the LBaaS offering in a manner that not only addresses their specific use case, but is generic enough for other users to leverage.
Further, this general approach may be used to extend other API sets beyond LBaaS as well.
With this, we also want to highlight the contribution of OpenContrail developer, Varun Lodaya who has played a critical role in the design and development of this enhancement
Abstract :
With LBaaS, OpenStack attempts to define a single set of APIs to consume load balancing functionality regardless of the implementation of those APIs. This allows the OpenStack operator flexibility in choosing the load balancing implementation on the backend as well as in making changes to that back end. This has allowed various vendors to provide their southbound LBaaS drivers for example F5, A10, HAProxy, NGINX, etc. Now, while each of these load balancers offers many varied features, the northbound LBaaS APIs (v1.0) are a bit limited. It is perhaps impractical to provide APIs for every feature provided by all the load balancers. Therefore, there is a need to provide a way to support load balancer functionality beyond what is made available via the LBaaS v1.0 APIs.
OpenContrail user, Symantec had the following LBaaS use cases that translated to a requirement to exercise additional functionality beyond what is available via LBaaS v1. In response to that requirement, Symantec developers designed the LBaaS Custom Attributes support.
The following are the use cases in the words of Cloud Platform Engineering developer, Varun Lodaya, from Symantec:
Use Cases:
● Enable our cloud users to manage all their LBaaS features/configurations themselves.
● Empower the cloud providers to decide what capabilities they want their users to have since they know their infrastructure capabilities and limitations the best.
● Support tenant SSL certs with custom attributes.
Design :
We came up with a modular design that would help to cater to all of the above use cases. Following is the design flow:
● Based on the LBaaS driver that the cloud provider is using, the cloud provider will identify the additional features they want to expose to their users. These additional features will be made available to the users via custom attributes. The cloud provider will then generate a validation file that contains all the custom attributes (corresponding to the features) they want to provide to their users. In addition to the list of additional features, the cloud provider will also specify any limits they might want to enforce associated with those custom attributes.
● This validation file will be used and enforced by OpenContrail when facilitating the invocation of the custom attributes.
● When users exercise the load balancing functionality via the LBaaS APIs, they will invoke additional functionality by specifying a list of keyvalue dicts while configuring the LBaaS pool. This list of keyvalue dicts is saved in a database.
● When the LBaaS VIP is created, the OpenContrail Service Monitor process reads the custom attributes from the lb_pool object and validates them against the corresponding custom validation file.
● If validation fails, service_monitor process moves the vip/service_instance to “Error” state with the corresponding error message. It’s users responsibility to then go back and correct their custom attributes to pass validation.
● If validation passes, the custom attributes are pushed down to the corresponding drivers. In case of OpenContrail’s implementation of LBaaS using HAProxy, these custom attributes get pushed to the vRouter from where they get applied to the corresponding HAProxy process.
● This custom attributes extension could also be used to support tenant level SSL certs with LBaaSv1. Users could manage their certs in different ways, one of those ways being the Openstack project Barbican.
● Once they have their certificate pem files ready, they can provide the certificate references as custom attributes to OpenContrail.
● OpenContrail then downloads the certificates via references provided in custom attributes and updates the corresponding southbound driver with the SSL certificates.
● Currently, we support Openstack Barbican as certificate manager and OpenContrail code fetches the certificates and private keys from Barbican, but, code is generic enough to be able to plugin any cert_manager_driver which could then fetch the certificates from thirdparty certificate managers.
Usage:
CLI:
Neutron CLI to provide custom attributes:
neutron lb-pool-create --name Test_Pool --subnet-id <subnetid>
--lb-method ROUND_ROBIN --protocol HTTP --custom-attributes type=dict
list=true client_timeout=50000,
tls_container= http://<barbican_ep>/v1/containers/ <container_ref_uuid>
neutron lb-pool-update <pool-id> --custom-attributes type=dict
list=true server_timeout=100000
Barbican CLI to manage certs :
barbican secret store --payload-content-type='text/plain'
--name='certificate' --payload="$(cat server.crt)"
barbican secret store --payload-content-type='text/plain'
--name='private_key' --payload="$(cat server.key)"
barbican secret container create --name='tls_container'
--type='certificate' --secret="certificate=$(barbican secret list |
awk '/ certificate / {print $2}')" --secret="private_key=$(barbican
secret list | awk '/ private_key / {print $2}')"
API:
Neutron API to provide custom attributes:
curl -i -X POST http://<neutron_end_point>:9696/v2.0/ports.json -H
"UserAgent: pythonneutronclient" -H "ContentType:
application/json" -H "Accept: application/json" -H "X-Auth-Token:
0cb1bad6081e4ca383495a3f5a3ea718" -d '{"port": {"network_id":
"9be1ce8e-5226-4046-be82-100fcd041dc1", "fixed_ips": [{"subnet_id":
"c2968821-e52b-4c2e-a895-4ded7abf2edb", "ip_address":
"192.168.1.5"}], "custom_attributes": [{"client_timeout=50000",
"tls_container=http://<barbican_ep>/v1/containers/<container_uuid>"}]
, "admin_state_up": true}}'
Barbican API to create private_key secrets :
curl -i -X POST https://<barbicanapi_end_point>:9311/v1/secrets -H
"content-type:application/json" -H "X-Auth-Token:$TOKEN" \
-d '{"name": "Private_Key", "payload": "<private_key>",
"payload_content_type": "text/plain"}'
Barbican API to create certificate secrets:
curl -i -X POST https://<barbicanapi_end_point>:9311/v1/secrets -H
"content-type:application/json" -H "X-Auth-Token:$TOKEN" \
-d '{"name": "Certificate", "payload": "<certificate>",
"payload_content_type": "text/plain"}'
Barbican API to create pem containers:
curl -i -X POST https://<barbicanapi_end_point>:9311/v1/containers -H
"content-type:application/json" -H "X-Auth-Token:$TOKEN" \
-d '{"name": "tls_container", "type": "certificate", "secret_refs": \
[{"name": "private_key", "secret_ref": "<key_ref>"},{"name": \
"certificate", "secret_ref": "<certificate_ref>"}]}'
Barbican API to update secret ACLs:
curl -i -X PUT \https://<barbicanapi_end_point>:9311/v1/secrets/<secret_uuid>/acls \
-H "content-type: application/json" -H "X-Auth-Token:$TOKEN" \
-d '{"read": {"users": "[<user_uuids>]"}}'
EXAMPLE:
How to use Custom Attributes for SSL Cert Support with HAProxy:
Steps:
1) Add the new config file which contains keystone auth credentials to
/etc/contrail/contrail-vrouter-agent.conf file as follows:
cat /etc/contrail/contrail-vrouter-agent.conf | grep B3 lb_custom
[ SERVICE INSTANCE]
# Path to the script which handles the netns commands
netns_command = /usr/bin/opencontrail-vrouter-netns
lb_custom_attr_conf_path = /etc/contrail/contrail-vrouter-custom-attr.conf
2) The keystone file contains the following by default:
/etc/contrail/contrail-vrouter-custom-attr.conf
[DEFAULT]
[KEYSTONE]
keystone_endpoint=http://172.16.38.189:5000
barbican_endpoint=http://172.16.38.188:9311
domain_name=default
username=admin
password=abc123
project_name=demo
keystone_version=v3
[CERT]
#cert_manager=Barbican_Cert_Manager
cert_manager=Generic_Cert_Manager
3) Restart contrail vrouter agent for it to read this new config.
root@ubuntu : /var/log/keystone # service supervisor-vrouter restart
supervisor-vrouter stop / waiting
supervisor-vrouter start / running , process 18287
Steps 4 to 7 would vary based on the driver selected in the config.
Barbican Cert Manager Flow:
4) Create barbican secrets first. Make sure payloadcontenttype is ‘text/plain’.
barbican secret store --payload-content-type= 'text/plain' --name= 'certificate'
--payload= "$(cat ssl.crt)"
barbican secret store --payload-content-type= 'text/plain' --name = 'private_key'
--payload= "$(cat ssl.key)"
5) Create the barbican container referencing both the secrets
barbican container create --name= 'tls_container' --secret= "certificate=$(barbican
secret list | awk '/ certificate / {print $2}')" --secret= "private_key=$(barbican
secret list | awk '/ private_key / {print $2}')"
6) Now create the load balancer pool with tls_container as the custom-attribute as follows:
neutron lb-pool-create --subnet-id c41ec07c-9330-4469-b7f7-33fd4f29fce1 --lb-method
ROUND_ROBIN --protocol HTTPS --name TestPool --custom-attributes type = dict list = true
tls_container = http://172.16.38.188:9311/v1/containers/f3c48a4b-efab-4050-9c6e-289fb6c10168
7) Create the load balancer VIP now
neutron lb-vip-create --subnet-id c41ec07c-9330-4469-b7f7-33fd4f29fce1 --protocol HTTP
--protocol-port 443 --name TestVip TestPool
Basic Cert Manager Flow:
4) Store secrets in shared folder.
5) Now create the load balancer pool with tls_container as the custom-attribute as follows:
neutron lb-pool-create --subnet-id c41ec07c-9330-4469-b7f7-33fd4f29fce1 --lb-method
ROUND_ROBIN --protocol HTTPS --name TestPool --custom-attributes type = dict list = true
tls_container =/var/lib/contrail/shared_crts/crt.pem
7) Create the load balancer VIP now
neutron lb-vip-create --subnet-id c41ec07c-9330-4469--b7f7-33fd4f29fce1 --protocol HTTP
--protocol-port 443 --name TestVip TestPool
8) Monitor Logs by tailing the below file:
/var/log/contrail/haproxy_parse.log
BLUEPRINT and SOURCE CODE:
https://blueprints.launchpad.net/opencontrail/+spec/lbaas-custom-attr-support
https://bugs.launchpad.net/opencontrail/+bug/1475393
https://bugs.launchpad.net/opencontrail/+bug/1546253
https://bugs.launchpad.net/opencontrail/+bug/1547645
