Experimenting with OpenStack* Sahara* on Docker* Containers

PREFACE

Docker* is an emerging technology that has become very popular recently in the market. It provides a flexible architecture to deploy applications. OpenStack* is another hot technology on the market. It has been available for several years, became more stable and also added more features support in recent releases. Sahara is a project to bring Big Data technology (Hadoop*, Spark*, …etc) into OpenStack*. It would be a perfect match when we consider the use of Docker instead of Hypervisors in the OpenStack. Docker + OpenStack can provide a better resource utilization and also may have a better performance compare with Hypervisors like KVM*, VMware*,etc. On the other hand, when we consider Big Data solutions in the cloud, people always have performance concerns about Hypervisor and Bare Metal. Docker is a good solution to solve this performance concern. This blog is a tutorial to help people to enable Docker in OpenStack Sahara*. During our installation, we suffered several issues and had assistance from Docker support or Nova Docker driver support. I also listed them in this blog as tips if you are interested in Docker and OpenStack as well.

PREREQUISITES

Hardware Configuration

We used 6 Intel based Servers with the same hardware configuration to build up OpenStack environment. Listed below is the machine details as a reference.

Machine Numbers x 6: Controller w/ Computing Node x 1, Computing Nodes x 5

Hardware Details
ItemsDetails
CPUIntel Xeon X5670 2.93Ghz
Memory64GB Memory(1333Mhz 8GB x 8)
Storage1TB SATA HDD

 

 

The commands we used to check the hardware configuration.

The Commands check the configuration
CPU Information# cat /proc/cpuinfo
Memory Information# cat /proc/meminfo
More Momory Information# dmidecode
Disk Information# fdisk -l

Software Configuration

Below is the software configuration we used in our experimental environment. At the beginning, we used an older version of Docker (v0.9), but the support functionality from Docker is not very robust since Docker is an incubation project. We would like to recommend the latest Docker to be used in your environment. A newer release can provide more features with OpenStack and also more stability. In our environment, we use Docker v1.3, it works well with OpenStack Juno. The operating system is CentOS, you can also use Ubuntu. The support from Docker and OpenStack should be the same.

Software Details
SoftwareVersion
Operation SystemCentOS 7.0
Dockerv1.3
OpenStackJuno

Network Topology

The network is an important issue if you would like to run in a complex or production environment. In our experiment, there is no need to run in a complex environment. We choose to use a simple network topology to support Docker in OpenStack. We only used one network interface in each machine. It’s a minimal solution but also enough to support OpenStack. If you are considering having complete OpenStack feature support, you may need more networks.

Network NameSubnetDetails
Public Network192.168.0.0/16Used for Floating IP Assigning
Private Network10.0.0.0/16Used for Private IP Assigning

Logical Architecture and Working Flow

The  graph below describes the logical architecture and the working flow in Sahara and the Nova Docker Driver. In Sahara, we use a CDH Plugin as an example to build up a Hadoop cluster and run Hadoop jobs in it. CDH Plugins use the existing OpenStack project, Heat, Nova, Neutron, Glance, etc., to provision a cluster and use Cloudera Manager to install the rest of the Hadoop services.

 
OpenStack Sahara Working Flow with Docker 

Step1: Call Heat to provision a cluster by using Nova API and other OpenStack services.

Step2: Enable Cloudera Manager Service

Step3: Call CM API to install the other services in the cluster
 

OpenStack ENVIRONMENT

We use OpenStack Juno as our experimental platform. OpenStack Juno has enabled Sahara project, but it remove novadocker driver support from Nova project to Stackforge. We installed the novadocker driver and modified it to support Docker v1.3. The detail configuration will be described in below.

Step 1: Software Repositories

Update the current packages.

# sudo yum update -y

Setup RDO repositories.

# sudo yum install –y https://rdo.fedorapeople.org/rdo-release.rpm

Please see https://repos.fedorapeople.org/repos/openstack/openstack-juno/ to download different OpenStack distributions.

In this case, we use https://repos.fedorapeople.org/repos/openstack/openstack-juno/rdo-release-juno-1.noarch.rpm for our experiments.

Step 2: Install Packstack Installer

Install Packstack installer from RDO repo.

# sudo yum install -y openstack-openstack

Step 3: Edit Packstack Configuration File (Optional)

Generate a configuration file.

# packstack --gen-answer-file=$answer_file_template

Customize the answer file for your needs.

Please go to the reference chapter for an answer file example.

Step 4: Run Packstack to install OpenStack

Run below command to install OpenStack via Packstack with an answer file.

packstack --anser-file=$answer_file_template

Step 5: Install Sahara

  1. Install Sahara package

  2. Edit Sahara Configuration files in
    For more information, please follow http://docs.openstack.org/developer/sahara/userdoc/configuration.guide.html

  3. Create Database Schema

  4. Start sahara service

  5. Set sahara on run level

Step 6: Wait for the installation finish

After the installation has been done, enjoy your OpenStack environment.

CONFIGURE WITH DOCKER

The Docker driver is a hypervisor driver for OpenStack Nova. It was introduced with OpenStack Havana. Although it has been removed in Juno, we can still use it in Juno, with some modification. It is also expected the driver will return to mainline Nova in Kilo release.

Nova Docker driver Working Flow

The Nova driver embeds an HTTP client to talk with Docker’s internal REST API thru a Unix socket. The driver will fetch images from OpenStack Glance and load them into the Docker file system. Images can use the ‘docker save’ command to export a Docker Image to Glance and build a Docker container in Docker registry.

How Nova-Docker works in OpenStack

Configure OpenStack to enable Docker

  1. Install Docker at first.
    Option 1 - Automatically Install from repo
    # sudo yum install docker
    Option 2 - Manually Install the latest Docker
    # wget https://get.docker.com/builds/Linux/x86_64/docker-latest -O docker
    # chmod +x docker
    # sudo ./docker -d &
    For more information, please refer to https://docs.docker.com/installation/binaries/

    For RHEL6, you will need RHEL 6.5 or higher, with a RHEL 6 kernel version 2.6.32-431 or higher as this has specific kernel fixes to allow Docker to work. For more details, just refer to link above. In order for Nova to communicate with Docker over its local socket, add nova to the docker group and restart the compute service to pick up the change:
    # usermod -G docker nova
    # service openstack-nova-computerestart

  2. Install Nova Docker Driver:
    # pip install -e git+https://github.com/stackforge/nova-docker#egg=novadocker
    Install the required modules:
    # cd src/novadocker/
    # python setup.py install

  3. Nova Configuration
    Nova must to be configured to use nova docker driver. Edit “/etc/nova/nova.conf” to configure below options:
    [DEFAULT]
    compute_driver = novadocker.virt.docker.DockerDriver
    Create the directory /etc/nova/rootwrap.d(consistent with the “filters_path” in the file /etc/nova/rootwrap.conf), inside the directory create a file “docker.filters” with the following content:
    [Filters]
    # nova/virt/docker/driver.py:'ln', '-sf', '/var/run/netns/.*'
    ln: CommandFilter, /bin/ln, root

  4. Glance Configuration
    Configure the options below in glance conf.
    [DEFAULT]
    container_formats = ami,ari,aki,bare,ovf,docker

How to use Docker in OpenStack

Below is an example in how to use the Docker image in OpenStack. You can create a custom Docker image and upload the image by using below commands.

  1. Search a docker image available in Docker public registry
    # docker search $image_name

  2. Pull the image
    # docker pull $tags/$image_name

  3. Save the image and register it in Glance
    # docker save #tags/#image_name | glance image-create --is-public=True --container-format=docker --disk-format=raw --name $tags/$image_name

  4. Boot the instance using Docker image
    # nova boot --image “samalba/hipache” --flavor m1.tiny test

  5. Check the instance is booted
    # nova list

  6. Check the instance in Docker
    # docker ps

BUILD A CUSTOM DOCKER IMAGE

You may want to build your own Docker image for OpenStack. Docker can build an image automatically by reading the instructions from a Dockerfile. A Dockerfile is a text document that contains all the commands you would like to execute in the image. By calling “docker build” command, you can build your own image.

How to Build Docker Image

# sudo docker build -t $tags/$image_name /path/to/dockerfiledir

Note: Docker will re-use the intermediate images, accelerating the building of an image significantly. Please make sure all the required commands are at the top of the Dockerfile. For those changeable commands, please put them at the bottom of the Dockerfile. This method can help in building the image by using cache to run the usual commands.

For a Dockerfile usage, please refer to http://docs.docker.com/reference/builder/

For a complete Dockerfile example, please check the reference.

TROUBLE SHOOTING

  1. The number of vCPU is always “1” for every computing node.
    Solution:
    There is a different concept between Docker and Hypervisor. Please update the source code below in the nova docker driver.
    stats = {
        'vcpus' : 1,  # Change the number to apply the number of vCPU
        'vcpus_used': 0,
        ...
    }

  2. When Remotely running the command “ls .ssh/authorized_keys” -fails during starting the instances.
    This command is for cloud-init to generating authorized_keys in the instance. But Docker cannot support this feature. For this situation, please MARK the command in “/usr/lib/python2.7/site-packages/sahara/service/engine.py”

  3. Remote login failed by using private key.
    Modify _ssh.connect(host, username=username, password=”xxx”, sock=proxy) in “/usr/lib/python2.7/site-packages/sahara/utils/ssh_remote.py”
    Please make sure the password is also set for Docker image.

  4. When you remotely run the command “sudo hostname” it fails.
    Docker cannot support modifying the hostname before Docker v1.2. Please upgrade Docker to v1.2 or later.
    Workaround: There is a workaround to customize the host file in Docker manually. Please refer to: http://jasonincode.com/customizing-hosts-file-in-docker/#.VFl1DPmUdZu

  5. When you remotely run the command “sudo mv etc-host /etc/hosts” it fails.
    Docker cannot support modifying /etc/hosts before DOcker v1.2. Please upgrade Docker to v1.2 or later.

  6. After upgrade to Docker v1.2, “sudo mv etc-host /etc/hosts” responds the device is busy.

    Please use “sudo cp etc-hosts /etc/hosts” to replace the command “sudo mv etc-hosts /etc/hosts” in sahara/service/engine.py of Sahara source code. Another way to do is to set the hosts manually when all the instances launched.

  7. My instances cannot reach each other.
    Please make sure all the settings are fine in /etc/hosts and proxy variables: http_proxy, https_proxy, and no_proxy

  8. Could not open session when run the command “service cloudera-scm-server-db”
    By default Docker cannot allow the authorization to create the Database using the command. To fix, add the “Privileged” parameter in nova docker driver and set it to True to solve the issue.
    Workaround: A workaround is to modify “/etc/security/limits.d/xxx.conf” after the instance launched and set the value from “hard”/”-” to “soft” to avoid the issue.

  9. How to check if CM can be reachable?
    Use the command:
    # curl -X GET -u "admin:admin" -l http://$cm_host:7180/api/v7/tools/echo?message=hello

  10. CM response “Connection refused”.
    Please check the firewall has been passed for Docker.
    # iptables -t nat -A OUTPUT -j DOCKER

  11. The log response ApiException:{}(error 500)
    Add extra time in sahara/plugins/cdh/deploy.py of Sahara source code, default is 300 secs for timeout. Another root cause could be proxy issue. Please make sure you have correct settings for http_proxy, https_proxy, and no_proxy in the environment.

  12. There are several ports that cannot be accessed when starting the Cluster.
    Please expose all the necessary ports in Dockerfile.
    Or you can add “Publish-all-ports” parameter in client.py of nova docker driver source code, please make sure set it to true.

  13. There is no storage space in Data Node.
    By default, Docker uses a 10GB Root Disk and a reserved space also needs to be set for non-DFS usage in CM.
    Please change the parameter when you launch Docker binary. For more information, please refer to https://github.com/snitm/docker/blob/master/daemon/graphdriver/devmapper/README.md
    Another way to do is to set reserved space to smaller in CM HDFS Configuration.

  14. My instance cannot access files for swift package in Sahara.

    Please make sure your instances can reach the internet at first.

    Or you can set up your own site (like ftp or nfs) to get the necessary files. Please also remember to change the swift package url in node group templates of Sahara.

  15. There is no cloudera-scm-agent running on the host.
    Sometimes cloudera-scm-agent may not started automatically or get an error when starting. Please restart the service manually by using the command: “sudo service cloudera-scm-agent restart”

  16. Docker cannot connect to proxy.
    Please use HTTP_PROXY when you launch Docker process in the background, for example: “sudo HTTP_PROXY=xxx ./docker -d &”. Then you create the Docker image by using this process with HTTP_PROXY.

  17. Container cannot start in other Computing Nodes except the Controller?
    Docker Image must be copied to individual Computing Nodes manually. Please also don’t forget to register the image to Docker register and using “docker images” to confirm the image is existing in the Docker. 

standard
Para obter informações mais completas sobre otimizações do compilador, consulte nosso aviso de otimização.