VENOM & OpenStack: All You Need to Know

All of our infrastructure is now patched against the new VENOM vulnerability, which means that all of our customers are safe from this problem. However, we wanted to release a blog post to go in details on some common questions regarding this new vulnerability if you use OpenStack in any way.

It's extremely important to know that 99% of OpenStack users are affected, due to the fact that most OpenStack clouds run KVM which is affected by this issue, so you must be very careful.

OpenStack users

If your operator says that you're safe from it, however, your instance action log does not show any signs of reboot or suspend/resume, you must double check with them, as a reboot or suspend/resume is required in order for this fix to apply.

You can check this using the OpenStack CLI (or any other way you wish). As we see in the following from a server on our infrastructure, it was suspended/resumed after 13th of May 2015, the day the issue went public.

You can also check your providers control panel, or in our case inside our CloudConsole “History” tab for the server. As a user, you cannot do much other than contact your provider to make sure they take care of this for you.

OpenStack operators

In this section, we refer to operators as anyone who's involved in making sure an OpenStack cloud stays up, stable and remains secure. It's extremely important to patch against this as the problem is that by gaining access to the hypervisor through a virtual machine, a user can effectively take over the entire cloud, as typically compute nodes have SSH public key authentication pre-configured in most cases.

In addition, you open up access to your cloud's message queue which means that a malicious user can start sending messages on the queue which could be executed by other nodes on the cloud, not to mention gaining access to your management network, all in all, you really should be patching this as soon as possible.

Update QEMU

The first step is to make sure that you're running the latest patched qemu on your compute nodes. As different Linux distributions use different versioning schemes, it's best to check with your documentation for the patched version. We've provided a list of a few security bulletins from different distributions here:

Ubuntu
RedHat
For reference, you can update your Ubuntu system by simply running apt-get update and then followed by apt-get upgrade. Once done, you should check that you're up to date by running the following (example is on Ubuntu 14.04 LTS):

$ dpkg -l | grep qemu-system-x86
ii qemu-system-x86 2.0.0+dfsg-2ubuntu1.11 amd64 QEMU full system emulation binaries (x86)

Reboot or Suspend/Resume All Instances

Once you do the upgrade, the process which is running your virtual machine remains the same exact one that was vulnerable, therefore, you have to restart the process so that it loads the new patched version.

You can either choose to reboot the servers which will kill the qemu-system-x86_64 process and start the new patched one again.

Alternatively, you can choose to suspend and resume, which saves the content of the instances' memory on disk and kills the qemu-system-x86_64 process when suspending, then starts a new process and reads the saved contents of memory in again.

The advantage of suspend and resume is that you do not lose the state of the virtual machine, so no actual reboot happens, reducing potential headaches of rebooting servers. You can do both using the OpenStack API, Horizon or CLI clients. This remains up to you on how you'd like to automate that process.

Verify Instances

The most important thing is to make sure that all of our work was done properly and that we did not miss any instances, we recommend running the following which will check for all qemu processes that have not yet been restarted to use the new version.

$ pgrep qemu-system | xargs -n 1 lsof -p | grep txt | grep deleted

The command above should return nothing at all. If it returns a result such as the following, it means that the qemu instance with PID 407 is running an old release. It's important to note that this command gives the right answer only if you have already upgraded your qemu.

$ pgrep qemu-system | xargs -n 1 lsof -p | grep txt
qemu-syst 407 libvirt-qemu txt REG 9,1 6155432 64359274 /usr/bin/qemu-system-x86_64 (deleted)

Hopefully this article has been helpful in giving you all the information you need to know about it and how to address it, feel free to leave any comments or questions below if you need help with anything specific!