Openstack on OpenPOWER

These are notes on how the OSUOSL is setting up Openstack on OpenPOWER (as of Feb 2019).

Summary

We use Chef as our configuration management tool of choice and have created an environment using the OpenStack Chef Cookbooks. The compute nodes are running CentOS 7 and the controller node is an x86 virtual machine running CentOS 7. All storage for the cluster is powered via a Ceph cluster backed by a 40Gb network.

We have created our own site specific wrapper Openstack cookbook that also includes any changes needed to make Openstack run on the OpenPOWER platform on CentOS 7. It currently requires some private cookbooks to run however we are working on removing those requirements.

Hardware Stack

  • Controller (1)
    • One (1) x86_64 Virtual Machine running on Ganeti with KVM
    • 12g RAM / 20g disk / 4 vCPU
  • Compute (8)
    • Five (5) 8247-22L POWER8 systems w/ 512g RAM
    • Three (3) 9006-12P POWER9 systems w/ 512g RAM
  • Storage (5)
    • Five (5) 8001-22C POWER8 systems
    • 256g RAM
    • 8 x 8TB SATA / 4 x 240GB SSD

Software Stack

Here is the current deployment of OpenStack on OpenPOWER we’re using:

  • Host Operating System
    • CentOS 7
    • Compute nodes running a mainline 4.14.x kernel
  • OpenStack
    • OpenStack Ocata Release
    • Packages from RDO
  • Ceph
    • Ceph Luminous Release

Specific Changes

Here are a listing of specific changes we made to get this working on CentOS 7 on OpenPOWER.

Building of ppc64le binaries

RDO technically doesn’t include ppc64le in its distribution, however the vast majority of the packages are noarch so this isn’t a problem. But there are a few cases where this isn’t right. To do that, we’ve pulled the SRPMs from http://vault.centos.org/ and published them in our own repositories at http://ftp.osuosl.org/pub/osl/repos/yum/openpower/centos-7/ppc64le/.

Turn off SMT

The SMT feature in the PPC64 cpu needs to be turned off via running /sbin/ppc64_cpu --smt=off. Otherwise the Little Endian guests will not run properly.

Loading kvm-hv module

The kvm-hv module will need to be loaded and the kvm-pr should not be loaded.