diff --git b/.gitignore a/.gitignore new file mode 100644 index 0000000..53b7af4 --- /dev/null +++ a/.gitignore @@ -0,0 +1,3 @@ +*.swp +*.swo +*~ diff --git b/GITSALT.md a/GITSALT.md new file mode 100644 index 0000000..6615098 --- /dev/null +++ a/GITSALT.md @@ -0,0 +1,98 @@ + +## Git backend tutorial + +### 1. Install salt master and register minion + +```bash +sudo apt-get install salt-minion +sudo apt-get install salt-master +``` + +#### Edit /etc/salt/minion to set master to 127.0.0.1 +#### Open 4505 and 4506 with ufw allow. +#### Restart master and minion. Accept minion key with salt-key -A. +#### Use -l debug option to show debug messages. + +### 2. Install pygit2 + +#### 2.1 Without adding the repo, installing pygit is a bit difficult. Use this: + +```bash +sudo add-apt-repository ppa:dennis/python +sudo apt-get update +sudo apt-get install python-pygit2 +``` + +#### 2.2 Copy keys. + +### 3. Modify the *master* config file: + +```yaml +fileserver_backend: + - git + +gitfs_remotes: + - git@git.ik.bme.hu:circle/salt.git: + - pubkey: /root/.ssh/git.pub + - privkey: /root/.ssh/git + +pillar_roots: + base: + - /home/cloud/salt/pillar + +gitfs_root: salt +``` + +### 4. Clone pillar to /home/cloud/ +```bash +git clone `https://git.ik.bme.hu/circle/salt.git +``` + +### 5. Finish: call salt '*' state.sls allinone (or whatever you need) + +### the *master* config file: + +#### The default git provider is pygit2. You can change that to dulwich ot gitpython. + +```yaml +gitfs_provider: dulwich +``` + +#### Include git in the fileserver_backend list: + +```yaml +fileserver_backend: + - git +``` + +#### Specify one or more git://, https://, file://, or ssh:// URLs in gitfs_remotes to configure which repositories to cache and search for requested files: + +```yaml +gitfs_remotes: + - git@git.ik.bme.hu:circle/salt.git +``` + +> The gitfs_remotes option accepts an ordered list of git remotes to cache and search, in listed order, for requested files. + +#### Serving from subdirectory + +```yaml +gitfs_root: foo/baz +``` + +#### Other options + +Its possible to change branches, enviroments + +Change branch: + +```yaml +gitfs_base: salt-base +``` + + +### Tutorial with more information: +http://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html + +### Local gitfs issue: +https://github.com/saltstack/salt/issues/6660 diff --git b/README.md a/README.md new file mode 100644 index 0000000..2e3b19b --- /dev/null +++ a/README.md @@ -0,0 +1,199 @@ +# CIRCLE Project - Salt Installer + +## OS Support + +* Red Hat Linux family: + * Red Hat Enterprise Linux 7+ + * Cent OS 7+ + * Scientific Linux 7+ +* Debian Linux family: + * Debian linux 8+ + * Ubuntu linux 14.04 LTS + +## Prerequisites + +### Red Hat family + +Install EPEL repository (if the link is broken, please contact us): +```bash +sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm +``` + +Install some important packages: +```bash +sudo yum install python2-pip gcc vim git +``` + +### Debian family + +Install some important packages: +```bash +sudo apt-get update +sudo apt-get install python-pip vim git +``` + +## Install Salt + +```bash +sudo pip install salt==2014.7.1 +``` + +## Get the installer +Clone circle installer git repository into cloud home + +```bash +git clone https://git.ik.bme.hu/circle/salt.git +``` + +## Change variables +Modify installer.sls file +``` +vim salt/pillar/installer.sls +``` + +Most used variables +------------------- +* **proxy_secret**: This is used to provide cryptographic signing, and should be set to a unique, unpredictable value. +* **secret_key**: This is used to provide cryptographic signing, and should be set to a unique, unpredictable value. +* **deployment_type**: local (development) or production +* **admin_user**: user name to login in as admin on the site +* **admin_pass**: password to login in as admin on the site +* **database**: + * **password**: database user’s password +* **amqp**: + * **password**: amqp user’s password + * **host**: amqp server IP - usually runs at localhost +* **graphite**: + * **password**: graphite user’s password + * **host**: graphite server IP - usually runs at localhost +* **nfs**: + * **enabled**: nfs is enabled + * **server**: nfs server’s hostname + * **network**: nfs server’s network to access files + * **directory**: this directory will be shared +* **storagedriver**: + * **queue_name**: the server’s hostname +* **fwdriver**: + * **queue_name**: the server’s hostname + * **gateway**: the server’s gateway + * **external_net**: the server’s network + * **external_if**: the server’s network interface + +Other variables +--------------- + +* user: user who will install the software +* time zone: the server’s time zone, format is region/city +* amqp: + * user: amqp user + * port: amqp server’s port + * vhost: virtual host - specifies the namespace for entities (exchanges and queues) referred to by the protocol +* agent: + * repo_revision: revision +* agentdriver: + * repo_revision: revision +* cache: cache url - usually pylibmc://127.0.0.1:11211/ +* database: + * name: django database’s name + * user: database user +* fwdriver: + * repo_revision: revision + * user: fwdriver user name + * vm_if: vm interface + * vm_et: vm network + * management_if: management interface + * reload_firewall_timeout: timeout for synchronous firewall reload +* graphite: + * user: graphite user + * port: graphite server’s port + * secret_key: graphite’s secret key +* manager: + * repo_revision: revision +* monitor-client: + * repo_revision: revision +* storage-driver: + * repo_revision: revision +* vm-driver: + * repo_revision: revision +* vnc-driver: + * repo_revision: revision + +## Install Circle +Run the following installation command: + +```bash +sudo salt-call state.sls allinone --local --file-root=/home/$USER/salt/salt --pillar-root=/home/$USER/salt/pillar +``` +After this finished, you have to get "Failed: 0" message. +If installer fails, please visit the [Troubleshooting](#troubleshooting) paragraph. + +After install, delete agent.conf or agent.service file: + +If you have upstart: +```bash +sudo rm -f /etc/init/agent.conf +``` +Or if you have systemd: +```bash +sudo rm -f /etc/systemd/system/agent.service +``` + +## Quickstart - Standalone Node + +### Login +Log in into the Circle website with admin (the site is accessable on the 443 port). Name and password is in the `salt/pillar/installer.sls`. + +### Create Node +To run virtual machines, we need to create nodes - and add to the system. Click on the new icon in the dashboard, Nodes menu. + +#### Configure Node + +To standalone configuration, type the current machine's hostname to Host/name, MAC address to Host/MAC, IP to HOST/IP. Choose managed-vm as VLAN. + +#### Activate Node + +Click on the 'Activate' icon to use the Node. + +### Start Virtual Machine + +To create new Virtual Machine, we use Templates - images based on previously saved VMs. Currently we haven't got any template - so let's create a new one. Click on Templates/new icon and choose 'Create a new base VM without disk'. + +#### Configure Template + +Set name, CPU and RAM settings, architecture. Check in the boot menu box, select network and lease, write down, which operating system will you use. Finally, create a template. +> The rows marked with astersk need to be filled. + + + +#### Add disk + +Currently we don't have any disks attached to our VM. To add, click on the Resources menu, 'create disk' icon, set the name and size. + + + +#### Attach ISO + +To install an OS, we can use ISO images, to boot from. Click on 'download disk' and type the ISO's URL. + + + +### Start Virtual Machine +Finally, we can run the machine. Click on 'deploy' and start it. You can choose, on which node do you want to run. + + + +## Troubleshooting ## + +### Portal won't load +Maybe port 443 is closed. Check and open it. + + +### Portal won't load on Ubuntu 14.04 +```bash +sudo service nginx restart +``` + +### Cannot reach the internet on VM-s on distro from Red Hat family +```bash +sudo systemctl restart systemd-sysctl +``` diff --git b/_static/images/configure_node.jpg a/_static/images/configure_node.jpg new file mode 100644 index 0000000..965d10c Binary files /dev/null and a/_static/images/configure_node.jpg differ diff --git b/_static/images/disk.jpg a/_static/images/disk.jpg new file mode 100644 index 0000000..573e3b5 Binary files /dev/null and a/_static/images/disk.jpg differ diff --git b/_static/images/iso.jpg a/_static/images/iso.jpg new file mode 100644 index 0000000..44d53f7 Binary files /dev/null and a/_static/images/iso.jpg differ diff --git b/_static/images/ubuntu.png a/_static/images/ubuntu.png new file mode 100644 index 0000000..db54910 Binary files /dev/null and a/_static/images/ubuntu.png differ diff --git b/install a/install new file mode 100755 index 0000000..7c600ad --- /dev/null +++ a/install @@ -0,0 +1,22 @@ +#!/bin/sh + +if [ $(id -u) -ne 0 ]; then + RED_UNDERLINED='\033[4;31m' + NC='\033[0m' # No Color + echo -e $RED_UNDERLINED"Please run as root or use sudo!"$NC + exit +fi + +FULLPATH=$(readlink -m $0) +PREFIX=$(dirname $FULLPATH) + +pip install -r $PREFIX/requirements.txt + +$PREFIX/kvm-ok > /dev/null +retv=$? +EXTRAPARAMS="" +if [ $retv -eq 0 ]; then + EXTRAPARAMS="--kvm-present" +fi + +python $PREFIX/install.py $EXTRAPARAMS $@ diff --git b/install.py a/install.py new file mode 100644 index 0000000..c6a0fde --- /dev/null +++ a/install.py @@ -0,0 +1,216 @@ +import salt.client +from salt import config +from salt.log.setup import setup_console_logger +from os.path import join, abspath, dirname +from netifaces import ifaddresses, gateways, AF_INET +from netaddr import IPNetwork +import socket +import yaml +import random +import os +import getpass +from halo import Halo +import argparse + + +PREFIX = dirname(__file__) + + +def get_timezone(): + localtime = '/etc/localtime' + try: + zonefile = abspath(os.readlink(localtime)) + zone_parts = zonefile.split('/') + return join(zone_parts[-2], zone_parts[-1]) + except Exception: + return 'Europe/Budapest' + + +def get_gateway(): + return gateways()['default'][AF_INET] + + +def get_default_gw(): + return get_gateway()[0] + + +def get_interface(): + return get_gateway()[1] + + +def get_ip_with_mask(intf): + ip = ifaddresses(intf)[AF_INET][0] + return str(IPNetwork(join(ip['addr'], ip['netmask']))) + + +def get_hostname(): + return str(socket.gethostname().split('.')[0]) + + +def print_warning(text): + RED_UNDERLINED = '\033[4;31m' + NC = '\033[0m' # No Color + print(RED_UNDERLINED + text + NC) + + +def input_password_with_retype(): + pw = getpass.getpass("Enter admin password:") + if len(pw) == 0: + print_warning('Please enter a non-empty password!') + return ('', False) + pw2 = getpass.getpass("Retype password:") + status = pw == pw2 + if not status: + print_warning('The passwords are different.') + return (pw, status) + + +def input_admin_password(): + pw, status = input_password_with_retype() + while not status: + pw, status = input_password_with_retype() + return pw.encode('utf8') + + +def yaml_pretty_dump(data, file, **extra): + yaml.dump(data, file, encoding='utf-8', default_flow_style=False, **extra) + + +def dump_errors(result): + # Filter errors only + errors = {} + for key, data in result.iteritems(): + if not data['result']: + errors[key] = data + with open(join(PREFIX, 'errors.yml'), 'w') as f: + yaml_pretty_dump(errors, f) + + +class KeyStore: + """ Loads, stores, generates, and saves secret keys """ + def __init__(self, keyfile): + self.keyfile = keyfile + self.data = {} + try: + with open(keyfile) as f: + self.data = yaml.safe_load(f) + except Exception: + pass + + def gen_key(self, length): + s = "abcdefghijklmnopqrstuvwxyz01234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ" + return "".join(random.sample(s, length)) + + def get_key(self, name): + key = self.data.get(name) + if key is None: + key = self.gen_key(16) + self.data[name] = key + return key + + def save(self): + with open(self.keyfile, 'w') as f: + yaml.dump(self.data, f) + +parser = argparse.ArgumentParser() +parser.add_argument('--kvm-present', action='store_true', + help='Installs with KVM hypervisor otherwise with QEMU.') +parser.add_argument('--dev', action='store_true', + help='Installs Develpment version') +parser.add_argument('--local', action='store_true', + help='Installs Develpment version') +args = parser.parse_args() + +if args.dev or args.local: + deployment_type = 'local' +else: + deployment_type = 'production' + +KEYFILE = join(PREFIX, '.circlekeys') +ks = KeyStore(KEYFILE) + +installer_sls = { + 'user': 'cloud', + 'proxy_secret': ks.get_key('proxy_secret'), + 'secret_key': ks.get_key('secret_key'), + 'timezone': get_timezone(), + 'deployment_type': deployment_type, + 'admin_user': 'admin', + 'admin_pass': input_admin_password(), + 'database': { + 'name': 'circle', + 'user': 'circle', + 'password': ks.get_key('database_password'), + }, + 'amqp': { + 'user': 'cloud', + 'password': ks.get_key('amqp_password'), + 'host': '127.0.0.1', + 'port': 5672, + 'vhost': 'circle', + }, + 'graphite': { + 'user': 'monitor', + 'password': ks.get_key('graphite_password'), + 'host': '127.0.0.1', + 'port': 5672, + 'vhost': 'monitor', + 'queue': 'monitor', + 'secret_key': ks.get_key('graphite_secret_key'), + }, + 'cache': 'pylibmc://127.0.0.1:11211/', + 'nfs': { + 'enabled': True, + 'server': '127.0.0.1', + 'network': '127.0.0.0/8', + 'directory': '/datastore', + }, + 'storagedriver': { + 'queue_name': get_hostname(), + }, + 'fwdriver': { + 'gateway': get_default_gw().encode('utf-8'), + 'external_if': get_interface().encode('utf-8'), + 'external_net': get_ip_with_mask(get_interface()).encode('utf-8'), + 'queue_name': get_hostname(), + 'management_if': 'ethy', + 'trunk_if': 'linkb', + }, + 'vmdriver': { + 'hypervisor_type': 'kvm' if args.kvm_present else 'qemu', + }, +} + +ks.save() # Save secret keys + +# Make installer.sls +INSTALLERT_SLS = join(PREFIX, 'pillar/installer.sls') +with open(INSTALLERT_SLS, 'w') as f: + yaml_pretty_dump(installer_sls, f) + +# NOTE: default logfile is '/var/log/salt/minion' +opts = config.minion_config('') +opts['file_client'] = 'local' +# NOTE: False will cause salt to only display output +# for states that failed or states that have changes +opts['state_verbose'] = False +opts['file_roots'] = {'base': [join(PREFIX, 'salt')]} +opts['pillar_roots'] = {'base': [join(PREFIX, 'pillar')]} +setup_console_logger(log_level='info') +caller = salt.client.Caller(mopts=opts) +# Run install with salt +with Halo(text='Installing', spinner='dots'): + result = caller.function('state.sls', 'allinone', with_grains=True) + +# Count errors and print to console +error_num = 0 +for key, data in result.iteritems(): + if not data['result']: + print('Error in state: %s' % key) + error_num += 1 + +if error_num == 0: + print('Succesfully installed!') +else: + print_warning('%i error occured during install!' % error_num) + dump_errors(result) diff --git b/kvm-ok a/kvm-ok new file mode 100755 index 0000000..5040935 --- /dev/null +++ a/kvm-ok @@ -0,0 +1,105 @@ +#!/bin/sh +# +# kvm-ok - check whether the CPU we're running on supports KVM acceleration +# Copyright (C) 2008-2010 Canonical Ltd. +# +# Authors: +# Dustin Kirkland <kirkland@canonical.com> +# Kees Cook <kees.cook@canonical.com> +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 3, +# as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +set -e + +assert_root() { + if [ "$(id -u)" != "0" ]; then + echo "INFO: For more detailed results, you should run this as root" + echo "HINT: sudo $0" + exit 1 + fi +} + +verdict() { + # Print verdict + if [ "$1" = "0" ]; then + echo "KVM acceleration can be used" + exit 0 + else + echo "KVM acceleration can NOT be used" + exit 1 + fi +} + +# check cpu flags for capability +virt=$(egrep -m1 -w '^flags[[:blank:]]*:' /proc/cpuinfo | egrep -wo '(vmx|svm)') || true +[ "$virt" = "vmx" ] && brand="intel" +[ "$virt" = "svm" ] && brand="amd" + +if [ -z "$virt" ]; then + echo "INFO: Your CPU does not support KVM extensions" + assert_root + verdict 1 +fi + +# Now, check that the device exists +if [ -e /dev/kvm ]; then + echo "INFO: /dev/kvm exists" + verdict 0 +else + echo "INFO: /dev/kvm does not exist" + echo "HINT: sudo modprobe kvm_$brand" +fi + +assert_root + +# Prepare MSR access +msr="/dev/cpu/0/msr" +if [ ! -r "$msr" ]; then + modprobe msr +fi +if [ ! -r "$msr" ]; then + echo "You must be root to run this check." >&2 + exit 2 +fi + +echo "INFO: Your CPU supports KVM extensions" + +disabled=0 +# check brand-specific registers +if [ "$virt" = "vmx" ]; then + BIT=$(rdmsr --bitfield 0:0 0x3a 2>/dev/null || true) + if [ "$BIT" = "1" ]; then + # and FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX clear (no tboot) + BIT=$(rdmsr --bitfield 2:2 0x3a 2>/dev/null || true) + if [ "$BIT" = "0" ]; then + disabled=1 + fi + fi + +elif [ "$virt" = "svm" ]; then + BIT=$(rdmsr --bitfield 4:4 0xc0010114 2>/dev/null || true) + if [ "$BIT" = "1" ]; then + disabled=1 + fi +else + echo "FAIL: Unknown virtualization extension: $virt" + verdict 1 +fi + +if [ "$disabled" -eq 1 ]; then + echo "INFO: KVM ($virt) is disabled by your BIOS" + echo "HINT: Enter your BIOS setup and enable Virtualization Technology (VT)," + echo " and then hard poweroff/poweron your system" + verdict 1 +fi + +verdict 0 diff --git b/pillar/agent.sls a/pillar/agent.sls new file mode 100644 index 0000000..7aa0a36 --- /dev/null +++ a/pillar/agent.sls @@ -0,0 +1,3 @@ +agent: + repo_name: https://git.ik.bme.hu/circle/agent.git + repo_revision: master diff --git b/pillar/agentdriver.sls a/pillar/agentdriver.sls new file mode 100644 index 0000000..f425fa9 --- /dev/null +++ a/pillar/agentdriver.sls @@ -0,0 +1,3 @@ +agentdriver: + repo_name: https://git.ik.bme.hu/circle/agentdriver.git + repo_revision: master diff --git b/pillar/amqp.sls a/pillar/amqp.sls new file mode 100644 index 0000000..fe70428 --- /dev/null +++ a/pillar/amqp.sls @@ -0,0 +1,14 @@ +amqp: + user: cloud + password: password + host: localhost + port: 5672 + vhost: circle + +graphite: + user: monitor + password: monitor + host: localhost + port: 5672 + vhost: monitor + queue: monitor diff --git b/pillar/common.sls a/pillar/common.sls new file mode 100644 index 0000000..0e7a732 --- /dev/null +++ a/pillar/common.sls @@ -0,0 +1,24 @@ +user: cloud +cache: pylibmc://127.0.0.1:11211/ +proxy_secret: xooquageire7uX1 +secret_key: Ga4aex3Eesohngo +timezone: Europe/Budapest + +deployment_type: local + +admin_user: admin +admin_pass: 4j23oihreehfd + +database: + name: circle + user: circle + password: hoGei6paiN0ieda + +graphite: + secret_key: ahf2aim7ahLeo8n + +nfs: + enabled: false + server: localhost + network: 192.168.1.0/24 + directory: /datastore diff --git b/pillar/dnsdriver.sls a/pillar/dnsdriver.sls new file mode 100644 index 0000000..cc383b7 --- /dev/null +++ a/pillar/dnsdriver.sls @@ -0,0 +1,5 @@ +dnsdriver: + repo_name: https://git.ik.bme.hu/circle/dnsdriver.git + repo_revision: master + dns_db_dir: /var/lib/circle/dnsdriver + address_list: 127.0.0.1 diff --git b/pillar/firewall.sls a/pillar/firewall.sls new file mode 100644 index 0000000..ff1d9ac --- /dev/null +++ a/pillar/firewall.sls @@ -0,0 +1,21 @@ +fwdriver: + repo_name: https://git.ik.bme.hu/circle/fwdriver.git + repo_revision: master + + user: fw + + queue_name: cloud + + vm_if: vm + vm_net: 192.168.2.254/24 + vm_net_ip: 192.168.2.254 + vm_net_mask: 255.255.255.0 + + management_if: eth5 + management_net: 192.168.1.254/24 + + external_if: eth0 + external_net: 10.0.0.97/16 + gateway: 10.0.255.254 + + reload_firewall_timeout: 120 diff --git b/pillar/installer.sls a/pillar/installer.sls new file mode 100644 index 0000000..239afab --- /dev/null +++ a/pillar/installer.sls @@ -0,0 +1,50 @@ +#user: cloud + +#proxy_secret: xooquageire7uX1 +#secret_key: Ga4aex3Eesohngo +#timezone: Europe/Budapest + +#deployment_type: local + +#admin_user: admin +#admin_pass: 4j23oihreehfd + +#database: +# name: circle +# user: circle +# password: hoGei6paiN0ieda + +#amqp: +# user: cloud +# password: password +# host: 127.0.0.1 +# port: 5672 +# vhost: circle + +#graphite: +# user: monitor +# password: monitor +# host: 127.0.0.1 +# port: 5672 +# vhost: monitor +# queue: monitor +# secret_key: ahf2aim7ahLeo8n + +#cache: pylibmc://127.0.0.1:11211/ + +#nfs: +# enabled: true +# server: 10.0.0.115 +# network: 192.168.1.0/24 +# directory: /datastore + +#storagedriver: +# queue_name: cloud-6605 + +#fwdriver: +# queue_name: cloud-6605 +# gateway: 10.0.255.254 +# external_net: 10.0.0.115/16 +# external_if: eth0 +# trunk_if: linkb +# management_if: ethy diff --git b/pillar/manager.sls a/pillar/manager.sls new file mode 100644 index 0000000..00f841f --- /dev/null +++ a/pillar/manager.sls @@ -0,0 +1,4 @@ +manager: + repo_name: https://git.ik.bme.hu/circle/cloud.git + repo_revision: master + diff --git b/pillar/monitor-client.sls a/pillar/monitor-client.sls new file mode 100644 index 0000000..56bef4b --- /dev/null +++ a/pillar/monitor-client.sls @@ -0,0 +1,3 @@ +monitor-client: + repo_name: https://git.ik.bme.hu/circle/monitor-client.git + repo_revision: master diff --git b/pillar/storagedriver.sls a/pillar/storagedriver.sls new file mode 100644 index 0000000..7cafb0f --- /dev/null +++ a/pillar/storagedriver.sls @@ -0,0 +1,4 @@ +storagedriver: + repo_name: https://git.ik.bme.hu/circle/storagedriver.git + repo_revision: master + queue_name: storageserver diff --git b/pillar/top.sls a/pillar/top.sls new file mode 100644 index 0000000..9ab9a55 --- /dev/null +++ a/pillar/top.sls @@ -0,0 +1,15 @@ +base: + '*': + - vmdriver + - amqp + - agentdriver + - agent + - storagedriver + - vncproxy + - monitor-client + - vmdriver + - firewall + - manager + - common + - dnsdriver + - installer diff --git b/pillar/vmdriver.sls a/pillar/vmdriver.sls new file mode 100644 index 0000000..5107834 --- /dev/null +++ a/pillar/vmdriver.sls @@ -0,0 +1,4 @@ +vmdriver: + repo_name: https://git.ik.bme.hu/circle/vmdriver.git + repo_revision: master + hypervisor_type: kvm diff --git b/pillar/vncproxy.sls a/pillar/vncproxy.sls new file mode 100644 index 0000000..e88be6c --- /dev/null +++ a/pillar/vncproxy.sls @@ -0,0 +1,3 @@ +vncproxy: + repo_name: https://git.ik.bme.hu/circle/vncproxy.git + repo_revision: master diff --git b/requirements.txt a/requirements.txt new file mode 100644 index 0000000..9694b20 --- /dev/null +++ a/requirements.txt @@ -0,0 +1,4 @@ +salt==2014.7.1 +netaddr==0.7.14 +netifaces==0.10.6 +halo==0.0.7 diff --git b/salt/agentdriver/configuration.sls a/salt/agentdriver/configuration.sls new file mode 100644 index 0000000..d847eb4 --- /dev/null +++ a/salt/agentdriver/configuration.sls @@ -0,0 +1,44 @@ +/home/{{ pillar['user'] }}/.virtualenvs/agentdriver/bin/postactivate: + file.managed: + - source: salt://agentdriver/files/postactivate + - template: jinja + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 700 + +/etc/incron.d/agentdriver: + file.managed: + - source: salt://agentdriver/files/agentdriver.incron + - template: jinja + - user: root + - group: root + +{% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} +/etc/systemd/system/agentdriver.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/agentdriver/miscellaneous/agentdriver.service + +{% else %} + +/etc/init/agentdriver.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/agentdriver/miscellaneous/agentdriver.conf +{% endif %} + +{% if grains['os_family'] == 'RedHat' %} +incrond: +{% else %} +incron: +{% endif %} + service: + - full_restart: true + - enable: true + - running + - watch: + - file: /etc/incron.d/agentdriver diff --git b/salt/agentdriver/files/agentdriver.incron a/salt/agentdriver/files/agentdriver.incron new file mode 100644 index 0000000..bfd10f7 --- /dev/null +++ a/salt/agentdriver/files/agentdriver.incron @@ -0,0 +1 @@ +/var/lib/libvirt/serial IN_CREATE setfacl -m u:{{ pillar['user'] }}:rw $@/$# diff --git b/salt/agentdriver/files/postactivate a/salt/agentdriver/files/postactivate new file mode 100644 index 0000000..7bd0330 --- /dev/null +++ a/salt/agentdriver/files/postactivate @@ -0,0 +1,3 @@ +export AMQP_URI=amqp://{{ pillar['amqp']['user'] }}:{{ pillar['amqp']['password'] }}@{{ pillar['amqp']['host'] }}:{{ pillar['amqp']['port'] }}/{{ pillar['amqp']['vhost'] }} +export CACHE_URI={{ pillar['cache'] }} + diff --git b/salt/agentdriver/gitrepo.sls a/salt/agentdriver/gitrepo.sls new file mode 100644 index 0000000..8804504 --- /dev/null +++ a/salt/agentdriver/gitrepo.sls @@ -0,0 +1,11 @@ +include: + - common + +gitrepo_agentdriver: + git.latest: + - name: {{ pillar['agentdriver']['repo_name'] }} + - rev: {{ pillar['agentdriver']['repo_revision'] }} + - target: /home/{{ pillar['user'] }}/agentdriver + - user: {{ pillar['user'] }} + - require: + - pkg: git diff --git b/salt/agentdriver/init.sls a/salt/agentdriver/init.sls new file mode 100644 index 0000000..18d7a09 --- /dev/null +++ a/salt/agentdriver/init.sls @@ -0,0 +1,42 @@ +include: + - agentdriver.gitrepo + - agentdriver.virtualenv + - agentdriver.configuration + +agentdriver: + pkg.installed: + - pkgs: + - git + - ntp + - incron + {% if grains['os_family'] == 'RedHat' %} + - python2-pip + - libmemcached-devel + - python-devel + - python-virtualenvwrapper + - zlib-devel + {% else %} + - libmemcached-dev + - python-dev + - virtualenvwrapper + - python-pip + - zlib1g-dev + {% endif %} + - require_in: + - git: gitrepo_agentdriver + - virtualenv: virtualenv_agentdriver + user: + - present + - name: {{ pillar['user'] }} + - gid_from_name: True + - shell: /bin/bash + - groups: + {% if grains['os_family'] == 'RedHat' %} + - wheel + {% else %} + - sudo + {% endif %} + - require_in: + - git: gitrepo_agentdriver + - virtualenv: virtualenv_agentdriver + diff --git b/salt/agentdriver/virtualenv.sls a/salt/agentdriver/virtualenv.sls new file mode 100644 index 0000000..8d23ebc --- /dev/null +++ a/salt/agentdriver/virtualenv.sls @@ -0,0 +1,6 @@ +virtualenv_agentdriver: + virtualenv.managed: + - name: /home/{{ pillar['user'] }}/.virtualenvs/agentdriver + - requirements: /home/{{ pillar['user'] }}/agentdriver/requirements.txt + - user: {{ pillar['user'] }} + - no_chown: true diff --git b/salt/allinone.sls a/salt/allinone.sls new file mode 100644 index 0000000..50afd85 --- /dev/null +++ a/salt/allinone.sls @@ -0,0 +1,11 @@ +include: + - profile + - agentdriver + - manager + - graphite + - monitor-client + - storagedriver + - vmdriver + - vncproxy + - fwdriver + - network diff --git b/salt/common.sls a/salt/common.sls new file mode 100644 index 0000000..a128098 --- /dev/null +++ a/salt/common.sls @@ -0,0 +1,3 @@ +git: + pkg.installed + diff --git b/salt/dnsdriver/configuration.sls a/salt/dnsdriver/configuration.sls new file mode 100644 index 0000000..5fc574f --- /dev/null +++ a/salt/dnsdriver/configuration.sls @@ -0,0 +1,43 @@ +/var/lib/circle/dnsdriver: + file.directory: + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 755 + - makedirs: True + + +/var/lib/circle/dnsdriver/makefile: + file.managed: + - source: salt://dnsdriver/files/makefile + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 755 + + +/etc/systemd/system/dnscelery.service: + file.managed: + - source: salt://dnsdriver/files/dnscelery.service + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 755 + - template: jinja + + +/home/{{ pillar['user'] }}/.virtualenvs/dnsdriver/bin/postactivate: + file.managed: + - source: salt://dnsdriver/files/postactivate + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 755 + - template: jinja + - require: + - virtualenv: virtualenv_dnsdriver + +tinydns_conf: + file.managed: + - name: /etc/ndjbdns/tinydns.conf + - source: salt://dnsdriver/files/tinydns.conf + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 755 + - template: jinja diff --git b/salt/dnsdriver/files/dnscelery.service a/salt/dnsdriver/files/dnscelery.service new file mode 100644 index 0000000..34bdfb0 --- /dev/null +++ a/salt/dnsdriver/files/dnscelery.service @@ -0,0 +1,19 @@ +[Unit] +Description=DNS driver +Wants=network.target +After=network.target + +[Service] +User={{ pillar["user"] }} +Group={{ pillar["user"] }} + +KillSignal=SIGTERM +TimeoutStopSec=600 +Restart=always + +WorkingDirectory=/home/{{ pillar["user"] }}/dnsdriver + +ExecStart=/bin/bash -c "source /home/{{ pillar["user"] }}/.virtualenvs/dnsdriver/bin/activate; source /home/{{ pillar["user"] }}/.virtualenvs/dnsdriver/bin/postactivate; exec celery -A dnscelery worker --loglevel=info -n $(/bin/hostname -s).dns" + +[Install] +WantedBy=multi-user.target diff --git b/salt/dnsdriver/files/makefile a/salt/dnsdriver/files/makefile new file mode 100644 index 0000000..c25a0f2 --- /dev/null +++ a/salt/dnsdriver/files/makefile @@ -0,0 +1,2 @@ +default: + tinydns-data diff --git b/salt/dnsdriver/files/postactivate a/salt/dnsdriver/files/postactivate new file mode 100644 index 0000000..c635c40 --- /dev/null +++ a/salt/dnsdriver/files/postactivate @@ -0,0 +1,5 @@ +export AMQP_URI=amqp://{{ pillar['amqp']['user'] }}:{{ pillar['amqp']['password'] }}@{{ pillar['amqp']['host'] }}:{{ pillar['amqp']['port'] }}/{{ pillar['amqp']['vhost'] }} +export DNS_DB_DIR={{ pillar['dnsdriver']['dns_db_dir'] }} + +# extra paramaters for dnscelery +#export EXTRA= diff --git b/salt/dnsdriver/files/tinydns.conf a/salt/dnsdriver/files/tinydns.conf new file mode 100644 index 0000000..7b42716 --- /dev/null +++ a/salt/dnsdriver/files/tinydns.conf @@ -0,0 +1,67 @@ +# +# tinydns.conf: this file is part of the djbdns project. +# +# Here we define some variables vital for running tinydns. +# +# Things to remember: +# +# - Lines starting with `#' are comments, thus ignored. +# - Blank lines are blank, thus ignored. +# - Do not leave blank spaces around `=' sign while defining a variable. +# + +# Maximum number of bytes that could be allocated if required. +# +DATALIMIT=300000 + +# No of bytes to allocate for the cache. This may not exceed DATALIMIT +# +# CACHESIZE=100000 + +# Address to listen on for incoming connections. It could be comma separated +# list of IP addresses. +# +# IP=127.0.0.1[,x.x.x.x,...] +# +IP={{ pillar['dnsdriver']['address_list'] }} + +# Address to use while sending out-going requests. 0.0.0.0 means machines +# primary IP address. +# +# IPSEND=0.0.0.0 + +# A non-root user whose privileges should be acquired by tinydns. +# Default: daemon +# See: $ id -u daemon +# +UID=2 + +# A non-root group whose privileges should be acquired by tinydns. +# Default: daemon +# See: $ id -g daemon +# +GID=2 + +# ROOT: is the new root & working directory for tinydns. +# Obviously, the user whose ID is mentioned above MUST be able to read from +# this directory. +# +# Also, this is where `data' and `data.cdb' files should reside. +# +ROOT={{ pillar['dnsdriver']['dns_db_dir'] }} + +# If HIDETTL is set, tinydns always uses a TTL of 0 in its responses. +# +# HIDETTL= + +# If FORWARDONLY is set, tinydns treats the servers/roots as a list of IP +# addresses for other caches, not root servers. It forwards queries to those +# caches the same way a client does, rather than contacting a chain of servers +# according to NS records. +# +# FORWARDONLY= + +# If DEBUG_LEVEL is set, tinydns displays helpful debug messages to +# the console. +# +DEBUG_LEVEL=1 diff --git b/salt/dnsdriver/gitrepo.sls a/salt/dnsdriver/gitrepo.sls new file mode 100644 index 0000000..fdad2b1 --- /dev/null +++ a/salt/dnsdriver/gitrepo.sls @@ -0,0 +1,11 @@ +include: + - common + +gitrepo_dnsdriver: + git.latest: + - name: {{ pillar['dnsdriver']['repo_name'] }} + - rev: {{ pillar['dnsdriver']['repo_revision'] }} + - target: /home/{{ pillar['user'] }}/dnsdriver + - user: {{ pillar['user'] }} + - require: + - pkg: git diff --git b/salt/dnsdriver/init.sls a/salt/dnsdriver/init.sls new file mode 100644 index 0000000..09de94d --- /dev/null +++ a/salt/dnsdriver/init.sls @@ -0,0 +1,35 @@ +include: + - dnsdriver.gitrepo + - dnsdriver.virtualenv + - dnsdriver.configuration + + +dnsdriver: + pkg.installed: + - pkgs: + - ndjbdns + - make + - python-virtualenvwrapper + - require_in: + - virtualenv: virtualenv_dnsdriver + - file: tinydns_conf + + +dnscelery: + service.running: + - enable: True + - watch: + - pkg: dnsdriver + - sls: dnsdriver.gitrepo + - sls: dnsdriver.virtualenv + - sls: dnsdriver.configuration + + +tinydns: + service.running: + - enable: True + - watch: + - pkg: dnsdriver + - sls: dnsdriver.gitrepo + - sls: dnsdriver.virtualenv + - sls: dnsdriver.configuration diff --git b/salt/dnsdriver/virtualenv.sls a/salt/dnsdriver/virtualenv.sls new file mode 100644 index 0000000..ec063dd --- /dev/null +++ a/salt/dnsdriver/virtualenv.sls @@ -0,0 +1,8 @@ +virtualenv_dnsdriver: + virtualenv.managed: + - name: /home/{{ pillar['user'] }}/.virtualenvs/dnsdriver + - requirements: /home/{{ pillar['user'] }}/dnsdriver/requirements.txt + - user: {{ pillar['user'] }} + - no_chown: true + - require: + - git: gitrepo_dnsdriver diff --git b/salt/fwdriver/configuration.sls a/salt/fwdriver/configuration.sls new file mode 100644 index 0000000..f822f00 --- /dev/null +++ a/salt/fwdriver/configuration.sls @@ -0,0 +1,108 @@ +include: + - openvswitch + +/home/{{ pillar['fwdriver']['user'] }}/.virtualenvs/fw/bin/postactivate: + file.managed: + - source: salt://fwdriver/files/postactivate + - template: jinja + - user: {{ pillar['fwdriver']['user'] }} + - group: {{ pillar['fwdriver']['user'] }} + - mode: 700 + +{% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} +/etc/systemd/system/firewall.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['fwdriver']['user'] }}/fwdriver/miscellaneous/firewall.service + +/etc/systemd/system/firewall-init.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: salt://fwdriver/files/firewall-init.service +{% else %} +/etc/init/firewall.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['fwdriver']['user'] }}/fwdriver/miscellaneous/firewall.conf + +/etc/init/firewall-init.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['fwdriver']['user'] }}/fwdriver/miscellaneous/firewall-init.conf +{% endif %} + +/etc/dhcp: + file.directory: + - mode: 755 + +/etc/dhcp/dhcpd.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: salt://fwdriver/files/dhcpd.conf + +/etc/dhcp/dhcpd.conf.generated: + file.managed: + - user: {{ pillar['fwdriver']['user'] }} + - group: {{ pillar['fwdriver']['user'] }} + +{% if grains['os_family'] != 'RedHat' and grains['os'] != 'Debian' %} +/etc/init.d/isc-dhcp-server: + file.symlink: + - target: /lib/init/upstart-job + - force: True +{% endif %} + +/etc/sysctl.d/60-circle-firewall.conf: + file.managed: + - user: root + - group: root + - contents: "net.ipv4.ip_forward=1\nnet.ipv6.conf.all.forwarding=1" + +/etc/sudoers.d/fwdriver: + file.managed: + - user: root + - group: root + - mode: 400 + - template: jinja + - source: salt://fwdriver/files/sudoers + + +{% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} +systemd-sysctl: + cmd.run: + - name: /bin/systemctl restart systemd-sysctl + service.running: + - watch: + - file: /etc/sysctl.d/60-circle-firewall.conf + - require: + - cmd: systemd-sysctl +{% endif %} + +{% if grains['os_family'] == 'RedHat' %} +/root/firewall-init.te: + file.managed: + - source: salt://fwdriver/files/firewall-init.te + - template: jinja + - mode: 644 + +firewall-init_semodule: + cmd.run: + - cwd: /root + - user: root + - name: checkmodule -M -m -o firewall-init.mod firewall-init.te; semodule_package -o firewall-init.pp -m firewall-init.mod; semodule -i firewall-init.pp + - unless: semodule -l |grep -qs ^firewall-init + - require: + - file: /root/firewall-init.te + +{% endif %} + diff --git b/salt/fwdriver/files/dhcpd.conf a/salt/fwdriver/files/dhcpd.conf new file mode 100644 index 0000000..0bfdad1 --- /dev/null +++ a/salt/fwdriver/files/dhcpd.conf @@ -0,0 +1,5 @@ +ddns-update-style none; +default-lease-time 60000; +max-lease-time 720000; +log-facility local7; +include "/etc/dhcp/dhcpd.conf.generated"; diff --git b/salt/fwdriver/files/firewall-init.service a/salt/fwdriver/files/firewall-init.service new file mode 100644 index 0000000..9650b5f --- /dev/null +++ a/salt/fwdriver/files/firewall-init.service @@ -0,0 +1,15 @@ +[Unit] +Description=CIRCLE firewall init +After=network.target +#Before=firewall.service + +[Service] +User=root +Group=root + +Type=oneshot +ExecStart=/bin/bash -c "/bin/systemctl restart openvswitch{%if grains['os']=='Debian'%}-switch{%endif%} ; /sbin/ip netns add fw || true; ovs-vsctl del-br firewall || true; /sbin/ip netns exec fw sysctl -f /etc/sysctl.d/60-circle-firewall.conf ; /sbin/ip netns exec fw ip link set lo up" + +[Install] +WantedBy=multi-user.target + diff --git b/salt/fwdriver/files/firewall-init.te a/salt/fwdriver/files/firewall-init.te new file mode 100644 index 0000000..f7ea4bd --- /dev/null +++ a/salt/fwdriver/files/firewall-init.te @@ -0,0 +1,22 @@ + +module firewall-init 1.0; + +require { + type ifconfig_t; + type ifconfig_var_run_t; + type virtio_device_t; + type root_t; + class dir mounton; + class chr_file { read write }; +} + +#============= ifconfig_t ============== + +#!!!! This avc is allowed in the current policy +allow ifconfig_t ifconfig_var_run_t:dir mounton; + +#!!!! This avc is allowed in the current policy +allow ifconfig_t root_t:dir mounton; + +#!!!! This avc is allowed in the current policy +allow ifconfig_t virtio_device_t:chr_file { read write }; diff --git b/salt/fwdriver/files/isc-dhcp-server.conf a/salt/fwdriver/files/isc-dhcp-server.conf new file mode 100644 index 0000000..cccd427 --- /dev/null +++ a/salt/fwdriver/files/isc-dhcp-server.conf @@ -0,0 +1,59 @@ +description "ISC DHCP IPv4 server" +author "Stéphane Graber <stgraber@ubuntu.com>" + +start on runlevel [2345] +stop on runlevel [!2345] + +pre-start script + if [ ! -f /etc/default/isc-dhcp-server ]; then + echo "/etc/default/isc-dhcp-server does not exist! - Aborting..." + echo "Run 'dpkg-reconfigure isc-dhcp-server' to fix the problem." + stop + exit 0 + fi + . /etc/default/isc-dhcp-server + + if [ -f /etc/ltsp/dhcpd.conf ]; then + CONFIG_FILE=/etc/ltsp/dhcpd.conf + else + CONFIG_FILE=/etc/dhcp/dhcpd.conf + fi + if [ ! -f $CONFIG_FILE ]; then + echo "$CONFIG_FILE does not exist! - Aborting..." + echo "Please create and configure $CONFIG_FILE to fix the problem." + stop + exit 0 + fi + + if ! ip netns exec fw dhcpd -user dhcpd -group dhcpd -t -q -4 -cf $CONFIG_FILE > /dev/null 2>&1; then + echo "dhcpd self-test failed. Please fix the config file." + echo "The error was: " + ip netns exec fw dhcpd -user dhcpd -group dhcpd -t -4 -cf $CONFIG_FILE + stop + exit 0 + fi +end script + +respawn +script + if [ -f /etc/ltsp/dhcpd.conf ]; then + CONFIG_FILE=/etc/ltsp/dhcpd.conf + else + CONFIG_FILE=/etc/dhcp/dhcpd.conf + fi + + . /etc/default/isc-dhcp-server + + # Allow dhcp server to write lease and pid file as 'dhcpd' user + mkdir -p /var/run/dhcp-server + chown dhcpd:dhcpd /var/run/dhcp-server + + # The leases files need to be root:root even when dropping privileges + [ -e /var/lib/dhcp/dhcpd.leases ] || touch /var/lib/dhcp/dhcpd.leases + chown root:root /var/lib/dhcp /var/lib/dhcp/dhcpd.leases + if [ -e /var/lib/dhcp/dhcpd.leases~ ]; then + chown root:root /var/lib/dhcp/dhcpd.leases~ + fi + + exec ip netns exec fw dhcpd -user dhcpd -group dhcpd -f -q -4 -pf /run/dhcp-server/dhcpd.pid -cf $CONFIG_FILE $INTERFACES +end script diff --git b/salt/fwdriver/files/postactivate a/salt/fwdriver/files/postactivate new file mode 100644 index 0000000..6317a7d --- /dev/null +++ a/salt/fwdriver/files/postactivate @@ -0,0 +1,4 @@ +export GATEWAY={{ pillar['fwdriver']['gateway'] }} +export AMQP_URI=amqp://{{ pillar['amqp']['user'] }}:{{ pillar['amqp']['password'] }}@{{ pillar['amqp']['host'] }}:{{ pillar['amqp']['port'] }}/{{ pillar['amqp']['vhost'] }} +export CACHE_URI={{ pillar['cache'] }} +export BRIDGE_TYPE=NONE diff --git b/salt/fwdriver/files/sudoers a/salt/fwdriver/files/sudoers new file mode 100644 index 0000000..63baee7 --- /dev/null +++ a/salt/fwdriver/files/sudoers @@ -0,0 +1,2 @@ +{{ pillar['fwdriver']['user'] }} ALL= (ALL) NOPASSWD: /sbin/ip netns exec fw ip addr *, /sbin/ip netns exec fw ip ro *, /sbin/ip netns exec fw ip link *, /sbin/ip netns exec fw ipset *, /usr/bin/ovs-vsctl, /sbin/ip netns exec fw iptables-restore -c, /sbin/ip netns exec fw ip6tables-restore -c, /etc/init.d/isc-dhcp-server restart, /sbin/ip link *, /sbin/iptables-restore -c, /sbin/ip6tables-restore -c, /sbin/ipset *, /bin/systemctl restart dhcpd +Defaults: fw !requiretty diff --git b/salt/fwdriver/gitrepo.sls a/salt/fwdriver/gitrepo.sls new file mode 100644 index 0000000..a298657 --- /dev/null +++ a/salt/fwdriver/gitrepo.sls @@ -0,0 +1,11 @@ +include: + - common + +gitrepo_fwdriver: + git.latest: + - name: {{ pillar['fwdriver']['repo_name'] }} + - rev: {{ pillar['fwdriver']['repo_revision'] }} + - target: /home/{{ pillar['fwdriver']['user'] }}/fwdriver + - user: {{ pillar['fwdriver']['user'] }} + - require: + - pkg: git diff --git b/salt/fwdriver/init.sls a/salt/fwdriver/init.sls new file mode 100644 index 0000000..8cc6210 --- /dev/null +++ a/salt/fwdriver/init.sls @@ -0,0 +1,51 @@ +include: + - fwdriver.gitrepo + - fwdriver.virtualenv + - fwdriver.configuration + +firewall: + pkg.installed: + - pkgs: + {% if grains['os_family'] == 'RedHat' %} + - zlib-devel + - python-virtualenvwrapper + - python-devel + - libmemcached-devel + - python2-pip + - dhcp + {% else %} + - zlib1g-dev + - virtualenvwrapper + - python-dev + - libmemcached-dev + - openvswitch-switch + - python-pip + {% if grains['os'] != 'Debian' %} + {# No such package in Debian Jessie! #} + - openvswitch-controller + {% endif %} + - isc-dhcp-server + {% endif %} + - git + - ntp + - iptables + - ipset + - require: + - user: {{ pillar['fwdriver']['user'] }} + - require_in: + - git: gitrepo_fwdriver + - virtualenv: virtualenv_fwdriver + + user: + - present + - name: {{ pillar['fwdriver']['user'] }} + - gid_from_name: True + service: + - enabled + - require: + - service: firewall-init + +firewall-init: + service: + - enabled + diff --git b/salt/fwdriver/virtualenv.sls a/salt/fwdriver/virtualenv.sls new file mode 100644 index 0000000..3c2b909 --- /dev/null +++ a/salt/fwdriver/virtualenv.sls @@ -0,0 +1,6 @@ +virtualenv_fwdriver: + virtualenv.managed: + - name: /home/{{ pillar['fwdriver']['user'] }}/.virtualenvs/fw + - requirements: /home/{{ pillar['fwdriver']['user'] }}/fwdriver/requirements.txt + - user: {{ pillar['fwdriver']['user'] }} + - no_chown: true diff --git b/salt/graphite/configuration.sls a/salt/graphite/configuration.sls new file mode 100644 index 0000000..a6cacb9 --- /dev/null +++ a/salt/graphite/configuration.sls @@ -0,0 +1,89 @@ +postactivate: + file.managed: + - name: /home/{{ pillar['graphite']['user'] }}/.virtualenvs/graphite/bin/postactivate + - source: salt://graphite/files/postactivate + - template: jinja + - user: {{ pillar['graphite']['user'] }} + - group: {{ pillar['graphite']['user'] }} + - mode: 700 + +requirements: + file.managed: + - name: /home/{{ pillar['graphite']['user'] }}/requirements.txt + - template: jinja + - source: salt://graphite/files/requirements.txt + - user: {{ pillar['graphite']['user'] }} + - group: {{ pillar['graphite']['user'] }} + - require: + - user: {{ pillar['graphite']['user'] }} + +{% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} + +/etc/systemd/system/graphite.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: salt://graphite/files/graphite.service + +/etc/systemd/system/graphite-carbon.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: salt://graphite/files/graphite-carbon.service + +{% else %} + +/etc/init/graphite.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: salt://graphite/files/graphite.conf + +/etc/init/graphite-carbon.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: salt://graphite/files/graphite-carbon.conf +{% endif %} + +/opt/graphite: + file.directory: + - makedirs: True + - user: {{ pillar['graphite']['user'] }} + - group: {{ pillar['graphite']['user'] }} + - require: + - user: {{ pillar['graphite']['user'] }} + +/opt/graphite/conf/carbon.conf: + file.managed: + - source: salt://graphite/files/carbon.conf + - user: {{ pillar['graphite']['user'] }} + - group: {{ pillar['graphite']['user'] }} + - template: jinja + - makedirs: True + - require: + - user: {{ pillar['graphite']['user'] }} + +/opt/graphite/conf/storage-schemas.conf: + file.managed: + - name: /opt/graphite/conf/storage-schemas.conf + - source: salt://graphite/files/storage-schemas.conf + - user: {{ pillar['graphite']['user'] }} + - group: {{ pillar['graphite']['user'] }} + - makedirs: True + - require: + - user: {{ pillar['graphite']['user'] }} + +/opt/graphite/webapp/graphite/local_settings.py: + file.managed: + - source: salt://graphite/files/local_settings.py + - user: {{ pillar['graphite']['user'] }} + - group: {{ pillar['graphite']['user'] }} + - template: jinja + - makedirs: True + - require: + - user: {{ pillar['graphite']['user'] }} diff --git b/salt/graphite/files/carbon.conf a/salt/graphite/files/carbon.conf new file mode 100644 index 0000000..966df09 --- /dev/null +++ a/salt/graphite/files/carbon.conf @@ -0,0 +1,374 @@ +[cache] +# Configure carbon directories. +# +# OS environment variables can be used to tell carbon where graphite is +# installed, where to read configuration from and where to write data. +# +# GRAPHITE_ROOT - Root directory of the graphite installation. +# Defaults to ../ +# GRAPHITE_CONF_DIR - Configuration directory (where this file lives). +# Defaults to $GRAPHITE_ROOT/conf/ +# GRAPHITE_STORAGE_DIR - Storage directory for whipser/rrd/log/pid files. +# Defaults to $GRAPHITE_ROOT/storage/ +# +# To change other directory paths, add settings to this file. The following +# configuration variables are available with these default values: +# +# STORAGE_DIR = $GRAPHITE_STORAGE_DIR +# LOCAL_DATA_DIR = STORAGE_DIR/whisper/ +# WHITELISTS_DIR = STORAGE_DIR/lists/ +# CONF_DIR = STORAGE_DIR/conf/ +# LOG_DIR = STORAGE_DIR/log/ +# PID_DIR = STORAGE_DIR/ +# +# For FHS style directory structures, use: +# +# STORAGE_DIR = /var/lib/carbon/ +# CONF_DIR = /etc/carbon/ +# LOG_DIR = /var/log/carbon/ +# PID_DIR = /var/run/ +# +#LOCAL_DATA_DIR = /opt/graphite/storage/whisper/ + +# Enable daily log rotation. If disabled, a kill -HUP can be used after a manual rotate +ENABLE_LOGROTATION = True + +# Specify the user to drop privileges to +# If this is blank carbon runs as the user that invokes it +# This user must have write access to the local data directory +USER = +# +# NOTE: The above settings must be set under [relay] and [aggregator] +# to take effect for those daemons as well + +# Limit the size of the cache to avoid swapping or becoming CPU bound. +# Sorts and serving cache queries gets more expensive as the cache grows. +# Use the value "inf" (infinity) for an unlimited cache size. +MAX_CACHE_SIZE = inf + +# Limits the number of whisper update_many() calls per second, which effectively +# means the number of write requests sent to the disk. This is intended to +# prevent over-utilizing the disk and thus starving the rest of the system. +# When the rate of required updates exceeds this, then carbon's caching will +# take effect and increase the overall throughput accordingly. +MAX_UPDATES_PER_SECOND = 500 + +# If defined, this changes the MAX_UPDATES_PER_SECOND in Carbon when a +# stop/shutdown is initiated. This helps when MAX_UPDATES_PER_SECOND is +# relatively low and carbon has cached a lot of updates; it enables the carbon +# daemon to shutdown more quickly. +# MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = 1000 + +# Softly limits the number of whisper files that get created each minute. +# Setting this value low (like at 50) is a good way to ensure your graphite +# system will not be adversely impacted when a bunch of new metrics are +# sent to it. The trade off is that it will take much longer for those metrics' +# database files to all get created and thus longer until the data becomes usable. +# Setting this value high (like "inf" for infinity) will cause graphite to create +# the files quickly but at the risk of slowing I/O down considerably for a while. +MAX_CREATES_PER_MINUTE = 50 + +LINE_RECEIVER_INTERFACE = 0.0.0.0 +LINE_RECEIVER_PORT = 2003 + +# Set this to True to enable the UDP listener. By default this is off +# because it is very common to run multiple carbon daemons and managing +# another (rarely used) port for every carbon instance is not fun. +ENABLE_UDP_LISTENER = False +UDP_RECEIVER_INTERFACE = 0.0.0.0 +UDP_RECEIVER_PORT = 2003 + +PICKLE_RECEIVER_INTERFACE = 0.0.0.0 +PICKLE_RECEIVER_PORT = 2004 + +# Set to false to disable logging of successful connections +LOG_LISTENER_CONNECTIONS = True + +# Per security concerns outlined in Bug #817247 the pickle receiver +# will use a more secure and slightly less efficient unpickler. +# Set this to True to revert to the old-fashioned insecure unpickler. +USE_INSECURE_UNPICKLER = False + +CACHE_QUERY_INTERFACE = 0.0.0.0 +CACHE_QUERY_PORT = 7002 + +# Set this to False to drop datapoints received after the cache +# reaches MAX_CACHE_SIZE. If this is True (the default) then sockets +# over which metrics are received will temporarily stop accepting +# data until the cache size falls below 95% MAX_CACHE_SIZE. +USE_FLOW_CONTROL = True + +# By default, carbon-cache will log every whisper update and cache hit. This can be excessive and +# degrade performance if logging on the same volume as the whisper data is stored. +LOG_UPDATES = False +LOG_CACHE_HITS = False +LOG_CACHE_QUEUE_SORTS = True + +# The thread that writes metrics to disk can use on of the following strategies +# determining the order in which metrics are removed from cache and flushed to +# disk. The default option preserves the same behavior as has been historically +# available in version 0.9.10. +# +# sorted - All metrics in the cache will be counted and an ordered list of +# them will be sorted according to the number of datapoints in the cache at the +# moment of the list's creation. Metrics will then be flushed from the cache to +# disk in that order. +# +# max - The writer thread will always pop and flush the metric from cache +# that has the most datapoints. This will give a strong flush preference to +# frequently updated metrics and will also reduce random file-io. Infrequently +# updated metrics may only ever be persisted to disk at daemon shutdown if +# there are a large number of metrics which receive very frequent updates OR if +# disk i/o is very slow. +# +# naive - Metrics will be flushed from the cache to disk in an unordered +# fashion. This strategy may be desirable in situations where the storage for +# whisper files is solid state, CPU resources are very limited or deference to +# the OS's i/o scheduler is expected to compensate for the random write +# pattern. +# +CACHE_WRITE_STRATEGY = sorted + +# On some systems it is desirable for whisper to write synchronously. +# Set this option to True if you'd like to try this. Basically it will +# shift the onus of buffering writes from the kernel into carbon's cache. +WHISPER_AUTOFLUSH = False + +# By default new Whisper files are created pre-allocated with the data region +# filled with zeros to prevent fragmentation and speed up contiguous reads and +# writes (which are common). Enabling this option will cause Whisper to create +# the file sparsely instead. Enabling this option may allow a large increase of +# MAX_CREATES_PER_MINUTE but may have longer term performance implications +# depending on the underlying storage configuration. +# WHISPER_SPARSE_CREATE = False + +# Only beneficial on linux filesystems that support the fallocate system call. +# It maintains the benefits of contiguous reads/writes, but with a potentially +# much faster creation speed, by allowing the kernel to handle the block +# allocation and zero-ing. Enabling this option may allow a large increase of +# MAX_CREATES_PER_MINUTE. If enabled on an OS or filesystem that is unsupported +# this option will gracefully fallback to standard POSIX file access methods. +WHISPER_FALLOCATE_CREATE = True + +# Enabling this option will cause Whisper to lock each Whisper file it writes +# to with an exclusive lock (LOCK_EX, see: man 2 flock). This is useful when +# multiple carbon-cache daemons are writing to the same files +# WHISPER_LOCK_WRITES = False + +# Set this to True to enable whitelisting and blacklisting of metrics in +# CONF_DIR/whitelist and CONF_DIR/blacklist. If the whitelist is missing or +# empty, all metrics will pass through +# USE_WHITELIST = False + +# By default, carbon itself will log statistics (such as a count, +# metricsReceived) with the top level prefix of 'carbon' at an interval of 60 +# seconds. Set CARBON_METRIC_INTERVAL to 0 to disable instrumentation +# CARBON_METRIC_PREFIX = carbon +# CARBON_METRIC_INTERVAL = 60 + +# Enable AMQP if you want to receve metrics using an amqp broker +# ENABLE_AMQP = False + +# Verbose means a line will be logged for every metric received +# useful for testing +# AMQP_VERBOSE = False + +# AMQP_HOST = localhost +# AMQP_PORT = 5672 +# AMQP_VHOST = / +# AMQP_USER = guest +# AMQP_PASSWORD = guest +# AMQP_EXCHANGE = graphite +# AMQP_METRIC_NAME_IN_BODY = False + +ENABLE_AMQP = True + +# Verbose means a line will be logged for every metric received +# useful for testing +AMQP_VERBOSE = False + +AMQP_HOST = {{ pillar['graphite']['host'] }} +AMQP_PORT = {{ pillar['graphite']['port'] }} +AMQP_VHOST = {{ pillar['graphite']['vhost'] }} +AMQP_USER = {{ pillar['graphite']['user'] }} +AMQP_PASSWORD = {{ pillar['graphite']['password'] }} +AMQP_EXCHANGE = {{ pillar['graphite']['queue'] }} +AMQP_METRIC_NAME_IN_BODY = True + + +# The manhole interface allows you to SSH into the carbon daemon +# and get a python interpreter. BE CAREFUL WITH THIS! If you do +# something like time.sleep() in the interpreter, the whole process +# will sleep! This is *extremely* helpful in debugging, assuming +# you are familiar with the code. If you are not, please don't +# mess with this, you are asking for trouble :) +# +# ENABLE_MANHOLE = False +# MANHOLE_INTERFACE = 127.0.0.1 +# MANHOLE_PORT = 7222 +# MANHOLE_USER = admin +# MANHOLE_PUBLIC_KEY = ssh-rsa AAAAB3NzaC1yc2EAAAABiwAaAIEAoxN0sv/e4eZCPpi3N3KYvyzRaBaMeS2RsOQ/cDuKv11dlNzVeiyc3RFmCv5Rjwn/lQ79y0zyHxw67qLyhQ/kDzINc4cY41ivuQXm2tPmgvexdrBv5nsfEpjs3gLZfJnyvlcVyWK/lId8WUvEWSWHTzsbtmXAF2raJMdgLTbQ8wE= + +# Patterns for all of the metrics this machine will store. Read more at +# http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol#Bindings +# +# Example: store all sales, linux servers, and utilization metrics +# BIND_PATTERNS = sales.#, servers.linux.#, #.utilization +# +# Example: store everything +# BIND_PATTERNS = # + +# To configure special settings for the carbon-cache instance 'b', uncomment this: +#[cache:b] +#LINE_RECEIVER_PORT = 2103 +#PICKLE_RECEIVER_PORT = 2104 +#CACHE_QUERY_PORT = 7102 +# and any other settings you want to customize, defaults are inherited +# from [carbon] section. +# You can then specify the --instance=b option to manage this instance + + + +[relay] +#LINE_RECEIVER_INTERFACE = 0.0.0.0 +#LINE_RECEIVER_PORT = 2013 +PICKLE_RECEIVER_INTERFACE = 0.0.0.0 +PICKLE_RECEIVER_PORT = 2014 + +# Set to false to disable logging of successful connections +LOG_LISTENER_CONNECTIONS = True + +# Carbon-relay has several options for metric routing controlled by RELAY_METHOD +# +# Use relay-rules.conf to route metrics to destinations based on pattern rules +#RELAY_METHOD = rules +# +# Use consistent-hashing for even distribution of metrics between destinations +#RELAY_METHOD = consistent-hashing +# +# Use consistent-hashing but take into account an aggregation-rules.conf shared +# by downstream carbon-aggregator daemons. This will ensure that all metrics +# that map to a given aggregation rule are sent to the same carbon-aggregator +# instance. +# Enable this for carbon-relays that send to a group of carbon-aggregators +#RELAY_METHOD = aggregated-consistent-hashing +RELAY_METHOD = rules + +# If you use consistent-hashing you can add redundancy by replicating every +# datapoint to more than one machine. +REPLICATION_FACTOR = 1 + +# This is a list of carbon daemons we will send any relayed or +# generated metrics to. The default provided would send to a single +# carbon-cache instance on the default port. However if you +# use multiple carbon-cache instances then it would look like this: +# +# DESTINATIONS = 127.0.0.1:2004:a, 127.0.0.1:2104:b +# +# The general form is IP:PORT:INSTANCE where the :INSTANCE part is +# optional and refers to the "None" instance if omitted. +# +# Note that if the destinations are all carbon-caches then this should +# exactly match the webapp's CARBONLINK_HOSTS setting in terms of +# instances listed (order matters!). +# +# If using RELAY_METHOD = rules, all destinations used in relay-rules.conf +# must be defined in this list +DESTINATIONS = 127.0.0.1:2004 + +# This defines the maximum "message size" between carbon daemons. +# You shouldn't need to tune this unless you really know what you're doing. +MAX_DATAPOINTS_PER_MESSAGE = 500 +MAX_QUEUE_SIZE = 10000 + +# Set this to False to drop datapoints when any send queue (sending datapoints +# to a downstream carbon daemon) hits MAX_QUEUE_SIZE. If this is True (the +# default) then sockets over which metrics are received will temporarily stop accepting +# data until the send queues fall below 80% MAX_QUEUE_SIZE. +USE_FLOW_CONTROL = True + +# Set this to True to enable whitelisting and blacklisting of metrics in +# CONF_DIR/whitelist and CONF_DIR/blacklist. If the whitelist is missing or +# empty, all metrics will pass through +# USE_WHITELIST = False + +# By default, carbon itself will log statistics (such as a count, +# metricsReceived) with the top level prefix of 'carbon' at an interval of 60 +# seconds. Set CARBON_METRIC_INTERVAL to 0 to disable instrumentation +# CARBON_METRIC_PREFIX = carbon +# CARBON_METRIC_INTERVAL = 60 + + +[aggregator] +LINE_RECEIVER_INTERFACE = 0.0.0.0 +LINE_RECEIVER_PORT = 2023 + +PICKLE_RECEIVER_INTERFACE = 0.0.0.0 +PICKLE_RECEIVER_PORT = 2024 + +# Set to false to disable logging of successful connections +LOG_LISTENER_CONNECTIONS = True + +# If set true, metric received will be forwarded to DESTINATIONS in addition to +# the output of the aggregation rules. If set false the carbon-aggregator will +# only ever send the output of aggregation. +FORWARD_ALL = True + +# This is a list of carbon daemons we will send any relayed or +# generated metrics to. The default provided would send to a single +# carbon-cache instance on the default port. However if you +# use multiple carbon-cache instances then it would look like this: +# +# DESTINATIONS = 127.0.0.1:2004:a, 127.0.0.1:2104:b +# +# The format is comma-delimited IP:PORT:INSTANCE where the :INSTANCE part is +# optional and refers to the "None" instance if omitted. +# +# Note that if the destinations are all carbon-caches then this should +# exactly match the webapp's CARBONLINK_HOSTS setting in terms of +# instances listed (order matters!). +DESTINATIONS = 127.0.0.1:2004 + +# If you want to add redundancy to your data by replicating every +# datapoint to more than one machine, increase this. +REPLICATION_FACTOR = 1 + +# This is the maximum number of datapoints that can be queued up +# for a single destination. Once this limit is hit, we will +# stop accepting new data if USE_FLOW_CONTROL is True, otherwise +# we will drop any subsequently received datapoints. +MAX_QUEUE_SIZE = 10000 + +# Set this to False to drop datapoints when any send queue (sending datapoints +# to a downstream carbon daemon) hits MAX_QUEUE_SIZE. If this is True (the +# default) then sockets over which metrics are received will temporarily stop accepting +# data until the send queues fall below 80% MAX_QUEUE_SIZE. +USE_FLOW_CONTROL = True + +# This defines the maximum "message size" between carbon daemons. +# You shouldn't need to tune this unless you really know what you're doing. +MAX_DATAPOINTS_PER_MESSAGE = 500 + +# This defines how many datapoints the aggregator remembers for +# each metric. Aggregation only happens for datapoints that fall in +# the past MAX_AGGREGATION_INTERVALS * intervalSize seconds. +MAX_AGGREGATION_INTERVALS = 5 + +# By default (WRITE_BACK_FREQUENCY = 0), carbon-aggregator will write back +# aggregated data points once every rule.frequency seconds, on a per-rule basis. +# Set this (WRITE_BACK_FREQUENCY = N) to write back all aggregated data points +# every N seconds, independent of rule frequency. This is useful, for example, +# to be able to query partially aggregated metrics from carbon-cache without +# having to first wait rule.frequency seconds. +# WRITE_BACK_FREQUENCY = 0 + +# Set this to True to enable whitelisting and blacklisting of metrics in +# CONF_DIR/whitelist and CONF_DIR/blacklist. If the whitelist is missing or +# empty, all metrics will pass through +# USE_WHITELIST = False + +# By default, carbon itself will log statistics (such as a count, +# metricsReceived) with the top level prefix of 'carbon' at an interval of 60 +# seconds. Set CARBON_METRIC_INTERVAL to 0 to disable instrumentation +# CARBON_METRIC_PREFIX = carbon +# CARBON_METRIC_INTERVAL = 60 diff --git b/salt/graphite/files/graphite-carbon.conf a/salt/graphite/files/graphite-carbon.conf new file mode 100644 index 0000000..d072843 --- /dev/null +++ a/salt/graphite/files/graphite-carbon.conf @@ -0,0 +1,20 @@ +description "CIRCLE Cloud Graphite monitoring server" + +start on runlevel [2345] +stop on runlevel [!2345] + +respawn +respawn limit 30 30 +setgid {{ pillar['graphite']['user'] }} +setuid {{ pillar['graphite']['user'] }} + +env HOME=/home/{{ pillar['graphite']['user'] }} +env GRAPHITE_ROOT=/opt/graphite +env PYTHONPATH=/opt/graphite/lib + + +script + . $HOME/.virtualenvs/graphite/local/bin/activate + cd /opt/graphite/bin/ + exec twistd --nodaemon --reactor=epoll --no_save carbon-cache +end script diff --git b/salt/graphite/files/graphite-carbon.service a/salt/graphite/files/graphite-carbon.service new file mode 100644 index 0000000..a246297 --- /dev/null +++ a/salt/graphite/files/graphite-carbon.service @@ -0,0 +1,14 @@ +[Unit] +Description=Graphite Carbon +After=network.target + +[Service] +User={{ pillar['graphite']['user'] }} +Group={{ pillar['graphite']['user'] }} +Environment=PYTHONPATH=/opt/graphite/lib GRAPHITE_ROOT=/opt/graphite +WorkingDirectory=/opt/graphite/bin/ +ExecStart=/bin/bash -c "source /etc/profile; workon graphite; exec twistd --nodaemon --reactor=epoll --no_save carbon-cache" +Restart=always + +[Install] +WantedBy=multi-user.target diff --git b/salt/graphite/files/graphite-unicode-fix.diff a/salt/graphite/files/graphite-unicode-fix.diff new file mode 100644 index 0000000..5094c18 --- /dev/null +++ a/salt/graphite/files/graphite-unicode-fix.diff @@ -0,0 +1,71 @@ +diff --git a/render/evaluator.py b/render/evaluator.py +index 70490a2..ee7cfd1 100644 +--- a/render/evaluator.py ++++ b/render/evaluator.py +@@ -37,7 +37,7 @@ def evaluateTokens(requestContext, tokens): + return float(tokens.number.scientific[0]) + + elif tokens.string: +- return str(tokens.string)[1:-1] ++ return unicode(tokens.string)[1:-1] + + elif tokens.boolean: + return tokens.boolean[0] == 'true' +diff --git a/render/glyph.py b/render/glyph.py +index a2cc893..7daadce 100644 +--- a/render/glyph.py ++++ b/render/glyph.py +@@ -181,7 +181,7 @@ class Graph: + self.drawRectangle( 0, 0, self.width, self.height ) + + if 'colorList' in params: +- colorList = unquote_plus( str(params['colorList']) ).split(',') ++ colorList = unquote_plus( unicode(params['colorList']) ).split(',') + else: + colorList = self.defaultColorList + self.colors = itertools.cycle( colorList ) +@@ -572,7 +572,7 @@ class LineGraph(Graph): + if 'yUnitSystem' not in params: + params['yUnitSystem'] = 'si' + else: +- params['yUnitSystem'] = str(params['yUnitSystem']).lower() ++ params['yUnitSystem'] = unicode(params['yUnitSystem']).lower() + if params['yUnitSystem'] not in UnitSystems.keys(): + params['yUnitSystem'] = 'si' + +@@ -630,11 +630,11 @@ class LineGraph(Graph): + self.setColor( self.foregroundColor ) + + if params.get('title'): +- self.drawTitle( str(params['title']) ) ++ self.drawTitle( unicode(params['title']) ) + if params.get('vtitle'): +- self.drawVTitle( str(params['vtitle']) ) ++ self.drawVTitle( unicode(params['vtitle']) ) + if self.secondYAxis and params.get('vtitleRight'): +- self.drawVTitle( str(params['vtitleRight']), rightAlign=True ) ++ self.drawVTitle( unicode(params['vtitleRight']), rightAlign=True ) + self.setFont() + + if not params.get('hideLegend', len(self.data) > settings.LEGEND_MAX_ITEMS): +@@ -1582,7 +1582,7 @@ class PieGraph(Graph): + if slice['value'] < 10 and slice['value'] != int(slice['value']): + label = "%.2f" % slice['value'] + else: +- label = str(int(slice['value'])) ++ label = unicode(int(slice['value'])) + extents = self.getExtents(label) + theta = slice['midAngle'] + x = self.x0 + (self.radius / 2.0 * math.cos(theta)) +diff --git a/render/hashing.py b/render/hashing.py +index 6575650..45f1bfe 100644 +--- a/render/hashing.py ++++ b/render/hashing.py +@@ -49,7 +49,7 @@ def stripControlChars(string): + + def compactHash(string): + hash = md5() +- hash.update(string) ++ hash.update(string.encode('utf-8')) + return hash.hexdigest() + diff --git b/salt/graphite/files/graphite.conf a/salt/graphite/files/graphite.conf new file mode 100644 index 0000000..9ad698b --- /dev/null +++ a/salt/graphite/files/graphite.conf @@ -0,0 +1,17 @@ +description "CIRCLE Cloud Graphite monitoring server" + +start on runlevel [2345] +stop on runlevel [!2345] + +respawn +respawn limit 30 30 +setgid {{ pillar['graphite']['user'] }} +setuid {{ pillar['graphite']['user'] }} + +env HOME=/home/{{ pillar['graphite']['user'] }} + +script + . $HOME/.virtualenvs/graphite/local/bin/activate + cd /opt/graphite/webapp/graphite + PYTHONPATH=/opt/graphite/webapp exec django-admin.py runserver [::]:8081 --settings=graphite.settings +end script diff --git b/salt/graphite/files/graphite.service a/salt/graphite/files/graphite.service new file mode 100644 index 0000000..24c78b8 --- /dev/null +++ a/salt/graphite/files/graphite.service @@ -0,0 +1,13 @@ +[Unit] +Description=Graphite +After=network.target + +[Service] +User={{ pillar['graphite']['user'] }} +Group={{ pillar['graphite']['user'] }} +WorkingDirectory=/opt/graphite/webapp/graphite +ExecStart=/bin/bash -c "source /etc/profile; workon graphite; PYTHONPATH=/opt/graphite/webapp exec django-admin.py runserver [::]:8081 --settings=graphite.settings" +Restart=always + +[Install] +WantedBy=multi-user.target diff --git b/salt/graphite/files/local_settings.py a/salt/graphite/files/local_settings.py new file mode 100644 index 0000000..4dafc80 --- /dev/null +++ a/salt/graphite/files/local_settings.py @@ -0,0 +1,204 @@ +## Graphite local_settings.py +# Edit this file to customize the default Graphite webapp settings +# +# Additional customizations to Django settings can be added to this file as well + +##################################### +# General Configuration # +##################################### +# Set this to a long, random unique string to use as a secret key for this +# install. This key is used for salting of hashes used in auth tokens, +# CRSF middleware, cookie storage, etc. This should be set identically among +# instances if used behind a load balancer. +SECRET_KEY = "{{ pillar['graphite']['secret_key'] }}" + +# In Django 1.5+ set this to the list of hosts your graphite instances is +# accessible as. See: +# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-ALLOWED_HOSTS +#ALLOWED_HOSTS = [ '*' ] + +# Set your local timezone (Django's default is America/Chicago) +# If your graphs appear to be offset by a couple hours then this probably +# needs to be explicitly set to your local timezone. +TIME_ZONE = "{{ pillar['timezone'] }}" + +# Override this to provide documentation specific to your Graphite deployment +#DOCUMENTATION_URL = "http://graphite.readthedocs.org/" + +# Logging +#LOG_RENDERING_PERFORMANCE = True +#LOG_CACHE_PERFORMANCE = True +#LOG_METRIC_ACCESS = True + +# Enable full debug page display on exceptions (Internal Server Error pages) +#DEBUG = True + +# If using RRD files and rrdcached, set to the address or socket of the daemon +#FLUSHRRDCACHED = 'unix:/var/run/rrdcached.sock' + +# This lists the memcached servers that will be used by this webapp. +# If you have a cluster of webapps you should ensure all of them +# have the *exact* same value for this setting. That will maximize cache +# efficiency. Setting MEMCACHE_HOSTS to be empty will turn off use of +# memcached entirely. +# +# You should not use the loopback address (127.0.0.1) here if using clustering +# as every webapp in the cluster should use the exact same values to prevent +# unneeded cache misses. Set to [] to disable caching of images and fetched data +#MEMCACHE_HOSTS = ['10.10.10.10:11211', '10.10.10.11:11211', '10.10.10.12:11211'] +#DEFAULT_CACHE_DURATION = 60 # Cache images and data for 1 minute + + +##################################### +# Filesystem Paths # +##################################### +# Change only GRAPHITE_ROOT if your install is merely shifted from /opt/graphite +# to somewhere else +#GRAPHITE_ROOT = '/opt/graphite' + +# Most installs done outside of a separate tree such as /opt/graphite will only +# need to change these three settings. Note that the default settings for each +# of these is relative to GRAPHITE_ROOT +#CONF_DIR = '/opt/graphite/conf' +#STORAGE_DIR = '/opt/graphite/storage' +#CONTENT_DIR = '/opt/graphite/webapp/content' + +# To further or fully customize the paths, modify the following. Note that the +# default settings for each of these are relative to CONF_DIR and STORAGE_DIR +# +## Webapp config files +#DASHBOARD_CONF = '/opt/graphite/conf/dashboard.conf' +#GRAPHTEMPLATES_CONF = '/opt/graphite/conf/graphTemplates.conf' + +## Data directories +# NOTE: If any directory is unreadable in DATA_DIRS it will break metric browsing +#WHISPER_DIR = '/opt/graphite/storage/whisper' +#RRD_DIR = '/opt/graphite/storage/rrd' +#DATA_DIRS = [WHISPER_DIR, RRD_DIR] # Default: set from the above variables +#LOG_DIR = '/opt/graphite/storage/log/webapp' +#INDEX_FILE = '/opt/graphite/storage/index' # Search index file + + +##################################### +# Email Configuration # +##################################### +# This is used for emailing rendered Graphs +# Default backend is SMTP +#EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' +#EMAIL_HOST = 'localhost' +#EMAIL_PORT = 25 +#EMAIL_HOST_USER = '' +#EMAIL_HOST_PASSWORD = '' +#EMAIL_USE_TLS = False +# To drop emails on the floor, enable the Dummy backend: +#EMAIL_BACKEND = 'django.core.mail.backends.dummy.EmailBackend' + + +##################################### +# Authentication Configuration # +##################################### +## LDAP / ActiveDirectory authentication setup +#USE_LDAP_AUTH = True +#LDAP_SERVER = "ldap.mycompany.com" +#LDAP_PORT = 389 +# OR +#LDAP_URI = "ldaps://ldap.mycompany.com:636" +#LDAP_SEARCH_BASE = "OU=users,DC=mycompany,DC=com" +#LDAP_BASE_USER = "CN=some_readonly_account,DC=mycompany,DC=com" +#LDAP_BASE_PASS = "readonly_account_password" +#LDAP_USER_QUERY = "(username=%s)" #For Active Directory use "(sAMAccountName=%s)" +# +# If you want to further customize the ldap connection options you should +# directly use ldap.set_option to set the ldap module's global options. +# For example: +# +#import ldap +#ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_ALLOW) +#ldap.set_option(ldap.OPT_X_TLS_CACERTDIR, "/etc/ssl/ca") +#ldap.set_option(ldap.OPT_X_TLS_CERTFILE, "/etc/ssl/mycert.pem") +#ldap.set_option(ldap.OPT_X_TLS_KEYFILE, "/etc/ssl/mykey.pem") +# See http://www.python-ldap.org/ for further details on these options. + +## REMOTE_USER authentication. See: https://docs.djangoproject.com/en/dev/howto/auth-remote-user/ +#USE_REMOTE_USER_AUTHENTICATION = True + +# Override the URL for the login link (e.g. for django_openid_auth) +#LOGIN_URL = '/account/login' + + +########################## +# Database Configuration # +########################## +# By default sqlite is used. If you cluster multiple webapps you will need +# to setup an external database (such as MySQL) and configure all of the webapp +# instances to use the same database. Note that this database is only used to store +# Django models such as saved graphs, dashboards, user preferences, etc. +# Metric data is not stored here. +# +# DO NOT FORGET TO RUN 'manage.py syncdb' AFTER SETTING UP A NEW DATABASE +# +# The following built-in database engines are available: +# django.db.backends.postgresql # Removed in Django 1.4 +# django.db.backends.postgresql_psycopg2 +# django.db.backends.mysql +# django.db.backends.sqlite3 +# django.db.backends.oracle +# +# The default is 'django.db.backends.sqlite3' with file 'graphite.db' +# located in STORAGE_DIR +# +#DATABASES = { +# 'default': { +# 'NAME': '/opt/graphite/storage/graphite.db', +# 'ENGINE': 'django.db.backends.sqlite3', +# 'USER': '', +# 'PASSWORD': '', +# 'HOST': '', +# 'PORT': '' +# } +#} +# + + +######################### +# Cluster Configuration # +######################### +# (To avoid excessive DNS lookups you want to stick to using IP addresses only in this entire section) +# +# This should list the IP address (and optionally port) of the webapp on each +# remote server in the cluster. These servers must each have local access to +# metric data. Note that the first server to return a match for a query will be +# used. +#CLUSTER_SERVERS = ["10.0.2.2:80", "10.0.2.3:80"] + +## These are timeout values (in seconds) for requests to remote webapps +#REMOTE_STORE_FETCH_TIMEOUT = 6 # Timeout to fetch series data +#REMOTE_STORE_FIND_TIMEOUT = 2.5 # Timeout for metric find requests +#REMOTE_STORE_RETRY_DELAY = 60 # Time before retrying a failed remote webapp +#REMOTE_FIND_CACHE_DURATION = 300 # Time to cache remote metric find results + +## Remote rendering settings +# Set to True to enable rendering of Graphs on a remote webapp +#REMOTE_RENDERING = True +# List of IP (and optionally port) of the webapp on each remote server that +# will be used for rendering. Note that each rendering host should have local +# access to metric data or should have CLUSTER_SERVERS configured +#RENDERING_HOSTS = [] +#REMOTE_RENDER_CONNECT_TIMEOUT = 1.0 + +# If you are running multiple carbon-caches on this machine (typically behind a relay using +# consistent hashing), you'll need to list the ip address, cache query port, and instance name of each carbon-cache +# instance on the local machine (NOT every carbon-cache in the entire cluster). The default cache query port is 7002 +# and a common scheme is to use 7102 for instance b, 7202 for instance c, etc. +# +# You *should* use 127.0.0.1 here in most cases +#CARBONLINK_HOSTS = ["127.0.0.1:7002:a", "127.0.0.1:7102:b", "127.0.0.1:7202:c"] +#CARBONLINK_TIMEOUT = 1.0 + +##################################### +# Additional Django Settings # +##################################### +# Uncomment the following line for direct access to Django settings such as +# MIDDLEWARE_CLASSES or APPS +#from graphite.app_settings import * + diff --git b/salt/graphite/files/postactivate a/salt/graphite/files/postactivate new file mode 100644 index 0000000..7bd0330 --- /dev/null +++ a/salt/graphite/files/postactivate @@ -0,0 +1,3 @@ +export AMQP_URI=amqp://{{ pillar['amqp']['user'] }}:{{ pillar['amqp']['password'] }}@{{ pillar['amqp']['host'] }}:{{ pillar['amqp']['port'] }}/{{ pillar['amqp']['vhost'] }} +export CACHE_URI={{ pillar['cache'] }} + diff --git b/salt/graphite/files/requirements.txt a/salt/graphite/files/requirements.txt new file mode 100644 index 0000000..da1099e --- /dev/null +++ a/salt/graphite/files/requirements.txt @@ -0,0 +1,14 @@ +Django==1.7 +Twisted<12.0 +python-memcached +txAMQP +simplejson +django-tagging +gunicorn +pytz +pyparsing +whisper +scandir +carbon==1.0.2 +cairocffi==0.9.0 +graphite-web==1.0.2 diff --git b/salt/graphite/files/storage-schemas.conf a/salt/graphite/files/storage-schemas.conf new file mode 100644 index 0000000..9640b2f --- /dev/null +++ a/salt/graphite/files/storage-schemas.conf @@ -0,0 +1,7 @@ +[carbon] +pattern = ^carbon\. +retentions = 60:90d + +[default] +pattern = .* +retentions = 60s:1d,240s:1w,1h:30d,6h:1y diff --git b/salt/graphite/files/syncdb.sh a/salt/graphite/files/syncdb.sh new file mode 100644 index 0000000..9fc426b --- /dev/null +++ a/salt/graphite/files/syncdb.sh @@ -0,0 +1,4 @@ +#!/bin/bash +source /home/{{ pillar['graphite']['user'] }}/.virtualenvs/graphite/bin/activate; +cd /opt/graphite/webapp/graphite/ +PYTHONPATH=/opt/graphite/webapp django-admin.py syncdb --settings=graphite.settings --noinput diff --git b/salt/graphite/init.sls a/salt/graphite/init.sls new file mode 100644 index 0000000..b56b163 --- /dev/null +++ a/salt/graphite/init.sls @@ -0,0 +1,41 @@ +include: + - graphite.rabbitmq + - graphite.virtualenv + - graphite.configuration + +graphite: + pkg.installed: + - pkgs: + - git + - ntp + {% if grains['os_family'] == 'RedHat' %} + - python2-pip + - pycairo + - python-devel + - python-virtualenvwrapper + - dejavu-sans-fonts + {% else %} + - python-pip + - python-cairo + - python-dev + - virtualenvwrapper + {% endif %} + - require: + - user: {{ pillar['graphite']['user'] }} + - require_in: + - virtualenv: virtualenv_graphite + - service: graphite + - service: graphite-carbon + user: + - present + - name: {{ pillar['graphite']['user'] }} + - gid_from_name: True + + service: + - running + - enable: True + +graphite-carbon: + service: + - running + - enable: True diff --git b/salt/graphite/rabbitmq.sls a/salt/graphite/rabbitmq.sls new file mode 100644 index 0000000..197af3f --- /dev/null +++ a/salt/graphite/rabbitmq.sls @@ -0,0 +1,21 @@ +rabbitmq-server_monitor: + pkg.installed: + - name: rabbitmq-server + service: + - running + - name: rabbitmq-server + - require: + - pkg: rabbitmq-server + +rabbitmq_user_monitor: + rabbitmq_user.present: + - name: {{ pillar['graphite']['user'] }} + - password: {{ pillar['graphite']['password'] }} + +virtual_host_monitor: + rabbitmq_vhost.present: + - name: {{ pillar['graphite']['vhost']}} + - user: {{ pillar['graphite']['user'] }} + - conf: .* + - write: .* + - read: .* diff --git b/salt/graphite/virtualenv.sls a/salt/graphite/virtualenv.sls new file mode 100644 index 0000000..4795f16 --- /dev/null +++ a/salt/graphite/virtualenv.sls @@ -0,0 +1,42 @@ +virtualenv_graphite: + virtualenv.managed: + - name: /home/{{ pillar['graphite']['user'] }}/.virtualenvs/graphite + - requirements: /home/{{ pillar['graphite']['user'] }}/requirements.txt + - user: {{ pillar['graphite']['user'] }} + - require: + - user: {{ pillar['graphite']['user'] }} + - file: /home/{{ pillar['graphite']['user'] }}/requirements.txt + - file: /opt/graphite + +global-site-packages: + file.absent: + - name: /home/{{pillar['graphite']['user'] }}/.virtualenvs/graphite/lib/python2.7/no-global-site-packages.txt + - require: + - virtualenv: virtualenv_graphite + +unicode-fix-diff: + file.managed: + - name: /home/{{pillar['graphite']['user'] }}/graphite-unicode-fix.diff + - template: jinja + - source: salt://graphite/files/graphite-unicode-fix.diff + - user: {{ pillar['graphite']['user'] }} + - group: {{ pillar['graphite']['user'] }} + +unicode-fix: + cmd.run: + - user: {{ pillar['graphite']['user'] }} + - cwd: /opt/graphite/webapp/graphite + - name: patch -N -p1 < /home/{{pillar['graphite']['user'] }}/graphite-unicode-fix.diff + - onlyif: patch -N --dry-run --silent -p1 < /home/{{pillar['graphite']['user'] }}/graphite-unicode-fix.diff + - require: + - virtualenv: virtualenv_graphite + - user: {{ pillar['graphite']['user'] }} + - file: unicode-fix-diff + +salt://graphite/files/syncdb.sh: + cmd.script: + - template: jinja + - user: {{ pillar['graphite']['user'] }} + - require: + - virtualenv: virtualenv_graphite + - user: {{ pillar['graphite']['user'] }} diff --git b/salt/manager/agentgit.sls a/salt/manager/agentgit.sls new file mode 100644 index 0000000..68e923d --- /dev/null +++ a/salt/manager/agentgit.sls @@ -0,0 +1,11 @@ +include: + - common + +agentgit: + git.latest: + - name: {{ pillar['agent']['repo_name'] }} + - rev: {{ pillar['agent']['repo_revision'] }} + - target: /home/{{ pillar['user'] }}/agent/agent-linux + - user: {{ pillar['user'] }} + - require: + - pkg: git diff --git b/salt/manager/configuration.sls a/salt/manager/configuration.sls new file mode 100644 index 0000000..1d00b76 --- /dev/null +++ a/salt/manager/configuration.sls @@ -0,0 +1,91 @@ +manager_postactivate: + file.managed: + - name: /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/postactivate + - source: salt://manager/files/postactivate + - template: jinja + - user: {{ pillar['user'] }} + - mode: 700 + +portal.conf: + file.managed: + {% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} + - name: /etc/systemd/system/portal.service + {% else %} + - name: /etc/init/portal.conf + {% endif %} + - user: root + - group: root + - template: jinja + {% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} + + {% if pillar['deployment_type'] == 'production' %} + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/portal-uwsgi.service + {% else %} + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/portal.service + {% endif %} + + {% else %} + + {% if pillar['deployment_type'] == 'production' %} + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/portal-uwsgi.conf + {% else %} + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/portal.conf + {% endif %} + + {% endif %} + +{% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} +/etc/systemd/system/manager.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/manager.service + +/etc/systemd/system/managercelery@.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/managercelery@.service + +{% else %} + +/etc/init/manager.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/manager.conf + +/etc/init/mancelery.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/mancelery.conf + +/etc/init/moncelery.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/moncelery.conf + +/etc/init/slowcelery.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/circle/miscellaneous/slowcelery.conf +{% endif %} + +salt://manager/files/init.sh: + cmd.script: + - template: jinja + - user: {{ pillar['user'] }} + - stateful: true + - require: + - virtualenv: virtualenv_manager + - file: /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/postactivate + - user: {{ pillar['user'] }} diff --git b/salt/manager/files/compile.sh a/salt/manager/files/compile.sh new file mode 100644 index 0000000..96129d1 --- /dev/null +++ a/salt/manager/files/compile.sh @@ -0,0 +1,25 @@ +#!/bin/bash +cd /home/{{ pillar['user'] }}/circle/circle/ +source /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/activate +source /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/postactivate +MANAGE="python /home/{{ pillar['user'] }}/circle/circle/manage.py" +bower install + +$MANAGE compileless +$MANAGE compilejsi18n -o dashboard/static/jsi18n + +COLLECTED=$($MANAGE collectstatic --noinput | + awk '/static files copied to/ {print $1}') + +OLD_SHA=$(sha1sum locale/hu/LC_MESSAGES/*.mo) +$MANAGE compilemessages +NEW_SHA=$(sha1sum locale/hu/LC_MESSAGES/*.mo) + +echo "$COLLECTED $NEW_SHA $OLD_SHA" +if [ "$NEW_SHA" != "$OLD_SHA" -o "$COLLECTED" -ne 0 ]; then + CHANGED=yes +else + CHANGED=no +fi + +echo "changed=$CHANGED comment='copied: $COLLECTED'" diff --git b/salt/manager/files/init.sh a/salt/manager/files/init.sh new file mode 100644 index 0000000..aaa92a3 --- /dev/null +++ a/salt/manager/files/init.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +source /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/activate +source /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/postactivate +{% set fw = pillar['fwdriver'] %} + +HOSTNAME=$(hostname -s) + +EXTRAPARAMS="" +if [ "{{ pillar['vmdriver']['hypervisor_type'] }}" = "kvm" ]; then + EXTRAPARAMS="--kvm-present" +fi + +exec python /home/{{ pillar['user'] }}/circle/circle/manage.py init \ + --external-net={{ fw['external_net'] }} \ + --management-net={{ fw['management_net'] }} \ + --vm-net={{ fw['vm_net'] }} \ + --admin-user={{ pillar['admin_user'] }} \ + --admin-pass={{ pillar['admin_pass'] }} \ + --datastore-queue={{ pillar['storagedriver']['queue_name'] }} \ + --firewall-queue={{ fw['queue_name'] }} \ + --external-if={{ fw['external_if'] }} \ + --management-if={{ fw['management_if'] }} \ + --vm-if={{ fw['vm_if'] }} \ + --node-hostname=$HOSTNAME \ + --node-mac="99:AA:BB:CC:DD:EE" \ + --node-ip="127.0.0.1" \ + --node-name=$HOSTNAME \ + $EXTRAPARAMS diff --git b/salt/manager/files/nginx-default-site.conf a/salt/manager/files/nginx-default-site.conf new file mode 100644 index 0000000..edf924e --- /dev/null +++ a/salt/manager/files/nginx-default-site.conf @@ -0,0 +1,48 @@ +ignore_invalid_headers on; + +server { + listen 443 ssl default; + ssl on; + ssl_certificate /etc/ssl/certs/circle.pem; + ssl_certificate_key /etc/ssl/certs/circle.pem; + +{% if pillar['deployment_type'] == "production" %} + location /media { + alias /home/{{ pillar['user'] }}/circle/circle/media; # your Django project's media files + } + location /static { + alias /home/{{ pillar['user'] }}/circle/circle/static_collected; # your Django project's static files + } +{% endif %} + + location / { +{% if pillar['deployment_type'] == "production" %} + uwsgi_pass unix:///tmp/uwsgi.sock; + include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually +{% else %} + proxy_pass http://localhost:8080; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header Host $http_Host; + proxy_set_header X-Forwarded-Protocol https; +{% endif %} + } + + location /vnc/ { + proxy_pass http://localhost:9999; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + + # WebSocket support (nginx 1.4) + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + } +} + + +server { + listen 80 default; + rewrite ^ https://$host/; # permanent; +} + diff --git b/salt/manager/files/nginx.conf a/salt/manager/files/nginx.conf new file mode 100644 index 0000000..b926731 --- /dev/null +++ a/salt/manager/files/nginx.conf @@ -0,0 +1,44 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ +# * Official Russian Documentation: http://nginx.org/ru/docs/ + +user nginx; +worker_processes 1; + +error_log /var/log/nginx/error.log; +#error_log /var/log/nginx/error.log notice; +#error_log /var/log/nginx/error.log info; + +pid /run/nginx.pid; + + +events { + worker_connections 1024; +} + + +http { + include /etc/nginx/mime.types; + default_type application/octet-stream; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + access_log /var/log/nginx/access.log main; + + sendfile on; + #tcp_nopush on; + + #keepalive_timeout 0; + keepalive_timeout 65; + + #gzip on; + + index index.html index.htm; + + # Load modular configuration files from the /etc/nginx/conf.d directory. + # See http://nginx.org/en/docs/ngx_core_module.html#include + # for more information. + include /etc/nginx/conf.d/*.conf; +} diff --git b/salt/manager/files/nginx.te a/salt/manager/files/nginx.te new file mode 100644 index 0000000..345f666 --- /dev/null +++ a/salt/manager/files/nginx.te @@ -0,0 +1,16 @@ + +module nginx 1.0; + +require { + type initrc_tmp_t; + type httpd_t; + type initrc_t; + class sock_file write; + class unix_stream_socket connectto; +} + +#============= httpd_t ============== +allow httpd_t initrc_t:unix_stream_socket connectto; + +#!!!! This avc is allowed in the current policy +allow httpd_t initrc_tmp_t:sock_file write; diff --git b/salt/manager/files/pg_hba.conf a/salt/manager/files/pg_hba.conf new file mode 100644 index 0000000..a183ef7 --- /dev/null +++ a/salt/manager/files/pg_hba.conf @@ -0,0 +1,9 @@ +# TYPE DATABASE USER ADDRESS METHOD + +# "local" is for Unix domain socket connections only +local all all peer +# IPv4 local connections: +host all all 127.0.0.1/32 md5 +# IPv6 local connections: +host all all ::1/128 md5 + diff --git b/salt/manager/files/postactivate a/salt/manager/files/postactivate new file mode 100644 index 0000000..20ae4f3 --- /dev/null +++ a/salt/manager/files/postactivate @@ -0,0 +1,36 @@ +# DO NOT EDIT THIS FILE +export AMQP_URI='amqp://{{ pillar['amqp']['user'] }}:{{ pillar['amqp']['password'] }}@{{ pillar['amqp']['host'] }}:{{ pillar['amqp']['port'] }}/{{ pillar['amqp']['vhost'] }}' +export CACHE_URI='{{ pillar['cache'] }}' + +export DJANGO_SETTINGS_MODULE='circle.settings.{{ pillar['deployment_type'] }}' +export DJANGO_TIME_ZONE=UTC +export DJANGO_DB_HOST='localhost' +export DJANGO_DB_PASSWORD='{{ pillar['database']['password'] }}' +export DJANGO_FIREWALL_SETTINGS='{"dns_ip": "8.8.8.8", "dns_hostname": + "localhost", "dns_ttl": "300", "reload_sleep": "10", + "rdns_ip": "8.8.8.8", "default_vlangroup": "portforward"}' +export DJANGO_ALLOWED_HOSTS='*' +export DJANGO_MEMCACHED='localhost:11211' + +#export DJANGO_SAML=TRUE +#export DJANGO_URL='<%= @django_url %>' +#export DJANGO_SAML_ATTRIBUTE_MAPPING='{"mail": ["email"], "sn": ["last_name"], "eduPersonPrincipalName": ["username"], "givenName": ["first_name"]}' +#export DJANGO_SAML_GROUP_OWNER_ATTRIBUTES='eduPersonScopedAffiliation' +#export DJANGO_SAML_GROUP_ATTRIBUTES='eduPersonScopedAffiliation' + +export GRAPHITE_HOST='localhost' +export GRAPHITE_PORT='8081' +export GRAPHITE_HOST='{{ pillar['graphite']['host'] }}' +export GRAPHITE_AMQP_PORT='{{ pillar['graphite']['port'] }}' +export GRAPHITE_AMQP_USER='{{ pillar['graphite']['user'] }}' +export GRAPHITE_AMQP_PASSWORD='{{ pillar['graphite']['password'] }}' +export GRAPHITE_AMQP_QUEUE='{{ pillar['graphite']['queue'] }}' +export GRAPHITE_AMQP_VHOST='{{ pillar['graphite']['vhost'] }}' +export SECRET_KEY='{{ pillar['secret_key'] }}' + +export PROXY_SECRET='{{ pillar['proxy_secret'] }}' + +export DEFAULT_FROM_EMAIL='root@localhost' + +#LOCAL="/home//.virtualenvs/circle/bin/postactivate.local" +#test -f "$LOCAL" && . "$LOCAL" diff --git b/salt/manager/files/syncdb.sh a/salt/manager/files/syncdb.sh new file mode 100644 index 0000000..ce2d5ca --- /dev/null +++ a/salt/manager/files/syncdb.sh @@ -0,0 +1,21 @@ +#!/bin/bash + +source /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/activate >/dev/null 2>&1 +source /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/postactivate >/dev/null 2>&1 +MANAGE="python /home/{{ pillar['user'] }}/circle/circle/manage.py" + +OUT=$( $MANAGE migrate 2>&1) + +if [ $? -ne 0 ]; then + /usr/bin/python -c "import sys; import json; sys.stdout.write(json.dumps({'changed': False, 'comment': sys.stdin.read()}) + '\n')" <<< "$OUT" + exit 1 +fi + +COUNT=$(/bin/egrep " *Applying " -c <<< "$OUT") +if [ $? -eq 0 ]; then + CHANGED=yes +else + CHANGED=no +fi + +echo "changed=$CHANGED comment='Migrated: $COUNT'" diff --git b/salt/manager/gitrepo.sls a/salt/manager/gitrepo.sls new file mode 100644 index 0000000..5b1afb5 --- /dev/null +++ a/salt/manager/gitrepo.sls @@ -0,0 +1,11 @@ +include: + - common + +gitrepo: + git.latest: + - name: {{ pillar['manager']['repo_name'] }} + - rev: {{ pillar['manager']['repo_revision'] }} + - target: /home/{{ pillar['user'] }}/circle + - user: {{ pillar['user'] }} + - require: + - pkg: git diff --git b/salt/manager/init.sls a/salt/manager/init.sls new file mode 100644 index 0000000..e5fa39d --- /dev/null +++ a/salt/manager/init.sls @@ -0,0 +1,90 @@ +include: + - manager.pipeline + - manager.gitrepo + - manager.agentgit + - manager.postgres + - manager.rabbitmq + - manager.virtualenv + - manager.configuration + - manager.nginx + +manager: + pkg.installed: + - pkgs: + - postgresql + - git + - ntp + - rabbitmq-server + - memcached + - gettext + - wget + - swig + {% if grains['os_family'] == 'RedHat' %} + - python2-pip + - libffi-devel + - openssl-devel + - libmemcached-devel + - postgresql-devel + - postgresql-libs + - postgresql-server + - libxml2-devel + - libxslt-devel + - python-devel + - python-virtualenvwrapper + {% else %} + - python-pip + - libffi-dev + - libssl-dev + - libmemcached-dev + - libpq-dev + - libxml2-dev + - libxslt1-dev + - python-dev + - virtualenvwrapper + {% endif %} + - require_in: + - service: postgres_service + user: + - present + - name: {{ pillar['user'] }} + - gid_from_name: True + - shell: /bin/bash + - groups: + {% if grains['os_family'] == 'RedHat' %} + - wheel + {% else %} + - sudo + {% endif %} + + + service: + - running + - enable: True + - watch: + - file: manager_postactivate + {% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} + - file: /etc/systemd/system/manager.service + - file: /etc/systemd/system/managercelery@.service + {% else %} + - file: /etc/init/manager.conf + - file: /etc/init/mancelery.conf + - file: /etc/init/moncelery.conf + - file: /etc/init/slowcelery.conf + {% endif %} + - sls: manager.gitrepo + +portal: + service: + - running + - enable: True + - watch: + - file: manager_postactivate + - file: portal.conf + - sls: manager.gitrepo + +memcached: + service: + - running + - enable: True + - require: + - pkg: manager diff --git b/salt/manager/nginx.sls a/salt/manager/nginx.sls new file mode 100644 index 0000000..805d88a --- /dev/null +++ a/salt/manager/nginx.sls @@ -0,0 +1,103 @@ +nginx: + service.running: + - enable: True + - watch: + - pkg: nginx + - cmd: circlecert + - file: nginxdefault + - file: nginx_home_permission + {% if grains['os_family'] == 'RedHat' %} + - file: nginxconf + - cmd: nginx_no_private_temp + {% endif %} + pkg: + - installed + +nginx_home_permission: + file.directory: + - name: /home/{{ pillar['user'] }} + - user: {{ pillar['user'] }} + - dir_mode: 711 + +circlecert: + cmd.run: + {% if grains['os_family'] == 'RedHat' %} + - name: ./make-dummy-cert circle.pem + {% else %} + - name: openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout circle.key -out circle.crt -subj '/CN=localhost/O=My Company Name LTD./C=US' && cat circle.key circle.crt > circle.pem && rm circle.key circle.crt; chmod 600 circle.pem + {% endif %} + - cwd: /etc/ssl/certs/ + - creates: /etc/ssl/certs/circle.pem + +{% if grains['os_family'] == 'RedHat' %} +nginx_selinux_pkgs: + pkg.installed: + - pkgs: + - policycoreutils + - policycoreutils-python + +nginx_httpd_can_network_connect: + selinux.boolean: + - name: httpd_can_network_connect + - value: True + - persist: True + - require: + - pkg: nginx_selinux_pkgs + +nginx_httpd_read_user_content: + selinux.boolean: + - name: httpd_read_user_content + - value: True + - persist: True + - require: + - pkg: nginx_selinux_pkgs + +/root/nginx.te: + file.managed: + - source: salt://manager/files/nginx.te + - template: jinja + - mode: 644 + +nginx_semodule: + cmd.run: + - cwd: /root + - user: root + - name: checkmodule -M -m -o nginx.mod nginx.te; semodule_package -o nginx.pp -m nginx.mod; semodule -i nginx.pp + - unless: semodule -l |grep -qs ^nginx + - require: + - file: /root/nginx.te + - pkg: nginx_selinux_pkgs + +nginx_no_private_temp: + cmd.run: + - user: root + - name: sed -i "/PrivateTmp/d" /usr/lib/systemd/system/nginx.service + - require: + - pkg: nginx +{% endif %} + +nginxdefault: + file.managed: + {% if grains['os_family'] == 'RedHat' %} + - name: /etc/nginx/conf.d/default.conf + {% else %} + - name: /etc/nginx/sites-enabled/default + {% endif %} + - template: jinja + - source: salt://manager/files/nginx-default-site.conf + - user: root + - group: root + - require: + - pkg: nginx + +{% if grains['os_family'] == 'RedHat' %} +nginxconf: + file.managed: + - name: /etc/nginx/nginx.conf + - template: jinja + - source: salt://manager/files/nginx.conf + - user: root + - group: root + - require: + - pkg: nginx +{% endif %} diff --git b/salt/manager/pipeline.sls a/salt/manager/pipeline.sls new file mode 100644 index 0000000..dbb8120 --- /dev/null +++ a/salt/manager/pipeline.sls @@ -0,0 +1,28 @@ +{% if grains['os'] == 'Ubuntu' or grains['os'] == 'Debian' %} +nodejs-legacy: + pkg.installed +{% endif %} + +npm: + {% if grains['os'] == 'Ubuntu' or grains['os'] == 'Debian' %} + pkg.installed: + - require: + - pkg: nodejs-legacy + {% else %} + pkg.installed + {% endif %} + +bower: + npm.installed: + - require: + - pkg: npm + +less: + npm.installed: + - require: + - pkg: npm + +yuglify: + npm.installed: + - require: + - pkg: npm diff --git b/salt/manager/postgres.sls a/salt/manager/postgres.sls new file mode 100644 index 0000000..e1593f3 --- /dev/null +++ a/salt/manager/postgres.sls @@ -0,0 +1,52 @@ +{% if grains['os_family'] == 'RedHat' %} +postgresql-server: + pkg.installed + +postgresql_initdb: + cmd.run: + - cwd: / + - user: root + - name: postgresql-setup initdb + - unless: test -f /var/lib/pgsql/data/postgresql.conf + - env: + LC_ALL: C.UTF-8 + file.managed: + - name: /var/lib/pgsql/data/pg_hba.conf + - template: jinja + - source: salt://manager/files/pg_hba.conf + - user: postgres + - group: postgres + - mode: 600 + - require: + - cmd: postgresql_initdb +{% endif %} + +postgres_service: + service.running: + - name: postgresql + - enable: True + {% if grains['os_family'] == 'RedHat' %} + - require: + - file: postgresql_initdb + {% endif %} + +dbuser: + postgres_user.present: + - name: {{ pillar['database']['user'] }} + - password: {{ pillar['database']['password'] }} + - user: postgres + - require: + - service: postgresql + +database: + postgres_database.present: + - name: {{ pillar['database']['name'] }} + - encoding: UTF8 + - lc_ctype: en_US.UTF8 + - lc_collate: en_US.UTF8 + - template: template0 + - owner: {{ pillar['database']['user'] }} + - user: postgres + - require: + - service: postgresql + - postgres_user: dbuser diff --git b/salt/manager/rabbitmq.sls a/salt/manager/rabbitmq.sls new file mode 100644 index 0000000..ff8142c --- /dev/null +++ a/salt/manager/rabbitmq.sls @@ -0,0 +1,32 @@ +rabbitmq-server: + pkg.installed: + - name: rabbitmq-server + {% if grains['os_family'] == 'RedHat' %} + file.managed: + - name: /etc/rabbitmq/rabbitmq-env.conf + - contents: RABBITMQ_DIST_PORT=5671 + {% endif %} + service.running: + - enable: True + - require: + - pkg: rabbitmq-server + {% if grains['os_family'] == 'RedHat' %} + - file: rabbitmq-server + {% endif %} + +rabbitmq_user: + rabbitmq_user.present: + - name: {{ pillar['amqp']['user'] }} + - password: {{ pillar['amqp']['password'] }} + - require: + - service: rabbitmq-server + +virtual_host: + rabbitmq_vhost.present: + - name: {{ pillar['amqp']['vhost']}} + - user: {{ pillar['amqp']['user'] }} + - conf: .* + - write: .* + - read: .* + - require: + - service: rabbitmq-server diff --git b/salt/manager/virtualenv.sls a/salt/manager/virtualenv.sls new file mode 100644 index 0000000..5256f47 --- /dev/null +++ a/salt/manager/virtualenv.sls @@ -0,0 +1,47 @@ +include: + - common + +# m2crypto workaround +# /usr/include/openssl/opensslconf.h:31: Error: CPP #error +# ""This openssl-devel package does not work your architecture?"". +# Use the -cpperraswarn option to continue swig processing. + +{% if grains['os_family'] == 'RedHat' %} +m2crypto_swig_env: + environ.setenv: + - name: SWIG_FEATURES + - value: -D__x86_64__ +{% endif %} + +virtualenv_manager: + virtualenv.managed: + - name: /home/{{ pillar['user'] }}/.virtualenvs/circle + - requirements: /home/{{ pillar['user'] }}/circle/requirements/{{ pillar['deployment_type'] }}.txt + - user: {{ pillar['user'] }} + - cwd: /home/{{ pillar['user'] }}/circle/ + - no_chown: true + - require: + - git: gitrepo + {% if grains['os_family'] == 'RedHat' %} + - environ: m2crypto_swig_env + {% endif %} + +salt://manager/files/syncdb.sh: + cmd.script: + - template: jinja + - user: {{ pillar['user'] }} + - stateful: true + - require: + - virtualenv: virtualenv_manager + - file: /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/postactivate + - user: {{ pillar['user'] }} + +salt://manager/files/compile.sh: + cmd.script: + - template: jinja + - user: {{ pillar['user'] }} + - stateful: true + - require: + - virtualenv: virtualenv_manager + - file: /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/postactivate + - user: {{ pillar['user'] }} diff --git b/salt/monitor-client/configuration.sls a/salt/monitor-client/configuration.sls new file mode 100644 index 0000000..320acab --- /dev/null +++ a/salt/monitor-client/configuration.sls @@ -0,0 +1,25 @@ +/home/{{ pillar['user'] }}/.virtualenvs/monitor-client/bin/postactivate: + file.managed: + - source: salt://monitor-client/files/postactivate + - template: jinja + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 700 + +{% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} +/etc/systemd/system/monitor-client.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/monitor-client/miscellaneous/monitor-client.service + +{% else %} + +/etc/init/monitor-client.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/monitor-client/miscellaneous/monitor-client.conf +{% endif %} diff --git b/salt/monitor-client/files/postactivate a/salt/monitor-client/files/postactivate new file mode 100644 index 0000000..b72431c --- /dev/null +++ a/salt/monitor-client/files/postactivate @@ -0,0 +1,6 @@ +export GRAPHITE_HOST='{{ pillar['graphite']['host'] }}' +export GRAPHITE_PORT='{{ pillar['graphite']['port'] }}' +export GRAPHITE_AMQP_USER='{{ pillar['graphite']['user'] }}' +export GRAPHITE_AMQP_PASSWORD='{{ pillar['graphite']['password'] }}' +export GRAPHITE_AMQP_QUEUE='{{ pillar['graphite']['queue'] }}' +export GRAPHITE_AMQP_VHOST='{{ pillar['graphite']['vhost'] }}' diff --git b/salt/monitor-client/gitrepo.sls a/salt/monitor-client/gitrepo.sls new file mode 100644 index 0000000..998b9f9 --- /dev/null +++ a/salt/monitor-client/gitrepo.sls @@ -0,0 +1,11 @@ +include: + - common + +gitrepo_monitor-client: + git.latest: + - name: {{ pillar['monitor-client']['repo_name'] }} + - rev: {{ pillar['monitor-client']['repo_revision'] }} + - target: /home/{{ pillar['user'] }}/monitor-client + - user: {{ pillar['user'] }} + - require: + - pkg: git diff --git b/salt/monitor-client/init.sls a/salt/monitor-client/init.sls new file mode 100644 index 0000000..6bcef85 --- /dev/null +++ a/salt/monitor-client/init.sls @@ -0,0 +1,31 @@ +include: + - monitor-client.gitrepo + - monitor-client.virtualenv + - monitor-client.configuration + +monitor-client: + pkg.installed: + - pkgs: + - git + - ntp + - wget + {% if grains['os_family'] == 'RedHat' %} + - python2-pip + - python-devel + - python-virtualenvwrapper + {% else %} + - python-pip + - python-dev + - virtualenvwrapper + {% endif %} + - require_in: + - git: gitrepo_monitor-client + - virtualenv: virtualenv_monitor-client + service: + - running + - enable: True + - watch: + - pkg: monitor-client + - sls: monitor-client.gitrepo + - sls: monitor-client.virtualenv + - sls: monitor-client.configuration diff --git b/salt/monitor-client/virtualenv.sls a/salt/monitor-client/virtualenv.sls new file mode 100644 index 0000000..14bd447 --- /dev/null +++ a/salt/monitor-client/virtualenv.sls @@ -0,0 +1,6 @@ +virtualenv_monitor-client: + virtualenv.managed: + - name: /home/{{ pillar['user'] }}/.virtualenvs/monitor-client + - requirements: /home/{{ pillar['user'] }}/monitor-client/requirements.txt + - user: {{ pillar['user'] }} + - no_chown: true diff --git b/salt/network/files/fix_dhcp.sh a/salt/network/files/fix_dhcp.sh new file mode 100644 index 0000000..c4f1fc5 --- /dev/null +++ a/salt/network/files/fix_dhcp.sh @@ -0,0 +1,9 @@ +#!/bin/bash +sed -i '/HWADDR=.*/d' /etc/sysconfig/network-scripts/ifcfg-vm +sed -i -e \$aNM_CONTROLLED=\"no\" /etc/sysconfig/network-scripts/ifcfg-vm +/bin/systemctl daemon-reload +ifup vm +systemctl restart firewall +systemctl restart dhcpd +exit 0 + diff --git b/salt/network/files/fix_dhcp_Debian.conf a/salt/network/files/fix_dhcp_Debian.conf new file mode 100644 index 0000000..7f0f87b --- /dev/null +++ a/salt/network/files/fix_dhcp_Debian.conf @@ -0,0 +1,10 @@ +# systemd service file extras added by CIRCLE Salt installer: +# openvswitch and virtual network interface must be up before +# dhcpd is started + +[Unit] +After=openvswitch-switch.service +[Service] +ExecStartPre=-/sbin/ifup vm +{# TODO: change 'vm' to pillar['fwdriver']['vm_if'] ? #} +{# TODO: similar patch for firewall.service ? #} diff --git b/salt/network/files/network a/salt/network/files/network new file mode 100644 index 0000000..5e1e505 --- /dev/null +++ a/salt/network/files/network @@ -0,0 +1,2 @@ +NETWORKING_IPV6=yes +IPV6FORWARDING=yes diff --git b/salt/network/files/reload_firewall.sh a/salt/network/files/reload_firewall.sh new file mode 100644 index 0000000..eb36539 --- /dev/null +++ a/salt/network/files/reload_firewall.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +source /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/activate +source /home/{{ pillar['user'] }}/.virtualenvs/circle/bin/postactivate +python /home/{{ pillar['user'] }}/circle/circle/manage.py reload_firewall --sync --timeout={{ pillar['fwdriver']['reload_firewall_timeout'] }} diff --git b/salt/network/init.sls a/salt/network/init.sls new file mode 100644 index 0000000..18417a9 --- /dev/null +++ a/salt/network/init.sls @@ -0,0 +1,114 @@ +ovs-if: + cmd.run: + - name: ovs-vsctl add-port cloud vm tag=2 -- set Interface vm type=internal + - unless: ovs-vsctl list-ifaces cloud | grep "^vm$" + +vm: + network.managed: + - enabled: True + - type: eth + - proto: none + - ipaddr: {{ pillar['fwdriver']['vm_net_ip'] }} + - netmask: {{ pillar['fwdriver']['vm_net_mask'] }} + - pre_up_cmds: + {% if grains['os_family'] == 'RedHat' %} + - /bin/systemctl restart openvswitch + {% elif grains['os'] == 'Debian' %} + - /bin/systemctl restart openvswitch-switch + {% else %} + - /etc/init.d/openvswitch-switch restart + {% endif %} + - require: + - cmd: ovs-if + +{% if grains['os'] == 'Debian' %} +symlink_dhcpd: + file.symlink: + - name: /etc/init.d/dhcpd + - target: /etc/init.d/isc-dhcp-server + - force: True + cmd.run: + - name: /bin/systemctl daemon-reload + - require: + - file: symlink_dhcpd +{% endif %} + +firewall2: + service: + - name: firewall + - running + - require: + - network: vm + +reload_firewall: + cmd.script: + - name: salt://network/files/reload_firewall.sh + - template: jinja + - user: {{ pillar['user'] }} + - require: + - service: firewall2 + {% if grains['os'] == 'Debian' %} + - cmd: symlink_dhcpd + {% endif %} + +{% if grains['os_family'] == 'RedHat' %} +net_config: + file.managed: + - name: /etc/sysconfig/network + - source: salt://network/files/network + - user: root + - group: root + - mode: 644 + +fix_dhcp: + cmd.script: + - name: salt://network/files/fix_dhcp.sh + - require: + - cmd: reload_firewall + - file: net_config +{% endif %} + +isc-dhcp-server: + {% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} + cmd.run: + - name: /bin/systemctl restart dhcpd + {% if grains['os_family'] == 'RedHat' %} + - watch: + - cmd: fix_dhcp + {% elif grains['os'] == 'Debian' %} + - watch: + - cmd: fix_dhcp_daemon_reload + {% endif %} + {% endif %} + service.running: + - enable: True + {% if grains['os_family'] == 'RedHat' %} + - watch: + - cmd: fix_dhcp + {% elif grains['os'] == 'Debian' %} + - watch: + - cmd: fix_dhcp_daemon_reload + {% endif %} + {% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} + - name: dhcpd + - require: + - cmd: isc-dhcp-server + {% endif %} + +{% if grains['os'] == 'Debian' %} +{# For next reboot #} +after_openvswitch_conf: + file.managed: + - name: /etc/systemd/system/isc-dhcp-server.service.d/after_openvswitch.conf + - source: salt://network/files/fix_dhcp_Debian.conf + - user: root + - group: root + - template: jinja + - makedirs: True + +fix_dhcp_daemon_reload: + cmd.run: + - name: /bin/systemctl daemon-reload + - require: + - file: after_openvswitch_conf +{% endif %} diff --git b/salt/nfs-client/init.sls a/salt/nfs-client/init.sls new file mode 100644 index 0000000..80ff5c4 --- /dev/null +++ a/salt/nfs-client/init.sls @@ -0,0 +1,20 @@ +nfs-client: + pkg.installed: + - pkgs: + {% if grains['os_family'] == 'RedHat' %} + - nfs-utils + {% else %} + - nfs-common + {% endif %} + - require_in: + - mount: /datastore + +/datastore: + mount.mounted: + - device: {{ pillar['nfs']['server'] }}:/datastore + - fstype: nfs + - opts: rw,nfsvers=3,noatime + - dump: 0 + - pass_num: 2 + - persist: True + - mkmnt: True diff --git b/salt/node.sls a/salt/node.sls new file mode 100644 index 0000000..d1e8a54 --- /dev/null +++ a/salt/node.sls @@ -0,0 +1,6 @@ +include: + - profile + - agentdriver + - monitor-client + - vmdriver + - nfs-client diff --git b/salt/openvswitch/files/openvswitch-2.3.1-1.x86_64.rpm a/salt/openvswitch/files/openvswitch-2.3.1-1.x86_64.rpm new file mode 100644 index 0000000..f0e85b0 Binary files /dev/null and a/salt/openvswitch/files/openvswitch-2.3.1-1.x86_64.rpm differ diff --git b/salt/openvswitch/init.sls a/salt/openvswitch/init.sls new file mode 100644 index 0000000..f2a5fb7 --- /dev/null +++ a/salt/openvswitch/init.sls @@ -0,0 +1,19 @@ +{% if grains['os_family'] == "RedHat" %} +openvswitch: + pkg.installed: + - sources: + - openvswitch: salt://openvswitch/files/openvswitch-2.3.1-1.x86_64.rpm + cmd.run: + - name: mkdir /etc/openvswitch; restorecon -R /etc/openvswitch/ + - creates: /etc/openvswitch + - require: + - pkg: openvswitch + service: + - name: openvswitch + - running + - enable: True + - require: + - cmd: openvswitch + - required_in: + - cmd: ovs-bridge +{% endif %} diff --git b/salt/profile.sls a/salt/profile.sls new file mode 100644 index 0000000..b83f800 --- /dev/null +++ a/salt/profile.sls @@ -0,0 +1,10 @@ +{% if grains['os']=='Debian' %} +{# For non-interactive shells, virtualenvwrapper commands + ('workon' etc.) are not sourced automatically #} +/etc/profile: + file.append: + - text: + - "#Line below added for Debian by CIRCLE Salt installer" + - . /etc/bash_completion +{% endif %} + diff --git b/salt/storagedriver/configuration.sls a/salt/storagedriver/configuration.sls new file mode 100644 index 0000000..7932527 --- /dev/null +++ a/salt/storagedriver/configuration.sls @@ -0,0 +1,55 @@ +/home/{{ pillar['user'] }}/.virtualenvs/storagedriver/bin/postactivate: + file.managed: + - source: salt://storagedriver/files/postactivate + - template: jinja + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 700 + +{% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} +/etc/systemd/system/storagecelery@.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/storagedriver/miscellaneous/storagecelery@.service + +/etc/systemd/system/storage.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/storagedriver/miscellaneous/storage.service + +{% else %} + +/etc/init/storagecelery.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/storagedriver/miscellaneous/storagecelery.conf + +/etc/init/storage.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/storagedriver/miscellaneous/storage.conf +{% endif %} + +/datastore: + file.directory: + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 755 + +# will not be needed in the future, with new gc +/datastore/trash: + file.directory: + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 755 + - require: + - file: /datastore + diff --git b/salt/storagedriver/files/agentdriver.incron a/salt/storagedriver/files/agentdriver.incron new file mode 100644 index 0000000..bfd10f7 --- /dev/null +++ a/salt/storagedriver/files/agentdriver.incron @@ -0,0 +1 @@ +/var/lib/libvirt/serial IN_CREATE setfacl -m u:{{ pillar['user'] }}:rw $@/$# diff --git b/salt/storagedriver/files/exports.tmpl a/salt/storagedriver/files/exports.tmpl new file mode 100644 index 0000000..45e9a87 --- /dev/null +++ a/salt/storagedriver/files/exports.tmpl @@ -0,0 +1 @@ +{{ pillar['nfs']['directory'] }} {{ pillar['nfs']['network'] }}(rw,async,insecure,no_subtree_check,no_root_squash) diff --git b/salt/storagedriver/files/postactivate a/salt/storagedriver/files/postactivate new file mode 100644 index 0000000..7bd0330 --- /dev/null +++ a/salt/storagedriver/files/postactivate @@ -0,0 +1,3 @@ +export AMQP_URI=amqp://{{ pillar['amqp']['user'] }}:{{ pillar['amqp']['password'] }}@{{ pillar['amqp']['host'] }}:{{ pillar['amqp']['port'] }}/{{ pillar['amqp']['vhost'] }} +export CACHE_URI={{ pillar['cache'] }} + diff --git b/salt/storagedriver/gitrepo.sls a/salt/storagedriver/gitrepo.sls new file mode 100644 index 0000000..df0c4b4 --- /dev/null +++ a/salt/storagedriver/gitrepo.sls @@ -0,0 +1,11 @@ +include: + - common + +gitrepo_storagedriver: + git.latest: + - name: {{ pillar['storagedriver']['repo_name'] }} + - rev: {{ pillar['storagedriver']['repo_revision'] }} + - target: /home/{{ pillar['user'] }}/storagedriver + - user: {{ pillar['user'] }} + - require: + - pkg: git diff --git b/salt/storagedriver/init.sls a/salt/storagedriver/init.sls new file mode 100644 index 0000000..24f2e42 --- /dev/null +++ a/salt/storagedriver/init.sls @@ -0,0 +1,50 @@ +include: + - storagedriver.gitrepo + - storagedriver.virtualenv + - storagedriver.configuration + - storagedriver.nfs-server + + +{% if grains['os_family'] == 'RedHat' %} +ev_repo_for_storagedriver: + pkg.installed: + - name: centos-release-qemu-ev +{% endif %} + +storagedriver: + pkg.installed: + - pkgs: + - git + - ntp + {% if grains['os_family'] == 'RedHat' %} + - python2-pip + - libmemcached-devel + - python-devel + - python-virtualenvwrapper + - qemu-img-ev + - zlib-devel + {% else %} + - python-pip + - libmemcached-dev + - python-dev + - qemu-utils + - virtualenvwrapper + - zlib1g-dev + {% endif %} + - require_in: + - git: gitrepo_storagedriver + - virtualenv: virtualenv_storagedriver + {% if grains['os_family'] == 'RedHat' %} + - require: + - pkg: ev_repo_for_storagedriver + {% endif %} + +storage: + service: + - running + - enable: True + - watch: + - pkg: storagedriver + - sls: storagedriver.gitrepo + - sls: storagedriver.virtualenv + - sls: storagedriver.configuration diff --git b/salt/storagedriver/nfs-server.sls a/salt/storagedriver/nfs-server.sls new file mode 100644 index 0000000..0e4c543 --- /dev/null +++ a/salt/storagedriver/nfs-server.sls @@ -0,0 +1,35 @@ +{% if pillar['nfs']['enabled'] %} +rpcbind: + pkg: + - installed + + service: + - running + - require: + - pkg: rpcbind + +nfs-server: + service: + {% if grains['os_family'] != 'RedHat' %} + - name: nfs-kernel-server + {% endif %} + - running + - watch: + - file: /etc/exports + - require: + - service: rpcbind + pkg.installed: + {% if grains['os_family'] == 'RedHat' %} + - name: nfs-utils + {% else %} + - name: nfs-kernel-server + {% endif %} + +/etc/exports: + file.managed: + - template: jinja + - sources: + - salt://storagedriver/files/exports.tmpl + - require: + - pkg: nfs-server +{% endif %} diff --git b/salt/storagedriver/virtualenv.sls a/salt/storagedriver/virtualenv.sls new file mode 100644 index 0000000..11380eb --- /dev/null +++ a/salt/storagedriver/virtualenv.sls @@ -0,0 +1,6 @@ +virtualenv_storagedriver: + virtualenv.managed: + - name: /home/{{ pillar['user'] }}/.virtualenvs/storagedriver + - requirements: /home/{{ pillar['user'] }}/storagedriver/requirements/production.txt + - user: {{ pillar['user'] }} + - no_chown: true diff --git b/salt/top.sls a/salt/top.sls new file mode 100644 index 0000000..7ee3314 --- /dev/null +++ a/salt/top.sls @@ -0,0 +1,3 @@ +base: + '*': + - vim diff --git b/salt/vmdriver/configuration.sls a/salt/vmdriver/configuration.sls new file mode 100644 index 0000000..9e2bb61 --- /dev/null +++ a/salt/vmdriver/configuration.sls @@ -0,0 +1,38 @@ +include: + - openvswitch + +/home/{{ pillar['user'] }}/.virtualenvs/vmdriver/bin/postactivate: + file.managed: + - source: salt://vmdriver/files/postactivate + - template: jinja + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 700 + +{% set service_dir = "/etc/systemd/system/" if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' else "/etc/init/" %} +{% set service_files = (("vmcelery@.service", "netcelery@.service", "node.service") + if grains['os_family'] == 'RedHat' + or grains['os'] == 'Debian' else + ("vmcelery.conf", "netcelery.conf", "node.conf")) %} + +{% for file in service_files %} +{{ service_dir ~ file }}: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/vmdriver/miscellaneous/{{ file }} +{% endfor %} + +ovs-bridge: + cmd.run: + - name: ovs-vsctl add-br cloud + - unless: ovs-vsctl list-br | grep "^cloud$" + +/etc/sudoers.d/netdriver: + file.managed: + - source: salt://vmdriver/files/sudoers + - template: jinja + - user: root + - group: root + - mode: 600 diff --git b/salt/vmdriver/files/10.virt.rules a/salt/vmdriver/files/10.virt.rules new file mode 100644 index 0000000..e7b0e9a --- /dev/null +++ a/salt/vmdriver/files/10.virt.rules @@ -0,0 +1,10 @@ +polkit.addRule(function(action, subject) { + polkit.log("action=" + action); + polkit.log("subject=" + subject); + var now = new Date(); + polkit.log("now=" + now) + if ((action.id == "org.libvirt.unix.manage" || action.id == "org.libvirt.unix.monitor") && subject.isInGroup("wheel")) { + return polkit.Result.YES; + } + return null; +}); diff --git b/salt/vmdriver/files/apparmor-libvirt a/salt/vmdriver/files/apparmor-libvirt new file mode 100644 index 0000000..bf1a578 --- /dev/null +++ a/salt/vmdriver/files/apparmor-libvirt @@ -0,0 +1,11 @@ +# +# This profile is for the domain whose UUID matches this file. +# + +#include <tunables/global> + +profile LIBVIRT_TEMPLATE { + #include <abstractions/libvirt-qemu> + /var/lib/libvirt/serial/** rwk, + /dev/vhost-net rw, +} diff --git b/salt/vmdriver/files/org.libvirt.unix.manage.pkla a/salt/vmdriver/files/org.libvirt.unix.manage.pkla new file mode 100644 index 0000000..f45d630 --- /dev/null +++ a/salt/vmdriver/files/org.libvirt.unix.manage.pkla @@ -0,0 +1,7 @@ +[Allow cloud libvirt management permissions] +Identity=unix-user:cloud +Action=org.libvirt.unix.manage;org.libvirt.unix.monitor +ResultAny=yes +ResultInactive=yes +ResultActive=yes +{# TODO: change 'cloud' to ? #} diff --git b/salt/vmdriver/files/postactivate a/salt/vmdriver/files/postactivate new file mode 100644 index 0000000..fb24edb --- /dev/null +++ a/salt/vmdriver/files/postactivate @@ -0,0 +1,6 @@ +export AMQP_URI=amqp://{{ pillar['amqp']['user'] }}:{{ pillar['amqp']['password'] }}@{{ pillar['amqp']['host'] }}:{{ pillar['amqp']['port'] }}/{{ pillar['amqp']['vhost'] }} +export CACHE_URI={{ pillar['cache'] }} +export LIBVIRT_URI=qemu:///system +export HYPERVISOR_TYPE="{{ pillar['vmdriver']['hypervisor_type'] }}" +export NATIVE_OVS=True + diff --git b/salt/vmdriver/files/sudoers a/salt/vmdriver/files/sudoers new file mode 100644 index 0000000..eee7640 --- /dev/null +++ a/salt/vmdriver/files/sudoers @@ -0,0 +1,3 @@ +{{ pillar['user'] }} ALL = (ALL) NOPASSWD: /usr/bin/ovs-ofctl, /usr/bin/ovs-vsctl, /sbin/ip link set * +Defaults: {{ pillar['user'] }} !requiretty + diff --git b/salt/vmdriver/files/usr.lib.libvirt.virt-aa-helper a/salt/vmdriver/files/usr.lib.libvirt.virt-aa-helper new file mode 100644 index 0000000..e2ef374 --- /dev/null +++ a/salt/vmdriver/files/usr.lib.libvirt.virt-aa-helper @@ -0,0 +1,66 @@ +# Last Modified: Mon Jul 06 17:22:37 2009 +#include <tunables/global> + +/usr/lib/libvirt/virt-aa-helper { + #include <abstractions/base> + #include <abstractions/user-tmp> + + # needed for searching directories + capability dac_override, + capability dac_read_search, + + # needed for when disk is on a network filesystem + network inet, + + deny @{PROC}/[0-9]*/mounts r, + @{PROC}/[0-9]*/net/psched r, + owner @{PROC}/[0-9]*/status r, + @{PROC}/filesystems r, + + # for hostdev + /sys/devices/ r, + /sys/devices/** r, + /sys/bus/usb/devices/ r, + /sys/bus/usb/devices/** r, + deny /dev/sd* r, + deny /dev/dm-* r, + deny /dev/mapper/ r, + deny /dev/mapper/* r, + + /usr/lib/libvirt/virt-aa-helper mr, + /sbin/apparmor_parser Ux, + + /etc/apparmor.d/libvirt/* r, + /etc/apparmor.d/libvirt/libvirt-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]* rw, + + # For backingstore, virt-aa-helper needs to peek inside the disk image, so + # allow access to non-hidden files in @{HOME} as well as storage pools, and + # removable media and filesystems, and certain file extentions. A + # virt-aa-helper failure when checking a disk for backinsgstore is non-fatal + # (but obviously the backingstore won't be added). + audit deny @{HOME}/.* mrwkl, + audit deny @{HOME}/.*/ rw, + audit deny @{HOME}/.*/** mrwkl, + @{HOME}/ r, + @{HOME}/** r, + @{HOME}/.Private/** mrwlk, + @{HOMEDIRS}/.ecryptfs/*/.Private/** mrwlk, + + /var/lib/libvirt/images/ r, + /var/lib/libvirt/images/** r, + /var/lib/nova/images/** r, + /var/lib/nova/instances/_base/** r, + /var/lib/nova/instances/snapshots/** r, + /var/lib/eucalyptus/instances/**/disk* r, + /var/lib/eucalyptus/instances/**/loader* r, + /var/lib/uvtool/libvirt/images/** r, + /{media,mnt,opt,srv}/** r, + + /**.img r, + /**.qcow{,2} r, + /**.qed r, + /**.vmdk r, + /**.[iI][sS][oO] r, + /**/disk{,.*} r, + /datastore/** r, +} diff --git b/salt/vmdriver/files/vmdriver.te a/salt/vmdriver/files/vmdriver.te new file mode 100644 index 0000000..936df9b --- /dev/null +++ a/salt/vmdriver/files/vmdriver.te @@ -0,0 +1,22 @@ + +module vmdriver 1.1; + +require { + type virt_var_lib_t; + type svirt_tcg_t; + type svirt_t; + type default_t; + class sock_file { create unlink }; + class dir { write remove_name add_name }; + class lnk_file read; +} + +#============= svirt_tcg_t ============== +allow svirt_tcg_t virt_var_lib_t:dir { write remove_name add_name }; +allow svirt_tcg_t virt_var_lib_t:sock_file { create unlink }; + +#============= svirt_t ============== +allow svirt_t virt_var_lib_t:dir { write add_name }; +allow svirt_t virt_var_lib_t:sock_file create; +allow svirt_t default_t:lnk_file read; + diff --git b/salt/vmdriver/gitrepo.sls a/salt/vmdriver/gitrepo.sls new file mode 100644 index 0000000..47f1a93 --- /dev/null +++ a/salt/vmdriver/gitrepo.sls @@ -0,0 +1,11 @@ +include: + - common + +gitrepo_vmdriver: + git.latest: + - name: {{ pillar['vmdriver']['repo_name'] }} + - rev: {{ pillar['vmdriver']['repo_revision'] }} + - target: /home/{{ pillar['user'] }}/vmdriver + - user: {{ pillar['user'] }} + - require: + - pkg: git diff --git b/salt/vmdriver/init.sls a/salt/vmdriver/init.sls new file mode 100644 index 0000000..2022f34 --- /dev/null +++ a/salt/vmdriver/init.sls @@ -0,0 +1,91 @@ +include: + - vmdriver.libvirt + - vmdriver.gitrepo + - vmdriver.virtualenv + - vmdriver.configuration + + +{% if grains['os_family'] == 'RedHat' %} +ev_repo_for_vmdriver: + pkg.installed: + - name: centos-release-qemu-ev +{% endif %} + +vmdriver: + pkg.installed: + - pkgs: + - git + - python-augeas + - ntp + - wget + {% if grains['os_family'] == 'RedHat' %} + - qemu-kvm-ev + - python2-pip + - libmemcached-devel + - libvirt + - libvirt-daemon + - libvirt-daemon-kvm + - libvirt-python + - libxml2-devel + - libxslt-devel + - python-devel + - python-virtualenvwrapper + - qemu-img-ev + - zlib-devel + {% else %} + - qemu-kvm + - python-pip + - libmemcached-dev + - libvirt-bin + - libxml2-dev + - libxslt1-dev + - openvswitch-common + - openvswitch-switch + {% if grains['os'] != 'Debian' %} + {# No such package in Debian Jessie! #} + - openvswitch-controller + {% endif %} + - python-dev + - python-libvirt + - virtualenvwrapper + - qemu-utils + - zlib1g-dev + {% endif %} + - require_in: + - file: /etc/default/libvirt-bin + {% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} + - service: libvirtd + {% else %} + - file: /etc/apparmor.d/libvirt/TEMPLATE + - file: /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper + - file: /var/lib/libvirt/serial + - service: libvirt-bin + {% endif %} + - augeas: libvirtconf + - git: gitrepo_vmdriver + - virtualenv: virtualenv_vmdriver + {% if grains['os_family'] == 'RedHat' %} + - require: + - pkg: ev_repo_for_vmdriver + {% endif %} + +agentdriver_service: + service: + - name: agentdriver + - running + - enable: true + - watch: + - pkg: agentdriver + - sls: agentdriver.gitrepo + - sls: agentdriver.virtualenv + - sls: agentdriver.configuration + +node: + service: + - running + - enable: True + - watch: + - pkg: vmdriver + - sls: vmdriver.gitrepo + - sls: vmdriver.virtualenv + - sls: vmdriver.configuration diff --git b/salt/vmdriver/libvirt.sls a/salt/vmdriver/libvirt.sls new file mode 100644 index 0000000..5fa5c51 --- /dev/null +++ a/salt/vmdriver/libvirt.sls @@ -0,0 +1,126 @@ +augeas_dependency: + pkg.installed: + - pkgs: + - python-augeas + +libvirtconf: + augeas.change: + - context: /files/etc/libvirt/libvirtd.conf + - changes: + - set listen_tcp 1 + - set listen_tls 0 + - set auth_tcp "none" + +/etc/default/libvirt-bin: + file.append: + - text: libvirtd_opts="-d -l" + +{% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} +libvirtd: +{% else %} +libvirt-bin: +{% endif %} + service: + - running + - watch: + - file: /etc/default/libvirt-bin + - augeas: libvirtconf + +{% if grains['os_family'] == 'RedHat' %} +/usr/bin/kvm: + file.symlink: + - target: /usr/libexec/qemu-kvm + +/etc/polkit-1/rules.d/10.virt.rules: + file.managed: + - source: salt://vmdriver/files/10.virt.rules + - template: jinja + - mode: 644 + +polkit: + service: + - running + - watch: + - file: /etc/polkit-1/rules.d/10.virt.rules + +/root/vmdriver.te: + file.managed: + - source: salt://vmdriver/files/vmdriver.te + - template: jinja + - mode: 644 + +selinux_pkgs: + pkg.installed: + - pkgs: + - policycoreutils + - policycoreutils-python + +vmdriver_semodule: + cmd.run: + - cwd: /root + - user: root + - name: checkmodule -M -m -o vmdriver.mod vmdriver.te; semodule_package -o vmdriver.pp -m vmdriver.mod; semodule -i vmdriver.pp + - unless: semodule -l |grep -qs ^vmdriver + - require: + - file: /root/vmdriver.te + - pkg: selinux_pkgs + +{% elif grains['os'] == 'Debian' %} + +/usr/bin/kvm: + file.replace: + - pattern: -enable-kvm + - repl: "" + - watch: + - pkg: vmdriver + +policycoreutils: + pkg.installed + +{# Note: Debian Jessie has polkit 0.105, which uses pkla format instead of js #} +/etc/polkit-1/localauthority/50-local.d/org.libvirt.unix.manage.pkla: + file.managed: + - source: salt://vmdriver/files/org.libvirt.unix.manage.pkla + - user: root + - group: root + - template: jinja + +polkitd: + service: + - running + - watch: + - file: /etc/polkit-1/localauthority/50-local.d/org.libvirt.unix.manage.pkla + +{% else %} + +/etc/apparmor.d/libvirt/TEMPLATE: + file.managed: + - source: salt://vmdriver/files/apparmor-libvirt + - template: jinja + - mode: 644 + +/etc/apparmor.d/usr.lib.libvirt.virt-aa-helper: + file.managed: + - source: salt://vmdriver/files/usr.lib.libvirt.virt-aa-helper + - template: jinja + - mode: 644 + +apparmor: + service: + - reload: true + - running + - watch: + - file: /etc/apparmor.d/libvirt/TEMPLATE + - file: /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper +{% endif %} + +/var/lib/libvirt/serial: + file.directory: + - makedirs: True + {% if grains['os_family'] == 'RedHat' %} + - user: qemu + {% else %} + - user: libvirt-qemu + {% endif %} + - group: kvm + - mode: 755 diff --git b/salt/vmdriver/virtualenv.sls a/salt/vmdriver/virtualenv.sls new file mode 100644 index 0000000..c469b87 --- /dev/null +++ a/salt/vmdriver/virtualenv.sls @@ -0,0 +1,20 @@ +virtualenv_vmdriver: + virtualenv.managed: + - name: /home/{{ pillar['user'] }}/.virtualenvs/vmdriver + - requirements: /home/{{ pillar['user'] }}/vmdriver/requirements/production.txt + - user: {{ pillar['user'] }} + - no_chown: true + +{% set libvirt_dir = "/usr/lib64/python2.7/site-packages/" if grains['os_family'] == 'RedHat' else "/usr/lib/python2.7/dist-packages/" %} + +{% set targets = { 'libvirtmod_qemu.so': 'libvirtmod_qemu.x86_64-linux-gnu.so', + 'libvirtmod.so': 'libvirtmod.x86_64-linux-gnu.so' + } if grains['os'] == 'Debian' else {} %} + +{% for file in ("libvirtmod_qemu.so", "libvirtmod.so", "libvirt_qemu.py", "libvirt.py", "libvirt_qemu.pyc", "libvirt.pyc") %} +/home/{{ pillar['user'] }}/.virtualenvs/vmdriver/lib/python2.7/site-packages/{{ file }}: + file.symlink: + - target: {{ libvirt_dir + targets[file]|default(file) }} + - require: + - virtualenv: virtualenv_vmdriver +{% endfor %} diff --git b/salt/vncproxy/configuration.sls a/salt/vncproxy/configuration.sls new file mode 100644 index 0000000..dc4112c --- /dev/null +++ a/salt/vncproxy/configuration.sls @@ -0,0 +1,25 @@ +/home/{{ pillar['user'] }}/.virtualenvs/vncproxy/bin/postactivate: + file.managed: + - source: salt://vncproxy/files/postactivate + - template: jinja + - user: {{ pillar['user'] }} + - group: {{ pillar['user'] }} + - mode: 700 + +{% if grains['os_family'] == 'RedHat' or grains['os'] == 'Debian' %} +/etc/systemd/system/vncproxy.service: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/vncproxy/miscellaneous/vncproxy.service + +{% else %} + +/etc/init/vncproxy.conf: + file.managed: + - user: root + - group: root + - template: jinja + - source: file:///home/{{ pillar['user'] }}/vncproxy/miscellaneous/vncproxy.conf +{% endif %} diff --git b/salt/vncproxy/files/postactivate a/salt/vncproxy/files/postactivate new file mode 100644 index 0000000..841773f --- /dev/null +++ a/salt/vncproxy/files/postactivate @@ -0,0 +1 @@ +export PROXY_SECRET={{ pillar['proxy_secret'] }} diff --git b/salt/vncproxy/gitrepo.sls a/salt/vncproxy/gitrepo.sls new file mode 100644 index 0000000..a620212 --- /dev/null +++ a/salt/vncproxy/gitrepo.sls @@ -0,0 +1,11 @@ +include: + - common + +gitrepo_vncproxy: + git.latest: + - name: {{ pillar['vncproxy']['repo_name'] }} + - rev: {{ pillar['vncproxy']['repo_revision'] }} + - target: /home/{{ pillar['user'] }}/vncproxy + - user: {{ pillar['user'] }} + - require: + - pkg: git diff --git b/salt/vncproxy/init.sls a/salt/vncproxy/init.sls new file mode 100644 index 0000000..e28de25 --- /dev/null +++ a/salt/vncproxy/init.sls @@ -0,0 +1,35 @@ +include: + - vncproxy.gitrepo + - vncproxy.virtualenv + - vncproxy.configuration + +vncproxy: + pkg.installed: + - pkgs: + - git + - ntp + - wget + {% if grains['os_family'] == 'RedHat' %} + - python2-pip + - libffi-devel + - openssl-devel + - python-devel + - python-virtualenvwrapper + {% else %} + - python-pip + - libffi-dev + - libssl-dev + - python-dev + - virtualenvwrapper + {% endif %} + - require_in: + - git: gitrepo_vncproxy + - virtualenv: virtualenv_vncproxy + service: + - running + - enable: True + - watch: + - pkg: vncproxy + - sls: vncproxy.gitrepo + - sls: vncproxy.virtualenv + - sls: vncproxy.configuration diff --git b/salt/vncproxy/virtualenv.sls a/salt/vncproxy/virtualenv.sls new file mode 100644 index 0000000..2f1f2d1 --- /dev/null +++ a/salt/vncproxy/virtualenv.sls @@ -0,0 +1,6 @@ +virtualenv_vncproxy: + virtualenv.managed: + - name: /home/{{ pillar['user'] }}/.virtualenvs/vncproxy + - requirements: /home/{{ pillar['user'] }}/vncproxy/requirements.txt + - user: {{ pillar['user'] }} + - no_chown: true diff --git b/salt/win/repo/7zip/init.sls a/salt/win/repo/7zip/init.sls new file mode 100644 index 0000000..239ba9e --- /dev/null +++ a/salt/win/repo/7zip/init.sls @@ -0,0 +1,9 @@ +7zip: + 9.20.00.0: + installer: 'http://hivelocity.dl.sourceforge.net/project/sevenzip/7-Zip/9.20/7z920-x64.msi' + full_name: '7-Zip 9.20 (x64 edition)' + reboot: False + install_flags: ' /q ' + msiexec: True + uninstaller: 'http://hivelocity.dl.sourceforge.net/project/sevenzip/7-Zip/9.20/7z920-x64.msi' + uninstall_flags: ' /qn' diff --git b/salt/win/repo/msysgit/init.sls a/salt/win/repo/msysgit/init.sls new file mode 100644 index 0000000..1efa9a2 --- /dev/null +++ a/salt/win/repo/msysgit/init.sls @@ -0,0 +1,8 @@ +msysgit: + 1.9.4-preview20140815: + installer: 'https://github.com/msysgit/msysgit/releases/download/Git-1.9.4-preview20140815/Git-1.9.4-preview20140815.exe' + install_flags: ' /VERYSILENT /NOREBOOT' + full_name: 'Git version 1.9.4-preview20140815' + reboot: False + uninstaller: 'C:\Program Files (x86)\Git\unins000.exe' + uninstall_flags: ' /VERYSILENT /NOREBOOT' diff --git b/salt/win/repo/python2/init.sls a/salt/win/repo/python2/init.sls new file mode 100644 index 0000000..2a32e48 --- /dev/null +++ a/salt/win/repo/python2/init.sls @@ -0,0 +1,9 @@ +python2: + 2.7.8150: + full_name: 'Python 2.7.8 (64-bit)' + msiexec: True + installer: 'https://www.python.org/ftp/python/2.7.8/python-2.7.8.amd64.msi' + install_flags: '/qn /norestart' + uninstaller: 'https://www.python.org/ftp/python/2.7.8/python-2.7.8.amd64.msi' + uninstall_flags: '/qn' + reboot: False diff --git b/salt/win/repo/winrepo.p a/salt/win/repo/winrepo.p new file mode 100644 index 0000000..177568b Binary files /dev/null and a/salt/win/repo/winrepo.p differ diff --git b/salt/winagent/files/distutils.cfg a/salt/winagent/files/distutils.cfg new file mode 100644 index 0000000..baeab4a --- /dev/null +++ a/salt/winagent/files/distutils.cfg @@ -0,0 +1,2 @@ +[build] +compiler = mingw32 diff --git b/salt/winagent/init.sls a/salt/winagent/init.sls new file mode 100644 index 0000000..329a5f8 --- /dev/null +++ a/salt/winagent/init.sls @@ -0,0 +1,93 @@ +msysgit: + pkg.installed + +python2: + pkg.installed + +7zip: + pkg.installed + +get-pip.py: + file.managed: + - name: c:/get-pip.py + - source: https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py + - source_hash: md5=515f9476562994aa997df488c6c6c080 + +ez_setup.py: + file.managed: + - name: c:/ez_setup.py + - source: https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py + - source_hash: md5=4cfc24855347d1e01a73ff38830455b4 + +mingwget.zip: + file.managed: + - name: c:/mingwget.zip + - source: http://heanet.dl.sourceforge.net/project/mingw/Installer/mingw-get/mingw-get-0.6.2-beta-20131004-1/mingw-get-0.6.2-mingw32-beta-20131004-1-bin.zip + - source_hash: md5=971778e9330ae006aaeb2d63344be5f3 + +psutil.exe: + file.managed: + - name: c:/psutil.exe + - source: https://pypi.python.org/packages/2.7/p/psutil/psutil-2.1.3.win32-py2.7.exe + - source_hash: md5=57ded53eb8082c438f626c9e0de3357a + +distutils.cfg: + file.managed: + - name: C:\python27\Lib\distutils\distutils.cfg + - source: salt://winagent/files/distutils.cfg + - template: jinja + +getpip: + cmd.run: + - name: c:/python27/python.exe c:/get-pip.py + - unless: which pip + - require: + - pkg: python2 + - file: get-pip.py + - reload_modules: True + +easy_install: + cmd.run: + - name: c:/python27/python.exe c:/ez_setup.py > nul + - unless: which pip + - require: + - pkg: python2 + - file: ez_setup.py + - reload_modules: True + +unzip-mingw: + cmd.run: + - name: '"c:/Program Files/7-zip/7z.exe" x -o"C:\MinGW" -y c:/mingwget.zip' + - require: + - pkg: 7zip + - file: mingwget.zip + +install_gcc: + cmd.run: + - name: 'c:\MinGW\bin\mingw-get install gcc' + - require: + - cmd: unzip-mingw + - win_path: 'C:\MinGW\bin' + +pywin32: + file.managed: + - name: 'C:/pywin32.exe' + - source: http://softlayer-ams.dl.sourceforge.net/project/pywin32/pywin32/Build%20219/pywin32-219.win32-py2.7.exe + - source_hash: md5=f270e9f88155f649fc1a6c2f85aa128d + +install_pywin32: + cmd.run: + - name: 'c:/Python27/Scripts/easy_install c:/pywin32.exe' + - require: + - file: pywin32 + +git_clone: + cmd.run: + - name: '"C:\Program Files (x86)\Git\bin\git.exe" clone {{ pillar['agent']['repo_name'] }} c:/agent' + - require: + - pkg: msysgit + +pyinstaller_agent: + cmd.run: + - name: 'pyinstaller -F --hidden-import pkg_resources --hidden-import infi agent-winservice.py' + - cwd: 'c:/agent'