diff --git a/docs/deploy.rst b/docs/deploy.rst index 1e642c7..d8e7d8d 100644 --- a/docs/deploy.rst +++ b/docs/deploy.rst @@ -1,4 +1,151 @@ Deploy -======== +====== + +This tutorial describes the installation of a production environment. To +have a fully working environment, you have to set up the other components +as well. The full procedure is included in the :doc:`Puppet recipes +<puppet>` available for CIRCLE Cloud. + +This component should normally deployed to a single head node. +This is the web-based entry point to the end users, and also the manager of +the compute and storage nodes. + +Preparation +----------- + +To get the project running, launch a new Ubuntu 14.04 machine, and +log in to it over SSH. + + +.. warning:: + If the first character of the hostname should not be a digit, because + RabbitMQ won't work with it. + + The machine should have an :abbr:`fqdn (fully qualified domain name)`, + which shoud be correctly printed by :kbd:`hostname -f`. You can achieve + this with an IP address (e.g. 127.0.1.1) in :file:`/etc/hosts` having the + short hostname as first, and the fqdn as second alias). + + +Setting up required software +---------------------------- + +Update the package lists, and install the required system software:: + + sudo apt-get update + sudo apt-get install --yes virtualenvwrapper postgresql git \ + python-pip rabbitmq-server libpq-dev python-dev ntp memcached \ + libmemcached-dev gettext wget pwgen nginx + +Set up *PostgreSQL* to listen on localhost and restart it:: + + sudo sed -i /etc/postgresql/9.1/main/postgresql.conf -e '/#listen_addresses/ s/^#//' + sudo /etc/init.d/postgresql restart + +Also, create a new database and user:: + + pwgen 12 >pgpw + sudo -u postgres createuser -S -D -R circle + sudo -u postgres psql <<<"ALTER USER circle WITH PASSWORD '$(cat pgpw)';" + sudo -u postgres createdb circle -O circle + +Configure RabbitMQ: remove the guest user, add virtual host and user with +proper permissions:: + + pwgen 12 >rmqpw + sudo rabbitmqctl delete_user guest + sudo rabbitmqctl add_vhost circle + sudo rabbitmqctl add_user cloud $(cat rmqpw) + sudo rabbitmqctl set_permissions -p circle cloud '.*' '.*' '.*' + +Set up nginx to serve the CIRCLE portal. :: + + sudo tee /etc/nginx/conf.d/default.conf <<END + ignore_invalid_headers on; + server { + listen 443 ssl default; + ssl on; + ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem; + ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key; + location /static { + alias ${PWD}/circle/static_collected; # your Django project's static files + } + location / { + uwsgi_pass unix:///tmp/uwsgi.sock; + include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually + } + location /vnc/ { + proxy_pass http://localhost:9999; + proxy_set_header X-Real-IP \$remote_addr; + proxy_set_header Host \$host; + proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for; + # WebSocket support (nginx 1.4) + proxy_http_version 1.1; + proxy_set_header Upgrade \$http_upgrade; + proxy_set_header Connection "upgrade"; + } + } + + server { + listen 80 default; + rewrite ^ https://\$host/; # permanent; + } + END + sudo /etc/init.d/nginx restart + +.. warning:: + For a production deployment, you should use certificates issued by a + recognized certificate authority. Until you get it, you can use a + self-signed one automatically generated by the package. + +Setting up Circle itself +------------------------ + +Clone the git repository:: + + git clone https://git.ik.bme.hu/circle/cloud.git circle + +Set up *virtualenvwrapper* and the *virtual Python environment* for the +project:: + + source /etc/bash_completion.d/virtualenvwrapper + mkvirtualenv circle + +Set up default Circle configuration and activate the virtual environment:: + + cat >>/home/cloud/.virtualenvs/circle/bin/postactivate <<END + export DJANGO_SETTINGS_MODULE=circle.settings.production + export DJANGO_DB_HOST=localhost + export DJANGO_DB_PASSWORD=$(cat pgpw) + export DJANGO_FIREWALL_SETTINGS='{"dns_ip": "152.66.243.60", "dns_hostname": + "localhost", "dns_ttl": "300", "reload_sleep": "10", + "rdns_ip": "152.66.243.60", "default_vlangroup": "publikus"}' + export AMQP_URI='amqp://cloud:$(cat rmqpw)@localhost:5672/circle' + export CACHE_URI='pylibmc://127.0.0.1:11211/' + END + workon circle + cd ~/circle + +You should change DJANGO_FIREWALL_SETTINGS to your needs. + +Install the required Python libraries to the virtual environment:: + + pip install -r requirements.txt + +Sync the database and create a superuser:: + + circle/manage.py syncdb --all --noinput + circle/manage.py migrate --fake + circle/manage.py createsuperuser + +Copy the init files to Upstart's config directory and start the manager and +the portal application server:: + + sudo cp miscellaneous/mancelery.conf /etc/init/ + sudo start mancelery + sudo cp miscellaneous/portal-uwsgi.conf /etc/init/ + sudo start portal-uwsgi + + + -This is where you describe how the project is deployed in production. diff --git a/docs/install.rst b/docs/install.rst index 3c75574..9d69534 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -3,31 +3,40 @@ Installation of a development machine .. highlight:: bash +This tutorial describes the installation of a development environment. To +have a fully working environment, you have to set up the other components +as well. The full procedure is included in the :doc:`Puppet recipes +</puppet>` available for CIRCLE Cloud. + Preparation ----------- -To get the project running on a development machine, create a new Ubuntu 12.04 -instance, and log in to it over SSH. +To get the project running on a development machine, launch a new Ubuntu +14.04 machine, and log in to it over SSH. -To use *git* over *SSH*, we advise enabling SSH *agent forwarding*. -On your personal computer check if *ssh-agent* is running (the command should -print a process id):: - - $ echo $SSH_AGENT_PID - 1234 +.. info:: + To use *git* over *SSH*, we advise enabling SSH *agent forwarding*. + On your terminal computer check if *ssh-agent* is running (the command + should print a process id):: + + $ echo $SSH_AGENT_PID + 1234 + + If it is not running, you can configure your dektop environment to + automatically launch it. -If it is not running, you should set up your login manager or some other -solution to automatically launch it. + Add your private key to the agent (if it is not added by your desktop + environment):: -Add your private key to the agent (if it is not added by your desktop -environment):: + ssh-add [~/.ssh/path_to_id_rsa] - $ ssh-add [~/.ssh/path_to_id_rsa] + You can read and write all repositories over https, but you will have to + provide username and password for every push command. Log in to the new vm. The :kbd:`-A` switch enables agent forwarding:: - $ ssh -A cloud@host + ssh -A cloud@host You can check agent forwarding on the vm:: @@ -38,55 +47,55 @@ You can check agent forwarding on the vm:: If the first character of the hostname of the vm is a digit, you have to change it, because RabbitMQ won't work with it. :: - $ old=$(hostname) - $ new=c-${old} - $ sudo tee /etc/hostname <<<$new - $ sudo hostname $new - $ sudo sed -i /etc/hosts -e "s/$old/$new/g" + old=$(hostname) + new=c-${old} + sudo tee /etc/hostname <<<$new + sudo hostname $new + sudo sed -i /etc/hosts -e "s/$old/$new/g" Setting up required software ---------------------------- Update the package lists, and install the required system software:: - $ sudo apt-get update - $ sudo apt-get install --yes virtualenvwrapper postgresql git \ - python-pip rabbitmq-server libpq-dev python-dev ntp memcached \ - libmemcached-dev + sudo apt-get update + sudo apt-get install --yes virtualenvwrapper postgresql git \ + python-pip rabbitmq-server libpq-dev python-dev ntp memcached \ + libmemcached-dev Set up *PostgreSQL* to listen on localhost and restart it:: - $ sudo sed -i /etc/postgresql/9.1/main/postgresql.conf -e '/#listen_addresses/ s/^#//' - $ sudo /etc/init.d/postgresql restart + sudo sed -i /etc/postgresql/9.1/main/postgresql.conf -e '/#listen_addresses/ s/^#//' + sudo /etc/init.d/postgresql restart Also, create a new database and user:: - $ sudo -u postgres createuser -S -D -R circle - $ sudo -u postgres psql <<<"ALTER USER circle WITH PASSWORD 'circle';" - $ sudo -u postgres createdb circle -O circle + sudo -u postgres createuser -S -D -R circle + sudo -u postgres psql <<<"ALTER USER circle WITH PASSWORD 'circle';" + sudo -u postgres createdb circle -O circle Configure RabbitMQ: remove the guest user, add virtual host and user with proper permissions:: - $ sudo rabbitmqctl delete_user guest - $ sudo rabbitmqctl add_vhost circle - $ sudo rabbitmqctl add_user cloud password - $ sudo rabbitmqctl set_permissions -p circle cloud '.*' '.*' '.*' + sudo rabbitmqctl delete_user guest + sudo rabbitmqctl add_vhost circle + sudo rabbitmqctl add_user cloud password + sudo rabbitmqctl set_permissions -p circle cloud '.*' '.*' '.*' Enable SSH server to accept your name and address from your environment:: - $ sudo sed -i /etc/ssh/sshd_config -e '$ a AcceptEnv GIT_*' - $ sudo /etc/init.d/ssh reload + sudo sed -i /etc/ssh/sshd_config -e '$ a AcceptEnv GIT_*' + sudo /etc/init.d/ssh reload You should set these vars in your **local** profile:: - $ cat >>~/.profile <<'END' + cat >>~/.profile <<'END' export GIT_AUTHOR_NAME="Your Name" export GIT_AUTHOR_EMAIL="your.address@example.org" export GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME" export GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL" END - $ source ~/.profile + source ~/.profile Allow sending it in your **local** ssh configuration:: @@ -100,17 +109,23 @@ Setting up Circle itself Clone the git repository:: - $ git clone git@git.cloud.ik.bme.hu:circle/cloud.git circle + git clone https://git.ik.bme.hu/circle/cloud.git circle + +If you want to push back any modifications, it is possible to set SSH as the +push protocol:: + + cd circle + git remote set-url --push origin git@git.ik.bme.hu:circle/cloud.git Set up *virtualenvwrapper* and the *virtual Python environment* for the project:: - $ source /etc/bash_completion.d/virtualenvwrapper - $ mkvirtualenv circle + source /etc/bash_completion.d/virtualenvwrapper + mkvirtualenv circle Set up default Circle configuration and activate the virtual environment:: - $ cat >>/home/cloud/.virtualenvs/circle/bin/postactivate <<END + cat >>/home/cloud/.virtualenvs/circle/bin/postactivate <<END export DJANGO_SETTINGS_MODULE=circle.settings.local export DJANGO_DB_HOST=localhost export DJANGO_DB_PASSWORD=circle @@ -120,32 +135,32 @@ Set up default Circle configuration and activate the virtual environment:: export AMQP_URI='amqp://cloud:password@localhost:5672/circle' export CACHE_URI='pylibmc://127.0.0.1:11211/' END - $ workon circle - $ cd ~/circle + workon circle + cd ~/circle Install the required Python libraries to the virtual environment:: - $ pip install -r requirements/local.txt + pip install -r requirements/local.txt Sync the database and create a superuser:: - $ circle/manage.py syncdb --all --noinput - $ circle/manage.py migrate --fake - $ circle/manage.py createsuperuser --username=test --email=test@example.org + circle/manage.py syncdb --all --noinput + circle/manage.py migrate --fake + circle/manage.py createsuperuser --username=test --email=test@example.org You can now start the development server:: - $ circle/manage.py runserver '[::]:8080' + circle/manage.py runserver '[::]:8080' You will also need to run a local Celery worker:: - $ circle/manage.py celery worker -A manager.mancelery + circle/manage.py celery worker -A manager.mancelery .. note:: You might run the Celery worker (and also the development server) in GNU Screen, or use Upstart:: - $ sudo cp miscellaneous/mancelery.conf /etc/init/ - $ sudo start mancelery + sudo cp miscellaneous/mancelery.conf /etc/init/ + sudo start mancelery Building documentation ---------------------- @@ -153,14 +168,14 @@ Building documentation To build the *docs*, install *make*, go to the docs folder, and run the building process. :: - $ sudo apt-get install make - $ cd ~/circle/docs/ - $ make html + sudo apt-get install make + cd ~/circle/docs/ + make html You might also want to serve the generated docs with Python's development server:: - $ (cd _build/html && python -m SimpleHTTPServer 8080) + (cd _build/html && python -m SimpleHTTPServer 8080) Configuring vim --------------- @@ -168,16 +183,16 @@ Configuring vim To follow the coding style of the project more easily, you might want to configure vim like we do:: - $ mkdir -p ~/.vim/autoload ~/.vim/bundle - $ curl -Sso ~/.vim/autoload/pathogen.vim \ - https://raw.githubusercontent.com/tpope/vim-pathogen/master/autoload/pathogen.vim - $ cd ~/.vim; mkdir -p bundle; cd bundle && git clone \ - git://github.com/klen/python-mode.git - $ cat >>~/.vimrc <<END + mkdir -p ~/.vim/autoload ~/.vim/bundle + curl -Sso ~/.vim/autoload/pathogen.vim \ + https://raw.github.com/tpope/vim-pathogen/master/autoload/pathogen.vim + cd ~/.vim; mkdir -p bundle; cd bundle && git clone \ + git://github.com/klen/python-mode.git + cat >>~/.vimrc <<END filetype off call pathogen#infect() call pathogen#helptags() filetype plugin indent on syntax on END - $ sudo pip install pyflakes rope pep8 mccabe + sudo pip install pyflakes rope pep8 mccabe