Building a Python/Django Development Virtual Machine

Recently for a project, I needed to build a Python, Django, PostgreSQL, NGINX Development Virtual Machine(VM). Below are the steps that I followed to build this VM in VMware Fusion on MacbookPro (MBP). This post is as much about sharing my build experience as documenting the steps for my future use and potential automation.

The guidance for this procedure came from How To Install and Configure Django with Postgres, Nginx, and Gunicorn.

Installation

Ubuntu Server OS

The steps were very similar to the ones I covered in my prior post OpenStack: Quick Install using DevStack.

Download Ubuntu 12.04 "Precise Pangolin" x86_64 Minimal CD ISO Image mini.iso.

Start VMware Fusion and select Virtual Machine Library in Windows option on VMware Fusion toolbar. This will bring up Virtual Machine Library window showing all the Virtual Machine already available.

Click Add button and select New. This will bring up New Virtual Assistance Window showing Create New Virtual Machine.

Click Continue without disc as we will be using the downloaded ISO image. This will bring up Installation Media section.
Select Use operating system installation disc or image and click on arrows next to Choose a disc or disc image.... Select the Ubuntu ISO image and then click Continue.

Choose Operating System section should show Linux as Operating System and Ubuntu 64-bit as Version. Click Continue.

The Finish section will show Virtual Machine Summary, Click Finish. Select the location where we want to save the VM file and name the file.

A console window will be launched and OS install will start. Answered the prompts during the install process. Once update and reboot completes, a login prompt will appear. Log in to VM.

OpenSSH Server

After login, install OpenSSH Server to enable access to Ubuntu VM over SSH.

$sudo apt-get install openssh-server

Check whether SSH process is running.

$service ssh status

Either note down the IP address of VM from login screen (shown above) or using ifconfig command to be able to SSH into the VM remotely.

$ssh anil@172.16.191.158

You may need to remove SSH key if there is a fingerprint mismatch between the VM and remote client.

$ssh-keygen -R 172.16.191.158

Update Packages

You need to make sure all installed packages are current. Download any package updates and install.

anil@django:~$ sudo apt-get update
[sudo] password for anil: 
Hit http://us.archive.ubuntu.com precise Release.gpg
Get:1 http://us.archive.ubuntu.com precise-updates Release.gpg [198 B]
...

At this point VM is ready for installation of Python Virtualenv, Django, PostgreSQL, NGINX, and Gunicorn.

Python Virtualenv

Virtualenv is Virtual Python Environment builder to create separate Python environments. This enables to keep installations, dependencies, versions and permissions separate for different applications across different virtual environments.

Install python-virtualenv.

anil@django:~$ sudo apt-get install python-virtualenv
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  python-pip python-setuptools
The following NEW packages will be installed:
  python-pip python-setuptools python-virtualenv
...

Now, we need to create a virtual environment for our project (in this case, lendcafe) where we can install Python and Django packages.

anil@django:~$ sudo virtualenv /opt/lendcafe
New python executable in /opt/lendcafe/bin/python
Installing distribute...........................................................done.
Installing pip...............done.

You can name anything you like for your virtualenv.

Django

Django is a Python web framework. It enables rapid development of common web application tasks and adheres to DRY principle (Don't Repeat Yourself).

To install Django, first we need to activate virtualenv.

anil@django:~$ source /opt/lendcafe/bin/activate
(lendcafe)anil@django:~$ 

The activate script modifies shell prompt to show the currently active environment.

Install Django

(lendcafe)anil@django:~$ sudo pip install django
[sudo] password for anil: 
Downloading/unpacking django
  Downloading Django-1.6.2.tar.gz (6.6Mb): 6.6Mb downloaded
  Running setup.py egg_info for package django

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Installing collected packages: django
  Running setup.py install for django
    changing mode of build/scripts-2.7/django-admin.py from 644 to 755

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    changing mode of /usr/local/bin/django-admin.py to 755
Successfully installed django
Cleaning up...

PostgreSQL

PostgreSQL is an open source object-relational database.

Deactivate the virtual environment.

(lendcafe)anil@django:~$ deactivate
anil@django:~$ 

Install Python dependencies for PostgreSQL

anil@django:~$ sudo apt-get install libpq-dev python-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
...

Install PostgreSQL

anil@django:~$ sudo apt-get install postgresql postgresql-contrib
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
...

NGINX

NGINX is an open source HTTP server and reverse proxy. It is known for high performance and low resource utilization. Instead of relying on threads to handle requests, it uses event-driven asynchronous architecture.

Install NGINX

anil@django:~$ sudo apt-get install nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
...

Gunicorn

Gunicorn is a Python WSGI HTTP Server.

Activate virtualenv

anil@django:~$ source /opt/lendcafe/bin/activate
(lendcafe)anil@django:~$ 

Install Gunicorn within virtualenv

(lendcafe)anil@django:~$pip install gunicorn
Downloading/unpacking gunicorn
  Downloading gunicorn-18.0.tar.gz (366Kb): 366Kb downloaded
  Running setup.py egg_info for package gunicorn
  ...

Configuration

PostgreSQL

The default superuser for PostgreSQL is called postgres. We need to login as this user first.

anil@django:~$ sudo su - postgres
[sudo] password for anil: 
postgres@django:~$ 

The shell prompt should now show starting with postgres@....

Create a user. Answer the prompts. I decided to not make this user superuser, allow this new user to create databases or new roles.

postgres@django:~$ createuser -P
Enter name of role to add: anil
Enter password for new role: 
Enter it again: 
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n

Create a new database. I named this database to be lendcafe.

postgres@django:~$ createdb lendcafe

To grant new user access to this database, first access the PostgreSQL interactive terminal and then grant all privileges. Type \q to quit.

postgres@django:~$ psql
psql (9.1.11)
Type "help" for help.
postgres=# GRANT ALL PRIVILEGES ON DATABASE lendcafe TO anil;
GRANT
postgres=# \q
postgres@django:~$ 

Test the new user log in to database. If you are not already logged in the system as new user, you will get error as shown below. Login as new user or use su command.

postgres@django:~$ psql -d lendcafe -U anil
psql: FATAL:  Peer authentication failed for user "anil"
postgres@django:~$ su - anil
Password: 
anil@django:~$ psql -d lendcafe -U anil
psql (9.1.11)
Type "help" for help.
lendcafe=> 
lendcafe=> \q
anil@django:~$ 

Django

To create a Django project, first switch to the virtualenv directory created during installation and activate the virtualenv.

anil@django:~$ cd /opt/lendcafe
anil@django:/opt/lendcafe$ source /opt/lendcafe/bin/activate
(lendcafe)anil@django:/opt/lendcafe$    

Start a new Django project. If you receive permission denied error as shown below, change the ownership of your environment directory. Then start a new Django project again. Check to make sure the project directory was created in virtualenv directory.

(lendcafe)anil@django:/opt/lendcafe$ django-admin.py startproject lcproject
CommandError: [Errno 13] Permission denied: '/opt/lendcafe/lcproject'
(lendcafe)anil@django:/opt/lendcafe$ sudo chown -R anil:anil /opt/lendcafe
[sudo] password for anil: 
(lendcafe)anil@django:/opt/lendcafe$ django-admin.py startproject lcproject
(lendcafe)anil@django:/opt/lendcafe$ ls -la
total 28
drwxr-xr-x 7 anil anil 4096 Feb 21 11:52 .
drwxr-xr-x 3 root root 4096 Feb 20 10:37 ..
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 bin
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 include
drwxrwxr-x 3 anil anil 4096 Feb 21 11:52 lcproject
drwxr-xr-x 3 anil anil 4096 Feb 20 10:37 lib
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 local
(lendcafe)anil@django:/opt/lendcafe$ 

For Django to be able to communicate with PosgreSQL database, we need to install PostgresSQL adapter for the Python Psycopg.

(lendcafe)anil@django:/opt/lendcafe$ pip install psycopg2
Downloading/unpacking psycopg2
  Downloading psycopg2-2.5.2.tar.gz (685Kb): 685Kb downloaded
  Running setup.py egg_info for package psycopg2

Installing collected packages: psycopg2
...
Successfully installed psycopg2
Cleaning up...
(lendcafe)anil@django:/opt/lendcafe$ 

Edit the setting.py file in directory and subdirectory named same as your project name.

(lendcafe)anil@django:/opt/lendcafe$ nano lcproject/lcproject/settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'lendcafe',
        'USER': 'anil',
        'PASSWORD': 'myPassword',
        'HOST': 'localhost',
        'PORT': '',
    }
}

Run the following command to add Django specific database tables and configuration to database. I received the following error message when I tried to run the command.

(lendcafe)anil@django:/opt/lendcafe$ python lcproject/manage.py syncdb
Traceback (most recent call last):
  File "lcproject/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management

The StackOverflow discussion django import error - No module named core.management provides potential solutions to this error.

My issue turned out that I originally installed Django as root user while I was trying to run the above command as different user. When I ran the above command with sudo, I encountered a different error.

(lendcafe)anil@django:/opt/lendcafe/lcproject$ sudo python manage.py syncdb
Traceback (most recent call last):
  File "manage.py", line 11, in <module>
...
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 25, in <module>
raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named psycopg2

At this point, I decided to reinstall Django, this time without sudo.

(lendcafe)anil@django:/$ pip install django
Downloading/unpacking django
  Downloading Django-1.6.2.tar.gz (6.6Mb): 6.6Mb downloaded
  Running setup.py egg_info for package django

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Installing collected packages: django
  Running setup.py install for django
    changing mode of build/scripts-2.7/django-admin.py from 664 to 775

    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    changing mode of /opt/lendcafe/bin/django-admin.py to 775
Successfully installed django
Cleaning up...

Once I reinstalled Django, I was able to successfully add Django specific database tables.

(lendcafe)anil@django:/opt/lendcafe/lcproject$ python manage.py syncdb
Creating tables ...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'anil'): 
Email address: xyz@example.com
Password: 
Password (again): 
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
(lendcafe)anil@django:/opt/lendcafe/lcproject$ 

Gunicorn

Create a gunicorn_config.py file in /opt/lendcafe directory and add following entries in the file.

(lendcafe)anil@django:/opt/lendcafe$ nano gunicorn_config.py

command = '/opt/lendcafe/bin/gunicorn'
pythonpath = '/opt/lendcafe/lcproject'
bind = '127.0.0.1:8001'
workers = 5
user = 'anil'

Use the following command to run the server.

(lendcafe)anil@django:/opt/lendcafe$ /opt/lendcafe/bin/gunicorn -c /opt/lendcafe/gunicorn_config.py lcproject.wsgi
2014-02-21 15:41:13 [2892] [INFO] Starting gunicorn 18.0
2014-02-21 15:41:13 [2892] [INFO] Listening at: http://127.0.0.1:8001 (2892)
2014-02-21 15:41:13 [2892] [INFO] Using worker: sync
2014-02-21 15:41:13 [2897] [INFO] Booting worker with pid: 2897
2014-02-21 15:41:13 [2898] [INFO] Booting worker with pid: 2898
2014-02-21 15:41:13 [2899] [INFO] Booting worker with pid: 2899
2014-02-21 15:41:13 [2900] [INFO] Booting worker with pid: 2900
2014-02-21 15:41:13 [2901] [INFO] Booting worker with pid: 2901

Background the process by using ctrl + z and then typing bg followed by Enter. supervisord and screen can be used to manage Gunicorn start/restart. Refer to How to Install and Manage Supervisor on Ubuntu and Debian VPS for more information.

Enter the IP address and port specified in gunicorn_config.py file in a browser's address bar. If you see the message It worked! Congratulations on your first Django-powered page., you are good to go.

NGINX

Django settings for the project are stored in settings.py file. As NGINX will be used for static files, modify the location of STATIC_URL in /opt/lendcafe/lcproject/lcproject/settings.py file.

(lendcafe)anil@django:/opt/lendcafe/lcproject/lcproject$ nano settings.py

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.6/howto/static-files/

STATIC_URL = '/opt/lendcafe/static/'

Create a subdirectory static in /opt/lendcafe/ directory.

(lendcafe)anil@django:/opt/lendcafe$ mkdir static
(lendcafe)anil@django:/opt/lendcafe$ ls -la
total 36
drwxr-xr-x 8 anil anil 4096 Feb 22 15:26 .
drwxr-xr-x 3 root root 4096 Feb 20 10:37 ..
drwxr-xr-x 2 anil anil 4096 Feb 21 15:28 bin
-rw-rw-r-- 1 anil anil  133 Feb 21 15:41 gunicorn_config.py
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 include
drwxrwxr-x 3 anil anil 4096 Feb 21 11:52 lcproject
drwxr-xr-x 3 anil anil 4096 Feb 20 10:37 lib
drwxr-xr-x 2 anil anil 4096 Feb 20 10:37 local
drwxrwxr-x 2 anil anil 4096 Feb 22 15:26 static
(lendcafe)anil@django:/opt/lendcafe$ 

NGINX is a reverse proxy and HTTP server. There are two blocks, Server-block and Location-blocks in configuration file that we will be primarily working with. The server-block is very similar to virtual host and location-block to URI. There is much more information available in NGINX Beginner's Guide.

Create a new NGINX config file lendcafe in /etc/nginx/sites-available/ directory and enter the following configuration information.

(lendcafe)anil@django:/opt/lendcafe$ sudo nano /etc/nginx/sites-available/lendcafe

server {
    server_name localhost;

    location /static/ {
        alias /opt/lendcafe/static/;
    }

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
    }
}

Create a symbolic link in /etc/nginx/sites-enabled directory to the NGINX configuration file lendcafe.

(lendcafe)anil@django:/etc/nginx/sites-enabled$ sudo ln -s /etc/nginx/sites-available/lendcafe

Delete the symbolic link default.

(lendcafe)anil@django:/etc/nginx/sites-enabled$ sudo rm default

Restart NGINX service to changes to take effect.

(lendcafe)anil@django:/etc/nginx/sites-enabled$ sudo service nginx restart
Restarting nginx: nginx.
(lendcafe)anil@django:/etc/nginx/sites-enabled$ 

VM is setup for Python/Django web development.