Create a client site and vhost

Fen Labalme's picture

 

See Instead

GitLab verison

Use this page as a reference for CiviCRM still.

Overview

Generally, when we take on a new project, the project manager will fill create a System Trac ticket with a vhost request form, detailing information about the new project. That ticket will be assigned to Owen first, in order to prioritize and make sure all information is correct, and then it will be assigned to Openwall, who does their part of the setup here: http://civicactions.net/notebook/vhost-service-creation

Openwall will then assign the ticket to the IS group (a.k.a. owen), and the IS Group will do the following steps


See Also


Supporting Services

These basic support services enable the creation and maintenance of the client site and vhost.

Set Some Variables

To make the following scripts easy to use (you can just cut and paste them into your shell windows), set these variables in both your sandbox and on your client vhost (once you SSH into it).

PROJECT=project's "simple" name, basically the vhost username without the "w_"
SITENAME=${PROJECT}
DRUPAL_MAJOR="7"
DRUPAL_VERSION="7.0"
CIVICRM_MAJOR="v4.0"
CIVICRM_VERSION="v4.0.0"

Notes:

  • SITENAME is usually the same as PROJECT but a different variable is used as this is not always the case.
  • You will only need the CIVICRM_* variables if CiviCRM was requested

Create SVN Repository

Create an SVN repository for the project by running the createSvnRepos script on the SVN server:

ssh s_@svn.civicactions.net ./bin/createSvnRepos ${PROJECT}
svn mkdir https://svn.civicactions.net/repos/${PROJECT}/tags -m "tags dir"
svn mkdir https://svn.civicactions.net/repos/${PROJECT}/branches -m "branches dir"

Create Trac Project Wiki

Create a Trac project wiki (for use by the project team to manage the project and code) by running the Trac creation script on your sandbox (now screen enhanced to overcome dropped ssh connections):

ENVPATH="var/clients/${PROJECT}"
PROJECT_FULL_NAME="get Project's full name from vhost request form"
TECH_LEAD="benjamin"
PROJECT_MANAGER="justine"
ssh -t w_trac@trac.civicactions.net screen ./bin/tracnewsite.sh ${ENVPATH} "${PROJECT_FULL_NAME}" "${TECH_LEAD}" "${PROJECT_MANAGER}"

If you need to add permissions for a user from the command line (eg: your user does not have access to the admin tab) you can do:

ssh w_trac@trac.civicactions.net ./bin/trac-admin ${ENVPATH} permission add TRAC_USER_NAME TRAC_ADMIN

Where TRAC_USER_NAME is trac's name for the specific user. For example, this user might be owen or sumit.


Commit Client Code Template to Repository

This step creates a bare directory structure "template" for the client site code which is where all CivicActions development occurs for a client. We assume that you have already created your sandbox.

The following steps must be done from a developer workspace AKA "sandbox".

  • Create the directory structure template in your sandbox and import it into the PROJECT's SVN repository:
    cd ~/workspace
    rm -rf template
    svn export https://svn.civicactions.net/repos/drupal/${DRUPAL_MAJOR}/template
    svn import template https://svn.civicactions.net/repos/${PROJECT}/trunk -m "initial import"
    svn co https://svn.civicactions.net/repos/${PROJECT}/trunk ${SITENAME}.${DRUPAL_MAJOR}
    
  • Update htaccess and settings.inc file to reflect the client's SITENAME, PROJECT and "Allow from" Vhost IP address and commit the results:
    cd ${SITENAME}.${DRUPAL_MAJOR}
    VHOST_IP_ADDR=`host ${SITENAME}.civicactions.net | grep "has address" | cut -d ' ' -f 4`
    echo $VHOST_IP_ADDR
    # if the VHOST_IP_ADDR looks right then continue 
    sed -i -e "s/PROJECT/${PROJECT}/g" htaccess
    sed -i -e "s/VHOST_IP_ADDR/${VHOST_IP_ADDR}/g" htaccess
    sed -i -e "s/SITENAME/${SITENAME}/g" htaccess
    sed -i -e "s/SITENAME/${SITENAME}/g" settings.inc
    svn ci -m "updated with parameters specific to ${PROJECT}"
    
  • If CiviCRM has been requested, copy the civicrm.settings.inc file from the CiviCRM base directory in your sandbox to here:
    svn export https://svn.civicactions.net/repos/civicrm/${CIVICRM_MAJOR}/etc/civicrm.settings.inc
    svn add civicrm.settings.inc
    svn ci -m "adding base CiviCRM settings file"
    
  • (Optional) Add some modules that you know you'll want for the project (note: a properly configured drush will automagically "svn add" the downloaded modules for you):
    cd ~/workspace/${SITENAME}.${DRUPAL_MAJOR}/modules/contrib-stable
    drush dl admin adminrole civicactions environment_indicator sevenup
    cd ../../themes
    drush dl seven
    cd ..
    svn ci -m "basic modules and themes"
    
  • (Optional) Tell SVN to ignore the settings files (that have passwords in them) and files directory (this is best done after the *settings.php files and files/ directory have been created):
    cd ~/workspace/${SITENAME}.${DRUPAL_MAJOR}
    svn propset svn:ignore settings.php .
    svn propset svn:ignore civicrm.settings.php .
    svn propset svn:ignore files .
    svn ci -m "setting ignore props on settings files to protect passwords"
    
  • Note that we no longer cut the first tag, leaving this task to the PROJECT TL. This enables easier update by the TL of the LIVE site with the PROJECT's module set that can then be pushed easily to the DEV and QA sites.

Install Code on the Vhost

These steps will help you setup the development environment on a Vhost. As you'll see, Drupal, CiviCRM (if used) and client site code are installed and maintained in entirely separate directory trees (see also: Directory Structure for a Typical Virtual Host).


Install Client Code on the Vhost

SSH into the client vhost, which if you've installed the sandbox tools can be done from your command line with this simple command:

  cassh.sh ${PROJECT} [ ${SITENAME} ]

...where SITENAME is not required if the same as ${PROJECT}. Or you can "ssh w_PROJECT@SITENAME.civicactions.net" if you prefer. Then set some variables as you did in your sandbox to make it possible to cut and paste from these instructions).

PROJECT=project's "simple" name, basically the vhost username without the "w_"
SITENAME=${PROJECT}
DRUPAL_MAJOR="7"
DRUPAL_VERSION="7.0"
CIVICRM_MAJOR="v4.0"
CIVICRM_VERSION="v4.0.0"

Create the sites/ directory and install the client code:

mkdir sites
cd sites
svn co https://svn.civicactions.net/repos/${PROJECT}/trunk ${SITENAME}.civicactions.net
svn co https://svn.civicactions.net/repos/${PROJECT}/trunk ${SITENAME}-qa.civicactions.net
svn co https://svn.civicactions.net/repos/${PROJECT}/trunk ${SITENAME}-dev.civicactions.net
ln -s ${SITENAME}.civicactions.net default

Install Drupal on the vhost

To install the Drupal code base, run the following commands:

cd ~
mkdir -p drupal/nobackup
cd drupal/nobackup
svn checkout https://svn.civicactions.net/repos/drupal/${DRUPAL_MAJOR}/tags/${DRUPAL_VERSION}

which makes a ~/drupal directory containing a nobackup directory and installs the codebase within that directory. Items contained within a nobackup directory are not included in nightly backups, thus saving backup space and resources. Note: We shouldn't use Backup and Migrate module (in fact, it has broken sites in the past!) - we have nightly backups of our vhosts as part of our hosting package.

Next, run:

cd ~/drupal/nobackup/${DRUPAL_VERSION}
ln -s ~/sites sites
ln -s ~/sites/${SITENAME}-dev.civicactions.net/htaccess .htaccess
cd ~/public_html
for i in ${SITENAME}*; do rm -rf $i; ln -s ../drupal/nobackup/${DRUPAL_VERSION} $i; done

which links to our top level `sites/` directory and the `htaccess` file in the development tree. Our 'best practice' recommends using the `htaccess` in the development tree for rapid updates not tied to live site code releases'. The final line connects the Apache document roots to the Drupal installation.


Install CiviCRM on the vhost (if requested)

To install the CiviCRM code base, run the following commands (note that this is version v3.2.3 specifc - adjust accordingly):

cd ~
mkdir -p civicrm/nobackup
cd civicrm/nobackup
svn checkout https://svn.civicactions.net/repos/civicrm/${CIVICRM_MAJOR}/tags/${CIVICRM_VERSION}
svn export https://svn.civicactions.net/repos/civicrm/${CIVICRM_MAJOR}/etc/settings_location.php
replace PROJECT ${PROJECT} -- settings_location.php
ln -s ~/civicrm/nobackup/settings_location.php ${CIVICRM_VERSION}

The settings_location.php file must be referenced in every version of CiviCRM used as otherwise our symlinking setup confuses CiviCRM.

Next, symlink the CiviCRM code into the Drupal installs:

ln -s ~/civicrm/nobackup/${CIVICRM_VERSION} ~/sites/${SITENAME}-dev.civicactions.net/modules/civicrm
ln -s ~/civicrm/nobackup/${CIVICRM_VERSION} ~/sites/${SITENAME}-qa.civicactions.net/modules/civicrm
ln -s ~/civicrm/nobackup/${CIVICRM_VERSION} ~/sites/${SITENAME}.civicactions.net/modules/civicrm

Edit ~/.php/php.ini to set the memory_limit variable to at least 64MB. Note that 128MB recommended is CiviMail is to be used:

echo "memory_limit = 64M" >> ~/.php/php.ini

Housekeeping

Essential steps needed for site operation.

Setup Basic Authorization (BA) username and password

Run:

cat ~/.htpasswd

which displays the contents of .htpasswd. Be sure not to overwrite an existing password by re-defining one of the usernames in the file, unless that's intentional. Note that w_PROJECT may already be defined for phpmyadmin access.

Now run:

htpasswd ~/.htpasswd BASIC_AUTH_USERNAME

where BASIC_AUTH_USERNAME is the basic authorization username from the vhost-request form. When prompted for the password, enter the BASIC_AUTH_PASSWORD, also from the vhost-request form.

Publish the name/password on the Current Projects page.

Add TL's login name to ~/.pull_keys database.


Fix Permissions

fixwebperms.sh is your friend. Run:

cd ~
mkdir nobackup
fixwebperms.sh nobackup
fixwebperms.sh public_html
fixwebperms.sh drupal
fixwebperms.sh sites
fixwebperms.sh civicrm

Engage!

If D7, check php version! might need to do this, example on che anyway:

cd ~/bin
ls -l
rm -fv php
ln -s /opt/php/5.3.6-owlx2/bin/php-cgi fastcgi
ln -s /opt/php/5.3.6-owlx2/bin/php
ln -s /opt/php/5.3.6-owlx2/bin/php-cgi
ln -s /opt/php/5.3.6-owlx2/bin/php-cli
ls -l
PATH=$HOME/bin:$PATH
#   vim ~/.bash_profile   # Edit the file to make the same PATH line as above
php -v   # Should be 5.3.x

and edit ~/.

Make the sites come alive. Important note: if the site is on zinn-web, see comment below.

Initialize the LIVE Site

  • Set up the default settings.php file in preparation for the automatic install:
    cd ~/sites/${SITENAME}.civicactions.net
    cp ~/public_html/${SITENAME}/sites.orig/default/default.settings.php default.settings.php
    cp default.settings.php settings.php
    chmod 775 .
    chmod 664 settings.php
    
  • Optionally use drush site-install:
    PASSWORD=`grep password ~/.my.cnf | cut -d = -f 2`
    PORT=`grep port ~/.my.cnf | cut -d = -f 2`
    MAIL=${PROJECT}-int@lists.civicactions.net
    
    cd ~/public_html/${SITENAME}/sites/${SITENAME}.civicactions.net
    drush si --db-url=mysql://w_${PROJECT}:${PASSWORD}@local-db:${PORT}/live_drupal \
    --account-name=admin --account-pass=${PASSWORD} --account-mail=${MAIL} \
    --site-mail=${MAIL} --sites-subdir=${SITENAME}.civicactions.net
    
    Load the site and check it before continuing.
  • If not using drush si: In a web browser, visit: http://SITENAME.civicactions.net/
  • Follow the instructions entering the database info (which can be found in ~/.my.cnf) and Drupal should install successfully.
    • Database name: live_drupal
    • Database username: w_PROJECT (e.g. the vhost user name)
    • Database password: (the password from the ~/.my.cnf file)
    • Database host: local-db
    • Database port: (the port from the ~/.my.cnf file)
    • Table prefix: (leave blank)
  • Create the UID=1 "admin" account with:
    • Password: (the "password" in ~/.my.cnf)
    • Email: PROJECT-int@lists.civicactions.net (the Project's internal mailing list)
  • To save time and prevent duplication of effort, consider enabling and configuring some modules before the LIVE database is copied to QA and DEV, e.g.:
    cd ~/public_html/${SITENAME}/sites/${SITENAME}.civicactions.net
    drush en admin adminrole environment_indicator sevenup openid
    
    (Then configure.)

Create CivicActions-style settings.php

Save the settings.php created by the install procedure and create a "CivicActions-style" settings.php file with the appropriate content below for your Drupal version.

cd ~/sites/${SITENAME}.civicactions.net
chmod 775 .
rm default.settings.php
mv settings.php settings.php.orig

Create CivicActions-style settings.php for Drupal 6



Create CivicActions-style settings.php for Drupal 7

 
  array (
    'default' => 
    array (
      'database' => 'live_drupal',
      'username' => 'w_PROJECT',
      'password' => 'PASSWORD',
      'host' => 'local-db',
      'port' => 'PORT',
      'driver' => 'mysql',
      'prefix' => '',
    ),
  ),
);
$drupal_hash_salt = 'HASHSALT';

// These two lines should be commented out on the LIVE site:
// $conf['file_public_path'] = conf_path() .'/files';
// $update_free_access = TRUE;

// See http://drupal.org/documentation/modules/file if needed
// $conf['file_private_path'] = conf_path() .'/files/private';

Update stock settings.php file with live values

PASSWORD=`cat ~/.my.cnf | grep password | cut -d = -f 2`
PORT=`cat ~/.my.cnf | grep port | cut -d = -f 2`
HASHSALT=`egrep '^.drupal_hash_salt' settings.php.orig | cut -d \' -f 2`
replace PROJECT ${PROJECT} PASSWORD ${PASSWORD} PORT ${PORT} HASHSALT ${HASHSALT} -- settings.php

The default.settings.php file is needed for initializing the site (this can be removed later).

Add environment_indicator settings to settings.php (optional)

/**
 * environment_indicator: uncomment the lines relevant to the local instance
 */
// $conf['environment_indicator_text'] = 'LIVE SITE';
// $conf['environment_indicator_color'] = 'teal';

// $conf['environment_indicator_text'] = 'QA (Testing) SITE';
// $conf['environment_indicator_color'] = 'blue';

// $conf['environment_indicator_text'] = 'DEVelopment SITE';
// $conf['environment_indicator_color'] = 'dark-red';

// $conf['environment_indicator_text'] = 'LOCAL SANDBOX';
// $conf['environment_indicator_color'] = 'green';

Enable CiviCRM (if installed)

Logged in as the admin user, in a web browser, visit:

http://SITENAME.civicactions.net/sites/SITENAME.civicactions.net/modules/civicrm/install/index.php
  • Enter the CiviCRM Database Settings:
    • MySQL server: local-db:PORT (replace PORT wih the port from the ~/.my.cnf file)
    • MySQL username: w_PROJECT (e.g. the vhost user name)
    • MySQL password: (the password from the ~/.my.cnf file)
    • MySQL database: live_civicrm
  • Enter the Drupal Database Settings:
    • MySQL server: local-db:PORT (replace PORT wih the port from the ~/.my.cnf file)
    • MySQL username: w_PROJECT (e.g. the vhost user name)
    • MySQL password: (the password from the ~/.my.cnf file)
    • MySQL database: live_drupal
  • Re-check requirements, and install
  • Note: if CiviMail operation is desired, see CiviMail Configuration.

Create CivicActions-style civicrm.settings.php

Save the civicrm.settings.php created by the install procedure...

cd ~/sites/${SITENAME}.civicactions.net
chmod 775 .
mv civicrm.settings.php civicrm.settings.php.orig

...and create a "CivicActions-style" civicrm.settings.php file with the following content:



Then modify the template above with local values:

PASSWORD=`cat ~/.my.cnf | grep password | cut -d = -f 2`
PORT=`cat ~/.my.cnf | grep port | cut -d = -f 2`
SITEKEY=`php-cli -r 'echo md5(uniqid(rand(), true));'`
replace PROJECT ${PROJECT} PASSWORD ${PASSWORD} PORT ${PORT} SITEKEY ${SITEKEY} -- civicrm.settings.php

Copy LIVE settings files to QA and DEV Sites

Copy the settings files and files directory into the QA and DEV directories:

cd ~/sites/${SITENAME}.civicactions.net
cp *.php ../${SITENAME}-qa.civicactions.net
cp *.php ../${SITENAME}-dev.civicactions.net
replace live_ qa_ -- ../${SITENAME}-qa.civicactions.net/*.php
replace live_ dev_ -- ../${SITENAME}-dev.civicactions.net/*.php
replace '// $conf' '$conf' -- ../${SITENAME}-qa.civicactions.net/settings.php
replace '// $conf' '$conf' -- ../${SITENAME}-dev.civicactions.net/settings.php

Copy LIVE Database and Files to QA and DEV Sites

This will copy the LIVE Drupal (and CiviCRM, if installed) database, fixing some tables, sanitizing email addresses and changing all passwords to 'civicactions':

cd ~/sites/${SITENAME}.civicactions.net
cp -al files ../${SITENAME}-qa.civicactions.net
cp -al files ../${SITENAME}-dev.civicactions.net
pushdb --from live --to qa --oldPath ${SITENAME}.civicactions.net --newPath ${SITENAME}-qa.civicactions.net --all

IMPORTANT NOTE: Now, wait a minute. This is because pushdb creates a backup database with a timestamp, and because of this, pushdb can only be run once a minute. (Fen: fix this!)

The sleep command in the following line will do the waiting automatically for you

sleep 60 && pushdb --from live --to dev --oldPath ${SITENAME}.civicactions.net --newPath ${SITENAME}-dev.civicactions.net --all

Configure CiviCRM for QA and DEV Sites (if installed)

Configure the CiviCRM database for operation in its new locations:

for QADEV in qa dev; do
cd ~/public_html/${SITENAME}-${QADEV}/sites/${SITENAME}-${QADEV}.civicactions.net
drush civicrm-update-cfg -y
drush civicrm-cache-clear -y
drush cache-clear all
rm -rf files/civicrm/templates_c/*
rm -f files/upload/Config.IDS.ini
done

Final Steps

Only once the sites are ready to rock should cron be turned on.

Setup cron

Drupal comes with a `cron.php` file that is meant to be regularly invoked (say, every fifteen minutes) that in turn invokes any cron handlers of installed modules, creating serch indexes, gathering statistics, etc.

Run:

export EDITOR=/usr/bin/nano
crontab -e

and insert the following lines replacing SITENAME as appropriate:

MAILTO=cron-notify@civicactions.com,SITENAME-int@lists.civicactions.net
PATH=/home/clients/websites/w_project/bin:/bin:/usr/bin:/usr/local/bin

*/15 * * * * cd public_html/SITENAME/sites/SITENAME.civicactions.net && COLUMNS=80 drush -q cron
11 */4 * * * cd public_html/SITENAME-qa/sites/SITENAME-qa.civicactions.net && COLUMNS=80 drush -q cron
37 */8 * * * cd public_html/SITENAME-dev/sites/SITENAME-dev.civicactions.net && COLUMNS=80 drush -q cron

The `MAILTO` line defines email addresses which should receive cron error reports.

Note that in the example above, the LIVE site's cron.php runs every 15 minutes while the DEVelopment and QA instances only run every 4 and 8 hours respectively and at staggered times. In most cases, only the LIVE site needs the automated cron. The crons for DEVelopment and QA are set up for the sake of initial development, but should eventually be disabled. Finally, note that "PATH=/bin:/usr/bin:/usr/local/bin" is included so that the full path to drush does not need to be specified in each call.


Standard environment_indicator settings

Update (edit) the settings.php with the appropriate environment indicator overrides.

Tags: 

Groups audience: 

Openness: 

Public - accessible to all