Dev QA Live

Fen Labalme's picture

The ''DEV -> QA -> LIVE'' Release Process Overview

  • Sites are managed using SVN tags - for more on SVN, see svn-hints
  • Developers work in a personal "sandbox" (see creating-your-sandbox) on their personal desktop or laptop computer.
    • Development of code (e.g. modules) cannot happen directly on the PROJECT vhost as the "w_PROJECT" user does not have write permission to the SVN repositories
    • However, occasionally the TL (Tech Lead) will direct that certain types of database-only configuration changes can happen on the LIVE site.
    • Pre-QA changes on LIVE: some sprints may allow for pre-QA changes directly on LIVE. This is usually when the public and client are not actively using LIVE, and the pre-QA changes are safe and tested. In these cases QA testing still takes place on QA, but configuration is automatically pushed from LIVE to QA. After passing QA, a final deployment is made on the LIVE only if necessary. Engineers should use careful judgement and only add pre-QA changes on LIVE when confident of its safety and stability.
  • Developers use SSH to log into a client vhost (site); to create your SSH key, seeĀ howto-use-ssh.
  • As described in the Directory Structure document, a series of three sites (development testing, quality assurance, and live) are created for each client:
    1. DEV: As developers commit changes in their workspace/sandbox, they may update the PROJECT's DEV site ('SITENAME-dev.civicactions.net'). This is a "staging" area for accumulating updates, potentially from several developers, designers, etc., and integration/function testing can occur on the DEV site;
    2. QA: Once the development tree reaches the required level of stability and the Release Requirements have been met (see general-developer-process), a Release Candidate gets tagged and pulled to the QA site (SITENAME-qa.civicactions.net) for quality-assurance.
    3. LIVE: After passing QA, the LIVE site may be updated by QA's Release Candidate's tag. This is the top-level production site accessed by SITENAME.civicactions.net (or via a custom domain name).
  • See also: Drupal Quality Assurance with Dev QA Live

Update DEV

  • The dev tree can be updated regularly by the developer(s) working on a site and is not guaranteed to be stable
    • It is a good idea to periodically update the DEV files/ and database(s) from LIVE (see process for Update QA below).
  • When a developer runs the svn update command on both the dev tree and their personal workspace, the working files in each should be identical
    • Exception: config files, such as settings.php, will differ
  • To update the SERVER's DEV site, simply
    SANDBOX$ ssh w_${PROJECT}@${SITENAME}.civicactions.net
    SERVER$ cd ~/sites/${SITENAME}-dev.civicactions.net
    SERVER$ svn update
    SERVER$ fixwebperms.sh .
    
  • Single line shortcut for the last two lines: svn up; fixwebperms.sh .
  • Sometimes it helps to run fixwebperms twice, both before and after: fixwebperms.sh .; svn up; fixwebperms.sh .

Updating the DEV database and files/ dir from LIVE

This is a good practice, but not usually necessary with every code push. For instructions, see the "Update QA" section, below.

Cutting a SVN Tag for QA

  • Tag the dev release; from your SANDBOX (ie. from the workspace/project folder), run:
    SANDBOX$/$PROJECT svn update
    SANDBOX$/$PROJECT svn status -u
    
  • This latter command ensures that you don't have any conflicts, non-added or committed files, etc. Make sure you understand why each file listed is reported (such as the configuration files). The command svn help stat offer brief help - see the SVN Book for more detailed information.
    SANDBOX$ svn info | grep URL
    
  • This will tell you the URL in the repository your workspace is from
    • It will look like URL: https://svn.civicactions.net/repos/PROJECT/trunk
  • To create a tag, you copy the repository URL to a new tag URL, say: https://svn.civicactions.net/repos/PROJECT/tags/#.#-#
  • Note that if your username in your workspace is different than your SVN repository username ("USER") you will need to add "USER@" in front of the server name, e.g. https://USER@svn.civicactions.net/repos/PROJECT/trunk
    SANDBOX$ svn list https://svn.civicactions.net/repos/${PROJECT}/tags
    SANDBOX$ svn copy https://svn.civicactions.net/repos/${PROJECT}/trunk https://svn.civicactions.net/repos/${PROJECT}/tags/#.#-# -m "LOG MESSAGE"
    
  • The svn list command above is optional - it simply reports the current tags, useful so that you can name a tag using the same format as others.
  • A note on the tag naming convention: the first digit in the series is the major release, the second is for branches, the third, after the dash is for release candidates, numbering starts at 1 for release candidates. A normal sequence of releases might look like: 1.0-1, 2.0-1, 2.0-2 (second release candidate if first did not pass qa), 3.0-1. See Tag Naming Convention for more.

Update QA

  • At defined times the QA site can be updated with the (perhaps only temporarily) stable DEV that has been tagged (see below)
    • This usually happens when the Release Requirements have been met (see the General Developer Process) but could also happen:
      • At a defined code freeze date/time
      • Daily (perhaps by a cron job at 6:30 AM Pacific every morning)
  • There are three primary parts to updating the QA site: code, database and (of course) Quality Assurance

1. Updating the QA Database from LIVE

  • Use the pushdb command, which:
    1. backs up the "from" database(s) in ~/nobackup
    2. backs up the "to" database(s) in ~/nobackup
    3. deletes all tables in the "to" database
    4. loads the "to" database(s) with the contents of the "from" databases
    5. performs Post Update Configuration if the database is a Drupal database
      1. flush the cache* and settings tables
      2. update the system & files tables
      3. change user email addresses to qa-HASH@civicactions.com where HASH is the MD5 hash of the original email address. These email addresses forward to our ca-qa-test mailing list that you can request to be added to.
      4. change user passwords to civicactions
    SERVER$ pushdb --from live --to qa --oldPath ${SITENAME}.civicactions.net --newPath ${SITENAME}-qa.civicactions.net --all
    
  • If you don't want the emails and passwords changed, simply leave off the --all flag and answer all questions with "yes" manually
    • The --all flag answers "yes" to all questions and includes --who qa-HASH@civicactions.com --passwd civicactions
  • Note that this command is run on the vhost. SSH into the remote server to run this command.
  • If the LIVE site as a LIVESITENAME associated with it (like "example.org") run pushdb again with the --xFix option:
     SERVER$ pushdb -x qa -o ${LIVESITENAME} -n ${SITENAME}-qa.civicactions.net -y

2. Switching to the New Tag

  • Then go to the SERVER's QA site and svn switch to the same tag
    SANDBOX$ ssh w_${PROJECT}@${SITENAME}.civicactions.net
    SERVER$ cd ~/sites/${SITENAME}-qa.civicactions.net/
    SERVER$ svn switch https://svn.civicactions.net/repos/${PROJECT}/tags/#.#-#
    SERVER$ fixwebperms.sh .
    
  • Don't forget to run fixwebperms.sh after (and sometimes before) every operation that touches files used by the site!

3. Updating the QA files/ Directory

  • The QA files/ directory should be updated to include all files found on the LIVE site
    • The method below uses hard links which are fast, safe and lightweight
    SERVER$ cd ~/sites/${SITENAME}-qa.civicactions.net/
    SERVER$ rm -rf files
    SERVER$ cp -al ../${SITENAME}.civicactions.net/files .
    

Post QA Site Update Configuration

     cd ~/public_html/${SITENAME}-qa//sites/${SITENAME}-qa.civicactions.net
     drush cc all
  • You may also want to:
    • Go to /admin/build/modules and Save Configuration`
    • Run: /update.php to catch any update handlers than may be in the new code

Update the LIVE Site

Important: Someone capable of reverting the site/fixing bugs/backing out the change should be available for at least one hour after the change in case bugs are found.

  • When the QA site passes acceptance tests, the LIVE site can be updated from the same tag that the QA site has
  • Be sure the read ChangeManagement before updating the LIVE site.
  • As new code could potentially corrupt the database, be sure to backup the database first.
    SERVER$ pushdb --backup live
    
  • Before updating the LIVE site's code, it's important to ensure that there are no changes that have been made to the LIVE site that were not created through the standard dev->qa->live procedure described here. It's hard to believe, but it's been known to happen ;) Run:
    SERVER$ cd ~w_${PROJECT}/sites/${SITENAME}.civicactions.net/
    SERVER$ svn status -u
    
    • Be sure you understand why each file listed is shown. Ideally, there should be no output other than the configuration files (e.g. settings.php) and perhaps the files/ directory.
  • Finally (before starting your update) record the current tag in use on LIVE:
    SERVER$ svn info
    
  • The update procedure is similar to updating the code for the QA site except the files/ dir and database are already in place
    • Use the same tag that was used to create the QA site, e.g. ##.#-#, as that was what was tested
    SERVER$ cd ~w_${PROJECT}/sites/${SITENAME}.civicactions.net/
    SERVER$ svn switch https://svn.civicactions.net/repos/${PROJECT}/tags/#.#-#
    SERVER$ fixwebperms.sh .
    
  • Inform the PM that the site has been updated (usually best via the PROJECT-Int@lists.civicactions.com list).

Post LIVE Site Update Configuration

  • Go to /admin/build/modules and Save Configuration`
  • Run: /update.php to catch any update handlers than may be in the new code

Quality Assurance

See Quality Assurance for Basic Site Functionality Tests

these tests should always be run whenever the LIVE site is updated

Tags: 

Groups audience: 

- Private group -

Openness: 

Public - accessible to all