Using "drush make" to maintain the source tree for a Drupal site, and avoid killing kittens

Date: Fri Nov 05 2010 Drupal Planet »»»» drush
To avoid killing kittens through hacking Drupal core I've been maintaining a source tree using Mercurial (see: How to use Mercurial to hack Drupal Core without killing kittens). But, I had an inspired idea recently to see how well "drush make" could take care of the job and maybe stop using a private source tree. One of the key required features would be to maintain a set of local patches. I'm pleased to report some positive success and it looks like a useful method for a D6 to D7 upgrade.

The basic idea is to

  • Maintain a "drush make" file describing the source code for the site
  • Execute "drush make" every time you update the site
  • Use a shell script to handle some of the steps which "drush make" does not implement

To get started one generates a "drush make" file:


cd docroot-for-web-site
drush generate-makefile ../domain.make

The "drush make" project page contains some documentation that is almost useful. The generated make file has a PHP-like syntax that should feel familiar.

I am finding it helpful to follow a file and directory naming convention:-

  • domain.com: The docroot-for-web-site directory is named by the domain name of the website
  • domain.make: The make file gets named by removing the last element of the domain name, and appending "make"
  • domain.clone: A shell script to generate a clone of the website using "domain.make"
  • domain.patches: A directory containing a list of local patches

The "domain.make" file will need some modification before you can use it. It's going to start with the following:-


; This file was auto-generated by drush_make
core = 6.x

; Please fill the following out.
projects[pressflow][download][type] = ""; One of get, cvs, git, bzr or svn
projects[pressflow][download][url] = ""; The url of the download
projects[pressflow][type] = "core"

But this is incomplete. First, you must add a line reading "api = 2". Second is the access details for your Drupal core platform files.

As you see from the above I'm using Pressflow and the access method (type and url) is blank. Those values will have to be filled in to use Pressflow. If you're using straight Drupal, however, it's as simple as:


    projects[drupal][version] = 6.15

The working version to use the current version of pressflow is as follows. The URL below is found by starting at the pressflow.org website, going over to the launchpad.net site, and copying the download URL for the latest pressflow release.


projects[pressflow][download][type] = "get"
projects[pressflow][download][url] = "http://launchpad.net/pressflow/6.x/6.19.94/+download/pressflow-6.19.94.tar.gz"
projects[pressflow][type] = "core"

To specify a contributed module from drupal.org one simply adds a line reading


projects[] = "twitter"
projects[] = "oauth"

But sometimes it's more complex. For instance the XML Sitemap module is currently far enough along to warrant use on real sites:


projects[xmlsitemap][version] = 2.x-dev

A big advantage of "drush make" is it can handle downloading and installing third party libraries. With my former method I could set everything up in the source tree and maintain it in the mercurial repository. That is, until a new release of the module came out. Sometimes I found it helpful to completely delete a module from the mercurial repository, and install a fresh version from drupal.org. But if the given module had a third party library I'd have to remember to set it up, again.

With "drush make" one can do something like the following and the third party libraries will always be properly set up:


projects[] = "swftools"

; FlowPlayer Library required by SWFTools to play audio/video files
libraries[flowplayer][download][type] = get
libraries[flowplayer][download][url] = http ://www.opensourceopenminds.com/sites/default/files/releases/flowplayer-package.zip
libraries[flowplayer][destination] = modules/swftools/shared
libraries[flowplayer][directory_name] = flowplayer3
libraries[flowplayer][install_path] = sites/all

; 1PixelOut Library could be required by SWFTools to play audio files
libraries[onepixelout][download][type] = get
libraries[onepixelout][download][url] = http ://wpaudioplayer.com/wp-content/downloads/audio-player-standalone.zip
libraries[onepixelout][destination] = modules/swftools/shared
libraries[onepixelout][directory_name] = 1pixelout
libraries[onepixelout][install_path] = sites/all

You can specify download using an HTTP "get", or from source code repositories like "cvs" or "svn" or "git" (but not mercurial?). The tool can unpack the downloaded tarball or zipfile into a specified location. But it appears to not support moving individual files around after unpacking, so that has to be done as a later step in a shell script.

One might have to check carefully that "drush generate-makefile" successfully listed all the modules. In my case it did not list the "cck" module, but it's relatively easy to add missing modules once found.

I've created a "domain.clone" shell script that hits these points:-

  • "drush make" to set up the source code, followed by any individual shell commands to set up individual files; especially ensure to copy the sites/default/settings.php file
  • clone the contents of sites/default/files into sites/default/files of the newly built directory tree
  • Use the Unix patch command to apply a set of patches (if appropriate)

Drush make does support specifying patch files in the .make file, and those patch files can be retrieved via HTTP. That lets you apply patches from Drupal issue queues, for example. But so far as I can tell the patch in question has to be diff'd within the module directory, but on a locally generated patch it's easier generate a diff based on the whole directory tree.

The domain.clone shell script looks something like this:


#!/bin/sh
drush make domain.make new.domain.com
(cd domain.com/sites/default; tar cf - files) | (cd new.domain.com/sites/default; tar xf -)
cp domain.com/sites/default/settings.php new.domain.com/sites/default

foreach patch in domain.patches/*; do
  (cd new.domain.com; patch -p1) <$patch
done

Once the new.domain.com directory is built you can do one of two things. First is to point a virtual host at that directory and check it out using your web browser. Second is to replace the existing directory tree with the newly built one (a.k.a. "mv domain.com domain.old; mv new.domain.com domain.com"). Of course if the newly built directory tree requires that Drupal's update.php be run, do that (a.k.a. "drush updb").

An additional twist to this is using this to set up a test site. The above script clones the source files plus sites/default/files, but does not modify the $db_url in sites/default/settings.php meaning that the cloned site is running from the same database as the cloned site. To set up a test site there are a couple further steps required to ensure you're using a cloned database:

  • Change $db_url in sites/default/settings.php to point to a newly created database
  • Use mysqldump or similar method to clone the database into the newly created one

This method will ensure the code is the latest as recorded on drupal.org. To update to the latest contrib modules it's enough to re-run the domain.clone script, then do "drush updb". But to update to a newer Drupal release you have to revisit this line:


    projects[drupal][version] = 6.15

Changing that version number will update the downloaded Drupal version.