Links to know about
Music
The Chiara String Quartet
Chiara Quartet (MySpace)
Greenwood Music Camp
UNL School of Music
PHP
PEAR Installer Manifesto
phpDocumentor
PEAR
phar
docblock
PHP_Parser
PHP_Parser_DocblockParser
PHP_ParserGenerator
PHP_LexerGenerator
PEAR_PackageFileManager
PHP_Archive
Games_Chess
Blogs
Joshua Eichorn
Paul M. Jones
Davey Shafik
Popular Entries
Categories
Syndicate This Blog
Powered by
Sunday, February 11. 2007
Do you develop a website? It is infinitely better to synchronize live and development sites using the PEAR Installer
Recently, pear.php.net silently switched over to a revolutionary new way of maintaing the live website that has been extremely freeing. The same technique is being explored by the government of the state of Iowa and other large-scale sites with mission-critical servers. Before revealing the secret, let's learn about the problem.
One of the most common tasks that we experience as web developers is synchronizing a development web server with a live site. There are many solutions that have been tried before. Just a few:
- develop the code on the live server
- develop on a remote machine, and ftp the files to the site as they change
- develop using version control like CVS, and use a cron job to synchronize the website directly from CVS
- develop on a remote server with the same configuration, and use rsync to synchronize changes
Almost everyone has tried #1. The first time you accidentally erase a file, it will rid you of that habit in a hurry. Most developers use #2, and this has the benefit of having a backup on the development machine. The same principles hold for #3 and #4 but the changes are synced in a controlled way.
However, every single method described above has the potential for immediate and catastrophic failure, even with a backup.
Why?
Websites are not just the code that runs them. Websites consist of interdependency between code, data storage backend (LDAP/database/other), and also the actual hostname itself. Of the 4 methods listed above, not one addresses this issue. When upgrading code on the live server, if there are any database changes, these must be done by hand. Keeping track of links to use (for instance, when redirecting a user to a secure login form, one cannot use a relative href, but must use the full hostname) also requires some footwork. The most common solutions involve two strategies:
- use /etc/hosts to mimic the live server's DNS (alias hostnames to localhost)
- use a static configuration file that is not synchronized to define hostname/path differences
The first idea is good, but also means you will need another alias in order to rsync with the remote site (or a dedicated IP on the remote site). Chances are, if you have a dedicated machine for development, you also have a dedicated IP for the live server, and then this is a good solution for managing hostname differences between the development and live machine. The second solution is far more common, and much less robust. Any file that must be different between the development machine and the remote server is asking for trouble when using ftp or rsync. If the file is excluded from rsync/ftp, when it is erased, the file is gone, and must be stored somewhere else. This quickly introduces unwanted complexity in the maintenance of the website, and another failure path for the site.
The problems only multiply when using external depedencies. As many developers have discovered, it's all fine and good to try to avoid re-inventing the wheel. The problem happens when trying to upgrade an external library. Each library has different requirements, and upgrading often means a mysterious quagmire of erasing and adding files, as well as database changes or other tasks that make it onerous to upgrade. Actually, upgrading isn't so hard, but downgrading, should the upgrade break everything is really hard (Note: this is far more common if the package is not from pear.php.net - PEAR has a strong backwards compatibility policy designed to mitigate this issue).
The ultimate solution, as I outline in chapter 3 of The PEAR Installer Manifesto, is to use the PEAR Installer in conjunction with version control to manage the complexity of synchronizing a live website with the development machine.
A few years back, while designing the next incarnation of the PEAR Installer, I realized that its ability to do file transactions, intrinsic support for dependencies and versioning make it ideal for managing not just PHP libraries and applications, but also for managing complete websites, and not just the PHP files, but all file-related events. There were just a few elements missing: how to synchronize a database? It turns out that the answer is also PEAR-related, and was one of the features integrated in PEAR 1.4.0 and newer.
The code that runs http://pear.php.net is now completely encapsulated within the pearweb package (http://pear.php.net/pearweb). As part of the installation for the pearweb package, a post-installation script is available that initializes the database using MDB2_Schema, a cross-database package that can create or modify a database installation. As time passes, the pearweb package will be divided into several related packages, one for each sub-site within pear.php.net (there is already a subpackage for the installation .phar files, http://pear.php.net/pearweb_phars). This will allow modular development and other tried and true techniques applied to traditional programming environs.
The biggest difference between the old and the new method was exemplified a few weeks back. A regression was introduced in pearweb version 1.1.2 that broke the statistics graph for package downloads. Initially, I wasn't sure what the problem was caused by, so I first reverted to the previous version:
pear upgrade --force pearweb-1.1.1
Once the problem was traced to an extra line in one of the files, pearweb 1.1.3 was released, and the new version was synchronized with:
pear upgrade pearweb-1.1.3
Two lines of shell scripting and the entire website was fixed! In another early instance, an upgrade of the website resulted in the entire site being placed in the wrong directory due to a configuration value change in the PEAR Installer at pear.php.net. In 30 seconds, I was able to fix it with:
pear config-set web_dir /path/to/actual/pearweb
pear upgrade --force pearweb
This is the missing piece of the 4 most common synchronization methods: it's really hard to fix mistakes made. You will still make mistakes when using the PEAR Installer, the difference is that reverting them is a 1-liner and requires no sweat or fear that one will break something else in the process: each release of a package is a known quantity, and it will work the same way as it did last time. The major exception is database upgrades - these require special attention, as once the database changes, reverting to a previous version is no longer possible. However, designing modularly so that database changes only affect a single module within the website can even fix this problem.
For a more comprehensive analysis of the steps needed to maintain a live webserver successfully, like ways of providing a safety buffer when upgrading (redirecting users temporarily) and the nitty-gritty of packaging up a website using the PEAR Installer, buy my book, I promise it will be worth the cover price.
Don't be fooled by the title, this book is not about how to type "pear install Blah." As one of the people who wrote to me said:
Chapter 3 is of most interest to me - going beyond simple libraries to distributing full apps. As I run a development team of 8 managing mission critical sites, I'm trying hard to bring as much good software engineering methodology as I can in. I've taught everyone to use Subversion so my next issue is deployment. I'm obviously familiar now with basic package.xml stuff for libraries but not for full blown apps. I've been working on a small (php-based) app called **** on and off for a while, which automates some aspects of web app deployment (setting up the environment - databases etc., checking stuff out of SVN including deploying bits from PEAR and so on). However now that I have your book it seems that quite a bit of it can be done within PEAR - I had no idea it was that flexible!
The book uses the Chiara Quartet's website as an example, which was a from-scratch rewrite to fit this new model. Unfortunately since writing the chapter, we decided to outsource the website to another person who uses their own model of development. Fortunately, the design is much better than when I was maintaining the site 
. pear.php.net is more interesting, as it demonstrates how easy it is to move an existing site to the new model. All you really need is a package.xml, and possibly a PEAR channel server, although there are other ways to move the files around as well. Incidentally, Chapter 5, which is available for free on the packt website, describes in detail how to set up a channel server.
I really hope you will check out this method - it has made everything easier for me, and in a way represents the culmination of several years of blood, sweat and tears on my part pooling ideas for the PEAR Installer, implementing and regression testing them, and finally writing about them.
Happy PHPing!
pear upgrade --force pearweb-1.1.1
...and:
svn switch http://localhost/svnroot/pearweb/tags/1.1.1
SVN makes it very easy to create lots of tags that are very much like version numbers. If you need to downgrade, just svn switch to an earlier tag and things work out. For external libraries, you can simply include them as part of the repository, and update them when the package is updated, or you can use svn:externals to pin the library to an external SVN repository, possibly a specific revision, that is versioned along with the files.
SVN, of course, can't help you with database upgrades, but that's a knotty issue as you mentioned.
This is all in the spirit of friendly inquiry, because if it turns out that PEAR can serve as a PHP version of SVN/CVS, it will still be immensely helpful for the masses of us who are on webhosts who don't support version control systems.
If you have no external dependencies, it is the same.
However, in this situation:
you are using an external XML-RPC package that has a security vulnerability in version 1.0.2. Your website version 1.1.1 uses XML-RPC 1.0.2, and 1.1.2 uses XML-RPC 1.0.3. You need to temporarily revert to 1.1.1
The difference is huge. When you use svn switch, you are also reverting to the insecure XML-RPC. With pear upgrade pearweb-1.1.1, you are only reverting the code section that needs to be reverted.
Of course, you can also fix this problem by having separate repositories for each dependency, but then on upgrade, you have to remember which repository has been upgraded, and run a series of svn switch commands. Ultimately, this makes absolutely no sense, as svn was not designed to handle remote dependencies, and PEAR was.
So, it could be a huge difference, or none at all, it just depends on the project.
Great question, let's have some more like this!
SVN, actually, can automatically update external dependencies from external repositories using svn:externals (http://svnbook.red-bean.com/en/1.0/ch07s03.html), but you'd have to change the svn property, and then switch to that new revision, to keep the XML-RPC version up to date. It would look like:
1. On a developmental version or using the repobrowser, edit your 1.1.1 tag's svn:external definition to point from the 1.0.2 version to the 1.0.3 version using:
svn propset svn:externals libs/xml-rpc http://localhost/svnroot/pear/xml-rpc/tags/1.0.3
...making a new tag if you want your tags to be immutable.
2. svn switch'ing to that tag, SVN will handle the externals definition and load the new version of XML-RPC due to the changed svn:externals definition.
I must emphasize, however, this workflow makes no sense in the context of PEAR because PEAR doesn't run off SVN! You'd have to set up your own SVN repository to mooch packages off of. Conversely, though, in order to use PEAR's infrastructure, the package you want to use must be served off a PEAR server (I've considered setting one of those up for my own library, HTML Purifier, is it difficult to do?). But I believe PEAR can also piggy-back off of external tools like svn/cvs using post-install scripts, which makes it all the more powerful.
It should get you up and running in no time.
It is interesting to note that you can run a channel server out of a subdirectory, so you do not need a unique hostname, only a unique url. In other words "pear.example.com/subdirectory" is a valid channel name, and would run out of http://pear.example.com/subdirectory
Now the plan is to move also site structure and configs to a similar, PEAR-packaged format with its own custom role to sync it to a database...
This role is being used in production at the University of Nebraska-Lincoln internally, and works well, but it is an experimental concept, so I would be wary of auto-updating role/tasks. It's safest to do this in a post-install script like the PEAR website's script in cvs at pearweb/pearweb.php
However, using a custom task to automatically insert the contents of a file into a database is an interesting idea. It would be good to think about the security implications of doing this, and to make sure that the task only runs on packages from the official midgard channel. This could help to prevent any potential malicious use of the task by unrelated packages. The likelihood of a malicious attack is low, since it can by its nature be easily traced to its source and must be publicly released, but it is still a good idea to cover ones bases.
I'll address those two good points
1) I forgot
. The book demonstrates a technique to solve this issue using replacement tasks and custom configuration variables. This was what I was doing to maintain chiaraquartet.net. The custom file role is in two packages, http://pear.chiaraquartet.net/private/get/Role_Chiara-0.1.0.tgz and Role_Chiara2-0.1.0.tgz. I define 4 configuration variables, address_url, root_url, music_url and calendar_url. On the live server, these were set to addressbook.chiaraquartet.net, www.chiaraquartet.net, music.chiaraquartet.net and calendar.chiaraquartet.net. On my dev server, they were set to localhost/addressbook, localhost, localhost/music, and localhost/calendar.The code itself contained references to the URLs like so:
[a href="@ROOT_URL@/index.php"]Home[/a]
Then, in package.xml:
[file name="whatever.php"]
[tasks:replace from="@ROOT_URL@" to="root_url" type="pear-config" /]
[/file]
(replace [] with angle brackets < and >)
2) The significant and redeeming difference between the configuration file that is used in some setups and a PEAR configuration file is that the custom configuration file must be parsed every single time the server executes, otherwise there is no way to determine hostnames. Thus, if the file disappears, the entire site goes down, and all the links break. If the pear configuration file disappears, the site will continue to work normally, and the PEAR configuration file can be reconstructed quite easily using the pear config-create command, and then individually pear config-set each variable.
In other words, because the PEAR configuration file is never actually used by the website code, but only by the PEAR installer, it is far less mission-critical, and can be reconstructed. Although this is annoying, it doesn't result in heart-pounding sweaty-palmed scramble to fix it in the same way as a corrupted custom configuration file.
Another issue with the replace tasks is, regardless of the strength of this functionality, that you always have to install the website package, even to test only little changes on your development server. You can't run your dev server directly off the actual development code. This is of course true for all PEAR packages and has nothing to do with your article
.
However, the requirement to install in order to test is not onerous. I just maintain a terminal window that is open (or an Alt-F2 windows if in linux but not in X) where I have
pear up -f package.xml
and use this to update the local install. This is a good design practice, as it also verifies you have a valid configuration for the website's package.xml so that when you install on the live site, it actually works.
I'd be inclined to have PHING:
- do an SVN export
- apply configuration settings
- synchronise with the live server
So far I spotted no downsides compared to your solution and the bonus of not having a packaging step. Am I missing anything?
Phing, of course, also relies upon .xml files to do what it does, similar to PEAR, so there really is no difference in terms of packaging. Let me illustrate:
Phing:
- make xml file
- do the phing thing
PEAR:
- make xml file
- do the pear thing
In fact, the channel server is unnecessary, and is only useful for automatic upgrade cron jobs like at pear.php.net. What I did in order to upgrade the chiaraquartet.net site was I had a CVS repository outside the public_html directory, and when the time was ripe for a release, I did these steps:
$ cvs upd -P -d
$ pear package
$ pear upgrade package.xml
Why pear package? Then I have a .tgz sitting around, which makes these steps unnecessary when reverting to version 1.2.3:
$ cvs upd -R RELEASE_1_2_3
$ pear up -f package.xml
Instead, I could just:
$ pear up -f website-1.2.3.tgz
Also, Phing is not designed to handle these contingencies easily:
- revert to a previous version
- update only dependencies
- revert just a broken depedency
From the documentation, and what you've described, it sounds like you would need a separate phing .xml (or at least a separate section) for each of these tasks, all of which are just 1-liners when using PEAR.
So, Phing is a great tool but I wouldn't try to bend it to do what PEAR specializes in. You could, however, use the PEAR Package task to make your package.xml file to get started if you already have a phing build:
http://phing.info/docs/guide/2.2.0/chapters/appendixes/AppendixC-OptionalTasks.html#PearPackageTask
Let me put it another way: Phing can be installed/upgraded using the PEAR Installer. PEAR can be installed/upgraded using the PEAR Installer. Which tool is designed and trusted specifically to handle deployment of remote packages?
I'd like to expand on that and say that it's a good idea to use any package manager to manage your live website. The PEAR package manager is just one choice, and thanks to your efforts a really good choice.
We tend to use distribution native package managers so creating new servers is a snap, clone a fresh OS filesystem image in the SAN, assign it to a blade, boot up, set some machine specific configuration settings, use the distributions native package manager to fetch in the latest package for your server package with all the associated dependencies and we're good to go. Makes setting up new development and test systems a snap.
We're mainly a Python house currently, so we're using Pythons native package manager distutils to create OS packages. In the
todo list is to figure out a good way to automate this kind of thing for PHP. Are you by any chance aware of good tools to create rpm's and deb's from PEAR packages?
BTW. database versioning is also an interesting topic. It has two facets that make it a bit more difficult. One is the distributed versioning aspect where you have to synchronize versiondependencies across machine boundaries. Basically the same issue you get with any kind of RPC service (including webservices). The other problem is that you can't use the regular remove old version, install new version idiom due to the need to migrate data. I'm currently looking at using the database refactoring idiom to version the database. Basically you create scripts that take you from database version a to version b. When upgrading/downgrading your database you find a path through the refactoring scripts to the new version. This can facilitate some good tooling to automate the creation of the refactoring scripts.
The PEAR_Command_Packaging package (http://pear.php.net/PEAR_Command_Packaging) is designed specifically to support creation of .rpm packages based on PEAR packages. The package is designed to allow extension for other packaging formats, but nobody has stepped forward to implement them, implying demand for .rpm is much stronger than demand for .deb. If it would be useful to you to have .deb, I don't think it would be difficult to implement, I happen to be unfamiliar with the format (even though I use Kubuntu), and obviously have devoted my time to other things like maintaining PEAR, PhpDocumentor, etc.
Capistrano is designed mainly to handle single-chunk applications, whereas PEAR is designed to handle applications with many remote dependencies. As such, some tasks are difficult with Capistrano (often requiring new code to be written) such as upgrade/revert just a single dependency (this requires multiple capistrano scripts with internal symlinks for each dependency.
I would say Capistrano would be an interesting way to begin development, but I am curious how it would scale as the application becomes more complex. With PEAR, it is easy to split up an existing application into sub-packages, to add dependencies, and so on. Capistrano may require more tweaking, but time will tell.
Of course, one could always use the PEAR Installer locally to manage dependencies, and then deploy to a live site a complete application using Capistrano, but it should be noted that using a PHP-based tool and a ruby-based tool to do similar and possibly overlapping things is asking for trouble.
Lastly, PEAR runs identically on windows and unix, whereas capistrano is designed explicitly for a unix-based system, so if that matters to you (God help you if you're running a production website on windows!) then PEAR is clearly a better choice.
That's why i find PEAR installer only good for managing packages dependencies and initial project's repository setup when i need to put all external dependencies to some local folder of my working directory and commit them.
But seriously i do really think Capistrano is a better tool for projects deployment. How would one install the same copies of application to several web serves in a cluster with PEAR installer? And let's not forget Capistrano allows one to script each step of deployment very easily and customize it.
Correct me if i'm wrong please and PEAR installer can do all that for me too!
Anyway something similar to Capistrano written with PHP would be super useful.
I just __enjoyed__ your book (two times), and I am really exited about learning to utilize everything the PEAR installer offers. I have an application which can provide information as XML RPC to the end user. However I want to make it easier for the end user to display the information, and therefore I have written some helper files, which I want to distribute as a PEAR package.
Some of the files are library files, some are template files and then some are webfiles.
On page 154 you shortly mention something about web files, but I don't think I follow you completely. When I run the post-install script, I am asked where to put the webfiles. But there is not one place to put the webfiles. I have several development applications, and I cannot put them all in the same library
Can you give some more pointers on how to handle webpages?
I want to make some code available to my users. It is partly backend, templates and standard webfiles. The purpose is to make it really easy to install a standard implementation of the application, but I still want to make it easy to customize the installation and still be able to do safe updates.
Often one would like to customize something. For instance if you have a shopping cart, you might want to add an extra field and some rules, say:
cart.php - the standard implementation is edited to reflect the changes
// customized stuff in the standard implementation
$form->addElement('text', 'new_field', 'New field');
$form->addRule('new_field', 'required');
And you would probably want to change the css-file entirely to reflect the other aspects of your site.
But having customized stuff I cannot just update my package anymore without breaking the customization made by the user. How can one come around this problem?
Another thing - having the postinstall script write a configuration file is probably often something you want to do. Is there anyway to pull out the information from one's old configuration file and use it when running the postinstall script (so one can just push enter and not have to remember the values on updates?)
By the way, thank you for the really great work and the great book. Both are brilliant.
$pfm = new PEAR_PackageFileManager2();
$pfm->setOptions(
// ...
'exceptions' => array(),
// ...
);
// ...
$pfm->addReplacement('setup.php', 'pear-config', '@php_dir@', 'php_dir');
$pfm->addGlobalReplacement('package-info', '@package-version@', 'version');
$postInstall = $pfm->initPostinstallScript('setup.php');
$postInstall->addParamGroup(
'setup',
array(
$postInstall->getParam('webroot', 'Webroot'),
$postInstall->getParam('intraface', 'Private Key'),
),
'Settings'
);
$pfm->addPostInstallTask($postInstall, 'setup.php');
// ...
Links in this article
PEAR Installer Manifesto
Calendar
|
August '08 |
|
||||
| Mon | Tue | Wed | Thu | Fri | Sat | Sun |
| 1 | 2 | 3 | ||||
| 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| 11 | 12 | 13 | 14 | 15 | 16 | 17 |
| 18 | 19 | 20 | 21 | 22 | 23 | 24 |
| 25 | 26 | 27 | 28 | 29 | 30 | 31 |
Quicksearch
My Latest Releases
Top Exits
pecl.php.net (156)
www.php.net (149)
news.php.net (112)
php.net (103)