PEAR

It's been a while since I reported on any of my PHP activities, and this update is long overdue. I must apologize to Davey for not doing my 7 things, and to all of the facebook people for whom I didn't write 25 things. I am somehow not at all motivated to do this...

In any case, now that work on ext/phar has shifted primarily to maintenance mode, and namespaces are finally ancient history, I've shifted all of my coding energy to getting Pyrus, PEAR's next-generation installer, ready to ship. Pyrus is basically the PEAR installer completely redesigned for PHP 5.3, and it takes advantage of as many of PHP's new bells and whistles as makes sense. Some of the exciting features:

• Full support for .tar, .tgz, .tbz, .zip, and .phar archive formats out of the box, complete with signing via md5, sha1, sha256, sha512, or asymmetric true signature via openssl signature (all from ext/phar, which is built into PHP 5.3)
• primary registry is SQLite3-based, which is much more robust, and can be relocated very easily, plus support for XML file-based registry as well as PEAR 1.x's serialize-based registry
• redundant registries means ability to reconstruct a corrupted install
• configuration is stored with the installed files, no more hunting down errant config files, user preferences are stored in home directory or equivalent
• for developers: easy generation of package.xml straight from source code checkout
• also for developer: easy generation of channel REST files directly from release tarball, no need to install a package (like the old Chiara_PEAR_Server) or even run PHP on the channel web server.

There are many other great features for users who would like to extend Pyrus, including a seamless API for accessing package.xml which is identical regardless of whether the package.xml is from an .xml file, an archive, or an installed package's registry.

Of course, all of these features only matter if they work, and to move along that path, I've been spending my time unit testing up a storm. Using the run-tests command of the old PEAR installer, I'm writing and refining tests. With the addition of xdebug code coverage (pear run-tests -rx does the trick), it is possible to see where stuff is being executed. Unfortunately, I don't like the existing reporting tools (call me a snob), so I whipped up a very simple coverage analyzer and popped it into the subversion where all of PEAR2 is stored.

When I say simple, I mean here is the script for generating the source (svn link):

Highlighting is a little wonky because this is PHP 5.3+ code, but my blog is running 5.2.

The output is at http://pear.php.net/~greg/coverage/. This is a dumbed-down version of gcov.php.net, which I found extraordinarily helpful when writing the unit tests for ext/phar, but unlike gcov, there are some fantastically helpful features. The short list:

1. summary of all executed files and their coverage
2. project coverage (lines executed, lines covered, and the percentage of covered code)
3. annotated source showing the lines covered with hyperlinks to lists of tests that cover each line
4. this one excites me: per-test code coverage.

For instance, this report shows the tests that execute line 660 of PackageFile/v2.php, next this report shows the files that the PackageFile_v2/basic/file.phpt test actually executes with their proportional coverage, and finally this report shows

the lines in PackageFile/v2.php that are executed by the file.phpt test.

This last part is extremely useful to me, I have already used it to determine when a test is not executing the lines I expect it should be, and to fix problems such as using isset() instead of array_key_exists(), which would otherwise be very difficult to track down.

As you can also see, only 30% of the source to Pyrus is covered by the current tests, and so there is a long ways to go to full coverage. If any of this work interests you, feel free to contribute, PEAR2 has a much lower barrier to entry than PEAR, and as you can see, we need all the help we can get to bring this baby off the ground.

Why the tri-partite title?  I've been silent for a while on the blog, but the title sums up my life for the past several months.  What have I been doing?  Aside from some crazy music work learning a blisteringly difficult Elliot Carter String Quartet No. 4 as well as Steve Reich's Different Trains, Bartok's 6th Quartet, bringing back Korngold 3rd Quartet, learning Carter's Figment for solo cello, and performing Bach's 5th Suite for Unaccompanied Cello for the first time, I've also been pretty active in coding.  I acquired a new Macbook Pro with VMWare Fusion running Windows XP and Ubuntu, and it has made development super-easy.  I also put in a significant effort on the phone getting Barack his electoral vote in the Omaha area, which seems to have paid off.  So, here's the long version of what I've been doing.

Music and Macs

first the music

The past month has seen my time consumed learning Elliot Carter's 4th String Quartet for the 100th birthday thing in Boston.  Although I have not learned the others, a few veterans have reported that it is the most difficult.  I certainly found it to be the most difficult thing I've ever learned by a long shot.  It took me almost 15 hours of work to be able to play the first page (only 15 measures of music) at 3/4 of the printed tempo, and the piece is 35 pages in the cello part.  To contrast, page 35 of the 10 Celebrated Mozart Quartet cello part is the third movement of 6th string quartet in the book.  Carter 4th's cello part is as long as 6 Mozart quartet cello parts.  However, it is shorter than some of the 4 page Mozart quartets, because it is frickin fast and there are lots and lots of notes.  It is the most holy s*** hard piece I've ever learned, and for a while, I was scared it just wasn't going to be possible.

In addition, one of Carter's big compositional inventions is the metric modulation in which a tempo change has a distinct rhythmic relationship.  He uses some pretty normal ones in the first movement (the 16th note becomes the new triplet), but the other movements have fun ones such as the 16th note becoming the new septuplet.  Because most of my part is written in septuplets, this can be difficult to do.  Speeding up septuplets so that you fit 7 into what used to be the space of 6 septuplets is not an easy thing to learn at all.

...then the mac

...unless you have a metronome that can do the metric modulation.  We had been using a web-based metronome to plug into our studio's stereo to rehearse, because it had a more pleasing beat than any of our metronomes have, and it occurred to me that I could probably pretty easily write a metronome that is capable of doing metric modulations on my new Macbook Pro.

So, I took it upon myself to learn Objective-C and the Cocoa framework in Xcode to write a metronome.  It took me just over a day to learn Objective-C; it's very easy for a C coder to pick up, and the syntax is pretty simple.  Learning Cocoa fully took quite a bit longer, but after 3 days I had a working metronome that could do metric modulations (http://chiaraquartet.net/ChiaraMetronome1.0.0.dmg).  It had several flaws, not the least of which was the inability to insert a measure group, no way to save your work, and some odd behavior on the first beat (sometimes it is late and fast).

After a few more days of digging, I had figured out most of the quirks (and discovered bindings and Core Data, which simplified things) and had the interface greatly simplifed (http://chiaraquartet.net/ChiaraMetronome1.2.2.dmg).  I added the ability to start at any measure in the piece, and to have lead-in measures (which allow you to start the metronome, sit down, find the beat, and then start playing).

Then, I discovered document-based Core Data, which allowed saving files, resulting in the next generation (http://chiaraquartet.net/ChiaraMetronome2.0.0.dmg).

Finally, not to leave an unturned stone, I added the final features, the ability to specify whether a beat is accented, beated or silent, which allows implementation of uneven compound meters (like 5/8), and the ability to accelerate or decelerate tempo, and we have the final version http://chiaraquartet.net/ChiaraMetronome3.0.0.dmg.  This version contains 3 sample metronomes, the complete Elliot Carter 4th quartet we used to practice the tempo changes, a sample demonstrating accelerando/ritardando, and a sample demonstrating how the metronome can be used to practice complex polyrhythms like 6 against 7.

In my potentially humble opinion, the metronome is my crowning glory as a programmer, it is incredible for learning complex tempo relationships for composers like Carter and Bartok who call for it.  It's also important to note that this metronome should NEVER be used for learning music by composers such as Brahms or Mozart.  If I hear of anyone using it for that, I will hunt you down and make you listen to Beethoven 9 with your eyelids peeled back watching Teletubbies.

Short feature list:

• free, released as a beta with no implied license or warranty.  I still own the copyright, so you can't use it in your own programming projects or sell it without infringing on the copyright.  You can use it for your own practicing, and I will eventually release the source under an open-source friendly license once I have time to research the options.
• Please send me any metronome files you create, I'll start a public repository that people can download them from
• requires Mac OS X Leopard (version 10.5.0) or newer
• universal binary
• relative tempo slider can be used to practice under tempo and click up the metronome, preserving tempo relationships at tempo changes
• ability to program in endless tempo (0 bars) or a limited number of bars of a tempo
• ability to do compound meters, both even and uneven
• ability to do accelerando/ritardando, including ones that cross tempo group boundaries using the tempo calculator
• ability to start and end accelerando/ritardando mid-bar.  No support for mid-beat start/stop, but this can be simulated by doubling the number of beats and making the mid-beats silent.  For example: instead of 4 beats at 60 bpm, choose 8 beats, and make 2,4,6 and 8 silent, with double the tempo, 120 bpm, then start the accel on beat 4 to start on the and of 2.
• complete undo/redo support
• ability to split up a tempo group, insert a new tempo group, or copy a set of tempo groups and paste them (copy button)
• ability to play only a limited set of tempo groups, simply shift-click or command-click the tempo groups.  This can be used, for instance, to skip a repeat in practicing.
• ability to accent all downbeats with a simple click.
• start at any bar in the piece.
• lead-in bars to help prepare tempo prior to playing.

PEAR

Today, I upgrade pear.chiaraquartet.net to the latest version, 0.19.0, which Brett Bieber has been working hard to perfect.  The look is a bit cleaner, and I also replaced the defunct Crtx_Frontend_Chiara (or whatever it was called) with the newly minted Chiara_Pear_Server_Frontend_Web which is a direct port over of the old frontend with bugfixes and visual tweaks.

PEAR and phar

A high priority for PEAR is minting its next-generation installer, Pyrus.  Pyrus is based on PHP 5.3+, and thus I expended a tremendous amount of energy on the new phar extension,which will be enabled by default in PHP 5.3.  The phar extension provides several must-have features for Pyrus:

1. ability to run out-of-the-box applications without installing or extracting them (a la Java's jar) in zip, tar or native phar file format
2. ability to create data archives in tar or zip format and convert between them
3. ability to automatically handle oddities of running an application inside an uncompressed file
4. web-based applications without extraction

I'm very pleased this will be built into PHP 5.3, it cuts out oodles of code from Pyrus, and will significantly affect both its performance and memory footprint.

PEAR and namespaces. Oh MY!

Also important for PEAR2 is making it easier to work with the names, so I've been fighting in the namespace wars.  For a while there, it was looking like namespaces would go the way they had in the PHP 5 beta cycle: to the trash can.  With a little bit of work, we found a solution no one is 100% happy with, but having tried out several possibilities, I must say the solution we chose is something we can be proud of.  There have been many complainers, but the irony I have seen is that most assert they never planned to use namespaces in the first place, which begs the question: why flap your trap if you have no plan to use the thing?  In any case, you know where I stand on the issue.  I wrote the new manual as well as the first draft of the patch that made it possible to end the bickering on internals.  A big thanks to Dmitry Stogov and Stanislav Malyshev for their patience, and especially persistence in working out the solution.  Also thanks to many others who fielded opinions on-list, on IRC, and through private email.  Anyone who contributed a real concern rationally and sometimes forcefully deserves credit for the final implementation, and time will prove that it is a very good one for those who need it.

I am, of course, referring to the now infamous decision to use the backslash (\) as the namespace separator in PHP's implementation of namespaces.  I won't go into the rationale, except to say that PHP, unlike other languages, implements a very loose autoload implementation, which makes it impossible to do the most common solution to resolving a name conflict between a static method and a namespaced function - disallow that.  In other words, it's not possible to detect the conflict unless autoload is called on every potential namespaced function call, which introduces a potentially exponential hidden performance loss.  As Stanislav so eloquently put it, name conflicts have not caused major websites to go down, but performance issues have.  Thus, anything that calls autoload where it may be unnecessary is an unacceptable solution, and the only way to safely handle the ambiguity is to eliminate it.

That is what using the backslash does, because the syntax is different between a namespaced function call and a static method.  Complain or ridicule if you'd like, the rest of us will simply shut up and code.

This decision has several other benefits too, including the ability to audit someone else's code and figure out what you should grep for, a namespace or a class name.  This also translates into better autocomplete in future IDEs, and so on.

Wrap up

So, it's been a long blog post, but a lot has happened in the past several months, and it's time to get cracking on the future.  More will follow as time allows.

For those of you who were wondering where pear.chiaraquartet.net went, we suffered the same fate as Josh Eichorn's blog and other sites, but all has been restored.

Hello all.  It's been a very busy time for me, what with suddenly having a full gaggle of new cello students to teach at UNL as well as a fully loaded performance schedule for the quartet, but I have been finding time to code.  My priority has been the push towards PEAR2, which means that in addition to attempting to mediate the discussions of what should change from PEAR to PEAR2, I'm churning out code to turn its installer, Pyrus, from vaporware into actual code.

Along these lines, there has been significant progress.  The code has been developed on PHP 5.2, but will most likely target PHP 5.3 with the introduction of namespaces.  Pyrus completely revamps the way things are done.  By using SPL as well as extensions like XMLReader and XMLWriter, Pyrus manages to pack in far more features inside the code while literally cutting thousands of lines of code.  The biggest loser has been the package.xml handling code.  More than 2500 lines of code have gone byebye, but the API for reading from and creating package.xml is now as intuitive or more intuitive than simplexml.

In addition, using interfaces, a few design patterns and some clever iterators, adding support for a customized packaging solution is a matter of creating a class with a few lines of code that accepts file contents and where to save them.  Pyrus does the hard work of constructing the relative path of files within the archive and pre-processing their contents using file tasks.

The code is simple enough that I wrote a simple channel server with no dependencies except for Pyrus in less than 6 hours (just finished the initial work when my battery ran out today) that allows small channels to work without needing a database or even PHP on the channel server, as all the REST files and releases can be generated and then uploaded from a dev server.

In short, it's an exciting time to be involved in PEAR.  If you've been ignoring PEAR, perhaps it is time to take another look?

Just a quick note, I tried out the Pixy XSS and SQLI Scanner (http://pixybox.seclab.tuwien.ac.at/pixy/index.php) on a few simple PEAR files.  On the first, I got a java exception, on the second it was unable to resolve the simplest of includes (no ability to resolve include_path).

In short, the thing is useless for anything written using PEAR.  Fun!

It's been a while since my last blog post, I've found myself with more limited free time than usual and have thus had to focus on the really important things.  I was recently elected president of PEAR (I ran unopposed, so it's a durn good thing I was elected), and this has resulted in a flurry of activity.  A lot of energy is going into revamping the PEAR website, http://pear.php.net.  At this stage, most of the changes are not visible to the naked eye, as we are writing unit tests and refactoring gradually.  However, one change will be highly visible.

I'm excited to announce the first two mirrors of pear.php.net are now actively mirroring the installer REST files and actual .tgz files of package releases.  They are http://us.pear.php.net (provided by Joshua Eichorn and bluga.net) and http://de.pear.php.net (provided by Christian Weiske).  The option exists at a future date of mirroring the entire website, but this will not be possible without further changes to the infrastructure.

To try out one of the new mirrors, simply run:

pear config-set preferred_mirror us.pear.php.net

or

pear config-set preferred_mirror de.pear.php.net

Then, continue using the installer normally.  Users in Europe may notice a considerable performance increase by using de.pear.php.net.  Ain't progress great?

It's been a while since my last blog post, I've found myself with more limited free time than usual and have thus had to focus on the really important things. I was recently elected president of PEAR (I ran unopposed, so it's a durn good thing I was elected), and this has resulted in a flurry of activity. A lot of energy is going into revamping the PEAR website, http://pear.php.net. At this stage, most of the changes are not visible to the naked eye, as we are writing unit tests and refactoring gradually. However, one change will be highly visible.

I'm excited to announce the first two mirrors of pear.php.net are now actively mirroring the installer REST files and actual .tgz files of package releases. They are http://us.pear.php.net (provided by Joshua Eichorn and bluga.net) and http://de.pear.php.net (provided by Christian Weiske). The option exists at a future date of mirroring the entire website, but this will not be possible without further changes to the infrastructure.

To try out one of the new mirrors, simply run:

pear config-set preferred_mirror us.pear.php.net

or

pear config-set preferred_mirror de.pear.php.net

Then, continue using the installer normally. Users in Europe may notice a considerable performance increase by using de.pear.php.net. Ain't progress great?

Yes. If you have been living on a desert island and haven't already done so, it's time to take another look at http://pear.php.net to see what is up.

Newly elected PEAR Group member Josh Eichorn posted a blog entry, "How would you improve PEAR" recently. I was impressed with the response, it seems many people outside of PEAR are monitoring it and have thought about how to make it better. However, I was also not so impressed with the poor job we've done letting people know about the newest improvements to PEAR. In my comments, I listed as many as I could think of, but Josh pointed out that I would do well to post these comments in a more public setting, so here goes.

For those who are wondering, most of the suggestions made in the comments to Josh's blog entry are in fact already implemented or in the planning stages. There is really only one that is not:

1. SVN. CVS is here to stay, as we are dependent on cvs.php.net. HOWEVER, someone willing to donate dedicated hardware would pique our interest in having optional subversion hosting.

Things that are implemented already:

1. pear installer is capable of modifying php.ini, but it does require a binary package to be created with the dll.
2. PEAR_Exception is the error/exception base class.
3. Mentoring. The new constitution (http://pear.php.net/manual/en/constitution.php) requires mentoring of new developers.
4. package tags. This was recently implemented, in anticipation of the new collectives, and other useful purposes.
5. deprecated/unmaintained packages have been supported for a while now (see http://pear.php.net/DB for an example)
6. lightweight packages. See http://pear.php.net/MDB2 for an example. All of the database drivers are decoupled into subpackages. Structures_DataGrid is the most complex example of this de-coupling. The technical capability was not present before PEAR version 1.4.9 to handle this properly, and so most packages have not yet implemented this functionality.

The social issues are a real problem. I plan to make this my primary purpose as the PEAR president, separate from my work as a developer. We'll see how it goes, but I see a strong momentum shift in the makings here, which has been my dream for several years. It's exciting to see it moving forward so dramatically. I hope that more developers who see the potential and the things that can happen will join, it is only through the force of the will of people who believe in things like civility that PEAR will shift its culture to one that encourages innovation as well as stability.

One thing you can currently depend upon in PEAR that will never be changing is that packages marked stable will always work in a similar way, such that you won't find applications based on them suddenly breaking on upgrade. Innovation can always be provided without destroying the work of your users. Stay tuned for details of how things will be changing.

One of the paths that PEAR is actively recruiting developers for is converting a few key packages into new PHP5+ based code  (this means you). For example, HTTP_Request is a commonly used package that needs to be re-coded for PHP 5. We need a good database abstraction layer for PHP 5, and the PEAR Installer is almost ready for its next incarnation (codename Pyrus), details are on the roadmap (yes, another new feature of PEAR) at http://pear.php.net/bugs/roadmap.php?package=PEAR for version 2.0.0, and code in CVS at http://cvs.php.net/viewvc.cgi/pear-core/PEAR2. If you see a feature that you would like to implement, grab yourself a new account, assign it to yourself and upload a patch (yes, another new feature of PEAR).  If you do not already have an account, head on over to http://pear.php.net/account-request-vote.php.  This form can be used to get an account for voting in general elections as well as opening and commenting on bugs or uploading patches (yes, another new feature of PEAR).

It's a good time to get involved with PEAR, there's a lot of renewed vigor, and no ceiling in sight.

OK, today's release party wins.  I've just uploaded the 8th package for release, and it's a very significant day of releases.

First, the PEAR website now supports patch tracking in its bug tracker, as well as the ability to auto-generate a package.xml based on the roadmap, complete with release notes based on closed bugs.  This is pretty exciting stuff.  3 of the releases were devoted to the website pearweb package.

Next, I released PHP_ParserGenerator version 0.1.5, which fixes a bug and adds more display output

Finally, I just released the first full-featured parser for PHP, PHP_Parser, and its companion PHP_Parser_DocblockParser.  These two packages are the next incarnation of the once-defunct PHP_Parser package.  Based on a PHP_ParserGenerator-based grammar, the packages can parse any valid PHP 4, 5 or 6 (PHP 5.1.0+ is required for it to run, however).  Needless to say, I have phpDocumentor in mind for immediate usage.  The Docblock parser uses the docblock PECL extension to do its magic.  Full regression tests are available for both packages.  Documentation will be coming soon.

Note that all of the PHP_* packages are alpha state, so don't use them in production quite yet.  However, do start playing around with them!

As of February 22, 2007, I have called an official referendum on the future of PEAR.  There is a news item on the front page of pear.php.net with the same instructions in this message.  This election is only open to PEAR developers who have contributed to the development of a PEAR package at any time in history, but you must have the ability to log in at pear.php.net, and must have "pear.dev" karma.

Please go vote!

http://pear.php.net/election/info.php?election=6&vote=1

The choices are between:

1) don't change a thing

Keep the PEAR Group

2) Greg Beaver's constitution

As proposed on pear-dev

3) Anant Narayanan's constitution

As proposed in response to my original proposal on Dec. 30, 2006.

More information on each of these proposals is in the election description.

I have tried to present a clear picture of the differences between the proposals: each has merit, and should be considered carefully.  The result of this election will determine how PEAR moves forward on important issues such as the ultimate purpose of PEAR, how much control developers have versus the PEAR community at large, and other major issues such as granting CVS karma.

The ballot is secret, and is designed so that it is nearly impossible to reverse-engineer voters to votes in the database.  The election interface will not let you vote without logging in or without pear.dev karma.  If you believe you should have pear.dev karma to vote, and do not, please mail the pear-dev mailing list and we'll try to resolve the issue.

Go vote!

Recently, pear.php.net silently switched over to a revolutionary new way of maintaing the live website that has been extremely freeing. The same technique is being explored by the government of the state of Iowa and other large-scale sites with mission-critical servers. Before revealing the secret, let's learn about the problem.

One of the most common tasks that we experience as web developers is synchronizing a development web server with a live site. There are many solutions that have been tried before. Just a few:

1. develop the code on the live server
2. develop on a remote machine, and ftp the files to the site as they change
3. develop using version control like CVS, and use a cron job to synchronize the website directly from CVS
4. develop on a remote server with the same configuration, and use rsync to synchronize changes

Almost everyone has tried #1. The first time you accidentally erase a file, it will rid you of that habit in a hurry. Most developers use #2, and this has the benefit of having a backup on the development machine. The same principles hold for #3 and #4 but the changes are synced in a controlled way.

However, every single method described above has the potential for immediate and catastrophic failure, even with a backup.

Why?

Websites are not just the code that runs them. Websites consist of interdependency between code, data storage backend (LDAP/database/other), and also the actual hostname itself. Of the 4 methods listed above, not one addresses this issue. When upgrading code on the live server, if there are any database changes, these must be done by hand. Keeping track of links to use (for instance, when redirecting a user to a secure login form, one cannot use a relative href, but must use the full hostname) also requires some footwork. The most common solutions involve two strategies:

1. use /etc/hosts to mimic the live server's DNS (alias hostnames to localhost)
2. use a static configuration file that is not synchronized to define hostname/path differences

The first idea is good, but also means you will need another alias in order to rsync with the remote site (or a dedicated IP on the remote site). Chances are, if you have a dedicated machine for development, you also have a dedicated IP for the live server, and then this is a good solution for managing hostname differences between the development and live machine. The second solution is far more common, and much less robust. Any file that must be different between the development machine and the remote server is asking for trouble when using ftp or rsync. If the file is excluded from rsync/ftp, when it is erased, the file is gone, and must be stored somewhere else. This quickly introduces unwanted complexity in the maintenance of the website, and another failure path for the site.

The problems only multiply when using external depedencies. As many developers have discovered, it's all fine and good to try to avoid re-inventing the wheel. The problem happens when trying to upgrade an external library. Each library has different requirements, and upgrading often means a mysterious quagmire of erasing and adding files, as well as database changes or other tasks that make it onerous to upgrade. Actually, upgrading isn't so hard, but downgrading, should the upgrade break everything is really hard (Note: this is far more common if the package is not from pear.php.net - PEAR has a strong backwards compatibility policy designed to mitigate this issue).

The ultimate solution, as I outline in chapter 3 of The PEAR Installer Manifesto, is to use the PEAR Installer in conjunction with version control to manage the complexity of synchronizing a live website with the development machine.

A few years back, while designing the next incarnation of the PEAR Installer, I realized that its ability to do file transactions, intrinsic support for dependencies and versioning make it ideal for managing not just PHP libraries and applications, but also for managing complete websites, and not just the PHP files, but all file-related events. There were just a few elements missing: how to synchronize a database? It turns out that the answer is also PEAR-related, and was one of the features integrated in PEAR 1.4.0 and newer.

The code that runs http://pear.php.net is now completely encapsulated within the pearweb package (http://pear.php.net/pearweb). As part of the installation for the pearweb package, a post-installation script is available that initializes the database using MDB2_Schema, a cross-database package that can create or modify a database installation. As time passes, the pearweb package will be divided into several related packages, one for each sub-site within pear.php.net (there is already a subpackage for the installation .phar files, http://pear.php.net/pearweb_phars). This will allow modular development and other tried and true techniques applied to traditional programming environs.

The biggest difference between the old and the new method was exemplified a few weeks back. A regression was introduced in pearweb version 1.1.2 that broke the statistics graph for package downloads. Initially, I wasn't sure what the problem was caused by, so I first reverted to the previous version:

pear upgrade --force pearweb-1.1.1

Once the problem was traced to an extra line in one of the files, pearweb 1.1.3 was released, and the new version was synchronized with:

pear upgrade pearweb-1.1.3

Two lines of shell scripting and the entire website was fixed! In another early instance, an upgrade of the website resulted in the entire site being placed in the wrong directory due to a configuration value change in the PEAR Installer at pear.php.net. In 30 seconds, I was able to fix it with:

pear config-set web_dir /path/to/actual/pearweb
pear upgrade --force pearweb

This is the missing piece of the 4 most common synchronization methods: it's really hard to fix mistakes made. You will still make mistakes when using the PEAR Installer, the difference is that reverting them is a 1-liner and requires no sweat or fear that one will break something else in the process: each release of a package is a known quantity, and it will work the same way as it did last time. The major exception is database upgrades - these require special attention, as once the database changes, reverting to a previous version is no longer possible. However, designing modularly so that database changes only affect a single module within the website can even fix this problem.

For a more comprehensive analysis of the steps needed to maintain a live webserver successfully, like ways of providing a safety buffer when upgrading (redirecting users temporarily) and the nitty-gritty of packaging up a website using the PEAR Installer, buy my book, I promise it will be worth the cover price.

Don't be fooled by the title, this book is not about how to type "pear install Blah." As one of the people who wrote to me said:

Chapter 3 is of most interest to me - going beyond simple libraries to distributing full apps. As I run a development team of 8 managing mission critical sites, I'm trying hard to bring as much good software engineering methodology as I can in. I've taught everyone to use Subversion so my next issue is deployment. I'm obviously familiar now with basic package.xml stuff for libraries but not for full blown apps. I've been working on a small (php-based) app called **** on and off for a while, which automates some aspects of web app deployment (setting up the environment - databases etc., checking stuff out of SVN including deploying bits from PEAR and so on). However now that I have your book it seems that quite a bit of it can be done within PEAR - I had no idea it was that flexible!

The book uses the Chiara Quartet's website as an example, which was a from-scratch rewrite to fit this new model. Unfortunately since writing the chapter, we decided to outsource the website to another person who uses their own model of development. Fortunately, the design is much better than when I was maintaining the site . pear.php.net is more interesting, as it demonstrates how easy it is to move an existing site to the new model. All you really need is a package.xml, and possibly a PEAR channel server, although there are other ways to move the files around as well. Incidentally, Chapter 5, which is available for free on the packt website, describes in detail how to set up a channel server.

I really hope you will check out this method - it has made everything easier for me, and in a way represents the culmination of several years of blood, sweat and tears on my part pooling ideas for the PEAR Installer, implementing and regression testing them, and finally writing about them.

Happy PHPing!

I was surprised to hear a colleague today ask how to upgrade from PEAR 1.3.6 to the latest 1.5.0, as the security holes in PEAR 1.3.6 and earlier have been pulbicly known for well over a year now. When I inquired as to why he was using PEAR 1.3.6, he explained that Max OS X ships with PEAR 1.3.6. I won't get into the questionable business practice of shipping software with known security holes that have had known fixes for years. Instead, I would like to offer simple step-by-step instructions of how to upgrade from outdated buggy PEAR versions to the latest stable release, 1.5.0.

pear upgrade Console_Getopt-1.2.1 Archive_Tar-1.3.1 PEAR-1.4.3
pear upgrade PEAR

These two commands will be sufficient to upgrade any system running PEAR 1.3.3 or newer. If you are running a REALLY outdated version of PEAR, you are far better off starting over from scratch using http://pear.php.net/go-pear, which should be downloaded and saved as go-pear.php, then run via:

php go-pear.php

These simple steps will allow any system to be upgraded. Note that if you're running something like version 1.3.0 of PEAR, you could risk running:

pear upgrade PEAR-1.3.6

and then try upgrading to PEAR using the 2 lines listed above. But for God's sake, don't stick with PEAR 1.3.6 or earlier, that is just asking for trouble.

PEAR 1.5.0, released a little while ago, pushes the envelope for PEAR. This release can be downloaded at http://pear.php.net/get/PEAR-1.5.0.tgz, or installed with

pear upgrade PEAR

This release adds several interesting new features to PEAR, and provides a further stabilization of the code base. The most important change: as promised, PEAR now requires PHP 4.3.0 or newer to function.

This release adds a few small but important new features:

• package.xml 2.1 allows PECL zend extensions like xdebug to define themselves as such
• automatic enabling and disabling of pecl extensions in php.ini is now supported
• post-installation scripts are much more advanced in the CLI frontend
• deprecated packages are displayed as such by the installer
• TAP output is possible with the run-tests command

In addition, 46 bugs have been fixed since the last stable release, 1.4.11.

PEAR 1.5.0 maintains full backwards compatibility with PEAR 1.4.11 on every level, and only adds new features and greater stability of existing features, especially installation and uninstall with complex dependency trees. I highly recommend an upgrade.

I just pushed out a release of PHP_Archive 0.9.1. This release fixes a bug that affects 64-bit users of PHP 5.2 and newer. Basically, the unpack() function does not behave in a predictable way when unpacking large integers. This is a known bug, and is documented at the manual page, although I was unaware of the issue until today.

It turns out that this code behaves differently on most platforms:

On some systems, instead of int(3068571189), the output is int(-1226396107) even though the V argument specifically requests unsigned integers

It wouldn't matter, except that number happens to be the crc32 checksum of PEAR.php, and so a different number returned meant that the .phar that installs PEAR simply wouldn't work.

The workaround that allows .phars to work in all PHP versions is to convert the unpacked integer into an unsigned integer string using the equivalent of this code:

I hope this is useful to others who are working with unpack()/pack() - be wary, some things don't act like you think they might unless you have great knowledge of the deployment environments, and know what to expect.

Thanks to Ilia and Sara for providing the tips needed to solve this one

2006 was an interesting year for PEAR, a number of exciting events happened both inside PEAR's workings and in PHP at large that affected PEAR. I will attempt to give my best shot at a 20-20 hindsight look back at PEAR in the past year. I am hugely biased, as I don't maintain every package in PEAR, so this will be tilted towards the packages I do maintain and use.

Rather than try to summarize the releases of the past year, I'll let the internet do it. Click "Prev" to see them in sequential order. The same goes for PEPr proposals in the past year. As usual, the pear-general list was saturated with questions about HTML_QuickForm throughout the entire year. DB_DataObject and Structures_DataGrid appear to be a close second.

The PEAR Installer started out at version 1.4.6 on January 6. Shortly thereafter, I announced that PEAR 1.5.0 will no longer support PHP 4.2.0 or older with a few responses, some intriguing but most were not perturbed by the announcement. In addition, I discovered an arbitrary code execution vulnerability in PEAR versions 1.3.5 and older. Several packages were proposed and several were voted on. Around this time, Derick announced that the CVS tag RELEASE_1_0_4 mysteriously appeared on every single file in PHP's CVS, which was traced to a release of System_Command by Jan Schneider's sleuthing. Oddly enough, there were no hangings as a result, in a nice break from the flame war traditions. Otherwise, life was pretty normal in January.

February marked the beginning of people understanding that they needed to upgrade PEAR, and we saw some really ancient versions trying to upgrade. There were a lot of releases, but otherwise was business as usual. One interesting point was raised by Lukas Smith. There has been a lot of FUD regarding performance and PEAR packages. He cleared up the confusion with a series of benchmarks proving that things are looking up for database abstraction in PEAR. Meanwhile, I got all medieval on Pierre's behind regarding PEAR_Frontend_Web, but all turned out well in the end. Bertrand Gugger got a little confused about the date of April Fool's, and posted this interesting release announcement to pear-dev, and was chastised for the confusion. Pierre stepped up to the plate and delivered a home run when he added the ability to subscribe to bugs in the PEAR bug tracker.

March started with a timid request from Stefano Rausch wondering when package stats would be back up. Later, an interesting discussion of how to detect whether a file is inside include_path appeared, with the solution that Zend_Framework coming up as a winner. The first of the licensing queries appeared, in benign form. The question of PHP License compatibility would come to a head later on in the year. There was an interesting but flawed attempt to create namespaces in PHP code which ultimately fizzled. Scott Mattocks raised an interesting chicken-and-egg problem with proposing two connected packages: what if one package is accepted and the other isn't? I answered this with fuggedaboutit, no worries, and he seemed satisfied with the response. The month ended with a brief "isn't Zend Framework just PEAR for PHP 5 without the installer?" question. Nobody answered.

April was pretty quiet, although Lukas raised the ultimately defeated question of whether packages in PEAR must be prefixed with PEAR_ or P_ or WHATEVER_. Justin answered the question "is PEAR PHP5-compatible" quite nicely. And then we went to May.

Philippe poked PEAR to remind people that Google's Summer of Code didn't have any PEAR listings yet, and Lukas stepped up to the plate. Alan and I disturbed the waters by reminding people that PHP 6.0 will most likely kill off called static methods that are not static. Surprisingly, this indirectly led to a huge broohaha on php-internals over just what E_STRICT should mean, as many developers had not noticed this subtle change. On a bright note, Igor Feghali wrote a nice message saying he was accepted into a Google SoC project for MDB2_Schema. PhpDocumentor rose from the grave to emit a new release with a buttload of fixed bugs. Then, Sebastian release PHPUnit2 version 3.0.0alpha5, which caught me off guard, as I thought PEAR had decided on requiring a new package when the major version is bumped. We all ran around like chickens with our heads cut off for a little while, and then (skipping ahead to July) Sebastian announced he was pulling PHPUnit from PEAR.

June began my long months of crazy music time, and I disappeared from the PEAR world, and for a while, from the internet. However, this didn't stop my from creating PHP_ParserGenerator and PHP_LexerGenerator in my spare time. Oh, and Joe Stump provided a rather definitive answer to the question of "isn't Zend Framework just PEAR for PHP 5?" No.

July began, as I mentioned before, with Sebastian's stunning move to make PHPUnit it's own thing outside of PEAR, taking advantage of the channels feature I introduced in PEAR 1.4.X. This was closely followed by Lukas Smith's announcement of his retirement from PEAR effective October 1, although it appears he may have changed his mind since then. The jury is still out on that front. A few other small things happened, such as Olivier's interesting and ultimately ignored idea of how to "dynamize PEAR." Alexey started off his first PEPr proposal in a while with an E_STRICT bang, which ultimately succeeded. Markus Tacker tried his hand at a PEPr proposal to prefix packages with "P_". This led to the all time highest number of comments on a proposal, dragged on for 3 months, received both more +1 AND more -1 votes than any other package or proposal in history, and ultimately flopped. Oh well. Markus also suggested setting up an SVN mirror of the PEAR repository, and was greeted with resounding silence. Tough month! Some tentative motions were made towards starting a PEAR PR group. Nothing solid came out of it, unfortunately.

August saw the release of PEAR 1.4.11, the last stable release of PEAR until I push out 1.5.0 sometime very soon (Read: 1 month or so, if we're lucky). Otherwise, a pretty normal month.

In September, Martin decided to codify the requirement that all PEAR packages reside in a public SCM due to overwhelming public support. Then, a real bombshell hit. Pierre announced his retirement from PEAR. Shortly after, Scott started what turned out to be a snowball effect drawing people I had never heard of before out of the woodwork with all kinds of new and some very interesting ideas for PEAR's direction. Of course, all he was asking was how to re-populate the PEAR Group, leading to my favorite reply from Joe Stump: "So, what does the PEAR Group do, anyways? I'm up for it" . P.S. thanks Joe, you're a good sport. Lukas provided a nice history here, and opened up the doors to thinking about a new structure for PEAR. I added my 2 cents (OK, 2 dollars). David called for a meeting with a pretty looking agenda, which never fully materialized, I suspect due to timezone differences among other things. Many people requested a wiki for PEAR, which received both positive and negative reviews. Meanwhile, some little crust-monkey managed to silently delete all files containing "log" in their name from PEAR cvs, which affected PhpDocumentor, PEAR, Log (obviously), peardoc and a lot of other places for a while. Fortunately, life continued as usual on pear-general with people asking for help and developers (Mark in this case) providing helpful answers.

October rolled around, and I kicked things off with the release of PEAR 1.5.0a1 and a little announcement that pearweb (the code running pear.php.net) is now installable using the PEAR installer. Meanwhile, Craig Constantine stepped up to provide a central place to track the improvements to pearweb. Martin very casually announced a hardware upgrade to the box behind pear.php.net, which was actually a HUGE leap forward in the speed of the machine, and in turn what is possible. After that point, all the stability issues we were experiencing with the website literally vanished overnight. Go Martin! This month was laden with tons of releases, and there were some good signs that new users are starting to at least consider security when designing apps. Well, sort of. Oh, and I released PhpDocumentor 1.3.1, which was nice if you want to document PHP5-related stuff. Christian set about a task I assigned him with zeal, and released QA_Peardoc_Coverage to show us how we're doing as packages and as developers when it comes to documenting our PEAR stuff. I finally fixed statistics for packages at the end of the month, and we went our merry way into November. Oh and I almost forgot, Packt Publishing published PHP Programming with PEAR and my book the PEAR Installer Manifesto, and Apress published Foundations of PEAR: Rapid PHP Development. I suggested we deprecate XML-RPC access for pear.php.net (not the package), and in the worst moment, Adam Trachtenberg pointed out that go-pear was broken in PHP 5.2. I hurriedly committed a fix 4 days prior to release, and yet it never made it in, so those of you suffering can download http://pear.php.net/go-pear.phar or wait for 5.2.1.

November. Martin added the books I mentioned to the support page at pear.php.net. I created an interface for elections, which will be used very soon for a number of fun things (see December). Dan Convissor formally announced his being too busy for XML-RPC and DB. Other stuff happened too, I guess.

In December, I created some bug stats by package, and plopped them on each package's home page. This was pretty well-received once it got the inevitable kinks worked out. Around this same time, David Coallier and Christian came up with a nifty eye-candy google map of PEAR developers. Christian suggested we should track CVS commits, and I sort of shot it down, although if we do get a dedicated server to do the number crunching it still might work. PhpDocumentor acquired a new developer who actually does work (gasp). Next, I got a little crazy and added bug fixing stats per developer on each package's homepage. It was generally well-received, but a few objections convinced me to remove it from the package page, and leave it in the stats page. In addition, pear.php.net moved away from the dangerous cvs sync method to being synced from pear releases. As I mentioned in July, Lukas suggested PEARifying Doctrine as the PHP5 database handling package for PEAR. Finally, I proposed a constitution to define how PEAR should work, which hopefully will be voted on and decided real soon. In one of the more annoying threads of the year, Sylvain Beucler raised some of the questions that had been hashed, rehashed, and rerehashed on the php-internals list about the PHP License. In the end, Rasmus's say made the final cut and is going to be in the official FAQ. Justin went ahead and asked the GPL folks what they thought about the issue. Their reply was somewhat more succinct than we hoped for. Lastly, I committed the first working proof-of-concept of a PHP5 E_STRICT installer called "PEAR2" as a working title.

So, there you have it! 2007 will be a big year for PEAR, I hope the New Year will bring happiness to you and to your family, and that this retrospective is entertaining. I'm sure I missed some stuff, just post a comment if you see anything that should be in there.