Lot 49: Greg Beaver's blog - PEAR

Working towards Pyrus and PEAR2, check it out or miss out

nospam@example.com (Greg Beaver) — Sat, 22 Sep 2007 22:25:26 -0700

Hello all. It's been a very busy time for me, what with suddenly having a full gaggle of new cello students to teach at UNL as well as a fully loaded performance schedule for the quartet, but I have been finding time to code. My priority has been the push towards PEAR2, which means that in addition to attempting to mediate the discussions of what should change from PEAR to PEAR2, I'm churning out code to turn its installer, Pyrus, from vaporware into actual code.

Along these lines, there has been significant progress. The code has been developed on PHP 5.2, but will most likely target PHP 5.3 with the introduction of namespaces. Pyrus completely revamps the way things are done. By using SPL as well as extensions like XMLReader and XMLWriter, Pyrus manages to pack in far more features inside the code while literally cutting thousands of lines of code. The biggest loser has been the package.xml handling code. More than 2500 lines of code have gone byebye, but the API for reading from and creating package.xml is now as intuitive or more intuitive than simplexml.

In addition, using interfaces, a few design patterns and some clever iterators, adding support for a customized packaging solution is a matter of creating a class with a few lines of code that accepts file contents and where to save them. Pyrus does the hard work of constructing the relative path of files within the archive and pre-processing their contents using file tasks.

The code is simple enough that I wrote a simple channel server with no dependencies except for Pyrus in less than 6 hours (just finished the initial work when my battery ran out today) that allows small channels to work without needing a database or even PHP on the channel server, as all the REST files and releases can be generated and then uploaded from a dev server.

In short, it's an exciting time to be involved in PEAR. If you've been ignoring PEAR, perhaps it is time to take another look?

quick review of Pixy vulnerability scanner for PEAR users

nospam@example.com (Greg Beaver) — Fri, 22 Jun 2007 07:13:57 -0700

Just a quick note, I tried out the Pixy XSS and SQLI Scanner (http://pixybox.seclab.tuwien.ac.at/pixy/index.php) on a few simple PEAR files. On the first, I got a java exception, on the second it was unable to resolve the simplest of includes (no ability to resolve include_path).

In short, the thing is useless for anything written using PEAR. Fun!

pear.php.net is now mirrored

nospam@example.com (Greg Beaver) — Sat, 02 Jun 2007 08:27:04 -0700

It's been a while since my last blog post, I've found myself with more limited free time than usual and have thus had to focus on the really important things. I was recently elected president of PEAR (I ran unopposed, so it's a durn good thing I was elected), and this has resulted in a flurry of activity. A lot of energy is going into revamping the PEAR website, http://pear.php.net. At this stage, most of the changes are not visible to the naked eye, as we are writing unit tests and refactoring gradually. However, one change will be highly visible.

I'm excited to announce the first two mirrors of pear.php.net are now actively mirroring the installer REST files and actual .tgz files of package releases. They are http://us.pear.php.net (provided by Joshua Eichorn and bluga.net) and http://de.pear.php.net (provided by Christian Weiske). The option exists at a future date of mirroring the entire website, but this will not be possible without further changes to the infrastructure.

To try out one of the new mirrors, simply run:

pear config-set preferred_mirror us.pear.php.net

pear config-set preferred_mirror de.pear.php.net

Then, continue using the installer normally. Users in Europe may notice a considerable performance increase by using de.pear.php.net. Ain't progress great?

pear.php.net is now mirrored

nospam@example.com (Greg Beaver) — Sat, 02 Jun 2007 08:27:04 -0700

To try out one of the new mirrors, simply run:

pear config-set preferred_mirror us.pear.php.net

pear config-set preferred_mirror de.pear.php.net

Then, continue using the installer normally. Users in Europe may notice a considerable performance increase by using de.pear.php.net. Ain't progress great?

Is anything working in PEAR?

nospam@example.com (Greg Beaver) — Sat, 05 May 2007 21:38:10 -0700

Yes. If you have been living on a desert island and haven't already done so, it's time to take another look at http://pear.php.net to see what is up.

Newly elected PEAR Group member Josh Eichorn posted a blog entry, "How would you improve PEAR" recently. I was impressed with the response, it seems many people outside of PEAR are monitoring it and have thought about how to make it better. However, I was also not so impressed with the poor job we've done letting people know about the newest improvements to PEAR. In my comments, I listed as many as I could think of, but Josh pointed out that I would do well to post these comments in a more public setting, so here goes.

For those who are wondering, most of the suggestions made in the comments to Josh's blog entry are in fact already implemented or in the planning stages. There is really only one that is not:

SVN. CVS is here to stay, as we are dependent on cvs.php.net. HOWEVER, someone willing to donate dedicated hardware would pique our interest in having optional subversion hosting.

Things that are implemented already:

pear installer is capable of modifying php.ini, but it does require a binary package to be created with the dll.
PEAR_Exception is the error/exception base class.
Mentoring. The new constitution (http://pear.php.net/manual/en/constitution.php) requires mentoring of new developers.
package tags. This was recently implemented, in anticipation of the new collectives, and other useful purposes.
deprecated/unmaintained packages have been supported for a while now (see http://pear.php.net/DB for an example)
lightweight packages. See http://pear.php.net/MDB2 for an example. All of the database drivers are decoupled into subpackages. Structures_DataGrid is the most complex example of this de-coupling. The technical capability was not present before PEAR version 1.4.9 to handle this properly, and so most packages have not yet implemented this functionality.

The social issues are a real problem. I plan to make this my primary purpose as the PEAR president, separate from my work as a developer. We'll see how it goes, but I see a strong momentum shift in the makings here, which has been my dream for several years. It's exciting to see it moving forward so dramatically. I hope that more developers who see the potential and the things that can happen will join, it is only through the force of the will of people who believe in things like civility that PEAR will shift its culture to one that encourages innovation as well as stability.

One thing you can currently depend upon in PEAR that will never be changing is that packages marked stable will always work in a similar way, such that you won't find applications based on them suddenly breaking on upgrade. Innovation can always be provided without destroying the work of your users. Stay tuned for details of how things will be changing.

One of the paths that PEAR is actively recruiting developers for is converting a few key packages into new PHP5+ based code (this means you). For example, HTTP_Request is a commonly used package that needs to be re-coded for PHP 5. We need a good database abstraction layer for PHP 5, and the PEAR Installer is almost ready for its next incarnation (codename Pyrus), details are on the roadmap (yes, another new feature of PEAR) at http://pear.php.net/bugs/roadmap.php?package=PEAR for version 2.0.0, and code in CVS at http://cvs.php.net/viewvc.cgi/pear-core/PEAR2. If you see a feature that you would like to implement, grab yourself a new account, assign it to yourself and upload a patch (yes, another new feature of PEAR). If you do not already have an account, head on over to http://pear.php.net/account-request-vote.php. This form can be used to get an account for voting in general elections as well as opening and commenting on bugs or uploading patches (yes, another new feature of PEAR).

It's a good time to get involved with PEAR, there's a lot of renewed vigor, and no ceiling in sight.

holy release party, batman

nospam@example.com (Greg Beaver) — Mon, 05 Mar 2007 13:02:14 -0800

OK, today's release party wins. I've just uploaded the 8th package for release, and it's a very significant day of releases.

First, the PEAR website now supports patch tracking in its bug tracker, as well as the ability to auto-generate a package.xml based on the roadmap, complete with release notes based on closed bugs. This is pretty exciting stuff. 3 of the releases were devoted to the website pearweb package.

Next, I released PHP_ParserGenerator version 0.1.5, which fixes a bug and adds more display output

Finally, I just released the first full-featured parser for PHP, PHP_Parser, and its companion PHP_Parser_DocblockParser. These two packages are the next incarnation of the once-defunct PHP_Parser package. Based on a PHP_ParserGenerator-based grammar, the packages can parse any valid PHP 4, 5 or 6 (PHP 5.1.0+ is required for it to run, however). Needless to say, I have phpDocumentor in mind for immediate usage. The Docblock parser uses the docblock PECL extension to do its magic. Full regression tests are available for both packages. Documentation will be coming soon.

Note that all of the PHP_* packages are alpha state, so don't use them in production quite yet. However, do start playing around with them!

Vote in the first PEAR election

nospam@example.com (Greg Beaver) — Thu, 22 Feb 2007 15:45:12 -0800

As of February 22, 2007, I have called an official referendum on the future of PEAR. There is a news item on the front page of pear.php.net with the same instructions in this message. This election is only open to PEAR developers who have contributed to the development of a PEAR package at any time in history, but you must have the ability to log in at pear.php.net, and must have "pear.dev" karma.

Please go vote!

http://pear.php.net/election/info.php?election=6&vote=1

The choices are between:

1) don't change a thing

Keep the PEAR Group

2) Greg Beaver's constitution

As proposed on pear-dev

3) Anant Narayanan's constitution

As proposed in response to my original proposal on Dec. 30, 2006.

More information on each of these proposals is in the election description.

I have tried to present a clear picture of the differences between the proposals: each has merit, and should be considered carefully. The result of this election will determine how PEAR moves forward on important issues such as the ultimate purpose of PEAR, how much control developers have versus the PEAR community at large, and other major issues such as granting CVS karma.

The ballot is secret, and is designed so that it is nearly impossible to reverse-engineer voters to votes in the database. The election interface will not let you vote without logging in or without pear.dev karma. If you believe you should have pear.dev karma to vote, and do not, please mail the pear-dev mailing list and we'll try to resolve the issue.

Go vote!

Do you develop a website? It is infinitely better to synchronize live and development sites using the PEAR Installer

nospam@example.com (Greg Beaver) — Sun, 11 Feb 2007 10:11:14 -0800

Recently, pear.php.net silently switched over to a revolutionary new way of maintaing the live website that has been extremely freeing. The same technique is being explored by the government of the state of Iowa and other large-scale sites with mission-critical servers. Before revealing the secret, let's learn about the problem.

One of the most common tasks that we experience as web developers is synchronizing a development web server with a live site. There are many solutions that have been tried before. Just a few:

develop the code on the live server
develop on a remote machine, and ftp the files to the site as they change
develop using version control like CVS, and use a cron job to synchronize the website directly from CVS
develop on a remote server with the same configuration, and use rsync to synchronize changes

Almost everyone has tried #1. The first time you accidentally erase a file, it will rid you of that habit in a hurry. Most developers use #2, and this has the benefit of having a backup on the development machine. The same principles hold for #3 and #4 but the changes are synced in a controlled way.

However, every single method described above has the potential for immediate and catastrophic failure, even with a backup.

Why?

Websites are not just the code that runs them. Websites consist of interdependency between code, data storage backend (LDAP/database/other), and also the actual hostname itself. Of the 4 methods listed above, not one addresses this issue. When upgrading code on the live server, if there are any database changes, these must be done by hand. Keeping track of links to use (for instance, when redirecting a user to a secure login form, one cannot use a relative href, but must use the full hostname) also requires some footwork. The most common solutions involve two strategies:

use /etc/hosts to mimic the live server's DNS (alias hostnames to localhost)
use a static configuration file that is not synchronized to define hostname/path differences

The first idea is good, but also means you will need another alias in order to rsync with the remote site (or a dedicated IP on the remote site). Chances are, if you have a dedicated machine for development, you also have a dedicated IP for the live server, and then this is a good solution for managing hostname differences between the development and live machine. The second solution is far more common, and much less robust. Any file that must be different between the development machine and the remote server is asking for trouble when using ftp or rsync. If the file is excluded from rsync/ftp, when it is erased, the file is gone, and must be stored somewhere else. This quickly introduces unwanted complexity in the maintenance of the website, and another failure path for the site.

The problems only multiply when using external depedencies. As many developers have discovered, it's all fine and good to try to avoid re-inventing the wheel. The problem happens when trying to upgrade an external library. Each library has different requirements, and upgrading often means a mysterious quagmire of erasing and adding files, as well as database changes or other tasks that make it onerous to upgrade. Actually, upgrading isn't so hard, but downgrading, should the upgrade break everything is really hard (Note: this is far more common if the package is not from pear.php.net - PEAR has a strong backwards compatibility policy designed to mitigate this issue).

The ultimate solution, as I outline in chapter 3 of The PEAR Installer Manifesto, is to use the PEAR Installer in conjunction with version control to manage the complexity of synchronizing a live website with the development machine.

A few years back, while designing the next incarnation of the PEAR Installer, I realized that its ability to do file transactions, intrinsic support for dependencies and versioning make it ideal for managing not just PHP libraries and applications, but also for managing complete websites, and not just the PHP files, but all file-related events. There were just a few elements missing: how to synchronize a database? It turns out that the answer is also PEAR-related, and was one of the features integrated in PEAR 1.4.0 and newer.

The code that runs http://pear.php.net is now completely encapsulated within the pearweb package (http://pear.php.net/pearweb). As part of the installation for the pearweb package, a post-installation script is available that initializes the database using MDB2_Schema, a cross-database package that can create or modify a database installation. As time passes, the pearweb package will be divided into several related packages, one for each sub-site within pear.php.net (there is already a subpackage for the installation .phar files, http://pear.php.net/pearweb_phars). This will allow modular development and other tried and true techniques applied to traditional programming environs.

The biggest difference between the old and the new method was exemplified a few weeks back. A regression was introduced in pearweb version 1.1.2 that broke the statistics graph for package downloads. Initially, I wasn't sure what the problem was caused by, so I first reverted to the previous version:

pear upgrade --force pearweb-1.1.1

Once the problem was traced to an extra line in one of the files, pearweb 1.1.3 was released, and the new version was synchronized with:

pear upgrade pearweb-1.1.3

Two lines of shell scripting and the entire website was fixed! In another early instance, an upgrade of the website resulted in the entire site being placed in the wrong directory due to a configuration value change in the PEAR Installer at pear.php.net. In 30 seconds, I was able to fix it with:

pear config-set web_dir /path/to/actual/pearweb
pear upgrade --force pearweb

This is the missing piece of the 4 most common synchronization methods: it's really hard to fix mistakes made. You will still make mistakes when using the PEAR Installer, the difference is that reverting them is a 1-liner and requires no sweat or fear that one will break something else in the process: each release of a package is a known quantity, and it will work the same way as it did last time. The major exception is database upgrades - these require special attention, as once the database changes, reverting to a previous version is no longer possible. However, designing modularly so that database changes only affect a single module within the website can even fix this problem.

For a more comprehensive analysis of the steps needed to maintain a live webserver successfully, like ways of providing a safety buffer when upgrading (redirecting users temporarily) and the nitty-gritty of packaging up a website using the PEAR Installer, buy my book, I promise it will be worth the cover price.

Don't be fooled by the title, this book is not about how to type "pear install Blah." As one of the people who wrote to me said:

Chapter 3 is of most interest to me - going beyond simple libraries to distributing full apps. As I run a development team of 8 managing mission critical sites, I'm trying hard to bring as much good software engineering methodology as I can in. I've taught everyone to use Subversion so my next issue is deployment. I'm obviously familiar now with basic package.xml stuff for libraries but not for full blown apps. I've been working on a small (php-based) app called **** on and off for a while, which automates some aspects of web app deployment (setting up the environment - databases etc., checking stuff out of SVN including deploying bits from PEAR and so on). However now that I have your book it seems that quite a bit of it can be done within PEAR - I had no idea it was that flexible!

The book uses the Chiara Quartet's website as an example, which was a from-scratch rewrite to fit this new model. Unfortunately since writing the chapter, we decided to outsource the website to another person who uses their own model of development. Fortunately, the design is much better than when I was maintaining the site . pear.php.net is more interesting, as it demonstrates how easy it is to move an existing site to the new model. All you really need is a package.xml, and possibly a PEAR channel server, although there are other ways to move the files around as well. Incidentally, Chapter 5, which is available for free on the packt website, describes in detail how to set up a channel server.

I really hope you will check out this method - it has made everything easier for me, and in a way represents the culmination of several years of blood, sweat and tears on my part pooling ideas for the PEAR Installer, implementing and regression testing them, and finally writing about them.

Happy PHPing!

Mac OS X ships with security hole-laden PEAR - how to upgrade immediately

nospam@example.com (Greg Beaver) — Wed, 24 Jan 2007 21:35:00 -0800

I was surprised to hear a colleague today ask how to upgrade from PEAR 1.3.6 to the latest 1.5.0, as the security holes in PEAR 1.3.6 and earlier have been pulbicly known for well over a year now. When I inquired as to why he was using PEAR 1.3.6, he explained that Max OS X ships with PEAR 1.3.6. I won't get into the questionable business practice of shipping software with known security holes that have had known fixes for years. Instead, I would like to offer simple step-by-step instructions of how to upgrade from outdated buggy PEAR versions to the latest stable release, 1.5.0.

pear upgrade Console_Getopt-1.2.1 Archive_Tar-1.3.1 PEAR-1.4.3
pear upgrade PEAR

These two commands will be sufficient to upgrade any system running PEAR 1.3.3 or newer. If you are running a REALLY outdated version of PEAR, you are far better off starting over from scratch using http://pear.php.net/go-pear, which should be downloaded and saved as go-pear.php, then run via:

php go-pear.php

These simple steps will allow any system to be upgraded. Note that if you're running something like version 1.3.0 of PEAR, you could risk running:

pear upgrade PEAR-1.3.6

and then try upgrading to PEAR using the 2 lines listed above. But for God's sake, don't stick with PEAR 1.3.6 or earlier, that is just asking for trouble.

PEAR 1.5.0 (out today) pushes the envelope

nospam@example.com (Greg Beaver) — Tue, 23 Jan 2007 18:59:06 -0800

PEAR 1.5.0, released a little while ago, pushes the envelope for PEAR. This release can be downloaded at http://pear.php.net/get/PEAR-1.5.0.tgz, or installed with

pear upgrade PEAR

This release adds several interesting new features to PEAR, and provides a further stabilization of the code base. The most important change: as promised, PEAR now requires PHP 4.3.0 or newer to function.

This release adds a few small but important new features:

package.xml 2.1 allows PECL zend extensions like xdebug to define themselves as such
automatic enabling and disabling of pecl extensions in php.ini is now supported
post-installation scripts are much more advanced in the CLI frontend
deprecated packages are displayed as such by the installer
TAP output is possible with the run-tests command

In addition, 46 bugs have been fixed since the last stable release, 1.4.11.

PEAR 1.5.0 maintains full backwards compatibility with PEAR 1.4.11 on every level, and only adds new features and greater stability of existing features, especially installation and uninstall with complex dependency trees. I highly recommend an upgrade.

problem with PHP_Archive-based phars tracked to odd unpack() bug

nospam@example.com (Greg Beaver) — Fri, 05 Jan 2007 20:49:28 -0800

I just pushed out a release of PHP_Archive 0.9.1. This release fixes a bug that affects 64-bit users of PHP 5.2 and newer. Basically, the unpack() function does not behave in a predictable way when unpacking large integers. This is a known bug, and is documented at the manual page, although I was unaware of the issue until today.

It turns out that this code behaves differently on most platforms:

<?php
$a = pack('V', 3068571189);
var_dump(unpack('Va', $a));
?>

On some systems, instead of int(3068571189), the output is int(-1226396107) even though the V argument specifically requests unsigned integers

It wouldn't matter, except that number happens to be the crc32 checksum of PEAR.php, and so a different number returned meant that the .phar that installs PEAR simply wouldn't work.

The workaround that allows .phars to work in all PHP versions is to convert the unpacked integer into an unsigned integer string using the equivalent of this code:

<?php
$a = sprintf("%u", $a);
?>

I hope this is useful to others who are working with unpack()/pack() - be wary, some things don't act like you think they might unless you have great knowledge of the deployment environments, and know what to expect.

Thanks to Ilia and Sara for providing the tips needed to solve this one

2006: a PEAR retrospective

nospam@example.com (Greg Beaver) — Sat, 30 Dec 2006 13:07:23 -0800

2006 was an interesting year for PEAR, a number of exciting events happened both inside PEAR's workings and in PHP at large that affected PEAR. I will attempt to give my best shot at a 20-20 hindsight look back at PEAR in the past year. I am hugely biased, as I don't maintain every package in PEAR, so this will be tilted towards the packages I do maintain and use.

Rather than try to summarize the releases of the past year, I'll let the internet do it. Click "Prev" to see them in sequential order. The same goes for PEPr proposals in the past year. As usual, the pear-general list was saturated with questions about HTML_QuickForm throughout the entire year. DB_DataObject and Structures_DataGrid appear to be a close second.

The PEAR Installer started out at version 1.4.6 on January 6. Shortly thereafter, I announced that PEAR 1.5.0 will no longer support PHP 4.2.0 or older with a few responses, some intriguing but most were not perturbed by the announcement. In addition, I discovered an arbitrary code execution vulnerability in PEAR versions 1.3.5 and older. Several packages were proposed and several were voted on. Around this time, Derick announced that the CVS tag RELEASE_1_0_4 mysteriously appeared on every single file in PHP's CVS, which was traced to a release of System_Command by Jan Schneider's sleuthing. Oddly enough, there were no hangings as a result, in a nice break from the flame war traditions. Otherwise, life was pretty normal in January.

February marked the beginning of people understanding that they needed to upgrade PEAR, and we saw some really ancient versions trying to upgrade. There were a lot of releases, but otherwise was business as usual. One interesting point was raised by Lukas Smith. There has been a lot of FUD regarding performance and PEAR packages. He cleared up the confusion with a series of benchmarks proving that things are looking up for database abstraction in PEAR. Meanwhile, I got all medieval on Pierre's behind regarding PEAR_Frontend_Web, but all turned out well in the end. Bertrand Gugger got a little confused about the date of April Fool's, and posted this interesting release announcement to pear-dev, and was chastised for the confusion. Pierre stepped up to the plate and delivered a home run when he added the ability to subscribe to bugs in the PEAR bug tracker.

March started with a timid request from Stefano Rausch wondering when package stats would be back up. Later, an interesting discussion of how to detect whether a file is inside include_path appeared, with the solution that Zend_Framework coming up as a winner. The first of the licensing queries appeared, in benign form. The question of PHP License compatibility would come to a head later on in the year. There was an interesting but flawed attempt to create namespaces in PHP code which ultimately fizzled. Scott Mattocks raised an interesting chicken-and-egg problem with proposing two connected packages: what if one package is accepted and the other isn't? I answered this with fuggedaboutit, no worries, and he seemed satisfied with the response. The month ended with a brief "isn't Zend Framework just PEAR for PHP 5 without the installer?" question. Nobody answered.

April was pretty quiet, although Lukas raised the ultimately defeated question of whether packages in PEAR must be prefixed with PEAR_ or P_ or WHATEVER_. Justin answered the question "is PEAR PHP5-compatible" quite nicely. And then we went to May.

Philippe poked PEAR to remind people that Google's Summer of Code didn't have any PEAR listings yet, and Lukas stepped up to the plate. Alan and I disturbed the waters by reminding people that PHP 6.0 will most likely kill off called static methods that are not static. Surprisingly, this indirectly led to a huge broohaha on php-internals over just what E_STRICT should mean, as many developers had not noticed this subtle change. On a bright note, Igor Feghali wrote a nice message saying he was accepted into a Google SoC project for MDB2_Schema. PhpDocumentor rose from the grave to emit a new release with a buttload of fixed bugs. Then, Sebastian release PHPUnit2 version 3.0.0alpha5, which caught me off guard, as I thought PEAR had decided on requiring a new package when the major version is bumped. We all ran around like chickens with our heads cut off for a little while, and then (skipping ahead to July) Sebastian announced he was pulling PHPUnit from PEAR.

June began my long months of crazy music time, and I disappeared from the PEAR world, and for a while, from the internet. However, this didn't stop my from creating PHP_ParserGenerator and PHP_LexerGenerator in my spare time. Oh, and Joe Stump provided a rather definitive answer to the question of "isn't Zend Framework just PEAR for PHP 5?" No.

July began, as I mentioned before, with Sebastian's stunning move to make PHPUnit it's own thing outside of PEAR, taking advantage of the channels feature I introduced in PEAR 1.4.X. This was closely followed by Lukas Smith's announcement of his retirement from PEAR effective October 1, although it appears he may have changed his mind since then. The jury is still out on that front. A few other small things happened, such as Olivier's interesting and ultimately ignored idea of how to "dynamize PEAR." Alexey started off his first PEPr proposal in a while with an E_STRICT bang, which ultimately succeeded. Markus Tacker tried his hand at a PEPr proposal to prefix packages with "P_". This led to the all time highest number of comments on a proposal, dragged on for 3 months, received both more +1 AND more -1 votes than any other package or proposal in history, and ultimately flopped. Oh well. Markus also suggested setting up an SVN mirror of the PEAR repository, and was greeted with resounding silence. Tough month! Some tentative motions were made towards starting a PEAR PR group. Nothing solid came out of it, unfortunately.

August saw the release of PEAR 1.4.11, the last stable release of PEAR until I push out 1.5.0 sometime very soon (Read: 1 month or so, if we're lucky). Otherwise, a pretty normal month.

In September, Martin decided to codify the requirement that all PEAR packages reside in a public SCM due to overwhelming public support. Then, a real bombshell hit. Pierre announced his retirement from PEAR. Shortly after, Scott started what turned out to be a snowball effect drawing people I had never heard of before out of the woodwork with all kinds of new and some very interesting ideas for PEAR's direction. Of course, all he was asking was how to re-populate the PEAR Group, leading to my favorite reply from Joe Stump: "So, what does the PEAR Group do, anyways? I'm up for it" . P.S. thanks Joe, you're a good sport. Lukas provided a nice history here, and opened up the doors to thinking about a new structure for PEAR. I added my 2 cents (OK, 2 dollars). David called for a meeting with a pretty looking agenda, which never fully materialized, I suspect due to timezone differences among other things. Many people requested a wiki for PEAR, which received both positive and negative reviews. Meanwhile, some little crust-monkey managed to silently delete all files containing "log" in their name from PEAR cvs, which affected PhpDocumentor, PEAR, Log (obviously), peardoc and a lot of other places for a while. Fortunately, life continued as usual on pear-general with people asking for help and developers (Mark in this case) providing helpful answers.

October rolled around, and I kicked things off with the release of PEAR 1.5.0a1 and a little announcement that pearweb (the code running pear.php.net) is now installable using the PEAR installer. Meanwhile, Craig Constantine stepped up to provide a central place to track the improvements to pearweb. Martin very casually announced a hardware upgrade to the box behind pear.php.net, which was actually a HUGE leap forward in the speed of the machine, and in turn what is possible. After that point, all the stability issues we were experiencing with the website literally vanished overnight. Go Martin! This month was laden with tons of releases, and there were some good signs that new users are starting to at least consider security when designing apps. Well, sort of. Oh, and I released PhpDocumentor 1.3.1, which was nice if you want to document PHP5-related stuff. Christian set about a task I assigned him with zeal, and released QA_Peardoc_Coverage to show us how we're doing as packages and as developers when it comes to documenting our PEAR stuff. I finally fixed statistics for packages at the end of the month, and we went our merry way into November. Oh and I almost forgot, Packt Publishing published PHP Programming with PEAR and my book the PEAR Installer Manifesto, and Apress published Foundations of PEAR: Rapid PHP Development. I suggested we deprecate XML-RPC access for pear.php.net (not the package), and in the worst moment, Adam Trachtenberg pointed out that go-pear was broken in PHP 5.2. I hurriedly committed a fix 4 days prior to release, and yet it never made it in, so those of you suffering can download http://pear.php.net/go-pear.phar or wait for 5.2.1.

November. Martin added the books I mentioned to the support page at pear.php.net. I created an interface for elections, which will be used very soon for a number of fun things (see December). Dan Convissor formally announced his being too busy for XML-RPC and DB. Other stuff happened too, I guess.

In December, I created some bug stats by package, and plopped them on each package's home page. This was pretty well-received once it got the inevitable kinks worked out. Around this same time, David Coallier and Christian came up with a nifty eye-candy google map of PEAR developers. Christian suggested we should track CVS commits, and I sort of shot it down, although if we do get a dedicated server to do the number crunching it still might work. PhpDocumentor acquired a new developer who actually does work (gasp). Next, I got a little crazy and added bug fixing stats per developer on each package's homepage. It was generally well-received, but a few objections convinced me to remove it from the package page, and leave it in the stats page. In addition, pear.php.net moved away from the dangerous cvs sync method to being synced from pear releases. As I mentioned in July, Lukas suggested PEARifying Doctrine as the PHP5 database handling package for PEAR. Finally, I proposed a constitution to define how PEAR should work, which hopefully will be voted on and decided real soon. In one of the more annoying threads of the year, Sylvain Beucler raised some of the questions that had been hashed, rehashed, and rerehashed on the php-internals list about the PHP License. In the end, Rasmus's say made the final cut and is going to be in the official FAQ. Justin went ahead and asked the GPL folks what they thought about the issue. Their reply was somewhat more succinct than we hoped for. Lastly, I committed the first working proof-of-concept of a PHP5 E_STRICT installer called "PEAR2" as a working title.

So, there you have it! 2007 will be a big year for PEAR, I hope the New Year will bring happiness to you and to your family, and that this retrospective is entertaining. I'm sure I missed some stuff, just post a comment if you see anything that should be in there.

interesting, potentially critical bug in PEAR

nospam@example.com (Greg Beaver) — Tue, 19 Dec 2006 19:12:49 -0800

Recently, I noticed that in certain cases, PEAR appears to sort dependencies in slightly the wrong order when installing packages. For instance, while debugging the Structures_DataGrid package, which has a complex sub-package structure, I noticed that the Pager package was installed after the Structures_DataGrid_Renderers_Pager package. This disturbed me for two reasons:

Structures_DataGrid_Renderers_Pager depends on Pager, and so Pager should be installed first
There was no error.

After investigating (which in my case meant briefly recalling from memory how PEAR actually validates dependencies), I remembered that PEAR validates dependencies twice, once prior to download, and once prior to installation. By the time the dependencies are sorted, PEAR assumes that the sort algorithm properly sorts things. This is actually a pretty reasonable assumption considering the unit tests that are in place to test this. However, like all regression testing, the unit tests test boundaries and likely cases, but not all possible inputs.

I was unsure of how to proceed, until I picked up Kyle Loudon's Mastering Algorithms with C again. I purchased this excellent book last year to explore both common algorithms and their implementation right about the time I was most heavily into working on the phar extension. Phar is not dead, only postponed, for those of you wondering. The Graph section of the book has an implementation of topological sorting, and while glancing at the simple example of sorting classes and their prerequisites, it dawned on me that this is exactly what I needed.

Sorting packages with dependencies so that the dependencies are always installed first is a topological sort. So, I took a look around and found Sergio Carvallho's Structures_Graph package at pear.php.net (although it was not easy, pear.php.net's search facilities are pretty feeble now, stay tuned for improvements at some point in the near future). The package has a full implementation of topological sorting plus a few other things I won't need, but I now have a working implementation as part of the PEAR package.

This will unfortunately delay the release of PEAR 1.5.0 stable, as I consider this bug to be a critical bug. Why?

When PEAR installs dependent packages that are of dependency type "subpackage" it does a special trick. First, a little background:

Subpackage dependencies are designed to facilitate the splitting of large, bloated packages into lithe, efficient package and subpackages. In order to do this, PEAR must cleverly remove the files in the subpackage from the parent package prior to installation, otherwise we'll get a file conflict. If the packages are installed in the wrong order, this causes serious doodoo to hit the fan. Look for the fix in the next release candidate, PEAR 1.5.0RC3

Be careful of PEAR 1.4.4 and older installs when uninstalling a package

nospam@example.com (Greg Beaver) — Mon, 18 Dec 2006 12:59:11 -0800

Recently, a curious bug was opened at pear.php.net for the PEAR package (#9639). In it, a user was able to uninstall the Structures_DataGrid package, even though all of its subpackages were installed, which should have prevented uninstallation. As you can see in the comments, a test of installation followed by uninstallation using the latest PEAR worked with flying colors. Unfortunately, the package was installed with an older PEAR version.

This commit to PEAR/DependencyDB.php contains a subtle fix that allows successful registering all of a package's dependencies for uninstallation, something that was not available until PEAR 1.4.5. If you have installed any complex packages using PEAR 1.4.4 or earlier, I highly recommend that you re-install them using PEAR 1.5.0RC2, or the 1.5.0 stable which will be released shortly after the new year (barring any new bugs in 1.5.0RC2). To do this, you need not re-download all of the packages, simply run:

pear install --register-only --force <Packagename>

Where <Packagename> is the name of every package you installed with an older PEAR, or if you're not sure, the packages shown in when running "pear list"

This could be a huge pain in the rear end for some, and is why one of the must-have features I plan for PEAR 2.0 is a more robust registry that will allow reconstruction and repair without any of this monkey business. Stay tuned for details on this and other enhancements planned.

sample chapter of PEAR Installer Manifesto available

nospam@example.com (Greg Beaver) — Fri, 15 Dec 2006 15:34:22 -0800

Shameless promotion time!

Packt has put up a sample chapter, Chapter 5 "PEAR Channels: releasing to the world" up at http://www.packtpub.com/files/SampleChapter-The-PEAR-Installer-Manifesto.pdf. Check it out to see what you're missing if you don't already own the book.

This chapter describes in detail how to set up a channel using the Chiara_PEAR_Server package from pear.chiaraquartet.net, and why channels are one of the killer features in PEAR 1.4.x+