Lot 49: Greg Beaver's blog

2006 was an interesting year for PEAR, a number of exciting events happened both inside PEAR's workings and in PHP at large that affected PEAR. I will attempt to give my best shot at a 20-20 hindsight look back at PEAR in the past year. I am hugely biased, as I don't maintain every package in PEAR, so this will be tilted towards the packages I do maintain and use.

Rather than try to summarize the releases of the past year, I'll let the internet do it. Click "Prev" to see them in sequential order. The same goes for PEPr proposals in the past year. As usual, the pear-general list was saturated with questions about HTML_QuickForm throughout the entire year. DB_DataObject and Structures_DataGrid appear to be a close second.

The PEAR Installer started out at version 1.4.6 on January 6. Shortly thereafter, I announced that PEAR 1.5.0 will no longer support PHP 4.2.0 or older with a few responses, some intriguing but most were not perturbed by the announcement. In addition, I discovered an arbitrary code execution vulnerability in PEAR versions 1.3.5 and older. Several packages were proposed and several were voted on. Around this time, Derick announced that the CVS tag RELEASE_1_0_4 mysteriously appeared on every single file in PHP's CVS, which was traced to a release of System_Command by Jan Schneider's sleuthing. Oddly enough, there were no hangings as a result, in a nice break from the flame war traditions. Otherwise, life was pretty normal in January.

February marked the beginning of people understanding that they needed to upgrade PEAR, and we saw some really ancient versions trying to upgrade. There were a lot of releases, but otherwise was business as usual. One interesting point was raised by Lukas Smith. There has been a lot of FUD regarding performance and PEAR packages. He cleared up the confusion with a series of benchmarks proving that things are looking up for database abstraction in PEAR. Meanwhile, I got all medieval on Pierre's behind regarding PEAR_Frontend_Web, but all turned out well in the end. Bertrand Gugger got a little confused about the date of April Fool's, and posted this interesting release announcement to pear-dev, and was chastised for the confusion. Pierre stepped up to the plate and delivered a home run when he added the ability to subscribe to bugs in the PEAR bug tracker.

March started with a timid request from Stefano Rausch wondering when package stats would be back up. Later, an interesting discussion of how to detect whether a file is inside include_path appeared, with the solution that Zend_Framework coming up as a winner. The first of the licensing queries appeared, in benign form. The question of PHP License compatibility would come to a head later on in the year. There was an interesting but flawed attempt to create namespaces in PHP code which ultimately fizzled. Scott Mattocks raised an interesting chicken-and-egg problem with proposing two connected packages: what if one package is accepted and the other isn't? I answered this with fuggedaboutit, no worries, and he seemed satisfied with the response. The month ended with a brief "isn't Zend Framework just PEAR for PHP 5 without the installer?" question. Nobody answered.

April was pretty quiet, although Lukas raised the ultimately defeated question of whether packages in PEAR must be prefixed with PEAR_ or P_ or WHATEVER_. Justin answered the question "is PEAR PHP5-compatible" quite nicely. And then we went to May.

Philippe poked PEAR to remind people that Google's Summer of Code didn't have any PEAR listings yet, and Lukas stepped up to the plate. Alan and I disturbed the waters by reminding people that PHP 6.0 will most likely kill off called static methods that are not static. Surprisingly, this indirectly led to a huge broohaha on php-internals over just what E_STRICT should mean, as many developers had not noticed this subtle change. On a bright note, Igor Feghali wrote a nice message saying he was accepted into a Google SoC project for MDB2_Schema. PhpDocumentor rose from the grave to emit a new release with a buttload of fixed bugs. Then, Sebastian release PHPUnit2 version 3.0.0alpha5, which caught me off guard, as I thought PEAR had decided on requiring a new package when the major version is bumped. We all ran around like chickens with our heads cut off for a little while, and then (skipping ahead to July) Sebastian announced he was pulling PHPUnit from PEAR.

June began my long months of crazy music time, and I disappeared from the PEAR world, and for a while, from the internet. However, this didn't stop my from creating PHP_ParserGenerator and PHP_LexerGenerator in my spare time. Oh, and Joe Stump provided a rather definitive answer to the question of "isn't Zend Framework just PEAR for PHP 5?" No.

July began, as I mentioned before, with Sebastian's stunning move to make PHPUnit it's own thing outside of PEAR, taking advantage of the channels feature I introduced in PEAR 1.4.X. This was closely followed by Lukas Smith's announcement of his retirement from PEAR effective October 1, although it appears he may have changed his mind since then. The jury is still out on that front. A few other small things happened, such as Olivier's interesting and ultimately ignored idea of how to "dynamize PEAR." Alexey started off his first PEPr proposal in a while with an E_STRICT bang, which ultimately succeeded. Markus Tacker tried his hand at a PEPr proposal to prefix packages with "P_". This led to the all time highest number of comments on a proposal, dragged on for 3 months, received both more +1 AND more -1 votes than any other package or proposal in history, and ultimately flopped. Oh well. Markus also suggested setting up an SVN mirror of the PEAR repository, and was greeted with resounding silence. Tough month! Some tentative motions were made towards starting a PEAR PR group. Nothing solid came out of it, unfortunately.

August saw the release of PEAR 1.4.11, the last stable release of PEAR until I push out 1.5.0 sometime very soon (Read: 1 month or so, if we're lucky). Otherwise, a pretty normal month.

In September, Martin decided to codify the requirement that all PEAR packages reside in a public SCM due to overwhelming public support. Then, a real bombshell hit. Pierre announced his retirement from PEAR. Shortly after, Scott started what turned out to be a snowball effect drawing people I had never heard of before out of the woodwork with all kinds of new and some very interesting ideas for PEAR's direction. Of course, all he was asking was how to re-populate the PEAR Group, leading to my favorite reply from Joe Stump: "So, what does the PEAR Group do, anyways? I'm up for it" . P.S. thanks Joe, you're a good sport. Lukas provided a nice history here, and opened up the doors to thinking about a new structure for PEAR. I added my 2 cents (OK, 2 dollars). David called for a meeting with a pretty looking agenda, which never fully materialized, I suspect due to timezone differences among other things. Many people requested a wiki for PEAR, which received both positive and negative reviews. Meanwhile, some little crust-monkey managed to silently delete all files containing "log" in their name from PEAR cvs, which affected PhpDocumentor, PEAR, Log (obviously), peardoc and a lot of other places for a while. Fortunately, life continued as usual on pear-general with people asking for help and developers (Mark in this case) providing helpful answers.

October rolled around, and I kicked things off with the release of PEAR 1.5.0a1 and a little announcement that pearweb (the code running pear.php.net) is now installable using the PEAR installer. Meanwhile, Craig Constantine stepped up to provide a central place to track the improvements to pearweb. Martin very casually announced a hardware upgrade to the box behind pear.php.net, which was actually a HUGE leap forward in the speed of the machine, and in turn what is possible. After that point, all the stability issues we were experiencing with the website literally vanished overnight. Go Martin! This month was laden with tons of releases, and there were some good signs that new users are starting to at least consider security when designing apps. Well, sort of. Oh, and I released PhpDocumentor 1.3.1, which was nice if you want to document PHP5-related stuff. Christian set about a task I assigned him with zeal, and released QA_Peardoc_Coverage to show us how we're doing as packages and as developers when it comes to documenting our PEAR stuff. I finally fixed statistics for packages at the end of the month, and we went our merry way into November. Oh and I almost forgot, Packt Publishing published PHP Programming with PEAR and my book the PEAR Installer Manifesto, and Apress published Foundations of PEAR: Rapid PHP Development. I suggested we deprecate XML-RPC access for pear.php.net (not the package), and in the worst moment, Adam Trachtenberg pointed out that go-pear was broken in PHP 5.2. I hurriedly committed a fix 4 days prior to release, and yet it never made it in, so those of you suffering can download http://pear.php.net/go-pear.phar or wait for 5.2.1.

November. Martin added the books I mentioned to the support page at pear.php.net. I created an interface for elections, which will be used very soon for a number of fun things (see December). Dan Convissor formally announced his being too busy for XML-RPC and DB. Other stuff happened too, I guess.

In December, I created some bug stats by package, and plopped them on each package's home page. This was pretty well-received once it got the inevitable kinks worked out. Around this same time, David Coallier and Christian came up with a nifty eye-candy google map of PEAR developers. Christian suggested we should track CVS commits, and I sort of shot it down, although if we do get a dedicated server to do the number crunching it still might work. PhpDocumentor acquired a new developer who actually does work (gasp). Next, I got a little crazy and added bug fixing stats per developer on each package's homepage. It was generally well-received, but a few objections convinced me to remove it from the package page, and leave it in the stats page. In addition, pear.php.net moved away from the dangerous cvs sync method to being synced from pear releases. As I mentioned in July, Lukas suggested PEARifying Doctrine as the PHP5 database handling package for PEAR. Finally, I proposed a constitution to define how PEAR should work, which hopefully will be voted on and decided real soon. In one of the more annoying threads of the year, Sylvain Beucler raised some of the questions that had been hashed, rehashed, and rerehashed on the php-internals list about the PHP License. In the end, Rasmus's say made the final cut and is going to be in the official FAQ. Justin went ahead and asked the GPL folks what they thought about the issue. Their reply was somewhat more succinct than we hoped for. Lastly, I committed the first working proof-of-concept of a PHP5 E_STRICT installer called "PEAR2" as a working title.

So, there you have it! 2007 will be a big year for PEAR, I hope the New Year will bring happiness to you and to your family, and that this retrospective is entertaining. I'm sure I missed some stuff, just post a comment if you see anything that should be in there.

Recently, I noticed that in certain cases, PEAR appears to sort dependencies in slightly the wrong order when installing packages. For instance, while debugging the Structures_DataGrid package, which has a complex sub-package structure, I noticed that the Pager package was installed after the Structures_DataGrid_Renderers_Pager package. This disturbed me for two reasons:

1. Structures_DataGrid_Renderers_Pager depends on Pager, and so Pager should be installed first
2. There was no error.

After investigating (which in my case meant briefly recalling from memory how PEAR actually validates dependencies), I remembered that PEAR validates dependencies twice, once prior to download, and once prior to installation. By the time the dependencies are sorted, PEAR assumes that the sort algorithm properly sorts things. This is actually a pretty reasonable assumption considering the unit tests that are in place to test this.  However, like all regression testing, the unit tests test boundaries and likely cases, but not all possible inputs.

I was unsure of how to proceed, until I picked up Kyle Loudon's Mastering Algorithms with C again.  I purchased this excellent book last year to explore both common algorithms and their implementation right about the time I was most heavily into working on the phar extension.  Phar is not dead, only postponed, for those of you wondering.  The Graph section of the book has an implementation of topological sorting, and while glancing at the simple example of sorting classes and their prerequisites, it dawned on me that this is exactly what I needed.

Sorting packages with dependencies so that the dependencies are always installed first is a topological sort.  So, I took a look around and found Sergio Carvallho's Structures_Graph package at pear.php.net (although it was not easy, pear.php.net's search facilities are pretty feeble now, stay tuned for improvements at some point in the near future).  The package has a full implementation of topological sorting plus a few other things I won't need, but I now have a working implementation as part of the PEAR package.

This will unfortunately delay the release of PEAR 1.5.0 stable, as I consider this bug to be a critical bug.  Why?

When PEAR installs dependent packages that are of dependency type "subpackage" it does a special trick.  First, a little background:

Subpackage dependencies are designed to facilitate the splitting of large, bloated packages into lithe, efficient package and subpackages.  In order to do this, PEAR must cleverly remove the files in the subpackage from the parent package prior to installation, otherwise we'll get a file conflict.  If the packages are installed in the wrong order, this causes serious doodoo to hit the fan.  Look for the fix in the next release candidate, PEAR 1.5.0RC3

Recently, a curious bug was opened at pear.php.net for the PEAR package (#9639). In it, a user was able to uninstall the Structures_DataGrid package, even though all of its subpackages were installed, which should have prevented uninstallation. As you can see in the comments, a test of installation followed by uninstallation using the latest PEAR worked with flying colors. Unfortunately, the package was installed with an older PEAR version.

This commit to PEAR/DependencyDB.php contains a subtle fix that allows successful registering all of a package's dependencies for uninstallation, something that was not available until PEAR 1.4.5. If you have installed any complex packages using PEAR 1.4.4 or earlier, I highly recommend that you re-install them using PEAR 1.5.0RC2, or the 1.5.0 stable which will be released shortly after the new year (barring any new bugs in 1.5.0RC2). To do this, you need not re-download all of the packages, simply run:

pear install --register-only --force <Packagename>

Where <Packagename> is the name of every package you installed with an older PEAR, or if you're not sure, the packages shown in when running "pear list"

This could be a huge pain in the rear end for some, and is why one of the must-have features I plan for PEAR 2.0 is a more robust registry that will allow reconstruction and repair without any of this monkey business. Stay tuned for details on this and other enhancements planned.

Shameless promotion time!

Packt has put up a sample chapter, Chapter 5 "PEAR Channels: releasing to the world" up at http://www.packtpub.com/files/SampleChapter-The-PEAR-Installer-Manifesto.pdf.  Check it out to see what you're missing if you don't already own the book.

This chapter describes in detail how to set up a channel using the Chiara_PEAR_Server package from pear.chiaraquartet.net, and why channels are one of the killer features in PEAR 1.4.x+

Recently, I have been extremely active in helping to plan PEAR's future, and I have to say if you haven't noticed, things are hopping. Several significant contributions have also been made to the effort by established developers as well as several new developers who emerged from the woodwork. If you wish to see who is doing what, the pear-dev mailing list is a good place to start.

Most important on the list of improvements has been an upgrade of the hardware (thanks Martin) behind the nerve center of PEAR, pear.php.net's website. Thanks to pair Networks, we have super-fast hardware running now. This has allowed dreaming to happen that was not possible before.

Some of the exciting dreams that have already come to fruition:

• PEAR now only accepts new packages that are PHP 5+ (existing packages will not change)
• documentation coverage maps have already helped to increase the documentation (although a long ways to go remains)
• bug statistics on package homepages have significantly lowered the number of open bugs from 600ish to 400ish

Of course, I have not mentioned some of the minor but fun additions such as the most popular downloaded packages on the homepage, package download statistics being re-enabled, and David Coallier/Christian Weiske's recent addition of google mapping of where PEAR developers reside, but these are good as well.

On this list of major improvements yet to come:

• upgrade the bug tracker to allow roadmaps, and improve ease of use/robustness
• decide on a concrete governing structure for PEAR
• decide how PEAR 2.0 will work and what it even means

The last of these is of course the most important, and will be a collaboration between the installer and others. It will also mean determining what PEAR should be about, and this means a major shift is possible. I listed the others because without some kind of roadmapping capability, there is no way to document these requirements, and without a concrete governing structure, there is no way to make decisions in any kind of timely manner.

This is where you come in. I recently completed the beta-testing of PEAR's new election interface. This interface is a secret ballot voting system that will allow registered voters to help decide the outcome of PEAR's future. Most of the important decisions I mentioned will be decided by a general election, meaning that the users of PEAR, not just the developers, will have a voice in the direction PEAR decides to take.

Note: if you are already a PEAR developer, you are already registered to vote.  Accounts created for PEPr simply need to request voting karma (email pear-dev@lists.php.net to request voting karma).

How, you might ask, do I register to vote? Simple. Go on over to http://pear.php.net/account-request-vote.php and fill out the simple form. It will email you with a confirmation link. Visit this website, and you will be able to log in to vote in elections.

The first election will be to choose between several bug trackers, and I anticipate that the next will be to choose which of several possible directions PEAR could take is most important to you, so these are major decisions. If you use PHP, and want to see the standard package repository fit your needs, you will want to take a moment to register to vote. Registration is open all of the time, but you can only vote in active elections - once it's closed, you can't vote.

Register to vote, and tell your friends and colleagues that if they want to see their PHP life get even easier to do the same and have a voice in PEAR's future. Feel free to use this post as a template for blog posts/mailing list emails/whatever.