PHP

Recently noted by Derick Rethans in an informative blog post, there is (now was) a problem in the implementation of namespaces in the soon-to-be-released PHP 5.3.  I quote:

Take for example this code:

<?php
use PEAR::Date::Interval as Interval;
?>

In PHP 5.3 this would alias the class Interval in the namespace PEAR::Date to the class Interval. For now, this code would work just fine. However, if PHP would introduce a class "Interval" at some point in the future (and PHP can do this as it owns the global namespace) then the above code would suddenly stop working.

This morning, I realized that this would be very easy to fix, and posted a patch to internals@ with a brief explanation.  Dmitry committed a fix to PHP_5_3 and HEAD very quickly.  The issue raised by Derick is now obsolete, making namespaces not just more useful, but even more powerful than they were.

For instance, it is possible to "override" an internal class, perhaps to add missing functionality that will be present in a future release or fix a bug:

mydate.php:
<?php
namespace Foo;
class DateTime extends ::DateTime
{
// methods here
}
?>

<?php
include 'mydate.php';
use Foo::DateTime; // this essentially overrides internal DateTime class in the current script
$a = new DateTime('2006-05-04');
?>

Those who were lamenting the implementation of namespaces in PHP should take note: if you have a solution to make them better, the turnaround time between proposal to the list and commit can be on the order of a few hours if the idea clearly solves a major problem.

http://tek.phparch.com/mats/slides/greg_beaver-pear_phar.pdf

OK, so my selection of puns is not exactly "funny" per se, but it is true: php|tek is all about phar and variant spellings of the 4 letter word, such as "pear".  If you're in the audience for my talk about smart PHP application deployment using phar, PEAR and Pyrus, come say hello, I'm looking forward to meeting just about everyone I've argu^H^H^Hconversed with over the years, and of course you can also check out a copy of my book, I'll stuff a few into the luggage.  In other news, phar is tentatively enabled by default in PHP 5.3, marking the exciting next phase in all of the back-breaking work that has gone into it.  Check out the complete manual at http://php.net/phar.

Although I perform all the time as part of my day job as cellist of the Chiara Quartet, this will be a very different kind of performing experience and I'm looking forward to the challenge of revealing the true dynamism and excitement that I see in app deployment stuff I've been working on the past several years.

See you in Chicago!

I've been a bit busy, my last PHP post was in October (!) but I do have very good news. Following on the heels of my first php|arch article about the Phar extension in the January issue (http://phparch.com/c/magazine/issue/65), I've been extremely hard at work on perfecting this extension. As a testament to the strength and excitement surrounding the upcoming 2.0 release of the phar extension, my editor at php|arch became a developer on the phar extension after reading and working on the article (hi Steph).

Today, I released phar version 2.0.0a1, the first release of the newly upgraded extension. This extension is a wholly different animal from the last release, 1.2.3. While maintaining BC with the 1.2.x version of pecl/phar, this version adds unimaginable new facilities. The phar extension has transformed from something of a sideshow to a major player, and is being actively considered for inclusion in the core of PHP 5.3. Major new features in phar 2.0:

• Phar now supports tar and zip archives with the same API that is supports phar archives. Think of phar as something like the PDO of archive file formats. All phar archives must contain ".phar" in the filename to be executed
• Phar has a front controller for web applications that fully handles MIME types, supports mod_rewrite-like functionality with far more flexibility (a php callback is passed the url, it returns the actual file to load) and also the ability to deny access to portions of the phar that are not web-accessible (like library PHP code)
• Phar fully solves the problems with current directory and code loading/file loading that plague all other stream wrappers like PHP_Archive and PHK. This allows running complex applications like phpMyAdmin with no code modification directly from the original .tgz after a rename and the addition of a 20-line stub
• Phar supports phar:// stream wrappers in include_path for PHP 5.2+, so your require_once-based applications can be pharred up without code modification
• Phar supports creation and modification of data-only tar and zip archives (no executable phar stub) via the PharData class. Now you can use phar to create and even convert between file formats. Unlike the zip extension, phar supports bzip2 compression of files, and has no arbitrary limit on open file handles. Unlike the archive extension in pecl, phar fully supports windows, and is the first PHP extension to deliver read/write support for tar archives on both unix and windows.

In other words, phar 2.0 is the "holy crap" release of phar. The manual (which is currently almost fully updated to the API, but not yet rebuilt) is at http://www.php.net/phar. Another way to get to know phar is through the unit tests, which are included in the pecl tarball.

The current release can be downloaded from http://pecl.php.net/get/phar-2.0.0a1.tgz or installed via pecl install phar and windows dll can be downloaded from http://pecl4win.php.net/ext.php/php_phar.dll although it looks like building is having a few hiccups lately.

There are a few known issues to be worked out. One is compatibility with xdebug, which is a trivial problem involving extension load order. Once we work this out, xdebug will support stepping through the source of files within a phar archive in your IDE, a killer feature for those who need to debug things. In PHP 5.3, phar requires a small patch to support the phar stream wrapper in include_path.

...This brings me to PHP 5.3. I've been working hard with Dmitry Stogov, Marcus Boerger (the other phar lead) and a few other list contributors to perfect a patch that will bring stream wrapper support to include_path. This will mean that you can write a userspace stream wrapper, put its url in your include_path and start using it with relative includes such as what PEAR uses to load its code. This powerful feature will allow you to write your code in one way, and run it from several different locations. Expect to see this feature committed in the next week or so, barring major problems.

Lastly, I'm happy to announce that I will be a speaker at this year's php|tek conference in Chicago. It is the first time I've been able to pull myself away from a crazy-busy quartet schedule to attend a conference, and I look forward very much to meeting as many of you as possible while there. My talk will be on smart PHP application deployment, and covers PEAR, Phar, and the new Pyrus installer for PEAR2 (http://tek.phparch.com/c/schedule/talk/d2s6/1). Unfortunately, the talk is at the same as Josh Eichorn's talk on PEAR2 (http://tek.phparch.com/c/schedule/talk/d2s6/0), so we may need to work out some way to shuttle folks back and forth, we'll have to see .

Progress of the highest order is being made here, expect to hear more about the goings-on behind phar and PEAR2 as time permits.

So for the past half year or so, I have been trying on and off to install security updates for Microsoft .NET 1.1 and 2.0. Each time, it hadn't worked. I've now disabled automatic updates because they always fail on these two old updates.

Today, I thought maybe I would take a real crack at fixing this issue, and so did an extensive google search for any information on installing/uninstalling/repairing a .NET installation. There is, not surprisingly, a huge number of problems that people have experienced, this being a Microsoft component. I finally found a page on the Microsoft site that displays instructions on repairing .NET.

Unfortunately, none of the instructions work. I tried to uninstall the component, and it tells me source files are not available. I try re-installing from source, and it tells me that I can't install. I try manually deleting and it tells me the file is in use (yes, I've disabled anti-virus). Because this is an MS Windows machine, it's virtually impossible to tell which file is using the .NET stuff.

Basically, I'm not allowed to do anything productive with the Microsoft stuff on this computer. This incidentally also means I can't install VS 2005 because it needs a working .NET installation, so I can't debug my pecl extensions on Windows. My next computer is going to be a MacBook Pro, and I'm never looking back.

Hello all.  It's been a very busy time for me, what with suddenly having a full gaggle of new cello students to teach at UNL as well as a fully loaded performance schedule for the quartet, but I have been finding time to code.  My priority has been the push towards PEAR2, which means that in addition to attempting to mediate the discussions of what should change from PEAR to PEAR2, I'm churning out code to turn its installer, Pyrus, from vaporware into actual code.

Along these lines, there has been significant progress.  The code has been developed on PHP 5.2, but will most likely target PHP 5.3 with the introduction of namespaces.  Pyrus completely revamps the way things are done.  By using SPL as well as extensions like XMLReader and XMLWriter, Pyrus manages to pack in far more features inside the code while literally cutting thousands of lines of code.  The biggest loser has been the package.xml handling code.  More than 2500 lines of code have gone byebye, but the API for reading from and creating package.xml is now as intuitive or more intuitive than simplexml.

In addition, using interfaces, a few design patterns and some clever iterators, adding support for a customized packaging solution is a matter of creating a class with a few lines of code that accepts file contents and where to save them.  Pyrus does the hard work of constructing the relative path of files within the archive and pre-processing their contents using file tasks.

The code is simple enough that I wrote a simple channel server with no dependencies except for Pyrus in less than 6 hours (just finished the initial work when my battery ran out today) that allows small channels to work without needing a database or even PHP on the channel server, as all the REST files and releases can be generated and then uploaded from a dev server.

In short, it's an exciting time to be involved in PEAR.  If you've been ignoring PEAR, perhaps it is time to take another look?

A few minutes ago, I released phar version 1.2.1. phar is a PHP extension that allows the creation of complete filesystems within a single physical file.  In addition, a bootstrap stub written in PHP can be used to run a phar archive as if it were an executable file or shell script.  Documentation on its usage and how to create phars is at http://php.net/phar in the PHP manual.

This version of phar fixes a number of issues as documented in my last blog post, as well as a quite few issues that Marcus fixed in relation to setting the bootstrap stub, using phar's phar.extract_list INI parameter with an apache module and others (full list here).

In addition, I ran callgrind on the thing and found a few places it could use some trimming of the fat, and was able to cut down the number of stream operations by a factor of 2x when using phar in the most common use case, reading uncompressed files.  In addition, I cut another 2x for phars that have an MD5 or SHA1 signature.

This is quite a good release, looking forward to hearing your stories of using it.

Well it's been the craziest summer of my entire life (see Chiara Quartet website for the skinny). Of course, at the moment my schedule went into overdrive, two major issues "hit the fan" in the PHP world, in that Marcus proposed the phar extension for inclusion in the core of PHP, and Arnaud and I proposed major changes to the PEAR coding standards for the new PEAR2 repository. More on PEAR2 later.

phar is an interesting beast. Right before the summer began, I received a private email from a user who noticed that when trying to package together more than around 270 files into the same phar, his OS X would run out of available file handles to use. This is a problem that Marcus and I had suspected could be an issue based on the reports of alpha testers, but had not yet seen in the wild.

Marcus did a bunch of work on improving the stability of phar earlier in the spring and summer, bringing us to phar version 1.2.0 in late May, but did not have time to look at this complicated issue

So, early this week I sat down and buried myself inside the computer for many hours, finally achieving victory a few minutes ago. The process was arduous for many reasons. First, I needed to write better introspection, so I created a gdb helper file similar to the .gdbinit bundled with PHP that would allow me to browse open phars, open file handles, and a phar file entry very quickly.

Next, I had to track down the source of the problem. It turned out to be in the implementation of flushing to disk, which aggregates all unmodified and modified phar entries and then pushes them into a new phar on the disk, generates the manifest, calculates a hash, yadda yadda. After this act, every single file entry in the phar has an open file pointer for a temporary file containing the file's contents, even the ones that were unused!  This was bad, so I created a simple routine to check for unused files and close them during flush.

Thanks to our voluminous unit tests (with upwards of 90% code coverage according to gcov), I found a large issue in rename, and also discovered that I would need to implement a just-in-time re-opening of the file handle.  At the same time, I discovered a subtle coordination issue with compression where a newly compressed file was being treated as compressed before the compression could actually occur.  After fixing this, I finally had my patch.  Further real-life testing is needed before I trust that this is the version of phar that will be released as 1.2.1, but it's a grand step closer to fully robust.

Perhaps I'm the only person who thinks this is exciting, but it's been an issue since the earliest versions of phar with write support, and now is an issue no more.  This makes the next step of profiling and comparing to other solutions a lot more palatable to me, as I much prefer stability before optimization for all of the obvious reasons.

By the way, for those of you wondering how to make the transition from programming in PHP to programming PHP itself, I can't claim to be an expert, but I can tell you who is.  Sara Golemon.  Her book, Extending and Embedding PHP is an indispensable resource for anyone who wants to either write extensions or dig into the source of PHP itself.  Combine this with the (free) search facility of lxr.php.net and a clever use of grep, and you're well on your way to being a PHP internals coder with very little extra effort.  Sara's book is worth at least twice the cover price, and I recommend it even to those who have no interest in programming extensions, as understanding a bit (or more) of the internals of PHP is worth its weight in gold when you are making design choices for your applications or libraries.

A quick note about an obscure problem peculiar to PHP files that make use of PHP 5's new __HALT_COMPILER(); token.  Prior to PHP 5.2.2, if you included two files that contained the __HALT_COMPILER(); token, you would get a notice that __COMPILER_HALT_OFFSET__ was already defined.

This posed a rather large problem for the phar file format, which makes use of __HALT_COMPILER();.  Essentially, this eliminated any possibility of using phar as a library format.

However, never fear, your trusty Greg is here!

With this patch to PHP 5 and this patch to PHP 6, the problem is eliminated.  This patch will also affect applications like PHK that utilize the same principle.  Most of you won't notice the difference until Pyrus, the next generation of the PEAR Installer, is released.

In short, the thing is useless for anything written using PEAR.  Fun!

Shortly after my last blog post, I got a very helpful email from Rob Richards, who offered to help me with my trials and tribulations. He took a quick look at the schema for package.xml, and a sample package.xml and was able to find two small tweaks to make parsing actually work (gasp). This changes everything. I am abandoning the creation of a relax NG schema in favor of the battle-tested xsd. The error messages for xsd validation are far clearer than the rng ones. For instance with this script:

and the <date> tag moved to the wrong location, I get this error:

Line 565: Element '{http://pear.php.net/dtd/package-2.0}date': This element is not expected. Expected is ( {http://pear.php.net/dtd/package-2.0}stability ).

Fantastic! The error actually tells you what the problem is and gives a useful clue on how to fix it - this is *exactly* what I need to strip out the thousands of lines of unnecessary schema validation code in Pyrus (hallelujah).

I still haven't worked out the problems I was having with rng, perhaps another update on this later.

Update: I've found a solution that will work for Pyrus.  See my next blog post.

While working on Pyrus, the next generation installer for PEAR, I've been attempting to figure out how to leverage PHP's excellent new libxml-based XML validation to speed up package.xml validation in PEAR. To my great surprise and annoyance, I am starting to believe that this is the wrong thing to do.

Originally, I designed a W3C Schema for package.xml 2.0 using XMLSpy, but found that libxml is unable to handle the schema whatsoever. Instead, with the release of PEAR 1.4.0 I designed a schema validator based on the array functions of PHP and the unserialized array generated by XML_Serializer's XML_Unserializer class. Because this is not an elegant validator of W3C Schema or Relax NG, I called it "stupidSchemaValidate()". This is used to validate the basic structure of the XML level-by-level, and so requires many function calls.

I never considered using DTD because it can't handle namespaces.

Now that I am working on the PHP 5+ implementation of Pyrus, the first thing I thought I might do is create a Relax NG schema that the PHP libxml can handle. After an entire day of fighting with the thing, I've managed to discover more than 10 simple and valid Relax NG schema that simply don't work with the version of libxml distributed with PHP 5.2.3. In addition, with helpful error messages like "Expecting name, got nothing here," even with the use of libxml_use_internal_errors() I find the error reporting to be excruciatingly useless.

With my hacked-together stupidSchemaValidate(), I get wonderfully clear error messages like "unexpected <channel>, expecting one of <version>, <date> on line 23" which make it not just easy but simple to debug broken package.xml formatting.

Is there anyone out there who has had any luck getting a supposedly smart external validation tool to print remotely helpful error messages on validating a complex .xml with PHP?

It's been a while since my last blog post, I've found myself with more limited free time than usual and have thus had to focus on the really important things.  I was recently elected president of PEAR (I ran unopposed, so it's a durn good thing I was elected), and this has resulted in a flurry of activity.  A lot of energy is going into revamping the PEAR website, http://pear.php.net.  At this stage, most of the changes are not visible to the naked eye, as we are writing unit tests and refactoring gradually.  However, one change will be highly visible.

I'm excited to announce the first two mirrors of pear.php.net are now actively mirroring the installer REST files and actual .tgz files of package releases.  They are http://us.pear.php.net (provided by Joshua Eichorn and bluga.net) and http://de.pear.php.net (provided by Christian Weiske).  The option exists at a future date of mirroring the entire website, but this will not be possible without further changes to the infrastructure.

To try out one of the new mirrors, simply run:

pear config-set preferred_mirror us.pear.php.net

or

pear config-set preferred_mirror de.pear.php.net

Then, continue using the installer normally.  Users in Europe may notice a considerable performance increase by using de.pear.php.net.  Ain't progress great?

It's been a while since my last blog post, I've found myself with more limited free time than usual and have thus had to focus on the really important things. I was recently elected president of PEAR (I ran unopposed, so it's a durn good thing I was elected), and this has resulted in a flurry of activity. A lot of energy is going into revamping the PEAR website, http://pear.php.net. At this stage, most of the changes are not visible to the naked eye, as we are writing unit tests and refactoring gradually. However, one change will be highly visible.

I'm excited to announce the first two mirrors of pear.php.net are now actively mirroring the installer REST files and actual .tgz files of package releases. They are http://us.pear.php.net (provided by Joshua Eichorn and bluga.net) and http://de.pear.php.net (provided by Christian Weiske). The option exists at a future date of mirroring the entire website, but this will not be possible without further changes to the infrastructure.

To try out one of the new mirrors, simply run:

pear config-set preferred_mirror us.pear.php.net

or

pear config-set preferred_mirror de.pear.php.net

Then, continue using the installer normally. Users in Europe may notice a considerable performance increase by using de.pear.php.net. Ain't progress great?

Yes. If you have been living on a desert island and haven't already done so, it's time to take another look at http://pear.php.net to see what is up.

Newly elected PEAR Group member Josh Eichorn posted a blog entry, "How would you improve PEAR" recently. I was impressed with the response, it seems many people outside of PEAR are monitoring it and have thought about how to make it better. However, I was also not so impressed with the poor job we've done letting people know about the newest improvements to PEAR. In my comments, I listed as many as I could think of, but Josh pointed out that I would do well to post these comments in a more public setting, so here goes.

For those who are wondering, most of the suggestions made in the comments to Josh's blog entry are in fact already implemented or in the planning stages. There is really only one that is not:

1. SVN. CVS is here to stay, as we are dependent on cvs.php.net. HOWEVER, someone willing to donate dedicated hardware would pique our interest in having optional subversion hosting.

Things that are implemented already:

1. pear installer is capable of modifying php.ini, but it does require a binary package to be created with the dll.
2. PEAR_Exception is the error/exception base class.
3. Mentoring. The new constitution (http://pear.php.net/manual/en/constitution.php) requires mentoring of new developers.
4. package tags. This was recently implemented, in anticipation of the new collectives, and other useful purposes.
5. deprecated/unmaintained packages have been supported for a while now (see http://pear.php.net/DB for an example)
6. lightweight packages. See http://pear.php.net/MDB2 for an example. All of the database drivers are decoupled into subpackages. Structures_DataGrid is the most complex example of this de-coupling. The technical capability was not present before PEAR version 1.4.9 to handle this properly, and so most packages have not yet implemented this functionality.

The social issues are a real problem. I plan to make this my primary purpose as the PEAR president, separate from my work as a developer. We'll see how it goes, but I see a strong momentum shift in the makings here, which has been my dream for several years. It's exciting to see it moving forward so dramatically. I hope that more developers who see the potential and the things that can happen will join, it is only through the force of the will of people who believe in things like civility that PEAR will shift its culture to one that encourages innovation as well as stability.

One thing you can currently depend upon in PEAR that will never be changing is that packages marked stable will always work in a similar way, such that you won't find applications based on them suddenly breaking on upgrade. Innovation can always be provided without destroying the work of your users. Stay tuned for details of how things will be changing.

One of the paths that PEAR is actively recruiting developers for is converting a few key packages into new PHP5+ based code  (this means you). For example, HTTP_Request is a commonly used package that needs to be re-coded for PHP 5. We need a good database abstraction layer for PHP 5, and the PEAR Installer is almost ready for its next incarnation (codename Pyrus), details are on the roadmap (yes, another new feature of PEAR) at http://pear.php.net/bugs/roadmap.php?package=PEAR for version 2.0.0, and code in CVS at http://cvs.php.net/viewvc.cgi/pear-core/PEAR2. If you see a feature that you would like to implement, grab yourself a new account, assign it to yourself and upload a patch (yes, another new feature of PEAR).  If you do not already have an account, head on over to http://pear.php.net/account-request-vote.php.  This form can be used to get an account for voting in general elections as well as opening and commenting on bugs or uploading patches (yes, another new feature of PEAR).

It's a good time to get involved with PEAR, there's a lot of renewed vigor, and no ceiling in sight.