I've recently become enmeshed in developing Pyrus, PEAR's next generation installer (code coverage is moving up, 20% more coverage than my last blog post). This blog post details my journey in making the process of verifying the correctness of the development via testing and code coverage smoother, exponentially faster and more robust as a result.
One of the problems I found when designing the new code for PEAR 1.4.0 (back in the day) was that it was very difficult to determine whether changes would break things. The main problem revolves around the colossal size of the test suite. Because of the complexity of some of the tests (they must download files from a fake internet and then install them into a fake registry and then verify it actually works), the suite takes between 15 and 35 minutes to run, depending on your hardware configuration and PHP version.
This is a real problem when trying to develop with any kind of flow. If, after every change, one needs to sit through 35 minutes of tests, one will never develop anything of substance. Thus, typically I ran a subset of the tests, develop quickly, and then do a run of the entire test suite to make sure there are no unintended breaks in the other code. In addition, I had to simply assume that new tests I design actually execute the code in question, as it is not always possible to verify this directly in a passing or failing test.
The obvious solution here is to generate a code coverage report and go by that. This, however, still does not fix the problem of accidentally not running tests that are affected by a change, and generating a code coverage report for the entire project is even more time consuming.
What to do?
There are a couple of solutions. One is to set up a continuous integration server, and simply do minimal testing at home, and then commit-and-hope, fixing issues as they appear. Although this is a process that is used at php.net very successfully (see gcov.php.net), it doesn't fit my needs: I need to know right away whether I am testing the code I intend to test, and whether I am breaking anything else.
The solution I am toying with is a step beyond what PHPUnit has to offer currently (yes I am aware of other testing frameworks, for all of you doubters out there, and have nothing against them except that I personally find my development to be more lithe with the phpt test style). PHPUnit offers xdebug code coverage reports, and an option to do some fancy logging, to xml and to a database.
PEAR's run-tests command offers xdebug code coverage raw data, saved as a var_export()ed array, but no reporting. PHP's buit-in run-tests.php runner for phpt tests does not offer xdebug code coverage at all.
Neither of these solves the problems I mentioned above. What I dream of is a program that can automatically determine what files - test suites or code - have been modified, and automatically run tests that are affected by the changes, and then create a coverage report that I can view instantly.
Well, the dream is now a reality. Using a combination of the old and new, it is now possible to do near-instantaneous super-targeted test execution and instant code coverage reporting.
There are 4 simple steps and 1 bigger step involved in making this a reality:
- install PHP 5.3 or newer from either the recent RC2 (or stable release, if you're reading this in the future) or from CVS
- install PEAR 1.8.0 or newer
- svn checkout http://svn.pear.php.net/PEAR2/all
- copy all/Pyrus_Developer/src/Pyrus/Developer/CoverageAnalyzer/pear2coverage.phar.php to a web directory
Once you have done these four, you need to grab test-modified.php from http://cvs.php.net/viewvc.cgi/pear-core/test-modified.php?view=co&revision=1.4&content-type=text%2Fplain and change the paths for $codepath and $testpath to point at the project you are testing.
Next, run test-modified.php --norender. This will run the phpt tests, and create a pear2coverage.db sqlite3 database in the tests directory.
Finally, browse to pear2coverage.phar.php in your browser, enter the /full/path/to/pear2coverage.db and voila - a fully working instant coverage report.
This setup is the most involved part. As you develop, when you're ready to run tests, simply re-run php test-modified.php --norender, and the tests will be re-run, re-processed, and you will have a full coverage report available within about a minute's time.
I've been using this to develop Pyrus (located in your all checkout at all/Pyrus), and it's been great.
I have plans to examine the feasibility of processing the XML files that PHPUnit outputs, as this would allow generating coverage reports from phpunit-based tests as well. Unfortunately, I'm not sure the database format PHPUnit supports is simple enough to work for my needs, as it has much more detail (such as which classes/methods are declared in files), in order to support more abstract coverage reporting. Thus, I'm not likely to add support for that into this coverage reporter, unless I can be convinced otherwise.
This also demonstrates the phar extension's power. The web application is entirely contained within a phar archive, but web requests can be made for individual files directly, thus the generated html requests a .css file from within the phar, and it is served up by ext/phar with the correct mime type without any additional programming.
It's an exciting time to be developing in PHP, if you ask me.
