I've had little progress this week, and couldn't update my blog last week, due to final exams of my final semester. But since the exams are about to end, I think at least I can document my previous to last week's progress.

I analyzed a good number of python C modules using gcov manually. The results seem good and encouraging and gcov seems to be the right tool for the coverage. The fact that it requires modules to be linked in statically into Python is a bit of a trouble, but after having a detail discussion with my supervisor, Seth, we decided on assuming the following two points for my later work on integrating the gcov reports into figleaf:

1. The user has already compiled Python with modules statically linked in
2. Proper arguments were used with gcc during the compilation process so to let gcov track the coverage.
   

Lots of manual work during the past weeks. It is actually time to start coding all of it, and I've to start first with letting figleaf take care of the C code coverage report generation too. I've been looking through the figleaf code to figure out how to best achieve that goal, and will document any changes made here on the blog.

In the meantime, I've also started to look into improving the test coverage by writing new test cases. I found the unittest module to be interesting to start with as it has around half coverage, and would be interesting writing unit tests for the unittest module itself.

Gcov is a very handy tool. I've been using it for a while now to manually trace the C code coverage of different Python 2.6 modules.

Gcov Usage:

To use gcov with a C source file for code coverage, you have to compile your code with GCC and pass it two arguments at the compile time, "-fprofile-arcs -ftest-coverage". For example if you have a source file named "main.c", you would compile it as:

$gcc -o main -fprofile-arcs -ftest-coverage main.c

The compiler will compile your source file and produce another file named "main.gcno". Now run:

$gcov main

You will see that gcov will report 0% code coverage as you haven't run the compiled code yet. Run the compiled program and then use gcov for code coverage report again:

$./main
$gcov main

Depending on how much code in your program is executed, gcov will report the code coverage percentage, and will also produce two new files named "main.c.gcov" and "main.gcda". "main.c.gcov" is the file that holds the record of what lines were executed in your source file and how many times.

Using Gcov with Python:

There is a bit of a problem with Gcov. You need to statically link all of your code into your program in order to get code coverage analysis on it. Python doesn't do that automatically, and there is no easy configure switch to do it. So you have to manually play with the configurations to get different modules statically linked in. To compile a module statically, one way to do is to copy its entry from Modules/Setup into Modules/Setup.local, and put it under a "*static*" heading (without the quotes).

Lets say you want to compile the "mmap" module statically. First just start your python interpreter and import mmap. Now check for the __file__ attribute of the module:

>>>import mmap
>>>print mmap.__file__

The interpreter will print the module's location on the hard drive from where it was imported. Lets compile this module statically into the python interpreter now. We copy its entry from the Modules/Setup file in the python source tree into Modules/Setup.local, and put it under a *static* heading:

*static*
mmap mmapmodule.c -I$(prefix)/include -fprofile-arcs -ftest-coverage -L$(exec_prefix)/lib -lz -lgcov

The part after mmapmodule.c is to make this module work with gcov so we could get coverage report on it. Now compile your python interpreter and look for mmap's __file__ attribute again. This time you will see no such attribute defined for the module, which means it is built in this time.

Changing current directory into Modules and listing files will show that gcov has produced its data files for the mmapmodule. Run gcov on it to get the current coverage reports.

$gcov mmapmodule

You might get some code coverage for it since the import of the module into your python environment runs some of the initialization code. Had you run gcov on it without importing it first, you would have got 0% code coverage. Here is what I get:

File '/usr/include/sys/sysmacros.h'
Lines executed:0.00% of 6
/usr/include/sys/sysmacros.h:creating 'sysmacros.h.gcov'

File '/usr/include/sys/stat.h'
Lines executed:0.00% of 12
/usr/include/sys/stat.h:creating 'stat.h.gcov'

File './Modules/mmapmodule.c'
Lines executed:0.00% of 476
./Modules/mmapmodule.c:creating 'mmapmodule.c.gcov'
./Modules/mmapmodule.c:cannot open source file

Now lets run the mmap testing code and see how much of the actual code this test suite exercises:

$./python Lib/test/test_mmap.py -v
$cd Modules
$gcov mmapmodule

File '/usr/include/sys/sysmacros.h'
Lines executed:0.00% of 6
/usr/include/sys/sysmacros.h:creating 'sysmacros.h.gcov'

File '/usr/include/sys/stat.h'
Lines executed:0.00% of 12
/usr/include/sys/stat.h:creating 'stat.h.gcov'

File './Modules/mmapmodule.c'
Lines executed:71.64% of 476
./Modules/mmapmodule.c:creating 'mmapmodule.c.gcov'
./Modules/mmapmodule.c:cannot open source file

71.64% of the module is exercised by the test suite.

What's Next:

I have traced the code coverage of a number of modules by now. Next is to look for a way to easily compile a major number of modules statically into the Python interpreter so to easily get an automated code coverage report on it. This can be done by uncommenting appropriate entries for the modules in Module/Setup file, but I've been getting into a few problems with it right now. Solving that is the first task, and then the part of making figleaf do all of this automatically.

I've been discovering Gcov for the past week. Gcov is a C code coverage analysis tool developed by GNU. And its fun to use. :)

Well, I tried to do code coverage for a few modules. I started with zlib, well, because I found out someone had already used Gcov for its C coverage. Seems like Gcov isn't able to do code coverage for dynamically linked modules. So recompiling Python with statically linked modules is the first step for C coverage of the modules using Gcov. In case of zlib, it was easy meat. But then I started searching for ways to build whole of Python statically, with modules linked in statically as well. There is no supported way in Python build system for that, but a bit of manual hackery can be used to achieve the goal. You need to specify the modules you want to link statically in Modules/Setup.local. I need to do more research to come up with an efficient way of doing this.

Also found out a nice extension for Gcov called Lcov that can be used to generate nicely formated HTML reports of the coverage analysis.

I had a nice discussion with my project supervisor, Seth Lemons, a day back, during which we came up with a more clear strategy of how to go about achieving our goals (also thanks to Titus for his feedback). This also lead to a few modifications to the initial plan that I had proposed at the time of proposal submission. I think it's a good idea to document the strategy we came up with so we have it as a reference here to point to later on:

1. I am going to start with making C coverage analysis work for Python 2.6. The generic C coverage tool I've decided to use for this purpose is Gcov. I think it makes sense to first manually put Gcov to use with Python for the coverage analysis.
2. Next comes making figleaf generate a combined C + Python code coverage analysis report, by integrating the C coverage using Gcov with figleaf.
3. The part up till this point is for Python 2.6 version. So to migrate it to Py3k, I'll need to port figleaf to Py3k. A port of figleaf for Py3k is already available in a somewhat unmaintained form, developed by the author of figleaf (Titus Brown) himself. So I hope this task shouldn't be a tough one.
4. Once the above parts are completed, I can move to making the C code coverage working with Py3k again using Gcov.
5. A goal of the project was to increase the code coverage by writing new test cases as well. This task can go on in parallel during all of the above four phases of the project.

The decision to start with Python 2.6 before moving to Py3k was stimulated by two reasons:

1. Python 2.6 is still widely used so community can gain from the work performed for this version. Plus whatever is done for 2.6, can than be easily migrated to Py3k.
2. figleaf is currently stable for Python 2.6, so to start with the actual work instead of starting with porting figleaf to Py3k, it only makes sense to to use the current figleaf for Python 2.6 and use it for it's coverage analysis.

I haven't got much experience with Gcov, infact this is going to be my first time using it. So it is hard to estimate the time durations for the project milestones. But I hope once I get started with it, I'll be in a better position to judge the time constraints on the individual tasks.

...officially on 23rd May. I haven't had a head start yet though, mainly due to my college activities (final year project, presentations, mids etc), plus also because I was having discussion with my mentor and the community to get a good idea of what would be the right way of going through with my project, before I actually begin to write the code.

I've been testing different code coverage tools for Python in the past couple of weeks. The ones I specifically tested were coverage.py, Pycoco, and figleaf. All are actually pretty easy to use tools generating good reports. Though I found Pycoco to be slower mainly due to it downloading its own copy of Python and doing code coverage analysis on the downloaded source.

All of the above tools perform code coverage for Python 2.x, though figleaf has been ported by its create (Titus Brown) to Py3k, but I guess it pretty much in early stage. I haven't had it tried out yet but that's something on my priorities list.

Gcov is a code coverage tool that can be used with GCC to perform code coverage analysis on your C code. I'll be using it for C code coverage for Python 2.6. Later I'll extend it to Py3k.

I'll put more details on the blog and update regarding the development work, in the coming days.

GSoC2009's formal starting date for accepted students to start coding their projects is 23rd May. But to be actually able to start *coding* on that date, it is essential to know the tools and environment that you are going to be using and working on. Also important is to get a clear idea of what actually you will be writing the code for, what features your coded project will have, and how you plan to go about implementing them, making sure that your mentor and mentor organization are also clear on your goals and motives. So that's why the period till 23rd of May is called "The Community Bonding Period".

I'll be working with Python Software Foundation (PSF) to improve and analyze the code coverage of Python3k. I've already checkedout a local copy of Py3k's svn repository to get to know its directory structure and how things are managed. I've been on Python-dev mailing list for a long time, but it is time to start paying more attention to the discussions carried out on it. Also subscribed to Python-3000 and Python-ideas mailing lists.

An important resource for all the students working with PSF this year is the Python Developers Guide. It links to many resources for starting developers and reference material for a quick look up if you are stuck somewhere.

Python3k test suite is located in the Lib/tests directory. You can run all of the tests by running the command "./python Lib/tests/regrtest.py". Interestingly, two tests failed for me, test_distutils and test_socket. Running in verbose mode with the -v argument showed that distutils test was having some permission problem with the directories it created on my machine's /var/tmp directory. Interestingly, if I run the test as a root user, there is no problem as root doesn't need any kind of permissions to access a specific directory. But I am not supposed to run the tests as a root user. I've reported the problem on the Bug Tracker and hopefully the problem will be solved. The socket test had a weirder problem and I haven't dug it up yet.

Two important tools for python code coverage are figleaf and coverge.py. I'll be giving both the tools a detailed look because my summer project also involves comping up with a way for better report generation for Python code coverage.

Thanks to the folks on #python-dev on irc.freenode.net for listening to my questions regarding the source tree with patience. I hope to bug you guys even more, but only when it is necessary. ;)

Lots of students that didn't make it this year into GSoC are requesting for access to accepted students' proposals so they can have a look at what the accepted proposals looked like. So I thought I would post my proposal here.

Title: Code coverage analysis of and improvements to Python3k Core's testing framework
Student: muhammad shuaib khan

Abstract: This project proposes improvements to and extension of the Python3k testing framework by adding more test cases for increased code coverage, integration of the C code with the Python code coverage, and easy to generate integrated report.

Content:

GSoC Proposal
Name: Muhammad Shuaib Khan
Email: aries.shuaib@gmail.com, shuaib.khan@niit.edu.pk
Phone: +923435286646

Title

Code coverage analysis of and improvements to Python3k Core's testing framework

Synopsis

Testing phase of any big software project is critically essential in order to ensure flawless execution of the tool in real world enterprise environment. Python is no exception, and being a widely used programming language to code programs that are of critical importance to organizations, it is even more essential to have a solid testing framework for the language which is easy to use, improve, extend, and covers all the aspects of the core code. Python3k has a testing infrastructure that needs to be extended and improved. This would prove to be useful for code coverage analysis for both experienced core developers and new developers to ensure that the changes they have made to the code didn't break other pieces of the software.

Benefits to Community:

The need for a solid testing framework for Python interpreter and its branches is something that has been getting a good amount of attention in the recent times. Python3k has a relatively well defined test-cases development methodology as mentioned in the document http://svn.python.org/projects/python/branches/py3k/Lib/test/README. But it is important to notice that not all code gets tested by the testcases that are already laid out. Measuring the code coverage both of the Py3k C code and the python code, and getting the code coverage percentage to a higher numeral by writing new tests for the standard library and the core itself will be part of this project. The community will greatly benefit from this as they'll have an easy to use testing framework to ensure that their code hasn't broken down anything, and the confidence that all the aspects of the code have been tested for successful execution.


Deliverables:

The project is aiming to improve the code coverage of Py3k by writing new tests, and make it easy to analyze the code coverage achieved. Keeping the main motive in mind, the following deliverables can be chalked out:

1- Study the existing tests implemented and identify parts/modules that need improved testing.
2- Write new test cases to improve the code coverage.
3- Do the test coverage on Linux and Windows platforms. (Would be nice if done for Mac OS X as well, but I do not have access to this platform)
4- Integration of the C Code coverage with the Python code coverage, and generating an integrated report.

Description:

Whenever new code is added to Python as a module or as functionality to an existing module, it is of essence to add the corresponding test code as well that tests the added functionality for situations where the code might be used in ways the programmer didn't intend it to be used. But this is not the way things get done always and there is a good amount of code in the Py3k that needs test cases. As a GSoC project, I intend to improve the situation by analyzing the code coverage and add more tests for the C code and the python code, integrate the tests for both, and provide easy to generate integrated report mechanism

The guideline for test additions documents three different ways of adding the test cases. The unittest bases tests, doctest based tests, and the traditional way of test cases that uses hackish ways of testing such as comparing the output of functionality to expected output by comparing the two and looking for mismatches.

Related work:

Professor Titus Brown (http://ivory.idyll.org/about.html) was kind enough to point me to some existing documentation related to this project. Brett Cannon has been writing invaluable posts on his blog (http://sayspy.blogspot.com/) about Python test coverage and ways of improving it.

Biographical Information:

- I am undergraduate Computer Software Engineering student at National University of Sciences and Technology (NUST) - Pakistan.
- I was selected as a Summer Student for CERN Openlab in Geneva, Switzerland (http://tinyurl.com/caapau), where I worked on developing a testing framework in Python for a Linux performance monitoring tool called "Perfmon".
- I have been writing about open source software and have been published numerous times on Linux.com
* Lguest: A simple virtualization platform for Linux (http://www.linux.com/feature/126293) Feb 20, 2008
* Chess engines for Linux (http://www.linux.com/feature/60859) Mar 22, 2007
* CLI Magic: Linux troubleshooting tools 101 (http://www.linux.com/feature/60136) Feb 19, 2007
* A survey of open source cluster management systems (http://www.linux.com/feature/57073) Sep 21, 2006
* Setting up a Condor cluster (http://www.linux.com/feature/56747) Sep 01, 2006
- I've won a number of national software project and programming competitions.
- I'm also a Teaching Assistant (TA) at a national university where I conduct lab sessions of undergraduate students for Fundamentals of Programming and Object Oriented Programming courses.
- At my home institute, I've also been involved with a research group that targeted the development of an operating system with built in Grid management facilities.

This is my last semester of my bachelors degree and I've less courses to take. Only one and a half hour of lectures on weekdays on average. This gives me the opportunity to dedicate suitable time to my GSoC project. I'm familiar to all the major communication ways on the Internet and can be found on IRC, Skype, IM, Twitter. I've a good response time on the email.