I was thinking about another Kerala trip as my friend and NIT Calicut classmate Pramod invited me for his marriage on 1st May. Since it is a long trip of more than 24 hours from Pune to Kerala by bus, I thought I should plan some Free Software talk so that I have more motivation for such a hard trip. I was very happy to see great enthusiasm for gnome release party on my last visit.
I asked a few people if it would be possible to organize something on such a short notice. I was happy to get a positive response and we fixed the event for 28th and 29th at MES college of Engineering, Kuttippuram. But I can reach there only on Sunday if I leave Pune on Friday and take one day transit in Bangalore (I take this option because I get to meet my friends during day and I have to travel only at night). I asked around if someone can volunteer to take some basics session on Saturday so that particiants will be ready for the packaging session on the next day.
Nakul readily volunteered to take command line and shell scripting sessions. Since Ershad was coming, he agreed to take Free Software introduction. I have to mention his great grassroots Free Software promotion work here. Though he doesn’t post much about it on smc or plus lists I came to know about it from people who attended the sessions. (A request to Ershad to blog and talk about it on mailing lists). Since Labeeb was also there in Kerala he agreed to introduce Debian.
Haris volunteered to take a session on building from source. He is very enthusiastic and he writes beautiful blogs. Read his account of the two days here.
Since I was not present on day 1 you can read about it from Haris’ blog. Raju sir told me I could stay with him and I went to his place directly on Sunday morning. I reached Mannuthi at 3.45 am and he picked me up. I slept a bit and after breakfast we came to MES. One thing I really liked there was the fluroscent moon and star (I guess there were planets too) stickers on the wall! You feel like you are watching the sky at night!
We reached MES by 10.20 and they had already started discussing about personal privacy and tracking on the internet. I started with reading a note I wrote the previous night in the bus.
I asked them about what they learned on the previous day and I asked them a few questions like why we need to package software, who creates packages etc. I started discussing basic ideas around a package’s lifecycle from upstream tarball to a stable release. A few points I covered include “how and when a package first enters debian. Concepts of RFP, package maintainer, team maintained packages etc were covered.”
There are many Free Software projects out there and role of a distribution like Debian is to provide a collection of these software in an easy to distribute and manage format. Distributions like debian make sure the software is in good condition and pass it through thorough testing before it is given out as a supported software. Since there are many distributions out there with different policies and packaging formats (deb, rpm, ebuilds etc). Also each of this distributions may be including different versions of its dependencies. So people specialising in packaging makes the job of upstream developers easier by making an easy to install version of the software available. Distributions include many Free Software but every day more Free Software is released. So who creates these debian packages? How does a person ready to create a package know there is a new software? What do one do when a software they need is not available as a package? How can one install such a software if it is not packaged?
Most Free Software projects release their work as a tarballs – a compressed file of all the source code with instructions to compile them. Many times people read about new software from news sites and some of them savvy enough start building it from source. If you are already familiar with development tools it may not be difficult to do. Some of those users may create a debian package. Sometimes people just request a software to be packaged. It is done by submitting a wishlist bug against ‘wnpp’ pseudo package. wnpp stands for Work Needing and Prosective Packages. Such requests for packages are commonly referred as RFPs. So there are two ways a package enters debian.
1. Some one who know about debian packaging creates a debian package from a source tarball provided by upstream developers.
2. Someone files an RFP in debian bug tracking system and people come to know about it via wnpp and then creates a package.
For small packages a single person may create a debian package alone and maintains it by providing newer packages when some bug is fixed or a new version is released. Sometimes there are many applications created by one project (eg GNOME, KDE) or packages are similar (for example packages written in one particular language like perl, python, ruby) or they are used in one particular area (for example software useful for medical doctors or useful in schools), in these situations a team maintains these packages.
I really like this team maintenance idea because you can help updating any package when you have time and other team members take care of it when you are busy. With teams distributed around the world most of the time there is someone available usually to take care of the packages. But more hands to help is always welcomed by the teams as it helps spread the load to more people and reduces work for each person.
“when does a package move to testing. Package priority and condition of release critical bugs were explained.”
All packages are normally uploaded to unstable branch. When a package is uploaded, their maintainer tells how important this upload is. For normal uploads a priority of ‘low’ is set and for critical fixes ‘medium’ and for security fixes ‘high’ priorities are used. Packages are kept for review in unstable branch for 10 days for low priority packages, 5 days for medium priority packages and 2 days for high priority packages.
Many people including this author use unstable distribution as their main operating system every day. When they encounter bugs they report them. I have to clarify what unstable means here, it means always changing and each software itself is stable (either the upstream developers or debian developers decide it is stable enough for every day use). For software which not stable for every day use experimental branch is used.
After 10, 5 or 2 days depending on package’s priority their status is checked (this is done by a software called britney) to see if they can be moved to testing branch. First condition it checks is if there is any release critical bug reported against this package. Second condition is its dependencies are already in testing. If both these conditions are met it will be moved to testing. So new packages keep coming to unstable and testing all the time except when testing is frozen for next release. So we can call both unstable and testing branches as rolling releases.
Every time a stable version is released goals for next stable release is set by the project after discussions with different teams. Each team may want a particular version of their software in the next release or project as a whole may decide on a new feature. These release goals are collected by release team and published early in the release cycle. When testing is very close to achieving these release goals release team declares testing is frozen.
At this point britney keeps all packages in unstable and only bug fixes are allowed into testing. Sometimes release team allows a few exceptions, but usually freeze time is spent on fixing bugs by all developers. When all release critical bugs are fixed in testing a copy of testing is released as stable. At this time normal movement of packages from unstable to testing is resumed and preparation for next stable release starts. Stable release is officially supported by the project and recieves security updates only, no new versions are uploaded into stable.
“when does it goes to stable. I told them about release goals and freeze. I told them we release when we are ready and that means zero release critical bugs.”
This time I did not take my favorite package lekhonee-gnome and instead I chose to introduce gem2deb tool. We took mixlib-log gem and easily created a deb file with just one command gem2deb mixlib-log. But there was a catch, they were running squeeze and gem2deb was not present in the normal repositories.
I just told them to install gem2deb, some people tried to install it from sid repo but failed matching correct dependencies (most of systems had a messed up sources.list file, presumably from yesterday’s apt session). Some figured they can install it from source and they downloaded gem2deb tarball from github and followed instructions in README file to build it. First application of what they learned yesterday. And dpkg-buildpackage -us -uc command given there came in very handy when they had to rebuild mixlib-log package.
Every one created a deb package with one easy command ‘gem2deb mixlib-log’. On some systems proxy was not set correctly so gem2deb could not download the gem file using ‘gem fetch mixlib-log’. So gem file was downloaded from rubygems.org and them gem2deb was run against this file.
Every one used lintian to check what is missing in packaging as per policy and they used help of New Maintainer’s Guide to fix those errors and warnings.
[TODO] chosing a new gem file to package or update to gem2deb based packaging.
(Posting it as incomplete, I may add more points and few more details later)
30/04/2012: First draft, outline
01/05/2012: Expanded on debian development stages
02/05/2012: added link to Haris’ blog, editing changes, splitted big paragraphs to small paragraphs.
03/05/2012: Link to the note I wrote added.
10/05/2012: Diagram from Debian Handbook describing the release process added.
PS: I think, I better make a book out of it because its going to be a big post otherwise
Note: I was just trying to dump my thoughts as I wrote this and I was in a hurry on the way to a friend’s wedding, plus it was typed in my almost-dead ‘first android phone’ my phone. So excuse if you see anything not in place. Thanks to Shirish for some corrections.