Some hacking

I wanted to work on forgeplucker http://home.gna.org/forgeplucker/.

Stay focused on one project at a time. Break the project down into tiny chunks. -- ZenHabits.net

Setting up to hack on forgeplucker.

I saw esr's post announcing ForgePlucker and I went ahead a signed up for the mailing list, made an account with gnu gna, and went on irc (forgeplucker on irc.freenode.net) to say hi. One should note that while I used IRC a lot in the 1993-1997 span, I had nearly forgotten all the commands. I also didn't want to use mIRC illegally so I looked around and found irssi for debian stable. It took me a little while to figure out, but I got it working.

I noticed the docs are written in asciidoc. I'll have to check that out, but I already like it..

I also checked out the working code and noted it requires python 2.6, which, incidentally, is not installed on my debian stable. Arrghh! Not to be deterred, I elected to download Sun's VirtualBox and a netinstall CD of debian stable which I was going to upgrade to unstable, but somewhere in the debian install something fubared and I realized I should bite the bullet and just build python2.6 on the server.

I wanted to contribute 'something' so when I saw a bunch of typos in docs/handler-howto.txt I decided to fix a few. Then I made a patch file. Now: how to get out of the revelworks server? Unix 'mail' tool doesn't make send files obvious. So where to put it? I simply decided to put it on this server. Then I can get it from gmail and email it as an attachment. Until I figure out a better way.

Here it is: handler-howto.txt.patch.

Saturday, November 7, 2009. 2:48 AM.

Chatting with 'ptd' (He's in the UK) about adding bugs here http://sourceforge.net/tracker/?atid=1215676&group_id=286911&func=browse, the bug tracker for ForgePluckerTest at sf.net. And I entered a couple of bugs.

Python2.6 on Debian Stable

4:24 a.m. I still need to go compile python 2.6 on the revelworks.com server. Doing the tar vxfz Python - ./configure - make - make altinstall dance. Look at that! A bunch of stuff missing:

Now it's the next day, November 8 2009 at 2 am and I'm getting back to this. I tried to look for _bsddb which led to bsddb3... I went to comp.python.org... Confused!

So I searched for Berkeley db in apt-cache and found then apt-get install package db4.6-util. Gonna remake to see if anything happened.

Running ./configure. Makefile was created.

2:46 am. running make.

        Failed to find the necessary bits to build these modules:
        _bsddb             _tkinter           bsddb185
        bz2                dbm                dl
        gdbm               imageop            readline
        sunaudiodev
        To find the necessary bits, look in setup.py in detect_modules() for the module's name.
        

I go look in setup.py and I find:

exts.append(Extension('_bsddb', ['_bsddb.c'],
                      depends = ['bsddb.h'],
                      library_dirs=dblib_dir,
                      runtime_library_dirs=dblib_dir,
                      include_dirs=db_incs,
                      libraries=dblibs))

Wait, I need _bsddb.c. let me go look!

revelworks1:~/python26/Python-2.6.4# find . -name *bsddb*
./Lib/test/test_bsddb.py
./Lib/test/test_bsddb185.py
./Lib/test/test_bsddb3.py
./Lib/bsddb
./Doc/library/bsddb.rst
./PCbuild/_bsddb.vcproj
./Modules/bsddbmodule.c
./Modules/_bsddb.c
./Modules/bsddb.h
./PC/VS8.0/_bsddb44.vcproj
./PC/VS8.0/_bsddb.vcproj
./PC/VC6/_bsddb.dsp
./PC/VS7.1/_bsddb.vcproj

Wait another minute! I see _bsddb.c and bsddb.h right in the /Modules/ directory!

Back to setup.py

Let's make a long story short: there is something fishy with python 2.6 installation on Debian stable.

4:40 a.m.

I went and found out about the python buildbots. They did not include Debian stable. It is possible that there is a problem with building python 2.6 on debian stable.

Sunday, November 8, 2009. 10:26 pm.

VirtualBox

I am redoing the install of debian with virtualbox. Need a playground. Install crashed. Tried with Microsft Virtual PC 2007. Crashed too. Redownloaded debian netinst iso. Closed all programs and retried with virtual box. Seems to be working now. Whew! (turns out I was out of ram--netbook)

I tried to see how to ssh into that machine. Can't figure it out.

Oh well, I'll just work locally. apt-get install subversion

1:30 am. checked out python 2.6 via svn. So far so good. The first ./configure gave me a "no compiler found" or some such.

alt-get install gcc.

./configure again. Waiting. lots of stuff scrolling past.

make. Not found.

apt-get install make

make. Woa, lots of stuff scrolling past!

(As an aside, I really would like to be doing this through putty. It's not easy to cut and paste from the vm.)

ok. getting some output:

Failed to find the necessay bits to build these modules:
_bsddb             _curses             _curses_pane
_hashlib           _sqlite3            _ssl
_tkinter           bsddb185            bz2
dbm                gdbm                readline
sunaudiodev
To find the necessay bits, look in setup.py in detect_modules() for the module's name.

It's now 2:09 a.m.

apt-get install libssl-dev to satisfy the _ssl requirement

apt-get install libsqlite3-dev to satify the _sqlite3 requrement.

Ok, gonna see if anything changes.

make clean - 2:22 am - ./configure - 2:29 am - make 2:44 am.

Failed to find the necessay bits to build these modules:
_bsddb             _curses             _curses_pane
_tkinter           bsddb185            bz2
dbm                gdbm                readline
sunaudiodev
To find the necessay bits, look in setup.py in detect_modules() for the module's name.

Oh, _hashlib went away too. Nice.

apt-cache search sunaudiodev returns nothing.

STOP THE PRESSES! A moment at google reveals I am not the only person with this problem and someone else went through the same thing! See: http://blog.pythonaro.com/2008/10/horrible-hack-to-get-python-26-on.html!

3:00 am. I checked and all the items on line 2 are found with apt-cache search. So far so good.

apt-get install tk8.4-dev libgdbm-dev libdb-dev libreadline-dev libncurses-dev

3:06 am make

Failed to find the necessay bits to build these modules:
bsddb185            bz2                 sunaudiodev
To find the necessay bits, look in setup.py in detect_modules() for the module's name.

Well, that got me closer. It's 3:09 am. I'm going to bed.

Tuesday, November 10, 2009. Just listened to NPR news. President Obama just gave the eulogy for the soldiers killed at Fort Hood, TX.

I found another link on the web that shows how to get more python built on ubuntu. His recommendation is to install the python2.5 build dependencies thus:

apt-get build-dep python2.5

(note that build-dep builds a package dependencies.)

Woa, that's installing 192 packages! Let's see how that works. But to be honest, I think that was an overkill.

Also, he advocates patching setup.py to ignore a couple of dependencies. I don't really like that. There's a reason they are dependencies. Who knows what breaks if they are not there?

apt-get is still running, so in the meantime I went and asked a question at stackoverflow.com: Debian2.6 on Debian stable, where should the executable go? let's see what they say.

1:16 pm. Ok, apt-get completed. Now, doing make clean.

Running ./configure.

1:29 pm. make.

Lots of interesting (and conflicting) answers at stackoverflow.

Dangit, who knew compiling python on a virtualbox debian vm on a acer AspireOne netbook would be so slow. Oh wait...

1:45 pm. done makeing. Still unable to detect these two modules: bsddb185 and sunaudiodev. What to do now?

According to http://docs.python.org/library/sunaudio.html "Deprecated since version 2.6: The sunaudiodev module has been deprecated for removal in Python 3.0."

It does look like these two modules are old and deprecated and need not be installed. So I will apply a patch to get rid of these.

2:29 pm. Ok. patch applied to setup.py. Gotta go pick up Viggo.

6:51 p.m. Back from my fatherly duties.

make install then a lot of on-screen messages!

I wonder if there is a way to use paramiko to point to a debian install and do all this automatically.

Success! make install worked!

Python 2.6.4+ (release26-maint:76172M, Nov 10, 2009, 06:54:46)

Now all I've got to do is identify which packages I need to really install instead of the build-dep thing.

SSH into virtualbox

Wednesday, November 11, 2009

I want to figure out how to ssh into the virtualbox...

Thursday, November 12 2009, 2:09 am.

After much googling, I found this link: ssh into virtualbox guest but the instructions were strange and I had to go dig into the source code for the missing bits, and I had to make some changes upon copy/paste.

So I am going to rewrite the instructions from the link above, taking care to htmlize the brackets:

Edit C:/Documents and Settings/<your user name>/.VirtualBox/VirtualBox.xml

In between the tags <ExtraData> and </ExtraData>, You'll find there are already some ExtraDataItems. You may leave them there. Insert the following ExtraDataItems by copying and pasting the following text:

<ExtraDataItem name="VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort" value="2222" />
<ExtraDataItem name="VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort" value="22" />
<ExtraDataItem name="VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol" value="TCP" />

Close and reopen VirtualBox and boot your guest OS.

Then I pointed putty on my windows machine to 127.0.0.1 (port 2222) and after accepting the rsa key, I was in business! Thank you Charles Cavanaugh!

Now, I'm going to go to bed. Tomorrow. I want to redo it all to make sure I documented it right. Also, I want to try to isolate which of the packages installed during build-dep I actually need.

I also noticed that this pointed python to python2.6 and that's not what I wanted, so I probably need to use make altinstall.

12:55 pm on Thursday.

Just created another debian virtualbox guest. Installing Debian stable now.

Debian install complete at 1:17 pm. So 23 minutes total.

apt-get update

apt-get install ssh to get sshd running.

SSH works from putty at 127.0.0.1:2222

Streamlining building Python on Debian stable

Ok, so I look at apt-get build-dep python2.5 and see what it does.

Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  autoconf automake automake1.9 autotools-dev binutils blt blt-dev
  build-essential bzip2 cpp cpp-4.3 debhelper debiandoc-sgml defoma doc-base
  dpkg-dev dvipdfmx emacs22 emacs22-bin-common emacs22-common emacsen-common
  fontconfig-config g++ g++-4.3 gcc gcc-4.3 gettext ghostscript gs-common
  gsfonts gsfonts-x11 html2text intltool-debian lacheck latex-beamer
  latex-xcolor lesstif2 libasound2 libbluetooth-dev libbluetooth2 libbz2-dev
  libc6-dev libcompress-raw-zlib-perl libcompress-zlib-perl libcups2
  libcupsimage2 libdb4.5-dev libdigest-hmac-perl libdigest-sha1-perl libdrm2
  libexpat1 libffi-dev libffi5 libfile-remove-perl libfontconfig1 libfontenc1
  libfreetype6 libfreezethaw-perl libgdbm-dev libgif4 libgl1-mesa-glx
  libgmp3c2 libgomp1 libgs8 libhtml-parser-perl libhtml-tagset-perl
  libhtml-tree-perl libice-dev libice6 libio-compress-base-perl
  libio-compress-zlib-perl libio-stringy-perl libjpeg62 libkpathsea4
  libmail-box-perl libmail-sendmail-perl libmailtools-perl libmime-types-perl
  libmldbm-perl libmpfr1ldbl libncurses5-dev libncursesw5-dev
  libobject-realize-later-perl libpaper-utils libpaper1 libpng12-0 libpoppler3
  libpthread-stubs0 libpthread-stubs0-dev libreadline5-dev libroman-perl
  libsgmls-perl libsm-dev libsm6 libsp1c2 libsqlite3-dev libssl-dev
  libstdc++6-4.3-dev libsys-hostname-long-perl libt1-5 libtext-format-perl
  libtiff4 libtimedate-perl liburi-perl libuser-identity-perl libuuid-perl
  libx11-dev libxau-dev libxaw7 libxcb-xlib0-dev libxcb1-dev libxdamage1
  libxdmcp-dev libxfixes3 libxfont1 libxft2 libxi6 libxinerama1 libxmu6 libxp6
  libxpm4 libxrender1 libxt-dev libxt6 libxtst6 libxv1 libxxf86dga1
  libxxf86vm1 linux-libc-dev lmodern lsb-release make pgf po-debconf
  preview-latex-style prosper ps2eps psfontmgr sgml-base sgml-data sgmlspl
  sharutils sp tcl8.4 tcl8.4-dev tex-common texlive-base texlive-base-bin
  texlive-base-bin-doc texlive-common texlive-doc-base texlive-extra-utils
  texlive-fonts-recommended texlive-fonts-recommended-doc
  texlive-generic-extra texlive-generic-recommended texlive-humanities
  texlive-humanities-doc texlive-latex-base texlive-latex-base-doc
  texlive-latex-extra texlive-latex-extra-doc texlive-latex-recommended
  texlive-latex-recommended-doc texlive-pictures texlive-pictures-doc
  texlive-pstricks texlive-pstricks-doc texpower texpower-manual tipa tk8.4
  tk8.4-dev ttf-dejavu ttf-dejavu-core ttf-dejavu-extra x11-utils
  x11proto-core-dev x11proto-input-dev x11proto-kb-dev xaw3dg xbitmaps
  xfonts-encodings xfonts-utils xml-core xpdf xpdf-common xpdf-reader
  xpdf-utils xterm xtrans-dev zlib1g-dev
0 upgraded, 192 newly installed, 0 to remove and 0 not upgraded.
Need to get 304MB of archives.
After this operation, 664MB of additional disk space will be used.
Do you want to continue [Y/n]? Abort.

Oh damn, 664 Megs? And all that to satisfy the bz2 dependency? Double damn. Especially since a quick search for bz2 reveals that there is libbz2-dev item in there that looks like it might just work. I'll try that.

apt-get install libbz2-dev

Getting python2.6 source

Ok, we're going to go get the latest python 2.6 from svn trunk. Actually, no. We're going to get the latest stable 2.6.4 straight from python.org.

mkdir pythontemp
cd pythontemp
wget http://python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2

Output, for those who have never done that before:

--2009-11-13 02:12:48--  http://python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2
Resolving python.org... 82.94.164.162, 2001:888:2000:d::a2
Connecting to python.org|82.94.164.162|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11249486 (11M) [application/x-bzip2]
Saving to: `Python-2.6.4.tar.bz2'

100%[======================================================================================================>] 11,249,486   240K/s   in 56s

2009-11-13 02:13:45 (197 KB/s) - `Python-2.6.4.tar.bz2' saved [11249486/11249486]

Now, we decompress.

tar xjvf Python-2.6.4.tar.bz2
tar: bzip2: Cannot exec: No such file or directory
tar: Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error exit delayed from previous errors
debian:/pythontemp#

Oh, wait, we need to install package bzip2.

apt-get install bzip2

Now, let's decompress.

tar xjvf Python-2.6.4.tar.bz2

Ok, now we should have a Python-2.6.4 directory. Let's get in there: cd Python-2.6.4

Ooops, it's 2:30 am and I have a ton of stuff to do tomorrow. Good night!

Took a Breather

I skipped a few days. Ok, three weeks. But at least now Thanksgiving is out of the way.

I want to start looking at getting virtualenv working, as well as testing with nose.

Oh, and Distribute too.

I also want to look at python-dateutil.

Lastly I want to look at fabric + paramiko.

Ok. Back to python

So back to the directory ~/pythontemp/Python-2.6.4

apt-get install make

apt-get install gcc

apt-get install libssl-dev

apt-get install libsqlite3-dev

apt-get install tk8.4-dev libgdbm-dev libdb-dev libreadline-dev libncurses-dev

apt-get install libbz2-dev

Ok, now for the patch.

wget http://www.lysium.de/sw/python2.6-disable-old-modules.patch

This should have downloaded file python2.6-disable-old_modules.patch.

now apply the patch file:

Make sure you have patch on your system first.

apt-get install patch

Ok, now apply the path.

patch -p1 < python2.6-disable-old-modules.patch

Now we are ready for the ./configure make make install dance, but we're going to use checkinstall since we're going on Debian. I'm following instructions from this article: Make your own packages for Debian-based systems.

Let's install it.

apt-get install checkinstall

And now, let's do the python 2.6 install proper.

./configure

Great, no error. 5:01 am

make

Again, no error.

checkinstall

When asked "should I create a set of package docs" enter yes.

Description "python2.6". Note that this should be lowecase. Checkinstall will drop case anyway thought.

Pressed enter on the "press ENTER to create package" bit.

6:00 am. process complete. It says I can remove the package by issuing command dpkg -r python2.6 and that the new package is now at /pythontemp/Python-2.6.4/python_2.6.4-1_i386.deb

Let's try:

debian:/pythontemp/Python-2.6.4# python2.6
Python 2.6.4 (r264:75706, Nov 30 2009, 21:12:06)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

It's alive!

Damn I'm still awake. Better go sleep 3 hours before Viggo wakes me up.

Document Status

Last updated January 13, 2010.