Hacking log

Stay focused on one project at a time. Break the project down into tiny chunks. -- ZenHabits.net

January 6 2010

1:21 am. I want to reimage the debian stable at fsckvps.

Oh damn this is rich. When going to the fsckvps control panel, I got this: "we have a small problem with our database. Please try again later." A small database problem: a huge customer problem!

1:38 am. Looking for other VPS hosting. Cheapest.

provps.com - too expensive.

micfo.com - too expensive.

stardothosting.com - too expensive.

1:44 am tektonic.net not bad at $15/month.

By the way, I am looking at http://wiki.debian.org/DebianHosting.

1:47 am. Oh, vpslink.com starts at 7.95 /month. Not a lot of specs, but enough!

1:52 am. VPSlink had Seattle and new York data centers.

vpsville.com offers $8/month vps in Los Angeles data center.

Time to close my fsckvps account!

2:09 am. Found vpsvillage. They offer vsp for $6 /month.

vpsvillage order says: Note that according to demand, orders may take an average of 3 days to process. Please contact us if you will need an expedited configuration. WTF?

3:40 am. Took a break and cleaned up the kitchen. Doing some thinking about web hosting. Maybe I need to get rid of webfaction and just keep rapidvps.

Need to cancel the fsckvps.com account.

9:33 pm. Back at the computer.

Did some work in php for arcostream.com, specifically a wizard to customize jwplayer version 5. Using version 2.2 of swfobjects instead of version 1.5 like longtailvideo uses.

Oh, and I definitely want to check out invoicemachine.com. 3 free invoices per month they say.

January 7, 2010

2:26 am. fsckvps https://94.76.232.32:5656 username: node1. Just reimaged with debian stable 5 64 bit.

3:20 am. Doing job hunting. The reimage is complete. Logged in as root. password is ok.

9:58 pm. Got call about job today. Got email from Mutual of Omaha about job too. Gonna apply to Medtronic.

Satruday January 8. 2010

Had to use ipmitool to restart a server. Don't like it one bit. The server subsequently didn't restart. It was trying to install IE 8 but that evidently didn't go well. I drove downtown LA to 500 W 7th street and saw that the server was borked. I kicked it a few times and it came back to life. Damn Windows 2003 server.

Tuesday January 12, 2010

9:00pm. To make a long story short, I spoke with the CEO of tubefilter about dong some python work for them. They use CouchDB so I'm learning as much as I can about the thing. I think also they use django so I'll probably be putting one of those here at webfaction.

CouchDB seems very nice based on this video. (From http://blog.oskarsson.nu/2009/06/nosql-debrief.html.)

Fabric

11:58 pm Gonna install fabric on my local box. Hopefully in the python2.6 location.

As an aside, did you know that jEdit does automatic html tag completion when you're in a valid xhtml1.1 document?

On my local windows XP machine, I changed to the "c:\python26\Scripts\" directory. There I issued the easy_install-2.6 paramiko command.

FUBAR!

C:\Python26\Scripts>easy_install-2.6.exe paramiko
Searching for paramiko
Reading http://pypi.python.org/simple/paramiko/
Reading http://www.lag.net/~robey/paramiko/
Reading http://www.lag.net/paramiko/
Best match: paramiko 1.7.6
Downloading http://www.lag.net/paramiko/download/paramiko-1.7.6.zip
Processing paramiko-1.7.6.zip
Running paramiko-1.7.6\setup.py -q bdist_egg --dist-dir c:\docume~1\chrism~1\locals~1\temp\easy_install-yoiulg\paramiko-1.7.6\egg-dist-tmp-pfpbfs
warning: no files found matching 'user_rsa_key' under directory 'demos'
warning: no files found matching 'user_rsa_key.pub' under directory 'demos'
zip_safe flag not set; analyzing archive contents...
Adding paramiko 1.7.6 to easy-install.pth file

Installed c:\python26\lib\site-packages\paramiko-1.7.6-py2.6.egg
Processing dependencies for paramiko
Searching for pycrypto>=1.9
Reading http://pypi.python.org/simple/pycrypto/
Reading http://pycrypto.sourceforge.net
Reading http://www.amk.ca/python/code/crypto
Best match: pycrypto 2.0.1
Downloading http://www.amk.ca/files/python/crypto/pycrypto-2.0.1.tar.gz
Processing pycrypto-2.0.1.tar.gz
Running pycrypto-2.0.1\setup.py -q bdist_egg --dist-dir c:\docume~1\chrism~1\locals~1\temp\easy_install-nm62sp\pycrypto-2.0.1\egg-dist-tmp-847pkw
error: Setup script exited with error: Unable to find vcvarsall.bat

Well, that didn't work. Apparently this tried to install pycrypto-2.0.1 but failed to compile it. Of course it would, since I don't have a microsoft compiler on this machine. And paramiko is a requirement for fabric. Let me try to just easy_install fabric, see what it does.

C:\Python26\Scripts>easy_install-2.6 fabric
Searching for fabric
Reading http://pypi.python.org/simple/fabric/
Reading http://fabfile.org
Reading http://www.nongnu.org/fab/
Reading https://savannah.nongnu.org/projects/fab/
Best match: Fabric 0.9.0
Downloading http://pypi.python.org/packages/source/F/Fabric/Fabric-0.9.0.tar.gz#md5=696e42e6e771fa2a365cd03536d4cdcf
Processing Fabric-0.9.0.tar.gz
Running Fabric-0.9.0\setup.py -q bdist_egg --dist-dir c:\docume~1\chrism~1\locals~1\temp\easy_install-nmv49r\Fabric-0.9.0\egg-dist-tmp-zzlzv3
warning: no previously-included files matching '*' found under directory 'docs\_build'
zip_safe flag not set; analyzing archive contents...
Adding fabric 0.9.0 to easy-install.pth file
Installing fab-script.py script to C:\Python26\Scripts
Installing fab.exe script to C:\Python26\Scripts

Installed c:\python26\lib\site-packages\fabric-0.9.0-py2.6.egg
Processing dependencies for fabric
Searching for pycrypto>=1.9
Reading http://pypi.python.org/simple/pycrypto/
Reading http://pycrypto.sourceforge.net
Reading http://www.amk.ca/python/code/crypto
Best match: pycrypto 2.0.1
Downloading http://www.amk.ca/files/python/crypto/pycrypto-2.0.1.tar.gz
Processing pycrypto-2.0.1.tar.gz
Running pycrypto-2.0.1\setup.py -q bdist_egg --dist-dir c:\docume~1\chrism~1\locals~1\temp\easy_install-bl5fmp\pycrypto-2.0.1\egg-dist-tmp-wjpo1c
error: Setup script exited with error: Unable to find vcvarsall.bat

Oh Joy that went the same bad way.

Visual C++ 2008 Express Edition

Ok. After a little searching, I decided I needed to install Microsoft Visual C++ 2008 Express Edition. Now installing. Taking its sweet time. In the meantime I'm reading the fabric tutorial.

12:45 am. Wait, I just got a popup!

---------------------------
Microsoft Visual C++ 2008 Express Edition with SP1 Setup
---------------------------
Error 1935.An error occurred during the installation of assembly 'Microsoft.VisualStudio.VCProject,version="9.0.0.0",publicKeyToken="b03f5f7f11d50a3a",fileVersion="9.0.30729.1",culture="neutral"'. Please refer to Help and Support for more information. HRESULT: 0x80070020. 
---------------------------
OK   
---------------------------

Nice! not. Let's try again. Let's free up a little RAM.

It happened again. Exactly the same error message. I'm re-downloading from microsoft. 12:59 am Ok. Relauching vcsetup.exe.

Per this I'll try without panda antivirus.

1:08 am. It worked.

1:23am. I ran C:\Python26\Scripts>easy_install-2.6 fabric again and got the error again: unable to find vcvarsall.bat. Nice. Do I have to reboot?

I am so tempted to just do this on the debian virtual box.

Installed pip. no problem.

1:35 am. Added C:\Program Files\Microsoft Visual Studio 9.0\VC to my path environment variable.

Rebooted the box. vcvarsall.bat is found from the command line. Let's install fabric. No problem. Now for paramiko. No problem.

1:42 am. >>>import fabric and >>>import paramiko work.

Found an interesting little article on pip, virtualenv and fabric at Alec Clemesha's blog.

January 13, 2010

Making my first fabric file.

I skipped a bunch of trial and error things, including going through most of the fabric manual, and I made the following file:

from fabric.api import *
from fabric.contrib.console import confirm
import datetime

env.user = "root" 
env.password = "yourpassword"
env.hosts = ['0.0.0.0:22']
# env.hosts = ['example.com']  #sample

def write_log(msg):
    logfile = open("./log.txt", "a")
    line = str(datetime.datetime.now()) + " " + msg + "\n"
    logfile.write(line)
    logfile.close()
    
    
def ask(result, msg):
    if result.failed:
        #confirmed = confirm("Continue anyway?")
        confirmed = True  #automatically say yes.
    if result.failed and not confirmed:
        write_log("Aborting after failing to execute %s" % msg)
        abort("Aborting at user request.")
    else:
        write_log("Executed %s" % msg)

def my_task():
    with settings(warn_only = True):
        
        with cd("~"):
            
            cmd = 'apt-get -y update'
            ask(run(cmd), cmd)
            
            cmd = "rmdir -rf ~/pythontemp"
            ask(run(cmd), cmd)
            
            cmd = 'mkdir pythontemp'
            ask(run(cmd), cmd)
            
        with cd("~/pythontemp"):
            cmd = 'wget http://python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2'
            ask(run(cmd), cmd)
            
            cmd= "apt-get -y install bzip2"
            ask(run(cmd), cmd)
            
            cmd = 'tar xjvf Python-2.6.4.tar.bz2'
            ask(run(cmd), cmd)
        
        with cd("~/pythontemp/Python-2.6.4"):
            cmd = 'apt-get -y install make'
            ask(run(cmd), cmd)
            
            cmd = 'apt-get -y install gcc'
            ask(run(cmd), cmd)
            
            cmd = 'apt-get -y install libssl-dev'
            ask(run(cmd), cmd)
            
            cmd = 'apt-get -y install libsqlite3-dev'
            ask(run(cmd), cmd)
            
            cmd = 'apt-get -y install libbz2-dev'
            ask(run(cmd), cmd)
            
            cmd = 'apt-get -y install tk8.4-dev libgdbm-dev libdb-dev libreadline-dev libncurses-dev'
            ask(run(cmd), cmd)

            cmd = 'apt-get -y install patch'
            ask(run(cmd), cmd)
            
            cmd = 'wget http://www.lysium.de/sw/python2.6-disable-old-modules.patch'
            ask(run(cmd), cmd)

            cmd = 'patch -p1 < python2.6-disable-old-modules.patch'
            ask(run(cmd), cmd)

            cmd = 'apt-get -y install checkinstall'
            ask(run(cmd), cmd)

            cmd = './configure'
            ask(run(cmd), cmd)

            cmd = 'make'
            ask(run(cmd), cmd)

            cmd = 'checkinstall -y --pkgname=python2.6 --showinstall=no'
            ask(run(cmd), cmd)

I tried with the fsckvps Debian stable VPS (64 bit) and it worked. it failed to find a couple of dependencies (dl and imageop), but that didn't stop the build. If there is a need I can investigate further.

2:06 pm. Now using this fabric file to build python 2.6 on the rapidvps debian stable vps.

2:28 pm. That worked. I had to restart nginx but that was it.

Now for pip and virtualenv.

January 14, 2009

I spent (wasted) a couple of hours with AWS EC2.

12:49 am. I spent 3 minutes provisioning a vps server with vpsville at their los angeles data center.

I used the fabric script above to add python 2.6.4 on that machine.

I had to add the following lines to the fabric script at the beginning:

cmd = 'apt-get -y update'
ask(run(cmd), cmd)
cmd = 'apt-get install debian-keyring debian-archive-keyring'
ask(run(cmd), cmd)

1:04 am. Dangit, it's done setting up python already.

root@x:~# python
Python 2.6.4 (r264:75706, Jan 14 2010, 09:01:10)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

That was quick.

Maybe I should set up an open-source project on some forge out there.

Well, now it's later. January 26th.

I have been messing with a bunch of stuff, and I have now made a bitbucket project for the debian installer.

I'll update shortly.

I have added the steps necessary to install turbogears latest.

Install subversion

apt-get -y install subversion

Then install easy_install.

wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

I then installed virtualenv, although I have personal misgivings about the whole concept behind virtualenv. Isn't it yet another layer?

easy_install virtualenv

Then I did the install of latest trunk with pip.

easy_install pip
pip install -e svn+http://svn.turbogears.org/trunk
pip install -e svn+http://svn.turbogears.org/projects/tg.devtools/trunk

And that worked!

On to the tutorial!

I then created a directory: /tg (I know, I know, off the root, no good. meh.

then cd into that directory, then ran

paster quickstart

selected tester as project name. Then changed directory to /tg/tester and ran:

python setup.py develop
paster setup-app development.ini

Well, that didn't work. Got a lovely error trace:

Starting server in PID 6739.
Traceback (most recent call last):
  File "/usr/local/bin/paster", line 8, in 
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/usr/local/lib/python2.6/site-packages/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/local/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/usr/local/lib/python2.6/site-packages/paste/script/command.py", line 218, in run
    result = self.command()
  File "/usr/local/lib/python2.6/site-packages/paste/script/serve.py", line 303, in command
    serve()
  File "/usr/local/lib/python2.6/site-packages/paste/script/serve.py", line 287, in serve
    server(app)
  File "/usr/local/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 151, in server_wrapper
    **context.local_conf)
  File "/usr/local/lib/python2.6/site-packages/paste/deploy/util/fixtypeerror.py", line 57, in fix_call
    val = callable(*args, **kw)
  File "/usr/local/lib/python2.6/site-packages/paste/httpserver.py", line 1336, in server_runner
    serve(wsgi_app, **kwargs)
  File "/usr/local/lib/python2.6/site-packages/paste/httpserver.py", line 1285, in serve
    request_queue_size=request_queue_size)
  File "/usr/local/lib/python2.6/site-packages/paste/httpserver.py", line 1132, in __init__
    **threadpool_options)
  File "/usr/local/lib/python2.6/site-packages/paste/httpserver.py", line 1033, in __init__
    **threadpool_options)
  File "/usr/local/lib/python2.6/site-packages/paste/httpserver.py", line 615, in __init__
    self.add_worker_thread(message='Initial worker pool')
  File "/usr/local/lib/python2.6/site-packages/paste/httpserver.py", line 737, in add_worker_thread
    worker.start()
  File "/usr/local/lib/python2.6/threading.py", line 471, in start
    _start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread

Lovely... there was not enough ram in the VPS. Got it to work for a little while, but TG2 was trying to use 106 megs of ram, and the poor vps only had 64 available, so that didn't go very far...

7:00 am. Reimaging the vps with debian stable 5.0 .

January 26, 2010, 1:00 pm. I'm reimaging the fsckvps vps with debian 32 bits. I think though it's not a minimal install, and it has some stuff in there that I don't want, like apache.

I have a feeling fsckvps is oversold. SSH responsiveness is less than perfect.

ok. there is something super-weird. I uninstalled apache2 with

apt-get remove --purge apache2

...and apache is still there. Something is very wrong.

Document Status

Last updated January 13, 2010.

By Christopher Mahan (chris.mahan@gmail.com).