Friday, March 5, 2010

Rebuilding python from scratch again

I got scipy working a week or two ago, but doing so killed matplotib's tkagg. So, I switched to the MacOSX backend, which worked ok until I realized that the interactive (connect) features of macosx failed miserably. This led me to try to get matplotlib working.... which broke with those awful "symbol not found" errors in and, which I've determined all have to do with linking to the wrong library files.

The most worrisome part of this process was discovering that a full Time Machine recovery of /usr and /Library/Frameworks and /Library/Python did *not* restore python - it stayed dead with IDENTICAL errors. So there are probably additional layers of hidden links.

The process below is based on hyperjeff's blog post but differs substantially based on Sam Skillman's recommendations and the very big issue I ran into that my /usr/local files appeared to be corrupted. After this install, my path no longer includes /usr/local/bin and /sw has been moved to /_sw... hopefully one of these days I'll be ballsy enough to delete it.

  1. Install python 2.6.4
    1. Needed a clean terminal with no flags set at all. Don't know why - all I had set were a bunch of -arch x86_64 flags.

      export LD_LIBRARY_PATH="/usr/local/lib:/usr/X11/lib"
      ./configure --enable-framework=/Library/Frameworks MACOSX_DEPLOYMENT_TARGET=10.6
      make -j 17
      sudo make install

    2. Reset PYTHONPATH to blank
    3. alias clearflags='export CFLAGS=""; export CCFLAGS=""; export CXXFLAGS=""; export LDFLAGS=""; export FFLAGS="";' to make sure

  2. Install FFTW

    cd ~/tmp
    curl -O
    tar xf fftw-3.2.2.tar.gz
    cd fftw-3.2.2
    ./configure CC="gcc -arch x86_64" CXX="g++ -arch x86_64" CPP="gcc -E" CXXCPP="g++ -E"
    make -j 17
    sudo make install

  3. Install UMFPACK

    cd ~/tmp
    curl -O
    curl -O
    curl -O
    tar xf AMD.tar.gz
    tar xf UFconfig.tar.gz
    tar xf UMFPACK.tar.gz
    sed -ibck 's/F77 = f77/F77 = gfortran/' UFconfig/
    sed -ibck '299,303s/# //' UFconfig/
    cp UFconfig/UFconfig.h AMD/Include/
    cp UFconfig/UFconfig.h UMFPACK/Include/
    cd UMFPACK
    make -j 17
    make hb
    make clean

  4. Install numpy
    1. Set environment variables

      export CFLAGS="-arch x86_64"
      export FFLAGS="-m64"
      export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch x86_64"
      export PYTHONPATH="/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/"
      echo "[amd]
      library_dirs = /Users/adam/tmp/AMD/Lib
      include_dirs = /Users/adam/tmp/AMD/Include
      amd_libs = amd

      library_dirs = /Users/adam/tmp/UMFPACK/Lib
      include_dirs = /Users/adam/tmp/UMFPACK/Include
      umfpack_libs = umfpack" > site.cfg

    2. Setup & Install

      python build --fcompiler=gnu95
      sudo python install

    3. Test: python -c "import numpy"

  5. Install scipy. The important thing is to use g++-4.2 because g++-4.5 doesn't accept the -arch flag. Also, get rid of /sw if it's on your computer at all.

    sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyX
    cd ~/repos/scipy-0.7.1
    python clean
    rm -rf build
    FFLAGS="-m64" CFLAGS="-arch x86_64 -I/usr/local/include/freetype2 -I/usr/X11/include -L/usr/X11/lib" LDFLAGS="-Wall -undefined dynamic_lookup -bundle -lpng -arch x86_64" CXX="/usr/bin/g++-4.2" CC="/usr/bin/gcc-4.2" python build
    python install

    Test the install:

    python -c "import scipy, scipy.fftpack, scipy.interpolate"

    1. Install matplotlib. MAKE SURE /usr/bin/texbin is in front of /usr/local/bin and /sw/bin so that dvipng comes from MacTEX. I also ended up having to remove /usr/local/bin from my path completely

      sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyX
      cd ~/repos/matplotlib-svn
      python clean
      rm -rf build

    2. Do hyperjeff's recommended edits except don't use /usr/local because it's f'd:


      ## You shouldn't need to configure past this point (and yet…)

      CFLAGS="-arch i386 -arch x86_64 -I${PREFIX}/include -I${PREFIX}/include/freetype2 -isysroot /Developer/SDKs/MacOSX10.6.sdk"
      LDFLAGS="-arch i386 -arch x86_64 -L${PREFIX}/lib -syslibroot,/Developer/SDKs/MacOSX10.6.sdk"
      FFLAGS="-arch i386 -arch x86_64"


      wxagg = False

    3. Do the install (different from hyperjeff b/c I don't want root)
      sudo make -f make.osx fetch deps
      make -f mpl_build mpl_install
      python install

  6. Install setuptools
  7. easy_install ipython
  8. install everything else pythonically


Dennis said...

I got through step 4 and have Numpy going with only known fails and skips. In step 5, if I use the line that moves numpy to numpyX, scipy won't go - it is looking for the numpy module when it tries to python it?

Adam said...

Which python are you using? Make sure it's /Library/Frameworks/Python.framework/Versions/2.6/bin/python and not /usr/bin/python or something similar (though be aware that some may be symlinks to others)

Dennis said...

which python

Adam said...

What error does scipy die with?