Filled step plots in matplotlib

It's not possible to do a simple filled step plot in matplotlib using default
commands. Workaround:

def steppify(arr,isX=False,interval=0):
    """
    Converts an array to double-length for step plotting
    """
    if isX and interval==0:
        interval = abs(arr[1]-arr[0]) / 2.0
        newarr = array(zip(arr-interval,arr+interval)).ravel()
        return newarr

plot(xx,yy,linestyle='steps-mid',color='b',linewidth=1.5)
fill_between(steppify(xx[x1:x2],isX=True),
    steppify(yy[x1:x2])*0,
    steppify(yy[x1:x2]),
    facecolor='b',alpha=0.2)

IDL-to-Python

astrobetter started up an idl-to-python guide on their wiki.

command line & escape key

Problem: in both the bash command line and ipython, if I hit escape twice, I can never again do history-search-backwards (i.e. when you type part of a command and type "up" and reverse-search through your command history). Any idea how to fix this?

Rebuilding python from scratch again

I got scipy working a week or two ago, but doing so killed matplotib's tkagg. So, I switched to the MacOSX backend, which worked ok until I realized that the interactive (connect) features of macosx failed miserably. This led me to try to get matplotlib working.... which broke with those awful "symbol not found" errors in ft2font.so and _path.so, which I've determined all have to do with linking to the wrong library files.

The most worrisome part of this process was discovering that a full Time Machine recovery of /usr and /Library/Frameworks and /Library/Python did *not* restore python - it stayed dead with IDENTICAL errors. So there are probably additional layers of hidden links.

The process below is based on hyperjeff's blog post but differs substantially based on Sam Skillman's recommendations and the very big issue I ran into that my /usr/local files appeared to be corrupted. After this install, my path no longer includes /usr/local/bin and /sw has been moved to /_sw... hopefully one of these days I'll be ballsy enough to delete it.

1. Install python 2.6.4
   
   1. Needed a clean terminal with no flags set at all. Don't know why - all I had set were a bunch of -arch x86_64 flags.
      
export LD_LIBRARY_PATH="/usr/local/lib:/usr/X11/lib"
./configure --enable-framework=/Library/Frameworks MACOSX_DEPLOYMENT_TARGET=10.6
make -j 17
sudo make install

      
   2. Reset PYTHONPATH to blank
      
   3. alias clearflags='export CFLAGS=""; export CCFLAGS=""; export CXXFLAGS=""; export LDFLAGS=""; export FFLAGS="";' to make sure
      
   
   
2. Install FFTW
   
cd ~/tmp
curl -O http://www.fftw.org/fftw-3.2.2.tar.gz
tar xf fftw-3.2.2.tar.gz
cd fftw-3.2.2
clearflags
./configure CC="gcc -arch x86_64" CXX="g++ -arch x86_64" CPP="gcc -E" CXXCPP="g++ -E"
make -j 17
sudo make install

   
3. Install UMFPACK
   
cd ~/tmp
curl -O http://www.cise.ufl.edu/research/sparse/umfpack/current/UMFPACK.tar.gz
curl -O http://www.cise.ufl.edu/research/sparse/UFconfig/current/UFconfig.tar.gz
curl -O http://www.cise.ufl.edu/research/sparse/amd/current/AMD.tar.gz
tar xf AMD.tar.gz
tar xf UFconfig.tar.gz
tar xf UMFPACK.tar.gz
sed -ibck 's/F77 = f77/F77 = gfortran/' UFconfig/UFconfig.mk
sed -ibck '299,303s/# //' UFconfig/UFconfig.mk
cp UFconfig/UFconfig.h AMD/Include/
cp UFconfig/UFconfig.h UMFPACK/Include/
cd UMFPACK
make -j 17
make hb
make clean

   
4. Install numpy
   
   1. Set environment variables
      
export MACOSX_DEPLOYMENT_TARGET=10.6
export CFLAGS="-arch x86_64"
export FFLAGS="-m64"
export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch x86_64"
export PYTHONPATH="/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/"
echo "[amd]
library_dirs = /Users/adam/tmp/AMD/Lib
include_dirs = /Users/adam/tmp/AMD/Include
amd_libs = amd

[umfpack]
library_dirs = /Users/adam/tmp/UMFPACK/Lib
include_dirs = /Users/adam/tmp/UMFPACK/Include
umfpack_libs = umfpack" > site.cfg

      
   2. Setup & Install
      
python setup.py build --fcompiler=gnu95
sudo python setup.py install

      
   3. Test: python -c "import numpy"
      
   
   
5. Install scipy. The important thing is to use g++-4.2 because g++-4.5 doesn't accept the -arch flag. Also, get rid of /sw if it's on your computer at all.
   
sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyX
cd ~/repos/scipy-0.7.1
python setup.py clean
rm -rf build
clearflags
FFLAGS="-m64" CFLAGS="-arch x86_64 -I/usr/local/include/freetype2 -I/usr/X11/include -L/usr/X11/lib" LDFLAGS="-Wall -undefined dynamic_lookup -bundle -lpng -arch x86_64" CXX="/usr/bin/g++-4.2" CC="/usr/bin/gcc-4.2" python setup.py build
python setup.py install

   Test the install:
   
python -c "import scipy, scipy.fftpack, scipy.interpolate"

   
6. 1. Install matplotlib. MAKE SURE /usr/bin/texbin is in front of /usr/local/bin and /sw/bin so that dvipng comes from MacTEX. I also ended up having to remove /usr/local/bin from my path completely
      
sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyX
cd ~/repos/matplotlib-svn
python setup.py clean
rm -rf build
clearflags

      
   2. Do hyperjeff's recommended edits except don't use /usr/local because it's f'd:
      make.osx:
      
MACOSX_DEPLOYMENT_TARGET=10.6
PREFIX=/usr
PYTHON=/Library/Frameworks/Python.framework/Versions/Current/bin/python

## You shouldn't need to configure past this point (and yet…)

PKG_CONFIG_PATH="${PREFIX}/lib/pkgconfig"
CFLAGS="-arch i386 -arch x86_64 -I${PREFIX}/include -I${PREFIX}/include/freetype2 -isysroot /Developer/SDKs/MacOSX10.6.sdk"
LDFLAGS="-arch i386 -arch x86_64 -L${PREFIX}/lib -syslibroot,/Developer/SDKs/MacOSX10.6.sdk"
FFLAGS="-arch i386 -arch x86_64"

      setup.cfg:
      
wxagg = False

      
   3. Do the install (different from hyperjeff b/c I don't want root)
      sudo make -f make.osx fetch deps
make -f mpl_build mpl_install
python setup.py install

      
   
   
7. Install setuptools
   
8. easy_install ipython
   
9. install everything else pythonically
   

DS9 gaussian fitting

One thing DS9 desperately needs is an interactive gaussian fitter. I have NOT implemented one yet, but it is high on my to do list. Has anyone else (googlers?) tried or succeeded in implementing such a thing? Ideally, something with NO dependencies: if I write one, it will require python, numpy, and probably pyds9 - ick. Straight-up TCL would be very much preferable.

Another attempt to get 64 bit python on Snow Leopard

Sam Skillman posted his attempt to get 64 bit python on snow leopard. It worked, but you need to install python with --enable-framework and NOT --prefix and NOT --enable-universalSDK. It is 64-bit only, not universal: universal causes trouble.

Python 64-bit on Mac OS X 10.6 Snow Leopard

After yesterday's disastrous attempt to install various python packages, I started from scratch. First, I got rid of all of my python frameworks (backed up but removed from the path). Then, I compiled python 2.7 from scratch:

I got some help from http://blog.mahmoudimus.com/2009/12/python-2-6-4-and-twisted-9-on-os-x-10-6-snow-leopard/

./configure --enable-framework --enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk MACOSX_DEPLOYMENT_TARGET=10.6 --with-universal-archs=intel -with-readline-dir=/usr/local
make -j 17
make -j 17 test

make results:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             dl                 gdbm            
imageop            linuxaudiodev      ossaudiodev     
spwd               sunaudiodev                        
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

I'm not concerned about these - I don't use any of them and I assume I need to install some other packages to get them to work.

During make test, I had two failures that resulted in "python crash" pop-up boxes:

test_subprocess
.
    this bit of output is from a test of stdout in a different process ...
.
    this bit of output is from a test of stdout in a different process ...
test_sunaudiodev

Then, I got some malloc errors:

test_io

Testing large file ops skipped on darwin.
It requires 2147483648 bytes and a long time.
Use 'regrtest.py -u largefile test_io' to run it.

Testing large file ops skipped on darwin.
It requires 2147483648 bytes and a long time.
Use 'regrtest.py -u largefile test_io' to run it.
python.exe(22914,0x7fff70d3ebe0) malloc: *** mmap(size=9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(22914,0x7fff70d3ebe0) malloc: *** mmap(size=9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
python.exe(22914,0x7fff70d3ebe0) malloc: *** mmap(size=9223372036854775808) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
test_ioctl

PSDs, cross-correlation...

scipy is capable of doing fft-base cross-correlation, convolution, etc., but it requires the stsci package, which is not generally easy to install. For that matter, scipy can be a pain some of the time. So agpy now includes a 2D cross-correlation code and a power spectrum / power spectral density code. These are pure-numpy codes that should be easy to use without any other bothersome dependencies.

EDIT: I have them check for scipy (which can cause crashes if you have a bad scipy install, e.g. 32 bit executables on a 64 bit system) because scipy uses FFTW and numpy appears not to. Also, this code & related stuff has been discussed on astrobetyter

agpy
correlate2d
psds

Logarithmic Colormap / Other Colormap in Matplotlib

This is kind of a pain to find out:


from matplotlib.colors import LogNorm

im = imshow(.... cmap=... , norm=LogNorm(vmin=clevs[0], vmax=clevs[-1]))


It also works for contours, and can be particularly useful if you only want to display contours at a few levels, but you want the colormap to start at a different point. e.g.:

contour(xx,levels=[2,3,4,5,6,7,8,9,10],norm=matplotlib.colors.Normalize(vmin=0,vmax=10))

will start at light blue instead of dark blue in the default colormap

cython vs f2py

I had a go at optimizing some code this past week, and ended up learning to use both cython and f2py.

f2py is much easier to use. If you want to write a function in fortran and use it in python, all you do is write the code and add specifications using comments in the fortran code.

cython is more natural to code. The code style is C/fortran-like: think in terms of loops instead of arrays. The syntax is python-like, which makes coding somewhat clearer and simpler.

For my code, I found that cython was ~10% slower than fortran.

Check out the plfits in:
http://code.google.com/p/agpy/source/browse/#svn/trunk/

Year in review

It's been just over a year since I started this blog. I posted a LOT more than I thought I would. Also, I apparently dropped the tradition of reviewing beers. So I'll start by reviewing some beers!

Mountain Sun Hummingbird - a strong honey-ale (meade-beer?) brewed with orange blossom honey. Slightly darker than a typical Belgian Golden but similar in style. The beer is mildly sweet but very drinkable. Hmm... there are flavors I should mention but they escape me.

Mountain Sun/Avery Van Diemens - brewed with Tasmanian Pepper Berry, this is a very curious, semi-dark beer. When I first sampled it, I tasted a little bit of pepper kick at the end, but couldn't really identify any other flavors. Yesterday I sampled it next to a burrito with some somewhat spicy salsa. Somehow, since my mouth was already sensitized to spiciness, the flavor I got was root beer (sassafras?). It was... odd. Pretty good, but I didn't end up purchasing a glass.

Moving on to code. I don't know why I haven't mentioned this, but with my discovery of svn, I started uploading my code to the webternets: agpy is my Google Code page and includes a number of useful python codes, especially readcol and gaussfitter, which I have tested and used extensively since writing them. Python is still a long way from a cohesive astrolib code base, but with individual contributions, the STSCI development group, and APLpy underway, we're getting closer.

SUCCESS! 64 bit python with 64 bit tcl/tk!!!

After a long, tedious process (see previous posts), I got 64 bit python, 64 bit tcl/tk, and 64 bit tkinter all to work! I can now use the TkAgg backend in matplotlib!

Python 64 bit on Mac OS X: Sam Skillman's post

Tcl/Tk 64 bit: a post on the tcl/tk forums

tkinter 64 bit: python bug report 4017 (last two posts give the solution) and my posted solution

Installing 64 bit tcl/tk on Mac OS X

Everything is described in this post:
http://www.nabble.com/Error-compiling-tk-8.5.7-on-Mac-OS-X-10.5-td23790967.html


But here's a script too:

curl -O 'http://osdn.dl.sourceforge.net/sourceforge/tcl/t{cl,k}8.5.7-src.tar.gz'
for f in t*8.5.7*.gz; do tar zxf $f; done
cd tcl8.5.7/unix/
./configure --enable-framework --enable-64-bit
cd tk8.5.7/unix/
./configure --enable-framework --enable-64-bit
make -j 4 -C tcl8.5.7/unix
make -j 4 -C tk8.5.7/unix
sudo make install -C tcl8.5.7/unix
sudo make install -C tk8.5.7/unix


Concerns:
-might be necessary to do this in the macosx directory for some reason, though Aqua doesn't support 64 bits
-have to recompile python to get _tkinter to work (see a later post)

weekend

What did I do this weekend?
-upgraded PyRAF from 1.6.1 to 1.7
-switched matplotlib backends from TkAgg to Qt4
-put up a new article on molecularclouds


So... totally unproductive, but only in an awesome way?

I tried to switch PyRAF from X11 to Aqua a la Ticket 86 on the PyRAF trac site, but it failed with a can't-load error.

Also, updated readcol documentation so that it makes sense. Sam has used it, Robbie might.

BASH discoveries, readline questions

1. shopt -p
Maybe my hostname completion worked and then stopped working because the bash option hostcomplete was not set. Duh! Why? I don't know. Anyway, shopt -s hostcomplete solves the problem.

nocaseglob is also pretty cool (case insensitive tab completion)

2. it's really hard to search for readline stuff on google. Can anyone explain to me how BASH readline works? I would REALLY like to make bash readline work like ipython, in which you can start typing a command and hit the 'up' key to search through the history for anything beginning with the stuff you've typed up to that point. But I can't even find documentation for the ipython readline! Any hints, anyone?

3. my desktop at work blocks ssh connections. I can ssh into some computers and then into it, but not directly into it.

...as usual, I made a list where not-a-list would have sufficed, and I had to add the last thing because a 2-item list is dumb.

Python magic / advanced numpy indexing

Yeah, indexing python arrays should really be easy.

Stefan van der Walt's page


In [85]: bi = (f.bolo_indices[np.newaxis,:] + zeros([7751,1])).astype('int')

In [86]: whc = (whscan[:,np.newaxis] + zeros([1,107])).astype('int')

In [87]: array2d[whc,bi] = temp2d


Python: one-line arrays

Ahhh, refreshing:

whscan = asarray([arange(scanlen)+i for i in scans_info[:,0]]).ravel()


Not like IDL, which takes at least 4 lines b/c of the variable declaration. There's probably a better way to do that too.

Port vs Fink

Port failed to install:

adam-macbook ~$ sudo port install py25-matplotlib
---> Building py25-hashlib with target build
---> Staging py25-hashlib into destroot
---> Installing py25-hashlib 2.5.2_0
---> Activating py25-hashlib 2.5.2_0
---> Cleaning py25-hashlib
---> Fetching py25-zlib
---> Verifying checksum(s) for py25-zlib
---> Extracting py25-zlib
---> Configuring py25-zlib
---> Building py25-zlib with target build
---> Staging py25-zlib into destroot
---> Installing py25-zlib 2.5.2_0
---> Activating py25-zlib 2.5.2_0
---> Cleaning py25-zlib
---> Fetching py25-setuptools
---> Attempting to fetch setuptools-0.6c8.tar.gz from http://cheeseshop.python.org/packages/source/s/setuptools/
---> Verifying checksum(s) for py25-setuptools
---> Extracting py25-setuptools
---> Applying patches to py25-setuptools
---> Configuring py25-setuptools
---> Building py25-setuptools with target build
---> Staging py25-setuptools into destroot
---> Installing py25-setuptools 0.6c8_2
---> Activating py25-setuptools 0.6c8_2
---> Cleaning py25-setuptools
---> Fetching py25-tz
---> Attempting to fetch pytz-2007k.tar.bz2 from http://cheeseshop.python.org/packages/source/p/pytz/
---> Verifying checksum(s) for py25-tz
---> Extracting py25-tz
---> Configuring py25-tz
---> Building py25-tz with target build
---> Staging py25-tz into destroot
---> Installing py25-tz 2007k_0
---> Activating py25-tz 2007k_0
---> Cleaning py25-tz
---> Fetching py25-dateutil
---> Attempting to fetch python-dateutil-1.4.1.tar.gz from http://labix.org/download/python-dateutil
---> Verifying checksum(s) for py25-dateutil
---> Extracting py25-dateutil
---> Configuring py25-dateutil
---> Building py25-dateutil with target build
---> Staging py25-dateutil into destroot
---> Installing py25-dateutil 1.4.1_0
---> Activating py25-dateutil 1.4.1_0
---> Cleaning py25-dateutil
---> Fetching fftw-3
---> Attempting to fetch fftw-3.1.3.tar.gz from http://www.fftw.org/
---> Verifying checksum(s) for fftw-3
---> Extracting fftw-3
---> Configuring fftw-3
---> Building fftw-3 with target all
---> Staging fftw-3 into destroot
---> Installing fftw-3 3.1.3_0
---> Activating fftw-3 3.1.3_0
---> Cleaning fftw-3
---> Fetching py25-nose
---> Attempting to fetch nose-0.10.4.tar.gz from http://somethingaboutorange.com/mrl/projects/nose/
---> Verifying checksum(s) for py25-nose
---> Extracting py25-nose
---> Configuring py25-nose
---> Building py25-nose with target build
---> Staging py25-nose into destroot
---> Installing py25-nose 0.10.4_0
---> Activating py25-nose 0.10.4_0
---> Cleaning py25-nose
---> Fetching py25-numpy
---> Attempting to fetch numpy-1.2.0.tar.gz from http://downloads.sourceforge.net/numpy
---> Verifying checksum(s) for py25-numpy
---> Extracting py25-numpy
---> Applying patches to py25-numpy
---> Configuring py25-numpy
---> Building py25-numpy with target build
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0" && /opt/local/bin/python2.5 setup.py build " returned error 1
Command output: setup_package()
File "setup.py", line 89, in setup_package
configuration=configuration )
File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/core.py", line 184, in setup
return old_setup(**new_attr)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", line 151, in setup
dist.run_commands()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 974, in run_commands
self.run_command(cmd)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command
cmd_obj.run()
File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build.py", line 37, in run
old_build.run(self)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/build.py", line 112, in run
self.run_command(cmd_name)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", line 333, in run_command
self.distribution.run_command(command)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command
cmd_obj.run()
File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build_src.py", line 130, in run
self.build_sources()
File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build_src.py", line 147, in build_sources
self.build_extension_sources(ext)
File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build_src.py", line 250, in build_extension_sources
sources = self.generate_sources(sources, ext)
File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build_src.py", line 307, in generate_sources
source = func(extension, build_dir)
File "numpy/core/setup.py", line 87, in generate_config_h
raise SystemError,"Failed to test configuration. "\
SystemError: Failed to test configuration. See previous error messages for more information.

Error: The following dependencies failed to build: py25-numpy py25-tkinter tcl tk
Error: Status 1 encountered during processing.


I don't want to deal with it so I'm just using Fink.

Reinstalling a system

Python died tonight when I foolishly tried to update numpy or matplotlib. One was not compatible with the other in upgraded form, and the update for matplotlib wouldn't install because of GTK issues that were totally opaque.

So, I reinstalled EVERYTHING - fink AND macports - from scratch. If I don't get things running by morning I'm going to have to do a restore, which is ugly as death.

Things I need to do post-install:

sudo port install python_select
sudo python_select python25


test ipython
test matplotlib
test numpy

Python: setting matplotlib defaults

Setting matplotlib defaults is a lot more difficult than it should be.


matplotlib.defaultParams['image.origin']='lower'
matplotlib.defaultParams['image.interpolation']='nearest'