Blog moved!

This blog has moved!  It is now hosted at
http://keflavich.github.io/blog/

Repositories for observers

I should have posted these a while ago....
casaradio is a subversion repository for folks at The Center for Astrophysics and Space Astronomy at CU Boulder to post radio astronomy related codes. So far, emphasizes single dish (GBT, Arecibo), but will include EVLA, CARMA, and ALMA eventually.
aposoftware is a similar page, but is a mercurial repository and is meant to include instrument-specific software for the Apache Point Observatory 3.5m telescope. Right now includes a TUI script or two and the TSPEC and DIS IRAF-twodspec pipelines.

I'd be remiss to leave out the BGPS pipeline even though it's mentioned on the previous post.

Also, agpy is my personal code repository.

BGPS data paper published

Metalinking! The BGPS paper finally made it onto astro-ph today. It will be published in ApJS before the year's end.

Links to all of the published BGPS papers at the Bolocam Data Team website

And just because I want more linking, here they all are again:
The Bolocam Galactic Plane Survey I. Survey Description and Data Reduction arXiv
The Bolocam Galactic Plane Survey II. Catalog of the Image Data arXiv
The Bolocam Galactic Plane Survey III. Characterizing Physical Properties of Massive Star-Forming Regions in the Gemini OB1 Molecular Cloud arXiv
The Bolocam Galactic Plane Survey IV: λ = 1.1 and 0.35 mm Dust Continuum Emission in the Galactic Center Region

The same set of links is reproduced at the pipeline googlecode page.

Neat new things....

1. sptool is a quick way to compare standards to stellar spectra. Nice, I'd been looking for a tool like that.
2. GNU screen captions are useful especially when working in a screen-within-a-screen environment (who does that, really?)
3. finally got SPLAT to work... turns out I just hadn't reduced my damned data
4. kill -STOP and kill -CONT are really useful ways to pause programs that are sucking up resources if you want to resume them later. Haven't tried this on "real" code yet.

Filled step plots in matplotlib

It's not possible to do a simple filled step plot in matplotlib using default
commands. Workaround:

def steppify(arr,isX=False,interval=0):
    """
    Converts an array to double-length for step plotting
    """
    if isX and interval==0:
        interval = abs(arr[1]-arr[0]) / 2.0
        newarr = array(zip(arr-interval,arr+interval)).ravel()
        return newarr

plot(xx,yy,linestyle='steps-mid',color='b',linewidth=1.5)
fill_between(steppify(xx[x1:x2],isX=True),
    steppify(yy[x1:x2])*0,
    steppify(yy[x1:x2]),
    facecolor='b',alpha=0.2)

Histogram in Google Spreadsheet

It's not easy to make a histogram in google spreadsheets without replicating data. The "countif" function would be great, except it only allows very simple criteria. However, there's a workaround:
=count(Filter('Grades'!V2:V30,'Grades'!V2:V30>0.9))
=count(Filter('Grades'!V2:V30,'Grades'!V2:V30<0.9,'Grades'!V2:V30>0.8))

The Filter() function returns an array, which can be operated on like any other set of cells.

It's still not easy to make a nice-looking histogram, but the output of this process is at least usable.

IDL-to-Python

astrobetter started up an idl-to-python guide on their wiki.

Ghostscript error?

I've been receiving the following error when attempting to compile (ps2pdf) my w5 outflows paper:

Error: /rangecheck in --get--
Operand stack:
   pdfmark   --dict:20/25(ro)(L)--   --nostringval--   50
Execution stack:
   %interp_exit   .runexec2   --nostringval--   --nostringval--   --nostringval--   2   %stopped_push   --nostringval--   --nostringval--   --nostringval--   false   1   %stopped_push   1878   1   3   %oparray_pop   1877   1   3   %oparray_pop   1861   1   3   %oparray_pop   1755   1   3   %oparray_pop   --nostringval--   %errorexec_pop   .runexec2   --nostringval--   --nostringval--   --nostringval--   2   %stopped_push   --nostringval--   %finish_show   --nostringval--   --nostringval--   8   6   1   --nostringval--   (pdf_text_enum_t)   %op_show_continue   --nostringval--
Dictionary stack:
   --dict:1153/1684(ro)(G)--   --dict:0/20(G)--   --dict:71/200(L)--   --dict:125/300(L)--   --dict:44/200(L)--   --dict:138/224(L)--
Current allocation mode is local
Last OS error: 2
Current file position is 267478928
GPL Ghostscript 8.71: Unrecoverable error, exit code 1

I get the same error with Ghostscript 8.64, but on my laptop, using the fink version, it works. Similarly, there are errors with the postscript, so I'm led to believe it's an error in latex:

$ latex --version
pdfTeX 3.1415926-1.40.10-2.2 (TeX Live 2009)
kpathsea version 5.0.0
Copyright 2009 Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
Compiled with libpng 1.2.39; using libpng 1.2.39
Compiled with zlib 1.2.3; using zlib 1.2.3
Compiled with xpdf version 3.02pl3

No idea what the cause is but it's time to start documenting steps and looking for a workaround. Compiling on the lappy isn't a good option.

EVLA information

It has been really hard to find EVLA information like beam size, largest angular scale, sensitivity, etc. on the VLA pages because all of the google searches point to old VLA information. The most useful and recent EVLA information on beam size and largest angular scale is here

usetex failure in latex documents

When I use matplotlib's internal tex (rcParams['text.useTex']=False), the postscript files generated cause errors that look like this when you try to ps2pdf them:


ps2pdf h2co_pilot.ps
Error: /rangecheck in --get--
Operand stack:
--dict:20/25(ro)(L)-- --nostringval-- 71
Execution stack:
%interp_exit .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- --nostringval-- --nostringval-- false 1 %stopped_push 1862 1 3 %oparray_pop 1861 1 3 %oparray_pop 1845 1 3 %oparray_pop 1739 1 3 %oparray_pop --nostringval-- %errorexec_pop .runexec2 --nostringval-- --nostringval-- --nostringval-- 2 %stopped_push --nostringval-- %finish_show --nostringval-- --nostringval-- 9 6 0 --nostringval-- (pdf_text_enum_t) %op_show_continue --nostringval--
Dictionary stack:
--dict:1147/1684(ro)(G)-- --dict:0/20(G)-- --dict:70/200(L)-- --dict:116/300(L)-- --dict:44/200(L)-- --dict:25/42(L)--
Current allocation mode is local
Last OS error: 2
Current file position is 791626
GPL Ghostscript 8.64: Unrecoverable error, exit code 1
make: *** [h2co_pilot.pdf] Error 1


They will not open in MacOS's Preview.app either.

Solution: Make figures with rcParams['text.useTex'] = True

RATRAN on Mac OS X

Mac OS X doesn't like the defaults built in to RATRAN. It died unhappily with errors like:

ld_classic: can't locate file for: -lcrt0.o
and
ld: warning: in /usr/local/lib//libcfitsio.a, file is not of required architecture


In order to get it to run, I had to do the following:

1. Install CFITSIO with CFLAGS="-arch x86_64 -arch i386 -g -O2" to /usr/local/lib
   
2. Edit the sky/Makefile OPT variable (line 23) to read:
   OPT = -I. -O2 -fno-automatic -arch x86_64
   

Also, you need to set up system variables:

export RATRAN=/path/to/Ratran 
export RATRANRUN=/path/to/Ratran/run 

Gildas CLASS

It's absurdly difficult to find help on GILDAS Class, probably because you can't google "class" and most people probably don't label every piece of code with "GILDAS class".

Anyway, here are some scripts that I refer back to often:

file in August2009BGPS.dat
file out August2009fits.dat multiple
on error "file out August2009fits.dat"
say "READ IN FILES"
define character sourcelist*10[300]
accept sourcelist /column observed_sources.txt
on error "continue"
get 1001
set window -100 160
set mask -400 -100 160 400
set mode x -400 400
set align velocity
for i 1 to 161
say "Working on SOURCE "'i'
find /source 'sourcelist[i]' /telescope "CSO 4GHZ IF1" /offset 0 0 /quality 5
average
on error "@avplot2 'sourcelist[i]' 'i'; next"
base 3
line 0
min
plot
vis
write i
! on error "continue"
next


and


file in araya-2004.cls
find
define character filename*20
for i 1 to 20
say "Working on source "'i'
get next
let filename "araya-2004_"'i'".fits"
say "fits write "'filename'" /mode spectrum"
fits write 'filename' /mode spectrum
next

!file in araya-2002.cls
!find
!define character filename*20
!for i 1 to 42
! say "Working on source "'i'
! get i
! let filename "araya-2002_"'i'".fits"
! say "fits write "'filename'" /mode spectrum"
! fits write 'filename' /mode spectrum
!next


IRAF append/prepend/replace

IRAF has special syntax to append/replace, so you can do something like:
imarith *.fits - dark.fits ds_//*.fits
...though I can't remember right now how to do replacement etc.

command line & escape key

Problem: in both the bash command line and ipython, if I hit escape twice, I can never again do history-search-backwards (i.e. when you type part of a command and type "up" and reverse-search through your command history). Any idea how to fix this?

Rebuilding python from scratch again

I got scipy working a week or two ago, but doing so killed matplotib's tkagg. So, I switched to the MacOSX backend, which worked ok until I realized that the interactive (connect) features of macosx failed miserably. This led me to try to get matplotlib working.... which broke with those awful "symbol not found" errors in ft2font.so and _path.so, which I've determined all have to do with linking to the wrong library files.

The most worrisome part of this process was discovering that a full Time Machine recovery of /usr and /Library/Frameworks and /Library/Python did *not* restore python - it stayed dead with IDENTICAL errors. So there are probably additional layers of hidden links.

The process below is based on hyperjeff's blog post but differs substantially based on Sam Skillman's recommendations and the very big issue I ran into that my /usr/local files appeared to be corrupted. After this install, my path no longer includes /usr/local/bin and /sw has been moved to /_sw... hopefully one of these days I'll be ballsy enough to delete it.

1. Install python 2.6.4
   
   1. Needed a clean terminal with no flags set at all. Don't know why - all I had set were a bunch of -arch x86_64 flags.
      
export LD_LIBRARY_PATH="/usr/local/lib:/usr/X11/lib"
./configure --enable-framework=/Library/Frameworks MACOSX_DEPLOYMENT_TARGET=10.6
make -j 17
sudo make install

      
   2. Reset PYTHONPATH to blank
      
   3. alias clearflags='export CFLAGS=""; export CCFLAGS=""; export CXXFLAGS=""; export LDFLAGS=""; export FFLAGS="";' to make sure
      
   
   
2. Install FFTW
   
cd ~/tmp
curl -O http://www.fftw.org/fftw-3.2.2.tar.gz
tar xf fftw-3.2.2.tar.gz
cd fftw-3.2.2
clearflags
./configure CC="gcc -arch x86_64" CXX="g++ -arch x86_64" CPP="gcc -E" CXXCPP="g++ -E"
make -j 17
sudo make install

   
3. Install UMFPACK
   
cd ~/tmp
curl -O http://www.cise.ufl.edu/research/sparse/umfpack/current/UMFPACK.tar.gz
curl -O http://www.cise.ufl.edu/research/sparse/UFconfig/current/UFconfig.tar.gz
curl -O http://www.cise.ufl.edu/research/sparse/amd/current/AMD.tar.gz
tar xf AMD.tar.gz
tar xf UFconfig.tar.gz
tar xf UMFPACK.tar.gz
sed -ibck 's/F77 = f77/F77 = gfortran/' UFconfig/UFconfig.mk
sed -ibck '299,303s/# //' UFconfig/UFconfig.mk
cp UFconfig/UFconfig.h AMD/Include/
cp UFconfig/UFconfig.h UMFPACK/Include/
cd UMFPACK
make -j 17
make hb
make clean

   
4. Install numpy
   
   1. Set environment variables
      
export MACOSX_DEPLOYMENT_TARGET=10.6
export CFLAGS="-arch x86_64"
export FFLAGS="-m64"
export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch x86_64"
export PYTHONPATH="/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/"
echo "[amd]
library_dirs = /Users/adam/tmp/AMD/Lib
include_dirs = /Users/adam/tmp/AMD/Include
amd_libs = amd

[umfpack]
library_dirs = /Users/adam/tmp/UMFPACK/Lib
include_dirs = /Users/adam/tmp/UMFPACK/Include
umfpack_libs = umfpack" > site.cfg

      
   2. Setup & Install
      
python setup.py build --fcompiler=gnu95
sudo python setup.py install

      
   3. Test: python -c "import numpy"
      
   
   
5. Install scipy. The important thing is to use g++-4.2 because g++-4.5 doesn't accept the -arch flag. Also, get rid of /sw if it's on your computer at all.
   
sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyX
cd ~/repos/scipy-0.7.1
python setup.py clean
rm -rf build
clearflags
FFLAGS="-m64" CFLAGS="-arch x86_64 -I/usr/local/include/freetype2 -I/usr/X11/include -L/usr/X11/lib" LDFLAGS="-Wall -undefined dynamic_lookup -bundle -lpng -arch x86_64" CXX="/usr/bin/g++-4.2" CC="/usr/bin/gcc-4.2" python setup.py build
python setup.py install

   Test the install:
   
python -c "import scipy, scipy.fftpack, scipy.interpolate"

   
6. 1. Install matplotlib. MAKE SURE /usr/bin/texbin is in front of /usr/local/bin and /sw/bin so that dvipng comes from MacTEX. I also ended up having to remove /usr/local/bin from my path completely
      
sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyX
cd ~/repos/matplotlib-svn
python setup.py clean
rm -rf build
clearflags

      
   2. Do hyperjeff's recommended edits except don't use /usr/local because it's f'd:
      make.osx:
      
MACOSX_DEPLOYMENT_TARGET=10.6
PREFIX=/usr
PYTHON=/Library/Frameworks/Python.framework/Versions/Current/bin/python

## You shouldn't need to configure past this point (and yet…)

PKG_CONFIG_PATH="${PREFIX}/lib/pkgconfig"
CFLAGS="-arch i386 -arch x86_64 -I${PREFIX}/include -I${PREFIX}/include/freetype2 -isysroot /Developer/SDKs/MacOSX10.6.sdk"
LDFLAGS="-arch i386 -arch x86_64 -L${PREFIX}/lib -syslibroot,/Developer/SDKs/MacOSX10.6.sdk"
FFLAGS="-arch i386 -arch x86_64"

      setup.cfg:
      
wxagg = False

      
   3. Do the install (different from hyperjeff b/c I don't want root)
      sudo make -f make.osx fetch deps
make -f mpl_build mpl_install
python setup.py install

      
   
   
7. Install setuptools
   
8. easy_install ipython
   
9. install everything else pythonically
   

apache/php/mysql on snow leopard

This post helped me out

macvim crash

well, it finally happened.... my reliable, trusty editor crashed. That should be impossible. I am ready to call it quits for the week....

Process:         MacVim [650]
Path:            /Applications/Vim.app/Contents/MacOS/MacVim
Identifier:      org.vim.MacVim
Version:         7.2 (49)
Code Type:       X86 (Native)
Parent Process:  Vim [649]

Date/Time:       2010-02-25 13:12:43.001 -0700
OS Version:      Mac OS X 10.6.2 (10C540)
Report Version:  6

Interval Since Last Report:          871676 sec
Crashes Since Last Report:           26
Per-App Interval Since Last Report:  938504 sec
Per-App Crashes Since Last Report:   1
Anonymous UUID:                      03159B9E-2257-4E38-8C4A-4D4DAF5641A7

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: 0x000000000000000d, 0x0000000000000000
Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Thread 0 Crashed:  Dispatch queue: com.apple.main-thread
0   com.apple.CoreFoundation       0x99119480 __CFSetCallback + 0
1   com.apple.CoreFoundation       0x990c78bc ___CFBasicHashFindBucket1 + 444
2   com.apple.CoreFoundation       0x990cfaac CFBasicHashFindBucket + 252
3   com.apple.CoreFoundation       0x990e8293 CFSetGetValue + 131
4   com.apple.AppKit               0x961bae7e -[NSWindow _discardTrackingRect:] + 59
5   com.apple.AppKit               0x961badca -[NSView(NSInternal) _uninstallTrackingArea:] + 123
6   com.apple.AppKit               0x960d2c32 -[NSView(NSInternal) _uninstallRemovedTrackingAreas] + 293
7   com.apple.AppKit               0x960dac40 -[NSView(NSInternal) _updateTrackingAreas] + 646
8   com.apple.CoreFoundation       0x990ea4e0 CFArrayApplyFunction + 224
9   com.apple.AppKit               0x960daefb -[NSView(NSInternal) _updateTrackingAreas] + 1345
10  com.apple.CoreFoundation       0x990ea4e0 CFArrayApplyFunction + 224
11  com.apple.AppKit               0x960daefb -[NSView(NSInternal) _updateTrackingAreas] + 1345
12  com.apple.CoreFoundation       0x990ea4e0 CFArrayApplyFunction + 224
13  com.apple.AppKit               0x960daefb -[NSView(NSInternal) _updateTrackingAreas] + 1345
14  com.apple.AppKit               0x960da8db _handleInvalidCursorRectsNote + 392
15  com.apple.CoreFoundation       0x99135892 __CFRunLoopDoObservers + 1186
16  com.apple.CoreFoundation       0x990f218d __CFRunLoopRun + 557
17  com.apple.CoreFoundation       0x990f1864 CFRunLoopRunSpecific + 452
18  com.apple.CoreFoundation       0x990f1691 CFRunLoopRunInMode + 97
19  com.apple.HIToolbox            0x936f6f0c RunCurrentEventLoopInMode + 392
20  com.apple.HIToolbox            0x936f6bff ReceiveNextEventCommon + 158
21  com.apple.HIToolbox            0x936f6b48 BlockUntilNextEventMatchingListInMode + 81
22  com.apple.AppKit               0x960b0ac5 _DPSNextEvent + 847
23  com.apple.AppKit               0x960b0306 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 156
24  com.apple.AppKit               0x9607249f -[NSApplication run] + 821
25  com.apple.AppKit               0x9606a535 NSApplicationMain + 574
26  org.vim.MacVim                 0x0000238b _start + 209
27  org.vim.MacVim                 0x000022b9 start + 41

Thread 1:  Dispatch queue: com.apple.libdispatch-manager
0   libSystem.B.dylib              0x98d0c0ea kevent + 10
1   libSystem.B.dylib              0x98d0c804 _dispatch_mgr_invoke + 215
2   libSystem.B.dylib              0x98d0bcc3 _dispatch_queue_invoke + 163
3   libSystem.B.dylib              0x98d0ba68 _dispatch_worker_thread2 + 234
4   libSystem.B.dylib              0x98d0b4f1 _pthread_wqthread + 390
5   libSystem.B.dylib              0x98d0b336 start_wqthread + 30

Thread 2:
0   libSystem.B.dylib              0x98ce58da mach_msg_trap + 10
1   libSystem.B.dylib              0x98ce6047 mach_msg + 68
2   com.apple.CoreFoundation       0x990f277f __CFRunLoopRun + 2079
3   com.apple.CoreFoundation       0x990f1864 CFRunLoopRunSpecific + 452
4   com.apple.CoreFoundation       0x990f1691 CFRunLoopRunInMode + 97
5   com.apple.Foundation           0x91b24430 +[NSURLConnection(NSURLConnectionReallyInternal) _resourceLoadLoop:] + 329
6   com.apple.Foundation           0x91aeb8d8 -[NSThread main] + 45
7   com.apple.Foundation           0x91aeb888 __NSThread__main__ + 1499
8   libSystem.B.dylib              0x98d12fbd _pthread_start + 345
9   libSystem.B.dylib              0x98d12e42 thread_start + 34

Thread 3:
0   libSystem.B.dylib              0x98d04856 select$DARWIN_EXTSN + 10
1   com.apple.CoreFoundation       0x99131ddd __CFSocketManager + 1085
2   libSystem.B.dylib              0x98d12fbd _pthread_start + 345
3   libSystem.B.dylib              0x98d12e42 thread_start + 34

Thread 4:
0   libSystem.B.dylib              0x98d0b182 __workq_kernreturn + 10
1   libSystem.B.dylib              0x98d0b718 _pthread_wqthread + 941
2   libSystem.B.dylib              0x98d0b336 start_wqthread + 30

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00515db0  ebx: 0x990c7711  ecx: 0x00516460  edx: 0xbfffcabc
  edi: 0x00001041  esi: 0x00504270  ebp: 0xbfffca38  esp: 0xbfffc99c
   ss: 0x0000001f  efl: 0x00010246  eip: 0x99119480   cs: 0x00000017
   ds: 0x0000001f   es: 0x0000001f   fs: 0x00000000   gs: 0x00000037
  cr2: 0x97a20000

Command of the Day

ls | wc

and for when ls * fails

find . -name "*" | xargs -I {} mv {} away/

DS9 gaussian fitting

One thing DS9 desperately needs is an interactive gaussian fitter. I have NOT implemented one yet, but it is high on my to do list. Has anyone else (googlers?) tried or succeeded in implementing such a thing? Ideally, something with NO dependencies: if I write one, it will require python, numpy, and probably pyds9 - ick. Straight-up TCL would be very much preferable.

Another attempt to get 64 bit python on Snow Leopard

Sam Skillman posted his attempt to get 64 bit python on snow leopard. It worked, but you need to install python with --enable-framework and NOT --prefix and NOT --enable-universalSDK. It is 64-bit only, not universal: universal causes trouble.