IRIS2 Data Reduction - How to

On-line processing at the AAT

The latest version of ORAC-DR runs on aatlxa, a PC in the control room running Centos 5.4 Linux. Log in to this machine as

iris2red (and enter the iris2red password from the blue folder on the shelf above aatlxa)


oracdr -from 1 -loop wait &

then sit back and enjoy! Note that if iris2 has been set-up to run on aatvme14 (as opposed to aatvme10) then before starting oracdr you will need to as type:

setenv ORAC_DATA_IN /data_vme14/aatobs/iris2_data/YYMMDD

where YYMMDD is the current date. To re-process data from a different night, follow the steps below for off-line processing (except the source /applic/... step). If you wish to re-use dome flats from a previous night, follow these instructions.

Reduced data is written to /iris2_reduce/iris2red/<yymmdd>/. If this disk fills up, you should feel free to purge old data directories from previous IRIS2 runs. Do not execute any other Starlink tasks from this window, or it may cause ORAC-DR to fail, or give odd error messages.

Off-line processing on your AAT visitor account

To run ORAC-DR using your visitor account at the AAT, open up a terminal and type


Then type

oracdr_iris2 ut 

where ut is the UT date on which your data was taken, in the format YYYYMMDD. 

Then, define the IN and OUT folders

setenv ORAC_DATA_IN full directory path where data files are

setenv ORAC_DATA_OUT full directory path where output will go

oracdr command-line options

ORAC-DR will open up a GUI to report status, warnings, and errors, as well as any GAIA image displays (one for the most recent processed frame, and another for completed mosaics) or GWM plotting windows required.

ORAC-DR can be stopped and re-started easily. Simply click "Exit ORAC-DR" in the GUI. It can then be re-started with the oracdr command, with various command-line options. For instance, to re-start from observation number 57 while still taking data, type
oracdr -from 57 -loop wait & ).

You can use gaia to view the reduced images. Simply type

gaia &

 Reprocessing tricks

It is not uncommon at the AAT for exposure sequences to be curtailed by weather, or cloud may interfere with a few exposures from your sequence. Or you may decide you want to process the best seeing images from a sequence seperately. Can you do that? Yes you can - but because ORAC-DR is primarily driven by the content in the file headers, you need to update the raw data files files. There are three ways you can do this - modify the FITS headers directly using a Perl script; copy the raw FITS files to a new location, edit the headers and reprocess these files; or make ORAC-DR create NDF copies of the raw data and edit those.

Modify the raw FITS files in situ

There is now a Perl script which can do all this for you.

Usage: orac_fits_fix [-n NOFFSETS] [-r RECIPE] FITS_file1 ... FITS_filen
(if filenames are specified, the number of files should be multiples of NOFFSETS) 
Given a set of files, it will recreate the GRPNUM, GRPMEM etc. keywords so that they can be sequentially processed by recipes such as JITTER_SELF_FLAT, etc. You can also change the NOFFSETS keyword (by default it sets NOFFSETS to the number of files provided on the command line), which determines the number of images that must be acquired before a new flatfield and reduced mosaic is created.

For example, 
orac_fits_fix file[1-9] will fix headers to process as a group of 9 files in a mosaic;
orac_fits_fix -n 4 file[1-8] will fix headers to process as a group of 8 files, making a new flatfield and mosaic after every 4 files;
orac_fits_fix -r JITTER_SELF_FLAT_KEEPBAD file[1-8] will fix headers to make a mosaic of 8 files, with recipe JITTER_SELF_FLAT_KEEPBAD

Copying the raw data and editing the FITS Headers (e.g. in IRAF)

  1. Create a new directory for the files to edit, and copy them. For example suppose we had 15 H-band images taken on 09APR2003 as runs 94-108, with only runs 100-106 having data free from clouds:
    mkdir /iris2_raw/iris2red/030409
    cp /data_vme14/aatobs/iris2_data/030409/09apr010[0-6].fits /iris2_raw/iris2red/030409/
  2. Tell ORAC-DR where to find these files:
    setenv ORAC_DATA_IN /iris2_raw/iris2red/030409
  3. Edit the file headers. You can use whatever FITS header editor you prefer. Suppose you prefer to use IRAF - the hedit task in the images package can edit FITS headers. Start an xgterm, go to the raw data directory and start editing:
    xgterm &
    Then in the new xgterm:
    cd /iris2_raw/iris2red/030409
    hedit 09apr*.fits field=grpnum value=100 update+ verify-

    will change all the 09apr files to be in group 100 (instead of the group 94 they were before).
    hedit 09apr0100.fits field=grpmem value=1 update+ verify-
    hedit 09apr0101.fits field=grpmem value=2 update+ verify-
    hedit 09apr0106.fits field=grpmem value=7 update+ verify-

    will update the files 100-106 to be members 1-7 of group 100 (instead of the members 7-13 of group 94 they were before), and
    hedit 09apr*.fits field=noffsets value=7 update+ verify-
    will tell ORAC-DR's JITTER_SELF_FLAT recipe (the one originally specified when the data was taken) to make a flat and mosaic from 7 images, instead of the 15 originally specified. This example will not cover all cases - in some cases if you have complicated interleavings of objects and sky you won't be able to do pipeline reduction of partial data sets. JITTER_SELF_FLAT (especially with random dithering) however is pretty robust so long as you have more than 3 decent frames.
    Other FITS header entries you might need to edit are:
     ('OBJECT' or 'SKY' etc.)
    RUN <Run_number>
    OBSNUM <Run_number>
  4. Now reprocess the data (ensuring that you have not invoked CONVERT)
    oracdr -skip -list 100:105,106 &
    The -skip option will be necessary if your run numbers are not contiguous (since otherwise ORAC-DR will sit waiting for the next run, even if it's not there). Your newly processed data will appear in the same directory as usual.
  5. Now if you have redefined ORAC_DATA_IN, make ORAC-DR go back to looking for data in the usual place:

Editing the FITS headers as NDF files.

  1. Since ORAC-DR only ever operates on NDF copies of the raw data, you can make ORAC-DR create NDF copies of the raw data in the $ORAC_DATA_OUT directory using the QUICK_LOOK recipe:

    oracdr -skip -list 100:105,106 QUICK_LOOK &

    This leaves NDF copies of all the input images in $ORAC_DATA_OUT. You can now edit the fits headers of these files with the kappa fitsmod (link is external) task, but you must not do this from the window in which ORAC-DR is being run. Open a new terminal window, and type:

    cd /iris2_reduce/iris2red/030409 (or wherever $ORAC_DATA_OUT corresponds to)

    fitsmod 09apr0100 edit=U keyword=GRPNUM position=! value=100 comment=\$C

    etc. Scripts are available if you need to change the group membership, or number of offsets for a large number of files.

  2. The guidelines for the changes you make are the same as item 3 above, as are the ORAC-DR commands at item 4 to actually redo the processing.