Home > Info > Telescope & Instruments > Pipelines

Pipelines
Different pipelines exist for different instruments, based on their performance characteristics and what we want to do with the data in terms of archiving.

• RATCam pipeline
• SupIRCam pipeline
• RINGO pipeline
• FRODOSpec L2 pipeline

RATCam pipeline [go to RATCam page]

Basic instrumental reductions are applied to all RATCam images before the data are passed to users. This includes bias subtraction, trimming of the overscan regions and flat fielding. A library of the current calibration frames is maintained as part of the data archive and updated daily so that images are always reduced using the latest available flat-field image available at the time. Each of the operations are performed as described below.

Bias Subtraction
There is insufficient repeatable structure for bias frames to be useful. Bias subtraction is therefore based purely on analysis of the underscan region. Linear regression is used to determine a fit to the bias counts as a function of pixel row number and values deducted across the image according to this smooth function. Experience shows that RATCam does have a small ramp in the bias down each column and this first order fit is required. No attempt is made to remove any bias gradient in rows across the image.

Overscan Trimming
The overscan regions are trimmed off the image leaving a 2048x2048 (assuming on-chip binning was not used) pixel image.

Dark Substraction
This is not currently performed though the facility exists in the reduction pipeline. Experience has shown that when the camera is at normal operating temperature, dark current is not significant. If the camera temperature fluctuates for any reason, the dark current is sufficiently variable that a dark would need to be obtained with every frame and library dark frames are no use.

Flat Fielding
The appropriate master flat field is selected from the library to match the filter and binning configuration of the current exposure. In fact the library holds reciprocal flat-fields normalised to unity because of the computational efficiency of multiplying rather than dividing. The image data are therefore multiplied by the library flat.

Each twilight the instrument control software (ICS) attempts to update the oldest master flats in its library, by taking 3-5 raw sky flats for each filter/binning combination, giving preference to the most used. There usually isn't enough time to take sky flats for all combinations in one twilight session, so the next oldest in the list is attempted in following nights and so on. Usually the update is complete after 2-3 nights and the process starts again.

The master flat is derived from the median of each sky flat after each has been normalised to the common mean count level. New master flats, and their corresponding ratio image of new/old masters, are inspected for any corruption, inclusion of stars, etc.

Bad Pixel Mask
No cosmic ray rejection or bad pixel mask is applied since it is important for users performing accurate photometry to know exactly what masking has been applied. However, bad pixel masks have been generated and kindly made available by the Angstrom project, a gravitational lensing programme underway on the LT and RoboNet:

• unbinned bad pixel mask (13KB gzipped FITS, unzips to 8MB)
• 2x2 binned bad pixel mask (5KB gzipped FITS, unzips to 2MB)

Vignetting
The filter wheel slightly vignettes the optical beam to a different extent for each filter. In the extreme corners of the worst affected bands (i' and g') the flux is reduced by up to 15% compared to the unobstructed beam. In the other filters, obscuration is about 5% in the very corner of the observed field, falling to negligible values between 10 - 20 arcsec from the field edge.

The vignetting generally flat fields out very well and is rarely obvious in the reduced data, sometimes leaving distortions only in the range 3 - 5 arcsec from the field corner. Even where the data is well flat fielded though, noise characteristics of photon counting statistics could be affected in these regions of the frame.

Fringe Frames
We currently do not perform any automated defringing on CCD data before it is loaded into the archive. In order to help you defringe your own data, linked below are prepared master fringe frames created by stacking multiple deep integrations of blank fields. The master fringe frames are updated infrequently because the fringes on the CCD have been found to vary only on timescales of months. If you need access to the individual integrations which go into building these master frames, they are publicly available from the data archive. Simply select RATFringe from the Proposal ID drop-down list. You can therefore extract the most recent fringe frame from the archive at any time.

Legacy Fringe Frames. Prior to July 2006 SDSS-I, binned 2x2 (4MB gzipped FITS, unzips to 8MB) SDSS-Z, binned 2x2 (2.3MB gzipped FITS, unzips to 4MB) Legacy Fringe Frames. Created July/August 2006 SDSS-I, binned 1x1, stack of 46 x 300 second exposures, 13MB gzipped fits SDSS-Z, binned 2x2, stack of 27 x 300 second exposures, 3.1MB gzipped fits SDSS-I, binned 2x2, stack of 27 x 300 second exposures, 3.1MB gzipped fits SDSS-R, binned 2x2, stack of 27 x 300 second exposures, 2.6MB gzipped fits, fringes barely visible. SDSS-Z, binned 1x1, stack of 27 x 300 second exposures, 11MB gzipped fits Legacy Fringe Frames. Created September 2007 from data obtained Jan - Aug 2007. SDSS-Z, binned 1x1, stack of 36 x 300 second exposures, 15MB gzipped fits SDSS-Z, binned 2x2, stack of 33 x 300 second exposures, 4MB gzipped fits SDSS-I, binned 1x1, stack of 18 x 300 second exposures, 15MB gzipped fits SDSS-I, binned 2x2, stack of 34 x 300 second exposures, 4MB gzipped fits Legacy Fringe Frames. Created December 2007 from data obtained Sep - Dec 2007. These are best suited to any data obtained in 2007 after the September mirror recoating and servicing. I.e., the majority of semester 07B. SDSS-I, binned 1x1, stack of 216 x 300 second exposures, 15MB gzipped fits SDSS-I, binned 2x2, stack of 18 x 300 second exposures, 4MB gzipped fits SDSS-Z, binned 1x1, stack of 18 x 300 second exposures, 15MB gzipped fits SDSS-Z, binned 2x2, stack of 18 x 300 second exposures, 4MB gzipped fits Legacy Fringe Frames. Created Feb 2008 from data obtained Jan - Feb 2008. SDSS-I, binned 1x1, stack of 90 x 300 second exposures, 15MB gzipped fits Created Jan 2009 from data obtained throughout 2008. SDSS-I, binned 1x1, 90 x 300sec exposures from 08B, 15MB gzipped fits SDSS-I, binned 2x2, Rebinned version of above 1x1 data, 4MB gzipped fits SDSS-Z, binned 1x1, 72 x 300sec exposures from 08A and 08B, 15MB gzipped fits SDSS-Z, binned 2x2, Rebinned version of above 1x1 data, 4MB gzipped fits Created Jan 2010 from data obtained throughout 2009. SDSS-I, binned 1x1, 112 x 300sec exposures from 09A-09B, 15MB gzipped fits SDSS-I, binned 2x2, 93 x 300sec exposures from 09A-09B, 4MB gzipped fits SDSS-Z, binned 1x1, 122 x 300sec exposures from 09A-09B, 15MB gzipped fits SDSS-Z, binned 2x2, 36 x 300sec exposures from 09A-09B, 4MB gzipped fits

 

FITS Header Error Codes
Error codes are written into the FITS headers for specific error flags brought up during processing. These error flags are stored in the header keywords L1STATOV, L1STATZE, L1STATTR, L1STATFL and L1STATDA and follow this convention:

• -ve values are failure states (Special case: -1 = operation not attempted)
• +ve values are warning states (Special case: +1 = no errors or warnings)

Tables of failure and warning states are given below - but please note these are not exhaustive lists.

FITS HEADER FAILURE STATE CODES
Code #	Process Name	Error String
-12	DpRT_startup	Input is not a valid LT filename.
-14	DpRT_startup	According to filename flags, input has already been processed.
-15	DpRT_startup	Run_mode is not valid (%d). See dprt.h for valid values.
-18	DpRT_startup	Invalid exposure type flag.
-35	DpRT_zero	Cannot find zero (bias) frame file.
-37	DpRT_dark	Cannot find dark frame file.
-38	DpRT_flat	Mean counts in flat <= 0. Processing abandoned.
-40	DpRT_flat	Cannot find flatfield file.
-60	DpRT_fringe	Cannot find fringe file.
-61	DpRT_fringe	Fringe and data frame sizes do not match.
-63	DpRT_fringe	Correlation scaling failed. No defringing performed.
-73	DpRT_dark	Correlation scaling failed. No dedarking performed.
-261		Error opening FITS.
-263		Error getting header keywords.
-264		Non-square binning.
-268		Error reading image array.
-354	DpRT_make_bias	Could not open working directory.
-356	DpRT_make_bias	Failure to allocate memory for **fits_pointers.
-357	DpRT_make_bias	Failed to open FITS file.
-358	DpRT_make_bias	Failed to allocate memory to median_array or new_bias_array.
-359	DpRT_make_bias	Failed to allocate memory to temp_array.
-360	DpRT_make_bias	Failed to allocate memory to temp_array[%d].
-361	DpRT_make_bias	Failed to read image data from FITS file number %d.
-363	DpRT_make_bias	Poor stats in new_bias.
-453	DpRT_make_flat	Could not read file (outer loop).
-454	DpRT_make_flat	Could not open directory %s (inner loop).
-462	DpRT_make_flat	Fewer than %d good flats from which to make master flat.
-452	DpRT_make_flat	Could not read existing flat frame.
-356	DpRT_make_flat	Failure to allocate memory for **fits_pointers.
-364	DpRT_make_bias	New bias is highly deviant from the old one: mean abs dev = %.
-454	DpRT_make_flat	Could not open directory.
-457	DpRT_make_flat	Failed to open FITS file.
-458	DpRT_make_flat	Failed to allocate memory to median_array or [mean|median]_flat_array.
-459	DpRT_make_flat	Failed to allocate memory to temp_array.
-460	DpRT_make_flat	Failed to allocate memory to temp_array[%d].
-461	DpRT_make_flat	Failed to read image data from FITS file number %d.

FITS HEADER WARNING STATE CODES
Code #	Process Name	Error String
32	DpRT_startup	CCD not at thermal set point.
52	DpRT_output	No filter calibration data for filter.
53	DpRT_output	No filter calibration data.
62	DpRT_fringe	No correlation between data and fringe. Defringing will use simple exposure time scaing.
64	DpRT_fringe	Correlation scaling failed. Simple exposure time scaing will be used.
72	DpRT_dark	No correlation between data and dark. Dedarking will use simple exposure time scaing.
220	DpRT_init	No '.' character in flatfield filename.
225	DpRT_init	Divide-by-zeroes were safely trapped.
226	DpRT_init	Failed to trim the flat field.
230	DpRT_init	Failed to open FLATLIB directory.
232	filter_params	Failed to open filter parameter lookup table.
233	filter_params	Failed to parse filter config line.
231	filter_params	Failed to allocate name memory for filter.
234	filter_params	Filter config line has a name > 49 chars.
235	filter_params	Filter scale factor or ZP read from file is not valid for filter.
236	filter_params	Failed to allocate filter memory.
251	DpRT_init	Failed to allocate memory for *photstar_data. Calibration data will not be available.
252	DpRT_init	Failed to open photstar parameter lookup table.
351	DpRT_make_bias	Could not open log file.
352	DpRT_make_bias	Could not read existing bias frame.
353	DpRT_make_bias	Could not read file.
355	DpRT_make_bias	Too many good bias frames were found.
362	DpRT_make_bias	New bias is highly discrepant from library bias.
366	DpRT_make_bias	More bias frames were discarded than kept.
367	DpRT_make_bias	Fewer than %d good bias frames.
363	DpRT_make_flat	Poor stats in new [mean|median] flat.
364	DpRT_make_flat	New flat is highly deviant from the old one.
365	DpRT_make_flat	No suitable flat frames found in directory. %d were read but discarded.
453	DpRT_make_flat	Could not read file (inner loop).
500 - 507	DpRT_init	NULL pointer in dprt_*_lib.

 

SupIRCam pipeline [go to SupIRCam page]

All SupIRcam data since April 2007 has been run through a data reduction pipeline before being distributed to users. This pipeline performs linearity correction, dark current subtractions, flat-fields and sky subtraction. It is important to note that the sky image is generated by buidling a median stack of all the dither frames (suitably smoothed) within a single observation, typically either 5 or 9 frames. If there is a large extended source in the field, it is possible that this will be included in the sky image and therefore some small fraction of its flux be subtracted from each image. The sky image used is included in a FITS image extension called SKY within each distributed image for user inspection, and allowing users to remove the sky subtraction stage from the reduced image if they need.

As an example, to use ds9 to inspect the sky frame which was automatically generated and used in the reduction of frame s_e_20081010_1_1_1.fits, you could use the command
ds9 -zscale s_e_20081010_1_1_1.fits "s_e_20081010_1_1_1.fits[sky]"

By contrast the flat field used in the reduction is generated from a very large sample (many thousands) of recent observations and so is not affected by this problem (the flat field stability being much greater than the sky stability).

Data which predate the availabilty of this pipeline (i.e., prior to April 2007) are only available as raw data. They are not loaded into the main data archive but may be downloaded from Unloaded Data.

Any queries or comments on the data should be passed to LT Support in the usual fashion.

 

RINGO pipeline [go to RINGO page]
RINGO data are dark subtracted and flat-fielded before being distributed to the user. Error codes are identical those defined in the RATCam pipeline description.

For the subsequent stages of data reduction, please contact Iain Steele or Robert Smith for advice.

Data which predate the availabilty of the automated pipeline (i.e., prior to March 2007) are only available as raw data. They are not loaded into the main data archive but may be downloaded from Unloaded Data.

 

FRODOSpec L2 pipeline [go to FRODOSpec page]

The L2 pipeline reduction process uses three files, the "science" frame, the "continuum" frame and the "arc" frame. It is best visualised using this schematic:

1. Fibre tramline identification using the continuum frame (typically taken of the twilight sky) and polynomial mapping
   frfind, frclean, frtrace

   ---

   Fibre profile peak positions are determined and a pixel coordinate polynomial is fitted for each fibre.
   
   
   
2. Standard aperture flux extraction
   frextract

   ---

   Using the tramline mappings, the flux is extracted from each fibre in the science frame, continuum frame and arc frame.
   
   Under the criteria taken from the FRODOSpec specification, a fibre profile is contained spatially within 7 pixels.
   
   
   
3. Wavelength mapping on a fibre-to-fibre basis
   frarcfit

   ---

   The nearest-in-time arc file is extracted and emission line pixel positions identified. These positions are compared with a file containing known arc line positions and associated wavelengths. FRODOSpec is a bench mounted spectrograph, and as such, typical line positions vary of the order ~1px over a typical temperature range. The routine used is tolerant to this factor.
   
   A cross-correlation routine is first ran on each fibre to remove the constant offset. Unique pixel to wavelength solutions are then found for each fibre by polynomial interpolation.
   
   
   
4. Fibre transmission correction
   frcorrectthroughput

   ---

   Although the fibres are manufactured to the same standards and using the same methods, their transmittance is not always identical (due to small manufacturing defects). In order to correct for this, the extracted continuum frame (assumed to be uniformly illuminated) is taken and the total flux through each fibre determined. Transmission corrections can then be found and applied to the science frame.
   
   
   
5. Realignment of data to a common reference point and rebinning of data to a linear wavelength scale
   frrebin

   ---

   In order to obtain a single wavelength solution applicable to all fibres, each fibre is rebinned to a linear wavelength scale with the same starting/ending wavelength positions and same pixel to wavelength dispersions.
   
   
   
6. Sky fibre identification
   fridsky

   ---

   An iterative sigma-clipping procedure is used to determine if any sky fibres are present in the science frame.
   
   
   
7. Sky subtraction (if applicable)
   frsubsky

   ---

   The sky contribution to flux is removed from the science frame if the fridsky returns success in identifying sky fibres.
   
   
   
8. Formatting of data product
   frreformat

   ---

   A multipart file with the following data product is then formatted:

   HDU Index	EXTNAME	Details
   0	L1_RAW	Raw L1 reduced
   1	RSS_NONSS	Non sky subtracted row stacked spectra
   2	CUBE_NONSS	Non sky subtracted datacube
   3	RSS_SS	Sky subtracted row stacked spectra
   4	CUBE_SS	Sky subtracted datacube
   5	SPEC_NONSS	Non sky subtracted 1D spectrum
   6	SPEC_SS	Sky subtracted 1D spectrum
   7	COLDATACUBE_NONSS	Non sky subtracted collapsed datacube

   If sky fibres are not found in fridsky, the corresponding HDUs will be blank.
   
   Spectra are constructed using only the flux from the four brightest fibres, determined by first smoothing the data spectrally with a median boxcar (to restrain the influence of cosmic rays to total flux calculations). The outputted spectral data is not smoothed.
   
   

Although the FRODOSpec L2 reduction process has been written to produce reduced content for all science programmes, it is primarily meant for the qualification of how usable the data taken is, rather than to produce the optimal reduction product.

The L2 pipeline is specifically tailored for the observation of point sources. Those programmes with observations of extended sources may be best using an earlier reduction product to perform their analysis.

© ARI 2010