Pipelines

Different pipelines exist for different instruments, based on their performance characteristics and how the data is to be archived.

RATCam pipeline

Basic instrumental reductions are applied to all RATCam images before the data are passed to users. This includes bias subtraction, trimming of the overscan regions and flat fielding. A library of the current calibration frames is maintained as part of the data archive and updated daily so that images are always reduced using the latest available flat-field image available at the time. Each of the operations are performed as described below.

Bias Subtraction

There is insufficient repeatable structure for bias frames to be useful. Bias subtraction is therefore based purely on analysis of the underscan region. Linear regression is used to determine a fit to the bias counts as a function of pixel row number and values deducted across the image according to this smooth function. Experience shows that RATCam does have a small ramp in the bias down each column and this first order fit is required. No attempt is made to remove any bias gradient in rows across the image.

Overscan Trimming

The overscan regions are trimmed off the image leaving a 2048x2048 (assuming on-chip binning was not used) pixel image.

Dark Substraction

This is not currently performed though the facility exists in the reduction pipeline. Experience has shown that when the camera is at normal operating temperature, dark current is not significant. If the camera temperature fluctuates for any reason, the dark current is sufficiently variable that a dark would need to be obtained with every frame and library dark frames are no use.

Flat Fielding

The appropriate master flat field is selected from the library to match the filter and binning configuration of the current exposure. In fact the library holds reciprocal flat-fields normalised to unity because of the computational efficiency of multiplying rather than dividing. The image data are therefore multiplied by the library flat.

Each twilight the instrument control software (ICS) attempts to update the oldest master flats in its library, by taking 3-5 raw sky flats for each filter/binning combination, giving preference to the most used. There usually isn't enough time to take sky flats for all combinations in one twilight session, so the next oldest in the list is attempted in following nights and so on. Usually the update is complete after 2-3 nights and the process starts again.

The master flat is derived from the median of each sky flat after each has been normalised to the common mean count level. New master flats, and their corresponding ratio image of new/old masters, are inspected for any corruption, inclusion of stars, etc.

up to top of page

Bad Pixel Mask

No cosmic ray rejection or bad pixel mask is applied since it is important for users performing accurate photometry to know exactly what masking has been applied. However, bad pixel masks have been generated and kindly made available by the Angstrom project, a gravitational lensing programme underway on the LT and RoboNet:

Vignetting

The filter wheel slightly vignettes the optical beam to a different extent for each filter. In the extreme corners of the worst affected bands (i' and g') the flux is reduced by up to 15% compared to the unobstructed beam. In the other filters, obscuration is about 5% in the very corner of the observed field, falling to negligible values between 10 - 20 arcsec from the field edge.

The vignetting generally flat fields out very well and is rarely obvious in the reduced data, sometimes leaving distortions only in the range 3 - 5 arcsec from the field corner. Even where the data is well flat fielded though, noise characteristics of photon counting statistics could be affected in these regions of the frame.

Fringe Frames

We currently do not perform any automated defringing on CCD data before it is loaded into the archive. In order to help you defringe your own data, linked below are prepared master fringe frames created by stacking multiple deep integrations of blank fields. The master fringe frames are updated infrequently because the fringes on the CCD have been found to vary only on timescales of months. If you need access to the individual integrations which go into building these master frames, they are publicly available from the data archive. Simply select RATFringe from the Proposal ID drop-down list. You can therefore extract the most recent fringe frame from the archive at any time.

 

  • Legacy Fringe Frames. Prior to July 2006
  • Legacy Fringe Frames. Created July/August 2006
  • Legacy Fringe Frames. Created September 2007 from data obtained Jan - Aug 2007.
  • Legacy Fringe Frames. Created December 2007 from data obtained Sep - Dec 2007. These are best suited to any data obtained in 2007 after the September mirror recoating and servicing. I.e., the majority of semester 07B.
  • Legacy Fringe Frames. Created Feb 2008 from data obtained Jan - Feb 2008.
  • Created Jan 2009 from data obtained throughout 2008.
  • Created Jan 2010 from data obtained throughout 2009.

up to top of page

FITS Header Error Codes

Error codes are written into the FITS headers for specific error flags brought up during processing. These error flags are stored in the header keywords L1STATOV, L1STATZE, L1STATTR, L1STATFL and L1STATDA and follow this convention:

  • negative values are failure states (special case: -1 = operation not attempted)
  • positive values are warning states (special case: +1 = no errors or warnings)

Tables of failure and warning states are given below - but please note these are not exhaustive lists.

FITS HEADER FAILURE STATE CODES

Code #

Process Name

Error String

-12

DpRT_startup

Input is not a valid LT filename.

-14

DpRT_startup

According to filename flags, input has already been processed.

-15

DpRT_startup

Run_mode is not valid (%d). See dprt.h for valid values.

-18

DpRT_startup

Invalid exposure type flag.

-35

DpRT_zero

Cannot find zero (bias) frame file.

-37

DpRT_dark

Cannot find dark frame file.

-38

DpRT_flat

Mean counts in flat <= 0. Processing abandoned.

-40

DpRT_flat

Cannot find flatfield file.

-60

DpRT_fringe

Cannot find fringe file.

-61

DpRT_fringe

Fringe and data frame sizes do not match.

-63

DpRT_fringe

Correlation scaling failed. No defringing performed.

-73

DpRT_dark

Correlation scaling failed. No dedarking performed.

-261

Error opening FITS.

-263

Error getting header keywords.

-264

Non-square binning.

-268

Error reading image array.

-354

DpRT_make_bias

Could not open working directory.

-356

DpRT_make_bias

Failure to allocate memory for **fits_pointers.

-357

DpRT_make_bias

Failed to open FITS file.

-358

DpRT_make_bias

Failed to allocate memory to median_array or new_bias_array.

-359

DpRT_make_bias

Failed to allocate memory to temp_array.

-360

DpRT_make_bias

Failed to allocate memory to temp_array[%d].

-361

DpRT_make_bias

Failed to read image data from FITS file number %d.

-363

DpRT_make_bias

Poor stats in new_bias.

-453

DpRT_make_flat

Could not read file (outer loop).

-454

DpRT_make_flat

Could not open directory %s (inner loop).

-462

DpRT_make_flat

Fewer than %d good flats from which to make master flat.

-452

DpRT_make_flat

Could not read existing flat frame.

-356

DpRT_make_flat

Failure to allocate memory for **fits_pointers.

-364

DpRT_make_bias

New bias is highly deviant from the old one: mean abs dev = %.

-454

DpRT_make_flat

Could not open directory.

-457

DpRT_make_flat

Failed to open FITS file.

-458

DpRT_make_flat

Failed to allocate memory to median_array or [mean|median]_flat_array.

-459

DpRT_make_flat

Failed to allocate memory to temp_array.

-460

DpRT_make_flat

Failed to allocate memory to temp_array[%d].

-461

DpRT_make_flat

Failed to read image data from FITS file number %d.

up to top of page

FITS HEADER WARNING STATE CODES

Code #

Process Name

Error String

32

DpRT_startup

CCD not at thermal set point.

52

DpRT_output

No filter calibration data for filter.

53

DpRT_output

No filter calibration data.

62

DpRT_fringe

No correlation between data and fringe. Defringing will use simple exposure time scaing.

64

DpRT_fringe

Correlation scaling failed. Simple exposure time scaing will be used.

72

DpRT_dark

No correlation between data and dark. Dedarking will use simple exposure time scaing.

220

DpRT_init

No '.' character in flatfield filename.

225

DpRT_init

Divide-by-zeroes were safely trapped.

226

DpRT_init

Failed to trim the flat field.

230

DpRT_init

Failed to open FLATLIB directory.

232

filter_params

Failed to open filter parameter lookup table.

233

filter_params

Failed to parse filter config line.

231

filter_params

Failed to allocate name memory for filter.

234

filter_params

Filter config line has a name > 49 chars.

235

filter_params

Filter scale factor or ZP read from file is not valid for filter.

236

filter_params

Failed to allocate filter memory.

251

DpRT_init

Failed to allocate memory for *photstar_data. Calibration data will not be available.

252

DpRT_init

Failed to open photstar parameter lookup table.

351

DpRT_make_bias

Could not open log file.

352

DpRT_make_bias

Could not read existing bias frame.

353

DpRT_make_bias

Could not read file.

355

DpRT_make_bias

Too many good bias frames were found.

362

DpRT_make_bias

New bias is highly discrepant from library bias.

366

DpRT_make_bias

More bias frames were discarded than kept.

367

DpRT_make_bias

Fewer than %d good bias frames.

363

DpRT_make_flat

Poor stats in new [mean|median] flat.

364

DpRT_make_flat

New flat is highly deviant from the old one.

365

DpRT_make_flat

No suitable flat frames found in directory. %d were read but discarded.

453

DpRT_make_flat

Could not read file (inner loop).

500 - 507

DpRT_init

NULL pointer in dprt_*_lib.

up to top of page

RINGO2 pipeline

As RINGO2 produces approximately 8 CCD frames per second (one per rotor position) it is not practical to archive or distribute the raw data from the instrument. A initial data reduction pipeline at the telescope therefore stacks all the frames at a given rotor position within a multrun and normalizes by the number of frames in the stack. A multrun of any length will therefore produce 8 files, named p_*_N_0.fits where N varies from 1 to 8. These mean stacked frames are then transferred back to the data archive at LJMU, where a dark frame is subtracted and a flat-field correction is applied. This data (P_*_N_1.fits) is then made available to users for download.

As described in the above section, two types of standard star observations (zero-polarization and polarized) are necessary to reduce RINGO2 data, which should have been obtained with the same mechanical mount position angle as the science data. The mount angle is specified in the FITS header by keyword ROTANGLE.

  • Carry out sky-subtracted photometry on the frames corresponding to your zero polarized standard(s). You will have eight measurements (one per rotor position) per target.
  • Carry out sky-subtracted photometry on the frames corresponding to your polarized standard(s) and science target(s)
  • For each rotor position, divide the counts from the polarized standards and science targets by those from your zero-polarized standard. This process (analogous to flat-fielding) removes the effect of the instrumentatal polarization from your data.
  • For each polarized standard and science target use the equations outlined in Clarke & Neumayer 2002 to calulcate q and u, and hence the degree of polarization (P) and position angle (PA) relative to the mount. Note that rotor positions correspond to the nomenclature used in that paper according to the scheme:
    ROTOR POSITION C & N
    1 A1
    2 B1
    3 C1
    4 D1
    5 A2
    6 B2
    7 C2
    8 D2
  • Plot a graph of the observed vs actual values for the polarized standards. Place your measured value for your science target on the graph to calculate its true polarization.

IMPORTANT : Dates of hardware changes

Because the relative physical alignment between the polizer, the rotor and the CCD array is critical to the data reduction it is important to use only the correctly matched flat fields and standard stars to analyze science data. You cannot use standards before and after any hardware reconfiguration interchangably. The following list gives dates on which hardware were changed. Make sure your standards and science frames are not separated by any of these dates.

  • 2010-11-10 Foreoptics changed to elliminate vignetting.
  • 2011-02-15 - 2001-03-20 Drive belt broken. No data available from this time period.
  • 2011-03-20 Returned to service following belt repair and filter re-seating.
  • 2012-04-28 Cass bearing moved. MOUNT=0 before and after this date are not equivalent.

FRODOSpec pipeline

Data taken by FRODOSpec is reduced by two sequentially invoked pipelines. The first pipeline, known as the L1, is a CCD processing pipeline which performs bias subtraction, overscan trimming and CCD flat fielding. The second pipeline, known as the L2, performs the processes unique to IFS reduction. The L2 became operational on 9th July 2010, with the second version released in May 2011. The description in the following sections conform to the second version of the pipeline. Full details of the pipeline may be found in the paper Barnsley, Smith and Steele.

L2 Pipeline

L2 data products are eight part multi-extension FITS files with each extension containing a snapshot of the data taken at key stages of the reduction process. The lowest tier of reduction product available to the user is the L1 image. The output data product format is shown below.

 
HDU Index EXTNAME Details
0 L1_IMAGE L1 image
1 RSS_NONSS Non sky subtracted row stacked spectra
2 CUBE_NONSS Non sky subtracted datacube
3 RSS_SS Sky subtracted row stacked spectra
4 CUBE_SS Sky subtracted datacube
5 SPEC_NONSS Non sky subtracted 1D spectrum
6 SPEC_SS Sky subtracted 1D spectrum
7 COLCUBE_NONSS Non sky subtracted collapsed datacube image
 

If the sky-subtraction process is unsuccessful, the corresponding HDUs (*_SS) will be blank. One-dimensional spectra (SPEC_*) are constructed using only the flux from the brightest five fibres.

Data that has been processed successfully by both pipelines will have a filename ending in "_2.fits".

Pipeline in Detail

The L2 pipeline reduction process uses three files, the "target" frame, the "arc" frame and the "continuum" frame. The pipeline can be visualised using the following schematic:

FRODOspec L2 reduction pipeline flow chart

  1. Fibre tramline map generation using the continuum frame and polynomial tracing
    frfind, frclean, frtrace

    The positions of the fibre profile peaks are determined and polynomial traces determined for each.


  2. Standard aperture flux extraction
    frextract

    Using the tramline mappings, the flux is extracted from each fibre in the target frame, continuum frame and arc frame using a 5 partial pixel aperture.


  3. Wavelength mapping on a fibre-to-fibre basis
    frarcfit

    Candidate lines are found in the arc RSS frame and their positions compared with a reference list containing known arc line pixel positions and corresponding wavelengths. Lines are then matched and pixel to wavelength calibrations determined for each spectrum.


  4. Fibre transmission correction
    frcorrectthroughput

    Using a continuum RSS frame, fibre-to-fibre throughput differences in the target RSS frame are normalised.


  5. Rebinning of data to a linear wavelength calibration
    frrebin

    In order to obtain a single wavelength solution applicable to all fibres, the flux from each spectrum in the target RSS frame is rebinned using linear interpolation to a linear wavelength scale with the same starting/ending wavelength positions and pixel scales.


  6. Sky subtraction (if applicable)
    frsubsky

    If the routine can successfully identify sky-only fibres, the sky flux contribution is removed for all spectra in the target RSS frame.


  7. Formatting of output data product
    frreformat

    A data product with the output format shown above is constructed.

Extraction using the Starlink software

Starlink users may extract the appropriate extension using the CONVERT:FITS2NDF command, appending the filename with the extension they wish to extract in brackets. The result is a single extension .SDF file.

For example, to extract the raw image from the multipart FITS file r_e_20100311_1_1_1_2.fits use:

 > convert 
> fits2ndf "r_e_20100311_1_1_1_2.fits[0]"

Or alternatively, you can access the required HDU by using the corresponding EXTNAME key:

 > fits2ndf "r_e_20100311_1_1_1_2.fits[L1_IMAGE]"

Extraction using DS9

DS9 users may view the available extensions by adding -multiframe on the terminal command line:

 > ds9 -multiframe r_e_20100311_1_1_1_2.fits

A frame extraction can be achieved by adding the -frame and -savetofits parameters, specifying the frame to be extracted (frame 1 corresponds to the primary HDU):

 > ds9 -multiframe r_e_20100311_1_1_1_2.fits -frame 1 -savefits output.fits -quit 

up to top of page

SkyCam pipeline

When the enclosure is open, data is taken automatically once per minute with a 10 second exposure time. All data is automatically dark subtracted and flat-fielded and a world coordinate system fitted. SkycamZ data remains proprietary, but SkyCamA and SkyCamT data is then immediately publicly released both as JPEG and FITS files here. In addition both browsable and searchable image archives are available.