Pipelines

Different pipelines exist for different instruments, based on their performance characteristics and how the data is to be archived.

• Quick Links
• RATCam pipeline
• RINGO2 pipeline
• FRODOSpec L2 pipeline
• SkyCam pipeline

RATCam pipeline

Basic instrumental reductions are applied to all RATCam images before the data are passed to users. This includes bias subtraction, trimming of the overscan regions and flat fielding. A library of the current calibration frames is maintained as part of the data archive and updated daily so that images are always reduced using the latest available flat-field image available at the time. Each of the operations are performed as described below.

Bias Subtraction

There is insufficient repeatable structure for bias frames to be useful. Bias subtraction is therefore based purely on analysis of the underscan region. Linear regression is used to determine a fit to the bias counts as a function of pixel row number and values deducted across the image according to this smooth function. Experience shows that RATCam does have a small ramp in the bias down each column and this first order fit is required. No attempt is made to remove any bias gradient in rows across the image.

Overscan Trimming

The overscan regions are trimmed off the image leaving a 2048x2048 (assuming on-chip binning was not used) pixel image.

Dark Substraction

This is not currently performed though the facility exists in the reduction pipeline. Experience has shown that when the camera is at normal operating temperature, dark current is not significant. If the camera temperature fluctuates for any reason, the dark current is sufficiently variable that a dark would need to be obtained with every frame and library dark frames are no use.

Flat Fielding

The appropriate master flat field is selected from the library to match the filter and binning configuration of the current exposure. In fact the library holds reciprocal flat-fields normalised to unity because of the computational efficiency of multiplying rather than dividing. The image data are therefore multiplied by the library flat.

Each twilight the instrument control software (ICS) attempts to update the oldest master flats in its library, by taking 3-5 raw sky flats for each filter/binning combination, giving preference to the most used. There usually isn't enough time to take sky flats for all combinations in one twilight session, so the next oldest in the list is attempted in following nights and so on. Usually the update is complete after 2-3 nights and the process starts again.

The master flat is derived from the median of each sky flat after each has been normalised to the common mean count level. New master flats, and their corresponding ratio image of new/old masters, are inspected for any corruption, inclusion of stars, etc.

Bad Pixel Mask

No cosmic ray rejection or bad pixel mask is applied since it is important for users performing accurate photometry to know exactly what masking has been applied. However, bad pixel masks have been generated and kindly made available by the Angstrom project, a gravitational lensing programme underway on the LT and RoboNet:

• unbinned bad pixel mask (13KB gzipped FITS, unzips to 8MB)
• 2x2 binned bad pixel mask (5KB gzipped FITS, unzips to 2MB)

Vignetting

The filter wheel slightly vignettes the optical beam to a different extent for each filter. In the extreme corners of the worst affected bands (i' and g') the flux is reduced by up to 15% compared to the unobstructed beam. In the other filters, obscuration is about 5% in the very corner of the observed field, falling to negligible values between 10 - 20 arcsec from the field edge.

The vignetting generally flat fields out very well and is rarely obvious in the reduced data, sometimes leaving distortions only in the range 3 - 5 arcsec from the field corner. Even where the data is well flat fielded though, noise characteristics of photon counting statistics could be affected in these regions of the frame.

Fringe Frames

We currently do not perform any automated defringing on CCD data before it is loaded into the archive. In order to help you defringe your own data, linked below are prepared master fringe frames created by stacking multiple deep integrations of blank fields. The master fringe frames are updated infrequently because the fringes on the CCD have been found to vary only on timescales of months. If you need access to the individual integrations which go into building these master frames, they are publicly available from the data archive. Simply select RATFringe from the Proposal ID drop-down list. You can therefore extract the most recent fringe frame from the archive at any time.

Legacy Fringe Frames. Prior to July 2006 SDSS-I, binned 2x2 (4MB gzipped FITS, unzips to 8MB) SDSS-Z, binned 2x2 (2.3MB gzipped FITS, unzips to 4MB) Legacy Fringe Frames. Created July/August 2006 SDSS-I, binned 1x1, stack of 46 x 300 second exposures, 13MB gzipped fits SDSS-Z, binned 2x2, stack of 27 x 300 second exposures, 3.1MB gzipped fits SDSS-I, binned 2x2, stack of 27 x 300 second exposures, 3.1MB gzipped fits SDSS-R, binned 2x2, stack of 27 x 300 second exposures, 2.6MB gzipped fits, fringes barely visible. SDSS-Z, binned 1x1, stack of 27 x 300 second exposures, 11MB gzipped fits Legacy Fringe Frames. Created September 2007 from data obtained Jan - Aug 2007. SDSS-Z, binned 1x1, stack of 36 x 300 second exposures, 15MB gzipped fits SDSS-Z, binned 2x2, stack of 33 x 300 second exposures, 4MB gzipped fits SDSS-I, binned 1x1, stack of 18 x 300 second exposures, 15MB gzipped fits SDSS-I, binned 2x2, stack of 34 x 300 second exposures, 4MB gzipped fits Legacy Fringe Frames. Created December 2007 from data obtained Sep - Dec 2007. These are best suited to any data obtained in 2007 after the September mirror recoating and servicing. I.e., the majority of semester 07B. SDSS-I, binned 1x1, stack of 216 x 300 second exposures, 15MB gzipped fits SDSS-I, binned 2x2, stack of 18 x 300 second exposures, 4MB gzipped fits SDSS-Z, binned 1x1, stack of 18 x 300 second exposures, 15MB gzipped fits SDSS-Z, binned 2x2, stack of 18 x 300 second exposures, 4MB gzipped fits Legacy Fringe Frames. Created Feb 2008 from data obtained Jan - Feb 2008. SDSS-I, binned 1x1, stack of 90 x 300 second exposures, 15MB gzipped fits Created Jan 2009 from data obtained throughout 2008. SDSS-I, binned 1x1, 90 x 300sec exposures from 08B, 15MB gzipped fits SDSS-I, binned 2x2, Rebinned version of above 1x1 data, 4MB gzipped fits SDSS-Z, binned 1x1, 72 x 300sec exposures from 08A and 08B, 15MB gzipped fits SDSS-Z, binned 2x2, Rebinned version of above 1x1 data, 4MB gzipped fits Created Jan 2010 from data obtained throughout 2009. SDSS-I, binned 1x1, 112 x 300sec exposures from 09A-09B, 15MB gzipped fits SDSS-I, binned 2x2, 93 x 300sec exposures from 09A-09B, 4MB gzipped fits SDSS-Z, binned 1x1, 122 x 300sec exposures from 09A-09B, 15MB gzipped fits SDSS-Z, binned 2x2, 36 x 300sec exposures from 09A-09B, 4MB gzipped fits

FITS Header Error Codes

Error codes are written into the FITS headers for specific error flags brought up during processing. These error flags are stored in the header keywords L1STATOV, L1STATZE, L1STATTR, L1STATFL and L1STATDA and follow this convention:

• negative values are failure states (special case: -1 = operation not attempted)
• positive values are warning states (special case: +1 = no errors or warnings)

Tables of failure and warning states are given below - but please note these are not exhaustive lists.

FITS HEADER FAILURE STATE CODES
Code #	Process Name	Error String
-12	DpRT_startup	Input is not a valid LT filename.
-14	DpRT_startup	According to filename flags, input has already been processed.
-15	DpRT_startup	Run_mode is not valid (%d). See dprt.h for valid values.
-18	DpRT_startup	Invalid exposure type flag.
-35	DpRT_zero	Cannot find zero (bias) frame file.
-37	DpRT_dark	Cannot find dark frame file.
-38	DpRT_flat	Mean counts in flat <= 0. Processing abandoned.
-40	DpRT_flat	Cannot find flatfield file.
-60	DpRT_fringe	Cannot find fringe file.
-61	DpRT_fringe	Fringe and data frame sizes do not match.
-63	DpRT_fringe	Correlation scaling failed. No defringing performed.
-73	DpRT_dark	Correlation scaling failed. No dedarking performed.
-261		Error opening FITS.
-263		Error getting header keywords.
-264		Non-square binning.
-268		Error reading image array.
-354	DpRT_make_bias	Could not open working directory.
-356	DpRT_make_bias	Failure to allocate memory for **fits_pointers.
-357	DpRT_make_bias	Failed to open FITS file.
-358	DpRT_make_bias	Failed to allocate memory to median_array or new_bias_array.
-359	DpRT_make_bias	Failed to allocate memory to temp_array.
-360	DpRT_make_bias	Failed to allocate memory to temp_array[%d].
-361	DpRT_make_bias	Failed to read image data from FITS file number %d.
-363	DpRT_make_bias	Poor stats in new_bias.
-453	DpRT_make_flat	Could not read file (outer loop).
-454	DpRT_make_flat	Could not open directory %s (inner loop).
-462	DpRT_make_flat	Fewer than %d good flats from which to make master flat.
-452	DpRT_make_flat	Could not read existing flat frame.
-356	DpRT_make_flat	Failure to allocate memory for **fits_pointers.
-364	DpRT_make_bias	New bias is highly deviant from the old one: mean abs dev = %.
-454	DpRT_make_flat	Could not open directory.
-457	DpRT_make_flat	Failed to open FITS file.
-458	DpRT_make_flat	Failed to allocate memory to median_array or [mean|median]_flat_array.
-459	DpRT_make_flat	Failed to allocate memory to temp_array.
-460	DpRT_make_flat	Failed to allocate memory to temp_array[%d].
-461	DpRT_make_flat	Failed to read image data from FITS file number %d.

FITS HEADER WARNING STATE CODES
Code #	Process Name	Error String
32	DpRT_startup	CCD not at thermal set point.
52	DpRT_output	No filter calibration data for filter.
53	DpRT_output	No filter calibration data.
62	DpRT_fringe	No correlation between data and fringe. Defringing will use simple exposure time scaing.
64	DpRT_fringe	Correlation scaling failed. Simple exposure time scaing will be used.
72	DpRT_dark	No correlation between data and dark. Dedarking will use simple exposure time scaing.
220	DpRT_init	No '.' character in flatfield filename.
225	DpRT_init	Divide-by-zeroes were safely trapped.
226	DpRT_init	Failed to trim the flat field.
230	DpRT_init	Failed to open FLATLIB directory.
232	filter_params	Failed to open filter parameter lookup table.
233	filter_params	Failed to parse filter config line.
231	filter_params	Failed to allocate name memory for filter.
234	filter_params	Filter config line has a name > 49 chars.
235	filter_params	Filter scale factor or ZP read from file is not valid for filter.
236	filter_params	Failed to allocate filter memory.
251	DpRT_init	Failed to allocate memory for *photstar_data. Calibration data will not be available.
252	DpRT_init	Failed to open photstar parameter lookup table.
351	DpRT_make_bias	Could not open log file.
352	DpRT_make_bias	Could not read existing bias frame.
353	DpRT_make_bias	Could not read file.
355	DpRT_make_bias	Too many good bias frames were found.
362	DpRT_make_bias	New bias is highly discrepant from library bias.
366	DpRT_make_bias	More bias frames were discarded than kept.
367	DpRT_make_bias	Fewer than %d good bias frames.
363	DpRT_make_flat	Poor stats in new [mean|median] flat.
364	DpRT_make_flat	New flat is highly deviant from the old one.
365	DpRT_make_flat	No suitable flat frames found in directory. %d were read but discarded.
453	DpRT_make_flat	Could not read file (inner loop).
500 - 507	DpRT_init	NULL pointer in dprt_*_lib.

RINGO2 pipeline

As RINGO2 produces approximately 8 CCD frames per second (one per rotor position) it is not practical to archive or distribute the raw data from the instrument. A initial data reduction pipeline at the telescope therefore stacks all the frames at a given rotor position within a multrun and normalizes by the number of frames in the stack. A multrun of any length will therefore produce 8 files, named p_*_N_0.fits where N varies from 1 to 8. These mean stacked frames are then transferred back to the data archive at LJMU, where a dark frame is subtracted and a flat-field correction is applied. This data (P_*_N_1.fits) is then made available to users for download.

As described in the above section, two types of standard star observations (zero-polarization and polarized) are necessary to reduce RINGO2 data, which should have been obtained with the same mechanical mount position angle as the science data. The mount angle is specified in the FITS header by keyword ROTANGLE.

• Carry out sky-subtracted photometry on the frames corresponding to your zero polarized standard(s). You will have eight measurements (one per rotor position) per target.
• Carry out sky-subtracted photometry on the frames corresponding to your polarized standard(s) and science target(s)
• For each rotor position, divide the counts from the polarized standards and science targets by those from your zero-polarized standard. This process (analogous to flat-fielding) removes the effect of the instrumentatal polarization from your data.
• For each polarized standard and science target use the equations outlined in Clarke & Neumayer 2002 to calulcate q and u, and hence the degree of polarization (P) and position angle (PA) relative to the mount. Note that rotor positions correspond to the nomenclature used in that paper according to the scheme:

  ROTOR POSITION	C & N
  1	A1
  2	B1
  3	C1
  4	D1
  5	A2
  6	B2
  7	C2
  8	D2

• Plot a graph of the observed vs actual values for the polarized standards. Place your measured value for your science target on the graph to calculate its true polarization.

IMPORTANT : Dates of hardware changes

Because the relative physical alignment between the polizer, the rotor and the CCD array is critical to the data reduction it is important to use only the correctly matched flat fields and standard stars to analyze science data. You cannot use standards before and after any hardware reconfiguration interchangably. The following list gives dates on which hardware were changed. Make sure your standards and science frames are not separated by any of these dates.

• 2010-11-10 Foreoptics changed to elliminate vignetting.
• 2011-02-15 - 2001-03-20 Drive belt broken. No data available from this time period.
• 2011-03-20 Returned to service following belt repair and filter re-seating.
• 2012-04-28 Cass bearing moved. MOUNT=0 before and after this date are not equivalent.

FRODOSpec pipeline

Data taken by FRODOSpec is reduced by two sequentially invoked pipelines. The first pipeline, known as the L1, is a CCD processing pipeline which performs bias subtraction, overscan trimming and CCD flat fielding. The second pipeline, known as the L2, performs the processes unique to IFS reduction. The L2 became operational on 9th July 2010, with the second version released in May 2011. The description in the following sections conform to the second version of the pipeline. Full details of the pipeline may be found in the paper Barnsley, Smith and Steele.

L2 Pipeline

L2 data products are eight part multi-extension FITS files with each extension containing a snapshot of the data taken at key stages of the reduction process. The lowest tier of reduction product available to the user is the L1 image. The output data product format is shown below.

HDU Index	EXTNAME	Details
0	L1_IMAGE	L1 image
1	RSS_NONSS	Non sky subtracted row stacked spectra
2	CUBE_NONSS	Non sky subtracted datacube
3	RSS_SS	Sky subtracted row stacked spectra
4	CUBE_SS	Sky subtracted datacube
5	SPEC_NONSS	Non sky subtracted 1D spectrum
6	SPEC_SS	Sky subtracted 1D spectrum
7	COLCUBE_NONSS	Non sky subtracted collapsed datacube image

If the sky-subtraction process is unsuccessful, the corresponding HDUs (*_SS) will be blank. One-dimensional spectra (SPEC_*) are constructed using only the flux from the brightest five fibres.

Data that has been processed successfully by both pipelines will have a filename ending in "_2.fits".

Pipeline in Detail

The L2 pipeline reduction process uses three files, the "target" frame, the "arc" frame and the "continuum" frame. The pipeline can be visualised using the following schematic:

1. Fibre tramline map generation using the continuum frame and polynomial tracing
   frfind, frclean, frtrace

   ---

   The positions of the fibre profile peaks are determined and polynomial traces determined for each.
   
   
   
2. Standard aperture flux extraction
   frextract

   ---

   Using the tramline mappings, the flux is extracted from each fibre in the target frame, continuum frame and arc frame using a 5 partial pixel aperture.
   
   
   
3. Wavelength mapping on a fibre-to-fibre basis
   frarcfit

   ---

   Candidate lines are found in the arc RSS frame and their positions compared with a reference list containing known arc line pixel positions and corresponding wavelengths. Lines are then matched and pixel to wavelength calibrations determined for each spectrum.
   
   
   
4. Fibre transmission correction
   frcorrectthroughput

   ---

   Using a continuum RSS frame, fibre-to-fibre throughput differences in the target RSS frame are normalised.
   
   
   
5. Rebinning of data to a linear wavelength calibration
   frrebin

   ---

   In order to obtain a single wavelength solution applicable to all fibres, the flux from each spectrum in the target RSS frame is rebinned using linear interpolation to a linear wavelength scale with the same starting/ending wavelength positions and pixel scales.
   
   
   
6. Sky subtraction (if applicable)
   frsubsky

   ---

   If the routine can successfully identify sky-only fibres, the sky flux contribution is removed for all spectra in the target RSS frame.
   
   
   
7. Formatting of output data product
   frreformat

   ---

   A data product with the output format shown above is constructed.
   
   

Extraction using the Starlink software

Starlink users may extract the appropriate extension using the CONVERT:FITS2NDF command, appending the filename with the extension they wish to extract in brackets. The result is a single extension .SDF file.

For example, to extract the raw image from the multipart FITS file r_e_20100311_1_1_1_2.fits use:

 > convert 
 > fits2ndf "r_e_20100311_1_1_1_2.fits[0]"

Or alternatively, you can access the required HDU by using the corresponding EXTNAME key:

 > fits2ndf "r_e_20100311_1_1_1_2.fits[L1_IMAGE]"

Extraction using DS9

DS9 users may view the available extensions by adding -multiframe on the terminal command line:

 > ds9 -multiframe r_e_20100311_1_1_1_2.fits

A frame extraction can be achieved by adding the -frame and -savetofits parameters, specifying the frame to be extracted (frame 1 corresponds to the primary HDU):

 > ds9 -multiframe r_e_20100311_1_1_1_2.fits -frame 1 -savefits output.fits -quit 

SkyCam pipeline

When the enclosure is open, data is taken automatically once per minute with a 10 second exposure time. All data is automatically dark subtracted and flat-fielded and a world coordinate system fitted. SkycamZ data remains proprietary, but SkyCamA and SkyCamT data is then immediately publicly released both as JPEG and FITS files here. In addition both browsable and searchable image archives are available.