Data Products
Direct Links
- Daily Procedures
- Where Data are Stored Online
- Observations Log
- Downloading your data
- Calibrators and Standards
Daily Procedures
Raw data from all instruments are copied back daily to the ARI in Liverpool. On arrival here, both manual and automated data quality checks are run to allow us to monitor telescope performance. Where a reduction pipeline is available, the data are reduced and loaded into the searchable Data Archive. Data from all instruments (reduced if there is an automated pipeline and raw if there is not) are normally available for download by midday. An email is also sent to the PI notifying them of the new data. If you would also like an alert email sent to a Co-I please contact LT Support.
If any data is deemed not to have passed quality checking then the telescope time is refunded to the proposal (depending on time constraints set by the PI) and the PI notified in case they wish to reschedule the observation using the Phase2 UI.
Where Data are Stored Online
There are three web-based interfaces to allow PIs and CoIs access to the their data. The main distinguishing feature between these interfaces is the time-scale on which the data are made and then maintained available. All these data sources are password protected for as long as the data are still within their proprietary period, after which they become publicly available.
Separate from the three main user interfaces discussed here, a small amount of data which for one reason or other are not available in the main searchable data archive may be found in Unloaded Data. In most cases these are early comissioning data for particuilar instruments obtained when a reduction pipeline was not yet available or do not meet the quality control requirements for ingestion to the main archive. Most of the data included here are unreduced raw files direct from the instrument. Also included here are some very early (2004) LT data obtained before the robotic systems were comissioned. Most LT users will never want access to any of these files. For the majority of users, the most commonly used interface is Recent Data.
| Available | Until | Comments | |
|---|---|---|---|
| Quicklook | Real time. 5 minutes after observation | Three days | Monitor in real time what the telescope is doing for you. Also provides access to time critical data such as GRBs. |
| Archive | Next working day | Unlimited | The primary data repository. |
| Recent | Next working day | One month | A single place you can always look to see what data have been obtained for you in the past few nights |
Quicklook
- Available immediately during the night
- May only be partially reduced, will not always use the most up to date calibration files and may use reduction algorithms optimised for speed rather than accuracy
- FITS file names end in "_9.fits"
- Password protected
Searchable Archive
- Long term storage of all reduced LT data. Currently no time limit on storage
- Only contains those data which have been through the automated reduction pipelines. For instruments where no pipeline is currently in operation, alternative temporary measures are in place to provide the data to users: See Recent Data below
- Contains all reduced LT data taken since Sept 2004. If you need to recover data obtained for you before that date, please contact us.
- Reduced FITS file names end in "_1.fits" if they have undergone only basic instrumental reductions (bias, flat etc) or "_2.fits" if they include derived results such as object catalogues, extracted 1D spectra etc.
- Password protected
Recent Data
- Nightly gzipped tarballs, one for each observing proposal
- Exactly the same data as are loaded into the main data archive above, but are simply held here for a short while for more convenient download
- In the case of instruments for which we do not yet have a working reduction pipeline, the raw data will be held online here indefinately or until they can be reduced and loaded to the main data archive
- Reduced FITS file names end in "_1.fits" of "_2.fits" whilst unreduced FITS file names end in "_0.fits"
- Password protected
- Data are stored in subdirectories named by the local civil date of the start of that night. A full night's data are stored together in each subdirectory, not creating a new directory at midnight.
- For each night you will find
- All FITS files for the observing programme on that date
- A gzipped tar file of all the data bundled together to ease download
- An observing log
Observations Log
Observing logs are available on a per proposal basis so you can obtain a full list of all observations taken on a partiular night for a single proposal. These may be found in Recent Data. For example nightly logs for the RATCam photometric standards are accessible at http://150.204.240.8/data/webfiles/new/RATStand/.
Observing Log Error Codes
If there have been any problems during the image's passage through the Data Pipeline, error codes are added to the observing log, and more detailed codes are also written to custom FITS headers in the image. These are to alert your attention to more specific flags in the FITS header. The observing log codes are cumulative and are found in the log at the end of the line for each image. So for example, if a problem is encountered in both overscan subtraction and bias frame subtraction, the flag would be set to -6. (i.e., -2 + -4 )
| OBSERVING LOG CODES FOR RATCam PIPELINE | |
|---|---|
| Code # | Error String |
| -2 | Overscan subtraction |
| -4 | Bias frame subtraction |
| -8 | Trimming off the overscan |
| -16 | Flat fielding |
| -32 | Dark frame subtraction |
| -64 | Fringe subtraction |
Downloading Your Data
Data are available for immediate download over HTTP. Consult the Where Data are Stored section above to determine the access method best suited to you. Recent Data uses a static directory structure so that you always know where to look for your data. The Data Archive will automatically create hypertext links for you to all data that match your search criteria.
Accessing files / Passwords
In order to access FITS files, you will need to use the password which was provided to the PI in the notification email when the data were first loaded into the archive. The password is specific to the proposal ID (not the user ID) and any future or past data obtained for the proposal should have the same password. You can thus share your proposal password with different CoIs for each project. Data remain proprietary to the PI until one year after the expiry of the proposal. If a proposal has long term status, this may therefore be several years after the exposure was obtained. For a typical single semester proposal, it will be 12 - 18 months.
Calibrators and Standards
All standard calibration data (Photometric, polarimetric standards etc) taken with the LT are public and available for use by all observers. As with all other data, once reduced they can be accessed directly from the Data Archive or in the short term, the Recent Data pages. Each set of calibrators has its own specific proposal ID. The following table lists which proposal ID to interogate for each calibration standard. See the instrument specific pages for details of what standards are observed, zero points, colour transforations etc.
| PROPOSAL IDs FOR INSTRUMENTAL CALIBRATION DATA | |
|---|---|
| Proposal ID | Instrument and Data Type |
| RATStand (Formerly "RATCAM_STANDARDS") |
RATCam. Visible wavelength photometric standards |
| RATFringe (Formerly "RATCAM_FRINGE") |
RATCam. Deep i' and z' band blank fields for creation of fringe frames |
| RINGOStand (Formerly "RINGO_STAND") |
RINGO. Polarimetric standards. Both polarised and zero polarisation stars. Note: RINGO was decommissioned in 2009 and replaced with RINGO2. |
| SupIRStand (Formerly "SUPIRCAM_STANDARDS") |
SupIRCam. Near IR wavelength photometric standards. Note: SupIRCam was decommissioned in 2010. |
