The Imaging Pipeline

This page describes the SDSS imaging pipeline, which processes the raw telescope imaging to produce calibrated FITS files and catalogs of imaging parameters. Each step in the pipeline is outlined below.

More detail is available in the Algorithms section of this site. The Early Data Release (EDR) paper (Stoughton et al. 2002) is the fundamental resource for understanding the processing and data products from the SDSS, describing the pipelines and contents of generated data products. Successive data release papers describe changes between data releases – see SDSS Data Release Publications for a complete list, with links. The Data Release 9 paper (Ahn et al. 2012) describes some changes to astrometry algorithms that occurred with DR9 and later. The Data Release 13 paper (Albareti et al. in press) describes changes to the definition of primary and the photometric calibration.

In addition to these data release papers, the SDSS technical summary provides more general information on the survey. The SDSS Project Book, written for a funding proposal early in the survey, is the most exhaustive description of the survey but was last updated in 1997. The SAS datamodel has a table of the most commonly useful files in the SAS. Truly gory details can be found on Robert Lupton’s page of photo documents, including the draft photometric reduction paper as well as a treasure trove of photometric data reduction wisdom.

The remainder of this page starts with a brief overview of imaging data processing, followed by sections that describe the steps in data processing in detail. In addition to the more detailed description, each detail section provides references to papers that give additional details, and a table of the files associated with that step of the pipeline that can be found in the SAS. These tables include links to descriptions of the formats of those files and templates that can be used to generate SAS URLs for those files. The templates are in C printf() format, and can be used in C, bash, Python, and many other languages to automatically generate URLs.

Most of the catalog data (but not the images themselves) have been loaded into the Catalog Archive Server (CAS) database, which is accessible through SkyServer and CasJobs. Users are often better off obtaining SDSS data through a carefully constructed CAS query than they are downloading the data files from the SAS. Simple queries can be used to select just the objects and parameters of interest, while more complex queries can be used to do complex calculations on many objects, thereby avoiding the need to download the data on them at all.

Overview: Steps of the Imaging Pipeline

Imaging Observing
We used three instruments when collecting imaging data: the imaging camera mounted on the primary 2.5m SDSS telescope, which collected the imaging data themselves; the 0.5m photometric telescope, which collected images of photometric standard stars and reference fields; and a 10 micron all sky scanner
Imaging Data Reduction
The SDSS data processing factory used a collection of pipelines to process and calibrate the data from the imaging camera, ultimately producing a variety of data products including images with instrumental signatures removed, a photometric solution for the night, and a catalog of objects found in the data.
Resolution of Multiple Detections
The resolve step declares one observation on an object the primary observation, and others secondary, thereby avoiding unintentional duplication of objects. This step ultimately produces an astrometrically and photometrically calibrated catalog of objects found in the data from the imaging camera.
Recalibration
Improved photometric calibrations became possible at the end of the survey; we used a separate pipeline to refine the photometric calibrations.
Final Photometric Catalog
The results of the resolve and calibration stages are combined with the uncalibrated object catalogs to produce final catalogs.

Important Notes

  • In all of the tables below, add ‘https://data.sdss.org/sas/dr14/’ to all ‘URL format’ values to get the full URL.
  • Data reduction could occur multiple times for both images and spectra. Each time we repeated a data reduction, we labeled the output from that reduction process with a distinct rerun number.
  • In the case of imaging rerun numbers, different decades in the rerun number designate significant differences in the data reduction software. For example, rerun 1 and rerun 2 of a given run would have been processed by identical (or at least very similar) versions of the data reduction software, while rerun 40 would have been reduced with a significantly different version of the pipeline. However, the only relevant reruns for DR8 and later are 157 and 301, with the bulk of the data in rerun 301.

Imaging Observing

Imager

The imager collected survey data from the imager in drift scan (also called TDI) mode. We use the term ‘run’ to designate a single scan along a great circle. The data acquisition system divided the data from each CCD into frames, stored as FITS files; the observatory sent data to be processed as a collection of FITS files, each corresponding to an arc along the great circle 1361 rows (539 arcseconds) long from a single CCD. In addition to the images themselves, the data acquisition system also generated a variety of metadata and other engineering files, which included pixel statistics and a catalog of bright stars for each field.

SAS files generated in imager data collection
File Type in/out Description URL format format parameters
idReport out records runs collected on a night eboss/photo/redux/%d/%d/logs/idReport-%05d.par rerun, run, mjd
idR out raw imaging frames eboss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z run, camcol, run, filter, camcol, field
idGang out assorted data gathered by the data acquisition system eboss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit run, run, r(ow) or c(ol), row number or column number, frame
idFrameLog out camera position information for each frame eboss/photo/redux/%d/%d/logs/idFrameLog-%06d-%d.par rerun, run, run, crate
opCamera out imaging camera geometry eboss/photo/redux/%d/%d/logs/opCamera-%05d.par rerun, run, mjd (of generation)
opConfig out CCD readout configurations eboss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
opECalib out CCD electronic calibrations eboss/photo/redux/%d/%d/logs/opECalib-%05d.par rerun, run, mjd (of generation)

Infrared All-Sky Camera

Clouds can be clearly seen at night in a 10 micron all-sky camera. We used two such cameras at different times in the survey to monitor the sky for cloud (and therefore non-photometric conditions).

SAS files generated by the infrared all sky camera or scanner
File Type in/out Description URL format format parameters
cR out A single image from the infrared camera apo/ircam/%05d/cR%06d.fit mjd, UTC time (HHMMSS)
irsc out A log of statistics on ircam frames apo/ircam/%05d/irsc.log mjd

Other metadata

The observatory produces an assortment of engineering and other metadata.

Other SAS files generated in observing
File Type in/out Description URL format format parameters
night log out A prose account of the night apo/astrolog/%05d/manualLog-full.txt mjd
idWeather out A log of weather data eboss/photo/redux/%d/%d/logs/idWeather-%05d.par rerun, run, mjd

Imaging Data Reduction

Serial Stamp Collecting Pipeline (SSC)

The SSC repackages some of the data produced by the data acquisition system and produces postage stamps of bright reference stars.

SAS files read by the SSC pipeline
File Type in/out Description URL format format parameters
scPlan in the SSC processing plan eboss/photo/redux/%d/%d/ssc/scPlan.par run, rerun
idReport in records runs collected on a night eboss/photo/redux/%d/%d/logs/idReport-%05d.par rerun, run, mjd
opCamera in imaging camera geometry eboss/photo/redux/%d/%d/logs/opCamera-%05d.par rerun, run, mjd (of generation)
opConfig in CCD readout configurations eboss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
opECalib in CCD electronic calibrations eboss/photo/redux/%d/%d/logs/opECalib-%05d.par rerun, run, mjd (of generation)
opBC in CCD bad pixel file eboss/photo/redux/%d/%d/log/opBC-%d.par rerun, run, mjdbp (see fpPlan)
idR in raw imaging frames eboss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z run, camcol, run, filter, camcol, field
idGang in assorted data gathered by the data acquisition system eboss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit run, run, r(ow) or c(ol), row number or column number, frame
koCat in the catalog of known bright stars eboss/photo/redux/%d/%d/ssc/koCat-%06d.fit run, rerun, run

Postage Stamp Pipeline (PSP)

References: Stoughton et al. (2002), section 4.3.

The PSP measures the bias and global sky level, and fits a model for the point spread function for each field.

SAS files used or generated by the PSP pipeline
File Type in/out Description URL format format parameters
psParam in PSP tunable parameters eboss/photo/redux/%d/%d/photo/psParam.par run, rerun
psPlan in the PSP processing plan eboss/photo/redux/%d/%d/photo/psPlan.par run, rerun
opCamera in imaging camera geometry eboss/photo/redux/%d/%d/logs/opCamera-%05d.par rerun, run, mjd (of generation)
opConfig in CCD readout configurations eboss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
opECalib in CCD electronic calibrations eboss/photo/redux/%d/%d/logs/opECalib-%05d.par rerun, run, mjd (of generation)
opBC in CCD bad pixel file eboss/photo/redux/%d/%d/log/opBC-%d.par rerun, run, mjdbp (see fpPlan)
idB in bias vector eboss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit rerun, run, calibration run, filter, camcol
psField out the initial photometric calibration and point spread function fit by field (see the extractingPSF images page) eboss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field

Astrometric Pipeline (astrom)

References: Pier et al. (2003).

The astrometric pipeline calculates the astrometric solution. The astrometry page provides more information on the algorithms used and the proper interpretation of the output.

SAS files used or generated by the astrom pipeline
File Type in/out Description URL format format parameters
asParam in astrom tunable parameters eboss/photo/redux/%d/%d/astrom/asParam.par rerun, run
asPlan in the astrom processing plan eboss/photo/redux/%d/%d/astrom/asPlan.par rerun, run
opCamera in imaging camera geometry eboss/photo/redux/%d/%d/logs/opCamera-%05d.par rerun, run, mjd (of generation)
opConfig in CCD readout configurations eboss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
idReport in records runs collected on a night eboss/photo/redux/%d/%d/logs/idReport-%05d.par rerun, run, mjd
idWeather in A log of weather data eboss/photo/redux/%d/%d/logs/idWeather-%05d.par rerun, run, mjd
asTrans out transformation coefficients from row, column to great circle coordinates eboss/photo/redux/%d/%d/astrom/asTrans-%06d.fit rerun, run, run

Frames Pipeline (frames)

References: Stoughton et al. (2002), section 4.4.

The frames pipeline applies flat field and bias corrections to each frame, and interpolates values for pixels in bad columns and bleed trails and those corrupted by cosmic rays. In previous data releases, this pipeline would produce a ‘corrected frame’ or fpC file. These files are no longer produced. Instead, a frame file is produced by a different method, though it contains substantially the same information. The frames pipeline described here also detects bright objects and estimates the varying sky background, detects fainter objects, and measures a variety of properties for each object. Several algorithms pages, linked below, describe details of the processing performed by the frames pipeline.

SAS files used or generated by the frames pipeline
File Type in/out Description URL format format parameters
fpParam in frames tunable parameters eboss/photo/redux/%d/%d/photo/fpParam.par rerun, run
fpPlan in frames processing plan eboss/photo/redux/%d/%d/photo/fpPlan.par rerun, run
idReport in records runs collected on a night eboss/photo/redux/%d/%d/logs/idReport-%05d.par rerun, run, mjd
koCat in the catalog of known bright stars eboss/photo/redux/%d/%d/ssc/koCat-%06d.fit run, rerun, run
opConfig in CCD readout configurations eboss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
opECalib in CCD electronic calibrations eboss/photo/redux/%d/%d/logs/opECalib-%05d.par rerun, run, mjd (of generation)
opBC in CCD bad pixel file eboss/photo/redux/%d/%d/log/opBC-%d.par rerun, run, mjdbp (see fpPlan)
idB in bias vector eboss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit rerun, run, calibration run, filter, camcol
psField in the initial photometric calibration and point spread function fit by field (see the extractingPSF images page) eboss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field
asTrans in tranformation coefficients from row, column to great circle coordinates eboss/photo/redux/%d/%d/astrom/asTrans-%06d.fit rerun, run, run
fpFieldStat out field statistics eboss/photo/redux/%d/%d/objcs/%d/fpFieldStat-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field
fpObjc out the (uncalibrated) object catalog eboss/photo/redux/%d/%d/objcs/%d/fpObjc-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field
fpBIN out a 4×4 binned version of the corrected image after masking of objects and subtraction of sky, an image of the estimate of the sky, an image of the uncertainty in the sky estimate, and a table of bright star wings eboss/photo/redux/%d/%d/objcs/%d/fpBIN-%06d-%c%d-%04d.fit.gz rerun, run, camcol, run, filter, camcol, field
fpM out the frame masks (read using readAtlasImages) eboss/photo/redux/%d/%d/objcs/%d/fpM-%06d-%c%d-%04d.fit.gz rerun, run, camcol, run, filter, camcol, field
fpAtlas out the atlas images for all objects detected (read using readAtlasImages) eboss/photo/redux/%d/%d/objcs/%d/fpAtlas-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field

Resolve Multiple Detections

A given area on the sky may be observed by multiple runs. The final calibrated catalogs contain a field declaring whether the specific observation of the object is ‘primary’, and the canonical measurement of the object, or ‘secondary’, a duplicate. For EDR data through DR7 data, the EDR paper, section 4.7 details on how objects are resolved. From DR8 onwards, a new, more general resolve process accounts for the full set of SDSS runs. The DR8 through DR12 data all use exactly the same resolve; DR13 uses the same overall procedure, but has implemented new decisions about the ‘best’ field to use in each location on the sky, with no further changes as of DR14.

SAS files used or generated by the resolve stage
File Type in/out Description URL format format parameters
reObjGlobal out Global resolve status for every object in a field eboss/resolve/%s/%d/%d/resolve/%d/reObjGlobal-%06d-%d-%04d.fit resolve rerun, rerun, run, camcol, run, camcol, field
thingList out The full list of catalog entries for the resolved survey eboss/resolve/%s/thingList.fits resolve rerun
thingIndex out The full list of unique primary objects, gives position of an object in the thingList eboss/resolve/%s/thingIndex.fits resolve rerun
window_flist out The full list of fields used to determine the window function eboss/resolve/%s/window_flist.fits resolve rerun
window_unified out The full list of polygons determining the primary area of the window function eboss/resolve/%s/window_unified.fits resolve rerun
window_blist out The full list of balkans determining the primary area of the window function eboss/resolve/%s/window_blist.fits resolve rerun
window_bcaps out The full list of caps determining the primary area of the window function eboss/resolve/%s/window_bcaps.fits resolve rerun
window_bindx out Matches balkans to fields in the window function, sorted by balkans eboss/resolve/%s/window_bindx.fits resolve rerun
window_findx out Matches balkans to fields in the window function, sorted by sorted by fields eboss/resolve/%s/window_findx.fits resolve rerun

Photometric Recalibration (Ubercal and Hypercal)

References: Padmanabhan et al. (2008), Finkbeiner et al. (2015).

From DR7 through DR12, the “ubercal” recalibration recalibrated the survey by simultaneously solving for the calibration parameters and relative stellar fluxes using overlapping observations within SDSS. The algorithm decouples the problem of relative calibrations from that of absolute calibrations; the absolute calibration is reduced to determining a few numbers for the entire survey.

As of DR13, all of the g, r, i, z images and catalogs were recalibrated to Pan-STARRS using a procedure known as hypercal (‘hyper’ is used to denote that it uses information outside SDSS data. For the u band, the flat field was updated but the zero point was not. The calibration files below retain the same format as for previous data releases. See the photometric calibration algorithms page for more information.

SAS files generated in photometric recalibration
File Type in/out Description URL format format parameters
calibPhotomGlobal out Ubercal results (nanomaggies/count) and associated flat fields eboss/calib/%s/%d/%d/nfcalib/calibPhotomGlobal-%06d-%d.fits calibration rerun, rerun, run, run, camcol
calib out QA plots for calibration. boss/calib/%s/plots/calib-%06d-%c-%s.png calibration rerun, run, filter, type (flat, hist, run)

Photometric Catalogs

The final photometric catalogs contain the raw data that will be loaded into the CAS database.

SAS files produced by the final catalog

File Type in/out Description URL format format parameters
photoObj out Full, calibrated outputs of the imager photometric pipeline eboss/photoObj/%d/%d/%d/photoObj-%06d-%d-%04d.fits rerun, run, camcol, run, camcol, field
photoField out Summary outputs of the properties of every field eboss/photoObj/%d/%d/photoField-%06d-%d.fits rerun, run, run, camcol
photoRunAll out Summary information of the properties of every photometric run eboss/photoObj/photoRunAll-%s.fits release (dr12)

References