JP-3102: NIRSpec combined MOS/FS processing (#8467)

Co-authored-by: Howard Bushouse <[email protected]>

CHANGES.rst

@@ -27,6 +27,10 @@ assign_wcs
 - Update default parameters to increase the accuracy of the SIP approximation
   in the output FITS WCS. [#8529]
 
+- Added handling for fixed slit sources defined in a MSA metadata file, for combined
+  NIRSpec MOS and fixed slit observations. Slits are now appended to the data
+  product in the order they appear in the MSA file. [#8467]
+
 associations
 ------------
 
@@ -83,6 +87,10 @@ exp_to_source
   in order to support changes in `source_id` handling for NIRSpec MOS exposures
   that contain background and virtual slits. [#8442]
 
+- Update the top-level model exposure type from the slit exposure type,
+  to support processing for combined NIRSpec MOS and fixed slit
+  observations. [#8467]
+
 extract_1d
 ----------
 
@@ -108,11 +116,19 @@ extract_1d
 
 - Fix error in application of aperture correction to variance arrays. [#8530]
 
+- Removed a check for the primary slit for NIRSpec fixed slit mode:
+  all slits containing point sources are now handled consistently,
+  whether they are marked primary or not. [#8467]
+
 extract_2d
 ----------
 
 - Added handling for NIRCam GRISM time series pointing offsets. [#8449]
 
+- Added support for slit names that have string values instead of integer
+  values, necessary for processing combined NIRSpec MOS and fixed slit
+  data products. [#8467]
+
 flat_field
 ----------
 
@@ -122,9 +138,13 @@ flat_field
 - Update NIRSpec flatfield code for all modes to ensure SCI=ERR=NaN wherever the
   DO_NOT_USE flag is set in the DQ array. [#8463]
 
-- Updated the NIRSpec flatfield code to use the new format of the ``wavecorr`` 
+- Updated the NIRSpec flatfield code to use the new format of the ``wavecorr``
   wavelength zero-point corrections for point sources.  [#8376]
 
+- Removed a check for the primary slit for NIRSpec fixed slit mode:
+  all slits containing point sources are now handled consistently,
+  whether they are marked primary or not. [#8467]
+
 general
 -------
 
@@ -139,9 +159,22 @@ lib
 ---
 
 - Updated the ``wcs_utils.get_wavelength`` to use the new format
-  of the ``wavecorr`` wavelength zero-point corrections for point 
+  of the ``wavecorr`` wavelength zero-point corrections for point
   sources in NIRSpec slit data. [#8376]
 
+master_background
+-----------------
+
+- Removed a check for the primary slit for NIRSpec fixed slit mode:
+  all slits containing point sources are now handled consistently,
+  whether they are marked primary or not. [#8467]
+
+- Disabled support for master background correction for NIRSpec MOS
+  slits in the ``master_background``, called in ``calwebb_spec3``.
+  Master background correction for MOS mode should be performed
+  via ``master_background_mos``, called in ``calwebb_spec2``. [#8467]
+
+
 master_background_mos
 ---------------------
 
@@ -156,6 +189,10 @@ nsclean
   sparse matrix to perform linear algebra operations with a diagonal weight
   matrix. [#8547]
 
+- Added a check for combined NIRSpec MOS and fixed slit products: if fixed
+  slits are defined in a MOS product, the central fixed slit quadrant
+  is not automatically masked. [#8467]
+
 outlier_detection
 -----------------
 
@@ -198,6 +235,10 @@ photom
 - Ensure that NaNs in MRS photom files are not replaced with ones by
   pipeline code for consistency with other modes [#8453]
 
+- Removed a check for the primary slit for NIRSpec fixed slit mode:
+  all slits containing point sources are now handled consistently,
+  whether they are marked primary or not. [#8467]
+
 pipeline
 --------
 
@@ -215,6 +256,12 @@ pipeline
 - Added new optional step ``badpix_selfcal`` to the ``calwebb_spec2`` to self-calibrate
   bad pixels in IFU data. [#8500]
 
+- Added ``calwebb_spec2`` pipeline handling for combined NIRSpec MOS and
+  fixed slit observations. Steps that require different reference files
+  for MOS and FS are run twice, first for all MOS slits, then for all
+  FS slits. Final output products (``cal``, ``s2d``, ``x1d``) contain the
+  combined products. [#8467]
+
 pixel_replace
 -------------
 
@@ -303,7 +350,12 @@ wavecorr
 
 - Changed the NIRSpec wavelength correction algorithm to include it in slit WCS
   models and resampling.  Fixed the sign of the wavelength corrections. [#8376]
-
+
+- Added a check for fixed slits that already have source position information,
+  assigned via a MSA metafile, for combined NIRSpec MOS and fixed slit processing.
+  Point source position is calculated from dither offsets only for standard
+  fixed slit processing. [#8467]
+
 wfss_contam
 -----------
 
@@ -615,7 +667,6 @@ resample
 
 - Changed deprecated ``stpipe.extern.configobj`` to ``astropy.extern.configobj``. [#8320]
 
-
 residual_fringe
 ---------------
 

docs/jwst/data_products/msa_metadata.rst

@@ -43,7 +43,7 @@ The overall structure of the MSA FITS file is as follows:
 +-----+---------------+----------+-----------+--------------------+
 |  1  | SHUTTER_IMAGE | IMAGE    | float32   | 342 x 730          |
 +-----+---------------+----------+-----------+--------------------+
-|  2  | SHUTTER_INFO  | BINTABLE | N/A       | variable x 12 cols |
+|  2  | SHUTTER_INFO  | BINTABLE | N/A       | variable x 13 cols |
 +-----+---------------+----------+-----------+--------------------+
 |  3  | SOURCE_INFO   | BINTABLE | N/A       | variable x 8 cols  |
 +-----+---------------+----------+-----------+--------------------+
@@ -89,6 +89,8 @@ The structure of the ``SHUTTER_INFO`` table extension is as follows:
 +-------------------------------+-----------+----------------------+
 | PRIMARY_SOURCE                | string    | Primary source flag  |
 +-------------------------------+-----------+----------------------+
+| FIXED_SLIT                    | string    | Fixed slit name      |
++-------------------------------+-----------+----------------------+
 
 - SLITLET_ID: integer ID of each slitlet consisting of one or more
   open shutters.
@@ -102,17 +104,20 @@ The structure of the ``SHUTTER_INFO`` table extension is as follows:
   quadrant.
 - SOURCE_ID: unique integer ID for each source in each slitlet, used
   for matching to an entry in the ``SOURCE_INFO`` table.
-- BACKGROUND: boolean indicating whether the shutter is open to background
-  only or contains a known source.
+- BACKGROUND: Y or N. Y indicates that the shutter is open to background
+  only.
 - SHUTTER_STATE: OPEN or CLOSED. Generally will always be OPEN, unless
   the shutter is part of a long slit that is not contiguous.
 - ESTIMATED_SOURCE_IN_SHUTTER_X/Y: the position of the source within the
   shutter in relative units, where 0,0 is bottom-left, 0.5,0.5 is center,
   and 1,1 is upper-right, as planned in the MPT.
 - DITHER_POINT_INDEX: integer index of the nod sequence; matches to
   header keyword `PATT_NUM`.
-- PRIMARY_SOURCE: boolean indicating whether the shutter contains the
+- PRIMARY_SOURCE: Y or N. Y indicates that the shutter contains the
   primary science source.
+- FIXED_SLIT: string name of a fixed slit containing the source; set to
+  NONE for MSA slitlets. This column may not appear in older versions of
+  the MSA metadata files.
 
 It is the :ref:`assign_wcs <assign_wcs_step>` step in the
 :ref:`calwebb_spec2 <calwebb_spec2>` pipeline that opens and loads all
@@ -140,29 +145,29 @@ slitlet 2 is comprised of 3 shutters. Because a 3-point nod pattern has been use
 there are 3 different sets of metadata for each slitlet (one set for each dither/nod
 position) and hence a total of 9 entries (3 shutters x 3 dithers).
 
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
-| Slit | Meta |      |     |     |   Src  |     |       |  X    |  Y    | Dith | Pri |
-|      |      |      |     |     |        |     |       |       |       |      |     |
-| ID   | ID   | Quad | Row | Col |   ID   | Bkg | State |  pos  |  pos  | Pt   | Src |
-+======+======+======+=====+=====+========+=====+=======+=======+=======+======+=====+
-|   2  |   1  |  2   |  10 | 154 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  1   |  N  |
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
-|   2  |   1  |  2   |  10 | 155 |   42   |  N  | OPEN  | 0.399 | 0.702 |  1   |  Y  |
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
-|   2  |   1  |  2   |  10 | 156 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  1   |  N  |
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
-|   2  |   1  |  2   |  10 | 154 |   42   |  N  | OPEN  | 0.410 | 0.710 |  2   |  Y  |
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
-|   2  |   1  |  2   |  10 | 155 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  2   |  N  |
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
-|   2  |   1  |  2   |  10 | 156 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  2   |  N  |
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
-|   2  |   1  |  2   |  10 | 154 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  3   |  N  |
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
-|   2  |   1  |  2   |  10 | 155 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  3   |  N  |
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
-|   2  |   1  |  2   |  10 | 156 |   42   |  N  | OPEN  | 0.389 | 0.718 |  3   |  Y  |
-+------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
+| Slit | Meta |      |     |     |   Src  |     |       |  X    |  Y    | Dith | Pri | Fxd  |
+|      |      |      |     |     |        |     |       |       |       |      |     |      |
+| ID   | ID   | Quad | Row | Col |   ID   | Bkg | State |  pos  |  pos  | Pt   | Src | Slit |
++======+======+======+=====+=====+========+=====+=======+=======+=======+======+=====+======+
+|   2  |   1  |  2   |  10 | 154 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  1   |  N  | NONE |
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
+|   2  |   1  |  2   |  10 | 155 |   42   |  N  | OPEN  | 0.399 | 0.702 |  1   |  Y  | NONE |
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
+|   2  |   1  |  2   |  10 | 156 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  1   |  N  | NONE |
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
+|   2  |   1  |  2   |  10 | 154 |   42   |  N  | OPEN  | 0.410 | 0.710 |  2   |  Y  | NONE |
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
+|   2  |   1  |  2   |  10 | 155 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  2   |  N  | NONE |
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
+|   2  |   1  |  2   |  10 | 156 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  2   |  N  | NONE |
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
+|   2  |   1  |  2   |  10 | 154 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  3   |  N  | NONE |
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
+|   2  |   1  |  2   |  10 | 155 |    0   |  Y  | OPEN  |  NaN  |  NaN  |  3   |  N  | NONE |
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
+|   2  |   1  |  2   |  10 | 156 |   42   |  N  | OPEN  | 0.389 | 0.718 |  3   |  Y  | NONE |
++------+------+------+-----+-----+--------+-----+-------+-------+-------+------+-----+------+
 
 The values in the `slitlet_id` column show that we're only looking at table
 rows for slitlet 2, all of which come from MSA configuration (`msa_metadata_id`) 1.
@@ -229,6 +234,20 @@ from a virtual slitlet with `source_id` = -42 will look like:
 
   jw12345-o066_v000000042_nirspec_f170lp_g235m_x1d.fits
 
+Fixed Slits
+~~~~~~~~~~~
+
+It is possible to plan fixed slit sources alongside standard MOS targets. In this case,
+a unique `slitlet_id` is not assigned in the MSA file.  Instead, the slit is identified
+by the value of the `fixed_slit` column.  This value may be set to any of the NIRSpec
+fixed slit names used for science: S200A1, S200A2, S400A1, or S1600A1.
+
+Fixed slit targets must always have `primary_source` = "Y" and `background` = "N".
+They will never be extracted as background sources.
+
+The `shutter_quadrant`, `shutter_row`, and `shutter_column` fields are set to placeholder
+values. All other values have the same meaning and values as for MSA slitlets.
+
 
 The SOURCE_INFO Metadata
 ------------------------

docs/jwst/exp_to_source/main.rst

@@ -34,14 +34,21 @@ configuration of MSA slitlets with a source assigned to each slitlet.
 The source-to-slitlet linkage is carried along in the information contained
 in the MSA metadata file used during :ref:`calwebb_spec2 <calwebb_spec2>`
 processing. Each slitlet instance created by the :ref:`extract_2d <extract_2d_step>`
-step stores the source ID (a simple integer number) in the SOURCEID keyword of
+step stores the source name in the SRCNAME keyword of
 the SCI extension header for the slitlet. The ``exp_to_source`` tool uses
-the SOURCEID values to sort the data from each input product into an
+the SRCNAME values to sort the data from each input product into an
 appropriate source-based output product.
 
-NIRSpec Fixed-Slit
+If fixed slit targets are planned as part of a NIRSpec MOS exposure, they
+will also have sources identified by the MSA metadata file.  The associated
+SRCNAME values are used to sort the data from these slits in the same way the
+MSA slitlets are sorted.  In this combined mode, the products containing
+MOS slitlets are marked with EXP_TYPE = "NRS_MSASPEC" after sorting; the models
+containing fixed slits are marked with EXP_TYPE = "NRS_FIXEDSLIT".
+
+NIRSpec Fixed Slit
 ^^^^^^^^^^^^^^^^^^
-NIRSpec fixed-slit observations do not have sources identified with each
+NIRSpec fixed slit observations do not have sources identified with each
 slit, so the slit names, e.g. S200A1, S1600A1, etc., are mapped to predefined
 source ID values, as follows:
 

docs/jwst/flatfield/main.rst

@@ -78,12 +78,12 @@ pixel.  This interpolation requires knowledge of the dispersion direction,
 which is read from keyword "DISPAXIS."  See the Reference File section for
 further details.
 
-For NIRSpec Fixed-Slit and MOS exposures, an on-the-fly flat-field is
+For NIRSpec Fixed Slit (FS) and MOS exposures, an on-the-fly flat-field is
 constructed to match each of the slits/slitlets contained in the science
 exposure. For NIRSpec IFU exposures, a single full-frame flat-field is
 constructed, which is applied to the entire science image.
 
-NIRSpec NRS_BRIGHTOBJ data are processed just like NIRSpec Fixed-Slit
+NIRSpec NRS_BRIGHTOBJ data are processed just like NIRSpec fixed slit
 data, except that NRS_BRIGHTOBJ data are stored in a CubeModel,
 rather than a MultiSlitModel.  A 2-D flat-field image is constructed
 on-the-fly as usual, but this image is then divided into each plane of
@@ -92,9 +92,9 @@ the 3-D science data arrays.
 In all cases, there is a step option that allows for saving the
 on-the-fly flatfield to a file, if desired.
 
-NIRSpec Fixed-Slit Primary Slit
+NIRSpec Fixed Slit Primary Slit
 -------------------------------
-The primary slit in a NIRSpec fixed-slit exposure receives special handling.
+The primary slit in a NIRSpec fixed slit exposure receives special handling.
 If the primary slit, as given by the "FXD_SLIT" keyword value, contains a
 point source, as given by the "SRCTYPE" keyword, it is necessary to know the
 flatfield conversion factors for both a point source and a uniform source
@@ -104,7 +104,7 @@ is applied to the slit data, but that correction is not appropriate for the
 background signal contained in the slit, and hence corrections must be
 applied later in the :ref:`master background <master_background_step>` step.
 
-So in this case the `flatfield` step will compute 2D arrays of conversion
+In this case, the `flatfield` step will compute 2D arrays of conversion
 factors that are appropriate for a uniform source and for a point source,
 and store those correction factors in the "FLATFIELD_UN" and "FLATFIELD_PS"
 extensions, respectively, of the output data product. The point source
@@ -116,8 +116,15 @@ applied by the :ref:`wavecorr <wavecorr_step>` step to account for any
 source mis-centering in the slit and the flatfield conversion factors are
 wavelength-dependent. A uniform source does not require wavelength corrections
 and hence the flatfield conversions will differ for point and uniform
-sources. Any secondary slits that may be included in a fixed-slit exposure
+sources. Any secondary slits that may be included in a fixed slit exposure
 do not have source centering information available, so the
-:ref:`wavecorr <wavecorr_step>` step is not applied, and hence there's no
+:ref:`wavecorr <wavecorr_step>` step is not applied, and there is no
 difference between the point source and uniform source flatfield
 conversions for those slits.
+
+Fixed slits planned as part of a combined MOS and FS observation are an
+exception to this rule.  These targets may each be identified as
+point sources, with location information for each given in the
+:ref:`MSA metadata file <msa_metadata>`. Point sources in fixed slits planned
+this way are treated in the same manner as the primary fixed slit in standard
+FS observations.

docs/jwst/master_background/description.rst

@@ -295,27 +295,25 @@ source centering within the slit, hence slits containing uniform sources receive
 the same flat-field and photometric calibrations as background spectra and
 therefore don't require corrections for those two calibrations. Furthermore, the
 source position in the slit is only known for the primary slit in an exposure, so
-even if the secondary slits contain point sources, no wavelength correction can
-be applied, and therefore again the flat-field and photometric calibrations are
-the same as for background spectra. This means only the pathloss correction
-difference between uniform and point sources needs to be accounted for in the
-secondary slits.
+secondary slits are always handled as extended sources, no wavelength correction is
+applied, and therefore again the flat-field, photometric, and pathloss calibrations
+are the same as for background spectra.
 
-Therefore if the primary slit (as given by the FXD_SLIT keyword) contains a point source
-(as given by the SRCTYPE keyword) the corrections that need to be applied to the 2-D
-master background for that slit are:
+Fixed slits planned as part of a combined MOS and FS observation are an
+exception to this rule.  These targets may each be identified as
+point sources, with location information for each given in the
+:ref:`MSA metadata file <msa_metadata>`. Point sources in fixed slits planned
+this way are treated in the same manner as the primary fixed slit in standard
+FS observations.
+
+Therefore, if a fixed slit contains a point source (as given by the SRCTYPE keyword)
+the corrections that need to be applied to the 2-D master background for that slit are:
 
 .. math::
  bkg(corr) = bkg &* [flatfield(uniform) / flatfield(point)]\\
                  &* [pathloss(uniform) / pathloss(point)]\\
                  &* [photom(point) / photom(uniform)]
 
-For secondary slits that contain a point source, the correction applied to the
-2-D master background is simply:
-
-.. math::
- bkg(corr) = bkg * pathloss(uniform) / pathloss(point)
-
 The uniform and point source versions of the flat-field, pathloss, and photom
 corrections are retrieved from the input :ref:`cal <cal>` product. They
 are computed and stored there during the execution of each of those steps

docs/jwst/nsclean/main.rst

@@ -103,6 +103,10 @@ processing MOS and IFU images. The masked region is currently hardwired
 in the step to image indexes [1:2048, 923:1116], where the indexes are
 in x, y order and in 1-indexed values.
 
+Note, however, that it is possible to plan one or more fixed slit targets
+alongside MSA slitlets in MOS observations. In this situation, the fixed
+slit region is not automatically masked.
+
 Left/Right Reference Pixel Columns
 ----------------------------------
 Full-frame images contain 4 columns of reference pixels on the left and