Implement NRS MOS+FS processing

[stscijgbot-jp]

Issue JP-3102 was created on JIRA by Howard Bushouse:

As described in JSOCINT-618 (and a plethora of related tickets for several subsystems), Cal needs to figure out how we might be able processing NIRSpec MOS observations that also contain a selected source in one of the fixed slits. Hence, calwebb_spec2 would need to somehow process a given exposure as both MOS and FS. In this ticket we'll explore the differences between calibration of MOS and FS data and potential ways to overcome those differences in order to essentially treat the fixed slits as just additional MOS slitlets.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

As a starting point I used the exposure jw01128019001_0310i_00001_nrs1, which is a MOS mode exposure taken during commissioning, and did separate calwebb_spec2 processing runs on it, first as a standard MOS exposures, and second as a FS exposure, in order to compare the steps that are applied and the references files used by each step.

In order to test FS processing, I modified the following header keywords in the rate image:

• EXP_TYPE: NRS_MSASPEC -> NRS_FIXEDSLIT
• FXD_SLIT: Added to header with a value of "S200A1"
• SRCTYAPT: UNKNOWN -> POINT

I'm guessing there will also be issues with other keywords related to pointing information, such as APERNAME (set to "NRS_FIELD1_MSA4" for MOS), and all of the keyword values derived from that aperture, including V2_REF, V3_REF, V3I_YANG, RA_REF, DEC_REF, ROLL_REF, etc. This will need to be explored in more detail in terms of how those pointing-related values may affect each calwebb_spec2 step.
||Step||MOS||FS||Mismatch?||Notes||
|assign_wcs|CAM 0007
COLL 0007
DISP 0062
FORE 0049
FPA 0008
MSA 0008
OTE 0008
WAVERA 0006|0007
0007
0062
0049
0008
0008
0008
0006| |Suprisingly enough, both modes use all the same WCS-related ref files!|
|imprint|N/A|N/A| |Not executed|
|background|N/A|N/A| |Not executed|
|msa_flagging|0003|0001| |Step not executed for FS. Ref file is dummy and not used.|
|extract_2d|N/A|N/A| |No ref files used by step|
|srctype|N/A|N/A| |No ref files used by step|
|wavecorr|0004|0003|X| |
|flat_field|DFLAT 0001
FFLAT 0074
SFLAT 0166|DFLAT 0001
FFLAT 0059
SFLAT 0153| 
X
X| |
|pathloss|0002|0001|X| |
|barshadow|0001|N/A| |Step not applied to FS.|
|photom|PHOTOM 0015
AREA 0033|PHOTOM 0014
AREA 0058|X
X| |
|resample_spec|N/A|N/A| |No ref files used by step|
|extract_1d|EXTR1D 0003
APCORR 0004|EXTR1D 0008
APCORR 0003|X
X| |

 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The wavecorr step uses different information for MOS vs. FS to determine the location of the source within a slit. From the code comments:

The slit_x_position is determined in the following way.

MOS data:
``slit.source_xpos`` is used. It is read in from the msa_metadata_file
in the assign_wcs step.

FS data:
``input_model.meta.dither.x_offset``, populated by SDP, is used. The
value is in the telescope Ideal frame. To calculate the slit position,
it is transformed Ideal --> V2, V3 --> slit_frame. ```
For the MOS mode the `estimated_source_in_shutter_x` and `estimated_source_in_shutter_y` table entries are used. If the APT/MPT could compute entries for these items when adding the fixed slits to the MSA configuration data, the same MOS info could be used for both.

 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The srctype step uses the "stellarity" value for each source stored in the MSA configuration data file to determine point/extended type for each MOS source. For the FS mode, the SRCTYAPT keyword value (if set by the user) is used to determine point/extended type for the fixed slit designated as primary and sets all the others to EXTENDED. If the APT/MPT could supply stellarity info for at least the source contained in the equivalent of the primary slit, we could again use the MSA config data for both MOS and FS processing.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

In the flat_field step, both MOS and FS data use the same function "nirspec_fs_msa". This function contains one hook to handle the primary fixed slit in a special if it contains a point source, in which case it computes flat_field data arrays for both point and extended source (to be used later for master background subtraction). The remaining fixed slits and MOS slits are processed the same, modulo the use of different FFLAT and SFLAT reference data.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The pathloss step uses separate functions for MOS and FS data: "do_correction_mos" and "do_correction_fixedslit". The intermediate-level algorithms have significant differences, mainly due to the fact that for MOS slitlets the correction is highly dependent on the total number of shutters contained in the slitlet and which shutter has the source in it. Because the fixed slits are ... fixed, the application of the correction is far more straightforward. And of course again the contents of the reference files for MOS vs FS are quite different. Getting this step to work on a fixed slit that's masquerading as a MOS slitlet could be challenging.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The photom step uses different search criteria for FS vs. MOS modes when finding the applicable row of flux calibration data in the PHOTOM ref file. FS uses Filter, Grating, and Slit name, while MOS uses only Filter and Grating. Beyond that, the actual application of the flux calibration is the same for both FS and MOS slit instances and have the same dependencies on source type (point vs extended).

[stscijgbot-jp]

Comment by Megan Beddard on JIRA:

Howard Bushouse Is this ticket a sub task to completing JSOCINT-618? If so can you please link it as a child?

[stscijgbot-jp]

Comment by Ian Broussard on JIRA:

Megan Beddard  JWSTDMS-761 is the ticket tracking the DMS effort for JSOCINT-618.  It should be linked already to 618.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

I am working on a first draft of the code to support this mode, but I have some questions about the details of the processing.

The basic idea is currently:

• Inside the spec2 pipeline:
  ** The assign_wcs, bkg_subtract, extract_2d, and srctype steps proceed as usual.
  ** The input model is then split to process MOS slits together through these steps: master_background_mos, wavecorr, flat_field, pathloss, barshadow, photom.
  ** Then, FS slits are processed through the same steps, except master_background_mos, and appended to the output MOS model.  If save_results is set for these steps, the FS products are saved with an additional '_fs' suffix.
  ** The combined product is processed through resampling together.
  ** The product is split again for extraction, to use the correct default extraction and aperture reference files for each mode.  The extracted spectra are recombined before saving the output product.
• Inside the spec3 pipeline:
  ** Inputs are sorted by source.  Sources from fixed slits are then processed as fixed slit; sources from MOS slits are then processed as MOS.
  ** The outlier_detection, resample_spec, and extract_1d steps run as usual, with no further manipulation needed.

Questions for the NIRSpec team:

• If a fixed slit is not marked as a primary source in the MSA metadata file, should it be extracted? That is, should we extract background spectra from fixed slits that appear in the MSA file, or are they not likely to be useful?
• Is it desirable to extract S200B1, even though it is uncalibrated?  It is extracted in the ALLSLITS fixed slit case.
• How should master background subtraction in spec3 be handled? Should it just be skipped for this combined mode? Or should accumulate from all the spectra but apply only to fixed slits?  Or accumulate only from fixed slits and apply only to fixed slits? Or something else?

James Muzerolle Christian Hayes - please let me know if these questions are clear or if you need more information.  I'm happy to meet up offline to discuss if that would be useful.

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

Melanie Clarke, thanks for the summary.  I will bring these up with the NIRSpec team for discussion and get back to you, but I might follow up with you on a few clarifying questions in the meantime.

[stscijgbot-jp]

Comment by Jesse Doggett on JIRA:

I've modified SDP to add a column fixed_slit column to the MSA FITS file SHUTTER_INFO binary table extension. It will contain the fixed_slit column value in the PPS database table nirspec_msa_metadata for the visit of the MSA FITS file. (JSDP-2460)

Let me know if there is anything else CAL needs for supporting JSOCINT-618.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

An update for the NIRSpec team:

In separate conversations with the APT development team, it was stated that fixed slits will not be allowed to be planned as background in combined MOS/FS mode.  They will only appear in the MSA file only if they are planned with a source.  I've updated the draft code to only extract sources marked as primary.  In this case, there should be no special handling needed for S200B1: if it is planned (which it shouldn't be!), it will be extracted.

 

Also, a proposal for master background handling in spec3:

It is currently possible to use the master background step to remove a background from MOS spectra, if a background spectrum is manually specified in the association file or in the command line argument.  There is no special handling for point sources vs. extended sources, though, so this step is likely to do the wrong thing if the user is not very expert and very careful. 

I think it might be more sensible to disallow master background subtraction in spec3 for MOS slitlets: it should be done in spec2 only, in master_background_mos.  If we make this change, it should be possible to apply master background correction only to the FS slits in combined products in spec3.  Let me know if you like this approach, or prefer something else.

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

Melanie Clarke, thanks for the update.  Just to double check, am I correct that the remaining questions are:

• How should master background subtraction in spec3 be handled for the FS?
• Should master background be disallowed in spec3 for MOS slitlets?

I think for the second question, I like the approach of only allowing MOS master background in spec2, since there is a decent amount of bookkeeping that would be needed to perform the point/extended source handling properly in spec3.  But I'll bring that up for discussion and get back to you.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Thanks Christian Hayes - yes, those are the remaining questions.  When they are settled, it would also be helpful if the team could help with testing the new code, in case there is something else I've overlooked.

[stscijgbot-jp]

Comment by Jesse Doggett on JIRA:

I've run some MSA data through SDP to create MSA fits files with the new fixed_slit column in the SHUTTER_INFO extension table.  I put one at ████████████████████████████████████████████████████████████████ for you test reading the new format.

Note, it only has "NONE" values in the fixed slit column since we don't have any test data yet. However, could you please try and read it to confirm we're on the same page with the format and let me know if it's good for you or not?

Thanks.

 

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Thanks Jesse Doggett! I'll check it out today.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

The fixed_slit column looks as expected in the SHUTTER_INFO table.  This file fails to read in in the cal pipeline, because it lists multiple primary sources within the same slitlet_id, but that's unrelated to the new column.

I tried trimming the file down to keep only one of the sources listed in slitlet_id 2, and it then processes correctly through the cal pipeline, when used with a sample data set (jw02674004001_03101_00001_nrs2_rate.fits).  I reduced the data through the spec2 pipeline with this modified MSA file with both the current main development branch and with my draft branch for supporting combined MOS/FS branch, and the output was identical.  

I think this test means that this format should be okay for standard MOS observations.

[stscijgbot-jp]

Comment by Jesse Doggett on JIRA:

It seems concerning that we created a file that could not be read as is.

 

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

Melanie Clarke I discussed your questions with the NIRSpec team and we support disallowing master background subtraction for MOS slitlets in spec3.  Let me know if you need any other feedback, otherwise, we'd be happy to test this when it is ready!

[stscijgbot-jp]

Comment by Jesse Doggett on JIRA:

Thanks. We could test anytime given a cal build that has your changes. However, I think we're waiting for test data that's not coming until July.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Excellent, thanks Christian Hayes!  This should be close to done, then. I'll wrap things up and start moving it through reviews.  I'll tag you for a review on the PR if you have some time to take a look.

Jesse Doggett - can you give me some more details on where the file you made came from? I can try to help track down whether the multiple sources failure is a problem, or just an off-nominal test data condition.

[stscijgbot-jp]

Comment by Jesse Doggett on JIRA:

The file was automatically generated through SDP Level 1 processing of data directly off the telescope for observation jw00643029.

[stscijgbot-jp]

Comment by Jesse Doggett on JIRA:

What will happen if the MSA FITS files doesn't have a fixed_slit column for a MOS observation that does not use a fixed slit? Normally, SDP does not re-create existing MSA FITS files.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

The code is backwards compatible.  If there's no fixed_slit column, it will just handle all slits as regular MSA slitlets.

[stscijgbot-jp]

Comment by Jesse Doggett on JIRA:

That's awesome. Thanks.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed  by #8467

[stscijgbot-jp]

Comment by David Law on JIRA:

Christian Hayes I think this is now all done and ready to close?

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

David Law we've tested this with some hand-made MSA files and it works as expected, but I'm not sure if we wanted to keep this as RTT until the MOS+FS mode can be run on OTB as part of JSOCINT-618 so that we can do an end-to-end test with properly created MSA files (there was an initial test to try this, but it didn't include FS targets and I haven't seen a new test done yet).

[stscijgbot-jp]

Comment by David Law on JIRA:

Ok, in that case I'm going to assign it to you; please close or comment as necessary once the NIRSpec team is happy with how things are working on the OTB.