ETC verification
================

The "ETC Requirements Specification", VIS-SPE-4MOST-47110-1740-0001_
document describes how the ETC works and which TOAD data should be
used (e.g. average fibre area, effective telescope area, etc.). This
document lists some tests designed to check intermediate steps as well as
the final product (S/N curve) of the ETCs. This test report will be
only partially comprehensible without a close read of the
specification..

.. _VIS-SPE-4MOST-47110-1740-0001: https://ds-web.aip.de/docushare/dsweb/Services/Document-8238

The comparison data were obtained from TOAD. TOAD models the 4MOST
instrument extensively and as accurately as possible at this stage of
the development. TOAD is therefore considered to be closest to
reality. The version of the TOAD validation data, used in all tests,
can be found in the `Gitlab repository`_.

.. _Gitlab repository: https://gitlab.4most.eu/OpSys/etc

.. note::

   In some spectrograph arms, typically green/red, in the bottom
   panels one sees large P-Cygni-like residuals coming from emission
   or absorption lines. These residuals can get much higher than our
   limit of 5%. They are due to the difference in the resolution of
   the input spectra/transmission curves and the difference in the
   interpolation techniques. E.g. TOAD sky emission and transmission
   spectra have 650,000 resolution elements. 4FS has
   100,000. Therefore, these spikes in the residual are of no
   consequence and do not influence the results or conclusions.

The selected wavelengths of the tests are:

.. exec::

   from qmostetc.verification import obslist, VerificationBase
   print(".. list-table::")
   print("   :header-rows: 1\n")
   print("   * - spectrograph")
   print("     - arm")
   for pos in VerificationBase.pix_pos:
	print(f"     - Bin #{pos}")
   for spec, observatory in obslist.items():
       for arm, obs in observatory.items():
	   print(f"   * - {spec}")
	   print(f"     - {arm}")
	   for wl in obs.spectrograph.wavelength[VerificationBase.pix_pos]:
	       print(f"     - {wl:.2f}")

These pixels are indicated as stars in the verification plots.


Test #1: Coupling to fibre flux
-------------------------------

Testing if the coupling from total to fibre flux is correctly
done. Compares the number of electrons per pixel for each spectrograph
without sky. Full line is TOAD (reference), dashed line is 4FS
ETC. None of the selected pixels (indicated as stars) has a higher
than indicated tolerance.

The success condition of test 1 has been fulfilled, and 4FS is
consistent with TOAD.

Settings:

.. exec::

   from qmostetc.verification import verification_1
   for s in verification_1().summary_settings():
       print(f" * {s}")

The test includes variations of:

.. exec::

   from qmostetc.verification import verification_1
   for s in verification_1().summary_variations():
       print(f" * {s}")

.. plot::

   from qmostetc.verification import verification_1
   verification_1().plot()

Test #2: Re-binning and instrument convolving
---------------------------------------------

Here we test if the projection of emission lines on the instrument
pixel borders is correct. The plots below show the output spectra for
lines with varying center and line width. The test is about the FWHM
and the recovered flux of each line, which is within 5% (original
requirement is 10%).

Settings:

.. exec::

   from qmostetc.verification import verification_2
   for s in verification_2().summary_settings():
       print(f" * {s}")

The test includes variations of:

.. exec::

   from qmostetc.verification import verification_2
   for s in verification_2().summary_variations():
       print(f" * {s}")

.. plot::

   from qmostetc.verification import verification_2
   verification_2().plot()


Test #3: SNR scaling with exposure time
---------------------------------------

Testing if the SNR per pixel scales correctly with exposure time and
with NDIT. This compares a point source. Compares the SNR per pixel
for each spectrograph without sky flux for different exposure
times. Full line is TOAD (reference), dashed line is 4FS ETC. None of
the selected pixels (indicated as stars) has a higher than indicated
tolerance. The SNR of the long exposure time is within 1% of √2 times
the SNR of the short exposure time.

Settings:

.. exec::

   from qmostetc.verification import verification_3
   for s in verification_3().summary_settings():
       print(f" * {s}")

The test includes variations of:

.. exec::

   from qmostetc.verification import verification_3
   for s in verification_3().summary_variations():
       print(f" * {s}")

.. plot::

   from qmostetc.verification import verification_3
   verification_3().plot()



Test #4: Sky brightness
-----------------------

Testing the sky brightness. Assume a sky fibre observation, no target
flux.  Full line is TOAD (reference), dashed line is 4FS ETC. None of
the selected pixels (indicated as stars) has a higher than
indicated tolerance.

Settings:

.. exec::

   from qmostetc.verification import verification_4
   for s in verification_4().summary_settings():
       print(f" * {s}")

The test includes variations of:

.. exec::

   from qmostetc.verification import verification_4
   for s in verification_4().summary_variations():
       print(f" * {s}")

.. plot::

   from qmostetc.verification import verification_4
   verification_4().plot()


Test #5/7: "Real" spectra and end-to-end performance
----------------------------------------------------

These tests are performed with “real” spectra. First, the resulting
spectra are compared with TOAD. Full line is TOAD (reference), dashed
line is 4FS ETC (test #5). None of the selected pixels (indicated as
stars) has a higher than indicated tolerance.

Then, the SNR calculated by TOAD is converted into “Rules” and
“Rulesets” files, which are used by the ETC to calculate the exposure
time. The exposure time for the given SNR is recalculated by the ETC
and compared with the original time (20 min) (test #7). The calculated
exposure time together with its deviation is shown in the plot. It is
always within the required tolerance.

The wavelength ranges for the rules are:

.. exec::

   from qmostetc.verification import VerificationBase
   import astropy.units as u
   print(".. list-table::")
   print("   :header-rows: 1\n")
   print("   * - spectrograph")
   print("     - arm")
   print("     - wavelength range")

   for (spec, arm), (l_min, l_max) in VerificationBase.wl_range.items():
       print(f"   * - {spec}")
       print(f"     - {arm}")
       print(f"     - {l_min.to_value(u.nm):.1f} … {l_max.to(u.nm):.1f}")

Settings:

.. exec::

   from qmostetc.verification import verification_5
   for s in verification_5().summary_settings():
       print(f" * {s}")

The test includes variations of:

.. exec::

   from qmostetc.verification import verification_5
   for s in verification_5().summary_variations():
       print(f" * {s}")

.. plot::

   from qmostetc.verification import verification_5
   verification_5().plot()


Test #8: Extended source and airmass
------------------------------------

These tests are performed with “real” spectra in different target
shapes, magnitudes, seeings, airmasses and sky brightnesses. First,
the resulting spectra are compared with TOAD. Full line is TOAD
(reference), dashed line is 4FS ETC. None of the selected pixels
(indicated as stars) has a higher than indicated tolerance.

Then, the SNR calculated by TOAD is converted into “Rules” and
“Rulesets” files, which are used by the ETC to calculate the exposure
time. The exposure time for the given SNR is recalculated by the ETC
and compared with the original time (20 min). The calculated exposure
time together with its deviation is shown in the plot. It is always
within the required tolerance.

The wavelength ranges for the rules are:

.. exec::

   from qmostetc.verification import VerificationBase
   import astropy.units as u
   print(".. list-table::")
   print("   :header-rows: 1\n")
   print("   * - spectrograph")
   print("     - arm")
   print("     - wavelength range")

   for (spec, arm), (l_min, l_max) in VerificationBase.wl_range.items():
       print(f"   * - {spec}")
       print(f"     - {arm}")
       print(f"     - {l_min.to_value(u.nm):.1f} … {l_max.to(u.nm):.1f}")

Settings:

.. exec::

   from qmostetc.verification import verification_8
   for s in verification_8().summary_settings():
       print(f" * {s}")

The test includes variations of:

.. exec::

   from qmostetc.verification import verification_8
   for s in verification_8().summary_variations():
       print(f" * {s}")

.. plot::

   from qmostetc.verification import verification_8
   verification_8().plot()