Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tickets/SP-2004: move opsim simulation archive submodule out of rubin_scheduler into rubin_sim #451

Merged
merged 2 commits into from
Mar 27, 2025
Merged

tickets/SP-2004: move opsim simulation archive submodule out of rubin_scheduler into rubin_sim #451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ef22389
move opsim simulation archive submodule out of rubin_scheduler into r…
ehneilsen Mar 18, 2025
d1d49f7
Reduce memory request
ehneilsen Mar 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions batch/compile_prenight_metadata_cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
#SBATCH --account=rubin:developers # Account name
#SBATCH --job-name=auxtel_prenight_daily # Job name
#SBATCH --output=/sdf/data/rubin/shared/scheduler/prenight/sbatch/compile_prenight_metadata_cache.out # Output file (stdout)
#SBATCH --error=/sdf/data/rubin/shared/scheduler/prenight/sbatch/compile_prenight_metadata_cache.err # Error file (stderr)
#SBATCH --partition=milano # Partition (queue) names
#SBATCH --nodes=1 # Number of nodes
#SBATCH --ntasks=1 # Number of tasks run in parallel
#SBATCH --cpus-per-task=1 # Number of CPUs per task
#SBATCH --mem=4G # Requested memory
#SBATCH --time=1:00:00 # Wall time (hh:mm:ss)

echo "******** START of compile_prenight_metadata_cache.sh **********"

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

# SLAC S3DF - source all files under ~/.profile.d
if [[ -e ~/.profile.d && -n "$(ls -A ~/.profile.d/)" ]]; then
source <(cat $(find -L ~/.profile.d -name '*.conf'))
fi

source /sdf/group/rubin/sw/w_latest/loadLSST.sh
conda activate /sdf/data/rubin/shared/scheduler/envs/prenight
export AWS_PROFILE=prenight
WORK_DIR=$(date '+/sdf/data/rubin/shared/scheduler/prenight/work/compile_prenight_metadata_cache/%Y-%m-%dT%H%M%S' --utc)
echo "Working in $WORK_DIR"
mkdir ${WORK_DIR}
cd ${WORK_DIR}
printenv > env.out
compile_sim_archive_metadata_resource --append
echo "******* END of compile_prenight_metadata_cache.sh *********"
69 changes: 69 additions & 0 deletions batch/run_prenight_sims.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
#!/usr/bin/env bash
#SBATCH --account=rubin:developers # Account name
#SBATCH --job-name=auxtel_prenight_daily # Job name
#SBATCH --output=/sdf/data/rubin/shared/scheduler/prenight/sbatch/run_prenight_sims.out # Output file (stdout)
#SBATCH --error=/sdf/data/rubin/shared/scheduler/prenight/sbatch/run_prenight_sims.err # Error file (stderr)
#SBATCH --partition=milano # Partition (queue) names
#SBATCH --nodes=1 # Number of nodes
#SBATCH --ntasks=1 # Number of tasks run in parallel
#SBATCH --cpus-per-task=1 # Number of CPUs per task
#SBATCH --mem=4G # Requested memory
#SBATCH --time=1:00:00 # Wall time (hh:mm:ss)

echo "******** START of run_prenight_sims.sh **********"

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

# SLAC S3DF - source all files under ~/.profile.d
if [[ -e ~/.profile.d && -n "$(ls -A ~/.profile.d/)" ]]; then
source <(cat $(find -L ~/.profile.d -name '*.conf'))
fi

date --iso=s

source /sdf/group/rubin/sw/w_latest/loadLSST.sh
conda activate /sdf/data/rubin/shared/scheduler/envs/prenight

set -o xtrace

export AWS_PROFILE=prenight
WORK_DIR=$(date '+/sdf/data/rubin/shared/scheduler/prenight/work/run_prenight_sims/%Y-%m-%dT%H%M%S' --utc)
echo "Working in $WORK_DIR"
mkdir ${WORK_DIR}
cd ${WORK_DIR}

# Get ts_ocs_config
TS_CONFIG_OCS_VERSION=$(obs_version_at_time ts_config_ocs)
curl --location --output ts_config_ocs.zip https://github.com/lsst-ts/ts_config_ocs/archive/${TS_CONFIG_OCS_VERSION}.zip
unzip ts_config_ocs.zip
mv $(find . -maxdepth 1 -type d -name ts_config_ocs\*) ts_config_ocs

# Install required python packages
PACKAGE_DIR=$(readlink -f ${WORK_DIR}/packages)
mkdir ${PACKAGE_DIR}

# Get the scheduler version from the EFD and install it.
RUBIN_SCHEDULER_TAG=v$(obs_version_at_time rubin_scheduler)
pip install --no-deps --target=${PACKAGE_DIR} git+https://github.com/lsst/rubin_scheduler.git@${RUBIN_SCHEDULER_TAG}

# Cannot get ts_fbs_utils from the EFD, so just guess the highest semantic version tag in the repo.
TS_FBS_UTILS_TAG=$(curl -s https://api.github.com/repos/lsst-ts/ts_fbs_utils/tags | jq -r '.[].name' | egrep '^v[0-9]+.[0-9]+.[0-9]+$' | sort -V | tail -1)
pip install --no-deps --target=${PACKAGE_DIR} git+https://github.com/lsst-ts/ts_fbs_utils.git@${TS_FBS_UTILS_TAG}

# Get the scheduler configuration script
SCHEDULER_CONFIG_SCRIPT=$(scheduler_config_at_time latiss)

# Get the path to prenight_sim as provided by the current environment,
# so we do not accidentally run one from the adjusted PATH below.
PRENIGHT_SIM=$(which prenight_sim)

export PYTHONPATH=${PACKAGE_DIR}:${PYTHONPATH}
export PATH=${PACKAGE_DIR}/bin:${PATH}
printenv > env.out
date --iso=s
time ${PRENIGHT_SIM} --scheduler auxtel.pickle.xz --opsim None --script ${SCHEDULER_CONFIG_SCRIPT}
date --iso=s
echo "******* END of run_prenight_sims.sh *********"
4 changes: 3 additions & 1 deletion docs/api.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,6 @@ API

Self Calibration <selfcal-api>

Skybrightness <skybrightness-api>
Sim archive <sim-archive-api>

Skybrightness <skybrightness-api>
276 changes: 276 additions & 0 deletions docs/archive.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,276 @@
.. py:currentmodule:: rubin_sim.sim_archive

.. _archive:

========================================================
The Prototype OpSim Archive for ``schedview`` Dashboards
========================================================

Introduction
------------

Several tools will require an archive that provides access to simulations provided by ``rubin_scheduler``.
For example, the prenight briefing dashboard supplied by ``schedview`` is a tool for visualizing simualtions of a night of observing, and it requires access to such simulations.
There will eventually be other dashboards that will also need to read such tables for visualization, as well as tables describing visits completed by the actual instruments.
Users of these dashboards will need to use them to select which data sets are to be visualised: the dashboard code needs to provide both user interface elements that let the user select the desired table of visits, and also actually load the table itself.
The dashboard servers run within containers on kubernetes-based infrastructure, which operate best when persistent data is stored outside the containers themselves.
Therefore, these dashboards require an external (to the container) resource that supports searching for available simulations, and access to the data itself.

This archive design is intended primarily as a prototype, something to experiment with for a better informed development of requiriments.
So, flexibility and speed of implementation have been prioritized, with the intention that there be a significant rofactoring (or even outright replacement) when requirements have been more thoroughly developed.

Design
------

The archive itself is a directory tree.
The URI of an archive is the URI of the root of the directory tree as supported by the ``lsst.resources`` package.
Each simulation in the archive has its own directory in this directory tree, named according to the ISO-8601 date on which a simulation was added to the archive, and a simple incrementing index separating different simulations added on the same day, such that the format for the directory for a specific simulation is::

${ARCHIVE_URI}/{ISO_DATE}/{ADDITION_INDEX}

For example, if the URI of the archive is::

file:///my_data/sim_archive

then the URI of the third simulation added on June 21, 2030 will be::

file:///my_data/sim_archive/2030-06-21/3

Each simulation directory contains a metadata yaml file named ``sim_metadata.yaml``.
In the above example, the URI for this metadata file would be::

file:///my_data/sim_archive/2030-06-21/3/sim_metadata.yaml

A minimal ``sim_metadata.yaml`` file specifies the name of the sqlite3 database file with the visits.
For example, if the URI for the visit database in the above example is ``file:///my_data/sim_archive/2030-06-21/3/opsim.db``, then the minimal content of ``sim_metadata.yaml`` would be::

files:
observations:
name: 'opsim.db'

All other data in the metadata file is optional, but additional metadata will be required if the archived simulation is to be used for some use cases.
For example, if ``schedview``'s ``prenight`` dashboard is to be able to load the reward data, it must be able to locate the reward data from the metadata file, so that the metadata file needs to look something like this::

files:
observations:
name: 'opsim.db'
rewards:
name: 'rewards.h5'

Clients of the archive will also need to search available simulations for those meeting relevant criteria.
For example, the ``prenight`` dashboard will seach for simulations the include a desired night, in which case the range of nights covered by the simulation must be included.

A sample metadata file that includes an early guess at what the ``prenight`` dashboard will use looks like this::

files:
observations:
name: opsim.db
rewards:
name: rewards.h5
label: Notebook test on 2024-01-04 16:49:44.299
simulated_dates:
first: '2025-05-05'
last: '2025-05-05'

In the above:

``label``
Simulations will appear in drop-down section widgets in dashdoards such as the pre-night dashboard.
The ``label`` element in the determines how the simulation will appear in the dropdown.
In other applications, this element may also be used as plot annotations or column or row headings.

``simulation_dates``
Shows the range of dates covered by the simulation.
When the user specifies a night, the ``prenight`` dashboard will restrict the offered to those that cover the specified date.


Finally, a number of other elements may be included for debugging purposes.
A full file might look something like this::

files:
environment:
md5: 4381d7cc82049141c70216121e39f56d
name: environment.txt
notebook:
md5: 6b75c1dd8c4a3b83797c873f3270cc04
name: notebook.ipynb
observations:
md5: 1909d1afaf744ee50bdcf1a9625826ab
name: opsim.db
pypi:
md5: 9c86ea9b4e7aa40d3e206fad1a59ea31
name: pypi.json
rewards:
md5: 6d3c9d3e0dd7764ed60312e459586e1b
name: rewards.h5
scheduler:
md5: 5e88dfee657e6283dbc7a343f048db92
name: scheduler.pickle.xz
statistics:
md5: c515ba27d83bdbfa9e65cdefff2d9d75
name: obs_stats.txt
label: Notebook test on 2024-01-04 16:49:44.299
simulated_dates:
first: '2025-05-05'
last: '2025-05-05'
scheduler_version: 1.0.1.dev25+gba1ca4d.d20240102
sim_runner_kwargs:
mjd_start: 60800.9565967191
record_rewards: true
survey_length: 0.5155218997970223
tags:
- notebook
- devel
host: neilsen-nb
username: neilsen

This example has a number of additional elements useful for debugging, and which pehaps might be useful for future applictions, but which are not used (or planned to be used) by the prenight dashboard.

``files/*``
A number of other types of files associated with specific simulations may be included.
These may be useful in future applications, or for debugging only.
See below for descriptions of the extra types of files in this example.
``files/${TYPE}/md5``
Checksums for various files.
These can be useful both for checking for corruption, and for determining whether two simulations are identical without needing to download either.
``scheduler_version``
The version of the scheduler used to produce the simualtions.
``sim_runner_kwargs``
The arguments to the execution of ``sim_runner`` used to run the simulation.
``tags``
A list of ad-hoc keywords.
For example, simulations used to test a specific jira issue may all have the name of the issue as a keyword.
Simulations used to support a give tech note may have the name of the tech note.
``host``
The hostname on which the simulation was run.
``username``
The username of the user who ran the simulation.

Optional (for debugging or speculative future uses only) file types listed above are:

``environment``
The conda environment specification for the environment used to run the simulation.
``notebook``
The notebook used to create the simulation, for example as created using the ``%notebook`` jupyter magic.
``pypy``
The ``pypy`` package list of the environment used to run the simulation.
If the simulation is run using only conda-installed packages, this will be redundant with ``environment``.
``scheduler``
A python pickle of the scheduler, in the state as of the start of the simulation.
``statistics``
Basic statistics for the visit database.

Metadata cache
--------------

Reading each ``sim_metadata.yaml`` individually when loading metadata for a large number of simulations can be slow.
Therefore, metadata for sets of simulations can be compiled into a ``compiled_metadata_cache.h5`` file.
This file stores four tables in `hdf5` format: ``simulations``, ``files``, ``kwargs``, and ``tags``.
Each of these tables is indexed by the URI of a simulation.

The ``files`` table contains one column for each key in the ``files`` dictionary in the yaml metadata file for the simulation, providing the metadata needed to reconstruct this element of the dictionary.

The ``kwargs`` table contains one column for each key in the ``sim_runner_kwargs`` dictionary in the yaml metadata file for the simulation, providing the metadata needed to reconstruct this element of the dictionary.
If a keyword argument is not set, an `numpy.nan` value is stored in the table.

The ``tags`` table contains one column: ``tag``, and contains one row for each tag in each simulation.

The ``simulations`` table contains one column for every other keyword found in the metadata yaml files.
If a keyword argument is not set, an `numpy.nan` value is stored in the table.

The ``compile_sim_archive_metadata_resource`` command in ``rubin_sim`` maintains the ``compiled_metadata_cache.h5`` file in an archive.
By default, it reads every ``sim_metadata.yaml`` file in the archive and builds a corresponding cache hdf5 file from scratch.
If called with an ``--append`` flag, it reads an existing metadata cache file, reads ``sim_metadata.yaml`` files for simulations more recently added than the last file in the existing cache, appends them to the previous results from the cache, and writes the result to the cache.
The ``append`` flag therefore speeds up the update considerably, but does not update the cache for any changes to previously added simulations (including deletions).

The ``compile_sim_archive_metadata_resource`` needs to be run to update the cache.
Normall, a cron job will execute this command routinely to keep the cache reasonably up to date.
Because the tools read the metadata yaml files for any simulations added after the most recent cache update, it will function correctly even if the cache is out of date (but slower).

Automatic archiving of generated data
-------------------------------------

The ``rubin_sim`` package provides a tool to combine running a simulation and adding the results to an archive, including any metadata that can be derived automatically.
The ``rubin_sim.sim_archive.drive_sim`` function is wrapper around ``rubin_sim.scheduler.sim_runner`` that incorporates this metadata collection and the creation of the entry in an archive.
It takes all of the same arguments that ``sim_runner`` does, and passes them directly to ``sim_runner``.
In addition, it takes a few arguments that specify the archive into which it is to be added (``archive_uri``), the label to be included in the metadata (``label``), and the code used to run the simulation (ethier ``script`` or ``notebook``).
Details are available in the ``drive_sim`` docstring.

For example, if this code is put into a file and run as a script, it will run the specificed simulation and add it to the specified archive::

from astropy.time import Time

from rubin_sim.scheduler.example import example_scheduler
from rubin_sim.scheduler.model_observatory import ModelObservatory
from rubin_sim.sim_archive import drive_sim

sim_mjd_start = Time("2025-05-05").mjd + 0.5
# The start date of the simualtion.
# Offset by 0.5 to avoid starting late when the MJD rollover occurs during or
# after twilight. See dayObs in SITCOMTN-32: https://sitcomtn-032.lsst.io/ .

sim_length = 1.0
# Passed to sum_runner, in units of days.

archive_uri = "file:///sdf/data/rubin/user/neilsen/data/test_sim_archive/"
# The URI of the root of the archive. The trailing "/" is required.

observatory = ModelObservatory()
scheduler = example_scheduler()
scheduler.keep_rewards = True

results = drive_sim(
observatory=observatory,
scheduler=scheduler,
archive_uri=archive_uri,
label=f"Example simulation started at {Time.now().iso}.",
script=__file__,
tags=["example"],
mjd_start=sim_mjd_start,
survey_length=sim_length,
record_rewards=True,
)

The result looks like this::

bash$ ls /sdf/data/rubin/user/neilsen/data/test_sim_archive/2024-01-18/1
environment.txt example_archived_sim_driver.py obs_stats.txt opsim.db pypi.json rewards.h5 scheduler.pickle.xz sim_metadata.yaml
bash$ cat /sdf/data/rubin/user/neilsen/data/test_sim_archive/2024-01-18/1/sim_metadata.yaml
files:
environment:
md5: 33f94ddf8975f9641a1f524fd22e362e
name: environment.txt
observations:
md5: 8b1ee9a604a88d2708d2bfd924ac3cd9
name: opsim.db
pypi:
md5: 51a8deee5018f59f20d5741fd1a64778
name: pypi.json
rewards:
md5: 10e4ab9397382bfa108fa21354da3526
name: rewards.h5
scheduler:
md5: 35713860dc9ba7a425500f63939d0e02
name: scheduler.pickle.xz
script:
md5: b4a476a4fd1231ea1ca44149784f1c3f
name: example_archived_sim_driver.py
statistics:
md5: 7c6a6af38aff3ce4145146e35f929b47
name: obs_stats.txt
host: sdfrome002.sdf.slac.stanford.edu
label: Example simulation started at 2024-01-18 15:46:27.758.
scheduler_version: 1.0.1.dev25+gba1ca4d.d20240102
sim_runner_kwargs:
mjd_start: 60800.5
record_rewards: true
survey_length: 1.0
simulated_dates:
first: '2025-05-04'
last: '2025-05-04'
tags:
- example
username: neilsen

Alternately, the simulation can be run from a ``jupyter`` notebook similarly, excepet that instead of saving the script that generated the simulation, a notebook with the cells of notebook that created the simulation up to the cell that runs the simulation can be stored instead.
An example can be found in the ``archive/sim_and_archive.ipynb`` in the `rubin_sim_notebook github repository <https://github.com/lsst/rubin_sim_notebooks>`_.
Loading
Loading