Changes in the Tree

The definitions of many SDSS environment variables and sdss_access path names can change over time, as directory paths or file locations change as data gets shuffled around, or items get prepared for public release. sdss-tree provides a few convenience functions for tracking and identifying these changes across different SDSS data releases, or more specifically, across the different sdss-tree configurations.

View the history of a single parameter

To lookup the definition of a single environment variable or sdss_access path name, use the get_envvar_history or get_path_history convenience functions. These functions returns a dictionary of tree configurations with the value of the specified environment variable or sdss_access path name.

For example, to lookup all available defintions for the MANGA_HI environment variable, do

>>> from tree.tree import get_envvar_history, get_path_history

>>> # return all available definitions for the MANGA_HI environment variable
>>> get_envvar_history('MANGA_HI')
{'bosswork': None,
 'mpl3': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI',
 'mpl4': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI',
 'mpl5': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI',
 'mpl6': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI',
 'mpl7': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI',
 'mpl8': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI',
 'mpl9': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI',
 'mpl10': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI',
 'mpl11': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI',
 'dr7': None,
 'dr8': None,
 'dr9': None,
 'dr10': None,
 'dr11': None,
 'dr12': None,
 'dr13': None,
 'dr14': None,
 'dr15': '/Users/Brian/Work/sdss/sas/dr15/manga/HI',
 'dr16': '/Users/Brian/Work/sdss/sas/dr16/manga/HI',
 'dr17': '/Users/Brian/Work/sdss/sas/dr17/manga/HI',
 'sdss5': None,
 'sdsswork': '/Users/Brian/Work/sdss/sas/mangawork/manga/HI'}

For example, to lookup all available defintions for the mangaimage access path name, do

>>> from tree.tree import get_path_history

>>> # return all available definitions for the mangaimage access path name
>>> get_path_history('mangaimage')
{'bosswork': None,
 'mpl3': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/{dir3d}/images/{ifu}.png',
 'mpl4': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/{dir3d}/images/{ifu}.png',
 'mpl5': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/{dir3d}/images/{ifu}.png',
 'mpl6': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/{dir3d}/images/{ifu}.png',
 'mpl7': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/{dir3d}/images/{ifu}.png',
 'mpl8': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/images/{ifu}.png',
 'mpl9': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/images/{ifu}.png',
 'mpl10': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/images/{ifu}.png',
 'mpl11': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/images/{ifu}.png',
 'dr7': None,
 'dr8': None,
 'dr9': None,
 'dr10': None,
 'dr11': None,
 'dr12': None,
 'dr13': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/{dir3d}/images/{ifu}.png',
 'dr14': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/{dir3d}/images/{ifu}.png',
 'dr15': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/{dir3d}/images/{ifu}.png',
 'dr16': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/{dir3d}/images/{ifu}.png',
 'dr17': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/images/{ifu}.png',
 'sdss5': None,
 'sdsswork': '$MANGA_SPECTRO_REDUX/{drpver}/{plate}/images/{ifu}.png'}

Compute a complete changelog

To compute a complete changelog for all environment variables and sdss_access path names, across all releases, i.e. tree configurations, use the compute_changelog function. compute_changelog returns a dictionary of changes in the environment, or paths sections of the Tree, categorized as new, updated, or removed definitions.

To compute the changelog between DR17 and DR16

>>> from tree.changeloge import compute_changelog
>>> cl = compute_changelog('dr17', 'dr16')

>>> # show the envvar changes
>>> cl['environment']
{'new': {},
 'changes': {'SDSS': {'SDSS_QSO': '/dr17/sdss/qso/dr7_qsoals'},
  'APOGEE': {'APOGEE_FIRE_SIM': '/dr17/apogee/apogee/vac/apogee-fire_sim',
  'APOGEE_GC': '/dr17/apogee/apogee/vac/apogee-gc',
  'APOGEE_NC': '/dr17/apogee/apogee/vac/apogee-nc',
  'APOGEE_NET': '/dr17/apogee/apogee/vac/apogee-net',
  'APOGEE_PHOTVAR': '/dr17/apogee/apogee/vac/apogee-photvar',
  'APOGEE_SB2': '/dr17/apogee/apogee/vac/apogee-sb2',
  'APOGEE_WD_BINARY': '/dr17/apogee/apogee/vac/apogee-wd_binary',
  'APOGEE_DISTMASS': '/dr17/apogee/apogee/vac/apogee-distmass',
  'APOGEE_GRAVPOT16': '/dr17/apogee/apogee/vac/apogee-gravpot16'},
  'MANGA': {'MANGAPREIM_DIR': {'from': '$PRODUCT_ROOT/data/manga/mangapreim/tags/v2_5',
    'to': '$PRODUCT_ROOT/data/manga/mangapreim/tags/v2_9'},
  'MANGACORE_DIR': {'from': '$PRODUCT_ROOT/repo/manga/mangacore/tags/v1_6_2',
    'to': '$PRODUCT_ROOT/repo/manga/mangacore/tags/v1_9_1'},
  'MANGA_AGN': '/dr17/manga/agn',
  'MANGA_MANDALA': '/dr17/manga/mandala',
  'MANGA_SPECTRO_LENSING': '/dr17/manga/spectro/lensing',
  'MANGA_SPECZ': '/dr17/manga/spectro/specz'}},
'removed': {}}

>>> # show the path changes
>>> cl['paths']
{'new': {'apogee_astronn': '$APOGEE_ASTRONN/apogee_astroNN-{release}.fits',
  'apogee_distmass': '$APOGEE_DISTMASS/APOGEE_DistMass-{version}.fits',
  'apogee_fire_sim': '$APOGEE_FIRE_SIM/{firesimver}/{simulation}/lsr_{lsr}/apogee-{simulation}-lsr-{lsr}-rslice-{slice}.fits',
  'apogee_gc': '$APOGEE_GC/GC_{type}_VAC-{gcver}.fits',
  'apogee_gravpot16': '$APOGEE_GRAVPOT16/GravPot16_VAC_DR17.fits',
  'apogee_nc_abund': '$APOGEE_NC/{release}_nc_abund_{version}.fits',
  'apogee_net_II': '$APOGEE_NET/apogee_net_II-{version}.fits',
  'apogee_occam_cluster': '$APOGEE_OCCAM/occam_cluster-DR17.fits',
  'apogee_occam_member': '$APOGEE_OCCAM/occam_member-DR17.fits',
  'apogee_photvar': '$APOGEE_PHOTVAR/{version}/APOGEE-PhotVar-{version}.fits',
  ...,
  ...,
}

To output the changelog in string format, use the pprint=True keyword argument, e.g. compute_changelog('dr17', 'dr16', pprint=True), which pretty-prints the changelog into a nicer format. See Tree Evolution for an example pretty-printing of the changelog between DR17 and DR16.