API reference

cosmic-crunch is primarily a command-line tool (cosmic-crunch get and cosmic-crunch convert), but its download and conversion routines are importable as a library.

Downloading

cosmic_crunch.fetch: crawl the JPL GENESIS site and download COSMIC radio-occultation ASCII data files.

Ported from the v1 get_files.py script (v1.3.1). The v1 per-file metadata and AGPL header have been removed; the package is MIT-licensed (see LICENSE) and versioned via cosmic_crunch.__version__.

exception cosmic_crunch.fetch.NoDataFilesFoundError[source]

Bases: RuntimeError

Raised when a crawl completes successfully but finds zero data files.

This is the v2 “fail loud” fix (design spine): the v1 crawler returned an empty list and the script exited successfully having downloaded nothing when the site moved. A zero-result crawl is now a loud error, not a silent no-op.

cosmic_crunch.fetch.crawl_cosmic_urls(*args, **kwargs)[source]

Retry-wrapped, pickleable wrapper around _crawl_cosmic_urls() (safe to use as a multiprocessing worker).

Return type:

List[str]

cosmic_crunch.fetch.crawl_year_urls(*args, **kwargs)[source]

Retry-wrapped, pickleable wrapper around _crawl_year_urls().

Return type:

List[str]

cosmic_crunch.fetch.crawl_date_urls(*args, **kwargs)[source]

Retry-wrapped, pickleable wrapper around _crawl_date_urls().

Return type:

List[str]

cosmic_crunch.fetch.crawl_format_urls(*args, **kwargs)[source]

Retry-wrapped, pickleable wrapper around _crawl_format_urls().

Return type:

List[str]

cosmic_crunch.fetch.crawl_data_urls(*args, **kwargs)[source]

Retry-wrapped, pickleable wrapper around _crawl_data_urls().

Return type:

List[str]

cosmic_crunch.fetch.download_data_file(*args, **kwargs)[source]

Retry-wrapped, pickleable wrapper around _download_data_file().

Return type:

None

cosmic_crunch.fetch.crawl_site(processes=1, year_url_regex=None, date_url_regex=None)[source]

Crawl the full instrument tree (root -> postproc -> year -> date -> L2 -> format) and return the URL of every .txt.gz data file found. Raises NoDataFilesFoundError if the crawl finds nothing.

Custom year/date filters are threaded to the pool workers as bound arguments (functools.partial), never as rebound module globals, so they survive every multiprocessing start method (compiled patterns pickle cleanly).

Parameters:
Return type:

List[str]

Converting

cosmic_crunch.convert: convert JPL GENESIS COSMIC data files from gzip-compressed ASCII to the netCDF4 standard.

Ported from the v1 convert_files.py script (v1.3.3). The v1 per-file metadata and AGPL header have been removed; the package is MIT-licensed (see LICENSE) and versioned via cosmic_crunch.__version__.

cosmic_crunch.convert.read_cosmic_ascii_file(filename)[source]

Given the name of or path to a COSMIC ASCII file, this function reads data from the file into a dict of header fields and a dict of pandas.DataFrame`s and returns both `dict objects.

Parameters:

filename (str) – The filename of or path to the data file.

Returns:

The data file header, data, and whether the file is empty.

Return type:

Tuple[dict, dict, bool]

cosmic_crunch.convert.write_cosmic_netcdf4_file(filename, header, data, compress=False, complevel=7)[source]

Given a filename, header data, and a dict of datasets, this function creates a new netCDF4 file of the data.

Parameters:

  • filename (str) – The filename of or path to the data file.

  • header (dict) – The ASCII file header containing metadata about the dataset.

  • data (dict) – A dict of pandas.DataFrame objects of the file data.

  • compress (bool) – Whether to zlib-compress the variables. Lossless.

  • complevel (int) – The zlib compression level (1-9); ignored if not compressing.

Return type:

None

cosmic_crunch.convert.convert_cosmic_file(filename, skip_empty=False, compress=False, complevel=7)[source]

Given the filename of or path to a COSMIC ASCII data file, this function reads the file data and header and writes it to a new netCDF4 file.

Parameters:

  • filename (str) – The filename of or path to a COSMIC ASCII data file.

  • skip_empty (bool) – Whether skip conversion of files whose arrays are empty.

  • compress (bool) – Whether to zlib-compress the output variables. Lossless.

  • complevel (int) – The zlib compression level (1-9); ignored if not compressing.

Returns:

An integer completion code. 0: converted, 1: skipped, 2: error.

Return type:

int

cosmic_crunch.convert.crawl_convert(paths, processes=1, skip_empty=False, compress=False, complevel=7)[source]

Given the path to some COSMIC ASCII data file, this function creates a netCDF4 formatted copy inplace. Given the path to a root directory containing multiple COSMIC ASCII data files, this function crawls the directory, identifies each .txt.gz file, and creates a netCDF4 formatted copy inplace.

Parameters:

  • path (list) – The paths to COSMIC ASCII files or directories of them.

  • processes (int) – The number of multiprocessing workers to use.

  • skip_empty (bool) – Whether skip conversion of files whose arrays are empty.

  • compress (bool) – Whether to zlib-compress the output variables. Lossless.

  • complevel (int) – The zlib compression level (1-9); ignored if not compressing.

  • paths (Iterable)

Returns:

A list of integer completion codes.

Return type:

List[int]