cosmic-crunch¶
Download JPL GENESIS COSMIC radio-occultation ASCII data files and convert them to netCDF4.
cosmic-crunch crawls the JPL GENESIS data
archive, downloads the Level-2 ASCII occultation profiles, and (optionally)
converts them into self-describing netCDF4 files.
Problems this solves¶
Reach for this if you are trying to:
Bulk-download COSMIC-1 (FORMOSAT-3) GNSS radio-occultation (RO) data from the JPL GENESIS archive without hand-writing a crawler.
Convert COSMIC / GENESIS Level-2 ASCII occultation profiles to netCDF4 — the profile’s fields (temperature, pressure, refractivity, water vapour, …) as a self-describing, xarray-friendly file.
Resume an interrupted bulk pull — downloads are atomic and skip files already present, so you can just re-run.
Download only the slice you need — filter by instrument, year, or date (year and date by regex) instead of pulling the whole archive.
Site restructure note (2020 → 2026). JPL restructured the GENESIS site: the old crawl root (
/ftp/pub/genesis/glevels) is dead, so v1 of this tool silently “succeeded” while downloading nothing. v2 targets the current roothttps://genesis.jpl.nasa.gov/ftp/glevels/and fails loudly — a crawl that finds nothing exits non-zero instead of pretending to succeed.Mission status. COSMIC-1 (FORMOSAT-3, flight modules FM1–FM6, served here as
cosmic1/–cosmic6/) was decommissioned in 2020, so this is a static archive — the data has stopped changing. (COSMIC-2 is a different mission on a different archive and is out of scope.) The COSMIC program’s primary archive is UCAR’s CDAAC; this tool crawls JPL’s independent GENESIS processing of COSMIC-1, which is a separate archive with its own ASCII format.
Installation¶
pip install cosmic-crunch
This installs the cosmic-crunch command with two subcommands, get and
convert. Python 3.10+ is required.
To hack on the package itself, install from a clone instead:
pip install -e . (or pip install -e ".[test]" to run the test suite).
Usage¶
cosmic-crunch has a single entry point with two subcommands:
cosmic-crunch get # crawl the GENESIS site and download ASCII files
cosmic-crunch convert # convert downloaded ASCII files to netCDF4
cosmic-crunch get¶
usage: cosmic-crunch get [-h] [--base-url BASE_URL] [--logfile LOGFILE]
[--instrument INSTRUMENT] [--year_regex YEAR_REGEX]
[--date_regex DATE_REGEX] [--processes PROCESSES]
[--test] [--netcdf4] [--skip_empty] [--compress]
[--complevel COMPLEVEL]
flag |
description |
|---|---|
|
Override the crawl root. Precedence: flag > |
|
Instrument tree to crawl (substring filter). Defaults to |
|
Download only years matching this regular expression. |
|
Download only dates matching this regular expression. |
|
Worker processes for the |
|
Download a small subset (cosmic1, 2019-01-03, 10 files) as a smoke test. |
|
Convert the downloaded ASCII files to netCDF4 afterward. |
|
Skip converting files whose data arrays are all empty. |
|
Losslessly zlib-compress the netCDF4 output (off by default — see Compression). |
|
zlib level |
Downloads are atomic and resumable: each file is streamed to a .part
temporary and renamed into place only once complete, and files already present
with a matching size are skipped — so an interrupted bulk pull can simply be
re-run.
A successful run resembles:
$ cosmic-crunch get --year_regex=2006 --date_regex=2006-05-02 --netcdf4 --skip_empty --processes=4
Crawling all ./cosmic<#>/postproc: 100%|████████████████████| 6/6 [00:03<00:00, 1.61it/s]
Crawling all ./cosmic<#>/.../<year>: 100%|██████████████████| 6/6 [00:03<00:00, 1.59it/s]
Crawling all ./cosmic<#>/.../<date>: 100%|██████████████████| 3/3 [00:03<00:00, 1.17s/it]
Crawling all ./cosmic<#>/.../L2/<format>: 100%|█████████████| 4/4 [00:04<00:00, 1.09s/it]
Downloading data files: 100%|███████████████████████████████| 20/20 [00:26<00:00, 1.33s/it]
Converting ASCII to netCDF4: 100%|██████████████████████████| 20/20 [00:03<00:00, 6.32it/s]
ASCII to netCDF4 conversion summary:
- Successful conversions: 17
- Skipped conversions: 3
- Conversion errors: 0
- Total number of files: 20
Downloaded files are written under ./jpl_cosmic/<year>/<date>/txt/.
cosmic-crunch convert¶
usage: cosmic-crunch convert [-h] [--logfile LOGFILE] [--processes PROCESSES]
[--skip_empty] [--compress] [--complevel COMPLEVEL]
path [path ...]
Convert one or more ASCII .txt.gz files — or directories of them (crawled
recursively) — to netCDF4. netCDF4 files are written into a sibling nc/
directory (mirroring the txt/ layout), or beside the source file otherwise.
cosmic-crunch convert ./jpl_cosmic/2006/ --skip_empty --processes=4
Use from Python¶
The conversion machinery is importable as a library:
from cosmic_crunch import convert
# Convert a directory tree (or a single file) of ASCII profiles:
convert.crawl_convert(["./jpl_cosmic/2006/"], processes=4)
# Or parse one ASCII file directly into a header dict and a dict of
# pandas DataFrames (one per profile):
header, data, is_empty = convert.read_cosmic_ascii_file(
"20060501_0632co1_g35_2p6.L2.txt.gz"
)
The netCDF4 output nests each profile in a group (see the structure
below), so name the group when reading it back — a bare
xarray.open_dataset(path) shows only the global attributes:
import xarray as xr
profile = xr.open_dataset(
"20060501_0632co1_g35_2p6.L2.nc", group="COSMIC1-Profile"
)
Compression¶
Output is uncompressed by default. --compress turns on lossless zlib
compression (--complevel sets the level, default 7); stored values
round-trip bit-identically either way.
Compression is opt-in because it does not reliably shrink these files. A
COSMIC profile is many small float64 variables, and HDF5’s per-variable
chunk + filter overhead outweighs zlib’s savings on short profiles — enabling
compression can nearly double a small file. It only pays off for long
profiles and bulk archival (measured crossover ≈ 1000 levels). Enable it when
your profiles are large, and measure on a sample before committing an archive.
netCDF4 output structure¶
Each ASCII file becomes one netCDF4 file. The ASCII header becomes global
attributes; each DataType profile becomes a group whose variables are that
profile’s columns, indexed by an Index dimension:
netcdf 20060501_0632co1_g35_2p6.L2 {
// global attributes: ProductCreationTime, ShortName, DataSetID,
// PlatformShortName, Receiver, ... (the ASCII header)
group: COSMIC1-Profile {
dimensions: Index = <n> ;
variables: Height, Lat, Lon, Refractivity, Temperature, Pressure, WV Pressure ;
}
group: ECMWF-Profile {
dimensions: Index = <n> ;
variables: Height, Lat, Lon, Refractivity, Temperature, Pressure, WV Pressure ;
}
}
Missing values in the ASCII data (-9999) are stored as NaN.
Security note¶
v1 parsed header values with eval(), allowing arbitrary code execution from a
malicious or corrupted data file. v2 uses ast.literal_eval with a raw-string
fallback — header parsing can no longer execute code.
Development¶
pip install -e ".[test]"
python -m pytest -q # offline test suite (never touches the network)
ruff check .
License¶
MIT — see LICENSE. Built by Erick Shepherd.