← All projects

Project

ncarnate — Legacy Scientific Data to Modern netCDF4

I rebuilt ncarnate to convert supported HDF4 and netCDF data into netCDF4, verify stored values against the source, and reconstruct coordinates for supported satellite grids and swaths. The Python CLI and library are published on PyPI and conda-forge.

Code ↗ (opens in a new tab)Article →Docs & install ↗ (opens in a new tab)

Inside a conversion

Keep the measurements. Add the coordinates.

Explore the sea-ice file behind the article: compare stored values, inspect a grid cell, and see how ncarnate organizes the output.

AMSR-E · 19 June 2002ncarnate 2.3.0
SourceHDF4 / HDF-EOS2Measurements + projection metadata
ConvertednetCDF4Measurements + explicit coordinates

964,416 values compared · all equal
Full Arctic and Antarctic daily concentration arrays, checked against the original HDF4 file.

Sampled Arctic sea-ice grid: light cells contain more ice, dark blue is open water, gray is land, and purple marks missing data.

Inspect a cell

The measurements stay the same. ncarnate makes their locations explicit with x, y, latitude, and longitude arrays.

About this conversion

This inspector reads a recorded conversion made with ncarnate 2.3.0. The Python converter does not run in your browser. The two complete daily concentration arrays were compared before extracting this compact sample.

Data: NASA/NSIDC AE_SI12, from The HDF Group’s historical sample. The original and converted files are the same ones used for the article’s figure.

Download the original HDF4 file · Download inspector data · Reproduce the conversion

AMSR_E_L3_SeaIce12km_B02_20020619.hdf

Input: 60.2 MB · Output: 35.5 MB. These are whole-file sizes; the inspector displays two fields.

See both poles on the geographic mapsArctic and Antarctic sea-ice concentration on 19 June 2002, plotted from the converted file. Land is gray; missing data and areas outside coverage are purple.

Maps use the converted x/y coordinates and CF projection. Coastlines: Natural Earth. View full-size image.

Scientific data outlives the tools that made it. A satellite granule written in 2007 still holds real measurements, but it arrives in an aging container format with its geography left implicit in a block of metadata that general-purpose tools don’t reconstruct on their own. ncarnate converts supported files to modern netCDF4 and reconstructs coordinates for supported satellite grids and swaths, so compatible tools such as xarray, QGIS, or Panoply can locate the measurements.

For a hands-on walkthrough with an AMSR-E sea-ice granule, read Introducing ncarnate: converting HDF4 and netCDF to netCDF4, or browse the ncarnate documentation and installation guide.

Context

ncarnate began as a 2020 utility of mine, netcdf4_recompressor — a tool to rewrite a netCDF file with different compression. Two things were wrong with it. Its command line had been broken since it was written: a stray debugging line made it silently do nothing. And it only ever handled netCDF, never the HDF4/HDF-EOS2 files its own documentation claimed to support. In 2026 I rebuilt it around a larger goal than compression — read the legacy earth-science formats and emit netCDF4 that today’s tools can use directly.

The problem

There are two hard parts, and they pull in different directions.

The first is fidelity. Recompressing or converting a scientific file must change how it is stored and nothing about what it says. Packed integers, fill values, and scale factors have to survive; a conversion that quietly re-quantizes a measurement has corrupted the data it claims to preserve.

The second is geolocation. HDF-EOS2 doesn’t store latitude and longitude for every pixel. A grid is described by projection parameters; a swath carries coarse geolocation plus rules for interpolating it. Human-readable it is not, and until those coordinates are made explicit and standard, the file won’t line up on a map in a general-purpose tool.

Approach

The fidelity side is a contract enforced mechanically: converting changes storage, never data. Stored values are preserved value-for-value — integer and packed values bit-for-bit, while floating-point checks treat NaN representations and signed zero as equivalent. Fill values and scale factors are carried across as declarations rather than applied, and every output is checked value-for-value against its source before it is allowed to replace anything — the original is never destroyed on a failed run.

The geolocation side is the harder part. ncarnate parses the HDF-EOS2 StructMetadata and reconstructs CF-convention coordinates. Grid projections (polar-stereographic, geographic, and the EASE-Grid equal-area projection) become CF grid mappings plus one-dimensional x/y and two-dimensional lat/lon, inverse-projected through PROJ via pyproj. Swath geolocation is attached as CF coordinates, and where a swath stores its geography at coarser resolution than its data (a 5 km grid under a 1 km field), the missing coordinates are interpolated — through three-dimensional Cartesian space rather than raw latitude and longitude, so the result stays correct across the antimeridian and near the poles, which is exactly where polar-orbiting satellites spend their time.

To check the reconstruction against something other than itself, I compared its output for an AMSR-E sea-ice granule against The HDF Group’s own independent conversion of the same file; the coordinates agree to about a hundred-thousandth of a degree.

Outcome

ncarnate is live on PyPI under the MIT license — pip install ncarnate, CLI and library both. It converts supported HDF4/HDF-EOS2 and netCDF files to recompressed netCDF4 and adds CF coordinates for supported satellite grids and swaths. The format-support notes describe the supported cases and limitations.

It’s also on conda-forge — conda install -c conda-forge ncarnate. Conda-forge supplies the native HDF4, netCDF, and PROJ libraries together, and the recipe went through a conda-forge maintainer’s review before it was merged. With pip, no separate native-library installation is needed when binary wheels are available for your platform and Python version; if pip needs to build pyhdf from source, you will need the HDF4 library and build tools — see the installation guidance.

Skills & tools