Understanding NSIDC DAAC Earthdata Cloud S3 Buckets and Paths

The NSIDC DAAC Earthdata Cloud collections are stored in Amazon Simple Storage Service (Amazon S3), a secure, scalable object storage service. If you plan to work directly with data in the cloud or use command-line tools, it’s helpful to understand how S3 buckets and URLs are structured.

This guide explains:

How NSIDC DAAC organizes S3 bucket paths
How to interpret S3 paths, object keys, and prefixes
The difference between public and protected buckets
What’s required to access these files

What is a Bucket?

Amazon S3 stores data in buckets, which act like top-level folders or containers. Each bucket holds objects (files) along with their associated metadata, including permissions and descriptive details.

NSIDC DAAC’s Main S3 Buckets

Protected bucket: nsidc-cumulus-prod-protected
Public bucket: nsidc-cumulus-prod-public

Buckets contain many datasets, organized into a folder-like hierarchy by product and date.

Objects: Files Inside a Bucket

An object is a single file in a bucket. Every object has:

Bucket name – the top-level container (e.g., nsidc-cumulus-prod-protected)
Object key – the path to the file within the bucket
Metadata – details such as size, permissions, or tags

Example: Full S3 Path for a SMAP SPL3SMP_E File

s3://nsidc-cumulus-prod-protected/SMAP/SPL3SMP_E/006/2025/09/04/SMAP_L3_SM_P_E_20250904_R19240_001.h5

Tree breakdown:

s3://nsidc-cumulus-prod-protected/      ← bucket  
 └── SMAP/                              ← prefix (collection name)
     └── SPL3SMP_E/                     ← prefix (product short name)  
         └── 006/                       ← prefix (collection version)  
             └── 2025/                  ← prefix (year)  
                 └── 09/                ← prefix (month)  
                     └── 04/            ← prefix (day)  
                         └── SMAP_L3_SM_P_E_20250904_R19240_001.h5  ← object (file)

In the full S3 path above, the object key is:

SMAP/SPL3SMP_E/006/2025/09/04/SMAP_L3_SM_P_E_20250904_R19240_001.h5

What is an Object Prefix?

An object prefix is the folder-like portion of the object key that organizes files. S3 does not have real directories, but prefixes make the bucket behave like one.

At NSIDC DAAC, data are organized hierarchically:

Mission (e.g., SMAP/, ATLAS/)
Product short name (e.g., SPL3SMP_E/, ATL06/)
Collection version (e.g., 006/, 007/)
Date (year → month → day)
File (object) – the actual science granule (e.g., .h5 file)

In the SMAP example above, the prefixes include:

SMAP/ → mission
SMAP/SPL3SMP_E/ → product
SMAP/SPL3SMP_E/006/ → version
SMAP/SPL3SMP_E/006/2025/ → year
SMAP/SPL3SMP_E/006/2025/09/ → month
SMAP/SPL3SMP_E/006/2025/09/04 → day

Why prefixes matter:

Organization – unsures data are grouped logically by mission, product, version, and time
Filtering – tools like AWS CLI, boto3, and s3fs can list objects by prefix so you only see what you need.
Efficiency – avoids scanning the entire bucket when querying data subsets

Anatomy of an S3 Path

General format:

s3://bucket-name/prefixes/filename.ext

Part	Description
`bucket-name`	Top-level container. (e.g., `nsidc-cumulus-prod-protected`
`prefixes`	Directory-like structure organizing datasets by collection, version, date, etc.
`filename.ext`	The actual file/object (science file)

S3 vs HTTPS URLs

Every file in Earthdata Cloud can be referenced in two ways:

URL Type	Example	When to Use
S3 URL	`s3://nsidc-cumulus-prod-protected/SMAP/SPL3SMP_E/006/2025/09/04/SMAP_L3_SM_P_E_20250904_R19240_001.h5`	For use with AWS CLI, SDKs, or Python tools like `s3fs` or `boto3`). Requires Earthdata Login authentication and temporary AWS credentials.
HTTPS URL	`https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/SMAP/SPL3SMP_E/006/2025/09/04/SMAP_L3_SM_P_E_20250904_R19240_001.h5`	Use in a browser or with `wget`/`curl`. Requires Earthdata Login credentials (username/password or `.netrc` configuration).

Both URLs point to the same file, but differ in workflow.

S3 URLs are optimized for cloud-native workflows and large-scale automation.
HTTPS URLs are simpler for downloading files locally but still require Earthdata Login.

For a quick guide on when to use S3 vs HTTPS, see: https://nsidc.org/data/user-resources/help-center/creating-text-files-https-and-s3-urls-earthdata-cloud-data-access

Public vs. Protected Buckets

The main differences between the public and protected buckets are in their contents and the credentials required for access.

Comparison

Feature	Public Bucket (`nsidc-cumulus-prod-public`)	Protected Bucket (`nsidc-cumulus-prod-protected`)
Contents	- Granule-level metadata (XML), may include DAAC-enriched details (e.g., archive date/time) - Thumbnail/browse images (JPEG) typically generated by NSIDC DAAC as low-resolution previews of science data	- Primary science data files (e.g., HDF5, NetCDF, etc.) - Ancillary files (e.g., DMR++ files) stored alongside science data - Additional related science files delivered with the granule
Purpose	Preview or discover granules before downloading full data	Access and analyze science-quality data
Authentication	Not required	Required (via Earthdata Login → temporary AWS credentials)
Ease of Access	Directly accessible via browser, `curl`, or `wget`	Requires setup of AWS CLI/SDKs or authenticated tools
Typical Use Case	Quickly browse metadata or view preview images	Download, analyze, or process science data sets
Example Path	`s3://nsidc-cumulus-prod-public/ATLAS/ATL06/006/`	`s3://nsidc-cumulus-prod-protected/ATLAS/ATL06/006/`

Key Takeaways

Buckets are the top-level containers in Amazon S3.
Objects are individual files identified by their bucket name and object key.
Prefixes organize data logically within a bucket.
Use S3 URLs for cloud workflows and automation, HTTPS URLs for simple downloads.
The public bucket provides metadata and preview images without authentication.
The protected bucket holds science data files and requires Earthdata Login credentials.