Understanding NSIDC DAAC Earthdata Cloud S3 Buckets and Paths

The NSIDC DAAC Earthdata Cloud collections are stored in Amazon Simple Storage Service (Amazon S3), a secure, scalable object storage service. If you plan to work directly with data in the cloud or use command-line tools, it’s helpful to understand how S3 buckets and URLs are structured.

This guide explains:

• How NSIDC DAAC organizes S3 bucket paths
• How to interpret S3 paths, object keys, and prefixes
• The difference between public and protected buckets
• What’s required to access these files

What is a Bucket?

Amazon S3 stores data in buckets, which act like top-level folders or containers. Each bucket holds objects (files) along with their associated metadata, including permissions and descriptive details.

NSIDC DAAC’s Main S3 Buckets

• Protected bucket: nsidc-cumulus-prod-protected
• Public bucket: nsidc-cumulus-prod-public

Buckets contain many datasets, organized into a folder-like hierarchy by product and date.

Important Note: NSIDC DAAC Buckets Are Non-Listable

When working with NSIDC DAAC S3 data, it’s important to know that our S3 buckets are intentionally non-listable. This applies to both public and protected buckets.

What “non-listable” means

In many cloud storage systems, you can run a command (for example, aws s3 ls) to see all of the files in a bucket. For NSIDC DAAC buckets, this operation is disabled by policy. If you try it, you’ll likely see a 403 Permission Denied error.

This does not mean your credentials are broken or that access to the data is blocked—it simply reflects the bucket policy.

Why this matters

Because the buckets are non-listable, tools or scripts that depend on directory-style browsing or recursive syncing will not work. Instead, you'll need to use direct paths to objects instead of trying to browse the bucket. 

Objects: Files Inside a Bucket

An object is a single file in a bucket. Every object has:

• Bucket name – the top-level container (e.g., nsidc-cumulus-prod-protected)
• Object key – the path to the file within the bucket
• Metadata – details such as size, permissions, or tags

Example: Full S3 Path for a SMAP SPL3SMP_E File

s3://nsidc-cumulus-prod-protected/SMAP/SPL3SMP_E/006/2025/09/04/SMAP_L3_SM_P_E_20250904_R19240_001.h5

Tree breakdown: 

s3://nsidc-cumulus-prod-protected/      ← bucket  
 └── SMAP/                              ← prefix (collection name)
     └── SPL3SMP_E/                     ← prefix (product short name)  
         └── 006/                       ← prefix (collection version)  
             └── 2025/                  ← prefix (year)  
                 └── 09/                ← prefix (month)  
                     └── 04/            ← prefix (day)  
                         └── SMAP_L3_SM_P_E_20250904_R19240_001.h5  ← object (file)  

In the full S3 path above, the object key is:

SMAP/SPL3SMP_E/006/2025/09/04/SMAP_L3_SM_P_E_20250904_R19240_001.h5

What is an Object Prefix?

An object prefix is the folder-like portion of the object key that organizes files. S3 does not have real directories, but prefixes make the bucket behave like one.

At NSIDC DAAC, data are organized hierarchically:

1. Mission (e.g., SMAP/, ATLAS/)
2. Product short name (e.g., SPL3SMP_E/, ATL06/)
3. Collection version (e.g., 006/, 007/)
4. Date (year → month → day)
5. File (object) – the actual science granule (e.g., .h5 file)

In the SMAP example above, the prefixes include:

• SMAP/ → mission
• SMAP/SPL3SMP_E/ → product
• SMAP/SPL3SMP_E/006/ → version
• SMAP/SPL3SMP_E/006/2025/ → year
• SMAP/SPL3SMP_E/006/2025/09/ → month
• SMAP/SPL3SMP_E/006/2025/09/04 → day

Why prefixes matter:

• Organization –  unsures data are grouped logically by mission, product, version, and time
• Filtering –  tools like AWS CLI, boto3, and s3fs can list objects by prefix so you only see what you need.
• Efficiency –  avoids scanning the entire bucket when querying data subsets

Anatomy of an S3 Path

General format:

s3://bucket-name/prefixes/filename.ext

S3 vs HTTPS URLs

Every file in Earthdata Cloud can be referenced in two ways:

Both URLs point to the same file, but differ in workflow.

• S3 URLs are optimized for cloud-native workflows and large-scale automation.
• HTTPS URLs are simpler for downloading files locally but still require Earthdata Login.

For a quick guide on when to use S3 vs HTTPS, see: https://nsidc.org/data/user-resources/help-center/creating-text-files-https-and-s3-urls-earthdata-cloud-data-access

Public vs. Protected Buckets

The main differences between the public and protected buckets are in their contents and the credentials required for access.

Comparison

Key Takeaways

• Buckets are the top-level containers in Amazon S3.
• Objects are individual files identified by their bucket name and object key.
• Prefixes organize data logically within a bucket.
• Use S3 URLs for cloud workflows and automation, HTTPS URLs for simple downloads.
• The public bucket provides metadata and preview images without authentication.
• The protected bucket holds science data files and requires Earthdata Login credentials.
• NSIDC DAAC buckets are non-listable - you cannot browse them with ls or sync, but direct object access using known S3 paths will work. 

Last Updated: Sep 2025