FAQs About NSIDC DAAC's Earthdata Cloud Migration
What is NASA’s Earthdata Cloud?
Earthdata Cloud is a NASA Earth Science Data Systems program that enables new methods of data analysis and distribution, while preserving most existing analysis and distribution methods. Having data in the cloud enables efficient use of large amounts of data collections and collaborative work with these data.
Earthdata Cloud is hosted in Amazon Web Services (AWS), with DAAC tools and services co-located next to the data. Earthdata Cloud creates a common infrastructure across the Earth Science Data and Information System (ESDIS), including the DAACs.
Why is NASA migrating Earth science data to Earthdata Cloud?
With the impending arrival of new, high-data-volume Earth observation missions, NASA’s ability to effectively ingest, process, and archive large amounts of data requires the most cost-effective, flexible, and scalable data-management architectures and technologies. To meet these demands, NASA's Earth Science Data Systems (ESDS) Program is implementing a strategic vision to develop and operate multiple components of NASA's Earth Observing System Data and Information System (EOSDIS) in a commercial cloud environment.
EOSDIS provides end-to-end capabilities for managing NASA Earth science data from satellites, aircraft, in-situ measurements, and other sources, and the migration of EOSDIS data into the Earthdata Cloud, the first and currently the largest cloud project at NASA. This migration benefits users by giving them new ways to access NASA’s collection of Earth science datasets, improves the efficiency of data systems operations, increases user autonomy, maximizes flexibility, and offers shared services and controls. The Earthdata Cloud is a key component of the ESDS Transform to Open Science (TOPS) program, which provides the visibility, advocacy, and community resources to support and enable the shift to open science. TOPS, in turn, is part of NASA’s Open-Source Science Initiative, which promotes the open sharing of software, data, and knowledge (algorithms, papers, documents, ancillary information) as early as possible in the scientific process.
Why is NSIDC DAAC moving data to the cloud?
As the data volume of Earth observation missions continues to increase, NASA’s ability to effectively ingest, process, and archive large amounts of data requires the most cost-effective, flexible, and scalable data management architectures and technologies. To meet these demands, NASA's Earth Science Data Systems (ESDS) Program is implementing a strategic vision to develop and operate multiple components of NASA's Earth Observing System Data and Information System (EOSDIS) in a commercial cloud environment. This migration benefits users by giving them new ways to access NASA’s collection of Earth science data sets, improves the efficiency of data systems operations, increases user autonomy, maximizes flexibility, and offers shared services and controls. As part of this strategy, NASA’s Distributed Active Archive Centers, including NSIDC DAAC, are actively migrating data holdings into the Earthdata AWS Cloud over the next several years.
How is NSIDC DAAC migrating to Earthdata Cloud?
Data sets managed by NSIDC DAAC are being migrated mission by mission to Earthdata Cloud, and this process is expected to take several years. Users can continue to access data using their current and familiar workflows, with the additional option of accessing data in Earthdata Cloud.
Throughout the Earthdata Cloud migration process, NSIDC DAAC will provide resources and user support to help users with both their current workflows, and to transition their workflows to the cloud if they want to do so.
What does migration to Earthdata Cloud mean for data users?
Users will continue to be able to access data with their current workflows, and receive support from NSIDC DAAC on those workflows. Migration of NSIDC DAAC data to Earthdata Cloud is expected to take several years. As data sets are migrated into the Earthdata Cloud, cloud access will become a new option for accessing and working with NSIDC DAAC data, and users will have the opportunity to learn about the benefits and specifics of working in Earthdata Cloud.
If a data set is cloud accessible, you will see a new NASA Earthdata Cloud (AWS S3) option in the Data Access & Tools section of that particular data set landing page. You can learn more about accessing and downloading this data in our Earthdata Cloud Data Access Guide
Users who are ready to adapt their workflows to Earthdata Cloud can take advantage of some benefits beyond what is available with current methods, including:
- Improved Performance - Any user can quickly access and subset a large volume of data, as compared to accessing data and services on-premises
- Freedom from Data Transfers and Data Management - Users do not need to move, store, or manage large volumes of data
- Data co-location - Users can easily work with multiple EOSDIS data sets together.
How will data download change with Earthdata cloud?
Downloading data from the Earthdata Cloud archive to your local computer or storage system is and will continue to be free for users. Accessing the data in the cloud from us-west-2 region is also free. You just need an Earthdata login account and AWS credentials for access. There is no cost associated with these authentication steps.
For more information on how data download will and will not change under the Earthdata Cloud paradigm, see our NASA Earthdata Cloud Data Access Guide
What is “S3 access”?
Data that are migrated to Earthdata Cloud are stored in S3 “buckets,” which are similar to file folders; they store objects which consist of data and its descriptive metadata. S3 access refers to accessing NSIDC DAAC data stored in S3 buckets in the Earthdata Cloud, using your Earthdata Login and AWS credentials, and copying the data to an EC2 instance.
We provide step-by-step instructions of the prerequisites and workflows for S3 access in the “S3 Access to NSIDC DAAC Data” section of the NASA Earthdata Cloud Data Access Guide
What support is available for using data in Earthdata Cloud?
Please see our Earthdata Cloud Data Access Guide, which walks you through the steps needed to find and access NSIDC DAAC Earthdata Cloud data via AWS S3. You can also find this help article in the help section of every data set that’s been migrated to Earthdata Cloud.
If you have referenced the Data Access Guide but still have questions or require more personalized assistance, you can email NSIDC’s User Support Office (USO). They can help with accessing the public S3 bucket using Earthdata Login and AWS credentials and downloading data to an EC2 instance, as well as provide support for common access issues.
Additionally, you can visit the Earthdata Forum, a central online location where data users can interact with subject-matter experts from NASA’s Distributed Active Archive Centers (DAACs). For an introduction to the Earthdata Forum and tips on how to use it, watch the Earthdata Forum webinar.
How do I find data sets that are available in Earthdata Cloud?
You can find an assembled data set list of all NSIDC DAAC data publicly available on Earthdata Cloud in our NASA Earthdata Cloud Collection. This data collection includes access to cloud-based data sets as they’re published, as well as resources to discover, access and use Earthdata Cloud data. The collection will be continually updated as new cloud data and resources become available.
You can also find cloud-based data by going to Earthdata Search and logging in with your Earthdata Login credentials. Select the ‘Available from AWS Cloud’ filter in the sidebar. To narrow down to NSIDC DAAC data products, type ‘NSIDC’ in the search box in the top left corner. This will list all the NSIDC DAAC data products that are currently available in the Earthdata Cloud. You can also apply spatial, temporal or keyword filters to narrow down the list.
Please review the NASA Earthdata Cloud Data Access Guide to see additional options on searching for NSIDC DAAC-managed data in Earthdata Cloud.
How can I find out when a dataset will be available in Earthdata Cloud?
NSIDC DAAC does not currently have a specific timeframe for when particular data sets will be available in Earthdata Cloud.
We will publish data announcements on the website, and send supporting email announcements to users, when Earthdata Cloud data sets are published. You can see the announcements and sign up for them on the NASA Earthdata Cloud Collection page. The collection includes an RSS link that you can use to subscribe to announcements.
How do I access and work with data in the cloud?
There are a few ways to access data in the cloud, but NSIDC DAAC currently provides support for the S3 access method, where you access NSIDC DAAC data stored in S3 buckets in the Earthdata Cloud and copy the data to an EC2 instance. We provide step-by-step instructions of the prerequisites and workflows for S3 access in the “S3 Access to NSIDC DAAC Data” of the NASA Earthdata Cloud Data Access Guide.
Why do I see the same dataset listed twice in some search results?
For datasets available in the cloud, the same dataset is listed twice, one is the data in the operational Earthdata archive and the other is the data in Earthdata Cloud.