Knowledge Base

How to access data using an FTP client, the command line, wget or Python

To access data using an FTP client, the command line , wget, or a python script you first need to determine the FTP address associated with the data set of interest. This is needed for the 'Directory' if you use an FTP client, or in the steps below if you use the command line or wget. How to determine the FTP address is outlined below, followed by guidance on how to access the data using different methods.

How to determine the FTP address for a data set

There are two ways to determine the FTP address:

  1. If you subscribed to receive notifications about the data set, this address is provided in your data set registration confirmation email.
     
  2. Alternatively, the FTP links for NOAA data sets follow a pattern:
    ftp://sidads.colorado.edu/DATASETS/NOAA/XXXXXX

    where 'XXXXXX' refers to the NOAA data set ID, which starts with a 'G'. The data set ID can be found in the top left corner above the data set title on the data set landing page, as highlighted by the red box in the screenshot below.

    NOTE:  For data sets that are currently at version 1 then you only need the data set ID. However, for some data sets with a version number higher than 1 you will need to add a '_VX' where X refers to the version number (this does not apply to the Sea Ice Index, see list below). For example, G02202_V4 refers to version 4 of the NOAA/NSIDC Climate Data Record (CDR) data set. I have provided the FTP addresses for some our data sets below, as examples:

    - Sea Ice Index, Version 3: ftp://sidads.colorado.edu/DATASETS/NOAA/G02135
    - MASIE-NH, Version 1: ftp://sidads.colorado.edu/DATASETS/NOAA/G02186
    - IMS Daily Northern Snow and Ice Analysis, Version 1: ftp://sidads.colorado.edu/DATASETS/NOAA/G02156
    - SNODAS, Version 1: ftp://sidads.colorado.edu/DATASETS/NOAA/G02158
    - NOAA/NSIDC CDR of Passive Microwave Sea Ice Concentration, Version 4: ftp://sidads.colorado.edu/DATASETS/NOAA/G02202_V4

How to access data using an FTP Client

To access data using an FTP client (e.g. FileZilla, Cyberduck, FireFTP, etc.), please use the following information as input:
Host Name: sidads.colorado.edu
User Name: anonymous
Password: <your e-mail address>
Directory: /pub/DATASETS/NOAA/XXXXXX
(Copy the FTP address after the host name. 'XXXX’ may include the specific data set directory and subfolders within that.)

How to access data using the command line

To access data using the command line:

  1. Connect to our node by typing the following:
    ftp sidads.colorado.edu
     
  2. On connecting, it will ask you to log in with a Name, type:
    anonymous
     
  3. You should now be logged in, to navigate to the directory that contains the data you require, type:
    cd /DATASETS/NOAA/XXXX
    where 'XXXX' is the specific data set directory and any subfolders within that. For example, if you are looking for SNODAS masked data for January 2018 then you would type the following:
    cd /DATASETS/NOAA/G02158/masked/2018/01_Jan
     
  4. To see all the files in the folder, type:
    ls
     
  5. Set the transfer mode to binary by typing:
    binary
     
  6. To download a single file, type:
    get <filename>
    To download multiple files, type:
    mget <filename1> <filename2> <filename3>
    Or to download all the files in that folder, type:
    mget *
     
  7. To log out, type:
    quit

How to access data using wget

To download a single file from FTP using wget you can use the following command:

wget --ftp-user=anonymous ftp://sidads.colorado.edu/DATASETS/NOAA/<path to NOAA file>

Below is an example for downloading the Sea Ice Index daily data file for the northern hemisphere:

wget --ftp-user=anonymous ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/north/daily/data/N_seaice_extent_daily_v3.0.csv

To download all the files in one directory you can use the following command:

wget --ftp-user=anonymous -r -nd ftp://sidads.colorado.edu/DATASETS/NOAA/<path to NOAA data set directory>

Below is an example for downloading all the files from the northern hemisphere daily data directory for the Sea Ice Index:

wget --ftp-user=anonymous -r -nd ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/north/daily/data

How to access data using Python

The example script below can be used to download all the files in a directory on the FTP server. Copy the script into your own file (e.g. FTP_download.py), modify the first 3 variables as appropriate, then run the script.

#!/usr/bin/env python
# NSIDC
# Sample script to download all the files within one directory on the FTP server
#
# Requires Python3 and the ftplib and os libraries

from ftplib import FTP
import os

### The following 3 variables can be changed ###
# 1. Set the directory you would like to download the files to
destdir='path to directory'

# 2. Set the path to the FTP directory that contains the data you wish to download.
# This example is for the daily northern hemisphere data from the Sea Ice Index
# https://nsidc.org/data/g02135
directory = '/DATASETS/NOAA/G02135/north/daily/data'

# 3. Set the password which will be your email address
password = 'email_address'

############################################
### Don't need to change this code below ###
############################################
# FTP server
ftpdir = 'sidads.colorado.edu'

#Connect and log in to the FTP
print('Logging in')
ftp = FTP(ftpdir)
ftp.login('anonymous',password)

# Change to the directory where the files are on the FTP
print('Changing to '+ directory)
ftp.cwd(directory)

# Get a list of the files in the FTP directory
files = ftp.nlst()
files = files[2:]
print(files)

#Change to the destination directory on own computer where you want to save the files
os.chdir(destdir)

#Download all the files within the FTP directory
for file in files:
    print('Downloading...' + file)
    ftp.retrbinary('RETR ' + file, open(file, 'wb').write)

#Close the FTP connection
ftp.quit()

Last Updated September 2021