Knowledge Base

How to access data using an FTP client, the command line, wget or Python

To access data using an FTP client, the command line , wget, or a python script you first need to determine the FTP address associated with the data set of interest. This is needed for the 'Directory' if you use an FTP client, or in the steps below if you use the command line or wget. If you subscribed to receive notifications about the data set, this address is provided in your data set registration confirmation email. Alternatively, it can be determined by clicking on the 'Go to FTP' button on the data set web page's Download Data tab; it's what is shown in the address bar at the top of the window that opens.

FTP Client

To access data using an FTP client (e.g. FileZilla, Cyberduck, FireFTP, etc.), please use the following information as input:
Host Name: sidads.colorado.edu
User Name: anonymous
Password: <your e-mail address>
Directory: /pub/DATASETS/XXXX
(Copy the FTP address after the host name. 'XXXX’ may include the data set product directory and the specific data set directory.)

Command Line

To access data using the command line:

  1. Connect to our node by typing the following:
    ftp sidads.colorado.edu
     
  2. On connecting, it will ask you to log in with a Name, type:
    anonymous
     
  3. You should now be logged in, to navigate to the directory that contains the data you require, type:
    cd /pub/DATASETS/NOAA/XXXX
    where 'XXXX' may include the data set product directory and the specific data set directory. For example, if you are looking for SNODAS masked data for January 2018 then you would type the following:
    cd /pub/DATASETS/NOAA/G02158/masked/2018/01_Jan
     
  4. To see all the files in the folder, type:
    ls
     
  5. Set the transfer mode to binary by typing:
    binary
     
  6. To download a single file, type:
    get <filename>
    To download multiple files, type:
    mget <filename1> <filename2> <filename3>
    Or to download all the files in that folder, type:
    mget *
     
  7. To log out, type:
    quit

wget

To download a single file from FTP using wget you can use the following command:

wget --ftp-user=anonymous ftp://sidads.colorado.edu/DATASETS/NOAA/<path to NOAA file>

Below is an example for downloading the Sea Ice Index daily data file for the northern hemisphere:

wget --ftp-user=anonymous ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/north/daily/data/N_seaice_extent_daily_v3.0.csv

To download all the files in one directory you can use the following command:

wget --ftp-user=anonymous -r -nd ftp://sidads.colorado.edu/DATASETS/NOAA/<path to NOAA data set directory>

Below is an example for downloading all the files from the northern hemisphere daily data directory for the Sea Ice Index:

wget --ftp-user=anonymous -r -nd ftp://sidads.colorado.edu/DATASETS/NOAA/G02135/north/daily/data

Python

The example script below can be used to download all the files in a directory on the FTP server. Copy the script into your own file (e.g. FTP_download.py), modify the first 3 variables as appropriate, then run the script.

#!/usr/bin/env python
# NSIDC
# Sample script to download all the files within one directory on the FTP server
#
# Requires Python3 and the ftplib and os libraries

from ftplib import FTP
import os

### The following 3 variables can be changed ###
# 1. Set the directory you would like to download the files to
destdir='path to directory'

# 2. Set the path to the FTP directory that contains the data you wish to download.
# This example is for the daily northern hemisphere data from the Sea Ice Index
# https://nsidc.org/data/g02135
directory = '/DATASETS/NOAA/G02135/north/daily/data'

# 3. Set the password which will be your email address
password = 'email_address'

############################################
### Don't need to change this code below ###
############################################
# FTP server
ftpdir = 'sidads.colorado.edu'

#Connect and log in to the FTP
print('Logging in')
ftp = FTP(ftpdir)
ftp.login('anonymous',password)

# Change to the directory where the files are on the FTP
print('Changing to '+ directory)
ftp.cwd(directory)

# Get a list of the files in the FTP directory
files = ftp.nlst()
files = files[2:]
print(files)

#Change to the destination directory on own computer where you want to save the files
os.chdir(destdir)

#Download all the files within the FTP directory
for file in files:
    print('Downloading...' + file)
    ftp.retrbinary('RETR ' + file, open(file, 'wb').write)

#Close the FTP connection
ftp.quit()

Last Updated May 2021