WebAPI functionality

Koen Hufkens

2024-08-19

ecmwfr

Programmatic interface to the ‘ECMWF’ web API services. Allows for easy downloads of ECMWF public data.

Setup

Before starting save your provided ECMWF key to your local keychain. In this example I use my R development account with the key hidden using Xs. The package does not allow you to use your key inline in scripts to limit security issues when sharing scripts on github or otherwise.

# set a key to the keychain
wf_set_key(user = "khrdev@outlook.com",
           key = "XXXXXXXXXXXXXXXXXXXXXX",
           service = "webapi")

# you can retrieve the key using
wf_get_key(user = "khrdev@outlook.com")

# the output should be the key you provided
# in this case represented by the fake X string.
# "XXXXXXXXXXXXXXXXXXXXXX"

Alternatively, you can call the wf_set_key() function with only the service argument. The function will then redirect you to the correct login page where you can find your login credentials, which can be copied and pasted into the command prompt.

# set a key to the keychain
wf_set_key(service = "webapi")

Before you can download any data you have to make sure to accept the terms and conditions here: https://apps.ecmwf.int/datasets/licences/general/.

ECMWF Data Requests (Public Data Sets)

To download data use the wf_request() function, together with your email and a request string syntax as documented. Instead of json formatting the function uses a simple R list for all the arguments.

The conversion from a MARS or python based query to the list format can be automated if you use the RStudio based Addin. By selecting and using Addin -> Mars to list (or ‘Python to list’) you dynamically convert queries copied from either ECMWF or CDS based services.

# this is an example of a request
request <- list(stream  = "oper",
                   levtype = "sfc",
                   param   = "167.128",
                   dataset = "interim",
                   step    = "0",
                   grid    = "0.75/0.75",
                   time    = "00",
                   date    = "2014-07-01/to/2014-07-02",
                   type    = "an",
                   class   = "ei",
                   area    = "73.5/-27/33/45",
                   format  = "netcdf",
                   target  = "tmp.nc")

# an example download using fw_request()
# using the above request list()
ncfile <- wf_request(user    = "khrdev@outlook.com",
           request  = request,
           transfer = TRUE,
           path     = tempdir(),
           verbose  = FALSE)

The download operation might take a while. A progress indicator will keep you informed on the status of your request. The download will expire after a default time out of one hour. You will be informed of where to track your job request on the ECMWF website, and provided with a download url which can be used with the wt_transfer() funtion to download the data on a later moment. Keep in mind that you can only have 3 active downloads and 20 job items queued!

If the download is successful you can visualize one layer in the retrieved stack of temperatures at 2 meters using the plotting routine below.

# Open NetCDF file and plot the data
# (trap read error on mac - if gdal netcdf support is missing)
r <- try(terra::rast(ncfile))

if(!inherits(r, "try-error")) {
  terra::plot(r[[1]], main = "2 meter temperature (Kelvin)")
  maps::map("world", add = TRUE)
}

ECMWF mars Requests

The ecmwfr package (wf_request()) also allows to submit mars requests. Note that this requires a member state login or a commercial user account!

The ecmwfr syntax for mars requests is identical to the ones to the public ECMWF API except that one has to specify dataset = "mars". The example below downloads the 2 meter temperature forecast (parameter 167) from the 00 UTC ECMWF HRES run January 22, 2019, only 12 ahead forecast (step). A NetCDF file with a regular 0.125 x 0.125 degree horizontal resolution for a custom area in northern Europe will be downloaded (area). Note that the grid specification is required if format = "netcdf" as grib to NetCDF conversion is only able for regular longitude-latitude grids.

# this is an example of a request
request <- list("dataset" = "mars",
                   "class"   = "od",
                   "date"    = "2019-01-22",
                   "expver"  = "1",
                   "levtype" = "sfc",
                   "param"   = "167.128",
                   "step"    = "0",
                   "stream"  = "oper",
                   "time"    = "12",
                   "type"    = "fc",
                   "grid"    = "0.125/0.125",
                   "area"    = "50/0/30/15",
                   "format"  = "netcdf",
                   "target"  = "mars-data.nc")

# Submit mars request and wait for netcdf file
wf_request(user    = "khrdev@outlook.com",
           transfer = TRUE,
           path     = "~",
           request  = request,
           verbose  = FALSE)

Listing Ancillary data

You can list ancillary data such as user information, datasets and services using the following calls.

# user info
user_info <- wf_user_info(user = "khrdev@outlook.com")
head(user_info)

# services
services <- wf_services(user = "khrdev@outlook.com")
head(services)

# datasets
datasets <- wf_datasets(user = "khrdev@outlook.com")
head(datasets)