Comparing NLDAS-2 and NLDAS-3 Precipitation Forcing Data

Author

Rishi Anand, Siddharth Chaudhary

Published

August 29, 2024

Run this notebook

You can launch this notebook in VEDA JupyterHub by clicking the link below.

Launch in VEDA JupyterHub (requires access)

Learn more

Inside the Hub

This notebook was written on the VEDA JupyterHub and as such is designed to be run on a jupyterhub which is associated with an AWS IAM role which has been granted permissions to the VEDA data store via its bucket policy. The instance used provided 16GB of RAM.

See (VEDA Analytics JupyterHub Access)[https://nasa-impact.github.io/veda-docs/veda-jh-access.html] for information about how to gain access.

Outside the Hub

The data is in a protected bucket. Please request access by emailng aimee@developmentseed.org or alexandra@developmentseed.org and providing your affiliation, interest in or expected use of the dataset and an AWS IAM role or user Amazon Resource Name (ARN). The team will help you configure the cognito client.

You should then run:

%run -i 'cognito_login.py'

About the Data

NLDAS is a widely used land modeling environment that generates estimates of land surface fluxes and states such as soil moisture, snow, and streamflow. These estimates are critical for drought and flood monitoring, water availability and water resource management, climate assessments, and other uses. NLDAS-3 is the next generation version of NLDAS-2, and offers significant improvements such as improved spatial resolution (12.5km to 1km), expanded domain (CONUS to North and Central America), reduced data latency (3.5 days to near real-time), and assimilation of NASA remote sensing data, among others. (see Earthdata VEDA Data Story (https://www.earthdata.nasa.gov/dashboard/stories/nldas)). Please note that the NLDAS-3 precipitation data provided here is a sample dataset still in development, and will not be the final NLDAS-3 product when it is released.

This notebook is intended to visualize and compare the NLDAS-2 and sample NLDAS-3 monthly-averaged precipitation forcing.

Approach

  1. Query metadata from the VEDA STAC API
  2. Query NLDAS-2 and NLDAS-3 Raster data drom the VEDA Raster API
  3. Display the data side-by-side for comparison
import requests
import folium
import folium.plugins
from folium import Map, TileLayer
from pystac_client import Client
import branca
import pandas as pd
import matplotlib.pyplot as plt

Querying the STAC API

# The data can be accessed without credentials via these links
STAC_API_URL = "https://openveda.cloud/api/stac"
RASTER_API_URL = "https://openveda.cloud/api/raster"

# The NLDAS-2 and NLDAS-3 are stored in these collections
collection_id3 = "nldas3"
collection_id2 = "nldas2"
# These lines store the metadata for each collection to allow us to query the STAC easily
collection3 = requests.get(f"{STAC_API_URL}/collections/{collection_id3}").json()
collection2 = requests.get(f"{STAC_API_URL}/collections/{collection_id2}").json()
# This date can be changed and is the date that will be used to generate the maps
date = '2021-02-01'
# This line searches for and stores the NLDAS-3 file information that matches the given date
response3 = requests.post(
    f"{STAC_API_URL}/search",
    json={
        "collections": [collection_id3],
        "query": {"datetime": {"eq": date+"T00:00:00"}},
        "limit": 100,
    },
).json()
items3 = response3["features"]
len(items3)
1
# This line searches for and stores the NLDAS-2 file information that matches the given date
response2 = requests.post(
    f"{STAC_API_URL}/search",
    json={
        "collections": [collection_id2],
        "query": {"datetime": {"eq": date+"T00:00:00"}},
        "limit": 100,
    },
).json()
items2 = response2["features"]
len(items2)
1
# Get the first matching file for each version and save/visualize it
item2 = items2[0]
item3 = items3[0]
item2, item3
({'id': 'nldas2_LIS_HIST_202102',
  'bbox': [-168.98000671647551,
   7.019999961559591,
   -51.93999145246978,
   72.06000091582078],
  'type': 'Feature',
  'links': [{'rel': 'collection',
    'type': 'application/json',
    'href': 'https://staging.openveda.cloud/api/stac/collections/nldas2'},
   {'rel': 'parent',
    'type': 'application/json',
    'href': 'https://staging.openveda.cloud/api/stac/collections/nldas2'},
   {'rel': 'root',
    'type': 'application/json',
    'href': 'https://staging.openveda.cloud/api/stac/'},
   {'rel': 'self',
    'type': 'application/geo+json',
    'href': 'https://staging.openveda.cloud/api/stac/collections/nldas2/items/nldas2_LIS_HIST_202102'},
   {'title': 'Map of Item',
    'href': 'https://staging.openveda.cloud/api/raster/collections/nldas2/items/nldas2_LIS_HIST_202102/map?bidx=1&assets=cog_default&unscale=False&colormap=%7B%221%22%3A+%5B120%2C+120%2C+120%5D%2C+%222%22%3A+%5B130%2C+65%2C+0%5D%2C+%223%22%3A+%5B66%2C+207%2C+56%5D%2C+%224%22%3A+%5B245%2C+239%2C+0%5D%2C+%225%22%3A+%5B241%2C+89%2C+32%5D%2C+%226%22%3A+%5B168%2C+0%2C+0%5D%2C+%227%22%3A+%5B0%2C+143%2C+201%5D%7D&max_size=1024&resampling=nearest&return_mask=True',
    'rel': 'preview',
    'type': 'text/html'}],
  'assets': {'cog_default': {'href': 's3://veda-data-store-staging/EIS/nldas2monthly/nldas2_LIS_HIST_202102.tif',
    'type': 'image/tiff; application=geotiff; profile=cloud-optimized',
    'roles': ['data', 'layer'],
    'title': 'Default COG Layer',
    'description': 'Cloud optimized default layer to display on map',
    'raster:bands': [{'scale': 1.0,
      'nodata': -9999.0,
      'offset': 0.0,
      'sampling': 'area',
      'data_type': 'float32',
      'histogram': {'max': 574.3709106445312,
       'min': -4.244315147399902,
       'count': 11.0,
       'buckets': [75701.0,
        14093.0,
        5868.0,
        2149.0,
        419.0,
        204.0,
        128.0,
        63.0,
        36.0,
        12.0]},
      'statistics': {'mean': 40.024527986379255,
       'stddev': 50.011235463160766,
       'maximum': 574.3709106445312,
       'minimum': -4.244315147399902,
       'valid_percent': 16.90532483552632}}]},
   'rendered_preview': {'title': 'Rendered preview',
    'href': 'https://staging.openveda.cloud/api/raster/collections/nldas2/items/nldas2_LIS_HIST_202102/preview.png?bidx=1&assets=cog_default&unscale=False&colormap=%7B%221%22%3A+%5B120%2C+120%2C+120%5D%2C+%222%22%3A+%5B130%2C+65%2C+0%5D%2C+%223%22%3A+%5B66%2C+207%2C+56%5D%2C+%224%22%3A+%5B245%2C+239%2C+0%5D%2C+%225%22%3A+%5B241%2C+89%2C+32%5D%2C+%226%22%3A+%5B168%2C+0%2C+0%5D%2C+%227%22%3A+%5B0%2C+143%2C+201%5D%7D&max_size=1024&resampling=nearest&return_mask=True',
    'rel': 'preview',
    'roles': ['overview'],
    'type': 'image/png'}},
  'geometry': {'type': 'Polygon',
   'coordinates': [[[-168.98000671647551, 7.019999961559591],
     [-51.93999145246978, 7.019999961559591],
     [-51.93999145246978, 72.06000091582078],
     [-168.98000671647551, 72.06000091582078],
     [-168.98000671647551, 7.019999961559591]]]},
  'collection': 'nldas2',
  'properties': {'proj:bbox': [-168.98000671647551,
    7.019999961559591,
    -51.93999145246978,
    72.06000091582078],
   'proj:epsg': 4326.0,
   'proj:shape': [1626.0, 2926.0],
   'end_datetime': '2021-02-28T00:00:00',
   'proj:geometry': {'type': 'Polygon',
    'coordinates': [[[-168.98000671647551, 7.019999961559591],
      [-51.93999145246978, 7.019999961559591],
      [-51.93999145246978, 72.06000091582078],
      [-168.98000671647551, 72.06000091582078],
      [-168.98000671647551, 7.019999961559591]]]},
   'proj:transform': [0.04000000521668002,
    0.0,
    -168.98000671647551,
    0.0,
    -0.0400000005868765,
    72.06000091582078,
    0.0,
    0.0,
    1.0],
   'start_datetime': '2021-02-01T00:00:00'},
  'stac_version': '1.0.0',
  'stac_extensions': ['https://stac-extensions.github.io/projection/v1.0.0/schema.json',
   'https://stac-extensions.github.io/raster/v1.1.0/schema.json']},
 {'id': 'nldas3_LIS_HIST_202102',
  'bbox': [-168.98000671647551,
   7.019999961559591,
   -51.93999145246978,
   72.06000091582078],
  'type': 'Feature',
  'links': [{'rel': 'collection',
    'type': 'application/json',
    'href': 'https://staging.openveda.cloud/api/stac/collections/nldas3'},
   {'rel': 'parent',
    'type': 'application/json',
    'href': 'https://staging.openveda.cloud/api/stac/collections/nldas3'},
   {'rel': 'root',
    'type': 'application/json',
    'href': 'https://staging.openveda.cloud/api/stac/'},
   {'rel': 'self',
    'type': 'application/geo+json',
    'href': 'https://staging.openveda.cloud/api/stac/collections/nldas3/items/nldas3_LIS_HIST_202102'},
   {'title': 'Map of Item',
    'href': 'https://staging.openveda.cloud/api/raster/collections/nldas3/items/nldas3_LIS_HIST_202102/map?bidx=1&assets=cog_default&unscale=False&colormap=%7B%221%22%3A+%5B120%2C+120%2C+120%5D%2C+%222%22%3A+%5B130%2C+65%2C+0%5D%2C+%223%22%3A+%5B66%2C+207%2C+56%5D%2C+%224%22%3A+%5B245%2C+239%2C+0%5D%2C+%225%22%3A+%5B241%2C+89%2C+32%5D%2C+%226%22%3A+%5B168%2C+0%2C+0%5D%2C+%227%22%3A+%5B0%2C+143%2C+201%5D%7D&max_size=1024&resampling=nearest&return_mask=True',
    'rel': 'preview',
    'type': 'text/html'}],
  'assets': {'cog_default': {'href': 's3://veda-data-store-staging/EIS/nldas3monthly/nldas3_LIS_HIST_202102.tif',
    'type': 'image/tiff; application=geotiff; profile=cloud-optimized',
    'roles': ['data', 'layer'],
    'title': 'Default COG Layer',
    'description': 'Cloud optimized default layer to display on map',
    'raster:bands': [{'scale': 1.0,
      'nodata': -9999.0,
      'offset': 0.0,
      'sampling': 'area',
      'data_type': 'float32',
      'histogram': {'max': 428.0843200683594,
       'min': -0.022795898839831352,
       'count': 11.0,
       'buckets': [217417.0,
        29857.0,
        15254.0,
        7576.0,
        3011.0,
        1122.0,
        488.0,
        77.0,
        9.0,
        2.0]},
      'statistics': {'mean': 30.96841488575868,
       'stddev': 41.008411449510426,
       'maximum': 428.0843200683594,
       'minimum': -0.022795898839831352,
       'valid_percent': 47.08281935307018}}]},
   'rendered_preview': {'title': 'Rendered preview',
    'href': 'https://staging.openveda.cloud/api/raster/collections/nldas3/items/nldas3_LIS_HIST_202102/preview.png?bidx=1&assets=cog_default&unscale=False&colormap=%7B%221%22%3A+%5B120%2C+120%2C+120%5D%2C+%222%22%3A+%5B130%2C+65%2C+0%5D%2C+%223%22%3A+%5B66%2C+207%2C+56%5D%2C+%224%22%3A+%5B245%2C+239%2C+0%5D%2C+%225%22%3A+%5B241%2C+89%2C+32%5D%2C+%226%22%3A+%5B168%2C+0%2C+0%5D%2C+%227%22%3A+%5B0%2C+143%2C+201%5D%7D&max_size=1024&resampling=nearest&return_mask=True',
    'rel': 'preview',
    'roles': ['overview'],
    'type': 'image/png'}},
  'geometry': {'type': 'Polygon',
   'coordinates': [[[-168.98000671647551, 7.019999961559591],
     [-51.93999145246978, 7.019999961559591],
     [-51.93999145246978, 72.06000091582078],
     [-168.98000671647551, 72.06000091582078],
     [-168.98000671647551, 7.019999961559591]]]},
  'collection': 'nldas3',
  'properties': {'proj:bbox': [-168.98000671647551,
    7.019999961559591,
    -51.93999145246978,
    72.06000091582078],
   'proj:epsg': 4326.0,
   'proj:shape': [1626.0, 2926.0],
   'end_datetime': '2021-02-28T00:00:00',
   'proj:geometry': {'type': 'Polygon',
    'coordinates': [[[-168.98000671647551, 7.019999961559591],
      [-51.93999145246978, 7.019999961559591],
      [-51.93999145246978, 72.06000091582078],
      [-168.98000671647551, 72.06000091582078],
      [-168.98000671647551, 7.019999961559591]]]},
   'proj:transform': [0.04000000521668002,
    0.0,
    -168.98000671647551,
    0.0,
    -0.0400000005868765,
    72.06000091582078,
    0.0,
    0.0,
    1.0],
   'start_datetime': '2021-02-01T00:00:00'},
  'stac_version': '1.0.0',
  'stac_extensions': ['https://stac-extensions.github.io/projection/v1.0.0/schema.json',
   'https://stac-extensions.github.io/raster/v1.1.0/schema.json']})
# Save the relevant statistics for each file
item_stats3 = item3['assets']['cog_default']['raster:bands'][0]['statistics']
rescale_values3 = item_stats3['minimum'], item_stats3['maximum']

item_stats2 = item2['assets']['cog_default']['raster:bands'][0]['statistics']
rescale_values2 = item_stats2['minimum'], item_stats2['maximum']
# Query the raster API for the NLDAS-3 tile
tiles3 = requests.get(
    f"{RASTER_API_URL}/collections/{collection_id3}/items/{item3['id']}/tilejson.json?"
    "&assets=cog_default"
    "&color_formula=gamma+r+1.05&colormap_name=rdbu_r"
    f"&rescale={rescale_values3[0]},{rescale_values3[1]}",
).json()
tiles3
{'tilejson': '2.2.0',
 'version': '1.0.0',
 'scheme': 'xyz',
 'tiles': ['https://staging.openveda.cloud/api/raster/collections/nldas3/items/nldas3_LIS_HIST_202102/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?assets=cog_default&color_formula=gamma+r+1.05&colormap_name=rdbu_r&rescale=-0.022795898839831352%2C428.0843200683594'],
 'minzoom': 0,
 'maxzoom': 24,
 'bounds': [-168.98000671647551,
  7.019999961559591,
  -51.93999145246978,
  72.06000091582078],
 'center': [-110.45999908447266, 39.540000438690186, 0]}
# Query the raster API for the NLDAS-2 tile
tiles2 = requests.get(
    f"{RASTER_API_URL}/collections/{collection_id2}/items/{item2['id']}/tilejson.json?"
    "&assets=cog_default"
    "&color_formula=gamma+r+1.05&colormap_name=rdbu_r"
    f"&rescale={rescale_values2[0]},{rescale_values2[1]}",
).json()
tiles2
{'tilejson': '2.2.0',
 'version': '1.0.0',
 'scheme': 'xyz',
 'tiles': ['https://staging.openveda.cloud/api/raster/collections/nldas2/items/nldas2_LIS_HIST_202102/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?assets=cog_default&color_formula=gamma+r+1.05&colormap_name=rdbu_r&rescale=-4.244315147399902%2C574.3709106445312'],
 'minzoom': 0,
 'maxzoom': 24,
 'bounds': [-168.98000671647551,
  7.019999961559591,
  -51.93999145246978,
  72.06000091582078],
 'center': [-110.45999908447266, 39.540000438690186, 0]}

Visualize NLDAS-2 and NLDAS-3

Notice the differences in extent and resolution between the two versions.

# Create and display a DualMap to visualize NLDAS-2 and NLDAS-3 data side-by-side
map_layer2 = TileLayer(
    tiles=tiles2["tiles"][0],
    attr="VEDA",
)
map_layer3 = TileLayer(
    tiles=tiles3["tiles"][0],
    attr="VEDA",
)
map_ = folium.plugins.DualMap(location=[39,-110], zoom_start=3,)
map_layer2.add_to(map_.m1)
map_layer3.add_to(map_.m2)
map_
Make this Notebook Trusted to load map: File -> Trust Notebook