google earth engine GEE

rslearn.data_sources.google_earth_engine.GEE

This data source is for ingesting images from Google Earth Engine (GEE).

It must be configured with the name of an ee.ImageCollection on GEE. Each ee.Image in the ee.ImageCollection is treated as a different data source item. A Cloud Storage bucket is also required to store the intermediate outputs from GEE export jobs.

During the prepare stage, it will first export the metadata (geometry and time range) of all ee.Image objects in the ee.ImageCollection. Then it will use this to build an rtree from which prepare requests can be satisfied.

During the ingest stage, it will start export jobs to export images to the bucket. Each worker will start one job and poll until it finishes before proceeding onto the next ee.Image to export. After the export finishes, the resulting GeoTIFF(s) are read and processed into the tile store. Note that export jobs can take several minutes to complete depending on the size of the image.

This data source does support direct materialization, which can greatly speed up materialization for sparse windows. Whereas exporting a 10Kx10K image may take 5000 EECU-seconds (and potentially several minutes), exporting a 256x256 image should take only a few seconds.

Configuration

{
  "class_path": "rslearn.data_sources.google_earth_engine.GEE",
  "init_args": {
    // Required name of the ee.ImageCollection, e.g. "COPERNICUS/S1_GRD".
    "collection_name": "COPERNICUS/S1_GRD",
    // Required name of the GCS bucket to use to store intermediate outputs from export
    // jobs. You could set up lifecycle rules on this bucket to delete outputs after 1
    // day.
    "gcs_bucket_name": "...",
    // Required service account name.
    "service_account_name": "...",
    // Required path to a local file containing the service account credentials.
    "service_account_credentials": "/etc/credentials/gee_credentials.json",
    // Required directory to store rtree index over the exported ee.Image metadata.
    "index_cache_dir": "cache/gee",
    // Optional filters to apply on the ee.ImageCollection. See Sentinel-1 example below.
    // Currently only equality filters are supported.
    "filters": null
  }
}

The available bands depends on the chosen ee.ImageCollection. Here is an example layer configuration for Sentinel-1. The filters match only ee.Image objects where the "transmitterReceiverPolarisation" attribute is ["VV", "VH"] and the "instrumentMode" attribute is "IW".

{
  "sentinel1": {
    "band_sets": [
      {
        "bands": ["VV", "VH"],
        "dtype": "uint16",
        "format": "geotiff"
      }
    ],
    "data_source": {
      "class_path": "rslearn.data_sources.google_earth_engine.GEE",
      "init_args": {
        "collection_name": "COPERNICUS/S1_GRD",
        "dtype": "float32",
        "filters": [
          ["transmitterReceiverPolarisation", ["VV", "VH"]],
          ["instrumentMode", "IW"]
        ],
        "gcs_bucket_name": "YOUR_BUCKET_NAME",
        "index_fname": "cache/sentinel1_index",
        "service_account_credentials": "/etc/credentials/gee_credentials.json",
        "service_account_name": "YOUR_SERVICE_ACCOUNT_NAME"
      },
      "query_config": {
        "max_matches": 1
      }
    },
    "type": "raster"
  }
}