Environmental Data Retrieval API SWG
Little, Chris (UK Met Office)
Yue, Peng (Wuhan University)
Blodgett, Dave (US Geological Survey (USGS))
For more information on the OGC Environmental Data Retrieval API, we recommend reading the following blog post: OGC Environmental Data Retrieval API: simple access to big data.
1. Purpose of the Standards Working Group
The Environmental Data Retrieval API Standards Working Group will standardize several APIs, defined with OpenAPI (Version 3), to retrieve various common data patterns from a relatively persistent data store. The data patterns will include, but are not restricted to, data at a point in space and time, time series at a point, data along a trajectory, which may be 2, 3, or 4 dimensional, and covering a specified polygon or rectangular tile.
The APIs will allow service users to retrieve, in effect, transient resources over a discrete sampling geometry, created by the service in response to a standardized query pattern.
The APIs will provide 'building blocks' allowing the construction of more complex services.
Much environmental data is truly 'Big Data', as the data cannot be readily copied and distributed in reasonable timescales for many uses. By specifying precisely just the data required in a convenient pattern familiar to the user, a service provider can extract and serve the requested data, simplying access for the user. This then hides the service implementation details, while being scaleable both in terms of the underlying data volumes and the number of supported simultaneous users.
By using stable and standardized service APIs based on simple data retrieval patterns, it is envisaged that access and use of data and information will be improved in different domains, including geospatial, facilitating more innovation and value. Lowering the barrier to and extending the reach of environmental data can result in increased use, new science as well as integration with non-traditional information communities.
The standard will be consistent with the strategic direction established by "OGC API - Features - Part 1: Core" and the future OGC API - Common as part of evolving OGC API efforts.
The service APIs will support both traditional and cloud-based approaches to computing and also enable a mix of public and private business models on a 'level playing field'. For example, no one country is capable of supplying weather forecast data at the highest useful resolutions for the whole globe. Therefore a distributed scalable approach is essential, enabling both advanced countries and the Least Developed Countries to contribute to global strategic initiatives of sustainability and development.
The Environmental Data Retrieval API will support and be consistent with the HTTP and HTTPS protocols that are ubiquitous on the Web, and, where available, will use existing industry standards commonly used by developers, rather than geospatial specific resources. For example, security will be addressed by the use of OpenAPI (Version 3), allowing the use of security schemes that are common and familiar to Web developers, such as OAuth2 or JWT.
The API will allow the server to provide details to guide the user of the API, such as allowing the choice of common, web-based, modern encodings such as JSON (GeoJSON/CoverageJSON), though this is not mandated.
The original requirements for this proposed standard were evolved in the Met Ocean Domain Working Group as the Weather on the Web (WotW). However, many of the data patterns are applicable in other environmental sciences, or even other domains. The data retrieval patterns to access weather and climate data have been stable for many years. Such patterns include, but are not restricted to, point data, time-series at a point, vertical profiles, trajectories (whether in 2, 3 or 4 dimensions), polygons, or rectangular tiles from a tileset.
There are two main use cases:
An end user requires some convenient data for a specific, familiar purpose, such as a forecast surface temperature at a point or over an area, or a timeseries of water flow at a river location, or wind speed and direction along a proposed trajectory or route. However the user does not need the full range of data that modern weather or climate predictions, or other Big Data systems, provide.
A service provider of many value-added products and information services needs an architecture that decouples the provision of data to many evolving distributed applications from the underlying stable, operational, centralized bulk data production systems.
The underlying bulk data may be based on regular calculation grids as used in numerical weather prediction or environmental modelling, or may be point-cloud observation based, or other structures.
The planned service APIs will return the requested data in a specified structured payload format, such as JSON and derivatives, but will not be limited to only JSON.
Queries against the service endpoints using the APIs will return all the information that users should need to use the services successfully.
There is no existing OGC standard that directly addresses the above requirements.
The use of the OpenAPI (Version 3) framework is proposed to define the APIs, which is consistent with the "OGC API - Features - Part 1: Core" specification. Previous experiments, proof-of-concepts and prototyping work have confirmed the feasibility of a consistent approach.
Once the SWG is established, a candidate standard will be developed within one year.
The SWG will only standardize data retrieval orientated APIs, not product-based APIs that need business specific logic and rules.
Initially, the SWG will not standardize any data patterns other than: point, timeseries, vertical profile, trajectory, corridor, polygon or rectangular tile. Other common patterns, such as a 2D plume, vertical cross-section, or aggregate patterns, may be considered later.
Initially, the SWG will not standardize data patterns that involve the return of ensemble values or probability distribution functions (the fifth dimension), but envisage that it should be straightforward to generalize to support these use cases.
The SWG will not specify mechanisms for finding data retrieval services and their endpoints.
As the standard will be modular and multi-part, using the concept of core and extensions will allow a customized approach to implementing service APIs by implementors and data/service providers. If a community needs to develop a profile, it should be specified and governed by that community.
The SWG work is based on:
The Met Ocean Domain WG work to specify various Best Practices;
The WCS Met Ocean Application profiles done in the WCS SWG https://portal.opengeospatial.org/files/?artifact_id=81773&version=2 , https://portal.opengeospatial.org/files/?artifact_id=81789&version=1 , https://portal.opengeospatial.org/files/?artifact_id=81778&version=1 ;
The OGC API - Features - Part 1: Core standard https://github.com/opengeospatial/ogcapi-features ;
A Met Ocean DWG Hackathon held in December 2018 in Washington DC, USA;
The OGC Hackathon held in June 2019 in London, England, UK;
The Weather on the Web API Engineering Report https://github.com/opengeospatial/Weather-on-the-Web-ER.
Any relevant published or draft Engineering Reports from the OGC Innovation Program.
A series of APIs will be standardized for different data retrieval patterns. The Met Ocean DWG and other interested parties have already prioritized the patterns at various OGC meetings.
The Met Ocean DWG will also produce a Best Practice document for implementing these APIs within an operational meteorological context.
API to retrieve data values at a specified location altitude and time (x,y,z,t). Several operational versions of this pattern already exist in different countries for several years.
API to retrieve a time series of values at a specified location and height (x,y,z), whether elevation or altitude with a specific vertical CRS. This pattern also has some operational implementations.
API to retrieve a vertical profile of values at a specified location and time (x,y,t).
API to retrieve an array of values across a rectangular area (tile). Operational immplementation of this pattern has started.
API to retrieve a set of values across a polygonal area.
API to retrieve a series of values along a specified trajectory, whether 2,3, or 4 dimension.
API to retrieve a series of values within a 'corridor', that is, a trajectory with a surrounding buffer region along its length.