The OGC Environmental Data Retrieval API: simple access to big data
Another step towards the creation of the OGC Environmental Data Retrieval (EDR) API Standard has been completed: the successful testing of some of the API’s capabilities during the OGC Environmental Data Retrieval API Sprint. The virtual event was held from March 18-20, 2020. A final report detailing the EDR API Sprint is now available on GitHub.
Why an Environmental Data Retrieval API?
The EDR API uses current web technologies and practices to enable end-users – or anyone with web development experience – to easily identify and retrieve a subset of data from ‘big data’ stores. The idea is to save those users interested in environmental (or other ‘big’) data from having to transfer and deal with datasets that inevitably contain data concerning areas or time periods that are irrelevant to their interests. Plus, as an API, it does this using an understood and established mode of interaction that comes with a shallow learning curve.
What have we achieved so far?
For the previous two years, the OGC MetOcean Domain Working Group (DWG) and the OGC EDR API Standards Working Group (SWG), have worked with the UK Met Office and NOAA on the development and testing of EDR API implementations. Testing also occurred at the Weather and Oceans APIs Hack held in December, 2018.
Why the OGC Environmental Data Retrieval API Sprint?
Previous to the March Sprint, extensive testing of the EDR API had been undertaken with respect to the extraction of information using points, point time series, and 2/3D polygons. It was the further testing of these capabilities that served as the focal point for the recent Sprint.
|Goals Achieved||Objectives Achieved|
Verify and validate requirements and methods for the query and filter operations of the EDR API
Develop client-side value-added applications that consume data from prototype implementations of the EDR API
Assess compatibility with the OGC API - Features and OGC API - Common standards
Develop prototype functionality for the EDR API through developing and running code
Progress EDR API conformance testing of point, point time series, and 2/3D polygon queries
In the lead up to the Sprint, interested parties were encouraged to generate trackable issues on the EDR API Sprint GitHub repository. At the (virtual) Sprint kickoff on March 18, a number of use cases were presented, and tools and client APIs demonstrated. Additionally, two publicly accessible EDR API implementations from the UK Met Office and US National Weather Service (NWS) were used during the Sprint, allowing validation of the API’s capabilities against real-world implementations.
Outcomes of the Sprint
The Sprint had around two dozen attendees from around the world, representing OGC, government departments, universities, and private companies. Participants made significant progress in advancing the specification across several key functional areas, with a number of GitHub issues being created and discussed throughout the Sprint. As a testament to the participants’ dedication, a number of these issues were still being discussed one month after the Sprint when all tickets were closed out to finalize the EDR API Sprint Final Report.
The final report, available on the EDR API Sprint GitHub, contains the details of what was achieved during the Sprint, including: development of a new EDR API server and client; automatic harvesting of metadata from aggregations of data stores to improve search; implementing trajectory queries against typhoon/hurricane data; putting the EDR API on an Esri REST API image server; a STAC (Spatio-Temporal Asset Catalog) for accessing meteorological real-time data stores in pygeoapi; and more.
Becoming an OGC Standard
After the testing undertaken during the Sprint, the following steps remain in the EDR API’s journey towards adoption as an OGC Standard:
- Generate an Engineering Report on the EDR API Sprint.
- Finalize updates to the current specification and update the existing documentation.
- Start the OGC review and adoption process.
- Conduct future sprints to further test and enhance the candidate standard. Potential topics under consideration include:
- Expanding metadata offerings: consider what metadata frameworks should be used, recommended, or mandated.
- Testing other means of sampling.
- Investigating how to integrate with other APIs (such as OGC API Features or Maps).
- Incorporating the needed security aspects (coordinated with the OGC Security DWG and OGC OWS Common - Security SWG).
- Incorporating a pub/sub mechanism within the EDR API specification.
- Investigating how to align with decision impact groups such as SmartCities and others.
- Present a candidate standard to the OGC Technical Committee during a future OGC Members Meeting.
The OGC EDR API is nearing the end of its journey to adoption as an OGC standard: from its early conception, to its design and development by the members of the Environmental Data Retrieval API Standards Working Group and MetOcean Domain Working Group, through to its subsequent testing across sprints, hacks, and early real-world implementations, the EDR API will soon become the latest of OGC’s web-friendly APIs.
However, like with all OGC Standards, its adoption does not mark the end of its journey: the standard will continue to be refined by OGC’s diverse community of members, while providing people from across the world Findable, Accessible, Interoperable, and Reusable (FAIR) access to environmental big data.
OGC Members that wish to contribute to, or keep abreast of, the development of the Environmental Data Retrieval API are welcome to join the Environmental Data Retrieval API SWG or the EDR API GitHub page. More information on the Sprint (as well as other OGC Sprints) is available on the OGC Sprints page, or on the EDR API Sprint GitHub page. Learn more about OGC’s emerging family of APIs at ogcapi.org.