Earlier in this series, I introduced three common challenges that I frequently encounter in geospatial integration projects. In the previous two posts, I explored the importance of knowing your data and creating standards-based metadata. In this final post, I turn to the data itself, exploring the importance of having structured, standards-based data, while recognizing that this is not always the reality.
When we want to integrate vector data (for example, points, polygons, or lines) from an unstructured format such as a PDF, video, or image, some pre-processing is required. This complicates the Extract-Transform-Load (ETL) process and may lead to information being lost along the way.
However, I have encountered cases where there was no alternative because the data producers could only provide information in those formats. In such cases, Artificial Intelligence (AI) can play a valuable role by helping extract structured elements from unstructured information, using algorithms such as Convolutional Neural Networks (CNNs) for image segmentation.
But even when we are talking about structured formats, not all formats are created equal.
In the image below, I show a ranking of data formats based on their interoperability potential. It is worth noting that these are just illustrative examples rather than a comprehensive list of all the existing vector data formats.
CSV is the least recommended format, even though it is a Standard (RFC 4180). The fact that it does not have a fixed schema makes it difficult to validate the information. This is particularly problematic for geometries, which can be encoded in several ways without providing any information about the Coordinate Reference Systems (CRSs).
Shapefile is a structured, open format; a whitepaper was published in 1998 that describes the format in detail, enabling anyone to implement it (and it is effectively supported by most geospatial tools). However, the specification is not a standard, as it is not developed and maintained by an independent standards organization. As a result, it does not offer the same level of assurance that formal standards do.
The remaining three formats at the higher end of the scale are standards and therefore recommended. GeoJSON and JSON-FG rank higher in terms of interoperability, since they support semantic validation through JSON Schema. JSON-FG, in particular, supports the definition of FeatureTypes, defined by FeatureSchemas.
A few caveats should be considered when selecting standards using this diagram. JSON-FG is a relatively new standard and may not yet be widely adopted, while GeoJSON always assumes the default CRS (WGS84).
Although it is valuable to recommend more interoperable formats, I acknowledge that users will often arrive with different ones. The good thing is that if we are working with structured formats, it is usually straightforward to convert them using tools such as GDAL.
Putting These Recommendations into Practice
For practical guidance, see these guidelines for data formats, filenames, and field names. This guide provides a step-by-step approach to transforming tabular data into a geospatial data format.
