Swimming in Data: OGC’s WaterML 2.0 Quenches New Zealand’s Thirst for Information Integration
By exploiting one of their most precious resources — water — OGC standards helped Kiwis conquer one of their most vexing challenges: disparate data sources
Like so many nations, New Zealand is engaged in a contentious game of environmental tug-of-war. On one side, ecological enthusiasts who want to preserve and protect the country’s lush, otherworldly landscape, which is beloved by visitors and locals alike for its majestic mountains, verdant valleys and bewitching beaches. On the other side, economic opportunists who want to capitalize on the country’s natural resources in order to make New Zealand as productive as it is pretty.
Along with arable land and precious mineral deposits, the rope on which Kiwis on both sides are constantly tugging comprises the country’s most valuable commodity: fresh water.
New Zealand’s abundant water resources — including over 425,000 kilometers of rivers and streams, approximately 4,000 lakes and more than 200 underground aquifers — are a double-edged sword. While protecting them is critical for public health, wildlife conservation, hydropower, recreation and tourism, exploiting them is essential to support livelihoods in exporting industries like dairy, meat, wool, horticulture and forestry.
Although they often butt heads over the best ways to use them, one thing on which all Kiwis can agree is how vital their water resources are in the first place. Which is why their central government in May 2020 announced a major initiative to restore and revitalize New Zealand’s waterways, called Te Mana o te Wai. A term borrowed from New Zealand’s indigenous Māori people, Te Mana o te Wai translates as “The Power of Water” and according to the Ministry for the Environment “reflects the recognition of fresh water as a natural resource whose health is integral to the social, cultural, economic and environmental well-being of communities.” The initiative, which establishes sweeping new water-management rules and regulations, is expected to generate NZ$3.8 billion in benefits by 2050.
Although Te Mana o te Wai is new, New Zealand’s commitment to water stewardship is not. In fact, it’s so baked into Kiwi culture that water often has been an effective conduit through which to pursue other national policies and priorities.
Such was the case in 2009, when New Zealand’s regional authorities began looking for ways to consolidate their disparate and de-centralized publicly-funded data in order to demonstrate their national value proposition. Data centralization was a heavy lift, and achieving it would require buy-in from New Zealand’s 16 regional and unitary councils, each of which collects and maintains its own data stores. To sell the concept across all 16 councils, the regional authorities decided to leverage something they all had in common: an appreciation for water conservation and quality. Led by Horizons Regional Council,which conceived the idea, and in partnership with the governments Ministry for the Environment, it launched the Land, Air, Water Aotearoa (LAWA) website, a centralized web-services platform for water quality information and maps. Developed using the Open Geospatial Consortium’s (OGC) WaterML 2.0 standard, it has made it easier than ever for Kiwis to find and share environmental information while also illustrating — in New Zealand and around the world — the value of federated data management.
Historically, New Zealand’s 16 regional and unitary councils have worked autonomously in a siloed fashion — especially when it comes to data collection and management.
“The real challenge here in New Zealand is that we’ve got 16 fiefdoms that have always worked independently of each other, particularly when it comes to timeseries environmental data, for which they have customized their own implementations to suit their own needs,” says Brent Watson, Manager of Environmental Data at the Horizons Regional Council, which serves nearly 240,000 citizens across over 22,000 square kilometers of territory. “What we’ve seen as a result is a whole series of different naming conventions and location protocols, which looks quite messy when you start collating or federating data.”
Because data across regions is incongruent, it also is vastly underutilized. Not only by local, regional and central government, but also by citizens.
“While we’re mandated by central government and acts of Parliament, it’s the communities we serve that ultimately fund our activities,” Watson explains. “Surfacing data and making it available to the people who are paying for it — enabling them to access information that supports the policy and regulatory frameworks of our organization so they can challenge the way we work, how we’re funded and what our priorities are — is fundamental to the democratic process.”
Against that backdrop of disorganization and opacity, and with a desire for increased knowledge and transparency, the Horizons Regional Council with central government support hatched the idea for LAWA.
Although the Horizons Regional Council conceived the idea in 2009, LAWA didn’t launch until 2014. Governance — not technology — was the principal reason why.
“The biggest barrier wasn’t getting the data. It was getting signoff from the council chief executives and chairs to actually be OK with sharing data,” explains Sean Hodges, Manager of Information Management at the Horizons Regional Council.
In fact, the regional councils launched an early version of LAWA, called LAWNZ, in 2010. All of the data it contained, however, was stored behind a password-protected firewall.
“The council chief executives and chairs weren’t comfortable leaving this information out in public,” Hodges continues. “It took two or three years for them to decide and agree — through many meetings, discussions and expressions of concern — to actually release their data publicly.”
As the project’s main sponsor and champion, the onus was on Horizons Regional Council Chief Executive Michael McCartney to convince his peers that transparency was an asset, not a liability. By exercising patience and persistence, he eventually succeeded, according to Hodges, who says council leadership finally came around; like getting into a hot bath, they just needed time to adjust.
“From a self-interest point of view, they recognized the value in demonstrating [for the public] what councils actually do,” observes Hodges, who says allowing citizens to engage with councils’ data gives them insight into councils’ activities — and, therefore, councils’ merit, which ultimately makes councils’ jobs easier.
Saved by Standards
Once their leadership signed off on the idea of putting their data out in front of the public, the councils set about creating a governance structure, including a way to extract, exchange and amalgamate data from their disparate systems. Initially, the process was manual.
“There was so much data to process that we had two people crunching numbers side by side in a room with no windows for six weeks,” Hodges recalls. “We knew there had to be a better way.”
The Horizons Regional Council and central government’s Ministry for the Environment agreed that the “better way” was to leverage open standards, including OGC Web Services (OWS) and OGC WaterML 2.0, a standard information model for the representation of water observations data, the goal of which is facilitating the exchange of water-related datasets across information systems.
“Therein lay the path to do the technical work, which involved working with the councils to get systems in place to deliver data, and with suppliers to get them to generate data according to the standards,” Hodges explains. “It took several years, but we now have suppliers who all are generally compliant with WaterML 2.0 and delivering payloads in a reasonably consistent way.”
The project — the actual rollout of which took just eight months from approval to delivery — is likely the first nationwide implementation of water resources unified via a common standards framework.
Since LAWA’s launch in 2014, the consistency of which Hodges speaks has unlocked practical benefits for government and citizens alike.
For government, open standards have streamlined data access and analysis in a way that has freed up precious staff resources and catalyzed a new commitment to quality management. “It has prompted a shift in the mindset of the organization,” Watson says. “The energy in our organization shifted from data extraction to quality assurance … It’s not until you see your own data compared against other agencies’ data that you start understanding where the pitfalls in your processes might be.”
Regional government has found LAWA so valuable, in fact, that it’s now evangelizing data federation to local and federal authorities, who are pursuing their own open data projects around not only water quality and environmental resource management, but also other verticals, like transportation.
The benefits are even more tangible for citizens, who can use LAWA for everything from recreation — e.g., checking water quality at their local swimming hole — to business: e.g., assessing the availability of water in a given location.
“There’s now a wealth of data for the public to actually use,” Hodges says. “And in fact, the public is probably in a better position to use this data under a ‘citizen science’ banner than government because citizens are personally vested in their own communities.”
Indeed, informed citizens who live and work in a given place can set their own priorities — and hold government accountable for executing against them — in ways that civil servants who regularly move jobs from one government agency to the next cannot. Especially when it comes to a resource as a critical as water, the result isn’t data for data’s sake, but rather informed, timely decisions that create positive outcomes for individuals, communities and economies.
Concludes Hodges, “Having accessible water data via a global standard — thereby removing any proprietary system dependency — enables and empowers communities to self-inform and make decisions that support their interest in the sustainability of their water resources.”