Showing posts with label government. Show all posts
Showing posts with label government. Show all posts

Wednesday, 1 April 2020

Highlights from building NSW Transport's Safer Roads portal

NSW Transport has built a powerful road querying portal and the technology is worth sharing with others.

Awesome road safety data

The NSW Transport's Safer Roads Program have collated awesome metrics for hundreds of road characteristics within our state. These are used to support evidence-based decisions and help apply treatments and fix problems before accidents happen. However, this data is spread across multiple datasets and has required significant technical expertise and time to access and understand it. Our challenge has been to make this data easily accessible, queriable, and presented in meaningful reports for road planners.

Safer roads portal



So we've built a web application to query and present road data. It supports queries such as:
“Find all 80km/hr road sections, with a risk rating between 1 and 3, with roadside obstacles within 10m of the road, along two selected routes in my council area.”
Results are displayed in real-time in a map and charts. Queries can be iteratively refined and improved by the analyst, and then printed as a PDF report.

Technical challenges

The technical obstacles we've faced are worth sharing with others wanting to tackle similar use cases.

Multiple misaligned models

Our users' queries need to access road attributes from multiple sources. It might track the road's centerline in one dataset and track each lane in another. And the roads are segmented differently in each dataset. The spatial queries across multiple layers are very CPU intensive and resulted in unacceptable query times for our state-wide dataset.
We addressed this by creating a master query layer, with roads divided into 100m segments, with each road segment aggregating all attributes from the source datasets.
We think we can improve this approach even further by moving to 100m x 100m map tiles for our query layer. This will make our query layer more tolerant of mismatched source layers and will allow us to integrate point and polygon layers.

Platform hacking

We built upon ESRI's ArcGIS Portal Web App Builder. It allowed us to quickly prototype a map and charts website. However, our use case pushed past the capabilities of ArcGIS Portal (and ESRI’s newer Experience Builder). We adding extensions and pulled in additional open source libraries. Notably:
  • We replaced graphs with the more powerful Chart.js.
  • We needed to support the circular refinement of queries, between both spatial and attribute queries, without re-starting the query. This required switching software to using a Model/View/Controller design paradigm, which also fixed up widget communication mixups we were having.
  • The size of our dataset resulted in significant performance challenges. Initial queries crashed browsers, and first-round optimisation still caused ~ 10 minute response times). However, our data is relatively static, and this has enabled us to introduce database optimisation, tiling, caching and clustering strategies to bring standard query times down to web usability norms.

Open source options

In retrospect, we've realised that we've needed more than the capabilities of Web App Builder, and that the open-source stack of software would have suited us better. It would:
  • Provide the full suite of capabilities we require.
  • Address limitations with our current platform that we are having to work around.
  • Be relatively easy to migrate to, comparable to upgrading to ESRI's latest Javscript API.
  • Align with government open source recommendations.
  • Allow other agencies to deploy our application without license restrictions.
  • Allowed us to scale without license restrictions.
  • Still facilitate integration with our ESRI based applications by using OGC open standards.
Integrating charts into a web mapping portal is something that appears to be missing from the open-source geospatial stack, and is something we could offer up for the greater good. So moving to a fully open source solution is something we are considering for future iterations.

Reusable?

So are there others trying to solve a similar use case who want a copy of our codebase? People interested in collaboration, providing a business case for us to share our code? I suspect so. At the very least, there are our state's regional road authorities, and probably also local government authorities. But I expect our equivalents all around the world would be interested. If you are one of these people, then please reach out to us.

About the author

Cameron Shorter was the geospatial business analysis on the project.

Saturday, 2 November 2019

Scrappy Collaboration

Coordinate changes over time due to tectonic plate movement.
Confession: We broke government conventions, worked beyond our agency’s mandate, attended an international technical committee meeting on personal time, uncovered significant negative impacts for unsuspecting users in Australia’s multi-million dollar mapping program, and retrospectively received glowing praise and thanks.

Welcome to “scrappy collaboration”.

$225 million has been committed to improving Australia’s GPS positioning tenfold, to 5cm accuracy. However technical shortcuts previously embedded in web-mapping software, which ignore tectonic plate movements, are causing meter level map misalignments in web-mapping! This is an international problem which is being exposed by Australia’s advanced mapping program. Needless to say, misaligned maps are confusing for users and unacceptable for Australian mapping agencies.

When NSW Spatial Services discovered the project risks involved, and the minimal existing focus on the problem, we joined other Australian states, and worked beyond our agency’s mandate, to look for a solution.

An interrelated problem cluster
With investigation, we unravelled a cluster of interrelated problems:
  • Some of which had deep technical challenges;
  • With varying implications to diverse user groups and use cases;
  • Which were difficult to understand and explain;
  • With experts regularly talking past each other, each considering a different part of the problem;
  • With multiple holistic strategies possible (each strategy offering significantly different costs and benefits for each stakeholder group);
  • With high impact implications for software, data and users;
  • Requiring national and international collaboration between diverse stakeholders to resolve effectively.
Not my responsibility
The investigation turned out to be a significant time sink, distracting us from our core roles. If following a government playbook, we should have left the responsibility to others: Our software vendor, international standards foundations, national mapping organisations, ...

But we knew we had a moral obligation to step up. Our team understood the overarching problem; had access to geodetic surveyors and software engineers; and had personal connections with international experts and influencers from open-source and standards communities.

A scrappy approach
A complex, interdependent problem cluster has a high barrier to entry. Each stakeholder only understands and can influence a subset of the issues. No one has a business case to solve the greater problem for everyone. This is difficult to solve with traditional top-down management approaches.
Instead, we adopted a scrappy, very personal approach, trading our time, and collected knowledge with collaborators to achieve common goals.
  • We very publicly admitted the embarrassing problems we were facing in technical forums, and asked for help.
  • We shamelessly drew down favours from our friends in open source communities, government agencies, industry and standards organisations.
  • We spent extra effort coordinating contributions, answering questions, and addressing related problems relevant to our collaborators (because that is what friends do).
  • We concisely compiled descriptions of the problems, along with suggested solutions.
  • Joel Haasdyk, our datum modernisation manager and geodetic surveyor, then used personal leave to travel halfway around the world to present recommendations to the standards technical committee. 
When collaborating, we applied many of the hallmark characteristics prevalent in open source communities:
  • Respect the time of your collaborators; be concise in messaging.
  • Lead by example; put in the hard work.
  • Reach out to people; be vulnerable and ask for help.
  • Be clear about what you need and from whom.
  • Care about your collaborators; help solve their problems too.
Tap into your humanity - it has tangible business benefits.

Impact
The impact we achieved with collaborators was apparent from the reports coming from the standards community. Scott Simmons, Executive Director of the OGC Standards Program reported:
“I don’t know if I have ever seen a better and more compelling tag-team describe and lead discussion on a topic than Joel and Roger's presentation on the time-dependent limitations inherent in most OGC spatial standards. I received many emails and face-to-face comments from attendees afterwards stating that it was so valuable to understand the problem and see that there is a solution. Resulting from the presentation, the majority of OGC Standards Working Groups now understand the necessity to update their standards to account for time-dependence of coordinates, with some already initiating updates.”
Future standards development will consider tectonic plate movement and will start to address Australia’s map misalignment problems impacting our datum modernisation program. We are at the start of the journey, there is still plenty more to do, but we now are heading in the right direction.

Counterintuitive solution
So, when faced with complex problems, consider expanding scope rather than constraining it. Consider solving the problems of others as well as your own. Attract the collective wisdom and resources of collaborators and you'll likely find your solutions are more impactful, strategic and sustainable.

About the author
Cameron Shorter has been a Geospatial Business Analyst at NSW Spatial Services.


Friday, 30 August 2019

Time-dependent datum problems

Technical shortcuts taken in mapping systems, which ignore tectonic plate movements, are causing meter level map misalignments, most notably in web-mapping. It is caused by a cluster of interrelated problems, which we attempt to comprehensively catalogue in this article. For a more concise summary, look here.
Caution: This article is full of acronyms and complex geospatial mapping concepts. It is primarily aimed at technical experts in this domain. It provides a snapshot in time, collated from a range of people. It is expected to have inaccuracies and not be up to date with latest thinking.

Main problems

Web-mapping’s illusion of accuracy
  • To date, web-mapping has provided an illusion of accuracy; all map layers have typically been aligned; but they have all been aligned to the WGS 84 epoch (and tectonic plate location) associated with the regional datum. This is being exposed as misaligned maps where a nation creates new national datum(s) as part of datum modernisation programs.
  • Australia is experiencing this problem now, due to an update of the nation datum from GDA94 to GDA2020.
WGS 84 datum
  • WGS 84 Web Mercator is the defacto standard used in web-mapping.
  • WGS 84 has inherent ambiguity which in practice introduces low accuracy:
    • It has multiple realizations (it has been re-defined six times to date).
    • In web-mapping, the dynamic WGS 84 datum is being used as if it were a static.
    • In contrast, GPS positioning delivers point positions in the WGS 84 dynamic datum at the ‘current’ epoch (date).
    • Different epochs of WGS 84 have been adopted, in practice, in different regions of the world.
    • It uses step-wise annual updates rather than being dynamic at any time-scale.
    • … and more
  • Static tiled basemaps around the world provides a major barrier to moving away from the WGS 84 Web Mercator projection, as all existing maps would need to be reprojected, or face misalignment with new datums.
  • Adopting a new web-mapping datum would be costly, requiring extensive retiling of all the world’s tiled maps and stored datasets.
Spatial Standards
  • To accurately describe coordinates, spatial standards describing position in a dynamic environment must accurately describe:
    • The coordinates (e.g. Latitude, Longitude, Height),
    • The datum (reference frame) and
    • The epoch (date) at which the coordinates are expressed. (This is often different to the epoch at which coordinate was first observed!)
  • Epoch is implicitly defined for a static datum (e.g. 1994.0 in GDA94) and needs to be explicitly defined in a dynamic (time-dependent) datum (e.g. ITRF1992@1994.0). Unfortunately, epoch is not incorporated in legacy standards, so updating the way in which dynamic datums are defined and applied must be addressed in the full stack of OGC standards.
  • Even the most widely used of OGC standards, like the Web Map Service (WMS) standard, do not account for time dependence when selecting a dynamic Coordinate Reference System (CRS).
  • File formats such as KML and GeoJSON only support storing coordinates in WGS 84. This needs to be considered if addressing future accuracy requirements.
Communication challenges
Collaborating to solve mapping challenges is hindered by communicating a difficult problem, with complex terminology, to a diverse audience. For example:
  • There is a general lack of use and understanding of ‘accuracy’, and how the choice of transformation and projection can affect accuracy.
  • Authoritative documents use terms differently in different documents, or use multiple terms for similar concepts. For example:
    •  “datum”, “reference frame”, and “CRS”, all have similar meanings.
    • There are several different concepts of epoch which are required to accurately describe coordinates in a dynamic datum. For example, a coordinate can be observed at epoch 2017.21, but recorded or expressed at epoch 2018.5 in a reference frame of ITRF2014.

Action Plan

Urgency
There is a need for urgency because:
  • Australia has 1.8 metre map misalignments now.
  • North America plans to introduce new datums in 2022.
  • High-precision mass-market positioning is just around the corner.
Technically
We need to:
  • Solve web-mapping dynamic/static misalignments.
  • Advance time-dependent mapping.
  • Update standards.
  • Roll into software.
  • Improve world mapping practices and user outcomes.
Community mobilisation
While we have technical challenges, we have a harder outreach and community mobilisation challenge. Success depends on collaboratively bringing software, data and geospatial communities with us. We need core buy-in from:
  • Leading tiled map providers,
  • Spatial data providers,
  • Software venders, and
  • Government mapping agencies.
We need clear messaging that Julie, the GIS officer, and Joe, the graduate software developer, can use to explain the datum modernisation story.

Web-mapping business requirements

Let’s revisit the business requirements of accurate web-mapping to help us assess the viability of current and proposed solutions.
Legend for current support:
  • Supported
  • Partially supported
  • Not supported
Geospatial requirements:
  1. Support high accuracy mapping.
  2. Account for time dependence resulting from tectonic plate movement.
  3. Provide accurate map alignment when displaying map layers from disparate sources.
  4. Support calculation, publishing and application of accuracy metadata.
  5. Datasets must have a nominated datum (reference frame) and epoch (date). Epoch is implicitly defined for a static datum and needs to be explicitly defined in a time-dependent datum.
Usability requirements:
  1. Datasets must be able to be transformed between datums and epochs.
  2. Software applications shall continue to be responsive and performant for users, including for low-spec clients, such as browsers and mobile apps.
  3. Web-mapping services shall continue to scale efficiently to support multiple users.
  4. Users shall continue to be able to save map vector data as static files (such as KML, GML, GeoJSON), and render at a later point in time.
  5. Accuracy information should be embedded in decision workflows.
  6. Spatial expertise required to be learned by software implementers should be minimised.
Software Implementation requirements for web-mapping:
  1. Tiled web-mapping shall continue to be supported to address performance and scaling. In practice, this freezes maps in time.
Derived CRS requirements:
  1. Adopt a CRS, datum and projection for publishing map layers with the following characteristics:
    1. Defined accurately.
    2. Receives accurate transformations from key source CRSs (in particular, from national/regional datums).
    3. Aligned with a conventional fixed epoch (date).
    4. Receives coordinates converted from other epochs.
    5. Applicable at global and local scales.
    6. Near-universally adopted within web-mapping contexts.

Main problems

Web-mapping’s illusion of accuracy
  • To date, web-mapping has provided an illusion of accuracy; all map layers have typically been aligned; but they have all been aligned to the WGS 84 epoch (and tectonic plate location) associated with the regional datum. This is being exposed as misaligned maps where a nation creates new national datum(s) as part of datum modernisation programs.
  • Australia is experiencing this problem now, due to an update of the nation datum from GDA94 to GDA2020.
WGS 84 datum
  • WGS 84 Web Mercator is the defacto standard used in web-mapping.
  • WGS 84 has inherent ambiguity which in practice introduces low accuracy:
    • It has multiple realizations (it has been re-defined six times to date).
    • In web-mapping, the dynamic WGS 84 datum is being used as if it were a static.
    • In contrast, GPS positioning delivers point positions in the WGS 84 dynamic datum at the ‘current’ epoch (date).
    • Different epochs of WGS 84 have been adopted, in practice, in different regions of the world.
    • It uses step-wise annual updates rather than being dynamic at any time-scale.
    • … and more
  • Static tiled basemaps around the world provides a major barrier to moving away from the WGS 84 Web Mercator projection, as all existing maps would need to be reprojected, or face misalignment with new datums.
  • Adopting a new web-mapping datum would be costly, requiring extensive retiling of all the world’s tiled maps and stored datasets.
Spatial Standards
  • To accurately describe coordinates, spatial standards describing position in a dynamic environment must accurately describe:
    • The coordinates (e.g. Latitude, Longitude, Height),
    • The datum (reference frame) and
    • The epoch (date) at which the coordinates are expressed. (This is often different to the epoch at which coordinate was first observed!)
  • Epoch is implicitly defined for a static datum (e.g. 1994.0 in GDA94) and needs to be explicitly defined in a dynamic (time-dependent) datum (e.g. ITRF1992@1994.0). Unfortunately, epoch is not incorporated in legacy standards, so updating the way in which dynamic datums are defined and applied must be addressed in the full stack of OGC standards.
  • Even the most widely used of OGC standards, like the Web Map Service (WMS) standard, do not account for time dependence when selecting a dynamic Coordinate Reference System (CRS).
  • File formats such as KML and GeoJSON only support storing coordinates in WGS 84. This needs to be considered if addressing future accuracy requirements.
Communication challenges
Collaborating to solve mapping challenges is hindered by communicating a difficult problem, with complex terminology, to a diverse audience. For example:
  • There is a general lack of use and understanding of ‘accuracy’, and how the choice of transformation and projection can affect accuracy.
  • Authoritative documents use terms differently in different documents, or use multiple terms for similar concepts. For example:
    • “datum”, “reference frame”, and “CRS”, all have similar meanings.
    • There are several different concepts of epoch which are required to accurately describe coordinates in a dynamic datum. For example, a coordinate can be observed at epoch 2017.21, but recorded or expressed at epoch 2018.5 in a reference frame of ITRF2014.

Main options

Option 1: Do Nothing - Low accuracy web mapping

  • Accept poor accuracy and misaligned maps in web-mapping.
  • This is unacceptable for Australia.
Option 2: Call a spade a spade
  • By convention, acknowledge that WGS 84 is currently being used as an accurate static datum in web-mapping.
  • This will likely be accompanied by defining new EPSG code(s) to recognise the currently implemented datum.
  • We would need to acknowledge that WGS 84, as used by the GPS system, is a different datum entirely.
  • In Australia, we would acknowledge that WGS 84 is aligned with our old GDA94 datum.
Option 3: Adopt a common fixed epoch for web mapping
  • Adopt a common fixed epoch to facilitate static tiled maps of the world. This epoch might be periodically re-baselined into pre-planned “epochs of convenience”.
  • (Currently, maps are tiled into different epochs in different regions.)
  • Note: This would involve retiling all the world’s basemaps to the new common epoch.
  • Note: if committing to retiling all maps, we can consider additional improvements to map tiles, such as moving away from the less accurate spherical assumptions of the earth used by WGS 84 Web Mercator.
Option 4: Adopt dynamic datum
  • Adopt a conventionally agreed datum which is explicitly dynamic and internationally recognised, such as ITRF. This will allow new data to be more easily moved through time back to the common epoch.
  • Note: this would involve retiling all the world’s basemaps to the new datum.
Option 5: Use client based datum/epoch conversion tools
  • Allow users to obtain coordinates in (any) desired datum and/or epoch by transforming (x,y) mouse coordinates on screen, rather than expressing and storing underlying base-maps in multiple datums. (Move Mohammad to the mountain rather than moving the mountain to Mohammad.)
Update spatial standards
  • Almost all options will require updates to standards to cater for time-dependence. For example:
  • Web Services such as WMS, WFS, etc., which accept a dynamic CRS as a parameter, will additionally need to accept a CRS epoch field.
  • Data transfer formats such as KML and GeoJSON which only store coordinates in the WGS 84 datum will need to be extended to support more accurate datum(s).
Clear messaging
  • Investing in clear messaging should help with communication challenges.
  • We should simplify developer and user experiences by abstracting complicated concepts behind simple implementations.

Action Plan

Urgency
There is a need for urgency because:
  • Australia has 1.8 metre map misalignments now.
  • North America plans to introduce new datums in 2022.
  • High-precision mass-market positioning is just around the corner.
Technically
We need to:
  • Solve web-mapping dynamic/static misalignments.
  • Advance time-dependent mapping.
  • Update standards.
  • Roll into software.
  • Improve world mapping practices and user outcomes.
Community mobilisation
While we have technical challenges, we have a harder outreach and community mobilisation challenge. Success depends on collaboratively bringing software, data and geospatial communities with us. We need core buy-in from:
  • Leading tiled map providers,
  • Spatial data providers,
  • Software venders, and
  • Government mapping agencies.
We need clear messaging that Julie, the GIS officer, and Joe, the graduate software developer, can use to explain the datum modernisation story.

Detailed problem list

In this section we provide a “shopping list” of problems which hinder the adoption of high-accuracy mapping. It builds from experiences in implementing Australia’s datum modernisation program and ties into international challenges. It touches on time-dependence, standards, web-mapping, and map misalignment challenges.

1. WGS 84 Datum ambiguity

As WGS 84 is the defacto standard used in web-mapping, it is worth understanding the relationship between projection, datum and CRS:
  • The WGS 84 Web Mercator projection (EPSG::3857), technically called the Pseudo Mercator projection, is based on
  • The WGS 84 Coordinate Reference System (EPSG::4326), which is based on
  • The WGS 84 Datum ensemble (EPSG::6326), which is an ensemble of
  • Six WGS 84 Datum realizations.
Figure: Basis of WGS 84 from EPSG Registry 
As explained by Roger Lott, the formal WGS 84 datum is not unique; it has been updated six times to date. Each update (called a realization) refines the datum’s alignment with the earth to account for improved measurements of the earth’s shape (which is different to tectonic plate movement). Each WGS 84 formal realization is dynamic. I.e. coordinates change with time. To be unambiguous, coordinate metadata needs to include coordinate epoch (date).

Problem 1.1 Non-unique datum:
The WGS 84 datum ensemble (EPSG::6326), represents any or all of the formal WGS 84 realizations, without distinction. It should be considered to be static, with metre level uncertainty.

Problem 1.2 WGS 84 coordinates are stepwise updated

Figure: Stepwise jumps in WGS 84

WGS 84 coordinates for ground stations and satellite ephemerides (and therefore the datum definition) are re-computed each year, mid-year. WGS 84 actually moves in a step-wise fashion. If this is not taken into account, then positioning problems can result. (Refer to ISO 19161-1.)

2. Modernising OGC Standards

To accurately describe coordinates, they must include the coordinates, a datum (reference frame) and epoch (date). Epoch is implicitly defined for a static datum and needs to be explicitly defined in a time-dependent datum.
The base OGC and ISO standards document, OGC Abstract Specification Topic 2: Referencing by coordinates, introduces modern geodetic time-dependent concepts, including:
  • Extension to describe dynamic geodetic reference frames.
  • “Datum ensembles” which allows grouping of related realizations of a reference frame.
  • In a dynamic CRS, the coordinate epoch is stored as an attribute of a set of coordinates, it is not part of the CRS. (A set of coordinates could be points, features, or a dataset.)
Problem 2.1: Coordinates in Dynamic CRS are ambiguous
To date, coordinates which have been stored in a dynamic CRS have been ambiguous, as the observation epoch typically has not been recorded, and epoch is not stored in the CRS either.
This is proposed to be addressed by including epoch with coordinates when using a dynamic CRS.

Problem 2.2 Implementation challenges with time-dependent standards
  • We will need to consider how new time-dependent coordinates can be introduced without breaking backward compatibility with existing standards, software, and datasets.
  • There is likely to be performance and implementability implications depending on whether epoch is tied to the level of a point, feature, or dataset.
These should be considered before continuing with standard definitions.

Problem 2.3: OGC standards which don’t address time dependence
Core OGC standards still don’t address time-dependent coordinate referencing. This includes standards such as the GML format, WMS, WFS and so on.

Problem 2.4 High impact to update software to support time dependence
Changing the core coordinate definitions to support time dependence is likely to have a significant impact on software implementations, and stored datasets.

Problem 2.5: KML standard uses WGS 84
The OGC KML 2.3 standard, initially defined by Google, is used widely for storing map data. By definition, KML stores coordinates in the ambiguous WGS 84 CRS (EPSG::4987), and doesn’t record epoch (date), resulting in dataset inaccuracies.

Problem 2.6: GeoJSON standard uses WGS 84
The current GeoJSON standard (dated August 2016) states:
The coordinate reference system for all GeoJSON coordinates is a geographic coordinate reference system, using the World Geodetic System 1984 (WGS 84) 
WGS 84 is not explicitly defined (by an EPSG code or similar) and could be any of the WGS 84 realisations. Likewise, epoch (date) is not recorded, resulting in further dataset inaccuracies.

Problem 2.7 Non-standard GeoJSON in use
While the GeoJSON standard only allows a CRS of WGS 84, “in the wild” implementations appear to allow implementation of different CRS values (as per earlier drafts from April 2006 of the GeoJSON standard). For instance, the OpenLayers browser based client supports a GeoJSON CRS attribute. See example:

Problem 2.8 Inconsistent reference to CRS
As explained in a Request for Change for content negotiation by CRS, inconsistency in referring to CRS is causing implementation problems:
Currently, many GeoJSON API based implementations support communication of CRS. The GeoJSON standard does not provide a standardized method of negotiating CRS. As a result, each implementation varies with respect to both how a CRS is requested, as well as the method used to communicate which CRS was used to locate the data returned. Most, if not all, implementations use the body of the request or response for communication.
There are three issues with this implementation option. Firstly, the name of the parameters used in requests and responses vary between implementations. For example, the variation of terms used to specify the name of the parameter include, but are not limited to, "SRS" (the acronym for Spatial Reference System) and "CRS". Secondly, the semantics of the value used in requests and responses also varies between implementations. For example, the variation of terms used to specify the name of the value for the World Geodetic System of 1984 include, but are not limited to, "WGS84", "4326", and "EPSG:4326". Finally, the value returned by a service does not necessarily support the ability to dereference the CRS.

3. Publishing static maps using a dynamic datum

Organisations have been inadvertently freezing dynamic maps in time when the maps are published. This prevents the feature’s coordinates from changing over time as should happen for coordinates stored in a dynamic datum.

Problem 3.1: Tiling dynamic datum maps
When maps are tiled, they are frozen in time. As the defacto standard for tiled maps is the dynamic WGS 84 Web Mercator projection, we are creating a static/dynamic map misalignment problem.

Problem 3.2: Publishing and distributing dynamic datum maps
Some organisations have been publishing map data in the ambiguous WGS 84 datum. Also, most web services are configured to allow users to download maps in various datums, including WGS 84. When downloaded, these maps are frozen in time, creating the static/dynamic map misalignment problem. The problem is often exacerbated by technology and data providers who encourage use of WGS 84 to facilitate interoperability with map tiles.

4. Web-mapping has been topologically aligned, but not accurate

To date, web-mapping has typically appeared more accurate than the metre level accuracy associated with the WGS 84 datum ensemble. Map layers have been topologically aligned with each other, creating an illusion of accuracy. However, all layers have all been equally misaligned. This occurs because:
  • National mapping agencies store maps accurately in their region’s official static datum.
  • Until recently, a region’s official datum only had one transformation to the WGS 84 datum.
  • As a time-dependent transformation was not applied, this locked the target datum to a point in time, effectively emulating a static datum.
  • For instance, the transformation from CRS GDA94 (EPSG::4283) to CRS WGS 84 (EPSG::4326), (Transform: EPSG::1150) locks transformed coordinates to the year 1994.0.
Problem 4.1: Coordinates systematically misaligned:
Maps sourced into the WGS 84 datum ensemble are regularly systematically misaligned.

Problem 4.2: Lack of awareness:
The poor accuracy of WGS 84 has been masked from users as they see topologically aligned maps. Effectively, there has been a communication gap between the geospatial community and software developers in understanding this problem.

Problem 4.3: WGS 84 being used as if it were an accurate static datum
In web-mapping, WGS 84 is regularly implemented as an accurate static datum. Datasets are consistently transformed from ‘ground truthed’ maps in national/regional datums via null transformation to WGS 84, resulting in consistently misaligned maps, which create the illusion of accuracy.

Problem 4.4: Misalignment with GPS sourced points
WGS 84 features sourced from current GPS devices (such as mobile phones), will be misaligned with WGS 84 maps transformed from a region’s official static datum. The tectonic plate coordinate shift will be exposed.

Problem 4.5: Different epochs in each region:
WGS 84 uses null transformations to datums at different epochs in different regions. In practice, this means WGS 84 is aligned with different epochs (dates) in different regions. For example:
  • Australia’s epoch of WGS 84 is aligned with GDA94 (EPSG::4283), which is aligned to ITRF92 at epoch 1994.0 (Transform EPSG::1150).
  • USA’s epoch of WGS 84 is aligned to CRS NAD83 (EPSG::4269) from 1986.0 (Transform EPSG::1188) etc.
This will make it difficult for the world to agree to lock in a previously used WGS 84 epoch for web-mapping.

5. Australia’s misaligned web-maps

Australia had defined our static GDA94 datum as coincident to ITRF1992 on 1994-01-01 which was coincident to WGS 84 (G730) at that time, leading to defining a null transformation between GDA94 and WGS 84 (EPSG::1150).
However, due to Australia’s fast-moving tectonic plate (~7cm per year), coordinates in the dynamic WGS 84 datum drift apart from static GDA94 coordinates. In 25 years, this equates to ~ 1.8 metres.
Modern GPS positioning, which uses WGS 84 coordinates, and is projected to have centimetre level accuracy within the next few years, now have metre level misalignment with GDA94 sourced basemaps.
We have “mountain to Mohammed or Mohammed to mountain” options to address this misalignment:
  1. Move basemaps and datum to align with current GPS (WGS 84) positions. (Move mountain to Mohammed).
  2. Adjust coordinates in GPS mobile devices to account for WGS 84 and GDA94 differences. (Mohammed goes to the mountain).
Australia chose option 1, to gazette a new official national datum, GDA2020, which will be coincident with WGS 84 (G1762) and ITRF2014 on 2020-01-01.
Figure: Misalignment of WGS 84 Web Mecator web-maps
when sourced from GDA94 and GDA2020

The following datum transformations have been defined:
  1. GDA94 - GDA2020 (EPSG::8048) accounts for Australia’s drift of ~ 1.8 metres.
  2. GDA94 - WGS 84 (EPSG::1150), a null transformation which assumes GDA94 is the same as WGS 84.
  3. GDA2020 - WGS 84 (EPSG::8450) a null transformation which assumes GDA2020 is the same as WGS 84.
Problem 5.1: Systematic misaligned maps
Combining WGS 84 map layers, sourced from GDA94 and GDA2020 via null transformations, results in the systematic misaligned by ~ 1.8 metres. (Previously these layers would have been aligned because they would have both been transformed from GDA94.)

Problem 5.2: Reverting back to GDA94 in web-mapping
To facilitate backward compatibility with existing WGS 84 based web-mapping services, Australia plan to continue aligning new WGS 84 based web-maps with the old GDA94 instead of with the new GDA2020 (as originally planned).

Problem 5.3: Equating a static and dynamic datum
Defining a static and dynamic datum as equivalent will inevitably lead to inaccuracies which will increase over time, and will need to be addressed in future.

Problem 5.4 Moving datums is costly
Moving to a new national/regional datum to improve accuracy requires significant effort, and in Australia’s case, we discovered that the widespread adoption of web-mapping introduced extra hurdles.
Notably, new coordinate operations, which support moving coordinates between epochs, can be implemented to address the same accuracy business drivers.
These options should be considered by those considering future updates to national/regional datums.

Problem 5.5 GDA2020 - WGS 84 transformation not implemented:
Some software providers (such as ArcGIS server), which is used by many Australian mapping agencies, have not implemented the GDA2020 - WGS 84 (EPSG::8450) transformation, restricting these agencies from supporting it. (Considering our web-mapping misalignment problem, this is probably a good thing.)

6. Reference frames and projections

Problem 6.1: WGS 84 being used instead of modern ITRF
The WGS 84 Reference Frame used in web mapping is not as accurate as the International Terrestrial Reference Frame (ITRF). Compared to WGS 84, ITRF is:
  • The baseline reference frame used in geodesy
  • The reference frame being adopted by national datum modernisation initiatives
  • Addresses time
  • Has more accurate transformations into it.
Future global datum usage should consider moving from WGS 84 to ITRF.

Note 6.2: Spherical instead of Ellipsoidal Datum:
The WGS 84 Web Mercator projection used in web-mapping is based on a sphere, not an ellipsoid. This results in coordinates being distorted by up to tens of kilometers. (Distortions increase the closer to the poles). Further, the projection is non-conformal, which means the amount of distortion in the x axis differs to the y axis. Consequently derived distances, areas, bearings and so on will also be distorted.
If the world moves from WGS 84 to another CRS for web-mapping, along with the global retiling of maps entailed, this would be an opportunity to reconsider moving to an ellipsoidal based projection.

7 Constructs to describe accuracy

  • Transforming coordinates between datums has associated inaccuracies, which differs depending on the transformation path taken, and the transformation path typically is not predetermined.
  • In contrast, converting coordinates from one reference system to another, (e.g. from latitude /longitude to easting /northing) is a mathematical procedure which does not introduce inaccuracy.
  • The lineage of how a dataset’s coordinates are derived is typically not recorded with the dataset.
  • By definition, transformations between datums are assigned an accuracy, but accuracy isn’t assigned to a datum per se.
  • Dataset formats and web service standards typically don’t record or publish accuracy statements.
Problem 7.1 Difficult to determine dataset accuracy
As standard spatial data formats and web service interfaces don’t define attributes for accuracy, it is difficult for users to determine the accuracy of a datasets. Equally, it is difficult to publish a dataset’s accuracy.

Problem 7.2 Inaccuracy introduced during round-trip transformations
There are multiple transformation paths between datums. For instance, transforming between a GDA94 source and GDA2020 target datum can apply:
  • EPSG::8048, Helmert 7 Parameter transformation (1cm accuracy)
  • EPSG::8446, NTv2 Conformal transformation (5cm accuracy)
  • EPSG::8447, NTv2 Conformal and Distortion transformation (5cm accuracy)
Further, transforms can be applied directly (eg: GDA94 -> GDA2020) or via a hub transformation (eg: GDA94 -> WGS 84 hub -> GDA2020). Direct or hub potentially provides different results with different accuracies.
Because transformation lineage is typically not recorded with a dataset, transforming back to a source datum can introduce inaccuracy due to a different reverse transformation being applied.
Note: For proj based implementations: the proj6 library will use the most accurate transformation, if available on the system, otherwise it will fallback to less accurate transformation available, and provide hints on where to download more accurate transformations.

Problem 7.3 Describing accuracy
We don’t have the means to convey to a data administrator that a datum includes inaccurate transformations into it (and hence should be avoided for accurate use cases) - E.g. for the WGS 84 datum ensemble.

Problem 7.4 Lack of awareness of accuracy
There appears to be a general lack of discussion and awareness about these accuracy challenges.

8. Communicating map concepts

Discussion about mapping is hindered by:
  • A complicated technical problem,
  • Being applied to solve multiple use cases, with differing requirements and a variety of implemented solutions.
  • Multiple user groups understanding different facets of the problem, often using differing terminology, and talking past each other.
  • Historical implementations sometimes clouding people’s understanding.
  • Message crafting typically being dominated by highly technical people with minimal access to technical writers, professional educators and communicators.
Problem 8.1 Confusing terminology
Understanding the intricacies of mapping is complicated by terminology which is often misused, overloaded or misunderstood.

Problem 8.2 Confused messaging
Based on questions and misunderstandings seen on email lists, there appears to be a lack of clear messaging and education around projections, datums, coordinate reference systems, transformations, EPSG codes and related concepts. Messaging would benefit from input by communication and education expertise, tailored for key user groups.

Problem 8.3 Abstract out complexity
We should simplify developer and user experiences by abstracting complicated concepts behind simple implementations.

Web-mapping terminology (simplified)

Understanding the intricacies of mapping is complicated by terminology which is often misused, overloaded or misunderstood.
Hopefully, these simplified explanations should help. For official definitions, refer to the OGC Abstract Specification Topic 2: Referencing by coordinates.
  • Datum (more recently called a Reference Frame): A reference frame is a mathematical model of the earth against which features can be represented as coordinates.
    • Within a static datum, a feature’s coordinates are locked in time, or locked to its tectonic plate, so the feature’s coordinates remain the same over time. For example, GDA2020 is a static datum for Australia.
    • Within a dynamic datum, coordinates are fixed to the earth as a whole. As the earth’s tectonic plates move (by a few centimeters a year), a feature’s coordinates also move. Examples include WGS 84 as currently used by Global Positioning Systems (GPS), and the current realization of the International Terrestrial Reference Frame (ITRF2014).
  • Datum Ensemble: A group of related realizations of a Reference Frame (datum) used for lower accuracy applications where differences are insignificant. The WGS 84 datum (EPSG::6326) used in web-mapping is a datum ensemble.
  • Coordinate SystemA set of mathematical rules which specify how coordinates are assigned to points.
  • Coordinate Reference System (CRS): A coordinate system that is related to the real world by a datum. A CRS is typically referenced in Web Service calls by an EPSG code. EPSG::4326 is the defacto standard CRS used in web-mapping and refers to the WGS 84 datum ensemble (EPSG::6326).
  • Epoch: A point in time, as applied to time dependent datums, expressed in decimal years. For example, 2017-03-25 is epoch 2017.23.
Multiple definitions of epoch are commonly used to describe different aspects of a coordinate and a CRS, which can lead to confusion:
    1. Observation epoch: The date at which the observation was made.
    2. Coordinate epoch: The date at which the coordinates are expressed.
    3. Realization epoch: Part of the Indicates in general the date at which a datum was defined.
    4. Reference epoch: The epoch at which time-dependent coordinate transformation parameters are defined, along with their rates of change. (The EPSG database currently calls this “Realization epoch”.)
  • Map projection: Coordinate conversion from the earth’s ellipsoidal coordinate system to a flat plane. WGS 84 Web Mercator (EPSG:3857) is the defacto standard projection used in web-mapping. (Its official name is “Pseudo Mercator”). WGS 84 Web Mercator is projected from the WG84 datum ensemble (EPSG::6326).
  • EPSG Codes: Online database that contains definitions of numerous datums and map projections, along with formulas to translate between them. Each is uniquely identified via an EPSG code.
  • Coordinate operations:
    • Coordinate conversion: Changes coordinates from one coordinate reference system to another coordinate reference system based on the same datum (reference frame).
    • Coordinate transformation: Changes coordinates from one coordinate reference system to another coordinate reference system which is based on another datum (reference frame).
    • Point Motion Operation: Changes coordinates within one coordinate reference system to account for the motion of the point within the CRS over a period of time.

Sunday, 7 May 2017

Building an Open Government Multistakeholder Forum

Background

The Australian government's Department of Prime Minister and Cabinet is asking for community input on the development of an Open Government Multistakeholder Forum: a group that will allow government and the community to ensure the commitments of Australia’s Open Government National Action Plan are realised. The plan's commitments involve strengthening and improving:
  • Transparency and accountability in business,
  • The availability of open data and the digital transformation of government services,
  • Access to government information,
  • Integrity in the public sector, and
  • Public participation and engagement.

Framing the question

Let’s be more specific by defining the question this forum should aim to solve:
How can decision makers make good decisions about complex topics which address community needs and reflect community values, incorporating community engagement and earning community trust?
Unfortunately, the utopian vision of open government has inconvenient practical limitations requiring a healthy dose of pragmatism in order to be effective. The following challenges exist:
  • How do you attract the attention and contributions from people, acknowledging that we are in an era of information overload and competing priorities?
  • How do you facilitate conversations about complex and intricate issues, acknowledging that it requires substantial time to fully understand and then debate concepts?
  • How can you assess comprehensive community opinion on complex subjects, acknowledging that most people don’t have time to engage in the study or debate.
  • How do you avoid inappropriate bias from vested interests, acknowledging that vocal minorities (companies or individuals) may promote self serving principles contrary to the interests of the majority?
  • How do you gain trust from communities in presented information?
  • How do you manage communication overhead from a large community, finding and promoting the best ideas? (Increasing the signal-to-noise ratio).
  • How can people with a good ideas know they will be heard, and hence be encouraged to participate, noting the best ideas for complex subjects will be time consuming to develop and express? (Increase voice-to-signal ratio).
  • How do you efficiently avoid conversations being derailed by inappropriate or disrespectful behaviour? (Don't feed the trolls)

Learn from Open Source

In answering these questions we can learn from the communities behind established Open Source Software projects. Most of the principles behind Open Government are inherited from Open Source best practice, which in turn are based on morals found in gift economies such as family gatherings and local community groups.
Established Open Source communities typically have embraced Merit-ocracy, Do-ocracy, Reciprocity, Respect, Modularity, Pragmatism, Sustainable Practices, and have attracted Strategic Funding. Each of these points should be considered in detail as to how they should be applied in the Open Government context.
Open Source projects typically are started by a small volunteer base. All the team know each other intimately. Leaders organically evolve, based on meritocracy and do-ocracy. If the project successfully grows, it builds a technical base which becomes too big to maintain on volunteer labour alone. The project either hits a glass ceiling, or attracts external funding to resource core activities (such as project coordination). Australian Open Government has grown to a size comparable to large established Open Source projects and should be managed and resourced accordingly.

Answering Specific Questions

Government have suggested a basis for founding an Open Government Multistakeholder Forum and asked for community input. Highlights include:
  • Purpose: [The scope of work described for forum members, covering review, building a reasoned opinion based upon analysis, debating and then explaining opinions as recommendations which would require a non-trivial amount of time for diligent members to complete properly.]
  • Ways of Working: The Forum [will] meet at least every two months, in a location rotated between capital cities.
  • Ways of Working: Forum members [will] not be remunerated, but that its community members be reimbursed by government for reasonable travel costs.
  • Structure: The Forum [will] comprise not more than 16 members ... with equal representation from government ... and the community.
  • Appointment Criteria:  The Forum [should] broadly reflect the [social] diversity of the Australian community. In particular ... women and men should hold at least 40 per cent of positions on the Forum. ...
Question 1: Are there any other functions the Forum should usefully perform?
Question 2: If you do not agree with the single forum model, how do you think another model should work?
Question 3: Are there other ways of working you think the Forum should usefully adopt?
Question 4: Are there any other criteria or guidelines that should inform the appointment of community members to the Forum?

Answers 1-4:
There is a significant mismatch between the apparent workload, the accountability that should exist, and the expectation that leadership should come from unfunded volunteers. This should challenge our underlying human morals and ethics.
  • For forum members to provide valuable leadership, they should allocate substantial time analysing, debating and deciding on issues. This would equate to multiple days per week, which is more than most volunteers have available to them. Note that you want to attract committed people already playing leadership roles in open project(s).
  • As well as being morally questionable, I'd expect that having the Australian government expect forum members to work for free is counter to Australian labour laws.
  • I would expect that nominated government employees would be paid to participate as part of their daily work. This will create a mismatch in effort and influence that can be applied by government employees verses community members.
  • If community members are instead paid to participate by an external employer, one should question likely conflicts of interest.
  • If continuing with unpaid forum membership, the scope of tasks expected of forum members should be reduced to be in line with typical capacity of volunteers. 
  • The "Purpose" for the Multistakeholder Forum should be accompanied by realistic time and task commitment expectations of forum members, such that candidates and the community have a clear understanding of what should be done.
  • As well as aiming for social diversity, the forum would benefit from having a multi-disciplinarian team. 
Question 5: How should a selection panel to recommend Forum appointments be composed?
Question 6: Should nominations to the Forum be published?
Question 7: Having regard to the desirability to appoint Forum members without unnecessary delay, is there a better way to administer the nominations and appointment process?

  • If forum membership is treated as a paid position, then candidate selection should be treated with the same privacy applied to standard job hiring.
  • If membership is unpaid, then a public process could be considered. For instance, each candidate could provide a brief statement about themselves. The community can then vote on potential candidates to help short list candidates. The final selection should go to the selection committee who considers community votes when doing face-to-face interviews.
Question 8: Should appointments to the Forum be staggered?
Question 9: How should any mid-term vacancies be filled?

  • More relevant is "How should the Forum be refreshed?" In order to ensure history from one panel to the next, only a maximum of half the panel should be updated in any election cycle.
Question 10: How can the Forum best hear and respond to the range of community perspectives on open government?

This question requires construction of a well thought out plan. Here are a few memory jolters (to be considered with comments above):
  • Define a clear vision and set of principles which casual members of the community can quickly assess and agree or disagree with.
  • Trace all actions back to these principles.
  • Acknowledge the diversity in your community's level of interest and time commitments and adjust accordingly. 
  • Make use of tools which efficiently capture crowd wisdom. Eg: Allow comments on posts, and allow communities to vote up/down comments based on merit.
  • Respect the time and value of your community. Treat them as partners rather than a survey point. Adopt tools and processes which enables the community to maximise each individual's value of each contribution.

Post Note

This response is based on my background working within Open Source communities, Open Standards communities, activist communities and local communities. Do these ideas make sense to people with different experiences?

Saturday, 26 November 2016

The elusive "Open Business"


Variants presented at:
  • The “Geo-enabling our communities” conference, hosted by the Australian/New Zealand Surveying & Spatial Sciences Institute, Canberra, Australia, 25 November 2016.
  • The International Digital Earth Symposium, Sydney, Australia, 6 April 2017.
  • The "Spatial Information Day" conference, Adelaide, Australia, 11 August 2017.
  • Slides available at: https://docs.google.com/presentation/d/19fwg23g86PKNNuVlH8BmcWRnA9nwL_tgowJOoFPOLdc/edit?usp=sharing 


The Open Source story about creating Free Software sounds a bit like a fairy tale. 
Highly motivated developers, 
joyfully beaver away, 
in the middle of the night,
to create high quality software systems,
which they give away for free.
While this simplistic recount is mostly true, 
it glosses over the many subtle details required to create a successful Open Source project. 

Why do so many people give away so much of their time?
Why are these volunteers so effective?
Why does open source work?
Why has the business world found the open source formula so hard to replicate?

Surprisingly, many of the answers are found of our core morals and ethics.

The question of Open verses Proprietary actually breaks down into a series of sub questions.
  1. Should you use Free Software or Free Data?
  2. Should you design systems using Open Architectures and Open Standards?
  3. Does it make sense to contribute back to communities?
  4. Is there a business case to help lead community initiatives?
  5. And if so, should you help scale community and tap into the world’s collective intelligence?

This is a big topic and we have limited time, so I will focus on some of the key messages, mostly at the “use and implement” end of the continuum.

Lets start by asking why you might use Open Source GIS Software?
If you are starting from scratch, the answer is simple. 
There is a comprehensive stack of mature, widely used and widely supported Open Source Geospatial applications, all available for free.
This is a screenshot from the OSGeo-Live software distribution. 
OSGeo-Live includes 50 of the best geospatial Open Source applications, along with sample data, project overviews, and quickstarts for each application.
Lets look at a few of the more popular applications:

QGIS is a desktop GIS application similar to ArcGIS with comparable features, but it free.

OpenLayers is similar to Google Maps API, or ESRI’s Javascript APIs, also free.

Cesium provides a 3 dimensional globe of the earth, like Google Earth, but free.

GeoServer is a map rendering server, similar to ArcGIS Server.
It is the reference implementation for a number of the OGC standards, and is … free.

PostGIS adds spatial functionality to the Postgres database.
It is comparable in maturity, stability, performance and features to Oracle Spatial and Microsoft SQL Server, except it is … free.

For free data, you can use Open Street Map, and Open Route Map. This data is typically pretty good, and suitable for most use cases, but still not as consistent as datasets such as Google Maps.

Ok, so the software and data can be free, but there is more to applications than just the purchase price.
There is deployment, maintenance, training, support.
And who are you going to call at 2am in the morning if something goes wrong?

And that is where companies like Jirotech, EnterpriseDB and Redhat step in.
They backfill the capabilities of organisations deploying these free applications with enterprise level support and services.

So we have covered the first obvious question, 
“Does open source compete favourably feature-for-feature?” It does.
But we have just started. When considering an organisations’ technical roadmap, there are more reasons for selecting Open strategies.
Lets start by considering some of the characteristics of the digital age.

And the amount of software created is innovating at a similar rate.

Odds are that any software you own will be out-innovated within a year or two.
Your software is not an asset!
Your software is a liability!
It needs to be updated, maintained, and integrated with new systems.
It is technical debt, and you should try to own as little of it as possible.
You can achieve this by purchasing Proprietary Software, by using Software as a Service, or by leveraging Open Source.

Because software is so time consuming to create and so easy to copy, it is excessively prone to monopolies.
This holds true for both proprietary and open source products. A product that becomes a little better than its competitors will attracts users, developers and sponsors, which in turn allows that product to grow and improve quickly, allowing it to attract more users. This highly sensitive, positive feedback leads to successful software projects becoming category killers.
Where Open Source and Proprietary business models differ is how they respond to monopolies. 
Proprietary companies are incentivised to lock out competition and increase prices as much as the market will bear. 
However, the open source licenses are structured such that multiple companies can support the same open source product, so the market self corrects any tendencies toward price-fixing.

This leads us to Vendor Lock-In. 
Vendor Lock-In occurs when replacing a vendor’s product would significantly impacts your business.
It is a significant risk, as vendors then have excessive influence on price and your future technical design options.
There are two key strategies to mitigate against vendor lock-in.
  1. Is to use open source, as multiple vendors can all support the same codebase.
  2. Is to design modular architectures based on open standards. 

Using modular architectures:
  • reduces system complexity,
  • which reduces technical risk,
  • and facilitates sustained innovation.

It means you can improve one module, without impacting the rest of your system.
This helps with maintenance, innovation, and keeping up with latest technologies.

Committing to and sustaining a modular architectures requires continual vigilance and forward thinking, especially when acquiring new systems.
There will always be quick fixes and vendors offering more features if you are prepared to accept a level of lock-in.
You should be considering:
  • Long term maintenance,
  • Ability to integrate with other systems, 
  • Obsolescence,
  • And the cost of a future exit strategy. 


What I’ve described so far is practical, main stream advice.
Using open standards, open source and open data is now promoted in government policies and purchasing guidelines, and can be justified based on sound traditional economics.
But the Open Source culture is not based on traditional economics.

Open Source and Open Data communities are usually founded on gift cultures, and continue to retain the principles of the gift culture in their DNA. 
If you wish to successfully engage with these open communities, 
If you wish to have these communities adopt and maintain your codebase,
It helps to understand and respect these gift cultures.
And this starts by understanding our human desires to do things intrinsically good and valuable.

Which brings us to the topic of motivation. While traditional carrot and stick incentives improve motivation for boring, mechanical type tasks, research has shown it to be counter-productive for higher order thinking, such as creating software.
Dan Pink has collated this motivational research into a compelling book called Drive where
he describes how we humans are wired with deeper and more effective motivations. Namely: …

Autonomy, the desire to be self directed.

Mastery, the urge to get better at stuff.

And purpose, the desire to do something with meaning and importance.
So if we facilitate the collaboration of highly motivated people, with the interconnectedness of the internet, and provide them with creative tools, amazing things happen.

Like:
  • Wikipedia which has displaced Encyclopedia Britannica as the authoritative information source,
  • And Linux which is the dominant operating system in IT service centres,
  • Open Street Map, which provides detailed maps of the entire world,
  • And the OSGeo-Live distribution of Open Source Geospatial Software, a project I’ve been involved in for close to 10 years and which has attracted hundreds of contributors.

So how does this translate to attracting and engaging communities?
Professor Charles Schweik tackled this question. He and his team studied thousands of Open Source projects to identify common characteristics of successful projects, and they came up with some interesting findings. Like:
  • Most successful open source projects are small, with just 1, 2 or 3 developers. This is surprising if your exposure to Open Source is through the media stories which almost exclusively reference large projects such as Linux or Android.
  • Also, most open source projects are abandoned. 5 out of 6 according to Charle's research. 

But this is not a weakness, the low success rate is actually a good thing.
Developers vote with their time, and only great projects survive.

Also, when your developers are also users, wanting to scratch an itch, they are the best qualified to decide what is best for a project.

And when your developers are motivated by Autonomy, Mastery and Purpose, they will be motivated to spend extra time to “Get things right” rather than compromise on quality.


What Charlie's team found from their research was that successful projects usually possess:
  • A clearly defined vision,
  • Clear utility,
  • And leaders who lead by doing.
Then as projects move into a growth phase, successful projects tend to:
  • Attract an active community.
  • Provide fine scaled task granularity, making it easier for people to contribute.
  • And often benefit from attracting financial backing.
Lets expand on this. What attracts community?

Attracting volunteers involves helping maximise the unique, intrinsic value a person can contribute based on their limited time available.
Effectively maximise the usefulness and moral return on effort.

This starts with a clear and compelling vision, inspiring enough that others want to adopt the vision and work to make it happen.
This should be followed by a practical and believable commitment to deliver on the vision. Typically this is demonstrated by delivering a “Minimum Viable Product”. 

Then you need to be in need of help, preferably accepting small modular tasks with a low barrier to entry, and ideally something which each person is uniquely qualified to provide.
If anyone could fix a widget, then maybe someone else will do it. But if you are one of a few people with the skills to do the fixing, then your gift of fixing is so much more valuable, and there is a stronger moral obligation for you to step up.

Attracting collaborators means new ideas, new ideologies, and new visions.
A successful project works out how to balance competing priorities of adding features, retaining quality, remaining sustainable, and staying on task.

As projects mature and increase in complexity, reliance on the experience of core contributors increases.
Eventually core tasks for large projects usually becomes more consuming than can be sustained by volunteers.
At this point, sponsorship really helps.

As an example, I’ll reference the OSGeo-Live project I’ve been involved in.
Ten years ago, the Open Source Geospatial Foundation was a collection of Open Source applications, but lacked consistent marketing and was difficult for new users to navigate and understand. 
So we proposed to package all the applications on a DVD, ready to run, with sample datasets and consistent documentation. This was our vision.
We then created a minimal first version of the distribution, demonstrating our commitment
As some of us were on the organising committee of the next international geospatial Open Source Conference, we committed to hand out the DVD at the conference, creating a targeted marketing pipeline. This provided clear value for the developers we were recruiting. 
Then we provided simple guides on how to write installers and documentation and went to the open source developers saying:
“If you package your application and write documentation, like this…, then you can tap into a targeted marketing pipeline”. This made it easy for developers to provide discrete and uniquely valuable contributions. 
And it worked. We have attracted 100s of volunteers, to package 50+ projects, with documentation translated into over 10 languages, which is updated every 6 months.

Ok, so maybe you might be thinking that giving back to open communities might be noble, worthy, the right thing to do.
But there is no way you’d be able to justify it to management. You wouldn’t be the first to face this dilemma. We regularly help organisations answer various permutations to this question.
The answer typically references “Opportunity Management”.
Opportunity Management is the reverse of Risk Management. However, instead of identifying what could go wrong and putting strategies in place to prevent it, you identify things that could go right, then put strategies in place to help make it happen.
Help an open source community, and the number of users, developer and sponsors will grow, and you will indirectly reap the benefits.

So what have we covered?
  • Software is a liability.
  • Minimise your technical debt.
  • Design modular architectures with Open Standards. 
  • It reduces vendor lock-in, increases maintainability, agility and ability to innovate.
  • There is a breadth of Open Source applications which are feature rich, mature and commercially supported.
  • And there is Open Data available to address many of your use cases.


To take things to the next level, to engage with Open Source communities and tap into their collective creativity, you should re-learn how gift cultures work.
The beautiful part to this is that it involves reconnecting with our inner morals and ethics, and doing the right thing.