Monday, 11 November 2019

TheGoodDocsProject Fixit workshop

Are you coming to the Write the Docs Conference in Sydney, Australia, 14 and 15 November? If so, I hope you will feel inspired to join our GoodDocsProject FixIt workshop.

We started TheGoodDocsProject to create best-practice templates and writing instructions for documenting open-source software. With our alpha release, we’ve created some common templates which we’d love people to test, break and improve.

Topics:
Diverse contributions encourage. We hope to cover:

  • As a user or developer or writer: Try writing instructions for your favourite application based on one of our templates. Give us feedback.
  • As a technical writer or information architect: Review one of our templates, try and find limitations, suggest modifications, offer sections to add, debate improvements.
  • You have another idea: Let’s hear it.

Schedule:

  • 5 min: Introduction.
  • 30 min: Break out into groups brainstorming topic.
  • 15 min: groups present back to everyone.
  • 30 min: Break out2.
  • 15 min: Groups present2.
  • 5 min: wrapup and thanks

Logistics:


Post Fixit:


Who are we:

  • Cameron Shorter, Business Analyst by day, Open Source Developer and accidental Technical Writer by night.
  • Jared Morgan, Senior Technical Writer at Squiz
  • Erin McKean, Docs Advocacy Program Manager at Google
  • Felicity Brand (attending remotely), Technical Writer for Google Season of Docs
  • Sanket Totewar, Technical Writer and a few more things
  • Others from TheGoodDocsProject

    Saturday, 2 November 2019

    Scrappy Collaboration

    Coordinate changes over time due to tectonic plate movement.
    Confession: We broke government conventions, worked beyond our agency’s mandate, attended an international technical committee meeting on personal time, uncovered significant negative impacts for unsuspecting users in Australia’s multi-million dollar mapping program, and retrospectively received glowing praise and thanks.

    Welcome to “scrappy collaboration”.

    $225 million has been committed to improving Australia’s GPS positioning tenfold, to 5cm accuracy. However technical shortcuts previously embedded in web-mapping software, which ignore tectonic plate movements, are causing meter level map misalignments in web-mapping! This is an international problem which is being exposed by Australia’s advanced mapping program. Needless to say, misaligned maps are confusing for users and unacceptable for Australian mapping agencies.

    When NSW Spatial Services discovered the project risks involved, and the minimal existing focus on the problem, we joined other Australian states, and worked beyond our agency’s mandate, to look for a solution.

    An interrelated problem cluster
    With investigation, we unravelled a cluster of interrelated problems:
    • Some of which had deep technical challenges;
    • With varying implications to diverse user groups and use cases;
    • Which were difficult to understand and explain;
    • With experts regularly talking past each other, each considering a different part of the problem;
    • With multiple holistic strategies possible (each strategy offering significantly different costs and benefits for each stakeholder group);
    • With high impact implications for software, data and users;
    • Requiring national and international collaboration between diverse stakeholders to resolve effectively.
    Not my responsibility
    The investigation turned out to be a significant time sink, distracting us from our core roles. If following a government playbook, we should have left the responsibility to others: Our software vendor, international standards foundations, national mapping organisations, ...

    But we knew we had a moral obligation to step up. Our team understood the overarching problem; had access to geodetic surveyors and software engineers; and had personal connections with international experts and influencers from open-source and standards communities.

    A scrappy approach
    A complex, interdependent problem cluster has a high barrier to entry. Each stakeholder only understands and can influence a subset of the issues. No one has a business case to solve the greater problem for everyone. This is difficult to solve with traditional top-down management approaches.
    Instead, we adopted a scrappy, very personal approach, trading our time, and collected knowledge with collaborators to achieve common goals.
    • We very publicly admitted the embarrassing problems we were facing in technical forums, and asked for help.
    • We shamelessly drew down favours from our friends in open source communities, government agencies, industry and standards organisations.
    • We spent extra effort coordinating contributions, answering questions, and addressing related problems relevant to our collaborators (because that is what friends do).
    • We concisely compiled descriptions of the problems, along with suggested solutions.
    • Joel Haasdyk, our datum modernisation manager and geodetic surveyor, then used personal leave to travel halfway around the world to present recommendations to the standards technical committee. 
    When collaborating, we applied many of the hallmark characteristics prevalent in open source communities:
    • Respect the time of your collaborators; be concise in messaging.
    • Lead by example; put in the hard work.
    • Reach out to people; be vulnerable and ask for help.
    • Be clear about what you need and from whom.
    • Care about your collaborators; help solve their problems too.
    Tap into your humanity - it has tangible business benefits.

    Impact
    The impact we achieved with collaborators was apparent from the reports coming from the standards community. Scott Simmons, Executive Director of the OGC Standards Program reported:
    “I don’t know if I have ever seen a better and more compelling tag-team describe and lead discussion on a topic than Joel and Roger's presentation on the time-dependent limitations inherent in most OGC spatial standards. I received many emails and face-to-face comments from attendees afterwards stating that it was so valuable to understand the problem and see that there is a solution. Resulting from the presentation, the majority of OGC Standards Working Groups now understand the necessity to update their standards to account for time-dependence of coordinates, with some already initiating updates.”
    Future standards development will consider tectonic plate movement and will start to address Australia’s map misalignment problems impacting our datum modernisation program. We are at the start of the journey, there is still plenty more to do, but we now are heading in the right direction.

    Counterintuitive solution
    So, when faced with complex problems, consider expanding scope rather than constraining it. Consider solving the problems of others as well as your own. Attract the collective wisdom and resources of collaborators and you'll likely find your solutions are more impactful, strategic and sustainable.

    About the author
    Cameron Shorter has been a Geospatial Business Analyst at NSW Spatial Services.


    Friday, 30 August 2019

    Time-dependent datum problems

    Technical shortcuts taken in mapping systems, which ignore tectonic plate movements, are causing meter level map misalignments, most notably in web-mapping. It is caused by a cluster of interrelated problems, which we attempt to comprehensively catalogue in this article. For a more concise summary, look here.
    Caution: This article is full of acronyms and complex geospatial mapping concepts. It is primarily aimed at technical experts in this domain. It provides a snapshot in time, collated from a range of people. It is expected to have inaccuracies and not be up to date with latest thinking.

    Main problems

    Web-mapping’s illusion of accuracy
    • To date, web-mapping has provided an illusion of accuracy; all map layers have typically been aligned; but they have all been aligned to the WGS 84 epoch (and tectonic plate location) associated with the regional datum. This is being exposed as misaligned maps where a nation creates new national datum(s) as part of datum modernisation programs.
    • Australia is experiencing this problem now, due to an update of the nation datum from GDA94 to GDA2020.
    WGS 84 datum
    • WGS 84 Web Mercator is the defacto standard used in web-mapping.
    • WGS 84 has inherent ambiguity which in practice introduces low accuracy:
      • It has multiple realizations (it has been re-defined six times to date).
      • In web-mapping, the dynamic WGS 84 datum is being used as if it were a static.
      • In contrast, GPS positioning delivers point positions in the WGS 84 dynamic datum at the ‘current’ epoch (date).
      • Different epochs of WGS 84 have been adopted, in practice, in different regions of the world.
      • It uses step-wise annual updates rather than being dynamic at any time-scale.
      • … and more
    • Static tiled basemaps around the world provides a major barrier to moving away from the WGS 84 Web Mercator projection, as all existing maps would need to be reprojected, or face misalignment with new datums.
    • Adopting a new web-mapping datum would be costly, requiring extensive retiling of all the world’s tiled maps and stored datasets.
    Spatial Standards
    • To accurately describe coordinates, spatial standards describing position in a dynamic environment must accurately describe:
      • The coordinates (e.g. Latitude, Longitude, Height),
      • The datum (reference frame) and
      • The epoch (date) at which the coordinates are expressed. (This is often different to the epoch at which coordinate was first observed!)
    • Epoch is implicitly defined for a static datum (e.g. 1994.0 in GDA94) and needs to be explicitly defined in a dynamic (time-dependent) datum (e.g. ITRF1992@1994.0). Unfortunately, epoch is not incorporated in legacy standards, so updating the way in which dynamic datums are defined and applied must be addressed in the full stack of OGC standards.
    • Even the most widely used of OGC standards, like the Web Map Service (WMS) standard, do not account for time dependence when selecting a dynamic Coordinate Reference System (CRS).
    • File formats such as KML and GeoJSON only support storing coordinates in WGS 84. This needs to be considered if addressing future accuracy requirements.
    Communication challenges
    Collaborating to solve mapping challenges is hindered by communicating a difficult problem, with complex terminology, to a diverse audience. For example:
    • There is a general lack of use and understanding of ‘accuracy’, and how the choice of transformation and projection can affect accuracy.
    • Authoritative documents use terms differently in different documents, or use multiple terms for similar concepts. For example:
      •  “datum”, “reference frame”, and “CRS”, all have similar meanings.
      • There are several different concepts of epoch which are required to accurately describe coordinates in a dynamic datum. For example, a coordinate can be observed at epoch 2017.21, but recorded or expressed at epoch 2018.5 in a reference frame of ITRF2014.

    Action Plan

    Urgency
    There is a need for urgency because:
    • Australia has 1.8 metre map misalignments now.
    • North America plans to introduce new datums in 2022.
    • High-precision mass-market positioning is just around the corner.
    Technically
    We need to:
    • Solve web-mapping dynamic/static misalignments.
    • Advance time-dependent mapping.
    • Update standards.
    • Roll into software.
    • Improve world mapping practices and user outcomes.
    Community mobilisation
    While we have technical challenges, we have a harder outreach and community mobilisation challenge. Success depends on collaboratively bringing software, data and geospatial communities with us. We need core buy-in from:
    • Leading tiled map providers,
    • Spatial data providers,
    • Software venders, and
    • Government mapping agencies.
    We need clear messaging that Julie, the GIS officer, and Joe, the graduate software developer, can use to explain the datum modernisation story.

    Web-mapping business requirements

    Let’s revisit the business requirements of accurate web-mapping to help us assess the viability of current and proposed solutions.
    Legend for current support:
    • Supported
    • Partially supported
    • Not supported
    Geospatial requirements:
    1. Support high accuracy mapping.
    2. Account for time dependence resulting from tectonic plate movement.
    3. Provide accurate map alignment when displaying map layers from disparate sources.
    4. Support calculation, publishing and application of accuracy metadata.
    5. Datasets must have a nominated datum (reference frame) and epoch (date). Epoch is implicitly defined for a static datum and needs to be explicitly defined in a time-dependent datum.
    Usability requirements:
    1. Datasets must be able to be transformed between datums and epochs.
    2. Software applications shall continue to be responsive and performant for users, including for low-spec clients, such as browsers and mobile apps.
    3. Web-mapping services shall continue to scale efficiently to support multiple users.
    4. Users shall continue to be able to save map vector data as static files (such as KML, GML, GeoJSON), and render at a later point in time.
    5. Accuracy information should be embedded in decision workflows.
    6. Spatial expertise required to be learned by software implementers should be minimised.
    Software Implementation requirements for web-mapping:
    1. Tiled web-mapping shall continue to be supported to address performance and scaling. In practice, this freezes maps in time.
    Derived CRS requirements:
    1. Adopt a CRS, datum and projection for publishing map layers with the following characteristics:
      1. Defined accurately.
      2. Receives accurate transformations from key source CRSs (in particular, from national/regional datums).
      3. Aligned with a conventional fixed epoch (date).
      4. Receives coordinates converted from other epochs.
      5. Applicable at global and local scales.
      6. Near-universally adopted within web-mapping contexts.

    Main problems

    Web-mapping’s illusion of accuracy
    • To date, web-mapping has provided an illusion of accuracy; all map layers have typically been aligned; but they have all been aligned to the WGS 84 epoch (and tectonic plate location) associated with the regional datum. This is being exposed as misaligned maps where a nation creates new national datum(s) as part of datum modernisation programs.
    • Australia is experiencing this problem now, due to an update of the nation datum from GDA94 to GDA2020.
    WGS 84 datum
    • WGS 84 Web Mercator is the defacto standard used in web-mapping.
    • WGS 84 has inherent ambiguity which in practice introduces low accuracy:
      • It has multiple realizations (it has been re-defined six times to date).
      • In web-mapping, the dynamic WGS 84 datum is being used as if it were a static.
      • In contrast, GPS positioning delivers point positions in the WGS 84 dynamic datum at the ‘current’ epoch (date).
      • Different epochs of WGS 84 have been adopted, in practice, in different regions of the world.
      • It uses step-wise annual updates rather than being dynamic at any time-scale.
      • … and more
    • Static tiled basemaps around the world provides a major barrier to moving away from the WGS 84 Web Mercator projection, as all existing maps would need to be reprojected, or face misalignment with new datums.
    • Adopting a new web-mapping datum would be costly, requiring extensive retiling of all the world’s tiled maps and stored datasets.
    Spatial Standards
    • To accurately describe coordinates, spatial standards describing position in a dynamic environment must accurately describe:
      • The coordinates (e.g. Latitude, Longitude, Height),
      • The datum (reference frame) and
      • The epoch (date) at which the coordinates are expressed. (This is often different to the epoch at which coordinate was first observed!)
    • Epoch is implicitly defined for a static datum (e.g. 1994.0 in GDA94) and needs to be explicitly defined in a dynamic (time-dependent) datum (e.g. ITRF1992@1994.0). Unfortunately, epoch is not incorporated in legacy standards, so updating the way in which dynamic datums are defined and applied must be addressed in the full stack of OGC standards.
    • Even the most widely used of OGC standards, like the Web Map Service (WMS) standard, do not account for time dependence when selecting a dynamic Coordinate Reference System (CRS).
    • File formats such as KML and GeoJSON only support storing coordinates in WGS 84. This needs to be considered if addressing future accuracy requirements.
    Communication challenges
    Collaborating to solve mapping challenges is hindered by communicating a difficult problem, with complex terminology, to a diverse audience. For example:
    • There is a general lack of use and understanding of ‘accuracy’, and how the choice of transformation and projection can affect accuracy.
    • Authoritative documents use terms differently in different documents, or use multiple terms for similar concepts. For example:
      • “datum”, “reference frame”, and “CRS”, all have similar meanings.
      • There are several different concepts of epoch which are required to accurately describe coordinates in a dynamic datum. For example, a coordinate can be observed at epoch 2017.21, but recorded or expressed at epoch 2018.5 in a reference frame of ITRF2014.

    Main options

    Option 1: Do Nothing - Low accuracy web mapping

    • Accept poor accuracy and misaligned maps in web-mapping.
    • This is unacceptable for Australia.
    Option 2: Call a spade a spade
    • By convention, acknowledge that WGS 84 is currently being used as an accurate static datum in web-mapping.
    • This will likely be accompanied by defining new EPSG code(s) to recognise the currently implemented datum.
    • We would need to acknowledge that WGS 84, as used by the GPS system, is a different datum entirely.
    • In Australia, we would acknowledge that WGS 84 is aligned with our old GDA94 datum.
    Option 3: Adopt a common fixed epoch for web mapping
    • Adopt a common fixed epoch to facilitate static tiled maps of the world. This epoch might be periodically re-baselined into pre-planned “epochs of convenience”.
    • (Currently, maps are tiled into different epochs in different regions.)
    • Note: This would involve retiling all the world’s basemaps to the new common epoch.
    • Note: if committing to retiling all maps, we can consider additional improvements to map tiles, such as moving away from the less accurate spherical assumptions of the earth used by WGS 84 Web Mercator.
    Option 4: Adopt dynamic datum
    • Adopt a conventionally agreed datum which is explicitly dynamic and internationally recognised, such as ITRF. This will allow new data to be more easily moved through time back to the common epoch.
    • Note: this would involve retiling all the world’s basemaps to the new datum.
    Option 5: Use client based datum/epoch conversion tools
    • Allow users to obtain coordinates in (any) desired datum and/or epoch by transforming (x,y) mouse coordinates on screen, rather than expressing and storing underlying base-maps in multiple datums. (Move Mohammad to the mountain rather than moving the mountain to Mohammad.)
    Update spatial standards
    • Almost all options will require updates to standards to cater for time-dependence. For example:
    • Web Services such as WMS, WFS, etc., which accept a dynamic CRS as a parameter, will additionally need to accept a CRS epoch field.
    • Data transfer formats such as KML and GeoJSON which only store coordinates in the WGS 84 datum will need to be extended to support more accurate datum(s).
    Clear messaging
    • Investing in clear messaging should help with communication challenges.
    • We should simplify developer and user experiences by abstracting complicated concepts behind simple implementations.

    Action Plan

    Urgency
    There is a need for urgency because:
    • Australia has 1.8 metre map misalignments now.
    • North America plans to introduce new datums in 2022.
    • High-precision mass-market positioning is just around the corner.
    Technically
    We need to:
    • Solve web-mapping dynamic/static misalignments.
    • Advance time-dependent mapping.
    • Update standards.
    • Roll into software.
    • Improve world mapping practices and user outcomes.
    Community mobilisation
    While we have technical challenges, we have a harder outreach and community mobilisation challenge. Success depends on collaboratively bringing software, data and geospatial communities with us. We need core buy-in from:
    • Leading tiled map providers,
    • Spatial data providers,
    • Software venders, and
    • Government mapping agencies.
    We need clear messaging that Julie, the GIS officer, and Joe, the graduate software developer, can use to explain the datum modernisation story.

    Detailed problem list

    In this section we provide a “shopping list” of problems which hinder the adoption of high-accuracy mapping. It builds from experiences in implementing Australia’s datum modernisation program and ties into international challenges. It touches on time-dependence, standards, web-mapping, and map misalignment challenges.

    1. WGS 84 Datum ambiguity

    As WGS 84 is the defacto standard used in web-mapping, it is worth understanding the relationship between projection, datum and CRS:
    • The WGS 84 Web Mercator projection (EPSG::3857), technically called the Pseudo Mercator projection, is based on
    • The WGS 84 Coordinate Reference System (EPSG::4326), which is based on
    • The WGS 84 Datum ensemble (EPSG::6326), which is an ensemble of
    • Six WGS 84 Datum realizations.
    Figure: Basis of WGS 84 from EPSG Registry 
    As explained by Roger Lott, the formal WGS 84 datum is not unique; it has been updated six times to date. Each update (called a realization) refines the datum’s alignment with the earth to account for improved measurements of the earth’s shape (which is different to tectonic plate movement). Each WGS 84 formal realization is dynamic. I.e. coordinates change with time. To be unambiguous, coordinate metadata needs to include coordinate epoch (date).

    Problem 1.1 Non-unique datum:
    The WGS 84 datum ensemble (EPSG::6326), represents any or all of the formal WGS 84 realizations, without distinction. It should be considered to be static, with metre level uncertainty.

    Problem 1.2 WGS 84 coordinates are stepwise updated

    Figure: Stepwise jumps in WGS 84

    WGS 84 coordinates for ground stations and satellite ephemerides (and therefore the datum definition) are re-computed each year, mid-year. WGS 84 actually moves in a step-wise fashion. If this is not taken into account, then positioning problems can result. (Refer to ISO 19161-1.)

    2. Modernising OGC Standards

    To accurately describe coordinates, they must include the coordinates, a datum (reference frame) and epoch (date). Epoch is implicitly defined for a static datum and needs to be explicitly defined in a time-dependent datum.
    The base OGC and ISO standards document, OGC Abstract Specification Topic 2: Referencing by coordinates, introduces modern geodetic time-dependent concepts, including:
    • Extension to describe dynamic geodetic reference frames.
    • “Datum ensembles” which allows grouping of related realizations of a reference frame.
    • In a dynamic CRS, the coordinate epoch is stored as an attribute of a set of coordinates, it is not part of the CRS. (A set of coordinates could be points, features, or a dataset.)
    Problem 2.1: Coordinates in Dynamic CRS are ambiguous
    To date, coordinates which have been stored in a dynamic CRS have been ambiguous, as the observation epoch typically has not been recorded, and epoch is not stored in the CRS either.
    This is proposed to be addressed by including epoch with coordinates when using a dynamic CRS.

    Problem 2.2 Implementation challenges with time-dependent standards
    • We will need to consider how new time-dependent coordinates can be introduced without breaking backward compatibility with existing standards, software, and datasets.
    • There is likely to be performance and implementability implications depending on whether epoch is tied to the level of a point, feature, or dataset.
    These should be considered before continuing with standard definitions.

    Problem 2.3: OGC standards which don’t address time dependence
    Core OGC standards still don’t address time-dependent coordinate referencing. This includes standards such as the GML format, WMS, WFS and so on.

    Problem 2.4 High impact to update software to support time dependence
    Changing the core coordinate definitions to support time dependence is likely to have a significant impact on software implementations, and stored datasets.

    Problem 2.5: KML standard uses WGS 84
    The OGC KML 2.3 standard, initially defined by Google, is used widely for storing map data. By definition, KML stores coordinates in the ambiguous WGS 84 CRS (EPSG::4987), and doesn’t record epoch (date), resulting in dataset inaccuracies.

    Problem 2.6: GeoJSON standard uses WGS 84
    The current GeoJSON standard (dated August 2016) states:
    The coordinate reference system for all GeoJSON coordinates is a geographic coordinate reference system, using the World Geodetic System 1984 (WGS 84) 
    WGS 84 is not explicitly defined (by an EPSG code or similar) and could be any of the WGS 84 realisations. Likewise, epoch (date) is not recorded, resulting in further dataset inaccuracies.

    Problem 2.7 Non-standard GeoJSON in use
    While the GeoJSON standard only allows a CRS of WGS 84, “in the wild” implementations appear to allow implementation of different CRS values (as per earlier drafts from April 2006 of the GeoJSON standard). For instance, the OpenLayers browser based client supports a GeoJSON CRS attribute. See example:

    Problem 2.8 Inconsistent reference to CRS
    As explained in a Request for Change for content negotiation by CRS, inconsistency in referring to CRS is causing implementation problems:
    Currently, many GeoJSON API based implementations support communication of CRS. The GeoJSON standard does not provide a standardized method of negotiating CRS. As a result, each implementation varies with respect to both how a CRS is requested, as well as the method used to communicate which CRS was used to locate the data returned. Most, if not all, implementations use the body of the request or response for communication.
    There are three issues with this implementation option. Firstly, the name of the parameters used in requests and responses vary between implementations. For example, the variation of terms used to specify the name of the parameter include, but are not limited to, "SRS" (the acronym for Spatial Reference System) and "CRS". Secondly, the semantics of the value used in requests and responses also varies between implementations. For example, the variation of terms used to specify the name of the value for the World Geodetic System of 1984 include, but are not limited to, "WGS84", "4326", and "EPSG:4326". Finally, the value returned by a service does not necessarily support the ability to dereference the CRS.

    3. Publishing static maps using a dynamic datum

    Organisations have been inadvertently freezing dynamic maps in time when the maps are published. This prevents the feature’s coordinates from changing over time as should happen for coordinates stored in a dynamic datum.

    Problem 3.1: Tiling dynamic datum maps
    When maps are tiled, they are frozen in time. As the defacto standard for tiled maps is the dynamic WGS 84 Web Mercator projection, we are creating a static/dynamic map misalignment problem.

    Problem 3.2: Publishing and distributing dynamic datum maps
    Some organisations have been publishing map data in the ambiguous WGS 84 datum. Also, most web services are configured to allow users to download maps in various datums, including WGS 84. When downloaded, these maps are frozen in time, creating the static/dynamic map misalignment problem. The problem is often exacerbated by technology and data providers who encourage use of WGS 84 to facilitate interoperability with map tiles.

    4. Web-mapping has been topologically aligned, but not accurate

    To date, web-mapping has typically appeared more accurate than the metre level accuracy associated with the WGS 84 datum ensemble. Map layers have been topologically aligned with each other, creating an illusion of accuracy. However, all layers have all been equally misaligned. This occurs because:
    • National mapping agencies store maps accurately in their region’s official static datum.
    • Until recently, a region’s official datum only had one transformation to the WGS 84 datum.
    • As a time-dependent transformation was not applied, this locked the target datum to a point in time, effectively emulating a static datum.
    • For instance, the transformation from CRS GDA94 (EPSG::4283) to CRS WGS 84 (EPSG::4326), (Transform: EPSG::1150) locks transformed coordinates to the year 1994.0.
    Problem 4.1: Coordinates systematically misaligned:
    Maps sourced into the WGS 84 datum ensemble are regularly systematically misaligned.

    Problem 4.2: Lack of awareness:
    The poor accuracy of WGS 84 has been masked from users as they see topologically aligned maps. Effectively, there has been a communication gap between the geospatial community and software developers in understanding this problem.

    Problem 4.3: WGS 84 being used as if it were an accurate static datum
    In web-mapping, WGS 84 is regularly implemented as an accurate static datum. Datasets are consistently transformed from ‘ground truthed’ maps in national/regional datums via null transformation to WGS 84, resulting in consistently misaligned maps, which create the illusion of accuracy.

    Problem 4.4: Misalignment with GPS sourced points
    WGS 84 features sourced from current GPS devices (such as mobile phones), will be misaligned with WGS 84 maps transformed from a region’s official static datum. The tectonic plate coordinate shift will be exposed.

    Problem 4.5: Different epochs in each region:
    WGS 84 uses null transformations to datums at different epochs in different regions. In practice, this means WGS 84 is aligned with different epochs (dates) in different regions. For example:
    • Australia’s epoch of WGS 84 is aligned with GDA94 (EPSG::4283), which is aligned to ITRF92 at epoch 1994.0 (Transform EPSG::1150).
    • USA’s epoch of WGS 84 is aligned to CRS NAD83 (EPSG::4269) from 1986.0 (Transform EPSG::1188) etc.
    This will make it difficult for the world to agree to lock in a previously used WGS 84 epoch for web-mapping.

    5. Australia’s misaligned web-maps

    Australia had defined our static GDA94 datum as coincident to ITRF1992 on 1994-01-01 which was coincident to WGS 84 (G730) at that time, leading to defining a null transformation between GDA94 and WGS 84 (EPSG::1150).
    However, due to Australia’s fast-moving tectonic plate (~7cm per year), coordinates in the dynamic WGS 84 datum drift apart from static GDA94 coordinates. In 25 years, this equates to ~ 1.8 metres.
    Modern GPS positioning, which uses WGS 84 coordinates, and is projected to have centimetre level accuracy within the next few years, now have metre level misalignment with GDA94 sourced basemaps.
    We have “mountain to Mohammed or Mohammed to mountain” options to address this misalignment:
    1. Move basemaps and datum to align with current GPS (WGS 84) positions. (Move mountain to Mohammed).
    2. Adjust coordinates in GPS mobile devices to account for WGS 84 and GDA94 differences. (Mohammed goes to the mountain).
    Australia chose option 1, to gazette a new official national datum, GDA2020, which will be coincident with WGS 84 (G1762) and ITRF2014 on 2020-01-01.
    Figure: Misalignment of WGS 84 Web Mecator web-maps
    when sourced from GDA94 and GDA2020

    The following datum transformations have been defined:
    1. GDA94 - GDA2020 (EPSG::8048) accounts for Australia’s drift of ~ 1.8 metres.
    2. GDA94 - WGS 84 (EPSG::1150), a null transformation which assumes GDA94 is the same as WGS 84.
    3. GDA2020 - WGS 84 (EPSG::8450) a null transformation which assumes GDA2020 is the same as WGS 84.
    Problem 5.1: Systematic misaligned maps
    Combining WGS 84 map layers, sourced from GDA94 and GDA2020 via null transformations, results in the systematic misaligned by ~ 1.8 metres. (Previously these layers would have been aligned because they would have both been transformed from GDA94.)

    Problem 5.2: Reverting back to GDA94 in web-mapping
    To facilitate backward compatibility with existing WGS 84 based web-mapping services, Australia plan to continue aligning new WGS 84 based web-maps with the old GDA94 instead of with the new GDA2020 (as originally planned).

    Problem 5.3: Equating a static and dynamic datum
    Defining a static and dynamic datum as equivalent will inevitably lead to inaccuracies which will increase over time, and will need to be addressed in future.

    Problem 5.4 Moving datums is costly
    Moving to a new national/regional datum to improve accuracy requires significant effort, and in Australia’s case, we discovered that the widespread adoption of web-mapping introduced extra hurdles.
    Notably, new coordinate operations, which support moving coordinates between epochs, can be implemented to address the same accuracy business drivers.
    These options should be considered by those considering future updates to national/regional datums.

    Problem 5.5 GDA2020 - WGS 84 transformation not implemented:
    Some software providers (such as ArcGIS server), which is used by many Australian mapping agencies, have not implemented the GDA2020 - WGS 84 (EPSG::8450) transformation, restricting these agencies from supporting it. (Considering our web-mapping misalignment problem, this is probably a good thing.)

    6. Reference frames and projections

    Problem 6.1: WGS 84 being used instead of modern ITRF
    The WGS 84 Reference Frame used in web mapping is not as accurate as the International Terrestrial Reference Frame (ITRF). Compared to WGS 84, ITRF is:
    • The baseline reference frame used in geodesy
    • The reference frame being adopted by national datum modernisation initiatives
    • Addresses time
    • Has more accurate transformations into it.
    Future global datum usage should consider moving from WGS 84 to ITRF.

    Note 6.2: Spherical instead of Ellipsoidal Datum:
    The WGS 84 Web Mercator projection used in web-mapping is based on a sphere, not an ellipsoid. This results in coordinates being distorted by up to tens of kilometers. (Distortions increase the closer to the poles). Further, the projection is non-conformal, which means the amount of distortion in the x axis differs to the y axis. Consequently derived distances, areas, bearings and so on will also be distorted.
    If the world moves from WGS 84 to another CRS for web-mapping, along with the global retiling of maps entailed, this would be an opportunity to reconsider moving to an ellipsoidal based projection.

    7 Constructs to describe accuracy

    • Transforming coordinates between datums has associated inaccuracies, which differs depending on the transformation path taken, and the transformation path typically is not predetermined.
    • In contrast, converting coordinates from one reference system to another, (e.g. from latitude /longitude to easting /northing) is a mathematical procedure which does not introduce inaccuracy.
    • The lineage of how a dataset’s coordinates are derived is typically not recorded with the dataset.
    • By definition, transformations between datums are assigned an accuracy, but accuracy isn’t assigned to a datum per se.
    • Dataset formats and web service standards typically don’t record or publish accuracy statements.
    Problem 7.1 Difficult to determine dataset accuracy
    As standard spatial data formats and web service interfaces don’t define attributes for accuracy, it is difficult for users to determine the accuracy of a datasets. Equally, it is difficult to publish a dataset’s accuracy.

    Problem 7.2 Inaccuracy introduced during round-trip transformations
    There are multiple transformation paths between datums. For instance, transforming between a GDA94 source and GDA2020 target datum can apply:
    • EPSG::8048, Helmert 7 Parameter transformation (1cm accuracy)
    • EPSG::8446, NTv2 Conformal transformation (5cm accuracy)
    • EPSG::8447, NTv2 Conformal and Distortion transformation (5cm accuracy)
    Further, transforms can be applied directly (eg: GDA94 -> GDA2020) or via a hub transformation (eg: GDA94 -> WGS 84 hub -> GDA2020). Direct or hub potentially provides different results with different accuracies.
    Because transformation lineage is typically not recorded with a dataset, transforming back to a source datum can introduce inaccuracy due to a different reverse transformation being applied.
    Note: For proj based implementations: the proj6 library will use the most accurate transformation, if available on the system, otherwise it will fallback to less accurate transformation available, and provide hints on where to download more accurate transformations.

    Problem 7.3 Describing accuracy
    We don’t have the means to convey to a data administrator that a datum includes inaccurate transformations into it (and hence should be avoided for accurate use cases) - E.g. for the WGS 84 datum ensemble.

    Problem 7.4 Lack of awareness of accuracy
    There appears to be a general lack of discussion and awareness about these accuracy challenges.

    8. Communicating map concepts

    Discussion about mapping is hindered by:
    • A complicated technical problem,
    • Being applied to solve multiple use cases, with differing requirements and a variety of implemented solutions.
    • Multiple user groups understanding different facets of the problem, often using differing terminology, and talking past each other.
    • Historical implementations sometimes clouding people’s understanding.
    • Message crafting typically being dominated by highly technical people with minimal access to technical writers, professional educators and communicators.
    Problem 8.1 Confusing terminology
    Understanding the intricacies of mapping is complicated by terminology which is often misused, overloaded or misunderstood.

    Problem 8.2 Confused messaging
    Based on questions and misunderstandings seen on email lists, there appears to be a lack of clear messaging and education around projections, datums, coordinate reference systems, transformations, EPSG codes and related concepts. Messaging would benefit from input by communication and education expertise, tailored for key user groups.

    Problem 8.3 Abstract out complexity
    We should simplify developer and user experiences by abstracting complicated concepts behind simple implementations.

    Web-mapping terminology (simplified)

    Understanding the intricacies of mapping is complicated by terminology which is often misused, overloaded or misunderstood.
    Hopefully, these simplified explanations should help. For official definitions, refer to the OGC Abstract Specification Topic 2: Referencing by coordinates.
    • Datum (more recently called a Reference Frame): A reference frame is a mathematical model of the earth against which features can be represented as coordinates.
      • Within a static datum, a feature’s coordinates are locked in time, or locked to its tectonic plate, so the feature’s coordinates remain the same over time. For example, GDA2020 is a static datum for Australia.
      • Within a dynamic datum, coordinates are fixed to the earth as a whole. As the earth’s tectonic plates move (by a few centimeters a year), a feature’s coordinates also move. Examples include WGS 84 as currently used by Global Positioning Systems (GPS), and the current realization of the International Terrestrial Reference Frame (ITRF2014).
    • Datum Ensemble: A group of related realizations of a Reference Frame (datum) used for lower accuracy applications where differences are insignificant. The WGS 84 datum (EPSG::6326) used in web-mapping is a datum ensemble.
    • Coordinate SystemA set of mathematical rules which specify how coordinates are assigned to points.
    • Coordinate Reference System (CRS): A coordinate system that is related to the real world by a datum. A CRS is typically referenced in Web Service calls by an EPSG code. EPSG::4326 is the defacto standard CRS used in web-mapping and refers to the WGS 84 datum ensemble (EPSG::6326).
    • Epoch: A point in time, as applied to time dependent datums, expressed in decimal years. For example, 2017-03-25 is epoch 2017.23.
    Multiple definitions of epoch are commonly used to describe different aspects of a coordinate and a CRS, which can lead to confusion:
      1. Observation epoch: The date at which the observation was made.
      2. Coordinate epoch: The date at which the coordinates are expressed.
      3. Realization epoch: Part of the Indicates in general the date at which a datum was defined.
      4. Reference epoch: The epoch at which time-dependent coordinate transformation parameters are defined, along with their rates of change. (The EPSG database currently calls this “Realization epoch”.)
    • Map projection: Coordinate conversion from the earth’s ellipsoidal coordinate system to a flat plane. WGS 84 Web Mercator (EPSG:3857) is the defacto standard projection used in web-mapping. (Its official name is “Pseudo Mercator”). WGS 84 Web Mercator is projected from the WG84 datum ensemble (EPSG::6326).
    • EPSG Codes: Online database that contains definitions of numerous datums and map projections, along with formulas to translate between them. Each is uniquely identified via an EPSG code.
    • Coordinate operations:
      • Coordinate conversion: Changes coordinates from one coordinate reference system to another coordinate reference system based on the same datum (reference frame).
      • Coordinate transformation: Changes coordinates from one coordinate reference system to another coordinate reference system which is based on another datum (reference frame).
      • Point Motion Operation: Changes coordinates within one coordinate reference system to account for the motion of the point within the CRS over a period of time.

    Thursday, 22 August 2019

    Misaligned web-maps? High-accuracy data must become time-dependent

    Misalignment of WGS84 web-maps when
    sourced from GDA94 and GDA2020.
    Current web-mapping implementations are compromising datum modernisation programs by introducing metre-level misalignments of data. It is the start of a suite of high-accuracy and time-dependent mapping challenges the world is about to face.

    While web-mapping has worked ‘well enough’ to date, it cannot support the high-accuracy requirements expected by modern users and data. It is hampered by the ambiguity inherent in the practical application of WGS84 as used by the Web Mercator projection, and lack of care with the recording and use of time metadata.

    Mass market positioning is now accurate enough to be affected by the earth’s measurable surface deformation and tectonic plate motion. Providing an accurate frame of reference for new users and applications has been a significant driver behind recent datum modernisation programs, for example in Australia and the United States. To succeed, these programs urgently require standards, software and practices to be updated and adopted.

    In general, the time-dependence of spatial data is not yet properly managed and is often poorly understood and communicated. While standards such as ISO 19111:2019, ISO 19127:2019 and ISO 19162:2019 have been recently updated to address this issue, there remains significant work to refine:
    • Downstream spatial standards;
    • EPSG definitions for datums, projections, Coordinate Reference Systems (CRS) and transformations (changes are in train);
    • Software;
    • Published and stored datasets; and
    • Education and outreach.
    These issues require international collaboration to be addressed effectively.

    Key technical problem:

    Web-mapping is generally published in the WGS84 Web Mercator Projection, which is based on the WGS84 datum ensemble (EPSG::6326). While this WGS84 ensemble represents any or all of the six formal realisations of the WGS84 dynamic (earth fixed datum), in practice WGS84 is generally used to mean either:
    1. The realisation (and epoch) most closely equivalent to the local national datum, for example when transforming accurate ‘ground-truthed’ maps and data into WGS84 for web-mapping. This usually relies on defined ‘null transformations’ to WGS84, as in Australia from GDA94 (EPSG::1150) and North America from NAD83 (EPSG::1188).
    2. The latest published WGS84 realisation, with the datum effectively static at the date of collection, as with single-point-positions determined directly via GPS, which also have an inherently low accuracy.
    While each WGS84 realisation in the ensemble has been well defined, the generic use of the datum ensemble in web-mapping should be considered to have an uncertainty of several metres. 

    The Australian datum modernisation initiative has exposed this uncertainty in WGS84 by introducing a new static national datum, ‘GDA2020’ at the epoch 2020.0. Coordinates in this datum are ~1.8 metre north-east of the previous national datum, GDA94, due to the 7cm per year tectonic motion of the Australian Plate. The new null transformation (EPSG::8450) from GDA2020 back to the WGS84 datum ensemble introduces systematic 1.8 metre web-mapping misalignments.

    Datum modernisation programs planned for other regions will face similar problems, including when the U.S. introduces four new time-dependent datums in 2022.  To map these back to a common datum and epoch for web-mapping will require a much better definition of the web-mapping datum, and a better application of time-dependence in data and transformations.

    Potential solutions:

    It is recognised that a long-term solution is required to account for time-dependence in web-mapping. Options include:
    • Define a specific WGS84 realisation for web-mapping at a fixed and conventional epoch (point in time). 
    • Adopt and define a conventional epoch per region to reflect current practice in web-mapping. 
    • Avoid creating new null-transformations to the WGS84 datum ensemble.
    • Migrate away from the defacto WGS84 Web-Mercator projection currently used for web-mapping, to a new convention likely based on the International Terrestrial Reference Frame (ITRF), at a defined epoch.
    • Update web-mapping applications to natively cater for time-dependence. 
    • These options must be considered in concert with the required development of standards and practices to handle time-dependence across all spatial data applications.
    In the short term, Australia still faces the immediate issue of 1.8 metre misalignments in web-mapping. The likely approach is to deprecate the newly introduced null transformation from GDA2020 (EPSG::8450) and replace it with a transformation which brings all local data back into alignment with the previous GDA94 datum, and with existing WGS84 web-mapping products, at epoch 1994.0. This would effectively acknowledge that WGS84 is being used in practice as a relatively accurate, static datum.

    Support for this short-term approach would be required from software providers, who would have to introduce the custom GDA2020 => GDA94 => WGS84 concatenated transformation as default, and/or to adopt any new EPSG updates at the first practicable opportunity.

    Call to action:

    While there are a few quick-win solutions, most of the challenges will be hard. They will require deep insight, strong leadership, sustained effort, and international collaboration to address them properly. Australia is facing these issues now, and the rest of the world are about to face them.
    We are cataloguing these challenges and possible solutions on behalf of Australia’s Intergovernmental Committee for Surveying and Mapping (ICSM). A draft will be published within a couple of weeks, by Friday 30 August 2019. It will then be presented at the next OGC Technical Committee meeting, September 2019 in Banff, Canada.
    We want to test and refine these ideas with the breadth of the geospatial community and find collaborators. If you can help then please:
    • Comment on the draft document when it comes out.
    • Contribute to discussions currently happening on the OSGeo proj email list.
    • Reach out to us at the technical committee meeting if you will be there.
    • Take this message to stakeholders you think should be involved.
    We look forward to hearing from you.

    About the authors:

    • Joel Haasdyk is a Geodetic Surveyor, and GDA2020 Implementation Manager at NSW Spatial Services, Australia. He will be presenting these ideas at the OGC TC meeting in Banff on behalf of Australia’s Intergovernmental Committee for Surveying and Mapping.
    • Cameron Shorter is a Geospatial Business Analyst for the GDA2020 datum modernisation program at NSW Spatial Services, Australia; he is a long-standing OSGeo Open Source community contributor and commentator.


    An early version of this document was published on the OGC Blog and more details can be found here.

    Thursday, 8 August 2019

    Two tech writers for OSGeo Season of Docs

    Felicity Brand and Swapnil Ogale have been selected to join Google's Season Of Docs, under the umbrella of the OSGeo Foundation. Felicity and Swapnil are both seasoned technical writers, keen to help, and keen to learn about working with Open Source communities. I'm sure you will make them feel welcome and will help them to help us.

    Felicity has opted to focus on reviewing OSGeo Quickstarts.

    Swapnil had opted to create writing templates for Open Source, but these tasks are being superseded by the emerging GoodDocsProject, so we plan to focus Swapnil in one of the other writing tasks on the OSGeo agenda. (More details to come).

    Of note, discussion about OSGeo's Season Of Docs has led to scores of writing volunteers signing up to progress documentation in GeoNetwork, QGIS, OSGeoLive and Open Source in general through TheGoodDocsProject. Come join our email list, or reach out to me, if you are interested.

    Monday, 5 August 2019

    Considering future Open Geospatial conferences in Oceania

    “What should we do about upcoming conferences for Free and Open Source Software for Geospatial (FOSS4G) and State of the Map (SotM) within our Oceania region?”


    Image by @mapmakerdavid
    We didn’t have an answer to this question at the last OSGeo Oceania board meeting. At the moment, no one is volunteering to chair. No one is backing a city with commitment, and that is a problem we hope to solve.

    Pertinent is that the international FOSS4G will be held somewhere outside of Europe or North America in 2021 and a call for a two-page Expressions Of Interest is coming out in September 2019 (soon). We have a very compelling case for bringing the international community to our region, if we try.

    Are you keen to see FOSS4G and SotM continue in our region? Do you have ideas about where and how? Would you like to help make it happen? Are you interested in mentoring or being mentored by a bunch of committed volunteers who have worked on prior FOSS4G events? If so, why don’t you talk to us? Come join our email list and introduce yourself. We particularly want to hear ideas about what we should do about FOSS4G in the next two years.

    Some history:
    • 2009: FOSS4G-International in Sydney. During the global financial crisis, we had lower attendance than expected, but managed to stay profitable when similar conferences were losing money.
    • 2018: FOSS4G SotM Oceania in Melbourne. Exceeded expectations for size, sponsorship, activities, and engagement.
    • 2019: FOSS4G SotM Oceania Wellington: Has already exceeded sponsorship targets and sold out early bird tickets within a week.
    • 2020: FOSS4G SotM Oceania: Where?
    • 2021: FOSS4G-International: Should we bid for it?


    Wednesday, 1 May 2019

    OSGeo selected for Season-of-Docs 2019

    The OSGeo Foundation has been selected to participate in Google's Season-of-Docs initiative. We've identified tasks based on OSGeoLive, QGIS and GeoNework.
    Our approach is a little different. It is not just about searching for paid superstar technical writers. (Although superstars are welcome). We also want to support and expand our existing volunteer community. These are ordinary people, gifting bursts of effort, toward small, discrete and achievable tasks, to collectively achieve an extraordinary impact.

    Ideas we'd like to explore:

    • Open source projects face sustainability challenges. How will docs developed during Season-of-Docs be maintained long term?
    • Can a writer’s expertise be amplified to help community users and developers write good docs more effectively and efficiently?
    • Could best practices developed in one project be applied to the greater open source eco-system?
    Are you interested? If so, please introduce yourself on our email list, or contact me directly.

    Cameron Shorter

    Thursday, 4 April 2019

    Amplifying Google's Season of Docs

    Google has initiated a Season of Docs collaborative program aimed at improving open source documentation. It provides a great opportunity for us involved in open source and documentation to focus on big communication challenges.

    Challenges

    • Vision: Does everyone know the characteristics of good open source documentation? Does anyone know? Let’s collate research and best practices into accessible guides and templates.
    • Targetted: Projects communicate with a range of personas. They have different technical backgrounds, attention spans and information needs. A range of documentation types are needed. Each requires different levels of depth, specificity, currency, narrative, personalisation, examples, and more.
    • Empower everyone to contribute: Good documentation benefits from cross-disciplinary contributions: from developers, users, domain experts, teachers, technical writers, graphic artists, and translators. Ramp-up time is high for any group attempting to learn another’s skill-set. Open source documentation processes are typically set up by developers, for developers, and have a significant barrier-to-entry for others. How can we fix that?
    • Attract volunteers: How can we apply our knowledge of collaborative and volunteer communities to attract documentation teams? How can we help volunteers maximise the impact and effectiveness of their contributions?
    • Current and Sustainable: How can we sustainably keep documentation synchronised with rapidly evolving software? How do we help users find current material and archive outdated docs? How do we minimise maintenance?

    What to focus on?

    To maximise value, I suggest promoting initiatives which:
    • Use and contribute to documentation best practices.
    • Use templates and guides for key documentation types to help cross-domain collaboration.
    • Attracts volunteers, from multiple user profiles.
    • Applies multi-directional mentoring.
    • Refines workflows and tools to reduce barriers-to-entry.
    • Focuses on a specific initiative which tests the bounds of these ideas for a specific project and captures lessons learned.
    This might involve:
    • Auditing existing documentation.
    • Defining a writing strategy.
    • Developing or refining a specific documentation type for a project.
    • Mentoring a project’s community, increasing the initiative's sustainability.
    • Contributing to best practices.
    If we can achieve this:
    • Writer contribution impact will be maximised,
    • Community writing will become more effective,
    • And documentation maintenance will become more sustainable.

    Further Reading

    The OSGeo Foundation's focus in Season of Docs 2019:
    General:
    Documentation Strategy:
    Research into building open source communities:

    Tuesday, 12 March 2019

    Google Season of Docs Announced

    Google has just launched a Season of Docs program, designed to bring open source and technical writer communities together, to the benefit of both. Awesome! I reckon our OSGeo projects should get involved, and I'm personally keen to be part of it. Anyone else want to join me?

    Sunday, 3 March 2019

    Starting build cycle for OSGeoLive 13.0

    Are you interested in joining our Open Source GIS experience? The build cycle for the next OSGeoLive release 13.0, is going to start. The final version is planned to be ready on 30th July 2019 in time for FOSS4G 2019 in Bucharest. Have a look at our schedule for the detailed steps.
    Till then we have many challenges - from simple to challenging, to help describe and train people in the value of Open Source GIS.
    For this release, we hope to help OSGeo Community projects join OSGeoLive, and work out how we can keep within our size constraints.
    We'd love to build a cloud version of OSGeoLive for this release. It would make OSGeoLive, and OSGeo software much more accessible. It seems elusively achievable, but we haven't worked it out yet. If you think you could help, we'd love to hear from you.

    Get involved

    Are you a programmer or packager with experience in one of the Open Source GIS applications. Are you a technical writer interested in making FOSS4G accessible to all? Are you interested in helping update translations to latest documents on Transifex.

    Contact us

    If you would like to add a new project, help us to get in the cloud or have another goal you would like to work on and get involved you are very welcome.

    Please join the OSGeoLive mailing list or meet us on IRC at our weekly meeting.

    About OSGeoLive

    OSGeoLive is a ​Lubuntu based distribution of Geospatial Open Source Software, available via a Live DVD, Virtual Machine and USB. You can use OSGeoLive to try a wide variety of open source geospatial software without installing anything.


    Friday, 22 February 2019

    Inspiring techies to become great writers

    How to attract writers into Open Source communities, have them amplify developers' written impact, reduce tech ramp-up, facilitate growth, while having fun improving the world.

    Reading time: 8 minutes

    Presented at Write The Docs - Sydney [slides]

    My story starts at the international conference for Free and Open Source Software for Geospatial when it was held here in Sydney, ten years ago. We wanted to build a distribution of the software to hand out to conference attendees. Back then when software was expensive to download, it was a useful handout, and it provided an attractive marketing pipeline for these free software projects.

    To build the first version of OSGeoLive we followed install instructions, configured applications, resolved dependencies and conflicts, asked questions on email lists, and finally, after much hard work, produced our first release. This was a very important first step as it showed that we had the commitment to follow through and deliver.
    Install Script
    However, it also highlighted many of our failings. We had followed a fully manual install process, so when the next software releases came out, we had to start the installation process again, from scratch. It was completely unsustainable. We needed to automate the install process, and we needed projects to show us how.
    But our first calls for help were embarrassingly underwhelming. You see, the perceived learning curve for packaging was considered unacceptably high and volunteers were not stepping up.
    To fix this, we wrote an example install script along with clear, step-by-step instructions. We then went back to developers with the message of "if you can spend a couple of hours writing a short install script which looks like this, then we will market your application through our OSGeoLive distribution". Small effort / High Value. And it worked! 28 projects packaged their applications for our version 2 release.
    Using a document template
    In following releases, we applied the same process to documentation. Our first docs were written by developers and as such were quite ... variable. So we created skeleton outlines for Project Overviews and Quickstarts, along with clear writing instructions. We then asked projects to write documentation in line with these outlines. We followed up with reviews, and a publishing pipeline, to produce consistent, quality material. So by applying an outline / write / review / publish process, we were able to achieve significant efficiencies by allowing experts to focus only on the bits they do best and thus reduce contribution effort.
    Translators later adopted the same formula to create multiple language variants.

    Unpacking the story

    Ok, let’s unpack this story from the perspective of technical writers. Firstly, we should acknowledge the significant efficiencies software enables. Four out of the ten richest people in the world are self-made software entrepreneurs.
    However, while proprietary businesses centralise captured wealth, open source software, like other creative commons works, is given away for free. This act of sharing has far-reaching consequences. Tools and knowledge created can be used and built upon by everyone. This democratises knowledge, wealth and power.
    So if you want to positively improve our world, supporting Open Source projects is a good place to start.

    The Research

    So what makes a successful Open Source project; one which is sustained in the long term? Research provides some answers, An analysis of over 170,000 Open Source projects to identify success factors, and a study of the success factors of episodic volunteering in Open Source both highlight the importance of:
    • Clear utility,
    • Lean governance,
    • Clear vision and goals,
    • Marketing and a quality website,
    • Good user and developer documentation,
    • Digestible information,
    • Guided introductions,
    • Simple workspaces,
    • Task–finding dashboards,
    • Financial backing,
    • Fine-scaled task granularity, making it easier for people to contribute.
    And on the social side:
    • Leaders who lead by doing, by putting in the hard work,
    • A virtuous and supportive community,
    • Attracting external contributors,
    • Advocating broadly, by all involved,
    • A sense of collective ownership,
    • Hosting of meetups,
    • Recognising all forms of contribution,
    • Personal invitations for help,
    • Showing appreciation.
    Notice the prevalence of communication within these characteristics?

    Communicators Needed

    Communicators helping the world
    Where I’m heading here is that while the Open Source story usually focuses on software developers, equally important are communicators and community builders. So for you folks who are technical writers, Open Source communities need you! Your involvement can significantly amplify a project’s impact, and by extension, your help will be:
    • Making the world more egalitarian,
    • Reducing wealth inequality,
    • And democratising knowledge and power.

    Bridging the tech gap

    I’d like to touch on inflection points. As a project’s technology and user base matures, community success factors move. An early inflection point with Open Source projects happens when a project attracts external contributors. These contributors increase the size of the workforce and add extra skills, but they lack the deep domain expertise of the core team.
    To attract contributors, projects should reduce ramp-up time; and good documentation is key. Poetically, the documenters who are most qualified to help are typically the least technical, and face the greatest ramp-up time.
    Johanna Botman
    To help explain this problem, I’d like to introduce you to Johanna Botman. She started her working career as an English teacher, then moved into IT, website design, and is now a geospatial officer in local council. I met her at a spatial conference’s community day when she joined other volunteers in updating our OSGeoLive documentation. It was a humbling experience for me. I saw such an accomplished person, with oodles of commitment, struggling to use our documentation tools.
    Johanna reviewed an early draft of this presentation and observed that ...
    Some of the issues relating to getting started are about bridging the gap between developers and writers. Developers write code in coding tools. They collaborate, they are used to versioning and are at home with unformatted raw text and automation tools. Writers? They work on Windows machines in Word – maybe in Google Docs if they are lucky. They don’t know about running build scripts, running Linux variants, Github, chat programs, HTML, RST, Wikis and the variants of markup languages. 
    It’s one thing to work with the Open Source software, another to write about it, and a third thing to work out how to write it so that it fits in with the project. It seems as if the developers have created the writing system in a way that they are used to working, not necessarily in a way that works for writers.
    And I think Johanna is right. I’ve found many users and writers who are keen to help but feel overwhelmed, don’t know where to start, don’t want to burden the existing community, and so are shy to ask questions.
    Writers need to be sought out, welcomed, encouraged, supported and publicly appreciated if projects are to attract good documenters to help them grow.
    In turn, I’d encourage writers to proactively reach out to development communities. I’d expect you’ll find them very welcoming and appreciative, especially if you are volunteering to tackle key documentation pain points. And together we need to help bridge the gap between writers and developers.

    Documentation Strategy

    So what should writers expect to see when joining a project. Typically, emerging projects have patchy documentation, in various stages of currency, completion, relevance, verbosity, and quality. The challenge is to consolidate it so that it becomes concise, intuitive, accurate, minimises learning and ramp up time, and is sustainably maintained to match continually evolving software releases. This is non-trivial and requires good writing skills, planning, leadership, commitment and follow through to do it well.
    The problems are that:
    • Software is complex. It’s hard to understand, and harder to explain.
    • Software continually evolves, meaning documentation needs to be continually maintained.
    • Projects have a range of audiences, with differing information needs, technical backgrounds, and attention spans.
    Daniele Procida provides valuable advice in an article titled:  What nobody tells you about documentation.
    Document structures recommended in What nobody tells you about documentation.
    Daniele is an open source developer, technical writer and teacher and he explains that:
    There is a secret that needs to be understood in order to write good software documentation: there isn’t one thing called documentation, there are four. They are: 
    • Tutorials, 
    • How-to guides, 
    • Explanations and 
    • Technical references. 
    They represent four different purposes or functions, and require four different approaches to their creation. Understanding the implications of this will help improve most software documentation - often immensely.
    Daniele is right, although I feel his list should be expanded to include other forums.
    Placing documentation types
    Raw image (in draw.io format)
    Credit to Edwin Dawson for original concept.

    Writers should identify and define the characteristics of their communication mediums.
    The front web page and elevator pitch should be concise, highly polished and target first-time visitors, while community forums are good for once off questions and can be rough around the edges.
    For each medium, define:
    • Personas for your target audience,
    • Document structure,
    • Technical depth,
    • Quality requirements, and
    • Maintenance strategy.
    Then aim to:
    • Write timeless material to minimise maintenance.
    • Only write as much as your team has the capacity to maintain.
    • Help techies become great writers. Provide skeleton docs and writing instructions, then review, mentor and publish.
    • Work within a release cycle, which ideally synchronises with the project.
    It’s worth noting that most of what I’m talking about here is relevant to all sorts of technical writing - not just for Open Source communities.

    Getting Started?

    Tech Writer Heros
    Image source
    Okay, so maybe you’d like to get involved and are wondering where to start? I’d suggest:
    • Focus on one project instead of many, as your ramp-up time will likely be high.
    • Acknowledge the value you bring. Use it as motivation because you’ll have many challenges and the inner motivation will really help.
    • Demonstrate commitment. It’s contagious.
    • Ask for help, you will need it, and you’ll likely be surprised by how supportive people will be.
    • Don’t be daunted by your lack of domain knowledge. It’s actually an asset when writing to a broad audience.
    • Help developers become great writers. Great impact comes from amplifying the effectiveness of others.
    • Be part of the team; help paint the vision; be inspirational.
    • And most of all, have fun while you are doing it. Because believe you me, it is hugely rewarding to share the team camaraderie involved in building something that is much bigger and more impactful than you could possibly create by yourself.

      Sunday, 20 January 2019

      Toward a universal Style Manual

      The Australian Digital Transformation Agency plan to improve the Australia Government's writing style manual. Great. Can I help?

      Worst user, best friend

      I'll be your most demanding user. I'm technical, have no writing training and won't want to read the manual. And yet, this manual should be written for me. Why? Because people like me write policies, code, manuals, business cases, standards, designs,  requirements, tutorials. We write for government, industry, open source and international organisations.

      Writers shouldn't read your manual

      Auto-correctors fix spelling and grammar; templates help writers with headings and writing tips. The best way to reach us technical writers is to embed style guidance into these tools and templates. (Could you please also fix their formatting bugs, which seem to be in every word template I've ever used.) 

      Go universal

      Please avoid the temptation to create an Australian specific Style Manual. The inevitable inefficiencies caused by integrating local and international style variants will be painful. Draw inspiration from the Creative Commons licenses. They are internationally recognised, widely used and widely understood.  

      Collaborate

      Collaboration mitigates against being out-innovated, as in the digital economy, collaboration out competes competition. There are more smart people in the rest of the world, thinking about your problem, than can ever be mustered in your own team.
      • Use, extend, create - in that order:
        • Use existing material if it exists;
        • Otherwise extend and give back;
        • As a last resort, create your own.
      • Evangelise; encourage other nations to adopt the manual. Like language, the manual becomes more valuable as more people use it.
      • Use collaborative tools and methodologies to engage communities and evolve material.
      • Share ownership and control.

      Modularity

      Let's make the style manual extensible, to support domain specific requirements and individual flair. For instance, the international manual will need to support toggling between American and British spelling, and agencies will want to add domain specific acronyms. 

      Don't finish

      People will continue innovating. Establish a sustainable project, owned by multiple stakeholders, with periodic release cycles, which will last beyond your current funding cycle.