Why is the QGIS docs team struggling?

The QGIS documentation team is struggling and needs help. This has been known for a while. The much harder question is “How do we help a mostly volunteer community?”.

Reading time: 25 minutes

Summary

Many have tried to help QGIS docs, with limited success. I’ve collated insightful quotes from a bunch of their stories and then postulate solutions. Surprisingly, the biggest problem isn’t a lack of tech writers or complicated tools (although they are factors).

Problems centre around:

• Poorly capturing community good-will and offers of assistance;
• A lack of direction;
• Struggling to keep up with a rapidly evolving software baseline;
• Insufficient writing expertise;
• A high technical barrier to entry;
• Documentation and training being generated outside of the core project;
• Awkward documentation tools and processes.

This leads to an immediate case to:

• Define and evangelise a vision and roadmap.
• Prioritise funding and lobby sponsors to resource the vision.
• Implement an information architecture review.
• Sustain a community evangelist/coordinator to attract and nurture a broader doc community.
• Sustain a trained technical writer to amplify the quality and effectiveness of the community.
• Attract external docs back into the core.

Medium-term:

• Ask the greater open-source community to address the usability of documentation tools and reduce the technical barrier to entry. Adopt improvements as they are developed.
• Align with best the practices evolving within TheGoodDocsProject.

While acknowledging the great work done to date, I feel the QGIS docs team has insufficient capacity and availability to skills to drive this agenda. Targeted and sustained investment should be applied to bring the quality of QGIS docs up to the quality of the software.

Observations

The challenge

As one of OSGeo’s Season of Docs administrators, I’ve been observing the QGIS documentation community for months. The Open Source Geospatial Foundation (OSGeo) was allocated two tech writers and we probably should have allocated one to QGIS. However, I recommended they work on GeoNetwork and OSGeoLive instead. I was concerned by:

• How daunting QGIS doc challenges were,
  
• A lack of clear direction within the QGIS docs project,
  
• The brief three-month window for Season-of-Docs, and
  
• The high risk that the writers’ efforts might not achieve tangible outcomes.
  

I noted:

The big challenge for QGIS is aggregating external content into the core docs from lots of satellite communities. It would be a huge win to get it done, but also very risky as it requires coordination and collaboration from so many external volunteers.

Harrissou added:

It's unfortunate to not assign a senior writer to QGIS. I was personally envisioning [Season of Docs] as a catalyzer, an opportunity to trigger mobilisation of the writing community, and to teach us actual and best practices. And maybe that experience would confirm to us that we need the profile [of person] you propose later.

So what is lacking, and what can be improved?

Kudos to the volunteers

Firstly, I’d like to acknowledge the value provided by QGIS documentation volunteers and help they provide to newbies who reach out. QGIS has a solid baseline of docs and dedicated but under-resourced volunteers. They face a difficult job keeping up with the more active, much larger, and better-resourced developer community. I don’t think external people appreciate the difficulty of the documentation challenge.

Season of Docs

Before Season-of-Docs’ writing period officially started (September 2019) we’d already attracted plenty of latent interest:

• A spin-off GeoNetwork documentation group of 4+ volunteers was meeting fortnightly. (Swapnil, a senior tech writer supports this team, as part of Season-of-Docs.)
  
• A spin-off GoodDocsProject, with 5+ senior tech writers, are creating best-practice templates and writing instructions for documenting open-source software.
• QGIS and GeoNetwork quickstarts were updated to the latest 13.0 OSGeoLive release. (Felicity is updating 50+ quickstarts for Season-of-Docs.)
  

Over 20 people volunteered to help out with OSGeo’s Season-of-Docs. 10+ of these people were interested in QGIS - more than for any of the other OSGeo projects. However, we’ve had lack-lustre success at capturing this initial enthusiasm. Why? I collate quotes and observations below.

Piers, small company, creating training material

Piers Higgs is CEO of a Gaia Resources, a small environmental consulting company. He and his team have developed QGIS training material which they publish for free as videos, a manual and data package. Piers notes:

• The thing I find strange is how many people are using our course now - there are people from all around the world now. Most of them aren't actually enviro's either - they are just people wanting any sort of resource to help them get into QGIS.
  

Piers articulately outlined how he’d love to share his material and continue to maintain it, noting also that he is time-poor. This is a hugely valuable offer, but there wasn’t someone from the community ready to catch this offer and work with him to the extent required.
Alexandre Neto noted:

• Because we don't have many writers (we have two very active people), it's quite hard to allocate [time for] that “king of merge” into what we already have. It looks like no one has interest in it, but it's not really the case. What we would prefer is to see companies create new sections, improve and reuse what we have in the training manual.
  

Like Piers, many of the people who volunteered to help with Season-of-Docs are similarly from small consulting companies in similar situations. I see this as untapped potential. Piers commented further:

• Yep, but how to tap this potential is pretty hard. Unless you have a TARDIS, or a cloning machine?
  
• This pretty much says why I can't get into this. I don't have the bandwidth and much of my drive is taken up running the business. The personality - well, Cameron, you have spades of that ;)
  
• I did find the whole thing really hard to actually understand what was needed and who was doing what. I guess being "outside the camp" for most of the QGIS stuff these days has made me realise how hard it is to find my way back in again.
  
• So it's one thing to have a bunch of time-poor people who are interested, let's assume we don't have a TARDIS or cloning machine to fix that. I did find that trying to work out what was going on and what Season-of-Docs is, who's doing what, etc was just too big a beast to deal with. It was an effective barrier to entry for someone new, and it's one of the reasons Cameron found it so hard to engage me - I had to keep asking him for clarifications on what all the lists are, where the documents are, who's who in the zoo, etc. It's just a little bit... chaotic. I will readily admit I lean towards OCD tendencies, but being time poor, time spent trying to understand what is going on is an effective barrier to entry. It became "too hard" very quickly.
  
• [Capturing offers of assistance and supporting and encouraging new volunteers] are things as a community we do pretty badly.
  
• My interactions with the main QGIS developers etc hasn't been very frequent, but it's been reasonable.
  
• ... So I think remembering everyone is a volunteer and will have different motivations is really important. I used to run a volunteer GIS group and keeping up ways in which time poor people can be involved is key - e.g. writing a chapter is a big ask, but editing or testing it might be smaller and easier for time poor people. Food for thought.
  

Andrew, power user, starting to help with docs

Andrew Jeffrey is a power QGIS user, and a bit more. He is not a programmer and is giving to QGIS through docs, coordinating a regional user group and qgis events. (Other potential volunteers have a similar profile.) Andrew painted a practical vision about how QGIS docs can be improved and proceded to write a getting started guide for the new users he’d been helping, and followed up with a QGIS quickstart for the OSGeoLive project. Andrew is the sort of person you’d want to encourage and support.
Andrew’s comments are revealing:

• I feel this review [of QGIS docs] was started with the meetings you coordinated at the start of the Season-of-Docs process Cameron and then lost momentum because no one took the lead when you started to focus on other things. I did try to rally people for the OSGeolive quickstart amendments but quickly lost interest in continually asking for input with no response.
  
• I haven't given a whole lot - but would like to do more. Things that stop me: What’s a priority? Docs, training material, screenshots? It would be helpful if a more senior doc mentor was able to say “this is the low hanging fruit” “that is a great way to get started”.
  
• The help I have received from QGIS doc folks has been good and available when I ask for it. The support in terms of sharing contributions via participants in the Season-of-Docs has been sporadic, but I understand everyone has time constraints and other commitments. Also even before tech writers were assigned to projects I was asking the list for feedback on documentation and received nothing. So my enthusiasm for the Season-of-Docs has dropped off because I didn't feel like I was getting as much out as I was putting in.
  

So while Andrew had some support, more support would likely help him feel more welcomed and would empower him to increase his productivity. Again, I think Andrew’s anecdotes hint at lost opportunities we don’t hear about.

Jared, a tech writer

Jared Morgan is a senior tech writer, curious about open source, who volunteered to help out. He started reviewing QGIS docs and received feedback from core contributors (Harrissou and Matteo). Alexandre noted that he missed seeing Jarad’s feedback. Unfortunately, this initiative hasn’t appeared to be sustained. It appears there hasn’t been sufficient bandwidth to nurture and sustain the goodwill.

Charlie, university courses

There were a bunch of offers from GeoForAll university members, suggesting that their tertiary training material be used. For instance, we could update the comprehensive GeoAccademy courses which are still based on the old QGIS 2.8 version. Unfortunately, the initial enthusiasm didn’t translate into tangible action. From my perspective, there appeared to be a very high barrier to entry. How can we help all these disparate organisations and fragmented initiatives to collaborate on a common base of material which is brought into the core QGIS docs? How can they become less brittle, so material continues to be updated when program funding finishes?
Professor Charlie Schweik suggested developing training material and textbooks in conjunction with universities, possibly making use of OpenStax. I’d suggest that his suggestion be aligned with maintaining core QGIS material, rather than creating a parallel initiative, and that the common material can be retasked for various educational courses.

Andreas, cataloging doc team challenges

The QGIS docs team discussed many of the challenges they are facing. Andreas Neumann summarises many of these:

• I agree that the documentation task seems to be overwhelming and might also be daunting for newcomers, volunteers and even paid people. I also agree that the team is under-resourced. … We already knew this. … it would be encouraging to hear more suggestions for how to improve the situation.
  
• Should the team focus on smaller chunks/goals in order to have better progress and a better success feeling?
  
• Are the tools too complicated?
  
• Is there not enough information provided by developers or organizations who fund new features?
  
• Another observation I have is that there is an awful lot of documentation about QGIS out there on the web, spread into many personal blog websites, company blog posts and news sites, youtube movies, social media posts, etc. etc. However, all of this vast and de-centralized information doesn't end up in our central documentation.
  

Anita, Nathan, tools and process limitations

Working out how to bring the world’s QGIS documentation back into the core looks to be a core challenge for QGIS. Anita Graser’s response provides insights:

• I tried [putting my doc updates into the core] but something is keeping me from doing it regularly. Thinking about it, reasons for me include:
  
• It's not always possible to simply copy a blog post to the documentation. The expected style (as in wording) of the text is different. The text should fit into the bigger picture. This often means a significant rewrite.
  
• Maybe just me but: I'm always uncertain of how to add figures and links correctly so that they are not broken in the built documentation.
  
• Lack of immediate feedback: When I post on a blog, the content is immediately online and - as feedback comes in - it's possible to make adjustments quickly. The above Pull Request was open for a month. (There were a lot of good discussions going on but it might feel more motivating to publish more quickly and improve incrementally).
  
  So the last two points come down to the process we currently have in place. Coming from a platform (Wordpress) where I can immediately see and verify the final results, the qgis.org documentation system makes me feel less certain about the quality of my edits and it takes much longer until corrections are visible online. (I know that I could build the documentation locally on my machine. I've tried with Richard's help in the past and failed to set it up on my machine.)
  

Nathan Woodrow, one of the core QGIS techies noted:

I personally find some of the technical issues as quite a blocker for people to help.  It's what stops me most of the time and I'm conformable with the tools, last time I tried on Windows I just gave up because it was too much work and I only have limited time these days.  I'm not sure what the solution here is but I don't know if moving to something like GitHub Markdown or Google Docs is the option, mainly because of it throws away a lot of the work we have already done.  Having said that though this is a pain point that might help address some of the community involvement if we can solve it.   It's not the only problem though like Alexandre said it's just not fun work at times and it can be hard to even write good docs when that is your job and you have a good platform to do it in.

Tim Sutton, one of the QGIS founders, reported:
Our main discussion points in the [QGIS Project Steering Committee] meeting were:

1. Current documentation approach is unsustainable (a few hardcore enthusiasts but not enough to cope with the rapid pace of development)
2. Inviting contributors needs to be substantially easier - I’m talking at the level of editing a google doc or word doc here. At the very least a GitHub markdown page that is instantly published as soon as you edit it.
3. Cameron has had a chat with me about employing technical writers to make the documents more cohesive - I think this is a good initiative but Cameron I think we need to get the fundamental issue of the editing platform sorted out first
4. Translations severely hamper our ability to switch to a more agile system (e.g. GitHub markdown based wiki or Google doc) - in the PSC call we want to surface the idea of doing away with translations and leave translation initiatives to outside communities (e.g. local user groups). PostgreSQL etc don't have the overhead of this.
5. Our documentation could be easier if the format was more structured - think something like editing a changelog entry here. Again we looked at the PostGIS / PostgreSQL examples here which have a very standardised format.

There were some sentiments in the PSC call to drop the documentation effort completely and leave it to all the various community members to deal with, but I think maybe my 1-5 points above make a better compromise of reduced overhead, more accessible platform for writing while still having docs in English at least. 

Stepping back from specific comments about tools and processes, I’m seeing a high effort-to-reward ratio for the external documentation community. Options to address this include:

1. QGIS docs core team to absorb the effort, either through funding or inspiring volunteers.
   
2. Help contributors get their content back into the core, likely with hand-holding, or possibly out-sourcing paid work to them.
   
3. Improve the efficiency of tools or improve our explanation of tools. While improved tools will be helpful, I think it is a generic problem faced by the whole open-source community. As such, I feel QGIS should reach out to the greater community to help solve it.

While there is acknowledgement that docs need improving, I feel there is a general under-appreciation of the effort required to:

1. Merge disparate documentation.
   
2. Increase doc quality from “verbose and okay” to “intuitive, obvious and concise”.
   

We need to start by articulating the problem and how we propose to solve it, which should help find both sponsors and volunteers.

QGIS Community questionnaire about docs

824 people answered a questionnaire about how you learn about QGIS. Anita Graser compiled results into the following tables.

Considering changes to doc writing based on survey insights varied from "do more" to "do less".
Tom Chadwin summarised the results as:

Questions 1-4 (quantitative)

• 70%’s first-choice is Googling/StackExchange, which dwarfs the < 20% choosing official docs
• The fact that Googling came top of search methods emphasizes that we need to pay attention to SEO in the official docs
• Fewer than 50% find their answers in the official docs “often” or “always”, 45% answering “sometimes”
• Official docs are underused – over 50% only consult monthly or less frequently (including “never”)

Question 5 (qualitative)

• Many find the official docs too abstract, and would prefer examples worked in
• Perhaps unsurprisingly, there is a lot of enthusiasm for video tutorials
• There is some criticism of the confusion of QGIS versions in the documentation, especially when deep-linked from a Google result

Paolo Cavallini argued:

To me this confirms my opinion: our manuals are of limited relevance to the community of users. IMHO we have two options here:

1. Re-haul the whole documentation so to make it the real reference.
2. Shrink it down to the bare minimum (mostly a list of the commands and functions available), leaving the fancy documentation out in the Wild World of the Internet.

Quite frankly (sorry, no offense for the huge and excellent work done until now), I do not see a realistic way of implementing (and, more importantly, to keep always up to date) the first option, so I tend to prefer the second one.

Summarising Alexandre Neto's longer analysis:

• It's clear that people often search for help a lot.
• In terms of QGIS Docs quality, ... [it] seems that definitely needs improving.
• As a documentation person myself, I naturally have to disagree with the idea that the project should ... simply resign to a shrunk version of the documentation and let the outside world provide the fancy answers to the users. ... IMHO, Good, precise, and updated documentation leads to more adoption and better user experience.
• An interesting fact: in two weeks open to answers, ... this questionnaire gathered more than 800 responses! To me, this alone says a bit about the importance that documentation has for our users.

Documentation best practices

I'm concerned people are searching for one approach to documentation when there should be multiple. In a highly regarded article in Tech Writing circles, Daniele Procida argues:

There is a secret that needs to be understood in order to write good software documentation: there isn’t one thing called documentation, there are four.
They are:

1. Tutorials, 
2. How-to guides, 
3. Explanations, and 
4. Technical references. 

They represent four different purposes or functions, and require four different approaches to their creation. Understanding the implications of this will help improve most software documentation - often immensely. ...

I expand on this in Inspiring techies to become great writers.

The audience for different doc-types have different needs:

• API References need to be accurate, unambiguous and up-to-date. Polish is a nice-to-have.
• Community forums are great for niche topics. Incorrect or dated information is tolerated.
• Quickstarts need to be accurate and polished, but need not reference the latest Bleeding Edge release. Aligning with the Long Term Release is acceptable.

This approach should be defined in an information architecture and an implementation strategy (which is yet to be created for QGIS). These should take inspiration from TheGoodDocsProject, an emerging community of technical writers building “best-practice templates and writing instructions for documenting open-source software.”

Matteo, what to focus on

Matteo from the core QGIS documentation team who volunteered to mentor QGIS Season Of Docs writers suggested:

• I really think that currently, we need to define precise roles (issue manager, reviewer, English reviewer, etc). IMHO the growing complexity of the last years made it difficult for us to convince other people to contribute (at least, I'm not able to convince people during training and other activities)
  
• +1 for the "community" evangelist (could be another role of above)
• -1 to change the framework (even if complex is too important)
• -1 to create other dedicated repositories with additional training material: just add another chapter to the existing manual
  
  Summary: without boring everyone that already knows the current situation, I really think we have to set up a clear workflow (for us [the QGIS documentation team] and newcomers) or else we will lose volunteers and other people that want to contribute to the project.

Andreas, Paulo and Tim, considering funding

Andreas Neumann, Paolo Cavallini and Tim Sutton weighed in on funding tech writers:

• Andreas: It is not primarily a problem of finding financial resources. Every year we assigned funds for documentation and in most years those funds haven't been used. Even if we would make more funds available to the team, I feel this wouldn't solve the problems the team is facing.
  
• Paulo: While I agree that we should keep on using our funds, and additional resources, as an incentive for documenters, I think this should be done with a clear plan in mind. If not done carefully, this move could discourage volunteers, not only in this area ("why should I volunteer, when another one is doing the same thing and is paid for this?"). Volunteer communities are hard to build, and easy to destroy. Replacing volunteers with employees can quickly become very expensive, and we should be sure we'll be able to raise enough money both in the short and in the long term to fully support the effort. I suggest working out a budget for this, to check how feasible this solution can be, before taking further steps.
  
• Tim: I have a different opinion on this. Based on our experience of paying developers I don’t think it has in any way reduced the volunteer contributions to the code base - on the contrary, it probably has incentivised those that we paid to donate lots more of their time. I am pretty sure that we will have similar experience in other areas of the project. I am more bullish on documentation and think that we should work enthusiastically to get one or more dedicated, full-time document writers in the QGIS project….over and over we here it is the most wanting part of the project. 

2019 QGIS Budgeted expenses

Highlights for me after discussing the 2019 QGIS budget with Tim Sutton were:

• The QGIS team run on an incredibly small and efficient budget. Strategic investment from external stakeholders should yield a significant return-on-investment.
• Programmers' daily rates were higher than tech-writers'. This concerns me as I question whether tech-writers' employed are suitably experienced. Typically, a good tech-writer is a programmer who has learned to write, or a writer who has learned to program.
• The €12,000 allocated to documentation won't go far if paid at standard tech-writer rates.
• Bug fixing (for programmers) was allocated five times more than docs (for writers).

Clarence, finding writers

To find writers, Clarence Cromwell, a tech writer, suggested:

Why don’t you reach out to the WriteTheDocs community. It has a slack group which includes a #job-posts-only channel. I’ve seen many budding tech writers asking how to break into tech writing. You could offer to mentor writers in git and software processes, in return for a review of documentation.

This is worth pursuing in order to bolster our existing tech writing team. However, for holistic documentation leadership, I feel we need more than individuals can provide on volunteer time alone. We could consider Google’s SeasonOfDocs model of paying a stipend (for a lead tech writer). Note: I feel this role needs sustained sponsorship; Google’s sponsorship is limited to three months.

Anne and Charlie’s research into open source success factors

There are insights from open-source research we can draw upon. Pertinent to this conversation is Charlie Schweik’s research into open source success factors and Anne Barcomb’s research into episodic volunteers. Their research highlights:
Factors which lead to a project’s success are:

• Leadership by doing.
  
• Clear vision.
  
• Well-articulated goals.
  
• Task granularity: Projects have small tasks ready for people who only can contribute small bits of time.
  
• Financial backing.
  

Successful strategies for working with episodic volunteers:

• Although Open Source episodic volunteers were unlikely to see their participation as influenced by social norms, personal invitation was a common form of recruitment, especially among non-code contributors.
  
• Episodic volunteers with intrinsic motives are more likely to intend to remain, compared to episodic volunteers with extrinsic motives.
  
• Episodic volunteers derive satisfaction from knowing that their work is used, enjoying the work itself, and feeling appreciated.
  
• Lower barriers to entry.
  
• Provide opportunities for social interactions.
  

I suspect the QGIS project has become so successful, and the community so large, that it appears daunting for someone on the fringe who might want to join. They don’t feel worthy, are not sure how to break into the inner circle, or feel someone else will do the work if they don’t. It will likely be worth rekindling a supportive and personal culture within our community.

Learning from the OSGeoLive experience

I think it is worth considering the formula used in the OSGeoLive project to attract hundreds of episodal contributors, many of whom have been working on docs. It is summarised here:

• Start with a clear and compelling vision; inspiring enough that others want to adopt it and work to make it happen.
  
• This should be followed by a practical and believable commitment to deliver on the vision. Typically this is demonstrated by delivering a “Minimum Viable Product”.
  
• Be in need of help, preferably accepting small modular tasks with a low barrier to entry, and ideally something which each person is uniquely qualified to provide. If anyone could fix a widget, then maybe someone else will do it. But if you are one of a few people with the skills to do the fixing, then your gift of fixing is so much more valuable, and there is a stronger moral obligation for you to step up.
  
• Ensure that every participant gets more out of the project than they put in.
  
• Avoid giving away free rides. If you are giving away something uniquely valuable; and it costs you time to provide that value for your volunteers; then it is ok to expect something of your volunteers if they wish to get something in return.
  
• Use templates and processes to facilitate domain experts working together.
  
• Reduce all barriers that may prevent people from contributing, in particular, by providing step-by-step instructions.
  
• Set a schedule and work to it.
  
• Talk with your community regularly, and promptly answer queries.
  
• And most of all, have fun while you are doing it. Because believe you me, it is hugely rewarding to share the team camaraderie involved in building something that is much bigger and better than you could possibly create by yourself.
  

My assessment

The QGIS documentation community appears overwhelmed and seems to need help with:

• Articulating doc challenges to the community, potential contributors, and potential sponsors;
• Defining a clear vision and roadmap;
  
• Coordination and project maintenance;
  
• Breaking large daunting challenges into small tasks that can be tackled easily by volunteers;
  
• Capturing community good-will and offers of assistance,
  
• Inviting people to get involved one-on-one and then mentoring them;
  
• Periodic catchups;
  
• Outreach and evangelising;
  
• Attracting satellite initiatives into the core;
  
• Keeping up with a rapidly innovating software baseline;
  
• Documentation tooling and processes;
• Sustaining initiatives, and orphaning unmaintained documentation.
  

Most importantly, I think the QGIS docs team is missing sufficient people with the bandwidth, drive and personality to drive this agenda. I feel there is likely to be quite a bit of inertia required to ramp-up such a team, but I think it is worth investing in, as I think QGIS will benefit greatly once it is set up.

Suggestions

These suggestions are presented in my proposed order of priority.

1. Community evangelist / coordinator

I believe QGIS should engage a “community evangelist and coordinator”, tasked with:

• Inspiring others.
  
• Capturing untapped goodwill from within the QGIS community and potential business sponsors.
  
• Embracing and extending QGIS’s supportive culture.
  
• Reaching out one-on-one and personally inviting people to join, then pro-actively supporting them during their onboarding experience.
  
• Helping to reduce barriers to entry.
  
• Defining and managing a roadmap, with milestones and schedules.
  
• Coordinating community collaboration.
  
• Supporting documentation development, deployment and reviews, as required.
  

We should look for someone who:

• Is friendly, approachable, and community-minded.
  
• Is likely a notable and experienced member of an open-source community, whose opinions are respected and hold weight within the community.
  
• Is technical enough that they can help a newbie with git and doc tools.
  
• Is business savvy and able to persuade business people about the value of collaboration.
  
• Presents competently at conferences.
  

Getting the right person for this role will be very important, as they will influence the culture of the rest of the team.
Sustained funding should be sourced for this role as it will be difficult to resource on volunteer labour alone.

2. Vision and roadmap

Common feedback from volunteers was not knowing how to give back. We appear to be lacking the vision and roadmap which open-source research suggests is important. Alexandre Neto noted:

• Unfortunately, this issue list is the only thing we have [re roadmap].
  

I suggest defining a vision and roadmap, which can then be referenced to help prioritise direction.

3. Information architecture review

I get the impression that QGIS documentation is quite good, but it hasn’t been audited by a senior technical writer/information architect. I suggest a senior information architect be engaged for a once-off engagement to set up QGIS’s approach to documentation. We should consider:

• Best practice document types, templates and writing styles, tailored for different target audiences.
• Documentation architecture.
• Quality expectations.
• Maintenance strategy.

Ideally this information architect would be the same person as the sustained technical writer role (in order to retain project knowledge).

4. Engage a technical writer

People from the core doc team have noted that much of the QGIS documentation is written by software developers, or power users (without formal writing training). For many, English is a second language.
I suggest a sustained technical writer role be set up to:

• Reviews all new documentation generated by the community.
  
• Work with the authors to ensure it fits with QGIS’s writing guidelines and quality standards.
  

This will require sustained resourcing, which I suggest be supported by a stipend, in-kind contribution from a company, or similar.

5. Attract external QGIS docs into the core

There has been discussion about the significant amount of external docs and training material which is not coming back into the core QGIS. I’d suggest:

• Publicly stating the value we place on internal rather than external documentation.
  
• Encourage sponsors, and those paying for training to make use of internal rather than external material.
  
• Reach out to external material providers and work with them to bring their material back into the core. Acknowledge extra effort required to do this. (It will be short term pain for long term gain.)
  
• Monitor community activity and opportunistically support people to bring their external docs back into the core.
  

6. Ignore the tools for the moment

It has been noted that the git/sphinx documentation toolchain is a barrier to entry for people coming into docs. While acknowledging the problem, I suggest leaving it for the moment as we have higher priority problems which we can resolve and should focus on. Leave this for the wider open-source documentation community to solve.

Thankyou Felicity

Felicity Brand,
Season of Docs Technical Writer

Thank you Felicity for everything you have given our OSGeoLive community during Google's Season of Docs. It has been much more than your review and update of 50 odd OSGeoLive Quickstarts:

• You have always been friendly, warm and supportive, even during the witching hour teleconference calls we asked you to join. 
• You have touched so many of us. With each of the fifty-odd quickstarts you updated, you were introduced to a new bunch of people.
• You've stepped up and volunteered to help with whatever needed doing, be it big and challenging, or small and mundane.
• You went well beyond your Season of Docs mandate, both within the OSGeoLive project and additionally in TheGoodDocsProject. (Your initial proposal didn't account for all the community interaction you ended up doing).
• You've brought a fresh perspective to many of our conversations, often introducing innovative ideas. The OSGeo Lexicon Committee started with your suggestion. 
• You balanced practical and sensible pragmatism with witty and playful humour. It made working with you fun instead of a chore.
• And you've managed to do all this while mothering, without missing a beat. I was impressed when you introduced your well-behaved toddler as she walked into one of our video conferences.

So on behalf of the OSGeoLive community, thank you.

On Fri 29 Nov 2019, Felicity Brand wrote:

Dear all,
The Season of Docs program officially ended yesterday.
Phew, what a ride! Thank you all for being so supportive and helpful.
It has felt like an intense period of learning and doing - I'll be glad to have a rest.
You can read my summary report.
I have really enjoyed this opportunity to learn about open source, your community and geospatial software. I have also picked up skills in ReST, sphinx and taken my git to the next level.
This is not goodbye from me. I will have a bit of a rest first (so
might be offline for a few weeks).

• There are 30 open pull requests for Quickstarts that I'm hoping you
• (and Cameron) can help me finalise.
• There are also comments on some of the Quickstarts that I want to
• follow up in a bit more detail.
• I discovered a doc issue in trac that I can probably address as well.

I know that the Google Code-In program might be working with some of the
Quickstarts, so that is an exciting opportunity to put some of them to the test.
Thanks
Felicity

Launching TheGoodDocsProject

Discussing templates

What is TheGoodDocsProject?

“Best practice templates and writing instructions for documenting open-source software, 
collated from the best ideas within the documentarian community.”

Incidentally, the templates will be directly applicable for other domains too. So far, we have draft templates for:

• Overviews
• Quickstarts
• References
• Discussions
• How-tos
• Logging
• References
• Tutorials

We officially launched our first 0.1 release at the recent WriteTheDocs - Australia conference and were honoured to have ~ 50 documentarians review our templates in a two-hour "fix-it" session.

What next?

• Over the next few weeks, we will be copying suggestions into our issue tracker.
• By ~ March 2020, we plan to publish a 0.2 release, incorporating suggestions to date.
• We hope many documentarians will join us to help improve these templates.
• And then, with a bit of luck, people will start using the templates.

Highlights from WriteTheDocs - Australia

A bunch of ideas were presented at the WriteTheDocs conference which I'd love to see captured in our templates and shared with others. These included:

Inclusivity Dave presented an amusing and informative lightning talk using sign language, presented through his (female) interpreter. He started, "You are probably wondering why I sound like a woman? I was born deaf ..." He has since volunteered to add best practice inclusivity into our templates.

Dave Parker

Including international communities Alec discussed cross-cultural and cross-language inclusivity, along with challenges and potential solutions. It will be good to see these ideas woven into our templates.

Alec Clews

Empowering writers Leticia talked in inspiring terms about practical techniques to empower technical writers. I've seen many dis-empowered technical writers and I'd love to see techniques to address this captured and disseminated.

Leticia Mooney

Psychological safety Riona challenged conventional thinking about open source communities. She discussed technical and social barriers to technical fields, and linked psychological safety, knowledge sharing and good documentation to building more egalitarian societies.

Riona MacNamara

Information architecture Elle discussed building information architectures, and avoiding social and technical biases. We have only scratched the surface of this topic in TheGoodDocsProject and more expertise is required.

Elle Geraghty

Storytelling Jon and Tim wove a compelling tag-team story about narrative in game design. If we can get technical writing to be as interesting as this, readers will absorb information more effectively.

Jon Manning and Tim Nugent

Tech Writing Workshop Sarah presented Google's internal Tech Writing 101 workshop for Developers, which Google plan to share publicly. It will be great to harmonise this with TheGoodDocsProject.

Sarah Maddox

Release Notes Alex drew inspiration from horror stories when explaining best practices for release notes. This is worth watching, just for entertainment value - but we should also be capturing these insights into our templates.

Alex Koren

The pitch Mat recorded a post-conference talk about harnessing the post-conference arfterglow (password: WTD2019). At the conference, he was leading conversations about building a business case for us all to take back to organisations explaining the value we all gain by collaboratively sharing and maintaining templates, including: Training users to become familiar with standard documentation patterns. Reducing ramp-up time for documentarians. Supporting cross-domain expertise.

Mat Walker

What next for you?

Are you inspired? Want to help writers become more effective, and by extension, help share the world's knowledge? Come introduce yourself to the rest of us at TheGoodDocsProject.

Starting an OSGeo Lexicon Committee

Disparate glossaries of geospatial terms are spread across OSGeo projects, OSGeo communities, the OGC and ISO TC211 standards organisations. Making sense of and managing these terms is messy. So this week we have initiated an OSGeo Lexicon Committee with goals of:

• Managing OSGeo related terminology, 
• Setting up and applying best practices, 
• Providing glossaries for publishing to various OSGeo projects and communities, 
• Facilitating translations between terms,
• Coordinating with OGC and ISO TC 211 standards communities.

Does this interest you? Check out our wiki page or contribute to discussions on our email list.

TheGoodDocsProject Fixit workshop

Are you coming to the Write the Docs conference in Sydney, Australia, 14 and 15 November? If so, I hope you will feel inspired to join our GoodDocsProject FixIt workshop.

We started TheGoodDocsProject to create best-practice templates and writing instructions for documenting open-source software. With our alpha release, we’ve created some common templates which we’d love people to test, break and improve.

Topics:
Diverse contributions encourage. We hope to cover:

• As a user or developer or writer: Try writing instructions for your favourite application based on one of our templates. Give us feedback.
• As a technical writer or information architect: Review one of our templates, try and find limitations, suggest modifications, offer sections to add, debate improvements.
• You have another idea: Let’s hear it.

Schedule:

• 5 min: Introduction.
• 30 min: Break out into groups brainstorming topic.
• 15 min: groups present back to everyone.
• 30 min: Break out2.
• 15 min: Groups present2.
• 5 min: wrapup and thanks

Logistics:

• We will have flip charts and pens for brainstorming, and
• We will use track changes and Google Docs. (Bring a laptop.)
• Source templates: https://github.com/thegooddocsproject/templates

Post Fixit:

• Join our email list or slack chat.
• Help us triage ideas into our kanban board and issue tracker.
• Contribute further.

Who are we:

• Cameron Shorter, Business Analyst by day, Open Source Developer and accidental Technical Writer by night.
• Jared Morgan, Senior Technical Writer at Squiz
• Erin McKean, Docs Advocacy Program Manager at Google
• Felicity Brand (attending remotely), Technical Writer for Google Season of Docs
• Sanket Totewar, Technical Writer and a few more things
• Others from TheGoodDocsProject

Scrappy Collaboration

Coordinate changes over time due to tectonic plate movement.

Confession: We broke government conventions, worked beyond our agency’s mandate, attended an international technical committee meeting on personal time, uncovered significant negative impacts for unsuspecting users in Australia’s multi-million dollar mapping program, and retrospectively received glowing praise and thanks.

Welcome to “scrappy collaboration”.

$225 million has been committed to improving Australia’s GPS positioning tenfold, to 5cm accuracy. However technical shortcuts previously embedded in web-mapping software, which ignore tectonic plate movements, are causing meter level map misalignments in web-mapping! This is an international problem which is being exposed by Australia’s advanced mapping program. Needless to say, misaligned maps are confusing for users and unacceptable for Australian mapping agencies.

When NSW Spatial Services discovered the project risks involved, and the minimal existing focus on the problem, we joined other Australian states, and worked beyond our agency’s mandate, to look for a solution.

An interrelated problem cluster
With investigation, we unravelled a cluster of interrelated problems:

• Some of which had deep technical challenges;
• With varying implications to diverse user groups and use cases;
• Which were difficult to understand and explain;
• With experts regularly talking past each other, each considering a different part of the problem;
• With multiple holistic strategies possible (each strategy offering significantly different costs and benefits for each stakeholder group);
• With high impact implications for software, data and users;
• Requiring national and international collaboration between diverse stakeholders to resolve effectively.

Not my responsibility
The investigation turned out to be a significant time sink, distracting us from our core roles. If following a government playbook, we should have left the responsibility to others: Our software vendor, international standards foundations, national mapping organisations, ...

But we knew we had a moral obligation to step up. Our team understood the overarching problem; had access to geodetic surveyors and software engineers; and had personal connections with international experts and influencers from open-source and standards communities.

A scrappy approach
A complex, interdependent problem cluster has a high barrier to entry. Each stakeholder only understands and can influence a subset of the issues. No one has a business case to solve the greater problem for everyone. This is difficult to solve with traditional top-down management approaches.
Instead, we adopted a scrappy, very personal approach, trading our time, and collected knowledge with collaborators to achieve common goals.

• We very publicly admitted the embarrassing problems we were facing in technical forums, and asked for help.
• We shamelessly drew down favours from our friends in open source communities, government agencies, industry and standards organisations.
• We spent extra effort coordinating contributions, answering questions, and addressing related problems relevant to our collaborators (because that is what friends do).
• We concisely compiled descriptions of the problems, along with suggested solutions.
• Joel Haasdyk, our datum modernisation manager and geodetic surveyor, then used personal leave to travel halfway around the world to present recommendations to the standards technical committee. 

When collaborating, we applied many of the hallmark characteristics prevalent in open source communities:

• Respect the time of your collaborators; be concise in messaging.
• Lead by example; put in the hard work.
• Reach out to people; be vulnerable and ask for help.
• Be clear about what you need and from whom.
• Care about your collaborators; help solve their problems too.

Tap into your humanity - it has tangible business benefits.

Impact
The impact we achieved with collaborators was apparent from the reports coming from the standards community. Scott Simmons, Executive Director of the OGC Standards Program reported:

“I don’t know if I have ever seen a better and more compelling tag-team describe and lead discussion on a topic than Joel and Roger's presentation on the time-dependent limitations inherent in most OGC spatial standards. I received many emails and face-to-face comments from attendees afterwards stating that it was so valuable to understand the problem and see that there is a solution. Resulting from the presentation, the majority of OGC Standards Working Groups now understand the necessity to update their standards to account for time-dependence of coordinates, with some already initiating updates.”

Future standards development will consider tectonic plate movement and will start to address Australia’s map misalignment problems impacting our datum modernisation program. We are at the start of the journey, there is still plenty more to do, but we now are heading in the right direction.

Counterintuitive solution
So, when faced with complex problems, consider expanding scope rather than constraining it. Consider solving the problems of others as well as your own. Attract the collective wisdom and resources of collaborators and you'll likely find your solutions are more impactful, strategic and sustainable.

About the author
Cameron Shorter has been a Geospatial Business Analyst at NSW Spatial Services.

Time-dependent datum problems

Technical shortcuts taken in mapping systems, which ignore tectonic plate movements, are causing meter level map misalignments, most notably in web-mapping. It is caused by a cluster of interrelated problems, which we attempt to comprehensively catalogue in this article. For a more concise summary, look here.

Caution: This article is full of acronyms and complex geospatial mapping concepts. It is primarily aimed at technical experts in this domain. It provides a snapshot in time, collated from a range of people. It is expected to have inaccuracies and not be up to date with latest thinking.

Main problems

Web-mapping’s illusion of accuracy

• To date, web-mapping has provided an illusion of accuracy; all map layers have typically been aligned; but they have all been aligned to the WGS 84 epoch (and tectonic plate location) associated with the regional datum. This is being exposed as misaligned maps where a nation creates new national datum(s) as part of datum modernisation programs.
• Australia is experiencing this problem now, due to an update of the nation datum from GDA94 to GDA2020.

WGS 84 datum

• WGS 84 Web Mercator is the defacto standard used in web-mapping.
• WGS 84 has inherent ambiguity which in practice introduces low accuracy:
• It has multiple realizations (it has been re-defined six times to date).
• In web-mapping, the dynamic WGS 84 datum is being used as if it were a static.
• In contrast, GPS positioning delivers point positions in the WGS 84 dynamic datum at the ‘current’ epoch (date).
• Different epochs of WGS 84 have been adopted, in practice, in different regions of the world.
• It uses step-wise annual updates rather than being dynamic at any time-scale.
• … and more
• Static tiled basemaps around the world provides a major barrier to moving away from the WGS 84 Web Mercator projection, as all existing maps would need to be reprojected, or face misalignment with new datums.
• Adopting a new web-mapping datum would be costly, requiring extensive retiling of all the world’s tiled maps and stored datasets.

Spatial Standards

• To accurately describe coordinates, spatial standards describing position in a dynamic environment must accurately describe:
• The coordinates (e.g. Latitude, Longitude, Height),
• The datum (reference frame) and
• The epoch (date) at which the coordinates are expressed. (This is often different to the epoch at which coordinate was first observed!)
• Epoch is implicitly defined for a static datum (e.g. 1994.0 in GDA94) and needs to be explicitly defined in a dynamic (time-dependent) datum (e.g. ITRF1992@1994.0). Unfortunately, epoch is not incorporated in legacy standards, so updating the way in which dynamic datums are defined and applied must be addressed in the full stack of OGC standards.
• Even the most widely used of OGC standards, like the Web Map Service (WMS) standard, do not account for time dependence when selecting a dynamic Coordinate Reference System (CRS).
• File formats such as KML and GeoJSON only support storing coordinates in WGS 84. This needs to be considered if addressing future accuracy requirements.

Communication challenges

Collaborating to solve mapping challenges is hindered by communicating a difficult problem, with complex terminology, to a diverse audience. For example:

• There is a general lack of use and understanding of ‘accuracy’, and how the choice of transformation and projection can affect accuracy.
• Authoritative documents use terms differently in different documents, or use multiple terms for similar concepts. For example:
•  “datum”, “reference frame”, and “CRS”, all have similar meanings.
• There are several different concepts of epoch which are required to accurately describe coordinates in a dynamic datum. For example, a coordinate can be observed at epoch 2017.21, but recorded or expressed at epoch 2018.5 in a reference frame of ITRF2014.

Action Plan

Urgency

There is a need for urgency because:

• Australia has 1.8 metre map misalignments now.
• North America plans to introduce new datums in 2022.
• High-precision mass-market positioning is just around the corner.

Technically

We need to:

• Solve web-mapping dynamic/static misalignments.
• Advance time-dependent mapping.
• Update standards.
• Roll into software.
• Improve world mapping practices and user outcomes.

Community mobilisation

While we have technical challenges, we have a harder outreach and community mobilisation challenge. Success depends on collaboratively bringing software, data and geospatial communities with us. We need core buy-in from:

• Leading tiled map providers,
• Spatial data providers,
• Software venders, and
• Government mapping agencies.

We need clear messaging that Julie, the GIS officer, and Joe, the graduate software developer, can use to explain the datum modernisation story.

Web-mapping business requirements

Let’s revisit the business requirements of accurate web-mapping to help us assess the viability of current and proposed solutions.

Legend for current support:

• Supported
• Partially supported
• Not supported

Geospatial requirements:

1. Support high accuracy mapping.
2. Account for time dependence resulting from tectonic plate movement.
3. Provide accurate map alignment when displaying map layers from disparate sources.
4. Support calculation, publishing and application of accuracy metadata.
5. Datasets must have a nominated datum (reference frame) and epoch (date). Epoch is implicitly defined for a static datum and needs to be explicitly defined in a time-dependent datum.

Usability requirements:

6. Datasets must be able to be transformed between datums and epochs.
7. Software applications shall continue to be responsive and performant for users, including for low-spec clients, such as browsers and mobile apps.
8. Web-mapping services shall continue to scale efficiently to support multiple users.
9. Users shall continue to be able to save map vector data as static files (such as KML, GML, GeoJSON), and render at a later point in time.
10. Accuracy information should be embedded in decision workflows.
11. Spatial expertise required to be learned by software implementers should be minimised.

Software Implementation requirements for web-mapping:

12. Tiled web-mapping shall continue to be supported to address performance and scaling. In practice, this freezes maps in time.

Derived CRS requirements:

13. Adopt a CRS, datum and projection for publishing map layers with the following characteristics:
1. Defined accurately.
2. Receives accurate transformations from key source CRSs (in particular, from national/regional datums).
3. Aligned with a conventional fixed epoch (date).
4. Receives coordinates converted from other epochs.
5. Applicable at global and local scales.
6. Near-universally adopted within web-mapping contexts.

Main problems

Web-mapping’s illusion of accuracy

• To date, web-mapping has provided an illusion of accuracy; all map layers have typically been aligned; but they have all been aligned to the WGS 84 epoch (and tectonic plate location) associated with the regional datum. This is being exposed as misaligned maps where a nation creates new national datum(s) as part of datum modernisation programs.
• Australia is experiencing this problem now, due to an update of the nation datum from GDA94 to GDA2020.

WGS 84 datum

• WGS 84 Web Mercator is the defacto standard used in web-mapping.
• WGS 84 has inherent ambiguity which in practice introduces low accuracy:
• It has multiple realizations (it has been re-defined six times to date).
• In web-mapping, the dynamic WGS 84 datum is being used as if it were a static.
• In contrast, GPS positioning delivers point positions in the WGS 84 dynamic datum at the ‘current’ epoch (date).
• Different epochs of WGS 84 have been adopted, in practice, in different regions of the world.
• It uses step-wise annual updates rather than being dynamic at any time-scale.
• … and more
• Static tiled basemaps around the world provides a major barrier to moving away from the WGS 84 Web Mercator projection, as all existing maps would need to be reprojected, or face misalignment with new datums.
• Adopting a new web-mapping datum would be costly, requiring extensive retiling of all the world’s tiled maps and stored datasets.

Spatial Standards

• To accurately describe coordinates, spatial standards describing position in a dynamic environment must accurately describe:
• The coordinates (e.g. Latitude, Longitude, Height),
• The datum (reference frame) and
• The epoch (date) at which the coordinates are expressed. (This is often different to the epoch at which coordinate was first observed!)
• Epoch is implicitly defined for a static datum (e.g. 1994.0 in GDA94) and needs to be explicitly defined in a dynamic (time-dependent) datum (e.g. ITRF1992@1994.0). Unfortunately, epoch is not incorporated in legacy standards, so updating the way in which dynamic datums are defined and applied must be addressed in the full stack of OGC standards.
• Even the most widely used of OGC standards, like the Web Map Service (WMS) standard, do not account for time dependence when selecting a dynamic Coordinate Reference System (CRS).
• File formats such as KML and GeoJSON only support storing coordinates in WGS 84. This needs to be considered if addressing future accuracy requirements.

Communication challenges

Collaborating to solve mapping challenges is hindered by communicating a difficult problem, with complex terminology, to a diverse audience. For example:

• There is a general lack of use and understanding of ‘accuracy’, and how the choice of transformation and projection can affect accuracy.
• Authoritative documents use terms differently in different documents, or use multiple terms for similar concepts. For example:
• “datum”, “reference frame”, and “CRS”, all have similar meanings.
• There are several different concepts of epoch which are required to accurately describe coordinates in a dynamic datum. For example, a coordinate can be observed at epoch 2017.21, but recorded or expressed at epoch 2018.5 in a reference frame of ITRF2014.

Main options

Option 1: Do Nothing - Low accuracy web mapping

• Accept poor accuracy and misaligned maps in web-mapping.
• This is unacceptable for Australia.

Option 2: Call a spade a spade

• By convention, acknowledge that WGS 84 is currently being used as an accurate static datum in web-mapping.
• This will likely be accompanied by defining new EPSG code(s) to recognise the currently implemented datum.
• We would need to acknowledge that WGS 84, as used by the GPS system, is a different datum entirely.
• In Australia, we would acknowledge that WGS 84 is aligned with our old GDA94 datum.

Option 3: Adopt a common fixed epoch for web mapping

• Adopt a common fixed epoch to facilitate static tiled maps of the world. This epoch might be periodically re-baselined into pre-planned “epochs of convenience”.
• (Currently, maps are tiled into different epochs in different regions.)
• Note: This would involve retiling all the world’s basemaps to the new common epoch.
• Note: if committing to retiling all maps, we can consider additional improvements to map tiles, such as moving away from the less accurate spherical assumptions of the earth used by WGS 84 Web Mercator.

Option 4: Adopt dynamic datum

• Adopt a conventionally agreed datum which is explicitly dynamic and internationally recognised, such as ITRF. This will allow new data to be more easily moved through time back to the common epoch.
• Note: this would involve retiling all the world’s basemaps to the new datum.

Option 5: Use client based datum/epoch conversion tools

• Allow users to obtain coordinates in (any) desired datum and/or epoch by transforming (x,y) mouse coordinates on screen, rather than expressing and storing underlying base-maps in multiple datums. (Move Mohammad to the mountain rather than moving the mountain to Mohammad.)

Update spatial standards

• Almost all options will require updates to standards to cater for time-dependence. For example:

• Web Services such as WMS, WFS, etc., which accept a dynamic CRS as a parameter, will additionally need to accept a CRS epoch field.
• Data transfer formats such as KML and GeoJSON which only store coordinates in the WGS 84 datum will need to be extended to support more accurate datum(s).

Clear messaging

• Investing in clear messaging should help with communication challenges.
• We should simplify developer and user experiences by abstracting complicated concepts behind simple implementations.

Action Plan

Urgency

There is a need for urgency because:

• Australia has 1.8 metre map misalignments now.
• North America plans to introduce new datums in 2022.
• High-precision mass-market positioning is just around the corner.

Technically

We need to:

• Solve web-mapping dynamic/static misalignments.
• Advance time-dependent mapping.
• Update standards.
• Roll into software.
• Improve world mapping practices and user outcomes.

Community mobilisation

While we have technical challenges, we have a harder outreach and community mobilisation challenge. Success depends on collaboratively bringing software, data and geospatial communities with us. We need core buy-in from:

• Leading tiled map providers,
• Spatial data providers,
• Software venders, and
• Government mapping agencies.

We need clear messaging that Julie, the GIS officer, and Joe, the graduate software developer, can use to explain the datum modernisation story.

Detailed problem list

In this section we provide a “shopping list” of problems which hinder the adoption of high-accuracy mapping. It builds from experiences in implementing Australia’s datum modernisation program and ties into international challenges. It touches on time-dependence, standards, web-mapping, and map misalignment challenges.

1. WGS 84 Datum ambiguity

As WGS 84 is the defacto standard used in web-mapping, it is worth understanding the relationship between projection, datum and CRS:

• The WGS 84 Web Mercator projection (EPSG::3857), technically called the Pseudo Mercator projection, is based on
• The WGS 84 Coordinate Reference System (EPSG::4326), which is based on
• The WGS 84 Datum ensemble (EPSG::6326), which is an ensemble of
• Six WGS 84 Datum realizations.

Figure: Basis of WGS 84 from EPSG Registry 

As explained by Roger Lott, the formal WGS 84 datum is not unique; it has been updated six times to date. Each update (called a realization) refines the datum’s alignment with the earth to account for improved measurements of the earth’s shape (which is different to tectonic plate movement). Each WGS 84 formal realization is dynamic. I.e. coordinates change with time. To be unambiguous, coordinate metadata needs to include coordinate epoch (date).

Problem 1.1 Non-unique datum:

The WGS 84 datum ensemble (EPSG::6326), represents any or all of the formal WGS 84 realizations, without distinction. It should be considered to be static, with metre level uncertainty.

Problem 1.2 WGS 84 coordinates are stepwise updated

Figure: Stepwise jumps in WGS 84

WGS 84 coordinates for ground stations and satellite ephemerides (and therefore the datum definition) are re-computed each year, mid-year. WGS 84 actually moves in a step-wise fashion. If this is not taken into account, then positioning problems can result. (Refer to ISO 19161-1.)

2. Modernising OGC Standards

To accurately describe coordinates, they must include the coordinates, a datum (reference frame) and epoch (date). Epoch is implicitly defined for a static datum and needs to be explicitly defined in a time-dependent datum.

The base OGC and ISO standards document, OGC Abstract Specification Topic 2: Referencing by coordinates, introduces modern geodetic time-dependent concepts, including:

• Extension to describe dynamic geodetic reference frames.
• “Datum ensembles” which allows grouping of related realizations of a reference frame.
• In a dynamic CRS, the coordinate epoch is stored as an attribute of a set of coordinates, it is not part of the CRS. (A set of coordinates could be points, features, or a dataset.)

Problem 2.1: Coordinates in Dynamic CRS are ambiguous

To date, coordinates which have been stored in a dynamic CRS have been ambiguous, as the observation epoch typically has not been recorded, and epoch is not stored in the CRS either.

This is proposed to be addressed by including epoch with coordinates when using a dynamic CRS.

Problem 2.2 Implementation challenges with time-dependent standards

• We will need to consider how new time-dependent coordinates can be introduced without breaking backward compatibility with existing standards, software, and datasets.
• There is likely to be performance and implementability implications depending on whether epoch is tied to the level of a point, feature, or dataset.

These should be considered before continuing with standard definitions.

Problem 2.3: OGC standards which don’t address time dependence

Core OGC standards still don’t address time-dependent coordinate referencing. This includes standards such as the GML format, WMS, WFS and so on.

Problem 2.4 High impact to update software to support time dependence

Changing the core coordinate definitions to support time dependence is likely to have a significant impact on software implementations, and stored datasets.

Problem 2.5: KML standard uses WGS 84

The OGC KML 2.3 standard, initially defined by Google, is used widely for storing map data. By definition, KML stores coordinates in the ambiguous WGS 84 CRS (EPSG::4987), and doesn’t record epoch (date), resulting in dataset inaccuracies.

Problem 2.6: GeoJSON standard uses WGS 84

The current GeoJSON standard (dated August 2016) states:

The coordinate reference system for all GeoJSON coordinates is a geographic coordinate reference system, using the World Geodetic System 1984 (WGS 84) 

WGS 84 is not explicitly defined (by an EPSG code or similar) and could be any of the WGS 84 realisations. Likewise, epoch (date) is not recorded, resulting in further dataset inaccuracies.

Problem 2.7 Non-standard GeoJSON in use

While the GeoJSON standard only allows a CRS of WGS 84, “in the wild” implementations appear to allow implementation of different CRS values (as per earlier drafts from April 2006 of the GeoJSON standard). For instance, the OpenLayers browser based client supports a GeoJSON CRS attribute. See example:

Problem 2.8 Inconsistent reference to CRS

As explained in a Request for Change for content negotiation by CRS, inconsistency in referring to CRS is causing implementation problems:

Currently, many GeoJSON API based implementations support communication of CRS. The GeoJSON standard does not provide a standardized method of negotiating CRS. As a result, each implementation varies with respect to both how a CRS is requested, as well as the method used to communicate which CRS was used to locate the data returned. Most, if not all, implementations use the body of the request or response for communication.
There are three issues with this implementation option. Firstly, the name of the parameters used in requests and responses vary between implementations. For example, the variation of terms used to specify the name of the parameter include, but are not limited to, "SRS" (the acronym for Spatial Reference System) and "CRS". Secondly, the semantics of the value used in requests and responses also varies between implementations. For example, the variation of terms used to specify the name of the value for the World Geodetic System of 1984 include, but are not limited to, "WGS84", "4326", and "EPSG:4326". Finally, the value returned by a service does not necessarily support the ability to dereference the CRS.

3. Publishing static maps using a dynamic datum

Organisations have been inadvertently freezing dynamic maps in time when the maps are published. This prevents the feature’s coordinates from changing over time as should happen for coordinates stored in a dynamic datum.

Problem 3.1: Tiling dynamic datum maps

When maps are tiled, they are frozen in time. As the defacto standard for tiled maps is the dynamic WGS 84 Web Mercator projection, we are creating a static/dynamic map misalignment problem.

Problem 3.2: Publishing and distributing dynamic datum maps

Some organisations have been publishing map data in the ambiguous WGS 84 datum. Also, most web services are configured to allow users to download maps in various datums, including WGS 84. When downloaded, these maps are frozen in time, creating the static/dynamic map misalignment problem. The problem is often exacerbated by technology and data providers who encourage use of WGS 84 to facilitate interoperability with map tiles.

4. Web-mapping has been topologically aligned, but not accurate

To date, web-mapping has typically appeared more accurate than the metre level accuracy associated with the WGS 84 datum ensemble. Map layers have been topologically aligned with each other, creating an illusion of accuracy. However, all layers have all been equally misaligned. This occurs because:

• National mapping agencies store maps accurately in their region’s official static datum.
• Until recently, a region’s official datum only had one transformation to the WGS 84 datum.
• As a time-dependent transformation was not applied, this locked the target datum to a point in time, effectively emulating a static datum.
• For instance, the transformation from CRS GDA94 (EPSG::4283) to CRS WGS 84 (EPSG::4326), (Transform: EPSG::1150) locks transformed coordinates to the year 1994.0.

Problem 4.1: Coordinates systematically misaligned:

Maps sourced into the WGS 84 datum ensemble are regularly systematically misaligned.

Problem 4.2: Lack of awareness:

The poor accuracy of WGS 84 has been masked from users as they see topologically aligned maps. Effectively, there has been a communication gap between the geospatial community and software developers in understanding this problem.

Problem 4.3: WGS 84 being used as if it were an accurate static datum

In web-mapping, WGS 84 is regularly implemented as an accurate static datum. Datasets are consistently transformed from ‘ground truthed’ maps in national/regional datums via null transformation to WGS 84, resulting in consistently misaligned maps, which create the illusion of accuracy.

Problem 4.4: Misalignment with GPS sourced points

WGS 84 features sourced from current GPS devices (such as mobile phones), will be misaligned with WGS 84 maps transformed from a region’s official static datum. The tectonic plate coordinate shift will be exposed.

Problem 4.5: Different epochs in each region:

WGS 84 uses null transformations to datums at different epochs in different regions. In practice, this means WGS 84 is aligned with different epochs (dates) in different regions. For example:

• Australia’s epoch of WGS 84 is aligned with GDA94 (EPSG::4283), which is aligned to ITRF92 at epoch 1994.0 (Transform EPSG::1150).
• USA’s epoch of WGS 84 is aligned to CRS NAD83 (EPSG::4269) from 1986.0 (Transform EPSG::1188) etc.

This will make it difficult for the world to agree to lock in a previously used WGS 84 epoch for web-mapping.

5. Australia’s misaligned web-maps

Australia had defined our static GDA94 datum as coincident to ITRF1992 on 1994-01-01 which was coincident to WGS 84 (G730) at that time, leading to defining a null transformation between GDA94 and WGS 84 (EPSG::1150).

However, due to Australia’s fast-moving tectonic plate (~7cm per year), coordinates in the dynamic WGS 84 datum drift apart from static GDA94 coordinates. In 25 years, this equates to ~ 1.8 metres.

Modern GPS positioning, which uses WGS 84 coordinates, and is projected to have centimetre level accuracy within the next few years, now have metre level misalignment with GDA94 sourced basemaps.

We have “mountain to Mohammed or Mohammed to mountain” options to address this misalignment:

1. Move basemaps and datum to align with current GPS (WGS 84) positions. (Move mountain to Mohammed).
2. Adjust coordinates in GPS mobile devices to account for WGS 84 and GDA94 differences. (Mohammed goes to the mountain).

Australia chose option 1, to gazette a new official national datum, GDA2020, which will be coincident with WGS 84 (G1762) and ITRF2014 on 2020-01-01.

Figure: Misalignment of WGS 84 Web Mecator web-maps
when sourced from GDA94 and GDA2020

The following datum transformations have been defined:

1. GDA94 - GDA2020 (EPSG::8048) accounts for Australia’s drift of ~ 1.8 metres.
2. GDA94 - WGS 84 (EPSG::1150), a null transformation which assumes GDA94 is the same as WGS 84.
3. GDA2020 - WGS 84 (EPSG::8450) a null transformation which assumes GDA2020 is the same as WGS 84.

Problem 5.1: Systematic misaligned maps

Combining WGS 84 map layers, sourced from GDA94 and GDA2020 via null transformations, results in the systematic misaligned by ~ 1.8 metres. (Previously these layers would have been aligned because they would have both been transformed from GDA94.)

Problem 5.2: Reverting back to GDA94 in web-mapping

To facilitate backward compatibility with existing WGS 84 based web-mapping services, Australia plan to continue aligning new WGS 84 based web-maps with the old GDA94 instead of with the new GDA2020 (as originally planned).

Problem 5.3: Equating a static and dynamic datum

Defining a static and dynamic datum as equivalent will inevitably lead to inaccuracies which will increase over time, and will need to be addressed in future.

Problem 5.4 Moving datums is costly

Moving to a new national/regional datum to improve accuracy requires significant effort, and in Australia’s case, we discovered that the widespread adoption of web-mapping introduced extra hurdles.

Notably, new coordinate operations, which support moving coordinates between epochs, can be implemented to address the same accuracy business drivers.

These options should be considered by those considering future updates to national/regional datums.

Problem 5.5 GDA2020 - WGS 84 transformation not implemented:

Some software providers (such as ArcGIS server), which is used by many Australian mapping agencies, have not implemented the GDA2020 - WGS 84 (EPSG::8450) transformation, restricting these agencies from supporting it. (Considering our web-mapping misalignment problem, this is probably a good thing.)

6. Reference frames and projections

Problem 6.1: WGS 84 being used instead of modern ITRF

The WGS 84 Reference Frame used in web mapping is not as accurate as the International Terrestrial Reference Frame (ITRF). Compared to WGS 84, ITRF is:

• The baseline reference frame used in geodesy
• The reference frame being adopted by national datum modernisation initiatives
• Addresses time
• Has more accurate transformations into it.

Future global datum usage should consider moving from WGS 84 to ITRF.

Note 6.2: Spherical instead of Ellipsoidal Datum:

The WGS 84 Web Mercator projection used in web-mapping is based on a sphere, not an ellipsoid. This results in coordinates being distorted by up to tens of kilometers. (Distortions increase the closer to the poles). Further, the projection is non-conformal, which means the amount of distortion in the x axis differs to the y axis. Consequently derived distances, areas, bearings and so on will also be distorted.

If the world moves from WGS 84 to another CRS for web-mapping, along with the global retiling of maps entailed, this would be an opportunity to reconsider moving to an ellipsoidal based projection.

7 Constructs to describe accuracy

• Transforming coordinates between datums has associated inaccuracies, which differs depending on the transformation path taken, and the transformation path typically is not predetermined.
• In contrast, converting coordinates from one reference system to another, (e.g. from latitude /longitude to easting /northing) is a mathematical procedure which does not introduce inaccuracy.
• The lineage of how a dataset’s coordinates are derived is typically not recorded with the dataset.
• By definition, transformations between datums are assigned an accuracy, but accuracy isn’t assigned to a datum per se.
• Dataset formats and web service standards typically don’t record or publish accuracy statements.

Problem 7.1 Difficult to determine dataset accuracy

As standard spatial data formats and web service interfaces don’t define attributes for accuracy, it is difficult for users to determine the accuracy of a datasets. Equally, it is difficult to publish a dataset’s accuracy.

Problem 7.2 Inaccuracy introduced during round-trip transformations

There are multiple transformation paths between datums. For instance, transforming between a GDA94 source and GDA2020 target datum can apply:

• EPSG::8048, Helmert 7 Parameter transformation (1cm accuracy)
• EPSG::8446, NTv2 Conformal transformation (5cm accuracy)
• EPSG::8447, NTv2 Conformal and Distortion transformation (5cm accuracy)

Further, transforms can be applied directly (eg: GDA94 -> GDA2020) or via a hub transformation (eg: GDA94 -> WGS 84 hub -> GDA2020). Direct or hub potentially provides different results with different accuracies.

Because transformation lineage is typically not recorded with a dataset, transforming back to a source datum can introduce inaccuracy due to a different reverse transformation being applied.

Note: For proj based implementations: the proj6 library will use the most accurate transformation, if available on the system, otherwise it will fallback to less accurate transformation available, and provide hints on where to download more accurate transformations.

Problem 7.3 Describing accuracy

We don’t have the means to convey to a data administrator that a datum includes inaccurate transformations into it (and hence should be avoided for accurate use cases) - E.g. for the WGS 84 datum ensemble.

Problem 7.4 Lack of awareness of accuracy

There appears to be a general lack of discussion and awareness about these accuracy challenges.

8. Communicating map concepts

Discussion about mapping is hindered by:

• A complicated technical problem,
• Being applied to solve multiple use cases, with differing requirements and a variety of implemented solutions.
• Multiple user groups understanding different facets of the problem, often using differing terminology, and talking past each other.
• Historical implementations sometimes clouding people’s understanding.
• Message crafting typically being dominated by highly technical people with minimal access to technical writers, professional educators and communicators.

Problem 8.1 Confusing terminology

Understanding the intricacies of mapping is complicated by terminology which is often misused, overloaded or misunderstood.

Problem 8.2 Confused messaging

Based on questions and misunderstandings seen on email lists, there appears to be a lack of clear messaging and education around projections, datums, coordinate reference systems, transformations, EPSG codes and related concepts. Messaging would benefit from input by communication and education expertise, tailored for key user groups.

Problem 8.3 Abstract out complexity

We should simplify developer and user experiences by abstracting complicated concepts behind simple implementations.

Web-mapping terminology (simplified)

Understanding the intricacies of mapping is complicated by terminology which is often misused, overloaded or misunderstood.

Hopefully, these simplified explanations should help. For official definitions, refer to the OGC Abstract Specification Topic 2: Referencing by coordinates.

• Datum (more recently called a Reference Frame): A reference frame is a mathematical model of the earth against which features can be represented as coordinates.
• Within a static datum, a feature’s coordinates are locked in time, or locked to its tectonic plate, so the feature’s coordinates remain the same over time. For example, GDA2020 is a static datum for Australia.
• Within a dynamic datum, coordinates are fixed to the earth as a whole. As the earth’s tectonic plates move (by a few centimeters a year), a feature’s coordinates also move. Examples include WGS 84 as currently used by Global Positioning Systems (GPS), and the current realization of the International Terrestrial Reference Frame (ITRF2014).
• Datum Ensemble: A group of related realizations of a Reference Frame (datum) used for lower accuracy applications where differences are insignificant. The WGS 84 datum (EPSG::6326) used in web-mapping is a datum ensemble.
• Coordinate System: A set of mathematical rules which specify how coordinates are assigned to points.
• Coordinate Reference System (CRS): A coordinate system that is related to the real world by a datum. A CRS is typically referenced in Web Service calls by an EPSG code. EPSG::4326 is the defacto standard CRS used in web-mapping and refers to the WGS 84 datum ensemble (EPSG::6326).
• Epoch: A point in time, as applied to time dependent datums, expressed in decimal years. For example, 2017-03-25 is epoch 2017.23.

Multiple definitions of epoch are commonly used to describe different aspects of a coordinate and a CRS, which can lead to confusion:

1. Observation epoch: The date at which the observation was made.
2. Coordinate epoch: The date at which the coordinates are expressed.
3. Realization epoch: Part of the Indicates in general the date at which a datum was defined.
4. Reference epoch: The epoch at which time-dependent coordinate transformation parameters are defined, along with their rates of change. (The EPSG database currently calls this “Realization epoch”.)

• Map projection: Coordinate conversion from the earth’s ellipsoidal coordinate system to a flat plane. WGS 84 Web Mercator (EPSG:3857) is the defacto standard projection used in web-mapping. (Its official name is “Pseudo Mercator”). WGS 84 Web Mercator is projected from the WG84 datum ensemble (EPSG::6326).
• EPSG Codes: Online database that contains definitions of numerous datums and map projections, along with formulas to translate between them. Each is uniquely identified via an EPSG code.
• Coordinate operations:
• Coordinate conversion: Changes coordinates from one coordinate reference system to another coordinate reference system based on the same datum (reference frame).
• Coordinate transformation: Changes coordinates from one coordinate reference system to another coordinate reference system which is based on another datum (reference frame).
• Point Motion Operation: Changes coordinates within one coordinate reference system to account for the motion of the point within the CRS over a period of time.