Tuesday 21 April 2015

Esri's claim at being good "Standards" citizens is questionable

I'm calling Esri out on their claim to be good "Open Standards" citizens. Esri is again abusing their market position to compromise established Open Spatial Standards, as described in an Open Letter from the OSGeo community. It starts:
We, the undersigned, are concerned that the current interoperability between LiDAR applications, through use of the open "LAS" format, is being threatened by Esri's introduction and promotion of an alternative "Optimized LAS" proprietary format. This is of grave concern given that fragmentation of the LAS format will reduce interoperability between applications and organisations, and introduce vendor lock-in. …
To be clear, Esri has extended LAS to create "Optimized LAS" which provides near identical features and performance to the existing and open LASzip format, both of which provide faster access and smaller file sizes to the LAS format. However, rather than collaborate with the open community, as has been repeatedly offered, "Optimiszed LAS" has been developed internally to Esri. It is neither published, nor open, which provides both technical as well as legal barriers for other applications reading and/or writing to this proprietary format. This creates a vendor lock-in scenario which is contrary to the principles of the Open Geospatial Consortium, the OSGeo Foundation, and many government IT procurement policies.

Esri responded to the open request to avoid fragmenting LiDAR standards with the following motherhood statement, which doesn't actually answer the key questions:
Regarding Dr. Anand’s concerns and the referenced letter below:
Esri has long understood the importance of interoperability between systems and users of geographic information and services. Esri has participated in the development of national, information community, OGC, and ISO TC 211 standards from the development of the US Spatial Data Transfer Standard in the 1980s through the development of OGC Geopackage today. As a sustaining member of ASPRS and a Principle member of OGC, Esri would gladly participate in efforts to further the development of open LIDAR and point cloud standards. Keep in mind that ASPRS owns and maintains LAS, along with other spatial information standards, and would have the lead in moving it into  OGC or ISO TC211 for further work if they so desired. Esri will continue to support and use the ASPRS LAS standard; the Optimized LAS (see FAQ at https://github.com/Esri/esri-zlas-io-library) is not intended to replace LAS but to enhance access to remotely stored LIDAR information for our users.
Lets refute Esri's statement line by line:

Esri has long understood the importance of interoperability between systems and users of geographic information and services.

  • Nice motherhood statement. Notice that Esri carefully selects the words "understood the importance" rather than "we commit to implementing".

Esri has participated in the development of national, information community, OGC, and ISO TC 211 standards from the development of the US Spatial Data Transfer Standard in the 1980s through the development of OGC Geopackage today.


As a sustaining member of ASPRS and a Principle member of OGC, Esri would gladly participate in efforts to further the development of open LIDAR and point cloud standards.

  • Nice statement, without any quantifiable commitment. Will Esri put it into practice? Track record suggests otherwise. As explained by Marin Isenburg, Esri has talked a lot about collaboration and being open, while in parallel creating a competing proprietary format. If Esri were seriously committed to open LiDAR standards, Esri would publish "Optimized LAS" under an Open License, and/or take "Optimized LAS" through a standards development process such as provided by the OGC. Esri would have also build upon the prior LASzip format rather than redeveloping equivalent functionality.

Keep in mind that ASPRS owns and maintains LAS, along with other spatial information standards, and would have the lead in moving it into  OGC or ISO TC211 for further work if they so desired. 
  • Again, if Esri had the best interests of ASPRS and Open Standards in mind (as you would expect from a sustaining member), then we'd expect Esri to donate their LAS improvements back to the ASPRS for safe keeping. Why is Esri keeping such improvements in an Esri proprietary format instead?
  • Esri would be also lobbying ASPRS to accept improvements to the LAS format. Has this happened? Lack of public discussion on this topic suggests otherwise.

Esri will continue to support and use the ASPRS LAS standard; the Optimized LAS (see FAQ at https://github.com/Esri/esri-zlas-io-library) is not intended to replace LAS but to enhance access to remotely stored LIDAR information for our users.

  • Esri is sidestepping the issue. The LAS standard needs improvements. These improvements have been implemented by the open LASzip format and also by Esri's proprietary Optimized LAS. One should be incorporated into a future LAS standard.
  • The question Esri fails to answer is why does Esri refuse to work in collaboration with the open community? Why has Esri developed their own Optimized LAS format instead of improving an existing standard format?
  • Esri's FAQ, explains that esri-zlas-io-library is stored on github under the Apache license, which would make you think the code is Open Source and hence the Optimized LAS format could be reverse engineered. This is not the case. Esri has only licensed the binaries under the Apache license such that it can't be reverse engineered or improved by the community. By the OSI definition, this is not Open Source Software
So I'm calling Esri out on their claim to being supporters of Open Standards. Please Esri, either clean up the way you behave, or come clean and admit that Esri abuses its market position to undermine Open Standards.


6 comments:

GO BANANA said...

Years ago, I was in a meeting with a bunch of ESRI VPs and such. One of them indicated that Jack once said "Own the market, own the standards."

Unknown said...

Strong medicine Cameron. Thanks for bringing this to light so succinctly and not mincing words. Monopolies went the way of the DoDo bird in the 19th century with US Steel and the railroads. The only monopoly we should be playing in the 21st c is the board game.

Anonymous said...

That is a shame....but really not surprising from ESRI! But they're a business and they're here to make money. Can't blame them for that.

I haven't heard ESRI claiming that they are a "good open source citizen", but if they have been then they need to reassess their message. IF, it is true that the open source community approached ESRI repeatedly to collaborate on improving LASzip, then a "good open source citizen" would have worked with the community to improve the standard and then adapted their product to support that standard. Not split off a very similar proprietary alternative that only they can use.

ESRI has forged the way for the GIS and spatial industry and have done plenty of good and still do. However they have the influence and power to advocate for simplifying, improving and developing open standards and with that the potential to be a seriously "good open source citizen".

But to be honest I don't think that ESRI will ever change their ways.

Good job keeping your finger on the pulse Cameron and trying to keep organisations accountable. If they are talking the talk then they need to walk the walk.

Anonymous said...

Cameron, I wholeheartedly agree with everything you wrote in this blog post.

Thanks for the clear words.

Unknown said...

Do you realise that ESRI charges extra to be able to use OGC standards? I noticed this a few years back when using Arc GIS Server Desktop.

We could view WMS basemaps, but couldn't view WFS maps. In order to get access to WFS maps, we would have needed to purchase the ArcGIS Data Interoperability extension.

Seems this is still the case according to ArcGIS 10.1 help: http://resources.arcgis.com/en/help/main/10.1/index.html#//00370000000p000000 . Unfortunately, I couldn't convince business to buy me the extra functionality - which in turn meant I didn't get to use WFS.

Not the sort of business model you'd expect from a company claiming to be champions of OGC Standards.

Cameron Shorter said...

Jack Dangermond, founder and president of Esri, has responded to our Open Letter positively. I look forward to seeing Esri follow through on this comment.

Jack: Thank you very much for your letter. I appreciate your comments and I will use them to improve our contributions to the GIS community, our products, and our processes. While my colleague Keith Ryden has already responded to your letter sent to the OGC community earlier I want to assure Esri continues to remain committed towards interoperability in all the product technology.

As for the LAS issue, we have had extensive discussions with our standards team and feel the community should use OGC to organize an open process leading to an open standard. OGC is looking to initiate a group to study the formalization of an open standard for storing Lidar Data. We have expressed our interest in the formation of such a group and would be willing to bring our work for discussion as part of this initiative. I believe that this would be the right avenue for focusing on the technical merits of different approaches. As long-term supporters of the OGC mission, we would hope that this group at OGC can work towards a solution that meets the requirements of the community at large.

Esri continues to have a strong relationship with ASPRS and it would be unthinkable for us to pursue something that would be contrary to ASPRS's commitments in the LIDAR space.

I have instructed my team to make sure that we all work together to improve any technical shortcomings in the current specification.

Thank you again for bringing some of the nuances of this discussion to my notice.

Warm regards,
Jack