Point cloud processing in QGIS and 3D data enhancements

• Read original article here

Point cloud processing in QGIS and 3D data enhancements
Crowdfunding Status: Ongoing
Pledge now
Summary
In the wake of successful crowdfunding campaigns in 2020 and 2021 , we are pleased to announce the launch of a new campaign to further improve QGIS capabilities in managing and analyzing point clouds.
The objectives of this campaign are to:
Add point cloud data management, transformation and analysis algorithms. The tools will be based on PDAL. For example, with the introduction of these tools, users will be able to use the QGIS Processing toolbox to create or merge point cloud tiles, create raster DEMs from ground points and other algorithms. We will also ensure that large (terabyte size) datasets can be handled well for both display and analysis.
Improvements of the Profile tool: the profile tool was introduced in QGIS 3.26 and proved to be a powerful way of interacting with the elevation data. With the new enhancements, you will be able to embed elevation profiles within your print layouts, export the profile to other formats (e.g. DXF or CSV) and much more.
3D map improvements: your data come to life when opened in 3D scenes. There will be more enhancements for handling large scenes, better navigation and measurement.
If we exceed the funding for the above requirements, we will spend the extra funds to introduce new terrain shading tools, improve the performance of profile rendering and add more data-defined control over 3D objects.
 
An example of raster and point cloud rendering in QGIS 3D map. Data: Defra and OS.
 
Example of classified point cloud data in QGIS. Data: OpenTopography.
 
The total cost of development will be 75000 EUR. If you want to make a contribution, please fill the pledge form . For more information about contribution and how it works, see below.
We are teaming up with North Road , Hobu for implementing these new features.
Timeline
The campaign to pledge funds is set to finish on October 24, 2022. Work will commence when the target for our crowdfunding campaign has been reached. We aim to deliver the promised features over QGIS 3.30 and 3.32 (due to be released in February and June 2023 respectively).
About us
This effort has been organised and led by Lutra Consulting in collaboration with North Road and Hobu . These are three highly respected, proven companies committed to open source GIS, all of them with a history of running successful campaigns to enhance Open Source GIS software.
     
     
Together, we’ve put forward a team of outstanding quality and complementary skills:
Lutra Consulting (based in the UK) has been contributing to QGIS for more than ten years and are leading development of the 3D map view within QGIS
North Road (based in Australia) are one of the leading forces in QGIS development, adding great amount of new features with a lot of focus on beautiful cartographical and analytical capabilities
Hobu (based in the US) has been extremely active in the fundamental FOSS4G libraries including GDAL, PROJ and others. More importantly, Hobu has been leading the development of PDAL library for nearly 10 years.
Crowdfunding
By supporting this project, you will help to bring working with elevation / 3D data in QGIS to the next level, with extra focus on enhancing functionality related to point clouds.
This project cannot go ahead without your support! We use crowdfunding to raise funds for projects like this one which benefit a wide community of users. The project will only proceed if sufficient funding can be raised before the project crowdfunding deadline. Please see our main crowdfunding page for more information about our crowdfunding initiative and how it works.
Pledging funds is safe and easy. Simply fill in the pledge form to state how much you want to pledge - pledges only become binding if the funding target is reached before the deadline.
If you are based in the US or Australia and have difficulty pledging funds to companies outside your country, please contact Hobu or North Road to facilitate your contribution.
Detailed proposal and deliverables
Point cloud processing in QGIS
Until now, our efforts have focused only on visualization of point clouds in QGIS. Tasks related to analysis of point cloud data have been left to third party tools. While some of these tools also include integration to QGIS, they are typically either limited in functionality or proprietary products with paid licenses.
As the display of point cloud data in QGIS has been maturing, we would love to focus on point cloud analysis as well! Fortunately we do not need to look far: QGIS already uses the PDAL library for handling point cloud data. PDAL is a state of the art open source library for manipulation of point clouds. Currently, QGIS uses only a small fraction of what the PDAL library actually offers. It was always our plan to introduce more of its functionality for direct use in QGIS.
PDAL allows users to build complex data processing workflows by connecting pieces of its functionality together. As of now, there are more than 70 filters available for manipulating point cloud data, together with dozens of drivers for read/write support of various formats.
While PDAL is a great engine for processing point clouds, the project is geared towards advanced users that understand various peculiarities of point cloud formats and are able to connect relatively low-level pieces of functionality to pipelines using JSON files.
We would like to make processing of point clouds accessible to a wide range of QGIS users, and so we will develop a set of behind-the-scenes command line tools that would wrap PDAL functionality in a more user friendly way and with sensible default options. Each tool will handle a particular task that the user needs to do (clip data, convert to raster DEM etc). While these command line tools could be used independently from QGIS, our priority is making them super-easy to access directly through QGIS!
PDAL also intentionally does not deal with parallelism and leaves any parallelization of the data processing to clients of the library. This is a fair thing to do, however for many users it is hard to deal with parallel runs of pipelines. We will make the best effort to make the tools do the processing in parallel (using all CPUs on a single machine), so that users do not need to worry and still use all the power of their machines.
Parallel processing of tiled point cloud data to a raster digital elevation model.
In QGIS we will introduce a new provider for the Processing toolbox that will make the algorithms accessible in a familiar user interface (e.g. GDAL). By implementing the point cloud manipulation tools in the Processing framework we’ll be providing users with a common interface for accessing these tools, and also allowing their use in advanced operations such as batch processing or when creating graphical models.
Two groups of algorithms will be provided for use in QGIS Processing. The first group focuses on basic manipulation of point cloud data:
Info: find out more details about a point cloud dataset (create a report in HTML).
Boundary and density: get the actual shape of the area covered by point cloud and provide statistics of density of data.
Merge: merge multiple point cloud files into a single file.
Tile: turn the input point cloud into multiple tiles.
Reproject: apply a horizontal CRS transform to a point cloud.
Source point cloud data (top-left), point cloud density (top-right), extracted boundary polygon (bottom-left), exported raster DEM (bottom-right).
The second group of tools will create derived data from a point cloud:
Export to raster using interpolation: for each raster cell it will use points from its neighborhood to determine the value.
Export to raster using triangulation: the whole dataset first gets triangulated, raster cells get values interpolated from triangles.
Export to vector layer: individual points will be saved as XYZ point geometries.
Mockup of the Processing toolbox integration with point cloud algorithms.
After the initial work is done and the underlying infrastructure is in place, adding additional point cloud algorithms to the toolbox will be easy! (Assuming the PDAL functionality already exists). We expect that the initial set of tools will quickly be expanded to include tools like classification, noise filtering, height above ground calculation, clustering, surface reconstruction and various others - all of these are already available in PDAL, they just won’t be available in QGIS from the very beginning.
A sample QGIS processing model to create a raster DEM and contours from a point cloud dataset when point cloud processing algorithms will be available in QGIS.
Handling large datasets
Point cloud data are typically distributed as a collection of multiple LAS/LAZ files. Loading larger amounts of point cloud files in QGIS is currently not very practical - there are many individual map layers to handle, combined with possible performance issues and/or issues styling each layer separately.
The idea is to make it possible to load big datasets consisting of many LAS/LAZ files as a
single point cloud layer in QGIS, avoiding the issues described above. When dealing with big datasets, QGIS will initially only show the extents of individual files, and only once zoomed in further would the actual data be loaded (this time needing to read only one or few files).
Point cloud tiles, when zoomed out showing tile names and displaying the tiles when zoomed closer.
Alternatively, if a thinned "overview" point cloud would be associated with the dataset, QGIS will be able to show it for better context instead of just showing extents of tiles.
A large point cloud dataset visualized as a set of tiles (1km2 each) compared to the rendering of the actual point cloud data (using a sub-sampled point cloud).
We will introduce the concept of "virtual point clouds", similar to virtual rasters (vrt files) you may know from GDAL. There will be a single file that would reference other point cloud files which should be treated as a single dataset.
A virtual point cloud file that references other files (e.g. LAS/LAZ or COPC).
The specification of the format of the file is still being discussed, but it will be designed to be easy to use so that it can be adopted also by other software projects.
The concept of virtual point clouds will be also used by the Processing toolbox: a virtual point cloud may be used as an input dataset, and processing algorithms may return virtual point clouds as well.
Elevation profile tool improvements
Since the introduction of the elevation profile tool in QGIS 3.26, we have received a lot of great feedback and we decided to further improve its capabilities.
One of the most requested features for the new profile tool was the ability to embed elevation profiles within print layouts. We’ll be adding this functionality, making it possible to add beautifully styled profiles in your print outputs. You’ll get full control over the appearance of your elevation charts, right down to customisation of the fonts, colors and line styles used for the chart axis and other elements themselves!
Of course, we’ll also make it possible for elevation profiles to be dynamically generated during atlas exports. For instance, you’ll be able to include a different, automatically generated elevation profile for each individual page of your atlas export, with no code required!
Another much requested feature has been the ability to export elevation profiles for use in other applications. We’ll make it possible to export your profiles to CSV and to DXF, so that you can use source data of profiles in 3rd party software.
Currently, the scale of horizontal and vertical axes in elevation profiles are always independent. While this is useful in some cases, in other cases the true (undistorted) shapes of objects are needed and it can get a bit difficult to set these scales in a way they perfectly match. To handle this, a new option will be added to lock the horizontal and vertical axes to have the same scale.
But that’s not all! We’re also planning some additional improvements to make the tool easier to use:
Improved layer selection for creation of layer profiles, and more intelligent default selection of layers to include in new plots
Remembering any layers you’ve checked in the elevation profile, so that these are automatically restored whenever a new profile is opened
Add a choice of units for distance axis (eg meters, kilometers, feet, etc)
Handle polylines that share a segment with the captured curve
Support interpolation of raster data to avoid blocky-looking results
3D map views
Projects that contain background map layers spanning the whole world often make the 3D view behave poorly. While fixing all the underlying issues is a difficult task that is beyond the scope of this project, most of the time the actual area of interest of a project is much smaller. We will introduce an option to limit the spatial extent of the 3D map view, which will fix the poor 3D view behavior.
Measuring distances in 3D map views is a desired functionality, and it has been available for a couple of QGIS releases. However the measure tool in 3D is far from ideal - measurement line is not following the mouse cursor (as is the case in 2D map views), measurement lines can easily get occluded by other 3D objects, and picked points may be hard to spot (they are not highlighted). We would like to fix these issues and this will get us closer also to future use cases such as support for editing in 3D.
We are also going to further improve camera controls to make camera movement even more intuitive. Finally, we will have a look at the growing list of options in the configuration dialog of 3D map views and improve its usability in various ways.
Stretch goals
There are various features that we would like to add, but they did not fit into the scope and budget. However, if more funds will be pledged than the planned budget, we will add those features as well. Here are the next targets:
79000 EUR - Terrain shading in 2D. Currently, we have hillshading support for raster layers and eye-dome shading for point cloud layers in 2D map canvas. This item will make three improvements:
Extend the shading to all map layers supporting elevation data - QGIS will also generate elevation maps (merged from all participating layers)
Provide terrain shading to any map layer
Add ambient occlusion effects for 2D maps.
81000 EUR - Incremental updates of elevation profiles while data are being fetched. As of now, the elevation profile window is empty while data is being fetched. This improvement will provide updates to the elevation profile as soon as some data is ready - similar to how 2D map canvas does incremental rendering of map layers.
83000 EUR - More data-defined controls for 3D points. We will add the possibility to have data-defined rotation and size of point features in 3D, which will allow better variability of features - especially useful when displaying trees for example.
An example of raster in 2D map canvas with ambient occlusion and shadow depth.