Writing a reader

Authors:Bradley Chambers, Scott Lewis
Contact:brad.chambers@gmail.com
Date:11/02/2017

PDAL’s command-line application can be extended through the development of reader functions. In this tutorial, we will give a brief example.

The header

First, we provide a full listing of the reader header.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// MyReader.hpp

#pragma once

#include <pdal/PointView.hpp>
#include <pdal/Reader.hpp>
#include <pdal/util/IStream.hpp>

namespace pdal
{
  class MyReader : public Reader
  {
  public:
    MyReader() : Reader() {};

    static void * create();
    static int32_t destroy(void *);
    std::string getName() const;

  private:
    std::unique_ptr<ILeStream> m_stream;
    point_count_t m_index;
    double m_scale_z;

    virtual void addDimensions(PointLayoutPtr layout);
    virtual void addArgs(ProgramArgs& args);
    virtual void ready(PointTableRef table);
    virtual point_count_t read(PointViewPtr view, point_count_t count);
    virtual void done(PointTableRef table);
  };
}

In your MyReader class, you will declare the necessary methods and variables needed to make the reader work and meet the plugin specifications.

1
2
3
    static void * create();
    static int32_t destroy(void *);
    std::string getName() const;

These methods are required to fulfill the specs for defining a new plugin.

1
2
3
    std::unique_ptr<ILeStream> m_stream;
    point_count_t m_index;
    double m_scale_z;

m_stream is used to process the input, while m_index is used to track the index of the records. m_scale_z is specific to MyReader, and will be described later.

1
2
3
4
5
    virtual void ready(PointTableRef table);
    virtual point_count_t read(PointViewPtr view, point_count_t count);
    virtual void done(PointTableRef table);
  };
}

Various other override methods for the stage. There are a few others that could be overridden, which will not be discussed in this tutorial.

Note

See ./include/pdal/Reader.hpp of the source tree for more methods that a reader can override or implement.

The source

Again, we start with a full listing of the reader source.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
// MyReader.cpp

#include "MyReader.hpp"
#include <pdal/pdal_macros.hpp>
#include <pdal/util/ProgramArgs.hpp>

namespace pdal
{
  static PluginInfo const s_info = PluginInfo(
    "readers.myreader",
    "My Awesome Reader",
    "http://link/to/documentation" );

  CREATE_SHARED_PLUGIN(1, 0, MyReader, Reader, s_info)

  std::string MyReader::getName() const { return s_info.name; }

  void MyReader::addArgs(ProgramArgs& args)
  {
    args.add("z_scale", "Z Scaling", m_scale_z, 1.0);
  }

  void MyReader::addDimensions(PointLayoutPtr layout)
  {
    layout->registerDim(Dimension::Id::X);
    layout->registerDim(Dimension::Id::Y);
    layout->registerDim(Dimension::Id::Z);
    layout->registerOrAssignDim("MyData", Dimension::Type::Unsigned64);
  }

  void MyReader::ready(PointTableRef)
  {
    SpatialReference ref("EPSG:4385");
    setSpatialReference(ref);
  }

  template <typename T>
  T convert(const StringList& s, const std::string& name, size_t fieldno)
  {
      T output;
      bool bConverted = Utils::fromString(s[fieldno], output);
      if (!bConverted)
      {
          std::stringstream oss;
          oss << "Unable to convert " << name << ", " << s[fieldno] << ", to double";
          throw pdal_error(oss.str());
      }

      return output;
  }


  point_count_t MyReader::read(PointViewPtr view, point_count_t count)
  {
    PointLayoutPtr layout = view->layout();
    PointId nextId = view->size();
    PointId idx = m_index;
    point_count_t numRead = 0;

    m_stream.reset(new ILeStream(m_filename));

    size_t HEADERSIZE(1);
    size_t skip_lines(std::max(HEADERSIZE, (size_t)m_index));
    size_t line_no(1);
    for (std::string line; std::getline(*m_stream->stream(), line); line_no++)
    {
      if (line_no <= skip_lines)
      {
        continue;
      }

      // MyReader format:  X::Y::Z::Data
      StringList s = Utils::split2(line, ':');

      unsigned long u64(0);
      if (s.size() != 4)
      {
        std::stringstream oss;
        oss << "Unable to split proper number of fields.  Expected 4, got "
            << s.size();
        throw pdal_error(oss.str());
      }

      std::string name("X");
      view->setField(Dimension::Id::X, nextId, convert<double>(s, name, 0));

      name = "Y";
      view->setField(Dimension::Id::Y, nextId, convert<double>(s, name, 1));

      name = "Z";
      double z = convert<double>(s, name, 2) * m_scale_z;
      view->setField(Dimension::Id::Z, nextId, z);

      name = "MyData";
      view->setField(layout->findProprietaryDim(name),
                     nextId,
                     convert<unsigned int>(s, name, 3));

      nextId++;
      if (m_cb)
        m_cb(*view, nextId);
    }
    m_index = nextId;
    numRead = nextId;

    return numRead;
  }

  void MyReader::done(PointTableRef)
  {
    m_stream.reset();
  }

} //namespace pdal

In your reader implementation, you will use a macro defined in pdal_macros.

1
2
3
4
  static PluginInfo const s_info = PluginInfo(
    "readers.myreader",
    "My Awesome Reader",
    "http://link/to/documentation" );

This macro registers the plugin with the PDAL code. In this case, we are declaring this as a SHARED plugin, meaning that it will be located external to the main PDAL installation. The macro is supplied with a version number (major and minor), the class of the plugin, the parent class (in this case, to identify it as a reader), and an object with information. This information includes the name of the plugin, a description, and a link to documentation.

Creating STATIC plugins requires a few more steps which will not be covered in this tutorial.

1
2
3
4
  void MyReader::addArgs(ProgramArgs& args)
  {
    args.add("z_scale", "Z Scaling", m_scale_z, 1.0);
  }

This method will process a options for the reader. In this example, we are setting the z_scale value to a default of 1.0, indicating that the Z values we read should remain as-is. (In our reader, this could be changed if, for example, the Z values in the file represented mm values, and we want to represent them as m in the storage model). addArgs will bind values given for the argument to the m_scale_z variable of the stage.

1
2
3
4
5
6
7
  void MyReader::addDimensions(PointLayoutPtr layout)
  {
    layout->registerDim(Dimension::Id::X);
    layout->registerDim(Dimension::Id::Y);
    layout->registerDim(Dimension::Id::Z);
    layout->registerOrAssignDim("MyData", Dimension::Type::Unsigned64);
  }

This method registers the various dimensions the reader will use. In our case, we are using the X, Y, and Z built-in dimensions, as well as a custom dimension MyData.

1
2
3
4
5
  void MyReader::ready(PointTableRef)
  {
    SpatialReference ref("EPSG:4385");
    setSpatialReference(ref);
  }

This method is called when the Reader is ready for use. It will only be called once, regardless of the number of PointViews that are to be processed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
  template <typename T>
  T convert(const StringList& s, const std::string& name, size_t fieldno)
  {
      T output;
      bool bConverted = Utils::fromString(s[fieldno], output);
      if (!bConverted)
      {
          std::stringstream oss;
          oss << "Unable to convert " << name << ", " << s[fieldno] << ", to double";
          throw pdal_error(oss.str());
      }

      return output;
  }

This is a helper function, which will convert a string value into the type specified when it’s called. In our example, it will be used to convert strings to doubles when reading from the input stream.

1
  point_count_t MyReader::read(PointViewPtr view, point_count_t count)

This method is the main processing method for the reader. It takes a pointer to a Point View which we will build as we read from the file. We initialize some variables as well, and then reset the input stream with the filename used for the reader. Note that in other readers, the contents of this method could be very different depending on the format of the file being read, but this should serve as a good start for how to build the PointView object.

1
2
3
    size_t HEADERSIZE(1);
    size_t skip_lines(std::max(HEADERSIZE, (size_t)m_index));
    size_t line_no(1);

In preparation for reading the file, we prepare to skip some header lines. In our case, the header is only a single line.

1
2
3
4
5
6
    for (std::string line; std::getline(*m_stream->stream(), line); line_no++)
    {
      if (line_no <= skip_lines)
      {
        continue;
      }

Here we begin our main loop. In our example file, the first line is a header, and each line thereafter is a single point. If the file had a different format the method of looping and reading would have to change as appropriate. We make sure we are skipping the header lines here before moving on.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
      StringList s = Utils::split2(line, ':');

      unsigned long u64(0);
      if (s.size() != 4)
      {
        std::stringstream oss;
        oss << "Unable to split proper number of fields.  Expected 4, got "
            << s.size();
        throw pdal_error(oss.str());
      }

Here we take the line we read in the for block header, split it, and make sure that we have the proper number of fields.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
      std::string name("X");
      view->setField(Dimension::Id::X, nextId, convert<double>(s, name, 0));

      name = "Y";
      view->setField(Dimension::Id::Y, nextId, convert<double>(s, name, 1));

      name = "Z";
      double z = convert<double>(s, name, 2) * m_scale_z;
      view->setField(Dimension::Id::Z, nextId, z);

      name = "MyData";
      view->setField(layout->findProprietaryDim(name),
                     nextId,
                     convert<unsigned int>(s, name, 3));

Here we take the values we read and put them into the PointView object. The X and Y fields are simply converted from the file and put into the respective fields. MyData is done likewise with the custom dimension we defined. The Z value is read, and multiplied by the scale_z option (defaulted to 1.0), before the converted value is put into the field.

When putting the value into the PointView object, we pass in the Dimension that we are assigning it to, the ID of the point (which is incremented in each iteration of the loop), and the dimension value.

1
2
3
      nextId++;
      if (m_cb)
        m_cb(*view, nextId);

Finally, we increment the nextId and make a call into the progress callback if we have one with our nextId. After the loop is done, we set the index and number read, and return that value as the number of points read. This could differ in cases where we read multiple streams, but that won’t be covered here.

When the read method is finished, the done method is called for any cleanup. In this case, we simply make sure the stream is reset.

Compiling and Usage

The MyReader.cpp code can be compiled. For this example, we’ll use cmake. Here is the CMakeLists.txt file we will use:

1
2
3
4
5
6
7
8
9
cmake_minimum_required(VERSION 2.8.12)
project(ReaderTutorial)

find_package(PDAL 1.6.0 REQUIRED CONFIG)

add_library(pdal_plugin_reader_myreader SHARED MyReader.cpp)
target_link_libraries(pdal_plugin_reader_myreader PRIVATE ${PDAL_LIBRARIES})
target_include_directories(pdal_plugin_reader_myreader PRIVATE
                            ${PDAL_INCLUDE_DIRS})

If this file is in the directory containing MyReader.hpp and MyReader.cpp, simply run cmake ., followed by make. This will generate a file called libpdal_plugin_reader_myreader.dylib.

Put this dylib file into the directory pointed to by PDAL_DRIVER_PATH, and then when you run pdal --drivers, you should see an entry for readers.myreader.

To test the reader, we will put it into a pipeline and output a text file.

Please download the pipeline-myreader.json and test-reader-input.txt files.

In the directory with those two files, run pdal pipeline pipeline-myreader.json. You should have an output file called output.txt, which will have the same data as in the input file, except in a CSV style format, and with the Z values scaled by .001.