Writing a writer

Authors:Bradley Chambers, Scott Lewis
Contact:brad.chambers@gmail.com
Date:10/26/2016

PDAL’s command-line application can be extended through the development of writer functions. In this tutorial, we will give a brief example.

The header

First, we provide a full listing of the writer header.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// MyWriter.hpp

#pragma once

#include <pdal/Writer.hpp>

#include <string>

namespace pdal{

  typedef std::shared_ptr<std::ostream> FileStreamPtr;

  class MyWriter : public Writer
  {
  public:
    MyWriter()
    {}

    static void  * create();
    static int32_t destroy(void *);
    std::string getName() const;

  private:
    virtual void addArgs(ProgramArgs& args);
    virtual void initialize();
    virtual void ready(PointTableRef table);
    virtual void write(const PointViewPtr view);
    virtual void done(PointTableRef table);

    std::string m_filename;
    std::string m_newline;
    std::string m_datafield;
    int m_precision;

    FileStreamPtr m_stream;
    Dimension::Id m_dataDim;
  };

} // namespace pdal

In your MyWriter class, you will declare the necessary methods and variables needed to make the writer work and meet the plugin specifications.

1
  typedef std::shared_ptr<std::ostream> FileStreamPtr;

FileStreamPtr is defined to make the declaration of the stream easier to manage later on.

    static void  * create();
    static int32_t destroy(void *);
    std::string getName() const;

These three methods are required to fulfill the specs for defining a new plugin.

    virtual void addArgs(ProgramArgs& args);
    virtual void initialize();
    virtual void ready(PointTableRef table);
    virtual void write(const PointViewPtr view);
    virtual void done(PointTableRef table);

These methods are used during various phases of the pipeline. There are also more methods, which will not be covered in this tutorial.

    std::string m_filename;
    std::string m_newline;
    std::string m_datafield;
    int m_precision;

    FileStreamPtr m_stream;
    Dimension::Id m_dataDim;

These are variables our Writer will use, such as the file to write to, the newline character to use, the name of the data field to use to write the MyData field, precision of the double outputs, the output stream, and the dimension that corresponds to the data field for easier lookup.

As mentioned, there cen be additional configurations done as needed.

The source

We will start with a full listing of the writer source.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
// MyWriter.cpp

#include "MyWriter.hpp"
#include <pdal/pdal_macros.hpp>
#include <pdal/util/FileUtils.hpp>
#include <pdal/util/ProgramArgs.hpp>

namespace pdal
{
  static PluginInfo const s_info = PluginInfo(
    "writers.mywriter",
    "My Awesome Writer",
    "http://path/to/documentation" );

  CREATE_SHARED_PLUGIN(1, 0, MyWriter, Writer, s_info);

  std::string MyWriter::getName() const { return s_info.name; }

  struct FileStreamDeleter
  {
    template <typename T>
    void operator()(T* ptr)
    {
      if (ptr)
      {
        ptr->flush();
        FileUtils::closeFile(ptr);
      }
    }
  };


  void MyWriter::addArgs(ProgramArgs& args)
  {
    // setPositional() Makes the argument required.
    args.add("filename", "Output filename", m_filename).setPositional();  
    args.add("newline", "Line terminator", m_newline, "\n");
    args.add("datafield", "Data field", m_datafield, "UserData");
    args.add("precision", "Precision", m_precision, 3);
  }

  void MyWriter::initialize()
  {
    m_stream = FileStreamPtr(FileUtils::createFile(m_filename, true),
      FileStreamDeleter());
    if (!m_stream)
    {
      std::stringstream out;
      out << "writers.mywriter couldn't open '" << m_filename <<
        "' for output.";
      throw pdal_error(out.str());
    }
  }

  void MyWriter::ready(PointTableRef table)
  {
    m_stream->precision(m_precision);
    *m_stream << std::fixed;

    Dimension::Id d = table.layout()->findDim(m_datafield);
    if (d == Dimension::Id::Unknown)
    {
      std::ostringstream oss;
      oss << "Dimension not found with name '" << m_datafield << "'";
      throw pdal_error(oss.str());
    }

    m_dataDim = d;

    *m_stream << "#X:Y:Z:MyData" << m_newline;
  }


  void MyWriter::write(PointViewPtr view)
  {
      for (PointId idx = 0; idx < view->size(); ++idx)
      {
        double x = view->getFieldAs<double>(Dimension::Id::X, idx);
        double y = view->getFieldAs<double>(Dimension::Id::Y, idx);
        double z = view->getFieldAs<double>(Dimension::Id::Z, idx);
        unsigned int myData = 0;

        if (!m_datafield.empty()) {
          myData = (int)(view->getFieldAs<double>(m_dataDim, idx) + 0.5);
        }

        *m_stream << x << ":" << y << ":" << z << ":"
          << myData << m_newline;
      }
  }


  void MyWriter::done(PointTableRef)
  {
    m_stream.reset();
  }

}

In the writer implementation, we will use a macro defined in pdal_macros, which is included in the include chain we are using.

  static PluginInfo const s_info = PluginInfo(
    "writers.mywriter",
    "My Awesome Writer",
    "http://path/to/documentation" );

  CREATE_SHARED_PLUGIN(1, 0, MyWriter, Writer, s_info);

Here we define a struct with information regarding the writer, such as the name, a description, and a path to documentation. We then use the macro to create a SHARED plugin, which means it will be external to the main PDAL installation. When using the macro, we specify the version (major and minor), the class of the plugin, the class of the parent (Writer, in this case), and the struct we defined earlier.

Creating STATIC plugins, which would be part of the main PDAL installation, is also possible, but requires some extra steps and will not be covered here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
  struct FileStreamDeleter
  {
    template <typename T>
    void operator()(T* ptr)
    {
      if (ptr)
      {
        ptr->flush();
        FileUtils::closeFile(ptr);
      }
    }
  };

This struct is used for helping with the FileStreamPtr for cleanup.

1
2
3
4
5
6
7
8
  void MyWriter::addArgs(ProgramArgs& args)
  {
    // setPositional() Makes the argument required.
    args.add("filename", "Output filename", m_filename).setPositional();  
    args.add("newline", "Line terminator", m_newline, "\n");
    args.add("datafield", "Data field", m_datafield, "UserData");
    args.add("precision", "Precision", m_precision, 3);
  }

This method defines the arguments the writer provides and binds them to private variables.

    {
      std::stringstream out;
      out << "writers.mywriter couldn't open '" << m_filename <<
        "' for output.";
      throw pdal_error(out.str());
    }
  }

  void MyWriter::ready(PointTableRef table)
  {
    m_stream->precision(m_precision);
    *m_stream << std::fixed;

    Dimension::Id d = table.layout()->findDim(m_datafield);
    if (d == Dimension::Id::Unknown)
    {
      std::ostringstream oss;

This method initializes our file stream in preparation for writing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
  void MyWriter::ready(PointTableRef table)
  {
    m_stream->precision(m_precision);
    *m_stream << std::fixed;

    Dimension::Id d = table.layout()->findDim(m_datafield);
    if (d == Dimension::Id::Unknown)
    {
      std::ostringstream oss;
      oss << "Dimension not found with name '" << m_datafield << "'";
      throw pdal_error(oss.str());
    }

    m_dataDim = d;

    *m_stream << "#X:Y:Z:MyData" << m_newline;

The ready method is used to prepare the writer for any number of PointViews that may be passed in. In this case, we are setting the precision for our double writes, looking up the dimension specified as the one to write into MyData, and writing the header of the output file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
  void MyWriter::write(PointViewPtr view)
  {
      for (PointId idx = 0; idx < view->size(); ++idx)
      {
        double x = view->getFieldAs<double>(Dimension::Id::X, idx);
        double y = view->getFieldAs<double>(Dimension::Id::Y, idx);
        double z = view->getFieldAs<double>(Dimension::Id::Z, idx);
        unsigned int myData = 0;

        if (!m_datafield.empty()) {
          myData = (int)(view->getFieldAs<double>(m_dataDim, idx) + 0.5);
        }

        *m_stream << x << ":" << y << ":" << z << ":"
          << myData << m_newline;
      }
  }

This method is the main method for writing. In our case, we are writing a very simple file, with data in the format of X:Y:Z:MyData. We loop through each index in the PointView, and for each one we take the X, Y, and Z values, as well as the value for the specified MyData dimension, and write this to the output file. In particular, note the reading of MyData; in our case, MyData is an integer, but the field we are reading might be a double. Converting from double to integer is done via truncation, not rounding, so by adding .5 before making the conversion will ensure rounding is done properly.

Note that in this case, the output format is pretty simple. For more complex outputs, you may need to generate helper methods (and possibly helper classes) to help generate the proper output. The key is reading in the appropriate values from the PointView, and then writing those in whatever necessary format to the output stream.

1
2
3
4
  void MyWriter::done(PointTableRef)
  {
    m_stream.reset();
  }

This method is called when the writing is done. In this case, it simply cleans up the output stream by resetting it.

Compiling and Usage

To compile this reader, we will use cmake. Here is the CMakeLists.txt file we will use for this process:

1
2
3
4
5
6
7
8
9
cmake_minimum_required(VERSION 2.8.12)
project(WriterTutorial)

find_package(PDAL 1.6.0 REQUIRED CONFIG)

add_library(pdal_plugin_writer_mywriter SHARED MyWriter.cpp)
target_link_libraries(pdal_plugin_writer_mywriter PRIVATE ${PDAL_LIBRARIES})
target_include_directories(pdal_plugin_writer_mywriter PRIVATE
    ${PDAL_INCLUDE_DIRS})

If this file is in the directory with the MyWriter.hpp and MyWriter.cpp files, simply run cmake . followed by make. This will generate a file called libpdal_plugin_writer_mywriter.dylib.

Put this dylib file into the directory pointed to by PDAL_DRIVER_PATH, and then when you run pdal --drivers, you will see an entry for writers.mywriter.

To test the writer, we will put it into a pipeline and read in a LAS file and covert it to our output format. For this example, use interesting.las, and run it through pipeline-mywriter.json.

If those files are in the same directory, you would just run the command pdal pipeline pipeline-mywriter.json, and it will generate an output file called output.txt, which will be in the proper format. From there, if you wanted, you could run that output file through the MyReader that was created in the previous tutorial, as well.