Package

root package

Permalink

package root

Visibility
  1. Public
  2. All

Value Members

  1. package geotrellis

    Permalink

  2. package io

    Permalink

  3. package vectorpipe

    Permalink

    VectorPipe is a library for mass conversion of Vector data into Mapbox VectorTiles.

    VectorPipe is a library for mass conversion of Vector data into Mapbox VectorTiles. It is powered by GeoTrellis and Apache Spark.

    Outline

    GeoTrellis and Spark do most of our work for us. Writing a main function that uses VectorPipe need not contain much more than:

    import geotrellis.proj4.WebMercator
import geotrellis.spark.tiling.{LayoutDefinition, ZoomedLayoutScheme}
import geotrellis.vectortile.VectorTile
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession
import vectorpipe._

implicit val ss: SparkSession = ...

val layout: LayoutDefinition =
  ZoomedLayoutScheme.layoutForZoom(15, WebMercator.worldExtent, 512)

/* An ORC file containing OSM data. */
val path: String = "s3://path/to/data.orc"

osm.fromORC(path) match {
  case Failure(_) => { /* Handle the error. Was your path correct? */ }
  case Success((nodes, ways, relations)) => {

    val features: RDD[OSMFeature] =
      osm.features(nodes, ways, relations).geometries

    val featGrid: RDD[(SpatialKey, Iterable[OSMFeature])] =
      grid(Clip.byHybrid, layout, features)

    val tiles: RDD[(SpatialKey, VectorTile)] =
      vectortiles(Collate.byAnalytics, layout, featGrid)

    // further processing / output
}

/* Nicely stop Spark */
ss.stop()

    Writing Portable Tiles

    This method outputs VectorTiles to a directory structure appropriate for serving by a Tile Map Server. The VTs themselves are saved in the usual .mvt format, and so can be read by any other tool. The example that follows writes tiles from above to an S3 bucket:

    import geotrellis.spark.io.s3._  // requires the `geotrellis-s3` library

/* How should a `SpatialKey` map to a filepath on S3? */
val s3PathFromKey: SpatialKey => String = SaveToS3.spatialKeyToPath(
  LayerId("sample", 1),  // Whatever zoom level it is
  "s3://some-bucket/catalog/{name}/{z}/{x}/{y}.mvt"
)

tiles.saveToS3(s3PathFromKey)

    Writing a GeoTrellis Layer of VectorTiles

    The disadvantage of the "Portable Tiles" approach is that there is no way to read the tiles back into a RDD[(SpatialKey, VectorTile)] and do Spark-based manipulation operations. To do that, the tiles have to be written as a "GeoTrellis Layer" from the get-go. The output of such a write are split and compressed files that aren't readable by other tools. This method compresses VectorTiles to about half the size of a normal .mvt.

    import geotrellis.spark._
import geotrellis.spark.io._
import geotrellis.spark.io.file._    /* When writing to your local computer */
import org.apache.spark.storage.StorageLevel

/* IO classes */
val catalog: String = "/home/you/tiles/"  /* This must exist ahead of time! */
val store = FileAttributeStore(catalog)
val writer = FileLayerWriter(store)

/* Almost certainly necessary, to save Spark from repeating effort */
val persisted = tiles.persist(StorageLevel.MEMORY_AND_DISK_SER)

/* Dynamically determine the KeyBounds */
val bounds: KeyBounds[SpatialKey] =
  persisted.map({ case (key, _) => KeyBounds(key, key) }).reduce(_ combine _)

/* Construct metadata for the Layer */
val meta = LayerMetadata(layout, bounds)

/* Write the Tile Layer */
writer.write(LayerId("north-van", 15), ContextRDD(persisted, meta), ZCurveKeyIndexMethod)

Ungrouped