Packages

case class Options(tiffExtensions: Seq[String] = ..., crs: Option[CRS] = None, timeTag: String = GEOTIFF_TIME_TAG_DEFAULT, timeFormat: String = GEOTIFF_TIME_FORMAT_DEFAULT, maxTileSize: Option[Int] = Some(DefaultMaxTileSize), numPartitions: Option[Int] = None, partitionBytes: Option[Long] = Some(DefaultPartitionBytes), chunkSize: Option[Int] = None, delimiter: Option[String] = None, getClient: () ⇒ S3Client = S3ClientProducer.get) extends RasterReader.Options with Product with Serializable

This case class contains the various parameters one can set when reading RDDs from S3 using Spark.

TODO: Add persistLevel option

tiffExtensions

Read all file with an extension contained in the given list.

crs

Override CRS of the input files. If None, the reader will use the file's original CRS.

timeTag

Name of tiff tag containing the timestamp for the tile.

timeFormat

Pattern for java.time.format.DateTimeFormatter to parse timeTag.

maxTileSize

Maximum allowed size of each tiles in output RDD. May result in a one input GeoTiff being split amongst multiple records if it exceeds this size. If no maximum tile size is specific, then each file is broken into 256x256 tiles. If None, then the whole file will be read in. This option is incompatible with numPartitions and anything set to that parameter will be ignored.

numPartitions

How many partitions Spark should create when it repartitions the data.

partitionBytes

Desired partition size in bytes, at least one item per partition will be assigned. If no size is specified, then partitions 128 Mb in size will be created by default. This option is incompatible with the numPartitions option. If both are set and maxTileSize isn't, then partitionBytes will be ignored in favor of numPartitions. However, if maxTileSize is set, then partitionBytes will be retained. If None and maxTileSize is defined, then the default partitionBytes' value will still be used. If maxTileSize is also None, then partitionBytes will remain None as well.

chunkSize

How many bytes should be read in at a time when reading a file. If None, then 65536 byte chunks will be read in at a time.

delimiter

Delimiter to use for S3 objet listings. This provides a way to further define what files should be read. If None, then only the prefix will be used when determing which files to read.

getClient

A function to instantiate an S3Client.

Linear Supertypes
Serializable, Serializable, Product, Equals, RasterReader.Options, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Options
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. Options
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Options(tiffExtensions: Seq[String] = ..., crs: Option[CRS] = None, timeTag: String = GEOTIFF_TIME_TAG_DEFAULT, timeFormat: String = GEOTIFF_TIME_FORMAT_DEFAULT, maxTileSize: Option[Int] = Some(DefaultMaxTileSize), numPartitions: Option[Int] = None, partitionBytes: Option[Long] = Some(DefaultPartitionBytes), chunkSize: Option[Int] = None, delimiter: Option[String] = None, getClient: () ⇒ S3Client = S3ClientProducer.get)

    tiffExtensions

    Read all file with an extension contained in the given list.

    crs

    Override CRS of the input files. If None, the reader will use the file's original CRS.

    timeTag

    Name of tiff tag containing the timestamp for the tile.

    timeFormat

    Pattern for java.time.format.DateTimeFormatter to parse timeTag.

    maxTileSize

    Maximum allowed size of each tiles in output RDD. May result in a one input GeoTiff being split amongst multiple records if it exceeds this size. If no maximum tile size is specific, then each file is broken into 256x256 tiles. If None, then the whole file will be read in. This option is incompatible with numPartitions and anything set to that parameter will be ignored.

    numPartitions

    How many partitions Spark should create when it repartitions the data.

    partitionBytes

    Desired partition size in bytes, at least one item per partition will be assigned. If no size is specified, then partitions 128 Mb in size will be created by default. This option is incompatible with the numPartitions option. If both are set and maxTileSize isn't, then partitionBytes will be ignored in favor of numPartitions. However, if maxTileSize is set, then partitionBytes will be retained. If None and maxTileSize is defined, then the default partitionBytes' value will still be used. If maxTileSize is also None, then partitionBytes will remain None as well.

    chunkSize

    How many bytes should be read in at a time when reading a file. If None, then 65536 byte chunks will be read in at a time.

    delimiter

    Delimiter to use for S3 objet listings. This provides a way to further define what files should be read. If None, then only the prefix will be used when determing which files to read.

    getClient

    A function to instantiate an S3Client.

Value Members

  1. val chunkSize: Option[Int]
  2. val crs: Option[CRS]
    Definition Classes
    Options → Options
  3. val delimiter: Option[String]
  4. val getClient: () ⇒ S3Client
  5. val maxTileSize: Option[Int]
  6. val numPartitions: Option[Int]
  7. def parseTime(tags: Tags): ZonedDateTime
    Definition Classes
    Options
  8. val partitionBytes: Option[Long]
  9. val tiffExtensions: Seq[String]
  10. val timeFormat: String
    Definition Classes
    Options → Options
  11. lazy val timeFormatter: DateTimeFormatter
    Definition Classes
    Options
  12. val timeTag: String
    Definition Classes
    Options → Options