object HadoopGeoTiffRDD
Allows for reading of whole or windowed GeoTiff as RDD[(K, V)]s through Hadoop FileSystem API.
- Alphabetic
- By Inheritance
- HadoopGeoTiffRDD
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
case class
Options(tiffExtensions: Seq[String] = ..., crs: Option[CRS] = None, timeTag: String = GEOTIFF_TIME_TAG_DEFAULT, timeFormat: String = GEOTIFF_TIME_FORMAT_DEFAULT, maxTileSize: Option[Int] = Some(DefaultMaxTileSize), numPartitions: Option[Int] = None, partitionBytes: Option[Long] = Some(DefaultPartitionBytes), chunkSize: Option[Int] = None) extends RasterReader.Options with Product with Serializable
This case class contains the various parameters one can set when reading RDDs from Hadoop using Spark.
This case class contains the various parameters one can set when reading RDDs from Hadoop using Spark.
- tiffExtensions
Read all file with an extension contained in the given list.
- crs
Override CRS of the input files. If None, the reader will use the file's original CRS.
- timeTag
Name of tiff tag containing the timestamp for the tile.
- timeFormat
Pattern for java.time.format.DateTimeFormatter to parse timeTag.
- maxTileSize
Maximum allowed size of each tiles in output RDD. May result in a one input GeoTiff being split amongst multiple records if it exceeds this size. If no maximum tile size is specific, then each file is broken into 256x256 tiles. If None, then the whole file will be read in. This option is incompatible with numPartitions and anything set to that parameter will be ignored.
- numPartitions
How many partitions Spark should create when it repartitions the data.
- partitionBytes
Desired partition size in bytes, at least one item per partition will be assigned. If no size is specified, then partitions 128 Mb in size will be created by default. This option is incompatible with the numPartitions option. If both are set and maxTileSize isn't, then partitionBytes will be ignored in favor of numPartitions. However, if maxTileSize is set, then partitionBytes will be retained. If None and maxTileSize is defined, then the default partitionBytes' value will still be used. If maxTileSize is also None, then partitionBytes will remain None as well.
- chunkSize
How many bytes should be read in at a time when reading a file. If None, then 65536 byte chunks will be read in at a time.
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final val GEOTIFF_TIME_FORMAT_DEFAULT: String("yyyy:MM:dd HH:mm:ss")
- final val GEOTIFF_TIME_TAG_DEFAULT: String("TIFFTAG_DATETIME")
-
def
apply[K, V](path: Path, options: Options)(implicit sc: SparkContext, rr: RasterReader[Options, (K, V)]): RDD[(K, V)]
Creates a RDD[(K, V)] whose K and V depends on the type of the GeoTiff that is going to be read in.
Creates a RDD[(K, V)] whose K and V depends on the type of the GeoTiff that is going to be read in.
- path
Hdfs GeoTiff path.
- options
An instance of Options that contains any user defined or default settings.
-
def
apply[I, K, V](path: Path, uriToKey: (URI, I) ⇒ K, options: Options, geometry: Option[Geometry] = None)(implicit sc: SparkContext, rr: RasterReader[Options, (I, V)]): RDD[(K, V)]
Creates a RDD[(K, V)] whose K and V depends on the type of the GeoTiff that is going to be read in.
Creates a RDD[(K, V)] whose K and V depends on the type of the GeoTiff that is going to be read in.
This function has two modes of operation: When options.maxTileSize is set windows will be read from GeoTiffs and their size and count will be balanced among partitions using partitionBytes option. Resulting partitions will be grouped in relation to GeoTiff segment layout.
When maxTileSize is None the GeoTiffs will be read fully and balanced among partitions using either numPartitions or partitionBytes option.
- path
HDFS GeoTiff path.
- uriToKey
Function to transform input key basing on the URI information.
- options
An instance of Options that contains any user defined or default settings.
- geometry
An optional geometry to filter by. If this is provided, it is assumed that all GeoTiffs are in the same CRS, and that this geometry is in that CRS.
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
multiband[K](path: Path, options: Options)(implicit sc: SparkContext, rr: RasterReader[Options, (K, MultibandTile)]): RDD[(K, MultibandTile)]
Creates RDDs with the [(K, V)] values where V is a MultibandTile.
Creates RDDs with the [(K, V)] values where V is a MultibandTile. It assumes that the provided files are MultibandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- options
An instance of Options that contains any user defined or default settings.
-
def
multiband[I, K](path: Path, uriToKey: (URI, I) ⇒ K, options: Options)(implicit sc: SparkContext, rr: RasterReader[Options, (I, MultibandTile)]): RDD[(K, MultibandTile)]
Creates RDDs with the [(K, V)] values where V is a MultibandTile.
Creates RDDs with the [(K, V)] values where V is a MultibandTile. It assumes that the provided files are MultibandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- uriToKey
function to transform input key basing on the URI information.
- options
An instance of Options that contains any user defined or default settings.
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
singleband[K](path: Path, options: Options)(implicit sc: SparkContext, rr: RasterReader[Options, (K, Tile)]): RDD[(K, Tile)]
Creates RDDs with the [(K, V)] values where V is a Tile.
Creates RDDs with the [(K, V)] values where V is a Tile. It assumes that the provided files are SinglebandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- options
An instance of Options that contains any user defined or default settings.
-
def
singleband[I, K](path: Path, uriToKey: (URI, I) ⇒ K, options: Options)(implicit sc: SparkContext, rr: RasterReader[Options, (I, Tile)]): RDD[(K, Tile)]
Creates RDDs with the [(K, V)] values where V is a Tile.
Creates RDDs with the [(K, V)] values where V is a Tile. It assumes that the provided files are SinglebandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- uriToKey
function to transform input key basing on the URI information.
- options
An instance of Options that contains any user defined or default settings.
-
def
spatial(path: Path, uriToKey: (URI, ProjectedExtent) ⇒ ProjectedExtent, options: Options)(implicit sc: SparkContext): RDD[(ProjectedExtent, Tile)]
Creates RDDs with the [(K, V)] values being ProjectedExtent and Tile, respectively.
Creates RDDs with the [(K, V)] values being ProjectedExtent and Tile, respectively. It assumes that the provided files are SinglebandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- uriToKey
function to transform input key basing on the URI information.
- options
An instance of Options that contains any user defined or default settings.
-
def
spatial(path: Path, options: Options)(implicit sc: SparkContext): RDD[(ProjectedExtent, Tile)]
Creates RDDs with the [(K, V)] values being ProjectedExtent and Tile, respectively.
Creates RDDs with the [(K, V)] values being ProjectedExtent and Tile, respectively. It assumes that the provided files are SinglebandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- options
An instance of Options that contains any user defined or default settings.
-
def
spatial(path: Path)(implicit sc: SparkContext): RDD[(ProjectedExtent, Tile)]
Creates RDDs with the [(K, V)] values being ProjectedExtent and Tile, respectively.
Creates RDDs with the [(K, V)] values being ProjectedExtent and Tile, respectively. It assumes that the provided files are SinglebandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
-
def
spatialMultiband(path: Path, uriToKey: (URI, ProjectedExtent) ⇒ ProjectedExtent, options: Options)(implicit sc: SparkContext): RDD[(ProjectedExtent, MultibandTile)]
Creates RDDs with the [(K, V)] values being ProjectedExtent and MultibandTile, respectively.
Creates RDDs with the [(K, V)] values being ProjectedExtent and MultibandTile, respectively. It assumes that the provided files are MultibandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- uriToKey
function to transform input key basing on the URI information.
- options
An instance of Options that contains any user defined or default settings.
-
def
spatialMultiband(path: Path, options: Options)(implicit sc: SparkContext): RDD[(ProjectedExtent, MultibandTile)]
Creates RDDs with the [(K, V)] values being ProjectedExtent and MultibandTile, respectively.
Creates RDDs with the [(K, V)] values being ProjectedExtent and MultibandTile, respectively. It assumes that the provided files are MultibandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- options
An instance of Options that contains any user defined or default settings.
-
def
spatialMultiband(path: Path)(implicit sc: SparkContext): RDD[(ProjectedExtent, MultibandTile)]
Creates RDDs with the [(K, V)] values being ProjectedExtent and MultibandTile, respectively.
Creates RDDs with the [(K, V)] values being ProjectedExtent and MultibandTile, respectively. It assumes that the provided files are MultibandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
temporal(path: Path, uriToKey: (URI, TemporalProjectedExtent) ⇒ TemporalProjectedExtent, options: Options)(implicit sc: SparkContext): RDD[(TemporalProjectedExtent, Tile)]
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and Tile, respectively.
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and Tile, respectively. It assumes that the provided files are SinglebandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- uriToKey
function to transform input key basing on the URI information.
- options
An instance of Options that contains any user defined or default settings.
-
def
temporal(path: Path, options: Options)(implicit sc: SparkContext): RDD[(TemporalProjectedExtent, Tile)]
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and Tile, respectively.
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and Tile, respectively. It assumes that the provided files are SinglebandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- options
An instance of Options that contains any user defined or default settings.
-
def
temporal(path: Path)(implicit sc: SparkContext): RDD[(TemporalProjectedExtent, Tile)]
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and Tile, respectively.
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and Tile, respectively. It assumes that the provided files are SinglebandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
-
def
temporalMultiband(path: Path, uriToKey: (URI, TemporalProjectedExtent) ⇒ TemporalProjectedExtent, options: Options)(implicit sc: SparkContext): RDD[(TemporalProjectedExtent, MultibandTile)]
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and MultibandTile, respectively.
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and MultibandTile, respectively. It assumes that the provided files are MultibandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- uriToKey
function to transform input key basing on the URI information.
- options
An instance of Options that contains any user defined or default settings.
-
def
temporalMultiband(path: Path, options: Options)(implicit sc: SparkContext): RDD[(TemporalProjectedExtent, MultibandTile)]
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and MultibandTile, respectively.
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and MultibandTile, respectively. It assumes that the provided files are MultibandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
- options
An instance of Options that contains any user defined or default settings.
-
def
temporalMultiband(path: Path)(implicit sc: SparkContext): RDD[(TemporalProjectedExtent, MultibandTile)]
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and MultibandTile, respectively.
Creates RDDs with the [(K, V)] values being TemporalProjectedExtent and MultibandTile, respectively. It assumes that the provided files are MultibandGeoTiffs.
- path
Hadoop path to recursively search for GeoTiffs.
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
- object Options extends Serializable