class S3LayerWriter extends LayerWriter[LayerId]

Handles writing Raster RDDs and their metadata to S3.

Linear Supertypes
LayerWriter[LayerId], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. S3LayerWriter
  2. LayerWriter
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new S3LayerWriter(attributeStore: AttributeStore, bucket: String, keyPrefix: String, putObjectModifier: (PutObjectRequest) ⇒ PutObjectRequest = identity, s3Client: ⇒ S3Client = S3ClientProducer.get(), runtime: ⇒ IORuntime = IORuntimeTransient.IORuntime)

    attributeStore

    AttributeStore to be used for storing raster metadata

    bucket

    S3 bucket to be written to

    keyPrefix

    S3 prefix to write the raster to

    putObjectModifier

    Function that will be applied ot S3 PutObjectRequests, so that they can be modified (e.g. to change the ACL settings)

    s3Client

    A function which returns an S3 Client (real or mock) into-which to save the data

    runtime

    A function to get IORuntime

Value Members

  1. val attributeStore: AttributeStore
    Definition Classes
    S3LayerWriter → LayerWriter
  2. def overwrite[K, V, M](id: LayerId, rdd: RDD[(K, V)] with Metadata[M])(implicit arg0: AvroRecordCodec[K], arg1: Boundable[K], arg2: Encoder[K], arg3: Decoder[K], arg4: ClassTag[K], arg5: AvroRecordCodec[V], arg6: ClassTag[V], arg7: Encoder[M], arg8: Decoder[M], arg9: Component[M, Bounds[K]], arg10: Semigroup[M]): Unit

    Update persisted layer without checking for possible.

    Update persisted layer without checking for possible.

    Warning: using this method may result in data loss. Unlike the LayerWriter.update this method will not check for existing records before writing the update. Use this as optimiation when you are certain that update will not overlap with existing records.

    Additional care is needed in cases where KeyIndex may map multiple, distinct, values of K to single index. This is likely with spatio-temproral layers. In these cases overwrite will replace the whole record, possibly overwriting (K,V) pairs with K is not contained in update RDD.

    The method will throw if:

    • Specified layer does not exist
    • Change the Avro schema of records is detected
    • Update RDD Bounds are outside of layer index Bounds

    Updates with empty Bounds will be ignored.

    Definition Classes
    S3LayerWriter → LayerWriter
  3. def rddWriter: S3RDDWriter
  4. def update[K, V, M](id: LayerId, rdd: RDD[(K, V)] with Metadata[M], mergeFunc: (V, V) ⇒ V)(implicit arg0: AvroRecordCodec[K], arg1: Boundable[K], arg2: Encoder[K], arg3: Decoder[K], arg4: ClassTag[K], arg5: AvroRecordCodec[V], arg6: ClassTag[V], arg7: Encoder[M], arg8: Decoder[M], arg9: Component[M, Bounds[K]], arg10: Semigroup[M]): Unit

    Update persisted layer, merging existing value with updated value.

    Update persisted layer, merging existing value with updated value.

    The layer metadata may change as result of the update to reflect added values.

    The method will throw if:

    • Specified layer does not exist
    • Change the Avro schema of records is detected
    • Update RDD Bounds are outside of layer index Bounds

    Updates with empty Bounds will be ignored.

    Definition Classes
    S3LayerWriter → LayerWriter
  5. def write[K, V, M](id: LayerId, layer: RDD[(K, V)] with Metadata[M], keyIndexMethod: KeyIndexMethod[K])(implicit arg0: AvroRecordCodec[K], arg1: Encoder[K], arg2: ClassTag[K], arg3: AvroRecordCodec[V], arg4: ClassTag[V], arg5: Encoder[M], arg6: Component[M, Bounds[K]]): Unit
    Definition Classes
    LayerWriter
  6. def write[K, V, M](id: LayerId, layer: RDD[(K, V)] with Metadata[M], keyIndex: KeyIndex[K])(implicit arg0: AvroRecordCodec[K], arg1: Encoder[K], arg2: ClassTag[K], arg3: AvroRecordCodec[V], arg4: ClassTag[V], arg5: Encoder[M], arg6: Component[M, Bounds[K]]): Unit
    Definition Classes
    LayerWriter
  7. def writer[K, V, M](keyIndex: KeyIndex[K])(implicit arg0: AvroRecordCodec[K], arg1: Encoder[K], arg2: ClassTag[K], arg3: AvroRecordCodec[V], arg4: ClassTag[V], arg5: Encoder[M], arg6: Component[M, Bounds[K]]): Writer[LayerId, RDD[(K, V)] with Metadata[M]]
    Definition Classes
    LayerWriter
  8. def writer[K, V, M](keyIndexMethod: KeyIndexMethod[K])(implicit arg0: AvroRecordCodec[K], arg1: Encoder[K], arg2: ClassTag[K], arg3: AvroRecordCodec[V], arg4: ClassTag[V], arg5: Encoder[M], arg6: Component[M, Bounds[K]]): Writer[LayerId, RDD[(K, V)] with Metadata[M]]
    Definition Classes
    LayerWriter