lakehouse.interface
- class lakehouse.interface.Interface(spark: SparkSession, **options: Dict[str, Any])
Bases:
objectA generic class as imterface for etl classes
- spark
Spark Session as provided to process the data
- Type:
SparkSession
- \*\*options
Kwargs, Any options provided into the class
- Type:
Dict[str, Any]
- catalog
Name of the created catalog recognized by spark e.g. from Hive Metastore or Unity Catalogue
- Type:
str
- source_schema
Name of the source_schema
- Type:
str
- target_schema
Name of the target_schema
- Type:
str
- __init__(spark: SparkSession, **options: Dict[str, Any]) None
Initializes the Loader class with user-provided options.
- Parameters:
spark (SparkSession) – existing Spark Session
**options (Dict[str, Any]) – Kwargs, Any options provided into the class
- Kwargs options:
catalog (str): Name of the created catalog recognized by spark e.g. from Hive Metastore or Unity Catalogue, required source_schema (str): Name of the source_schema, required target_schema (str): Name of the target_schema, required