BigData / Apache Parquet Interview Questions
How do you read and write Parquet files in PySpark?
Spark provides first-class Parquet support via the DataFrameReader and DataFrameWriter APIs.
Read:
# Read a single file or directory of Parquet files df = spark.read.parquet("s3://my-bucket/events/") # With options df = (spark.read .option("mergeSchema", "true") .parquet("hdfs:///datalake/transactions/"))
Write:
# Overwrite with Snappy compression (default) df.write.mode("overwrite").parquet("s3://my-bucket/output/") # Partition by date and region, use ZSTD (df.write .partitionBy("date", "region") .option("compression", "zstd") .mode("append") .parquet("s3://my-bucket/partitioned/"))
Register as temp view for SQL:
df.createOrReplaceTempView("events") spark.sql("SELECT date, SUM(revenue) FROM events GROUP BY date").show()
Which PySpark write mode appends new data to an existing Parquet directory without overwriting existing files?
What method creates a SQL-queryable alias for a Spark DataFrame?
More Related questions...
