BigData / Apache Spark
1. What is Apache spark?
Apache Spark is an open-source cluster computing framework for real-time processing. Spark provides an interface for programming entire clusters with implicit data parallelism and fault-tolerance.
2. Key features of Apache Spark.
Speed : Spark runs faster than Hadoop MapReduce for large-scale data through controlled partitioning. Spark manages data using partitions that help parallelize distributed data processing with minimal network traffic. Real Time Computation : Spark perform computation in real-time and has less lat...
3. Does Apache spark provide real-time processing?
Yes. Apache spark supports real-time processing through spark streaming .
4. Components of Apache Spark.
Spark Core contains the basic functionality of Spark, including components for task scheduling, memory management, fault recovery, interacting with storage systems, and more. Spark Core is also home to the API that defines resilient distributed datasets (RDDs), which are Spark's main programming ...
5. What is Resilient Distribution Datasets (RDD)?
Resilient Distribution Datasets (RDD), a fault-tolerant assortment of operational elements that run parallel. The partitioned data in RDD is immutable and distributed.
6. What is transformations and actions in the RDDs?
Transformations are functions executed on demand, to produce a new RDD. All transformations are followed by actions. Some examples of transformations include map, filter and reduceByKey. Actions are the results of RDD computations or transformations. After an action is performed, the data from RD...
7. What is the role of Spark Engine?
Spark Engine schedules, distributes and monitors the data application across the spark clusters.
8. Name the operations that Apache Spark RDD supports.
Transformation and Action .
9. What is Spark Driver?
Spark Driver program runs on the master node of the machine and declares transformations and actions on data RDDs. The driver also delivers the RDD graphs to Master, where the standalone cluster manager runs.
10. What is Spark shell?
Sparks shell provides a simple way to learn the API, as well as a powerful tool to analyze data interactively. It is available in either Scala or Python. Scala: ./bin/spark-shell Python: ./bin/pyspark
11. What is Dataset in Apache spark?
After Spark 2.0, RDDs are replaced by Dataset, which is strongly-typed like an RDD, but with richer optimizations. Dataset is the Sparks primary abstraction of distributed collection of items. Datasets can be created from Hadoop InputFormats (such as HDFS files) or by transforming other Datasets.
12. Explain spark streaming.
Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Kinesis, or TCP sockets, and can be processed using complex algorithms expressed with high...
13. Types of shared variables in Spark.
Spark supports 2 types of shared variables: broadcast variables and accumulators. Broadcast variables allow the programmer to keep a read-only variable cached on each machine to give every node a copy of a large input dataset in an efficient manner. Accumulators are variables that are only "added...
14. Different Spark cluster managers.
Spark standalone cluster, Apache MESOS, Hadoop YARN, and kubernetes.
15. Different spark shells.
Spark-shell with Scala support, PySpark with python support, and SparkR with R support.
16. Advantages of Spark SQL.
Spark SQL executes faster than Hive. Hive code can be easily migrated to Spark SQL. Spark SQL enables real time querying capabilities.
17. What is MLlib in Spark?
Spark MLlib is used to perform machine learning using its inbuilt algorithms and utilities. It consists of 2 packages. spark.mllib contains the original API built on top of RDDs. spark.ml provides higher level API built on top of data frames for constructing ML pipelines. spark.ml is the primary ...
18. Explain Datasets and DataFrames in Apache spark.
A Dataset is a distributed collection of data. Dataset is a new interface added in Spark 1.6 that provides the benefits of RDDs such as strong typing, ability to use powerful lambda functions. A Dataset can be constructed from JVM objects and then manipulated using functional transformations such...
19. Explain vectorAssembler in MLlib.
VectorAssembler is a transformer that combines a given list of columns into a single vector column. VectorAssembler accepts the following input column types: all numeric types, boolean type, and vector type. In each row, the values of the input columns will be concatenated into a vector in the sp...
20. How do I read multiline JSON in Apache Spark?
Spark 2.2 introduced multiLine option which can be used to load JSON. val vaDF = spark.read.option("multiLine",true).json("vectorAssemblerTest.data")
21. Difference between map and flatmap transformations.
map(func) returns a new distributed dataset formed by passing each element of the source through a function func. flatMap(func) is similar to map, except that each input item can be mapped to 0 or more output items so that func should return a Seq rather than a single item.
