Prev Next

BigData / Apache Spark

Components of Apache Spark.

Spark Core contains the basic functionality of Spark, including components for task scheduling, memory management, fault recovery, interacting with storage systems, and more. Spark Core is also home to the API that defines resilient distributed datasets (RDDs), which are Spark's main programming abstraction. RDDs represent a collection of items distributed across many compute nodes that can be manipulated in parallel.

Spark SQL works with structured data. It allows querying data via SQL as well as the Apache Hive variant of SQL—called the Hive Query Language (HQL) and it supports many sources of data, including Hive tables, Parquet, and JSON.

Spark Streaming is a Spark component that enables processing of live streams of data. Examples of data streams include log files generated by production web servers or queues of messages containing status updates posted by users of a web service.

Spark comes with a library containing common machine learning (ML) functionality, called MLlib. MLlib provides various machine learning algorithms, including classification, regression, clustering, and collaborative filtering, as well as supporting functions such as model evaluation and data import. It also provides some lower-level ML primitives, including a generic gradient descent optimization algorithm. All of these methods are designed to scale out across a cluster.

GraphX is a library for manipulating graphs and performing graph-parallel computations.

To achieve this while maximizing flexibility, Spark can run over a variety of cluster managers, including Hadoop YARN, Apache Mesos, and a simple cluster manager included in Spark itself called the Standalone Scheduler.

It's right time to invest in Cryptocurrencies! Earn free bitcoins up to $250 now by signing up.

Earn bitcoins upto $250 (free), invest in other Cryptocurrencies when you signup with blockfi. Use the referral link: https://blockfi.com/?ref=f2ab78ba

Using BlockFi, don't just buy crypto - start earning on it. Open an interest account with up to 8.6% APY, trade currencies, or borrow money without selling your assets.

Invest now!!! Get Free equity stock (US,UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

More Related questions...

Show more question and Answers...

TensorFlow

Comments & Discussions