Interview Questions.

Question 1. What Is Shark?

Answer :

Most of the data customers know best SQL and are not proper at programming. Shark is a device, developed for individuals who are from a database history - to get entry to Scala MLib talents through Hive like SQL interface. Shark tool helps data customers run Hive on Spark - imparting compatibility with Hive metastore, queries and data.

Question 2. Most Of The Data Users Know Only Sql And Are Not Good At Programming. Shark Is A Tool, Developed For People Who Are From A Database Background - To Access Scala Mlib Capabilities Through Hive Like Sql Interface. Shark Tool Helps Data Users Run Hive On Spark - Offering Compatibility With Hive Metastore, Queries And Data.

Answer :

Sensor Data Processing –Apache Spark’s ‘In-memory computing’ works pleasant right here, as facts is retrieved and mixed from extraordinary resources.
Spark is favored over Hadoop for actual time querying of records
Stream Processing – For processing logs and detecting frauds in live streams for alerts, Apache Spark is the quality solution.
Python Interview Questions
Question three. What Is A Sparse Vector?

Answer :

sparse vector has two parallel arrays –one for indices and the alternative for values. These vectors are used for storing non-zero entries to store area.

Question four. What Is Rdd?

Answer :

RDDs (Resilient Distributed Datasets) are fundamental abstraction in Apache Spark that constitute the information entering the system in item layout. RDDs are used for in-reminiscence computations on massive clusters, in a fault tolerant way. RDDs are examine-most effective portioned, collection of information, which are –

Immutable – RDDs cannot be altered.
Resilient – If a node keeping the partition fails the other node takes the information.
Python Tutorial
Question five. Explain About Transformations And Actions In The Context Of Rdds.

Answer :

Transformations are capabilities done on demand, to supply a new RDD. All adjustments are observed with the aid of moves. Some examples of variations include map, filter and reduceByKey.

Actions are the results of RDD computations or variations. After an movement is accomplished, the statistics from RDD actions again to the nearby gadget. Some examples of moves include lessen, collect, first, and take.

Core Java Interview Questions
Question 6. What Are The Languages Supported By Apache Spark For Developing Big Data Applications?

Answer :

Scala, Java, Python, R and Clojure

Question 7. Can You Use Spark To Access And Analyse Data Stored In Cassandra Databases?

Answer :

Yes, it is possible in case you use Spark Cassandra Connector.

Core Java Tutorial JDBC Interview Questions
Question 8. Is It Possible To Run Apache Spark On Apache Mesos?

Answer :

Yes, Apache Spark can be run at the hardware clusters managed through Mesos.

Question nine. Explain About The Different Cluster Managers In Apache Spark

Answer :

The three one-of-a-kind clusters managers supported in Apache Spark are:

YARN
Apache Mesos -Has wealthy resource scheduling competencies and is properly ideal to run Spark in conjunction with different applications. It is tremendous while numerous customers run interactive shells as it scales down the CPU allocation among instructions.
Standalone deployments – Well acceptable for brand spanking new deployments which simplest run and are smooth to set up.
DB2 Using SQL Interview Questions
Question 10. How Can Spark Be Connected To Apache Mesos?

Answer :

To join Spark with Mesos-

Configure the spark driving force software to connect with Mesos. Spark binary package must be in a location on hand with the aid of Mesos. (or)
Install Apache Spark within the equal place as that of Apache Mesos and configure the assets ‘spark.Mesos.Executor.Domestic’ to factor to the region wherein it's miles hooked up.
JDBC Tutorial
Question 11. How Can You Minimize Data Transfers When Working With Spark?

Answer :

Minimizing records transfers and avoiding shuffling allows write spark packages that run in a fast and dependable way. The numerous approaches wherein facts transfers may be minimized whilst running with Apache Spark are:

Using Broadcast Variable- Broadcast variable complements the performance of joins among small and huge RDDs.
Using Accumulators – Accumulators help replace the values of variables in parallel at the same time as executing.
The maximum common way is to avoid operations ByKey, repartition or any other operations which trigger shuffles.
Domain Name System(DNS) Interview Questions
Question 12. Why Is There A Need For Broadcast Variables When Working With Apache Spark?

Answer :

These are read most effective variables, found in-memory cache on each gadget. When working with Spark, utilization of broadcast variables removes the need to deliver copies of a variable for every undertaking, so facts may be processed quicker. Broadcast variables help in storing a lookup desk inside the reminiscence which complements the retrieval efficiency while as compared to an RDD lookup ().

Python Interview Questions
Question thirteen. Is It Possible To Run Spark And Mesos Along With Hadoop?

Answer :

Yes, it's miles viable to run Spark and Mesos with Hadoop by means of launching each of these as a separate provider at the machines. Mesos acts as a unified scheduler that assigns duties to both Spark or Hadoop.

DB2 Using SQL Tutorial
Question 14. What Is Lineage Graph?

Answer :

The RDDs in Spark, depend on one or more other RDDs. The illustration of dependencies in between RDDs is called the lineage graph. Lineage graph records is used to compute each RDD on call for, in order that every time a part of continual RDD is lost, the statistics that is misplaced can be recovered the use of the lineage graph facts.

Question 15. How Can You Trigger Automatic Clean-united statesIn Spark To Handle Accumulated Metadata?

Answer :

You can trigger the clean-u.S.A.With the aid of putting the parameter ‘spark.Cleanser.Ttl’ or by way of dividing the lengthy going for walks jobs into exclusive batches and writing the middleman results to the disk.

Hibernate Interview Questions
Question sixteen. Explain About The Major Libraries That Constitute The Spark Ecosystem

Answer :

Spark MLib- Machine mastering library in Spark for commonly used learning algorithms like clustering, regression, category, and many others.
Spark Streaming – This library is used to manner real time streaming data.
Spark GraphX – Spark API for graph parallel computations with basic operators like joinVertices, subgraph, aggregateMessages, and many others.
Spark SQL – Helps execute SQL like queries on Spark data the use of trendy visualization or BI equipment.
Hibernate Tutorial
Question 17. What Are The Benefits Of Using Spark With Apache Mesos?

Answer :

It renders scalable partitioning among various Spark instances and dynamic partitioning among Spark and other large records frameworks.

MYSQL DBA Interview Questions
Question 18. What Is The Significance Of Sliding Window Operation?

Answer :

Sliding Window controls transmission of records packets among various pc networks. Spark Streaming library affords windowed computations where the differences on RDDs are applied over a sliding window of statistics. Whenever the window slides, the RDDs that fall in the precise window are mixed and operated upon to provide new RDDs of the windowed DStream.

Core Java Interview Questions
Question 19. What Is A Dstream?

Answer :

Discretized Stream is a sequence of Resilient Distributed Databases that represent a movement of facts. DStreams may be constituted of diverse sources like Apache Kafka, HDFS, and Apache Flume. DStreams have two operations –

Transformations that produce a brand new DStream.
Output operations that write records to an external gadget.
Java Swing Tutorial
Question 20. When Running Spark Applications, Is It Necessary To Install Spark On All The Nodes Of Yarn Cluster?

Answer :

Spark want no longer be established whilst walking a activity below YARN or Mesos because Spark can execute on pinnacle of YARN or Mesos clusters without affecting any change to the cluster.

Java Swing Interview Questions
Question 21. What Is Catalyst Framework?

Answer :

Catalyst framework is a brand new optimization framework found in Spark SQL. It lets in Spark to robotically transform SQL queries through including new optimizations to construct a quicker processing machine.

Question 22. Name A Few Companies That Use Apache Spark In Production.

Answer :

Pinterest, Conviva, Shopify, Open Table

Scala Tutorial
Question 23. Which Spark Library Allows Reliable File Sharing At Memory Speed Across Different Cluster Frameworks?

Answer :

Tachyon

Work On Interesting Data Science Projects using Spark to construct an impressive task portfolio!

SQLite Interview Questions
Question 24. Why Is Blinkdb Used?

Answer :

BlinkDB is a question engine for executing interactive SQL queries on massive volumes of statistics and renders question effects marked with significant mistakes bars. BlinkDB helps users stability ‘question accuracy’ with response time.

JDBC Interview Questions
Question 25. How Can You Compare Hadoop And Spark In Terms Of Ease Of Use?

Answer :

Hadoop MapReduce requires programming in Java that's hard, though Pig and Hive make it appreciably less difficult. Learning Pig and Hive syntax takes time. Spark has interactive APIs for special languages like Java, Python or Scala and additionally includes Shark i.E. Spark SQL for SQL lovers - making it comparatively less complicated to apply than Hadoop.

Apache ZooKeeper Tutorial
Question 26. What Are The Common Mistakes Developers Make When Running Spark Applications?

Answer :

Developers often make the error of-

Hitting the net carrier several times by means of the usage of a couple of clusters.
Run the whole thing on the local node instead of dispensing it.
Developers need to be careful with this, as Spark uses reminiscence for processing.

Apache Spark Interview Questions
Question 27. What Is The Advantage Of A Parquet File?

Answer :

Parquet record is a columnar format file that enables –

Limit I/O operations
Consumes less area
Fetches most effective required columns.
DB2 Using SQL Interview Questions
Question 28. What Are The Various Data Sources Available In Sparksql?

Answer :

Parquet document
JSON Datasets
Hive tables
Hyper SQL Database Tutorial
Question 29. How Spark Uses Hadoop?

Answer :

Spark has its own cluster control computation and mainly uses Hadoop for storage.

Scala Interview Questions
Question 30. What Are The Key Features Of Apache Spark That You Like?

Answer :

Spark offers superior analytic alternatives like graph algorithms, device mastering, streaming data, and so on
It has integrated APIs in more than one languages like Java, Scala, Python and R
It has top overall performance gains, as it enables run an software in the Hadoop cluster ten times quicker on disk and one hundred instances faster in memory.
Question 31. What Do You Understand By Pair Rdd?

Answer :

Special operations can be done on RDDs in Spark using key/value pairs and such RDDs are referred to as Pair RDDs. Pair RDDs allow customers to access every key in parallel. They have a reduceByKey () method that collects facts based totally on every key and a be a part of () technique that combines exclusive RDDs together, based on the factors having the same key.

Question 32. Which One Will You Choose For A Project –hadoop Mapreduce Or Apache Spark?

Answer :

As it is recognized that Spark uses memory rather than network and disk I/O. However, Spark uses huge quantity of RAM and calls for committed system to provide powerful consequences. So the decision to apply Hadoop or Spark varies dynamically with the requirements of the assignment and budget of the employer.

Apache ZooKeeper Interview Questions
Question 33. Explain About The Different Types Of Transformations On Dstreams?

Answer :

Stateless Transformations- Processing of the batch does no longer depend upon the output of the preceding batch. Examples – map (), reduceByKey (), filter ().
Stateful Transformations- Processing of the batch depends at the middleman effects of the preceding batch. Examples –Transformations that rely upon sliding home windows.
Domain Name System(DNS) Interview Questions
Question 34. Explain About The Popular Use Cases Of Apache Spark

Answer :

Apache Spark is in particular used for

Iterative gadget studying.
Interactive facts analytics and processing.
Stream processing
Sensor data processing
Question 35. Is Apache Spark A Good Fit For Reinforcement Learning?

Answer :

No. Apache Spark works properly simplest for easy machine getting to know algorithms like clustering, regression, classification.

Question 36. What Is Spark Core?

Answer :

It has all of the basic functionalities of Spark, like - memory control, fault restoration, interacting with garage systems, scheduling obligations, etc.

Hibernate Interview Questions
Question 37. How Can You Remove The Elements With A Key Present In Any Other Rdd?

Answer :

Use the subtractByKey () function

Question 38. What Is The Difference Between Persist() And Cache()

Answer :

persist () allows the consumer to specify the storage degree while cache () makes use of the default storage stage.

Question 39. What Are The Various Levels Of Persistence In Apache Spark?

Answer :

Apache Spark mechanically persists the intermediary statistics from numerous shuffle operations, however it's far frequently suggested that users name persist () approach on the RDD in case they plan to reuse it. Spark has various endurance ranges to keep the RDDs on disk or in reminiscence or as a mixture of both with special replication stages.

The diverse storage/persistence ranges in Spark are -

MEMORY_ONLY
MEMORY_ONLY_SER
MEMORY_AND_DISK
MEMORY_AND_DISK_SER, DISK_ONLY
OFF_HEAP
Question 40. How Spark Handles Monitoring And Logging In Standalone Mode?

Answer :

Spark has a web based consumer interface for tracking the cluster in standalone mode that shows the cluster and process statistics. The log output for every process is written to the paintings directory of the slave nodes.

MYSQL DBA Interview Questions
Question forty one. Does Apache Spark Provide Check Pointing?

Answer :

Lineage graphs are usually beneficial to get better RDDs from a failure however that is generally time ingesting if the RDDs have long lineage chains. Spark has an API for take a look at pointing i.E. A REPLICATE flag to persist. However, the choice on which records to checkpoint - is decided via the consumer. Checkpoints are useful whilst the lineage graphs are lengthy and have wide dependencies.

Question forty two. How Can You Launch Spark Jobs Inside Hadoop Mapreduce?

Answer :

Using SIMR (Spark in MapReduce) customers can run any spark activity inner MapReduce without requiring any admin rights.

Java Swing Interview Questions
Question forty three. How Spark Uses Akka?

Answer :

Spark uses Akka essentially for scheduling. All the people request for a challenge to master after registering. The grasp just assigns the venture. Here Spark uses Akka for messaging between the people and masters.

Question 44. How Can You Achieve High Availability In Apache Spark?

Answer :

Implementing single node recuperation with local record system
Using StandBy Masters with Apache ZooKeeper.
Question 45. Hadoop Uses Replication To Achieve Fault Tolerance. How Is This Achieved In Apache Spark?

Answer :

Data garage model in Apache Spark is based on RDDs. RDDs assist reap fault tolerance through lineage. RDD always has the statistics on the way to build from different datasets. If any partition of a RDD is lost due to failure, lineage allows build only that particular misplaced partition.

Question 46. Explain About The Core Components Of A Distributed Spark Application.

Answer :

Driver- The method that runs the primary () approach of this system to create RDDs and carry out differences and movements on them.
Executor –The worker strategies that run the man or woman duties of a Spark activity.
Cluster Manager-A pluggable thing in Spark, to launch Executors and Drivers. The cluster supervisor allows Spark to run on pinnacle of other outside managers like Apache Mesos or YARN.
Question forty seven. What Do You Understand By Lazy Evaluation?

Answer :

Spark is highbrow inside the manner in which it operates on data. When you inform Spark to operate on a given dataset, it heeds the commands and makes a word of it, in order that it does not neglect - however it does nothing, until asked for the final result. When a change like map () is called on a RDD-the operation isn't carried out without delay. Transformations in Spark are not evaluated until you perform an motion. This enables optimize the overall facts processing workflow.

Question 48. Define A Worker Node.

Answer :

A node that could run the Spark utility code in a cluster can be referred to as as a employee node. A worker node will have a couple of employee which is configured by way of putting the SPARK_ WORKER_INSTANCES property inside the spark-env.Sh report. Only one worker is commenced if the SPARK_ WORKER_INSTANCES assets is not described.

Question forty nine. What Do You Understand By Schemardd?

Answer :

An RDD that includes row items (wrappers round primary string or integer arrays) with schema statistics approximately the sort of facts in every column.

Question 50. What Are The Disadvantages Of Using Apache Spark Over Hadoop Mapreduce?

Answer :

Apache spark does now not scale nicely for compute intensive jobs and consumes huge wide variety of device assets. Apache Spark’s in-reminiscence functionality at times comes a major roadblock for value efficient processing of large information. Also, Spark does have its personal document management device and for this reason wishes to be integrated with other cloud based totally statistics systems or apache hadoop.

Question fifty one. Is It Necessary To Install Spark On All The Nodes Of A Yarn Cluster While Running Apache Spark On Yarn ?

Answer :

No , it isn't necessary because Apache Spark runs on pinnacle of YARN.

Question fifty two. What Do You Understand By Executor Memory In A Spark Application?

Answer :

Every spark software has identical constant heap size and fixed range of cores for a spark executor. The heap size is what known as the Spark executor reminiscence that's managed with the spark.Executor.Reminiscence property of the –executor-reminiscence flag. Every spark application could have one executor on every worker node. The executor reminiscence is essentially a degree on how a whole lot reminiscence of the worker node will the software utilize.

Question 53. What Does The Spark Engine Do?

Answer :

Spark engine schedules, distributes and monitors the records utility throughout the spark cluster.

Question 54. What Makes Apache Spark Good At Low-latency Workloads Like Graph Processing And Machine Learning?

Answer :

Apache Spark shops statistics in-reminiscence for faster model building and schooling. Machine getting to know algorithms require more than one iterations to generate a ensuing most beneficial version and in addition graph algorithms traverse all of the nodes and edges.These low latency workloads that need multiple iterations can lead to expanded performance. Less disk access and  managed community traffic make a large distinction when there is lots of records to be processed.

Question 55. Is It Necessary To Start Hadoop To Run Any Apache Spark Application ?

Answer :

Starting hadoop isn't always manadatory to run any spark software. As there is no seperate storage in Apache Spark, it makes use of Hadoop HDFS however it isn't mandatory. The facts can be saved in neighborhood document system, may be loaded from nearby record device and processed.

Question fifty six. What Is The Default Level Of Parallelism In Apache Spark?

Answer :

If the user does now not explicitly specify then the quantity of partitions are taken into consideration as default degree of parallelism in Apache Spark.

Question fifty seven. Explain About The Common Workflow Of A Spark Program

Answer :

The foremost step in a Spark application entails developing enter RDD's from outside statistics.
Use diverse RDD adjustments like clear out() to create new transformed RDD's primarily based on the business logic.
Persist() any intermediate RDD's which might must be reused in future.
Launch numerous RDD actions() like first(), depend() to start parallel computation , which will then be optimized and completed by Spark.
Question 58. Name A Few Commonly Used Spark Ecosystems.

Answer :

Spark SQL (Shark)

Spark Streaming

GraphX

MLlib

SparkR

Question fifty nine. What Is “spark Sql”?

Answer :

Spark SQL is a Spark interface to paintings with established as well as semi-established facts. It has the capability to load data from more than one structured sources like “textual content documents”, JSON files, Parquet files, amongst others. Spark SQL presents a special form of RDD known as SchemaRDD. These are row gadgets, where each item represents a document.

Question 60. Can We Do Real-time Processing Using Spark Sql?

Answer :

Not directly but we can check in an current RDD as a SQL table and trigger SQL queries on pinnacle of that.

Question sixty one. What Is Spark Sql?

Answer :

SQL Spark, better known as Shark is a novel module added in Spark to paintings with established information and carry out based information processing. Through this module, Spark executes relational SQL queries at the facts. The center of the element helps an altogether one-of-a-kind RDD known as SchemaRDD, composed of rows objects and schema objects defining records form of each column inside the row. It is similar to a table in relational database.

Question sixty two. What Is A Parquet File?

Answer :

Parquet is a columnar layout file supported by means of many different statistics processing structures. Spark SQL performs each examine and write operations with Parquet report and recollect it's one of the best huge information analytics format to this point.

Question sixty three. List The Functions Of Spark Sql.

Answer :

Spark SQL is able to:

Loading records from an expansion of structured sources
Querying records the use of SQL statements, each inner a Spark application and from outside tools that connect to Spark SQL via general database connectors (JDBC/ODBC). For instance, the use of enterprise intelligence gear like Tableau
Providing wealthy integration between SQL and normal Python/Java/Scala code, which includes the capability to sign up for RDDs and SQL tables, disclose custom features in SQL, and greater
Question 64. What Is Spark?

Answer :

Spark is a parallel records processing framework. It lets in to increase speedy, unified massive facts utility integrate batch, streaming and interactive analytics.

Question sixty five. What Is Hive On Spark?

Answer :

Hive is a component of Hortonworks’ Data Platform (HDP). Hive gives an SQL-like interface to data stored inside the HDP. Spark customers will automatically get the entire set of Hive’s rich features, including any new features that Hive might introduce in the destiny.

The essential task around enforcing the Spark execution engine for Hive lies in question planning, in which Hive operator plans from the semantic analyzer which is translated to a challenge plan that Spark can execute. It also consists of query execution, wherein the generated Spark plan receives actually completed inside the Spark cluster.

Question sixty six. What Is A “parquet” In Spark?

Answer :

“Parquet” is a columnar layout file supported with the aid of many information processing systems. Spark SQL plays both read and write operations with the “Parquet” file.

Question 67. What Are Benefits Of Spark Over Mapreduce?

Answer :

Due to the availability of in-memory processing, Spark implements the processing around 10-100x faster than Hadoop MapReduce. MapReduce uses staying power garage for any of the information processing duties.

Unlike Hadoop, Spark offers in-constructed libraries to carry out more than one obligations shape the same core like batch processing, Steaming, Machine studying, Interactive SQL queries. However, Hadoop most effective supports batch processing.
Hadoop is highly disk-structured while Spark promotes caching and in-memory facts storage
Spark is able to appearing computations more than one times at the equal dataset. This is known as iterative computation even as there may be no iterative computing carried out by way of Hadoop.
Question sixty eight. How Sparksql Is Different From Hql And Sql?

Answer :

SparkSQL is a unique issue on the spark Core engine that assist SQL and Hive Query Language with out converting any syntax. It’s feasible to enroll in SQL table and HQL desk.