{"id":22,"date":"2023-06-13T15:50:21","date_gmt":"2023-06-13T15:50:21","guid":{"rendered":"https:\/\/farrukhnaveed.co\/blog\/?p=22"},"modified":"2023-09-22T23:06:11","modified_gmt":"2023-09-22T17:36:11","slug":"introduction-to-apache-spark","status":"publish","type":"post","link":"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/","title":{"rendered":"Introduction to Apache Spark"},"content":{"rendered":"\n

Introduction<\/h2>\n\n\n\n

Apache Spark<\/a>  is a robust framework for processing data, enabling swift execution of tasks on extensive datasets. It excels in distributing data processing across multiple computers, both independently and in conjunction with other distributed computing tools. These crucial attributes make it indispensable in the realms of big data and machine learning, where immense computational power is essential for handling vast data repositories. Additionally, Spark simplifies programming for developers through its user-friendly API, relieving them of the complexities associated with distributed computing and processing large-scale data.<\/p>\n\n\n\n


Since its inception at the esteemed AMPLab at U.C. Berkeley in 2009, Apache Spark has evolved into a prominent distributed processing framework for big data. Its versatility is evident in its ability to be deployed in various manners, offering native support for popular programming languages like Java, Scala, Python, and R. Furthermore, Spark encompasses a wide range of functionalities, including SQL, streaming data processing, machine learning, and graph processing. Its extensive adoption spans across diverse sectors, including banking, telecommunications, gaming, government agencies, and major tech titans like Apple, IBM, Meta, and Microsoft.<\/p>\n\n\n\n

Spark RDD<\/h2>\n\n\n\n

At the core of Apache Spark lies the fundamental notion of the Resilient Distributed Dataset (RDD), which serves as a programming abstraction to represent an unchanging collection of objects that can be divided across a computing cluster. These RDDs enable operations to be split and executed in parallel batches across the cluster, facilitating rapid and scalable parallel processing. Apache Spark transforms the user’s commands for data processing into a Directed Acyclic Graph (DAG), which acts as the scheduling layer determining the sequencing and allocation of tasks across nodes.<\/p>\n\n\n\n

RDDs can be effortlessly generated from various sources such as plain text files, SQL databases, NoSQL repositories like Cassandra and MongoDB, Amazon S3 buckets, and many more. The Spark Core API heavily relies on the RDD concept, offering not only traditional map and reduce functionalities but also integrated support for data set joining, filtering, sampling, and aggregation.<\/p>\n\n\n\n

Spark runs in a distributed fashion by combining a driver<\/em> core process that splits a Spark application into tasks and distributes them among many executor<\/em> processes that do the work. These executors can be scaled up and down as required for the application\u2019s needs.<\/p>\n\n\n\n

Spark SQL<\/h2>\n\n\n\n

Spark SQL<\/a> has become more and more important to the Apache Spark project. It is the interface most commonly used by today\u2019s developers when creating applications. Spark SQL is focused on the processing of structured data, using a dataframe approach borrowed from R and Python (in Pandas). But as the name suggests, Spark SQL also provides a SQL2003-compliant interface for querying data, bringing the power of Apache Spark to analysts as well as developers.<\/p>\n\n\n\n

Alongside standard SQL support, Spark SQL provides a standard interface for reading from and writing to other datastores including JSON, HDFS, Apache Hive, JDBC, Apache ORC, and Apache Parquet, all of which are supported out of the box. Other popular data stores\u2014Apache Cassandra, MongoDB, Apache HBase, and many others\u2014can be used by pulling in separate connectors from the Spark Packages<\/a> ecosystem. Spark SQL allows user-defined functions (UDFs) to be transparently used in SQL queries.Nominations are open for the 2024 Best Places to Work in IT<\/a><\/p>\n\n\n\n

Selecting some columns from a dataframe is as simple as this line of code:<\/p>\n\n\n\n

<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
citiesDF.select(<\/span>"<\/span>name<\/span>"<\/span>, <\/span>"<\/span>pop<\/span>"<\/span>)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Using the SQL interface, we register the dataframe as a temporary table, after which we can issue SQL queries against it:<\/p>\n\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
citiesDF.createOrReplaceTempView(<\/span>"<\/span>cities<\/span>"<\/span>)<\/span><\/span>\nspark.sql(<\/span>"<\/span>SELECT name, pop FROM cities<\/span>"<\/span>)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Spark MLlib and MLflow<\/h2>\n\n\n\n
Apache Spark also bundles libraries for applying machine learning and graph analysis techniques to data at scale. MLlib<\/a> includes a framework for creating machine learning pipelines, allowing for easy implementation of feature extraction, selections, and transformations on any structured dataset. MLlib comes with distributed implementations of clustering and classification algorithms such as k-means clustering and random forests that can be swapped in and out of custom pipelines with ease. Models can be trained by data scientists in Apache Spark using R or Python, saved using MLlib, and then imported into a Java-based or Scala-based pipeline for production use.<\/p>\n\n\n\n
An open source platform for managing the machine learning life cycle, MLflow<\/a> is not technically part of the Apache Spark project, but it is likewise a product of Databricks and others<\/a> in the Apache Spark community. The community has been working on integrating MLflow with Apache Spark to provide MLOps<\/a> features like experiment tracking, model registries, packaging, and UDFs that can be easily imported for inference at Apache Spark scale and with traditional SQL statements.<\/p>\n\n\n\n
Structured Streaming<\/h2>\n\n\n\n
Structured Streaming<\/a> is a high-level API that allows developers to create infinite streaming dataframes and datasets. As of Spark 3.0, Structured Streaming is the recommended way of handling streaming data within Apache Spark, superseding the earlier Spark Streaming<\/a> approach. Spark Streaming (now marked as a legacy component) was full of difficult pain points for developers, especially when dealing with event-time aggregations and late delivery of messages.<\/p>\n\n\n\n
All queries on structured streams go through the Catalyst query optimizer, and they can even be run in an interactive manner, allowing users to perform SQL queries against live streaming data. Support for late messages is provided by watermarking messages and three supported types of windowing techniques: tumbling windows, sliding windows, and variable-length time windows with sessions.<\/p>\n\n\n\n
In Spark 3.1 and later, you can treat streams as tables, and tables as streams. The ability to combine multiple streams with a wide range of SQL-like stream-to-stream joins creates powerful possibilities for ingestion and transformation. Here\u2019s a simple example of creating a table from a streaming source:<\/p>\n\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
val<\/span> df <\/span>=<\/span> spark.readStream<\/span><\/span>\n  .format(<\/span>"<\/span>rate<\/span>"<\/span>)<\/span><\/span>\n  .option(<\/span>"<\/span>rowsPerSecond<\/span>"<\/span>, <\/span>20<\/span>)<\/span><\/span>\n  .load()<\/span><\/span>\n<\/span>\ndf.writeStream<\/span><\/span>\n  .option(<\/span>"<\/span>checkpointLocation<\/span>"<\/span>, <\/span>"<\/span>checkpointPath<\/span>"<\/span>)<\/span><\/span>\n  .toTable(<\/span>"<\/span>streamingTable<\/span>"<\/span>)<\/span><\/span>\n<\/span>\nspark.read.table(<\/span>"<\/span>myTable<\/span>"<\/span>).show()<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Structured Streaming is the future of streaming applications with the Apache Spark platform, so if you\u2019re building a new streaming application, you should use Structured Streaming. The legacy Spark Streaming APIs will continue to be supported, but the project recommends porting over to Structured Streaming, as the new method makes writing and maintaining streaming code a lot more bearable.<\/p>\n\n\n\n
Delta Lake<\/h2>\n\n\n\n
Like MLflow, Delta Lake<\/a> is technically a separate project from Apache Spark. Over the past couple of years, however, Delta Lake has become an integral part of the Spark ecosystem, forming the core of what Databricks calls the Lakehouse Architecture<\/a>. Delta Lake augments cloud-based data lakes with ACID transactions, unified querying semantics for batch and stream processing, and schema enforcement, effectively eliminating the need for a separate data warehouse for BI users. Full audit history and scalability to handle exabytes of data are also part of the package.<\/p>\n\n\n\n
And using the Delta Lake format (built on top of Parquet files) within Apache Spark is as simple as using the delta<\/code> format:<\/p>\n\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
df <\/span>=<\/span> spark.readStream.format(<\/span>"<\/span>rate<\/span>"<\/span>).load()<\/span><\/span>\n<\/span>\nstream <\/span>=<\/span> df <\/span><\/span>\n  .writeStream<\/span><\/span>\n  .format(<\/span>"<\/span>delta<\/span>"<\/span>) <\/span><\/span>\n  .option(<\/span>"<\/span>checkpointLocation<\/span>"<\/span>, <\/span>"<\/span>checkpointPath<\/span>"<\/span>) <\/span><\/span>\n  .start(<\/span>"<\/span>deltaTable<\/span>"<\/span>)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Running Apache Spark<\/h2>\n\n\n\n
At a fundamental level, an Apache Spark application consists of two main components: a driver<\/em>, which converts the user\u2019s code into multiple tasks that can be distributed across worker nodes, and executors<\/em>, which run on those worker nodes and execute the tasks assigned to them. Some form of cluster manager is necessary to mediate between the two.<\/p>\n\n\n\n
Out of the box, Apache Spark can run in a stand-alone cluster mode that simply requires the Apache Spark framework and a Java Virtual Machine on each node in your cluster. However, it\u2019s more likely you\u2019ll want to take advantage of a more robust resource management or cluster management system to take care of allocating workers on demand for you.<\/p>\n\n\n\n
In the enterprise, this historically meant running on Hadoop YARN<\/a> (YARN is how the Cloudera and Hortonworks distributions run Spark jobs), but as Hadoop has become less entrenched, more and more companies have turned toward deploying Apache Spark on Kubernetes<\/a>. This has been reflected in the Apache Spark 3.x releases, which improve the integration with Kubernetes including the ability to define pod templates for drivers and executors and use custom schedulers such as Volcano<\/a>.<\/p>\n\n\n\n
If you seek a managed solution, then Apache Spark offerings can be found on all of the big three clouds: Amazon EMR<\/a>, Azure HDInsight<\/a>, and Google Cloud Dataproc<\/a>.<\/p>\n\n\n\n
Databricks Lakehouse Platform<\/h2>\n\n\n\n
Databricks<\/a>, the company that employs the creators of Apache Spark, has taken a different approach than many other companies founded on the open source products of the Big Data era. For many years, Databricks has offered a comprehensive managed cloud service that offers Apache Spark clusters, streaming support, integrated web-based notebook development, and proprietary optimized I\/O performance over a standard Apache Spark distribution. This mixture of managed and professional services has turned Databricks into a behemoth in the Big Data arena, with a valuation estimated at $38 billion in 2021. The Databricks Lakehouse Platform is now available on all three major cloud providers and is becoming the de facto<\/em> way that most people interact with Apache Spark.<\/p>\n\n\n\n
Apache Spark tutorials<\/h2>\n\n\n\n
Ready to dive in and learn Apache Spark? We recommend starting with the Databricks learning portal<\/a>, which will provide a good introduction to the framework, although it will be slightly biased towards the Databricks Platform. For diving deeper, we\u2019d suggest the Spark Workshop<\/a>, which is a thorough tour of Apache Spark\u2019s features through a Scala lens. Some excellent books are available too. Spark: The Definitive Guide<\/a> is a wonderful introduction written by two maintainers of Apache Spark. And High Performance Spark<\/a> is an essential guide to processing data with Apache Spark at massive scales in a performant way. Happy learning!<\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction Apache Spark  is a robust framework for processing data, enabling swift execution of tasks on extensive datasets. It excels in distributing data processing across […]<\/p>\n","protected":false},"author":2,"featured_media":24,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[14,19,13],"class_list":["post-22","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-engineering","tag-big-data","tag-data-engineering","tag-spark"],"yoast_head":"\nIntroduction to Apache Spark - Farrukh's Tech Space<\/title>\n<meta name=\"description\" content=\"Apache Spark is a powerful data processing framework that excels in distributing tasks across multiple computers. It's crucial in big data and machine learning due to its computational prowess. Spark also simplifies programming with a user-friendly API. Developed at U.C. Berkeley in 2009, it supports various languages, offers diverse functionalities, and is used across sectors by companies like Apple, IBM, and Microsoft.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introduction to Apache Spark - Farrukh's Tech Space\" \/>\n<meta property=\"og:description\" content=\"Apache Spark is a powerful data processing framework that excels in distributing tasks across multiple computers. It's crucial in big data and machine learning due to its computational prowess. Spark also simplifies programming with a user-friendly API. Developed at U.C. Berkeley in 2009, it supports various languages, offers diverse functionalities, and is used across sectors by companies like Apple, IBM, and Microsoft.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/\" \/>\n<meta property=\"og:site_name\" content=\"Farrukh's Tech Space\" \/>\n<meta property=\"article:published_time\" content=\"2023-06-13T15:50:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-09-22T17:36:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/what-is-apache-spark.b3a3099296936df595d9a7d3610f1a77ff0749df.png\" \/>\n\t<meta property=\"og:image:width\" content=\"779\" \/>\n\t<meta property=\"og:image:height\" content=\"370\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Farrukh Naveed Anjum\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Farrukh Naveed Anjum\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/\"},\"author\":{\"name\":\"Farrukh Naveed Anjum\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/168ccae4fac9c8db0c3d9314f61b018a\"},\"headline\":\"Introduction to Apache Spark\",\"datePublished\":\"2023-06-13T15:50:21+00:00\",\"dateModified\":\"2023-09-22T17:36:11+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/\"},\"wordCount\":1911,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#organization\"},\"keywords\":[\"Big Data\",\"Data Engineering\",\"Spark\"],\"articleSection\":[\"Data Engineering\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/\",\"name\":\"Introduction to Apache Spark - Farrukh's Tech Space\",\"isPartOf\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#website\"},\"datePublished\":\"2023-06-13T15:50:21+00:00\",\"dateModified\":\"2023-09-22T17:36:11+00:00\",\"description\":\"Apache Spark is a powerful data processing framework that excels in distributing tasks across multiple computers. It's crucial in big data and machine learning due to its computational prowess. Spark also simplifies programming with a user-friendly API. Developed at U.C. Berkeley in 2009, it supports various languages, offers diverse functionalities, and is used across sectors by companies like Apple, IBM, and Microsoft.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#website\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/\",\"name\":\"Farrukh Naveed Anjum Blogs\",\"description\":\"Empowering Software Architects with Knowledge on Big Data and AI\",\"publisher\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/farrukhnaveed.co\/blogs\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#organization\",\"name\":\"Farrukh Naveed Anjum Blogs\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg\",\"contentUrl\":\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg\",\"width\":1707,\"height\":2560,\"caption\":\"Farrukh Naveed Anjum Blogs\"},\"image\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/168ccae4fac9c8db0c3d9314f61b018a\",\"name\":\"Farrukh Naveed Anjum\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f5716d3d4b818819b6703f6583acc9888a62c5da7e3ad868085e0afec84c5a21?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f5716d3d4b818819b6703f6583acc9888a62c5da7e3ad868085e0afec84c5a21?s=96&d=mm&r=g\",\"caption\":\"Farrukh Naveed Anjum\"},\"sameAs\":[\"http:\/\/farrukhnaveed.co\"],\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/author\/farrukh\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Introduction to Apache Spark - Farrukh's Tech Space","description":"Apache Spark is a powerful data processing framework that excels in distributing tasks across multiple computers. It's crucial in big data and machine learning due to its computational prowess. Spark also simplifies programming with a user-friendly API. Developed at U.C. Berkeley in 2009, it supports various languages, offers diverse functionalities, and is used across sectors by companies like Apple, IBM, and Microsoft.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/","og_locale":"en_US","og_type":"article","og_title":"Introduction to Apache Spark - Farrukh's Tech Space","og_description":"Apache Spark is a powerful data processing framework that excels in distributing tasks across multiple computers. It's crucial in big data and machine learning due to its computational prowess. Spark also simplifies programming with a user-friendly API. Developed at U.C. Berkeley in 2009, it supports various languages, offers diverse functionalities, and is used across sectors by companies like Apple, IBM, and Microsoft.","og_url":"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/","og_site_name":"Farrukh's Tech Space","article_published_time":"2023-06-13T15:50:21+00:00","article_modified_time":"2023-09-22T17:36:11+00:00","og_image":[{"width":779,"height":370,"url":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/what-is-apache-spark.b3a3099296936df595d9a7d3610f1a77ff0749df.png","type":"image\/png"}],"author":"Farrukh Naveed Anjum","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Farrukh Naveed Anjum","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/#article","isPartOf":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/"},"author":{"name":"Farrukh Naveed Anjum","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/168ccae4fac9c8db0c3d9314f61b018a"},"headline":"Introduction to Apache Spark","datePublished":"2023-06-13T15:50:21+00:00","dateModified":"2023-09-22T17:36:11+00:00","mainEntityOfPage":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/"},"wordCount":1911,"commentCount":0,"publisher":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#organization"},"keywords":["Big Data","Data Engineering","Spark"],"articleSection":["Data Engineering"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/","url":"https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/","name":"Introduction to Apache Spark - Farrukh's Tech Space","isPartOf":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#website"},"datePublished":"2023-06-13T15:50:21+00:00","dateModified":"2023-09-22T17:36:11+00:00","description":"Apache Spark is a powerful data processing framework that excels in distributing tasks across multiple computers. It's crucial in big data and machine learning due to its computational prowess. Spark also simplifies programming with a user-friendly API. Developed at U.C. Berkeley in 2009, it supports various languages, offers diverse functionalities, and is used across sectors by companies like Apple, IBM, and Microsoft.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/farrukhnaveed.co\/blogs\/introduction-to-apache-spark\/"]}]},{"@type":"WebSite","@id":"https:\/\/farrukhnaveed.co\/blogs\/#website","url":"https:\/\/farrukhnaveed.co\/blogs\/","name":"Farrukh Naveed Anjum Blogs","description":"Empowering Software Architects with Knowledge on Big Data and AI","publisher":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/farrukhnaveed.co\/blogs\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/farrukhnaveed.co\/blogs\/#organization","name":"Farrukh Naveed Anjum Blogs","url":"https:\/\/farrukhnaveed.co\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg","contentUrl":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg","width":1707,"height":2560,"caption":"Farrukh Naveed Anjum Blogs"},"image":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/168ccae4fac9c8db0c3d9314f61b018a","name":"Farrukh Naveed Anjum","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/f5716d3d4b818819b6703f6583acc9888a62c5da7e3ad868085e0afec84c5a21?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f5716d3d4b818819b6703f6583acc9888a62c5da7e3ad868085e0afec84c5a21?s=96&d=mm&r=g","caption":"Farrukh Naveed Anjum"},"sameAs":["http:\/\/farrukhnaveed.co"],"url":"https:\/\/farrukhnaveed.co\/blogs\/author\/farrukh\/"}]}},"_links":{"self":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts\/22","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/comments?post=22"}],"version-history":[{"count":7,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts\/22\/revisions"}],"predecessor-version":[{"id":122,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts\/22\/revisions\/122"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/media\/24"}],"wp:attachment":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/media?parent=22"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/categories?post=22"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/tags?post=22"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Pandas API on Spark<\/h2>\n\n\n\n
The industry standard for data manipulation and analysis in Python is the Pandas<\/a> library. With Apache Spark 3.2, a new API was provided that allows a large proportion of the Pandas API to be used transparently with Spark. Now data scientists can simply replace their imports with import pyspark.pandas as pd<\/code> and be somewhat confident that their code will continue to work, and also take advantage of Apache Spark\u2019s multi-node execution. At the moment, around 80% of the Pandas API is covered, with a target of 90% coverage being aimed for in upcoming releases.<\/p>\n\n\n\n
Structured Streaming, by default, uses a micro-batching scheme of handling streaming data. But in Spark 2.3, the Apache Spark team added a low-latency Continuous Processing<\/a> mode to Structured Streaming, allowing it to handle responses with impressive latencies as low as 1ms and making it much more competitive with rivals such as Apache Flink<\/a> and Apache Beam<\/a>. Continuous Processing restricts you to map-like and selection operations, and while it supports SQL queries against streams, it does not currently support SQL aggregations. In addition, although Spark 2.3 arrived in 2018, as of Spark 3.3.2 in March 2023, Continuous Processing is still<\/em> marked as experimental.<\/p>\n\n\n\n
Behind the scenes, Apache Spark uses a query optimizer called Catalyst <\/a>that examines data and queries in order to produce an efficient query plan for data locality and computation that will perform the required calculations across the cluster. Since Apache Spark 2.x, the Spark SQL interface of dataframes and datasets (essentially a typed dataframe that can be checked at compile time for correctness and take advantage of further memory and compute optimizations at run time) has been the recommended approach for development. The RDD interface is still available, but recommended only if your needs cannot be addressed within the Spark SQL paradigm (such as when you must work at a lower level to wring every last drop of performance out of the system).<\/p>\n\n\n\n