{"id":92,"date":"2023-09-13T23:09:02","date_gmt":"2023-09-13T17:39:02","guid":{"rendered":"https:\/\/farrukhnaveed.co\/blogs\/?p=92"},"modified":"2023-09-13T23:18:44","modified_gmt":"2023-09-13T17:48:44","slug":"getting-started-with-pyspark","status":"publish","type":"post","link":"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/","title":{"rendered":"Getting Started with PySpark"},"content":{"rendered":"\n

Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you’re a data enthusiast or a seasoned developer looking to dive into the world of big data processing, this guide will help you get started with PySpark.<\/p>\n\n\n\n

What is PySpark?<\/h2>\n\n\n\n

PySpark is the Python library for Apache Spark, an open-source, distributed computing system that excels in processing vast amounts of data quickly and efficiently. It provides APIs in several languages, including Python, making it accessible to a wide range of developers.<\/p>\n\n\n\n

Installation<\/h2>\n\n\n\n

Before we dive into PySpark, you need to have Apache Spark installed on your machine. Follow these simple steps:<\/p>\n\n\n\n

    \n
  1. Install Java<\/strong>: PySpark requires Java, so make sure you have it installed on your system.<\/li>\n\n\n\n
  2. Download Apache Spark<\/strong>: Visit the official Apache Spark website (https:\/\/spark.apache.org\/downloads.html<\/a>) and download the latest version. Extract the downloaded file to your preferred location.<\/li>\n\n\n\n
  3. Set Environment Variables<\/strong>: Set the SPARK_HOME<\/code> and HADOOP_HOME<\/code> environment variables to point to the Spark and Hadoop directories, respectively.<\/li>\n\n\n\n
  4. Install PySpark<\/strong>: You can install PySpark using pip:<\/li>\n<\/ol>\n\n\n\n
    <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
    pip<\/span> <\/span>install<\/span> <\/span>pyspark<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
    Your First PySpark Code<\/h2>\n\n\n\n
    Now that you have PySpark installed, let’s write a simple PySpark script to get a taste of its capabilities. Open your Python editor or Jupyter Notebook and follow along.<\/p>\n\n\n\n
    <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
    # Import the necessary libraries<\/span><\/span>\nfrom<\/span> pyspark<\/span>.<\/span>sql <\/span>import<\/span> SparkSession<\/span><\/span>\n<\/span>\n# Create a SparkSession<\/span><\/span>\nspark <\/span>=<\/span> SparkSession<\/span>.<\/span>builder<\/span>.<\/span>appName<\/span>(<\/span>"<\/span>MyFirstPySparkApp<\/span>"<\/span>).<\/span>getOrCreate<\/span>()<\/span><\/span>\n<\/span>\n# Create a DataFrame<\/span><\/span>\ndata <\/span>=<\/span> <\/span>[(<\/span>"<\/span>Alice<\/span>"<\/span>,<\/span> <\/span>25<\/span>),<\/span> <\/span>(<\/span>"<\/span>Bob<\/span>"<\/span>,<\/span> <\/span>30<\/span>),<\/span> <\/span>(<\/span>"<\/span>Charlie<\/span>"<\/span>,<\/span> <\/span>35<\/span>)]<\/span><\/span>\ncolumns <\/span>=<\/span> <\/span>[<\/span>"<\/span>Name<\/span>"<\/span>,<\/span> <\/span>"<\/span>Age<\/span>"<\/span>]<\/span><\/span>\ndf <\/span>=<\/span> spark<\/span>.<\/span>createDataFrame<\/span>(<\/span>data<\/span>,<\/span> columns<\/span>)<\/span><\/span>\n<\/span>\n# Show the DataFrame<\/span><\/span>\ndf<\/span>.<\/span>show<\/span>()<\/span><\/span>\n<\/span>\n# Stop the SparkSession<\/span><\/span>\nspark<\/span>.<\/span>stop<\/span>()<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
    In this code:<\/p>\n\n\n\n
      \n
    • We import the SparkSession<\/code> class, which is the entry point for using PySpark.<\/li>\n\n\n\n
    • We create a SparkSession named “MyFirstPySparkApp.”<\/li>\n\n\n\n
    • We create a simple DataFrame containing name and age data.<\/li>\n\n\n\n
    • We display the DataFrame using the show()<\/code> method.<\/li>\n\n\n\n
    • Finally, we stop the SparkSession to release resources.<\/li>\n<\/ul>\n\n\n\n
      Running Your PySpark Code<\/h2>\n\n\n\n
      To run this code, save it in a .py<\/code> file and execute it using Python. Ensure that your Spark installation is correctly configured, and you have Java installed. <\/p>\n\n\n\n
      Following will be the output<\/p>\n\n\n\n
      <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
      +-------+---+<\/span><\/span>\n|<\/span>   <\/span>Name<\/span>|<\/span>Age<\/span>|<\/span><\/span>\n+-------+---+<\/span><\/span>\n|<\/span>  <\/span>Alice<\/span>|<\/span> <\/span>25<\/span>|<\/span><\/span>\n|<\/span>    <\/span>Bob<\/span>|<\/span> <\/span>30<\/span>|<\/span><\/span>\n|<\/span>Charlie<\/span>|<\/span> <\/span>35<\/span>|<\/span><\/span>\n+-------+---+<\/span><\/span>\n<\/span><\/code><\/pre><\/div>\n\n\n\n
      <\/p>\n\n\n\n
      To read data from a CSV file and apply aggregation on a PySpark DataFrame, you can follow these steps:<\/p>\n\n\n\n
      Step 1: Import the necessary libraries<\/h3>\n\n\n\n
      Start by importing the required libraries, including SparkSession<\/code> for initializing PySpark and functions<\/code> for aggregation functions like sum<\/code>, avg<\/code>, etc.<\/p>\n\n\n\n
      <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
      from<\/span> pyspark<\/span>.<\/span>sql <\/span>import<\/span> SparkSession<\/span><\/span>\nfrom<\/span> pyspark<\/span>.<\/span>sql <\/span>import<\/span> functions <\/span>as<\/span> F<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
      Step 2: Initialize a SparkSession<\/h3>\n\n\n\n
      Create a SparkSession<\/code>, which serves as the entry point to PySpark.<\/p>\n\n\n\n
      <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
      spark <\/span>=<\/span> SparkSession<\/span>.<\/span>builder<\/span>.<\/span>appName<\/span>(<\/span>"<\/span>CSVAggregation<\/span>"<\/span>).<\/span>getOrCreate<\/span>()<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
      Step 3: Read the CSV file into a DataFrame<\/h3>\n\n\n\n
      Use the read<\/code> method to read the CSV file into a PySpark DataFrame. Replace \"your_file.csv\"<\/code> with the path to your CSV file.<\/p>\n\n\n\n
      <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
      df <\/span>=<\/span> spark<\/span>.<\/span>read<\/span>.<\/span>csv<\/span>(<\/span>"<\/span>your_file.csv<\/span>"<\/span>,<\/span> <\/span>header<\/span>=True<\/span>,<\/span> <\/span>inferSchema<\/span>=True<\/span>)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
        \n
      • header=True<\/code> specifies that the first row in the CSV file contains column headers.<\/li>\n\n\n\n
      • inferSchema=True<\/code> attempts to infer the data types of the columns automatically.<\/li>\n<\/ul>\n\n\n\n
        Step 4: Apply Aggregation<\/h3>\n\n\n\n
        Now, you can apply aggregation functions to the DataFrame. Here’s an example of how to calculate the sum and average of a numerical column, such as “Amount”:<\/p>\n\n\n\n
        <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
        # Calculate the sum and average of the "Amount" column<\/span><\/span>\nsum_amount <\/span>=<\/span> df<\/span>.<\/span>agg<\/span>(<\/span>F<\/span>.<\/span>sum<\/span>(<\/span>"<\/span>Amount<\/span>"<\/span>).<\/span>alias<\/span>(<\/span>"<\/span>TotalAmount<\/span>"<\/span>))<\/span><\/span>\navg_amount <\/span>=<\/span> df<\/span>.<\/span>agg<\/span>(<\/span>F<\/span>.<\/span>avg<\/span>(<\/span>"<\/span>Amount<\/span>"<\/span>).<\/span>alias<\/span>(<\/span>"<\/span>AverageAmount<\/span>"<\/span>))<\/span><\/span>\n<\/span>\n# Show the results<\/span><\/span>\nsum_amount<\/span>.<\/span>show<\/span>()<\/span><\/span>\navg_amount<\/span>.<\/span>show<\/span>()<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
        In this code, we use agg()<\/code> to apply aggregation functions to the DataFrame. We use the alias()<\/code> method to provide meaningful names to the resulting columns.<\/p>\n\n\n\n
        Step 5: Display the Results<\/h3>\n\n\n\n
        Finally, display the results of the aggregation:<\/p>\n\n\n\n
        <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
        # Show the results<\/span><\/span>\nsum_amount<\/span>.<\/span>show<\/span>()<\/span><\/span>\navg_amount<\/span>.<\/span>show<\/span>()<\/span><\/span>\n<\/span><\/code><\/pre><\/div>\n\n\n\n
        Step 6: Stop the SparkSession<\/h3>\n\n\n\n
        Don’t forget to stop the SparkSession to release resources when you’re done:<\/p>\n\n\n\n
        <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
        spark<\/span>.<\/span>stop<\/span>()<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
        Here’s the complete code:<\/p>\n\n\n\n
        <\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
        from<\/span> pyspark<\/span>.<\/span>sql <\/span>import<\/span> SparkSession<\/span><\/span>\nfrom<\/span> pyspark<\/span>.<\/span>sql <\/span>import<\/span> functions <\/span>as<\/span> F<\/span><\/span>\n<\/span>\n# Initialize a SparkSession<\/span><\/span>\nspark <\/span>=<\/span> SparkSession<\/span>.<\/span>builder<\/span>.<\/span>appName<\/span>(<\/span>"<\/span>CSVAggregation<\/span>"<\/span>).<\/span>getOrCreate<\/span>()<\/span><\/span>\n<\/span>\n# Read the CSV file into a DataFrame<\/span><\/span>\ndf <\/span>=<\/span> spark<\/span>.<\/span>read<\/span>.<\/span>csv<\/span>(<\/span>"<\/span>your_file.csv<\/span>"<\/span>,<\/span> <\/span>header<\/span>=True<\/span>,<\/span> <\/span>inferSchema<\/span>=True<\/span>)<\/span><\/span>\n<\/span>\n# Calculate the sum and average of the "Amount" column<\/span><\/span>\nsum_amount <\/span>=<\/span> df<\/span>.<\/span>agg<\/span>(<\/span>F<\/span>.<\/span>sum<\/span>(<\/span>"<\/span>Amount<\/span>"<\/span>).<\/span>alias<\/span>(<\/span>"<\/span>TotalAmount<\/span>"<\/span>))<\/span><\/span>\navg_amount <\/span>=<\/span> df<\/span>.<\/span>agg<\/span>(<\/span>F<\/span>.<\/span>avg<\/span>(<\/span>"<\/span>Amount<\/span>"<\/span>).<\/span>alias<\/span>(<\/span>"<\/span>AverageAmount<\/span>"<\/span>))<\/span><\/span>\n<\/span>\n# Show the results<\/span><\/span>\nsum_amount<\/span>.<\/span>show<\/span>()<\/span><\/span>\navg_amount<\/span>.<\/span>show<\/span>()<\/span><\/span>\n<\/span>\n# Stop the SparkSession<\/span><\/span>\nspark<\/span>.<\/span>stop<\/span>()<\/span><\/span>\n<\/span><\/code><\/pre><\/div>\n\n\n\n
        <\/p>\n","protected":false},"excerpt":{"rendered":"
        Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you’re a […]<\/p>\n","protected":false},"author":1,"featured_media":98,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[14,5,13],"class_list":["post-92","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-engineering","tag-big-data","tag-python","tag-spark"],"yoast_head":"\nGetting Started with PySpark - Farrukh's Tech Space<\/title>\n<meta name=\"description\" content=\"Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you're a data enthusiast or a seasoned developer looking to dive into the world of big data processing, this guide will help you get started with PySpark.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Getting Started with PySpark\" \/>\n<meta property=\"og:description\" content=\"Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you're a data enthusiast or a seasoned developer looking to dive into the world of big data processing, this guide will help you get started with PySpark.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/\" \/>\n<meta property=\"og:site_name\" content=\"Farrukh's Tech Space\" \/>\n<meta property=\"article:published_time\" content=\"2023-09-13T17:39:02+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-09-13T17:48:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/09\/gg.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2240\" \/>\n\t<meta property=\"og:image:height\" content=\"1260\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Farrukh Naveed Anjum\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Getting Started with PySpark\" \/>\n<meta name=\"twitter:description\" content=\"Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you're a data enthusiast or a seasoned developer looking to dive into the world of big data processing, this guide will help you get started with PySpark.\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/09\/gg.png\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Farrukh Naveed Anjum\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/\"},\"author\":{\"name\":\"Farrukh Naveed Anjum\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/ce7d07e6a917b9b73aa79007a2357d29\"},\"headline\":\"Getting Started with PySpark\",\"datePublished\":\"2023-09-13T17:39:02+00:00\",\"dateModified\":\"2023-09-13T17:48:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/\"},\"wordCount\":496,\"publisher\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#organization\"},\"keywords\":[\"Big Data\",\"Python\",\"Spark\"],\"articleSection\":[\"Data Engineering\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/\",\"name\":\"Getting Started with PySpark - Farrukh's Tech Space\",\"isPartOf\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#website\"},\"datePublished\":\"2023-09-13T17:39:02+00:00\",\"dateModified\":\"2023-09-13T17:48:44+00:00\",\"description\":\"Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you're a data enthusiast or a seasoned developer looking to dive into the world of big data processing, this guide will help you get started with PySpark.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#website\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/\",\"name\":\"Farrukh Naveed Anjum Blogs\",\"description\":\"Empowering Software Architects with Knowledge on Big Data and AI\",\"publisher\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/farrukhnaveed.co\/blogs\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#organization\",\"name\":\"Farrukh Naveed Anjum Blogs\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg\",\"contentUrl\":\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg\",\"width\":1707,\"height\":2560,\"caption\":\"Farrukh Naveed Anjum Blogs\"},\"image\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/ce7d07e6a917b9b73aa79007a2357d29\",\"name\":\"Farrukh Naveed Anjum\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/bdf1af0d569259df562434e6dc99415a377c6fc053f9e1507aa34a6522561bb8?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/bdf1af0d569259df562434e6dc99415a377c6fc053f9e1507aa34a6522561bb8?s=96&d=mm&r=g\",\"caption\":\"Farrukh Naveed Anjum\"},\"description\":\"Full Stack Developer and Software Architect with 14 years of experience in various domains, including Enterprise Resource Planning, Data Retrieval, Web Scraping, Real-Time Analytics, Cybersecurity, NLP, ED-Tech, and B2B Price Comparison\",\"sameAs\":[\"https:\/\/farrukhnaveed.co\/blog\"],\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Getting Started with PySpark - Farrukh's Tech Space","description":"Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you're a data enthusiast or a seasoned developer looking to dive into the world of big data processing, this guide will help you get started with PySpark.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/","og_locale":"en_US","og_type":"article","og_title":"Getting Started with PySpark","og_description":"Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you're a data enthusiast or a seasoned developer looking to dive into the world of big data processing, this guide will help you get started with PySpark.","og_url":"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/","og_site_name":"Farrukh's Tech Space","article_published_time":"2023-09-13T17:39:02+00:00","article_modified_time":"2023-09-13T17:48:44+00:00","og_image":[{"width":2240,"height":1260,"url":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/09\/gg.png","type":"image\/png"}],"author":"Farrukh Naveed Anjum","twitter_card":"summary_large_image","twitter_title":"Getting Started with PySpark","twitter_description":"Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you're a data enthusiast or a seasoned developer looking to dive into the world of big data processing, this guide will help you get started with PySpark.","twitter_image":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/09\/gg.png","twitter_misc":{"Written by":"Farrukh Naveed Anjum","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/#article","isPartOf":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/"},"author":{"name":"Farrukh Naveed Anjum","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/ce7d07e6a917b9b73aa79007a2357d29"},"headline":"Getting Started with PySpark","datePublished":"2023-09-13T17:39:02+00:00","dateModified":"2023-09-13T17:48:44+00:00","mainEntityOfPage":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/"},"wordCount":496,"publisher":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#organization"},"keywords":["Big Data","Python","Spark"],"articleSection":["Data Engineering"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/","url":"https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/","name":"Getting Started with PySpark - Farrukh's Tech Space","isPartOf":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#website"},"datePublished":"2023-09-13T17:39:02+00:00","dateModified":"2023-09-13T17:48:44+00:00","description":"Python is an immensely popular programming language, and when it comes to processing large datasets, PySpark stands out as a powerful tool. Whether you're a data enthusiast or a seasoned developer looking to dive into the world of big data processing, this guide will help you get started with PySpark.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/farrukhnaveed.co\/blogs\/getting-started-with-pyspark\/"]}]},{"@type":"WebSite","@id":"https:\/\/farrukhnaveed.co\/blogs\/#website","url":"https:\/\/farrukhnaveed.co\/blogs\/","name":"Farrukh Naveed Anjum Blogs","description":"Empowering Software Architects with Knowledge on Big Data and AI","publisher":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/farrukhnaveed.co\/blogs\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/farrukhnaveed.co\/blogs\/#organization","name":"Farrukh Naveed Anjum Blogs","url":"https:\/\/farrukhnaveed.co\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg","contentUrl":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg","width":1707,"height":2560,"caption":"Farrukh Naveed Anjum Blogs"},"image":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/ce7d07e6a917b9b73aa79007a2357d29","name":"Farrukh Naveed Anjum","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/bdf1af0d569259df562434e6dc99415a377c6fc053f9e1507aa34a6522561bb8?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/bdf1af0d569259df562434e6dc99415a377c6fc053f9e1507aa34a6522561bb8?s=96&d=mm&r=g","caption":"Farrukh Naveed Anjum"},"description":"Full Stack Developer and Software Architect with 14 years of experience in various domains, including Enterprise Resource Planning, Data Retrieval, Web Scraping, Real-Time Analytics, Cybersecurity, NLP, ED-Tech, and B2B Price Comparison","sameAs":["https:\/\/farrukhnaveed.co\/blog"],"url":"https:\/\/farrukhnaveed.co\/blogs\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts\/92","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/comments?post=92"}],"version-history":[{"count":4,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts\/92\/revisions"}],"predecessor-version":[{"id":100,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts\/92\/revisions\/100"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/media\/98"}],"wp:attachment":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/media?parent=92"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/categories?post=92"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/tags?post=92"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}