{"id":106,"date":"2023-09-20T23:19:51","date_gmt":"2023-09-20T17:49:51","guid":{"rendered":"https:\/\/farrukhnaveed.co\/blogs\/?p=106"},"modified":"2023-09-20T23:19:54","modified_gmt":"2023-09-20T17:49:54","slug":"data-ingestion-from-kafka-to-mysql-using-python","status":"publish","type":"post","link":"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/","title":{"rendered":"Data Ingestion from Kafka to MySQL using Python"},"content":{"rendered":"\n<p>In today&#8217;s data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular choice for building real-time data pipelines due to its distributed and fault-tolerant nature. Meanwhile, MySQL remains a widely used relational database for storing and managing structured data. Combining these two technologies, we can create a robust solution for ingesting data from Kafka and inserting it into MySQL using Python.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Understanding the Components<\/h2>\n\n\n\n<p>Before we dive into the code, let&#8217;s understand the components involved in this process:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Kafka<\/strong>: Apache Kafka is a distributed streaming platform that allows you to publish and subscribe to streams of records, store them in a fault-tolerant manner, and process them in real-time. Kafka uses topics to categorize data streams, and producers write data to these topics, which can be consumed by consumers.<\/li>\n\n\n\n<li><strong>Python<\/strong>: We will use Python, a versatile and widely adopted programming language, to interact with both Kafka and MySQL. The <code>confluent-kafka-python<\/code> library will help us consume data from Kafka, while the <code>mysql-connector-python<\/code> library will allow us to interact with MySQL.<\/li>\n\n\n\n<li><strong>MySQL<\/strong>: MySQL is an open-source relational database management system known for its speed and reliability. We will store the data ingested from Kafka into a MySQL database table.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\">Setting Up the Environment<\/h2>\n\n\n\n<p>To get started, make sure you have Kafka and MySQL installed and running. You&#8217;ll also need Python installed on your system.<\/p>\n\n\n\n<p>Next, install the required Python libraries:<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"pip install confluent-kafka\npip install mysql-connector-python\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #D8DEE9FF\">pip install confluent<\/span><span style=\"color: #81A1C1\">-<\/span><span style=\"color: #D8DEE9FF\">kafka<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">pip install mysql<\/span><span style=\"color: #81A1C1\">-<\/span><span style=\"color: #D8DEE9FF\">connector<\/span><span style=\"color: #81A1C1\">-<\/span><span style=\"color: #D8DEE9FF\">python<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Writing Python Code<\/h2>\n\n\n\n<p>Let&#8217;s create a Python script to ingest data from Kafka and insert it into MySQL. Below is a step-by-step guide:<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"from confluent_kafka import Consumer, KafkaError\nimport mysql.connector\n\n# Kafka consumer configuration\nconf = {\n    'bootstrap.servers': 'localhost:9092',  # Replace with your Kafka broker(s)\n    'group.id': 'my-consumer-group',\n    'auto.offset.reset': 'earliest'  # Start from the beginning of the topic\n}\n\n# Create a Kafka consumer\nconsumer = Consumer(conf)\n\n# Subscribe to a Kafka topic\nconsumer.subscribe(['my-topic'])  # Replace with your Kafka topic name\n\n# MySQL database connection configuration\ndb_config = {\n    'host': 'localhost',  # Replace with your MySQL server hostname\n    'user': 'username',  # Replace with your MySQL username\n    'password': 'password',  # Replace with your MySQL password\n    'database': 'my_database'  # Replace with your database name\n}\n\n# Create a MySQL database connection\ndb_connection = mysql.connector.connect(**db_config)\ncursor = db_connection.cursor()\n\n# Main data ingestion loop\nwhile True:\n    msg = consumer.poll(1.0)  # Poll for Kafka messages\n    if msg is None:\n        continue\n    if msg.error():\n        if msg.error().code() == KafkaError._PARTITION_EOF:\n            print('Reached end of partition')\n        else:\n            print('Error while polling for messages: {}'.format(msg.error()))\n    else:\n        # Insert the Kafka message value into MySQL\n        data = msg.value().decode('utf-8')  # Assuming data is in UTF-8 encoding\n        insert_query = &quot;INSERT INTO my_table (data_column) VALUES (%s)&quot;\n        cursor.execute(insert_query, (data,))\n        db_connection.commit()\n        print('Inserted: {}'.format(data))\n\n# Clean up\ncursor.close()\ndb_connection.close()\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #81A1C1\">from<\/span><span style=\"color: #D8DEE9FF\"> confluent_kafka <\/span><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> Consumer<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> KafkaError<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> mysql<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9FF\">connector<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Kafka consumer configuration<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">conf <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">{<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">bootstrap.servers<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">localhost:9092<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #616E88\"># Replace with your Kafka broker(s)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">group.id<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">my-consumer-group<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">,<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">auto.offset.reset<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">earliest<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #616E88\"># Start from the beginning of the topic<\/span><\/span>\n<span class=\"line\"><span style=\"color: #ECEFF4\">}<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Create a Kafka consumer<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">consumer <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">Consumer<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">conf<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Subscribe to a Kafka topic<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">consumer<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">subscribe<\/span><span style=\"color: #ECEFF4\">([<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">my-topic<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">])<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #616E88\"># Replace with your Kafka topic name<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># MySQL database connection configuration<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">db_config <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">{<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">host<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">localhost<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #616E88\"># Replace with your MySQL server hostname<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">user<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">username<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #616E88\"># Replace with your MySQL username<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">password<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">password<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #616E88\"># Replace with your MySQL password<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">database<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">my_database<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #616E88\"># Replace with your database name<\/span><\/span>\n<span class=\"line\"><span style=\"color: #ECEFF4\">}<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Create a MySQL database connection<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">db_connection <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> mysql<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9FF\">connector<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">connect<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #81A1C1\">**<\/span><span style=\"color: #D8DEE9FF\">db_config<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">cursor <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> db_connection<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">cursor<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Main data ingestion loop<\/span><\/span>\n<span class=\"line\"><span style=\"color: #81A1C1\">while<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">True<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    msg <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> consumer<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">poll<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #B48EAD\">1.0<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #616E88\"># Poll for Kafka messages<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> msg <\/span><span style=\"color: #81A1C1\">is<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">None<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">continue<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> msg<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">error<\/span><span style=\"color: #ECEFF4\">():<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">if<\/span><span style=\"color: #D8DEE9FF\"> msg<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">error<\/span><span style=\"color: #ECEFF4\">().<\/span><span style=\"color: #88C0D0\">code<\/span><span style=\"color: #ECEFF4\">()<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">==<\/span><span style=\"color: #D8DEE9FF\"> KafkaError<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9FF\">_PARTITION_EOF<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">print<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">Reached end of partition<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #81A1C1\">else<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">            <\/span><span style=\"color: #88C0D0\">print<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">Error while polling for messages: <\/span><span style=\"color: #EBCB8B\">{}<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">format<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">msg<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">error<\/span><span style=\"color: #ECEFF4\">()))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">    <\/span><span style=\"color: #81A1C1\">else<\/span><span style=\"color: #ECEFF4\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #616E88\"># Insert the Kafka message value into MySQL<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        data <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> msg<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">value<\/span><span style=\"color: #ECEFF4\">().<\/span><span style=\"color: #88C0D0\">decode<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">utf-8<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">)<\/span><span style=\"color: #D8DEE9FF\">  <\/span><span style=\"color: #616E88\"># Assuming data is in UTF-8 encoding<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        insert_query <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">&quot;<\/span><span style=\"color: #A3BE8C\">INSERT INTO my_table (data_column) VALUES (<\/span><span style=\"color: #EBCB8B\">%s<\/span><span style=\"color: #A3BE8C\">)<\/span><span style=\"color: #ECEFF4\">&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        cursor<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">execute<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">insert_query<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">data<\/span><span style=\"color: #ECEFF4\">,))<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        db_connection<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">commit<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">        <\/span><span style=\"color: #88C0D0\">print<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #A3BE8C\">Inserted: <\/span><span style=\"color: #EBCB8B\">{}<\/span><span style=\"color: #ECEFF4\">&#39;<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">format<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">data<\/span><span style=\"color: #ECEFF4\">))<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #616E88\"># Clean up<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">cursor<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">close<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">db_connection<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #88C0D0\">close<\/span><span style=\"color: #ECEFF4\">()<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><\/div>\n\n\n\n<p>In this script:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We configure the Kafka consumer to connect to your Kafka broker and subscribe to a specific topic.<\/li>\n\n\n\n<li>We set up the MySQL database connection using the provided credentials and specify the database and table where we want to insert the data.<\/li>\n\n\n\n<li>Inside the main loop, we continuously poll for messages from Kafka. When a message is received, we decode it (assuming it&#8217;s in UTF-8 encoding) and insert it into the MySQL database table.<\/li>\n\n\n\n<li>We commit the transaction in the database, ensuring data integrity.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Running the Code<\/h2>\n\n\n\n<p>To run the script, simply execute it using Python:<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"python kafka_to_mysql.py\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #D8DEE9FF\">python kafka_to_mysql<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9FF\">py<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p>Ensure that your Kafka topic exists and has data to consume. As messages are ingested from Kafka, you will see them being inserted into your MySQL database.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>Ingesting data from Kafka and inserting it into MySQL using Python provides a powerful way to build real-time data pipelines for your applications. This approach allows you to process and analyze data in real-time, enabling faster and more informed decision-making. By following the steps and code provided in this article, you can get started on your journey to building robust data pipelines with Kafka and MySQL.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In today&#8217;s data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":107,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12,4],"tags":[14,16,5],"class_list":["post-106","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-engineering","category-python","tag-big-data","tag-mysql","tag-python"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Data Ingestion from Kafka to MySQL using Python - Farrukh&#039;s Tech Space<\/title>\n<meta name=\"description\" content=\"In today&#039;s data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular choice for building real-time data pipelines due to its distributed and fault-tolerant nature. Meanwhile, MySQL remains a widely used relational database for storing and managing structured data. Combining these two technologies, we can create a robust solution for ingesting data from Kafka and inserting it into MySQL using Python.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Data Ingestion from Kafka to MySQL using Python\" \/>\n<meta property=\"og:description\" content=\"In today&#039;s data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular choice for building real-time data pipelines due to its distributed and fault-tolerant nature. Meanwhile, MySQL remains a widely used relational database for storing and managing structured data. Combining these two technologies, we can create a robust solution for ingesting data from Kafka and inserting it into MySQL using Python.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/\" \/>\n<meta property=\"og:site_name\" content=\"Farrukh&#039;s Tech Space\" \/>\n<meta property=\"article:published_time\" content=\"2023-09-20T17:49:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-09-20T17:49:54+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/09\/111-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"912\" \/>\n\t<meta property=\"og:image:height\" content=\"424\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Farrukh Naveed Anjum\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Data Ingestion from Kafka to MySQL using Python\" \/>\n<meta name=\"twitter:description\" content=\"In today&#039;s data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular choice for building real-time data pipelines due to its distributed and fault-tolerant nature. Meanwhile, MySQL remains a widely used relational database for storing and managing structured data. Combining these two technologies, we can create a robust solution for ingesting data from Kafka and inserting it into MySQL using Python.\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/09\/111-1.jpg\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Farrukh Naveed Anjum\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/\"},\"author\":{\"name\":\"Farrukh Naveed Anjum\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/ce7d07e6a917b9b73aa79007a2357d29\"},\"headline\":\"Data Ingestion from Kafka to MySQL using Python\",\"datePublished\":\"2023-09-20T17:49:51+00:00\",\"dateModified\":\"2023-09-20T17:49:54+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/\"},\"wordCount\":466,\"publisher\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#organization\"},\"keywords\":[\"Big Data\",\"MySQL\",\"Python\"],\"articleSection\":[\"Data Engineering\",\"Python\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/\",\"name\":\"Data Ingestion from Kafka to MySQL using Python - Farrukh&#039;s Tech Space\",\"isPartOf\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#website\"},\"datePublished\":\"2023-09-20T17:49:51+00:00\",\"dateModified\":\"2023-09-20T17:49:54+00:00\",\"description\":\"In today's data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular choice for building real-time data pipelines due to its distributed and fault-tolerant nature. Meanwhile, MySQL remains a widely used relational database for storing and managing structured data. Combining these two technologies, we can create a robust solution for ingesting data from Kafka and inserting it into MySQL using Python.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#website\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/\",\"name\":\"Farrukh Naveed Anjum Blogs\",\"description\":\"Empowering Software Architects with Knowledge on Big Data and AI\",\"publisher\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/farrukhnaveed.co\/blogs\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#organization\",\"name\":\"Farrukh Naveed Anjum Blogs\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg\",\"contentUrl\":\"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg\",\"width\":1707,\"height\":2560,\"caption\":\"Farrukh Naveed Anjum Blogs\"},\"image\":{\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/ce7d07e6a917b9b73aa79007a2357d29\",\"name\":\"Farrukh Naveed Anjum\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/bdf1af0d569259df562434e6dc99415a377c6fc053f9e1507aa34a6522561bb8?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/bdf1af0d569259df562434e6dc99415a377c6fc053f9e1507aa34a6522561bb8?s=96&d=mm&r=g\",\"caption\":\"Farrukh Naveed Anjum\"},\"description\":\"Full Stack Developer and Software Architect with 14 years of experience in various domains, including Enterprise Resource Planning, Data Retrieval, Web Scraping, Real-Time Analytics, Cybersecurity, NLP, ED-Tech, and B2B Price Comparison\",\"sameAs\":[\"https:\/\/farrukhnaveed.co\/blog\"],\"url\":\"https:\/\/farrukhnaveed.co\/blogs\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Data Ingestion from Kafka to MySQL using Python - Farrukh&#039;s Tech Space","description":"In today's data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular choice for building real-time data pipelines due to its distributed and fault-tolerant nature. Meanwhile, MySQL remains a widely used relational database for storing and managing structured data. Combining these two technologies, we can create a robust solution for ingesting data from Kafka and inserting it into MySQL using Python.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/","og_locale":"en_US","og_type":"article","og_title":"Data Ingestion from Kafka to MySQL using Python","og_description":"In today's data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular choice for building real-time data pipelines due to its distributed and fault-tolerant nature. Meanwhile, MySQL remains a widely used relational database for storing and managing structured data. Combining these two technologies, we can create a robust solution for ingesting data from Kafka and inserting it into MySQL using Python.","og_url":"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/","og_site_name":"Farrukh&#039;s Tech Space","article_published_time":"2023-09-20T17:49:51+00:00","article_modified_time":"2023-09-20T17:49:54+00:00","og_image":[{"width":912,"height":424,"url":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/09\/111-1.jpg","type":"image\/jpeg"}],"author":"Farrukh Naveed Anjum","twitter_card":"summary_large_image","twitter_title":"Data Ingestion from Kafka to MySQL using Python","twitter_description":"In today's data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular choice for building real-time data pipelines due to its distributed and fault-tolerant nature. Meanwhile, MySQL remains a widely used relational database for storing and managing structured data. Combining these two technologies, we can create a robust solution for ingesting data from Kafka and inserting it into MySQL using Python.","twitter_image":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/09\/111-1.jpg","twitter_misc":{"Written by":"Farrukh Naveed Anjum","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/#article","isPartOf":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/"},"author":{"name":"Farrukh Naveed Anjum","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/ce7d07e6a917b9b73aa79007a2357d29"},"headline":"Data Ingestion from Kafka to MySQL using Python","datePublished":"2023-09-20T17:49:51+00:00","dateModified":"2023-09-20T17:49:54+00:00","mainEntityOfPage":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/"},"wordCount":466,"publisher":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#organization"},"keywords":["Big Data","MySQL","Python"],"articleSection":["Data Engineering","Python"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/","url":"https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/","name":"Data Ingestion from Kafka to MySQL using Python - Farrukh&#039;s Tech Space","isPartOf":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#website"},"datePublished":"2023-09-20T17:49:51+00:00","dateModified":"2023-09-20T17:49:54+00:00","description":"In today's data-driven world, real-time data processing is crucial for businesses to make informed decisions and stay competitive. Apache Kafka has emerged as a popular choice for building real-time data pipelines due to its distributed and fault-tolerant nature. Meanwhile, MySQL remains a widely used relational database for storing and managing structured data. Combining these two technologies, we can create a robust solution for ingesting data from Kafka and inserting it into MySQL using Python.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/farrukhnaveed.co\/blogs\/data-ingestion-from-kafka-to-mysql-using-python\/"]}]},{"@type":"WebSite","@id":"https:\/\/farrukhnaveed.co\/blogs\/#website","url":"https:\/\/farrukhnaveed.co\/blogs\/","name":"Farrukh Naveed Anjum Blogs","description":"Empowering Software Architects with Knowledge on Big Data and AI","publisher":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/farrukhnaveed.co\/blogs\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/farrukhnaveed.co\/blogs\/#organization","name":"Farrukh Naveed Anjum Blogs","url":"https:\/\/farrukhnaveed.co\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg","contentUrl":"https:\/\/farrukhnaveed.co\/blogs\/wp-content\/uploads\/2023\/06\/IMG_5018-scaled.jpg","width":1707,"height":2560,"caption":"Farrukh Naveed Anjum Blogs"},"image":{"@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/ce7d07e6a917b9b73aa79007a2357d29","name":"Farrukh Naveed Anjum","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/farrukhnaveed.co\/blogs\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/bdf1af0d569259df562434e6dc99415a377c6fc053f9e1507aa34a6522561bb8?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/bdf1af0d569259df562434e6dc99415a377c6fc053f9e1507aa34a6522561bb8?s=96&d=mm&r=g","caption":"Farrukh Naveed Anjum"},"description":"Full Stack Developer and Software Architect with 14 years of experience in various domains, including Enterprise Resource Planning, Data Retrieval, Web Scraping, Real-Time Analytics, Cybersecurity, NLP, ED-Tech, and B2B Price Comparison","sameAs":["https:\/\/farrukhnaveed.co\/blog"],"url":"https:\/\/farrukhnaveed.co\/blogs\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts\/106","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/comments?post=106"}],"version-history":[{"count":1,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts\/106\/revisions"}],"predecessor-version":[{"id":108,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/posts\/106\/revisions\/108"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/media\/107"}],"wp:attachment":[{"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/media?parent=106"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/categories?post=106"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/farrukhnaveed.co\/blogs\/wp-json\/wp\/v2\/tags?post=106"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}