Apache Iceberg

🚀 **New Blog Post Alert!** 🚀 Incremental Processing with Apache Iceberg & Spark In this post, I dive deep into how Apache Iceberg enhances the Medallion Architecture (you know, that Bronze, Silver, and Gold layers we all love), optimizing incremental data processing like never before. 🔑 Key takeaways: - Why incremental processing is critical for efficiency, timeliness, and scalability. - Real-world examples of transforming data between Bronze and Silver layers with Iceberg. - How Iceberg's versioning and metadata management make incremental loads faster and cost-efficient. - Best practices for controlling batch sizes and avoiding those dreaded performance bottlenecks. If you're working with big data and thinking of moving to apache iceberg, this is a must-read. Check it out 👉 https://lnkd.in/dBhCYsF3 #ApacheIceberg #BigData #MedallionArchitecture #Spark #DataEngineering #IncrementalProcessing #ETL,Apache Iceberg

Who’s ready to learn how to build a production-grade #ApacheIceberg lakehouse? 🙋♂️🙋♀️ We’re excited to offer hands-on Iceberg training led by Roy Hasson, VP of Product for Upsolver, at our #ChillDataSummit next week! With Roy’s deep expertise in data and his approachable teaching style, this is a rare chance to learn from an industry pro who welcomes every question and shares his knowledge generously! 🙌 🎫 Chill Data Summit on Tour, London 📅 Tuesday 17th September 2024 🕘 9 AM - 3.30 PM (BST) 📍 Shoreditch Studios, 37 Bateman's Row, London EC2A 3HH This is an amazing opportunity to level up your skills and get ahead of the game with #ApacheIceberg — don't miss out 👉 https://lnkd.in/eKzNgtVw #DataEngineering #Training #Events

👋🏽 Check out our latest white paper - "Introducing Apache Iceberg: The Case for an Open Data Lakehouse Powered by Cloudera". This paper covers how data teams can build and implement an open data lakehouse architecture across cloud and on-premises environments to satisfy virtually any analytic workload. 💡Did you know? At Cloudera, we are proud of our open-source roots and committed to enriching the community. Since 2021, we have contributed to the growing Iceberg community with hundreds of contributions across Impala, Hive, Spark, and Iceberg. In early 2022, we enabled a Technical Preview of Apache Iceberg in Cloudera Data Platform allowing Cloudera customers to realize the value of Iceberg’s schema evolution and time travel capabilities in our Data Warehousing, Data Engineering and Machine Learning services. We've come a long way, baby! 🙌🏼 The full white paper can be accessed here 👉 https://bit.ly/CLDRIceberg #cloudera #apacheiceberg #iceberg #datawarehousing #dataengineering #ML #machinelearning #hive #impala #whitepaper #datalakehouse #opensource #analytics #cloud #hybridcloud

This tutorial walks you through working with an Iceberg table saved to MinIO using Dremio. The open source community around Apache Iceberg is active, vibrant and robust. New functionality and integrations are being added constantly. Iceberg’s rapid adoption means that there are plenty of compatible application frameworks and learning resources available. MinIO is built to power data lakes and the analytics and AI that runs on top of them. Read this ebook for the full details (60 Pages).

Interesting research paper on how Apple has implemented Apache Iceberg on top of their Peta-byte scale Lake House and how the implemented some optimizations to improve the performance further like storage partition joins and lazy materializations. Remember that they are also pushing for DataFusion Comet for having a DataFusion based native executor for Spark. #analytics #iceberg https://lnkd.in/dkeSU9Q7

🚀 Exciting News! 🚀 After my last post somebody pointed out that my talk on **"Best Practices and Insights When Migrating to Apache Iceberg for Data Engineers"** is now available on YouTube! 🎥 In this session, I dive deep into the details of migrating to Apache Iceberg, sharing valuable tips and insights to help data engineers navigate the transition smoothly. **Spoiler Alert**: It's definitely worth watching until the end, where I share some **benchmark results** that you won't want to miss! 📊🔥 Check it out and let me know what you think! Your feedback and thoughts would be greatly appreciated. If you are thinking of migrating or already working with apache iceberg and want to talk , feel free to send me a message 🔗 https://lnkd.in/eJCeE6_t #DataEngineering #ApacheIceberg #Migration #TechTalk #Benchmarking #BestPractices #DataAnalytics #YouTube #Startburst #Trinofest

Hey, Data Engineers 🤓 Welcome to our new mini-series exploring the best practices for migrating from #Hive to #ApacheIceberg tables. In Part 1, we’ll explore the reasons why you should migrate from Hive to Iceberg. Let’s dive in! So what is Apache Iceberg? 🤷♂️ Apache Iceberg is an open table format that simplifies managing and optimizing large analytics tables on object stores such as Amazon S3. Unlike Hive, which requires extra coding for safe operations, Iceberg offers an elevated experience with features including: 💠 ACID transactions for safe data operations and the ability to insert, update, and delete 💠 Dynamic partitions that eliminate the need for static definitions 💠 Schema evolution to handle changes in data type and column names 💠 Pluggable storage formats like Parquet, Avro, Orc, and the up-and-coming Puffin 💠 Built-in storage optimizations for better performance Major cloud and data platforms like AWS, Snowflake, and Databricks are rapidly adopting Iceberg, driving the future of interoperable, open lakehouse architectures 🌐 Watch the accompanying video here 👉 https://lnkd.in/ebJshCi4 Join us for Day 2 tomorrow when we’ll be looking at the challenges of Hive-based data lakes. #DataEngineering #DataMigration #CloudArchitecture