Saturday, June 22, 2024
HomeBig DataHow Cloudinary remodeled their petabyte scale streaming knowledge lake with Apache Iceberg...

How Cloudinary remodeled their petabyte scale streaming knowledge lake with Apache Iceberg and AWS Analytics

This publish is co-written with Amit Gilad, Alex Dickman and Itay Takersman from Cloudinary. 

Enterprises and organizations throughout the globe wish to harness the ability of knowledge to make higher choices by placing knowledge on the middle of each decision-making course of. Information-driven choices result in simpler responses to sudden occasions, improve innovation and permit organizations to create higher experiences for his or her prospects. Nonetheless, all through historical past, knowledge companies have held dominion over their prospects’ knowledge. Regardless of the potential separation of storage and compute when it comes to structure, they’re usually successfully fused collectively. This amalgamation empowers distributors with authority over a various vary of workloads by advantage of proudly owning the information. This authority extends throughout realms similar to enterprise intelligence, knowledge engineering, and machine studying thus limiting the instruments and capabilities that can be utilized.

The panorama of knowledge expertise is swiftly advancing, pushed steadily by initiatives led by the open supply group usually and the Apache basis particularly. This evolving open supply panorama permits prospects full management over knowledge storage, processing engines and permissions increasing the array of accessible choices considerably. This strategy additionally encourages distributors to compete based mostly on the worth they supply to companies, fairly than counting on potential fusing of storage and compute. This fosters a aggressive atmosphere that prioritizes buyer acquisition and prompts distributors to distinguish themselves by means of distinctive options and choices that cater on to the particular wants and preferences of their clientele.

A contemporary knowledge technique redefines and allows sharing knowledge throughout the enterprise and permits for each studying and writing of a singular occasion of the information utilizing an open desk format. The open desk format accelerates corporations’ adoption of a contemporary knowledge technique as a result of it permits them to make use of numerous instruments on prime of a single copy of the information.

Cloudinary is a cloud-based media administration platform that gives a complete set of instruments and companies for managing, optimizing, and delivering photos, movies, and different media belongings on web sites and cell purposes. It’s broadly utilized by builders, content material creators, and companies to streamline their media workflows, improve consumer experiences, and optimize content material supply.

On this weblog publish, we dive into completely different knowledge elements and the way Cloudinary breaks the 2 considerations of vendor locking and price environment friendly knowledge analytics by utilizing Apache Iceberg, Amazon Easy Storage Service (Amazon S3), Amazon Athena, Amazon EMR, and AWS Glue.

Quick overview of Cloudinary’s infrastructure

Cloudinary infrastructure handles over 20 billion requests every day with each request producing occasion logs. Varied knowledge pipelines course of these logs, storing petabytes (PBs) of knowledge per thirty days, which after processing knowledge saved on Amazon S3, are then saved in Snowflake Information Cloud. These datasets function a crucial useful resource for Cloudinary inside groups and knowledge science teams to permit detailed analytics and superior use instances.

Till not too long ago, this knowledge was principally ready by automated processes and aggregated into outcomes tables, utilized by only some inside groups. Cloudinary struggled to make use of this knowledge for added groups who had extra on-line, actual time, lower-granularity, dynamic utilization necessities. Making petabytes of knowledge accessible for ad-hoc stories turned a problem as question time elevated and prices skyrocketed together with rising compute useful resource necessities. Cloudinary knowledge retention for the particular analytical knowledge mentioned on this publish was outlined as 30 days. Nonetheless, new use instances drove the necessity for elevated retention, which might have led to considerably larger price.

The information is flowing from Cloudinary log suppliers into information written into Amazon S3 and notified by means of occasions pushed to Amazon Easy Queue Service (Amazon SQS). These SQS occasions are ingested by a Spark software working in Amazon EMR Spark, which parses and enriches the information. The processed logs are written in Apache Parquet format again to Amazon S3 after which routinely loaded to a Snowflake desk utilizing Snowpipe.

Why Cloudinary selected Apache Iceberg

Apache Iceberg is a high-performance desk format for big analytic workloads. Apache Iceberg brings the reliability and ease of SQL tables to large knowledge, whereas making it doable for processing engines similar to Apache Spark, Trino, Apache Flink, Presto, Apache Hive, and Impala to soundly work with the identical tables on the identical time.

An answer based mostly on Apache Iceberg encompasses full knowledge administration, that includes easy built-in desk optimization capabilities inside an present storage resolution. These capabilities, together with the flexibility to make use of a number of engines on prime of a singular occasion of knowledge, helps keep away from the necessity for knowledge motion between numerous options.

Whereas exploring the assorted controls and choices in configuring Apache Iceberg, Cloudinary needed to adapt its knowledge to make use of AWS Glue Information Catalog, in addition to transfer a big quantity of knowledge to Apache Iceberg on Amazon S3. At this level it turned clear that prices can be considerably decreased, and whereas it had been a key issue for the reason that planning part, it was now doable to get concrete numbers. One instance is that Cloudinary was now in a position to retailer 6 months of knowledge for a similar storage value that was beforehand paid for storing 1 month of knowledge. This price saving was achieved by utilizing Amazon S3 storage tiers in addition to improved compression (Zstandard), additional enhanced by the truth that Parquet information have been sorted.

Since Apache Iceberg is nicely supported by AWS knowledge companies and Cloudinary was already utilizing Spark on Amazon EMR, they may combine writing to Information Catalog and begin a further Spark cluster to deal with knowledge upkeep and compaction. As exploration continued with Apache Iceberg, some attention-grabbing efficiency metrics have been discovered. For instance, for sure queries, Athena runtime was 2x–4x sooner than Snowflake.

Integration of Apache Iceberg

The combination of Apache Iceberg was finished earlier than loading knowledge to Snowflake. The information is written to an Iceberg desk utilizing Apache Parquet knowledge format and AWS Glue as the information catalog. As well as, a Spark software on Amazon EMR runs within the background dealing with compaction of the Parquet information to optimum measurement for querying by means of numerous instruments similar to Athena, Trino working on prime of EMR, and Snowflake.

Challenges confronted

Cloudinary confronted a number of challenges whereas constructing its petabyte-scale knowledge lake, together with:

  • Figuring out optimum desk partitioning
  • Optimizing ingestion
  • Fixing the small information drawback to enhance question efficiency
  • Cheaply sustaining Apache Iceberg tables
  • Selecting the best question engine

On this part, we describe every of those challenges and the options carried out to deal with them. Lots of the exams to test efficiency and volumes of knowledge scanned have used Athena as a result of it gives a easy to make use of, absolutely serverless, price efficient, interface with out the necessity to setup infrastructure.

Figuring out optimum desk partitioning

Apache Iceberg makes partitioning simpler for the consumer by implementing hidden partitioning. Slightly than forcing the consumer to provide a separate partition filter at question time, Iceberg tables could be configured to map common columns to the partition keys. Customers don’t want to keep up partition columns and even perceive the bodily desk format to get quick and correct question outcomes.

Iceberg has a number of partitioning choices. One instance is when partitioning timestamps, which could be finished by yr, month, day, and hour. Iceberg retains observe of the connection between a column worth and its partition with out requiring extra columns. Iceberg may also partition categorical column values by identification, hash buckets, or truncation. As well as, Iceberg partitioning is user-friendly as a result of it additionally permits partition layouts to evolve over time with out breaking pre-written queries. For instance, when utilizing every day partitions and the question sample modifications over time to be based mostly on hours, it’s doable to evolve the partitions to hourly ones, thus making queries extra environment friendly. When evolving such a partition definition, the information within the desk previous to the change is unaffected, as is its metadata. Solely knowledge that’s written to the desk after the evolution is partitioned with the brand new definition, and the metadata for this new set of knowledge is stored individually. When querying, every partition format’s respective metadata is used to determine the information that should be accessed; that is known as split-planning. Cut up-planning is certainly one of many Iceberg options which might be made doable as a result of desk metadata, which creates a separation between the bodily and the logical storage. This idea makes Iceberg extraordinarily versatile.

Figuring out the proper partitioning is essential when working with massive knowledge units as a result of it impacts question efficiency and the quantity of knowledge being scanned. As a result of this migration was from present tables from Snowflake native storage to Iceberg, it was essential to check and supply an answer with the identical or higher efficiency for the prevailing workload and kinds of queries.

These exams have been doable attributable to Apache Iceberg’s:

  1. Hidden partitions
  2. Partition transformations
  3. Partition evolution

These allowed altering desk partitions and testing which technique works finest with out knowledge rewrite.

Listed below are a couple of partitioning methods that have been examined:

  1. PARTITIONED BY (days(day), customer_id)
  2. PARTITIONED BY (days(day), hour(timestamp))
  3. PARTITIONED BY (days(day), bucket(N, customer_id))
  4. PARTITIONED BY (days(day))

Every partitioning technique that was reviewed generated considerably completely different outcomes each throughout writing in addition to throughout question time. After cautious outcomes evaluation, Cloudinary determined to partition the information by day and mix it with sorting, which permits them to kind knowledge inside partitions as can be elaborated within the compaction part.

Optimizing ingestion

Cloudinary receives billions of occasions in information from its suppliers in numerous codecs and sizes and shops these on Amazon S3, leading to terabytes of knowledge processed and saved day-after-day.

As a result of the information doesn’t are available a constant method and it’s not doable to foretell the incoming fee and file measurement of the information, it was essential to discover a approach of retaining price down whereas sustaining excessive throughput.

This was achieved by utilizing EventBridge to push every file acquired into Amazon SQS, the place it was processed utilizing Spark working on Amazon EMR in batches. This allowed processing the incoming knowledge at excessive throughput and scale clusters in line with queue measurement whereas retaining prices down.

Instance of fetching 100 messages (information) from Amazon SQS with Spark:

var consumer = AmazonSQSClientBuilder.customary().withRegion("us-east-1").construct()
var getMessageBatch: Iterable[Message] = DistributedSQSReceiver.consumer.receiveMessage(new ReceiveMessageRequest().withQueueUrl(queueUrl).withMaxNumberOfMessages(10)).getMessages.asScala
sparkSession.sparkContext.parallelize(10) .map(_ => getMessageBatch) .gather().flatMap(_.toList) .toList

When coping with a excessive knowledge ingestion fee for a selected partition prefix, Amazon S3 may doubtlessly throttle requests and return a 503 standing code (service unavailable). To deal with this state of affairs, Cloudinary used an Iceberg desk property known as write.object-storage.enabled, which includes a hash prefix into the saved Amazon S3 object path. This strategy was deemed environment friendly and successfully mitigated Amazon S3 throttling issues.

Fixing the small file drawback and enhancing question efficiency

In fashionable knowledge architectures, stream processing engines similar to Amazon EMR are sometimes used to ingest steady streams of knowledge into knowledge lakes utilizing Apache Iceberg. Streaming ingestion to Iceberg tables can undergo from two issues:

  • It generates many small information that result in longer question planning, which in flip can influence learn efficiency.
  • Poor knowledge clustering, which may make file pruning much less efficient. This usually happens within the streaming course of when there may be inadequate new knowledge to generate optimum file sizes for studying, similar to 512 MB.

As a result of partition is a key issue within the variety of information produced and Cloudinary’s knowledge is time based mostly and most queries use a time filter, it was determined to deal with the optimization of our knowledge lake in a number of methods.

First, Cloudinary set all the mandatory configurations that helped cut back the variety of information whereas appending knowledge within the desk by setting, which permits defining the default goal file measurement. Setting spark.sql.shuffle.partitions in Spark can cut back the variety of output information by controlling the variety of partitions used throughout shuffle operations, which impacts how knowledge is distributed throughout duties, consequently minimizing the variety of output information generated after transformations or aggregations.

As a result of the above strategy solely addressed the small file drawback however didn’t eradicate it totally, Cloudinary used one other functionality of Apache Iceberg that may compact knowledge information in parallel utilizing Spark with the rewriteDataFiles motion. This motion combines small information into bigger information to scale back metadata overhead and reduce the quantity of Amazon S3 GetObject API operation utilization.

Right here is the place it may well get difficult. When working compaction, Cloudinary wanted to decide on which technique to use out of the three that Apache Iceberg affords; every one having its personal benefits and downsides:

  1. Binpack – merely rewrites smaller information to a goal measurement
  2. Type – knowledge sorting based mostly on completely different columns
  3. Z-order – a method to colocate associated knowledge in the identical set of information

At first, the Binpack compaction technique was evaluated. This technique works quickest and combines small information collectively to achieve the goal file measurement outlined and after working it a big enchancment in question efficiency was noticed.

As talked about beforehand, knowledge was partitioned by day and most queries ran on a selected time vary. As a result of knowledge comes from exterior distributors and typically arrives late, it was observed that when working queries on compacted days, numerous knowledge was being scanned, as a result of the particular time vary may reside throughout many information. The question engine (Athena, Snowflake, and Trino with Amazon EMR) wanted to scan your complete partition to fetch solely the related rows.

To extend question efficiency even additional, Cloudinary determined to alter the compaction course of to make use of kind, so now knowledge is partitioned by day and sorted by requested_at (timestamp when the motion occurred) and buyer ID.

This technique is costlier for compaction as a result of it must shuffle the information to be able to kind it. Nonetheless, after adopting this kind technique, two issues have been noticeable: the identical queries that ran prior to now scanned round 50 p.c much less knowledge, and question run time was improved by 30 p.c to 50 p.c.

Cheaply sustaining Apache Iceberg tables

Sustaining Apache Iceberg tables is essential for optimizing efficiency, decreasing storage prices, and making certain knowledge integrity. Iceberg gives a number of upkeep operations to maintain your tables in fine condition. By incorporating these operations Cloudinary have been in a position to cost-effectively handle their Iceberg tables.

Expire snapshots

Every write to an Iceberg desk creates a brand new snapshot, or model, of a desk. Snapshots can be utilized for time-travel queries, or the desk could be rolled again to any legitimate snapshot.

Often expiring snapshots is advisable to delete knowledge information which might be now not wanted and to maintain the dimensions of desk metadata small. Cloudinary determined to retain snapshots for as much as 7 days to permit simpler troubleshooting and dealing with of corrupted knowledge which typically arrives from exterior sources and aren’t recognized upon arrival. SparkActions.get().expireSnapshots(iceTable).expireOlderThan(TimeUnit.DAYS.toMillis(7)).execute()

Take away outdated metadata information

Iceberg retains observe of desk metadata utilizing JSON information. Every change to a desk produces a brand new metadata file to supply atomicity.

Outdated metadata information are stored for historical past by default. Tables with frequent commits, like these written by streaming jobs, may must repeatedly clear metadata information.

Configuring the next properties will ensure that solely the newest ten metadata information are stored and something older is deleted.


Delete orphan information

In Spark and different distributed processing engines, when duties or jobs fail, they could depart behind information that aren’t accounted for within the desk metadata. Furthermore, in sure situations, the usual snapshot expiration course of may fail to determine information which might be now not essential and never delete them.

Apache Iceberg affords a deleteOrphanFiles motion that may deal with unreferenced information. This motion may take a very long time to finish if there are numerous information within the knowledge and metadata directories. A metadata or knowledge file is taken into account orphan if it isn’t reachable by any legitimate snapshot. The set of precise information is constructed by itemizing the underlying storage utilizing the Amazon S3 ListObjects operation, which makes this operation costly. It’s advisable to run this operation periodically to keep away from elevated storage utilization; nonetheless, too frequent runs can doubtlessly offset this price profit.

instance of how crucial it’s to run this process is to take a look at the next diagram, which exhibits how this process eliminated 112 TB of storage.

Rewriting manifest information

Apache Iceberg makes use of metadata in its manifest record and manifest information to hurry up question planning and to prune pointless knowledge information. Manifests within the metadata tree are routinely compacted within the order that they’re added, which makes queries sooner when the write sample aligns with learn filters.

If a desk’s write sample doesn’t align with the question learn filter sample, metadata could be rewritten to re-group knowledge information into manifests utilizing rewriteManifests.

Whereas Cloudinary already had a compaction course of that optimized knowledge information, they observed that manifest information additionally required optimization. It turned out that in sure instances, Cloudinary reached over 300 manifest information—which have been small, usually below 8Mb in measurement—and attributable to late arriving knowledge, manifest information have been pointing to knowledge in several partitions. This brought about question planning to run for 12 seconds for every question.

Cloudinary initiated a separate scheduled strategy of rewriteManifests, and after it ran, the variety of manifest information was decreased to roughly 170 information and because of extra alignment between manifests and question filters (based mostly on partitions), question planning was improved by thrice to roughly 4 seconds.

Selecting the best question engine

As a part of Cloudinary exploration aimed toward testing numerous question engines, they initially outlined a number of key efficiency indicators (KPIs) to information their search, together with assist for Apache Iceberg alongside integration with present knowledge sources similar to MySQL and Snowflake, the provision of an online interface for easy one-time queries, and price optimization. According to these standards, they opted to judge numerous options together with Trino on Amazon EMR, Athena, and Snowflake with Apache Iceberg assist (at the moment it was accessible as a Personal Preview). This strategy allowed for the evaluation of every resolution towards outlined KPIs, facilitating a complete understanding of their capabilities and suitability for Cloudinary’s necessities.

Two of the extra quantifiable KPIs that Cloudinary was planning to judge have been price and efficiency. Cloudinary realized early within the course of that completely different queries and utilization sorts can doubtlessly profit from completely different runtime engines. They determined to give attention to 4 runtime engines.

Engine Particulars
Snowflake native XL knowledge warehouse on prime of knowledge saved inside Snowflake
Snowflake with Apache Iceberg assist XL knowledge warehouse on prime of knowledge saved in S3 in Apache Iceberg tables
Athena On-demand mode
Amazon EMR Trino Opensource Trino on prime of eight nodes (m6g.12xl) cluster

The take a look at included 4 kinds of queries that symbolize completely different manufacturing workloads that Cloudinary is working. They’re ordered by measurement and complexity from the only one to essentially the most heavy and complicated.

Question Description Information scanned Returned outcomes set
Q1 Multi-day aggregation on a single tenant Single digit GBs <10 rows
Q2 Single-day aggregation by tenant throughout a number of tenant Dozens of GBs 100 thousand rows
Q3 Multi-day aggregation throughout a number of tenants Lots of of GBs <10 rows
This fall Heavy sequence of aggregations and transformations on a multi-tenant dataset to derive entry metrics Single digit TBs >1 billion rows

The next graphs present the fee and efficiency of the 4 engines throughout the completely different queries. To keep away from chart scaling points, all prices and question durations have been normalized based mostly on Trino working on Amazon EMR. Cloudinary thought-about Question 4 to be much less appropriate for Athena as a result of it concerned processing and reworking extraordinarily massive volumes of advanced knowledge.

Some necessary elements to think about are:

  • Price for EMR working Trino was derived based mostly on question period solely, with out contemplating cluster arrange, which on common launches in slightly below 5 minutes.
  • Price for Snowflake (each choices) was derived based mostly on question period solely, with out contemplating chilly begin (greater than 10 seconds on common) and a Snowflake warehouse minimal cost of 1 minute.
  • Price for Athena was based mostly on the quantity of knowledge scanned; Athena doesn’t require cluster arrange and the question queue time is lower than 1 second.
  • All prices are based mostly on record on-demand (OD) costs.
  • Snowflake costs are based mostly on Customary version.

The above chart exhibits that, from a price perspective, Amazon EMR working Trino on prime of Apache Iceberg tables was superior to different engines, in sure instances as much as ten instances inexpensive. Nonetheless, Amazon EMR setup requires extra experience and expertise in comparison with the no-code, no infrastructure administration supplied by Snowflake and Athena.

When it comes to question period, it’s noticeable that there’s no clear engine of alternative for all sorts of queries. Actually, Amazon EMR, which was essentially the most cost-effective choice, was solely quickest in two out of the 4 question sorts. One other attention-grabbing level is that Snowflake’s efficiency on prime of Apache Iceberg is nearly on-par with knowledge saved inside Snowflake, which provides one other nice choice for querying their Apache Iceberg data-lake. The next desk exhibits the fee and time for every question and product.

. Amazon EMR Trino Snowflake (XL) Snowflake (XL) Iceberg Athena
Query1 $0.01
5 seconds
8 seconds
8 seconds
11 seconds
Query2 $0.12
107 seconds
28 seconds
39 seconds
94 seconds
Query3 $0.17
147 seconds
120 seconds
211 seconds
26 seconds
Query4 $6.43
1,237 seconds
1,324 seconds
1,430 seconds

Benchmarking conclusions

Whereas each resolution presents its personal set of benefits and disadvantages—whether or not when it comes to pricing, scalability, optimizing for Apache Iceberg, or the distinction between open supply versus closed supply—the wonder lies in not being constrained to a single alternative. Embracing Apache Iceberg frees you from relying solely on a single resolution. In sure eventualities the place queries should be run steadily whereas scanning as much as lots of of gigabytes of knowledge with an purpose to evade warm-up intervals and hold prices down, Athena emerged as the only option. Conversely, when tackling hefty aggregations that demanded vital reminiscence allocation whereas being aware of price, the choice leaned in the direction of utilizing Trino on Amazon EMR. Amazon EMR was considerably extra price environment friendly when working longer queries, as a result of boot time price might be discarded. Snowflake stood out as an incredible choice when queries might be joined with different tables already residing inside Snowflake. This flexibility allowed harnessing the strengths of every service, strategically making use of them to swimsuit the particular wants of assorted duties with out being confined to a singular resolution.

In essence, the true energy lies within the potential to tailor options to various necessities, utilizing the strengths of various environments to optimize efficiency, price, and effectivity.


Information lakes constructed on Amazon S3 and analytics companies similar to Amazon EMR and Amazon Athena, together with the open supply Apache Iceberg framework, present a scalable, cost-effective basis for contemporary knowledge architectures. It allows organizations to rapidly assemble strong, high-performance knowledge lakes that assist ACID transactions and analytics workloads. This mix is essentially the most refined method to have an enterprise-grade open knowledge atmosphere. The supply of managed companies and open supply software program helps corporations to implement knowledge lakes that meet their wants.

Since constructing a knowledge lake resolution on prime of Apache Iceberg, Cloudinary has seen main enhancements. The information lake infrastructure allows Cloudinary to increase their knowledge retention by six instances whereas reducing the price of storage by over 25 p.c. Moreover, question prices dropped by greater than 25–40 p.c due to the environment friendly querying capabilities of Apache Iceberg and the question optimizations supplied within the Athena model 3, which is now based mostly on Trino as its engine. The power to retain knowledge for longer in addition to offering it to varied stakeholders whereas decreasing price is a key element in permitting Cloudinary to be extra knowledge pushed of their operation and decision-making processes.

Utilizing a transactional knowledge lake structure that makes use of Amazon S3, Apache Iceberg, and AWS Analytics companies can significantly improve a corporation’s knowledge infrastructure. This permits for stylish analytics and machine studying, fueling innovation whereas retaining prices down and permitting the usage of a plethora of instruments and companies with out limits.

Concerning the Authors

Yonatan Dolan is a Principal Analytics Specialist at Amazon Internet Providers. He’s situated in Israel and helps prospects harness AWS analytical companies to leverage knowledge, acquire insights, and derive worth. Yonatan is an Apache Iceberg evangelist.

Amit Gilad is a Senior Information Engineer on the Information Infrastructure staff at Cloudinar. He’s at the moment main the strategic transition from conventional knowledge warehouses to a contemporary knowledge lakehouse structure, using Apache Iceberg to boost scalability and suppleness.

Alex Dickman is a Employees Information Engineer on the Information Infrastructure staff at Cloudinary. He focuses on partaking with numerous inside groups to consolidate the staff’s knowledge infrastructure and create new alternatives for knowledge purposes, making certain strong and scalable knowledge options for Cloudinary’s various necessities.

Itay Takersman is a Senior Information Engineer at Cloudinary knowledge infrastructure staff. Targeted on constructing resilient knowledge flows and aggregation pipelines to assist Cloudinary’s knowledge necessities.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments