Sunday, May 26, 2024
HomeBig DataREA Group: An Lively Metadata Pioneer - Atlan

REA Group: An Lively Metadata Pioneer – Atlan

Activating and Governing a Rising Information Platform with Atlan

The Lively Metadata Pioneers collection options Atlan prospects who’ve lately accomplished an intensive analysis of the Lively Metadata Administration market. Paying ahead what you’ve realized to the following information chief is the true spirit of the Atlan neighborhood! So that they’re right here to share their hard-earned perspective on an evolving market, what makes up their fashionable information stack, revolutionary use instances for metadata, and extra.

On this installment of the collection, we meet Surj Rangi, Enterprise Cloud Information Architect, Piyush Dhir, Senior Technical Lead, and Danni Garcia, Product Supervisor, at REA Group, the operator of main residential and business property web sites, mortgage brokering companies, and extra. Surj, Piyush, and Danni share REA’s evolving information stack, their data-driven ambitions, and the factors and course of behind their alternative of Atlan.

This interview has been edited for brevity and readability.

May you inform us a bit about yourselves, your backgrounds, and what drew you to Information & Analytics?

Surj Rangi:

I’m Surj Rangi, Architect in Information Companies, and I’ve been at REA for 2 years now. I graduated in IT from the UK, then labored in a lot of consultancy companies in Information and Analytics and developed a powerful background in cloud platforms and information structure. I migrated to Australia about seven years in the past, with 20 years of expertise in information throughout numerous industries together with Media, Telecommunications, Finance, E-commerce and Banking.

I joined REA and was very eager on the function that I used to be provided and the crew I used to be coming into. What actually enticed me was working with an organization that had a startup mentality, and have been excited to push and ship outcomes. Beforehand, I’ve labored with massive banks the place there’s plenty of paperwork and issues take time, and I used to be excited to see how issues work at a spot like REA.

Piyush Dhir:

I’m a Senior Technical Lead at REA. My journey goes again to school after I was ending my Bachelors in Software program Engineering and wanted to decide about what I wished to do subsequent.

I began as an Android developer again when it appeared like all people’s subsequent factor was “What will be my subsequent Android challenge?” Once I was doing that, I got here throughout SQL Server, studying how it’s important to do operational modeling once you’re creating one thing like a front-end software. That’s how I made my first step into information. Since then, I’ve been working throughout a lot of totally different sorts of information groups.

My first information crew was a Information Administration crew for a public firm in Australia. They have been ranging from zero, constructing a whole greenfield ecosystem for his or her information utilizing the SAP merchandise. I spent about 5 years in that world, then moved into plenty of small corporations and massive corporations. I did a little bit of consulting, I labored for a financial institution within the center, after which lastly ended up at REA.

Once I first joined an information crew again in 2012, what actually stood out to me on the time was that information was stated to be “the brand new oil”, and that Information & Analytics have been going to be the following massive factor. Again then, some individuals began doing Machine Studying and enjoying round with R Studio, nevertheless it was by no means the “bread and butter” of any firm, simply a type of “north star” type of initiatives.

Instantly, now 10 years down the road, it’s turn into not solely the “bread and butter” of the corporate, nevertheless it’s a chance for monetization for lots of them, too. It’s good to see that transition taking place, and it’s been fascinating to look at.

Danni Garcia:

I’m a Product Supervisor in Information Companies with a particular background in Information research. I haven’t all the time been in Product. I’ve labored within the expertise business for nearly a decade now throughout many various areas and roles in each massive and small organizations, however I began out as a Information Analyst. 

Would you thoughts describing REA, and the way your information crew helps the group?


I believe it’s good to know that REA began in a storage in Australia within the early-to-mid ’90s, and since then the corporate has grown and scaled enormously throughout the globe. REA has a presence not solely in Australia, however Asia too and has sturdy ties with NewsCorp. We began by itemizing residential properties, and it’s grown from there to business properties and land, as effectively. We’ve additionally finished plenty of mergers and acquisitions.  For instance in Australia, we’ve purchased a agency known as Mortgage Selection that permits REA to be positioned not solely to promote listings, publications, and supply insights into property into the business in Australia, but additionally present mortgage dealer companies.

So if you wish to promote your property, REA offers the entire bundle. You’ll be able to promote your property, and when you want financing, we can assist you financial your subsequent funding.

We’ve gone by a protracted journey, and have had a Information Companies crew for a protracted time frame. Every part was decentralized, then it was centralized. Now it’s a little bit of a hybrid, the place we’ve got a centralized information crew constructing out the centralized information platform with key capabilities for use throughout the group, with decentralized information possession. We are attempting to align with a Information Mesh strategy by way of how we construct out our platform capabilities and adoption of “information as a product” throughout the group. 

We’re multi-cloud, each AWS and GCP, which brings its personal challenges, and we do every part from ingestion of information, event-driven structure to machine studying. We’re constructing information property to share with exterior corporations within the type of an information market.


Information Companies exists to assist the entire inside traces of companies  throughout  our group. We’re not an operational crew, however a foundational one, that builds information merchandise and capabilities to assist assist groups to allow them to efficiently leverage information for his or her merchandise. Our mission is to make it straightforward to grasp, shield and leverage REA information.


I’ll add that over the past couple of years, REA has predominantly seen themselves as a listings enterprise. It’s nonetheless a listings enterprise, offering the most effective listings info doable out to prospects and shoppers. However what’s occurred is that this wealthy information evolution helps our enterprise turn into data-driven. Among the information metrics you see on the REA web site and cellular software are largely derived from the work that the group has put in to develop our Information & Analytics and ML follow to drive higher determination making.

Now we have plenty of invaluable information. There are plenty of initiatives happening now to broaden the utilization of information, and over the following two years, we are going to develop our panorama and derive even higher outcomes for our prospects and shoppers. to grasp, leverage, then showcase information to our prospects and their prospects.

What does your information stack seem like?


Now we have a real-time ingestion platform known as Hydro utilizing MSK, which is a custom-built streaming platform. Then we’ve got our batch platform, which ingests batch information utilizing Breeze, constructed on Airflow. Our information lake resolution is BigQuery.


We have a look at ourselves as a poly-cloud firm, utilizing each AWS and Google Cloud Platform, for the time being.

From an AWS perspective, we’ve got most of our infrastructure workloads operating there. Now we have EC2 situations and RDS operating there. Now we have our personal VPC. Now we have a number of load balancers. 

From a Information and Analytics perspective, nearly all of our workloads are in GCP. We’re presently utilizing BigQuery as an information lake idea, and that’s the place most of our workloads run. We use SageMaker for ML, and there’s some groups which might be experimenting with BigQuery ML on the GCP facet, as effectively. We even have a self-managed Airflow occasion, in order that’s our information platform. 

We’re presently within the means of establishing our personal event-driven structure framework utilizing Kafka, which is on AWS MSK.

Aside from that, our Tableau entrance finish is used for reporting, so we’ve got each the Tableau desktop and the server model, for the time being.

Why seek for an Lively Metadata Administration resolution? What was lacking?


Now we have an present open-source information catalog that we’ve got been utilizing for a couple of years now. Adoption has not been nice. As we’ve scaled and grown, we realized that we would have liked one thing that’s extra related for the fashionable information stack, which is the route that we’re going in direction of. 

There’s additionally a stronger push in our business towards higher safety of information. We retailer plenty of personally identifiable information throughout the enterprise, and a few of our key methods we’ve got in Information Companies are that we wish to first perceive the info, shield it, then leverage it. We wish to have the ability to catalog our information, and perceive how dispersed it’s throughout our warehouses, numerous platforms, in batches, and streams.

Now we have plenty of information, e.g. we’ve bought over two petabytes of information in GCP BigQuery alone.  We wish to have the ability to perceive what information is, the place it’s put collectively, and apply extra rigor to it. Now we have good frameworks internally by way of governance, processes, and insurance policies, however we wish to have the suitable tech stack to assist us use this information.


There have been some technical limitations, as our earlier information catalog might solely assist BigQuery, however we actually wished to assist the route of the enterprise by way of scale and the way it will align extra broadly with our Information Imaginative and prescient and Technique.

Our technique desires to implement Information Mesh and ‘Information as a Product’ mindset throughout the group. Each crew owns information, they leverage it they usually have a accountability to handle it with governance frameworks.

So, so as to embed Information Governance practices and this cultural shift, we would have liked a instrument to assist the frameworks, metadata technique, and tagging technique. We additionally wanted an answer to centralize all our Information Property so we might have visibility of the place information is and the way it’s being categorised which helps our Privateness initiatives. 

We’re nonetheless on a metamorphosis journey at REA, which may be very thrilling. A brand new information catalog was an actual alternative to push ourselves additional into that transformation with a brand new Information Governance framework.

How did your analysis course of work? Did something stand out?


We did some market analysis, talking to Gartner and reviewing accessible tooling throughout the business. We might have clearly saved utilizing our present Information Catalog, however we wished to judge a large spectrum of instruments together with Atlan, Alation, and Open Metadata, to cowl Open Supply vs. Vendor managed.

We felt Atlan match the factors of a contemporary information stack, offering us the capabilities we’d like, similar to self-service tooling, an open API, and integrations to quite a lot of expertise stacks which have been all essential to us.

We had an overwhelmingly good expertise participating with Atlan, particularly with the Skilled Companies crew. The arrogance that they gave us within the tooling once we went by our use instances drove a sense of sturdy alignment between REA and Atlan.


We did a three-phase analysis course of. Initially we went out to the market, did a few of our personal analysis, attempting to grasp which corporations might match our use instances.

As soon as we did that, we went again and checked out totally different points similar to pricing and used that as a filtering mechanism. We additionally regarded on the future roadmap of these corporations to determine the place every firm may be going, which was our second filtering course of. Once we have been finished selecting our choices, we had to determine which one would go well with us greatest.

That’s once we did a lightweight proof of worth the place we created high-level analysis standards the place all people concerned might rating totally different capabilities from 1-10. The crew included a supply supervisor, a product supervisor, an architect, and builders, simply to get a holistic view of the expertise all people can be getting out of the instrument. After that scoring, we made a light-weight advice and offered it to our executives.

A few of what we have been within the analysis standards have been issues like understanding what information sources we might combine to, what safety regarded like, and ideas like extensibility so we could possibly be versatile sufficient to increase the catalog programmatically or through API. As a result of we’ve got our information platform operating on Airflow, we additionally wished to grasp how effectively every choice labored with that.

Then we additionally checked out roadmaps and requested ourselves what may occur sooner or later, and if one thing like Atlan’s funding in AI is one thing we should be trying into, and different future enhancements Atlan or different distributors might present. We have been attempting to get an understanding of the following two or three years, as a result of if we’re investing, we’re investing with a long-term perspective.


In the event you have a look at the time period “Information Catalog”, it’s been round for a really very long time. I’ve been working over 20 years, and I’ve used information catalogs for a very long time, however the evolution has been vital.

When Piyush, Danni and I have been distributors, that’s one thing we have been excited about. Would you like a conventional information catalog, which we’ve most likely seen in banks which have a powerful, ruled, centralized physique, or would you like one thing that’s evolving with the occasions, and evolving the place the business is heading?

I believe that’s why it was good to listen to from Atlan, and we appreciated the place they have been positioned in that evolution. We like that Atlan integrates with a lot of tech stacks. For instance, we use Nice Expectations for information high quality for the time being, however we’re contemplating Soda or Monte Carlo, and we realized Atlan already has an integration with Soda and Monte Carlo. We’re discovering extra examples of that, the place Atlan is turning into extra related.

Conversely, once we have been addressing personally identifiable info, we wished to have the ability to scan our information units. Atlan was fairly clear, saying “We’re not a scanning instrument, that’s not us.” It was good to have that differentiation. Once we checked out Open Metadata, they stated they’d scanning functionality, nevertheless it wasn’t as complete as we have been anticipating, and we all know now that this use case is in a special realm.

It’s good to have that readability, and know which route Atlan goes to go.

How do you propose on rolling Atlan out to your customers?


So usually in platforming and tooling, we’re very caught up specializing in the expertise and never specializing in the person expertise. That’s the place Atlan can actually assist.

We wish to create one thing that’s tangible, and that folks wish to use, so we will drive mass adoption of the platform. With our earlier catalog, we didn’t have a lot adoption, so we’re making {that a} success metric, and one of many nice options in Atlan is that we will customise it to satisfy the wants of differing personas. An idea that hasn’t been historically pushed within the Information Governance house!

We went out to the enterprise and undertook a giant train, interviewing our stakeholders and potential customers. Now, we actually perceive the use instances, scale and what our customers need from the Information Catalog. Our personas – analysts, producers, homeowners and customers will all be supported within the roll out of Atlan, ensuring that their expertise is custom-made inside the instrument they usually can all perceive and use information successfully for his or her roles. 

Photograph by Nico Smit on Unsplash



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments