Snowflake data sharing with Lyftrondata

Share on facebook
Share on twitter
Share on linkedin
Take a deep dive into Snowflake data sharing approach and how to leverage the powerful combination of Lyftrondata with Snowflake.
Snowflake data sharing approach
Snowflake data sharing with Lyftrondata
Benefits of Lyftrondata’s data sharing
Why data-driven enterprises do not need traditional ETL/ELT platforms anymore.
The evolution of the modern datahub.

Intra-organization use cases

An organization wants to create three different accounts due to different audit, logging, and security requirements for other groups. The three accounts are, Production account (PROD), Stage account (STAGE), and Development Account (DEV).

  • Production data is shared with STAGE.
  • STAGE has more source data loading into the table, validating the data loaded correctly.
  • STAGE shares the updated data with DEV.
  • DEV needs to modify data by conducting some transformation, and then shares it with PROD.

Based on the requirement, here is the Snowflake Data Sharing implementation.

In this big data era, data sharing is critical for corporate communication and operations across industries and organizations. The ability to make data easily consumable by people who can draw value from it remains rudimentary as data grows in size and volume. The majority of an organization’s data is locked away in numerous silos, with no easy method to make it accessible to potential consumers. Because massive datasets must now move across business borders by wire, the task of allowing data access has become substantially more difficult. As a result, the economic possibility for data sharing will continue to rise, creating unmet market needs for us.

Large datasets often have to be shared, whether between business departments within an enterprise or externally amidst several parties. The fact that Snowflake allows you to share data sets with anyone who has access to Snowflake accounts makes it the ideal cloud platform for data sharing.

Data Sharing in Snowflake enables account-to-account sharing of Snowflake database tables and views. The main participants on each side of a data-sharing relationship are the owner of the accounts at either end of the data-sharing relationship and any individual who shares specific accounts from your single master account with co-workers.

Snowflake data sharing approach

Snowflake Data Sharing is about taking your preferred account data model and exposing just the appropriate attributes across one or multiple accounts owned by separate organizations. That way each account contains the minimum amount of information for its use. No matter where your data is hosted, Snowflake has a solution that allows you to share it seamlessly between groups of people who need access. Simply set up a Data Sharing object within any table in your preferred account and click on the SHARES section in the left menu to start adding collaborators and configuring access privileges. Edit the container object’s metadata to tailor it exactly to your needs; if multiple accounts are involved, you can change the owner at any time!

Although a concept that may take some getting used to, shared data in the Snowflake Cost Table does not consume storage space for consumers. Therefore, Snowflake table storage expenses would not be reflected on your monthly bills. In addition, because no data is copied, arranging and making the most of sharing  is easily executed in minimal time, achieving instant results.

Moreover, by enabling data sharing from their Snowflake accounts, a large base of customers can easily be offered access to datasets from data producers. In comparison, companies that purchase data can then easily combine it with other datasets or information for further analysis and refining before utilizing it. For instance, a staffing company may have access to candidate records with related job applications. They could then mix these records and public profiles with their in-house client references about the candidates. This has the potential to make for more accurate referrals while retaining a competitive advantage if the new combined datasets were to remain exclusive.

Please note that “data sharing” in Snowflake is as simple as moving data within your hard disk or between folders. You can already do this without special software, but with Snowflake Data Warehouse, you get to do it on a much larger scale! With the way AWS S3 works, if you have multiple buckets, you can access files from the bucket of any other user so long as those files are shared; with Snowflake, setting up data sharing is similar. It allows users to move tables and their contained objects from one database into another. Similarly, you could move entire databases within objects into different clusters using native tools like BigQuery Transfer Tool because it allows for table transfer of entire BigQuery tables across different projects!

Lyft, shift, and load any data into Snowflake instantly

Snowflake data sharing with Lyftrondata

Lyftrondata makes three offers available to its customers: direct data share, data marketplace, and data exchange. There is no difference in the technical underpinnings between all three offers – their only distinction is that they each have their own unique functionality.

Direct data share
Direct data sharing is the most straightforward offer. It simply means that both provider and consumer have Snowflake accounts. The provider is a Snowflake account and the consumer is a reader account. This is created by the provider, the provider and consumer are data-sharing between the two accounts, either directly if they are within the same region, or indirectly via replication if they are not in the same region. You can follow the official documentation to configure direct data share by yourself without any other process involved.
Data marketplace
Snowflake (the account) is a centralized catalyst for your company’s data repository that allows you to access and publish data without having to worry about transforming or copying the data. With an instance of Snowflake, you can easily become a provider and publish your data to the marketplace, as well as a consumer who gains access to the published data. Snowflake is like the admin of the network; it helps keep everything secure.
Data exchange
Data exchange means private data sharing, while Public Data Exchange refers to a centralized marketplace for buying and selling information. Both options will create a data repository that can be accessed by only those invited to access it.

How Lyftrondata helps to transform your Snowflake

Benefits of Lyftrondata’s data sharing facilities

Snowflake Integration is taken seriously in Lyftrondata’s efforts to overcome the limitations of current ETL tools that stage flat files on a local disk before pushing them to Snowflake. Such a solution requires extra disk space for data loading and adds additional delays to the data pipeline. Lyftrondata takes a unique approach to data loading to the Snowflake Data Warehouse by utilizing new data streaming capabilities in Snowflake.

Streaming transformation pushdown turns your streaming database into a SQL data warehouse. Once data is loaded, transformations are applied to the streamed records on the fly as they are written into the database. No need to pre-load any data or manage space for temporary files. You can get all the advantages of a SQL data warehouse while still keeping your streaming application deployed continuously.

The use cases given below are some of the most prevalent ones among customers. They are designed to give you ideas on how to develop a data-sharing use case that meets your requirements.

Data shared by sourcing and staffing company
This use case is from an on-demand staffing provider that connects workers to businesses in need of their services. As a result of this platform, the company was able to connect 77 million registered workers, alongside 300 thousand job opportunities. The company is now looking at ways to move its data into Snowflake to collate datasets into one database for analysis. This approach permits its provider to share part of its analytic data directly with its corporate customer based on preset rules and criteria.
Data shared by data and technology company
Back to the use case in the introduction. This use case is for an analytical software provider that offers its customers a simple yet robust way to ingest and visualize geolocation data, alongside its analytics and insights technology. The diverse customer portfolio it boasts keeps track of millions of locations across every continent and every country in the world. Clients can choose between the Flexible Data Sharing Plan or Regular Data Sharing if they wish to combine it with other data sources like CRM or financial records to help them make more accurate business decisions. The data are shared as follows:
  • Production data is shared with STAGE
  • STAGE has more source data loading into the table, validating the data loaded correctly.
  • STAGE shares the updated data with DEV.
  • DEV needs to modify data by adding some transformation, and then shares it with PROD.
  • PROD shares its table TB1 to both stages via Share1.
  • STAGE creates a new table TBB using CTAS (Create Table As Select from TB1 of PROD)
  • STAGE shares TBB to DEV via Share2.
  • DEV creates a new table TBD using CTAS (Create Table As Select from TBB of STAGE)
  • DEV shares TBD to PROD via Share3.
  • PROD switches to using TBD of PROD instead of TB1
  • PROD creates a new table TB1_V2 using CTAS (Create Table As Select from TBD of DEV)

How Lyftrondata drives powerful BI acceleration

Lyftrondata’s data-sharing framework is built upon the secured data governance technology so that users with approved access can access the shared data in real-time without any movement of the original data. With this secured framework, you can create instant insights as well as additional business value while improving your data governance.

Lyftrondata’s modern architecture provides row and column level access security to users making it easier, and more secure than ever before, to share your data without writing a single piece of code. With Lyftrondata’s robust and secure framework, you can enable governance on your data hub and easily maintain one-to-one, one-to-many, and many-to-many relationships with your customers & partners.

Let’s get personal: See Lyftrondata on your data in a live demo

Schedule a free, no-strings-attached demo to discover how Lyftrondata can radically simplify data lake ETL in your organization.

Recent Posts