A No-Limits Data Lake To Power Intelligent Action
SIS provides extensive Data Lake services to the Microsoft Dynamics 365 Finance and CRM Community. If you are considering the benefits a modern data warehouse can provide, consider SIS. Our BI team can work with you on your cloud road map or extracting data from D365 Finance.
What is a Data Lake?
Azure Data Lake Analytics is an on-demand analytics job service that simplifies big data with no infrastructure to manage.
A Data Lake is a storehouse for large amounts of raw data. Data Lake stores can scale to terabytes or petabytes of data. This data usually comes from various heterogeneous sources as structured, semi-structured, or unstructured. The purpose of a Data Lake is to store data in its original state which is different from a traditional data warehouse. Traditional data warehouses transform data at the time of consumption.
A total Data Lake solution consists of storage and processing both. Data Lake storage is designed for infinite scalability, fault tolerance, and high-throughput consumption of data in any shape or size.
Below are key Data Lake cases:
-
Analytics
-
Reporting
-
On-premise data transfer
-
Cloud and IoT data transfer
-
Big data processing
Data Lake Advantages:
-
Raw data is not discarded
-
Users can access data and create queries
-
Many times, the process is faster than current ETL tools
-
Can store unstructured and semi-structured data
When to use a Data Lake
Data Lake usage includes data analytics, machine learning, and data exploration.
A Data Lake also performs as the data source for a data warehouse. Raw data is consumed into the Data Lake and transformed into a format that is structured and able to be queried. Generally, this process uses an ELT (extract-load-transform) pipeline, where data is consumed and then transformed. Relational source data can go straight into the data warehouse, through an ELT process, bypassing the Data Lake.
Data Lake stores are also used for event streaming and IoT scenarios. This is because they can process volumes of relational and non-relational data without schema definition or transformation. Data Lake stores are optimized for massive throughput and can handle large amounts of small writes at low latency.
Data Lake Considerations
-
Schema or descriptive metadata is necessary
-
Semantic consistency across the data is important
-
Governance is necessary for access control and privacy issues
-
Considerations need to be made for relational data
-
Visualization tools are necessary to provide integrated or holistic views across the organization
-
Governance is necessary to ensure the Data Lake does not become a dumping ground for data that is never actually analyzed or mined for insights
What SIS Can Do for D365 Client:
If a client has their own Azure data warehouse, SIS can integrate into that data warehouse.
If a client has Azure and no data warehouse but wants to move into Azure, SIS can build a
Data warehouse and report platform in Azure.
If a client has their own Azure data warehouse – and wants SIS IP integrated, SIS can install the IP into their Data Lake.
Data Lake & Dynamics 365 Finance
If you are a Dynamics 365 Finance or CE client or looking to become one, it is natural to want to harness your ERP data for operational and analytical reporting. The Data Lake architecture will help that and allow an affordable architecture to do it. Azure has all the tools to bring other data into the lake. Be it on-premise or in the cloud, and to secure your data. It will allow you to retire BYOD (Bring Your Own Data). And can be a central hub for a downstream data mart or enterprise data warehouse. With export to Data Lake many other use cases can be built up for:
-
Operational Reporting
-
Analytics
-
Data Mashup
-
Cost effective repository of structured and nonstructured data
-
Decision enablement
-
Data and a platform for machine learning
-
Modern Data Warehouse
-
Separation of Storage and Computer
