Cloud Computing
The Data Lake feature allows you to perform analytics on your data usage and prepare reports. Data Lake is a large repository that stores both structured and unstructured data. Data Lake Storage combines the scalability and cost benefits of object storage with the reliability and performance of the Big Data file system capabilities. The following illustration shows how Azure Data Lake stores all your business data and makes it available for analysis.
The goal of cloud computing is to make running a business easier and more efficient, whether it’s a small start-up or a large enterprise. Every business is unique and has different needs. To meet those needs, cloud computing providers offer a wide range of services. Cloud compute services including Virtual Machines, Containers, App Service and Serverless computing offer application development and deployment approaches if applied correctly can save time and money. Each service provides benefits as well as tradeoffs against other options. IT needs to have a good understanding of these compute services.
Virtual Machines (VM) is an emulation of a physical computer, which offers more control that comes with maintenance overhead.
Containers provide a consistent, isolated execution environment for applications. They are similar to VMs except they don’t require a guest operating system. Instead, the application and all its dependencies is packaged into a “container” and then a standard runtime environment is used to execute the app. This allows the container to start up in just a few seconds, because there’s no OS to boot and initialize. You only need the app to launch.
Serverless computing lets you run application code without creating, configuring, or maintaining a server. Each approach is optimized for specific use case. The core idea is that your application is broken into separate functions that run when triggered by some action. This is ideal for automated tasks.
Suppose you work in the analytics department of a large health system. Your organization’s IT infrastructure is hybrid both on-premise and cloud-based, and all data, including customer interactions and services information, resides in Azure SQL Data Warehouse. Your department analyzes customer services usage patterns and proposes inefficiencies in the processes based on your findings. You can achieve the desired results by using the robust machine learning and deep learning functions of Azure Databricks in conjunctions with the Azure SQL Data Warehouse.
Azure Databricks is a fully managed, cloud-based big data and machine learning platform. It enables developers to accelerate AI implementation by simplifying the process of building enterprise-grade production data applications. Built in a joint effort by Microsoft and the team that started Apache Spark, Azure Databricks provides data science and engineering teams with a single platform for big data processing and machine learning.
By combining an end-to-end, managed Apache Spark platform optimized for the cloud with the enterprise scale and security of the Azure platform, Azure Databricks makes it easy to run large-scale Spark workloads.
You can access SQL Data Warehouse from Azure Databricks by using the SQL Data Warehouse connector. SQL Data Warehouse connector is a data source implementation for Apache Spark that uses Azure Blob storage and PolyBase in SQL Data Warehouse to transfer large volumes of data efficiently between an Azure Databricks cluster and a SQL Data Warehouse instance.
Both the Azure Databricks cluster and the SQL Data Warehouse instance access a common Blob storage container to exchange data. In Azure Databricks, Spark jobs are triggered by the SQL Data Warehouse connector to read data from and write data to the Blob storage container. On the SQL Data Warehouse side, data loading and unloading operations performed by PolyBase are triggered by the SQL Data Warehouse connector through JDBC.
PolyBase is a technology that accesses data outside of a database via the T-SQL language. In Azure SQL Data Warehouse, you can import and export data to and from Azure Blob storage and Azure Data Lake Store.
Azure Data Factory is a cloud-based data integration service. It lets you create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation. Data Factory supports various data stores. In this case it uses Azure SQL Database as a data source.
Over the past few years Enterprise Cloud Strategy has become an integral part of IT strategy. There is a growing realization that cloud computing not only represents a set of technical opportunities for efficiencies and cost savings, but also provides the potential to significantly transform the scope of enterprise computing. In fact, many enterprises are finding that cloud computing offers entirely new business models, revenue streams, and vehicles for customer intimacy.
The goal of any enterprise strategy is to create competitive differentiation and advantage, and little doubt remains that IT has become a key element in modern strategy. IT now drives transformative innovation, making it possible for enterprises to compete more effectively by instantiating processes that deliver ongoing competitive advantage.
For any IT organization to be successful in delivering its enterprise strategy it must deliver on its Enterprise Cloud Strategy. There are many areas to be considered when executing a successful cloud strategy. In this blog I will highlight the key areas to be considered when delivering on a enterprise cloud strategy. In the later posts I will explain each area in detail.
1) Business drivers or motivation for Cloud
2) Organization’s Goals and Benefits
3) Risks and Challenges for the organization
4) Cloud is the main beneficiary of the following technology Innovations
5) Cloud Main Characteristics
6) Cloud Delivery Models
7) Cloud Deployment Models
8) Cloud Patterns
9) Cloud Technology Pillars
1) Create a clear vision for the cloud usage for your organization
2) Define clear business goals (growth, revenue, improvement, and/or differentiation)
3) Get agreement from the Leadership
4) Define a clear business case and link it to opportunities in areas like business-process improvement, business intelligence, and innovation
5) Define cloud deployment road-map
6) Define ROI in terms of revenue, expenses, growth potential and market share
7) Get detail cost analysis for the potential cloud solution for your organization’s environment
8) Communicate the road-map to internal groups involved in cloud deployment
9) Do pilot programs for cloud deployments
10) Create quantifiable metrics for cloud usages
11) Do an agile approach and improve release cycles
12) Adapt policies to ensure consistency in procurement, data usage, and integration.
Goal: IT leadership goal is to get fast access to technology insight so they can quickly transform their IT assets for greater value to the business.
Technologies: Key technologies driving the changes are Cloud Computing, Virtualization, Big data, Mobility and consumerization.
Challenges: Some of the key and common challenges facing the organizations today are Smaller IT budgets, Lack of skill set, Legacy integration, Deployment time, Transformation of big data into strategic assets, Security, Cloud approach, Mobile and consumer empowerment strategy.
Expected Results: IT is looking ways to transform these technologies into business insights so they can give their organizations the competitive advantage.
Many organizations big or small are struggling these days on how to transform their organizations including people, processes, and technologies. The transformation is due to this new disruptive technology stack which includes Social, Mobile, Analytic and Cloud (SMAC). These technologies and concepts have been around for a while but their usage in this combination is new.
All over in the media the advantages of this stack have been advertised with great promise. There is no shortage of articles in the media on how organizations in every industry have started using these technologies and their business units are benefiting from the value these technologies are providing both in agility and cost. Irrespective of this hype many organizations are having a hard time in understanding the applications of this technology mix in their organizations. It is very hard to look at this technology stack holistically from an implementations perspective and not worry about the cost. The cost alone is not justifiable and the risks are too high because some aspects of this technology are still maturing.
There are innovative ways in which this problem can be addressed. Different organizations have different strategies, people, processes and technology mix. These corporations have different positions in the market they compete. All these companies have one thing in common they are all facing similar challenges when it comes to transformation of their organizations to this new disruption. These challenges can be met and solved in a very systematic way so that their impact on the organization both from the cost and culture perspective can be absorbed.
Organizations can take certain steps to approach this problem. They can start with their corporate strategy. Some of the questions to ask are what are the capabilities the organization is looking to create and improve? Is the corporate strategy is mapped to the IT strategy? How much of this strategy is being followed? If there is no clear corporate strategy the organization should create one and map it to its IT strategy. If the initiative can’t be taken at the company level, the department(s) can create their department(s) strategy. This approach if taken and communicated in successful manner can become the catalyst for the change in the rest of the organization. Other questions to ask are what business and technology capabilities the current IT organization supports? What are the current systems in place that support different business units? What are the programs and projects in flight or in plan to be delivered?
The Corporate strategy, IT capabilities, business priorities and current programs will determine what approach the organization should take to start consuming the SMAC stack (Social, Mobile, Analytic and Cloud). Most all organizations have legacy infrastructure with huge investment and cannot be converted into the Cloud. Now a days vendors are offering cloud-based Services for almost all the services traditionally offered by the internal IT. For example there are Cloud-based back-up Services, Cloud-based Disaster Recovery Services, Cloud Gateways and Connections for Cloud storage, Cloud-based Automation tools. There are so many choices of services available in the market for organizations to pick. Some of these services are new and some have matured. All these services provide a good business case from cost and time perspective. Organizations can subscribe to these services based on their needs. The main factors to look at when deciding to use a cloud based service are the cost, time, impact on current systems and the risks. All these variables are manageable. Corporations can start collecting data on their implemented cloud based services both for efficiency measuring and cost benefits. They can use this data to build their case for transformation of other IT services into the Cloud.
The bottom line is that all organizations need to find ways to get into the cloud game otherwise they will be playing catch-up with their competitors and their business will lose out to new opportunities that are being presented due to this new distributive technology mix.
Technology Leader - CTO 