Data Virtualization Powering Many Dimensions of Digital Transformation: Insights from Denodo

As architects pursue their digital enterprise projects, they are finding data virtualization is proving a valued ally for delivering reliable data integration and data sharing. IDN explores data virtualization use cases across analytics, APIs, cloud, mobile, IoT and more with Denodo execs.

by Vance McCarthy

Tags: analytics, APIs, BI, big data, cloud, data virtualization, Denodo, integration, IoT, mobile, query, SaaS, search, security, SOA,

Suresh Chandrasekaran
senior vice president


"In an instant, your data warehouse project of the last 10 years has just got shot! Data virtualization provides a virtual data layer across the entire enterprise."

CLOUD-CON
Integration & Web APIs
Enterprise-Grade Integration Across Cloud and On-Premise
Online Conference

As architects pursue their digital enterprise projects, they are finding data virtualization is proving a valued ally for delivering reliable data integration and data sharing.

 

IDN talks with execs from Denodo to explore how data virtualization is changing the game for many key projects – APIs, big data, cloud, mobile, SaaS and even IoT.

 

Two data usage trends being driven by digital transformation are fueling interest in today’s modern data virtualization, Suresh Chandrasekaran, a senior vice president at Denodo told IDN.  These are:  Data Heterogeneity and Data Proliferation.

 

“As the model for the enterprise data warehouse (or mega data store) fell away, data heterogeneity has come to be accepted as the new normal,” he said.  “Hadoop appears, Spark appears, MongoDB appears, and Cassandra appears. And all over the map you have data everywhere now.  Then, of course, you have this data running a proliferation of cloud and mobile applications.” 

 

“So, the result: In an instant, your data warehouse project of the last 10 years has just got shot! This new world of distributed and diverse data, needed by many more apps and users, is now something that is real – and will never go away,” Chandrasekaran added.

 

Into this new ‘data everywhere’ world comes today’s approach to data virtualization, with key capabilities to provide an integration engine and an intelligent abstraction layer. “Data virtualization has expanded beyond data federation,” Chandrasekaran said. “It now provides a virtual data layer across the entire enterprise.”

 

The data virtualization architecture approach at Denodo consists of three main parts:

  • An abstraction (virtual) data layer 
  • Decoupling between source level information and consumer-level information
  • Ability to create integrated canonical views (or business-oriented views) of disparate data

 

Thanks to Denodo’s approach, the virtual (or data abstraction) layer has become more powerful and intelligent. The layer can support all types of data sharing – from batch to real-time; data governance; security policies for access control and even sports a smart metadata model that can make sense of data fields that use different terms but essentially contain the same or related data, Chandrasekaran told IDN. 

 

In fact, metadata is one of the important enablers for data virtualization, he added.  “Think about it, the Internet works so seamlessly because all the complexity is layered out. We do the same thing by first looking at data from the metadata perspective,” Chandrasekaran said.


He explained metadata’s importance with this example: “Let’s say a supplier has 200 attributes. Which portions do I want to expose for this data service? Which ones need to be masked or have restricted access?  With metadata, you can do multiple views of those 200 attributes, because they are costless to build or run,” he said.


These capabilities mean data virtualization is also changing how enterprise architects think when data has to be available – batch, real-time or phases in between. 

 

“Real-time integration still exists, but now using caching and in-memory technologies, real-time can be coupled with near real time, batch and even event-driven updates.  So this overall integration conversation has been engulfed by a greater acceptance that data virtualization can provide so much flexibility thanks to its capability to abstract and provide business views of disparate data,” he explained.

 

The result: With data virtualization, customers can get any data type anywhere it needs to be – and at anytime it needs to be there. Further, customers can perform on-going data discovery that works across legacy, cloud, mobile, Hadoop and other unstructured and streaming data sources.  Companies can even set access policies to manage and secure how data is exposed to users or applications.

 

Taking this architectural view, here is a list of how to apply data virtualization to many IDN projects. The list was compiled with the help of Chandrasekaran and Denodo CMO Ravi Shankar.

 

APIs and SOA
To support API and SOA services, many customers use data virtualization as a ‘data abstraction layer’ to aggregate and expose data (as a service or s stream) using APIs that developers use to access data they need for their applications.

 

One key consideration is how to publish a data service via data virtualization, to make it easy to consume data. Denodo supports REST, web services, XML over SOAP, among other approaches.  “We often see companies who have multiple users, 5 want data for a mobile app, 5 more from a custom app or whatever,” Chandrasekaran said. “Our publishing options support end user endpoints. SQL-oriented, SOAP, REST SharePoint, Ajax or we can even post an event into a message bus.

 

APIs are also important to capture metadata and ingest MD as canonical views, he added. “So, you are using APIs to bring in the types of metadata that an application likes to see, that sets my contract and my canonical business view,” he said.  So, with data virtualization, users maintain their virtual data layer as a data hierarchy – bringing together important data views: Master View, Source View and Operational View.

 

The result is easy data access with great context (and not a ton of coding). As an example: one of Denodo’s telco customers, Jazztel, built a portal that provides a 360-degree view of the customer. This provides customer service reps single-point access to a variety of customer data from different apps and data sources, (CRM, billing, etc.) and did so without complex coding.  This project reduced call wait times drastically since reps, “don’t have to waste time searching for information from different sources,” Shankar added.

 

Analytics and Big Data
Data virtualization provides a seamless way for companies to combine data and provide it to business users to do historic and even predictive analysis, using user-friendly queries (without a lot of coding or training).  In effect, data virtualization is helping enterprises modernize their data warehouses – without a ton of next-gen investments or re-architecting for Hadoop.

 

“Many companies leverage Big Data as a ‘cold storage’ mechanism to offload prior year sales data,” Shankar said. While this approach can save money, it is also limiting.  Often, companies only keep data for their current year in such a data warehouse because it can be expensive, Shankar said. “Business users need access to both current and prior year sales data to do historical analysis.”

One healthcare vendor is using Denodo to provide SQL access to many of the company’s product managers, thus abstracting them from the need to learn Pig, Hive and other complex tools.  “We let users query all this data using simple SQL. This means less sophisticated users can construct queries based on the approach they know, and we push those down to the data layer,” Chandrasekaran said.

 

Data virtualization also provides other analytic benefits to companies looking to capture tweets and weblogs and later analyze that data. “We can feed new types of data, such as streams or unstructured data, into traditional data stores or apps,” Chandrasekaran said. “As an example, let’s say you want to combine some social media comments with Salesforce.com or LinkedIn. We provide the interface (to these social streams), so the app can consume it as a common model,” he added.

 

Cloud Integration (iPaaS)
Denodo is not an iPaaS, but it can deploy in Amazon (AWS) and Microsoft Azure. Beyond that, data virtualization can also help iPaaS customers simplify integration and data sharing across multiple SaaS apps and on-prem apps, Shankar said.

 

“At times, our data virtualization co-exists with iPaaS, but the (integration) challenges we solve are slightly different,” Chandrasekaran said. “We are good at providing flexible data services that are defined and exposed as business-oriented canonicals. So, we provide data as a service, others apps focus on logic, process or business rules – and the two combined deliver the overall business service.”

With so many SaaS deployments in today’s enterprise, data virtualization often plays one key data-centric role. “It doesn’t make sense to pull all your (cloud) data into on-prem or to have multiple data stores, so we see data virtualization as a transition path to ease the transition and migration. Also, post transition to do the processing close to the data,” he said.

 

Denodo worked with one enterprise that had one request, described this way, “It would be great if I can point to one place that will give me a combo of flat files, CSV and other data that is still on-prem. That way I could avoid custom workarounds.”   In this case, data virtualization is a core platform for creating and managing data services that are easy to build and access. In this way, Denodo works well with iPaaS to enable a more ‘data-on-demand’ driven approach to integration-as-a-service.  

 

Data Views (with User-Centricity)
Another strong play for data virtualization is its ability to uncomplicate complex queries for end users.

 

These aren’t just dumbed-down queries.  Just the opposite in fact, as Chandrasekaran explains. The user simplicity is the result of well thought out data services architecture, along with canonical data models.

 

“Before a data service sends data to the user, the data virtualization’s virtual data layer aggregates data from multiple backend systems to create a physical or virtual data master.  Then, we create the data services (via the data services layer).  So, if I look at customer deployments, they talk about tiered deployments to create multiple layers. So, I create alias views, enrichments, canonical business views – whatever level of data you want, I can create services for those,” he said.

 

But what about cases where the user / analyst isn’t quite sure what they are looking for?  Data virtualization has an answer for that increasingly common use case, too.  And, thanks to a unique formula that brings together REST and metadata.

 

“Imagine a table with many columns and many rows. Every column has its own unique URL and every cell has its own unique URL because it is a RESTful endpoint. Now I have more than that – I have a whole set of ‘linked views’ that are also easy-to-access RESTful endpoints. The Denodo indexer can index everything. This means, the user can query their data services, even when they don’t know exactly what each data service offers.

 

Enterprise Cloud
As cloud applications proliferate, data becomes inherently distributed across many sources. DV provides a powerful way to combine the data from the cloud apps with on-premise apps to enable a hybrid computing environment. “One regional bank in Florida uses data virtualization as a logical data warehouse to bridge on-premise and cloud systems to support loan origination,” Shankar said.

 

Other companies are migrating from on-premise to the cloud. For such companies, DV abstracts the migration from the data consumers (i.e. business users), so that they continue to receive their data, while IT makes the transition from on-premise to the cloud. Hence, business proceeds uninhibited. As an example, a large mobile technology insurance company is using data virtualization to move it’s on-premise applications to cloud applications, he added.

 

Mobile & IoT

Data virtualization provides several key benefits to mobile app developers looking to access historic or contextual data for their apps.   According to Chandrasekaran, these benefits include:

 

(1) ease of access;
(2) ability to get real-time data without huge amounts of coding or latency; and
(3) power to customize the type and / or how much data is sent to a mobile app.

 

“You want to get data simply, and we can do that via REST and other APIs. But you also want the mobile application to be responsive to the context of what the developer or the end user wants to do,” Chandrasekaran said. 

 

As an example, he shared how a fast food company is working with Denodo to provide an interactive menu in their mobile app. “Say you want items with only so many calories or sugar or whatever. The mobile app lets the user set those levels, and we launch a query. This creates a data service that limits the results to just those items,” he added.

 

Data virtualization is also powering some major IoT projects, Shankar added. A global heavy equipment manufacturer receives sensor (IoT) data from their machines, which are loaded into a Hadoop system. Denodo combines the IoT data from the Hadoop with other data about the equipment, owners, parts, etc., which reside in back office systems to enable predictive analytics for business users to proactively recommend maintenance and parts replacement to their customers.  Notably, parts data gets combined with service records for service recommendations.

 

Security 
Many companies use data virtualization as a central area to store security policies. They enable SSO (single sign-on) through it.

“Autodesk uses data virtualization along with LDAP to secure the access to underlying systems and enforce a single point of security.” Shankar noted.

 

Overall, an approach that marries security with data virtualization can enhance protection for many firms. “This proves a great way to keep the security policies in one place to ensure the data consumers don’t directly access the underlying apps, thus providing a strong security. This has been specifically used in compliance scenarios,” he added.

Data Virtualization Lowers Cost, Boosts Agility for Data Sharing, Integration.

 

Taken in total, all these varied use cases for data virtualization are attracting attention – especially from enterprise architects and integration professionals. “We have helped customers reduce their integration costs by as much as 80 percent, and to use data as a strategic tool to better run their businesses,” Shankar said.

 

Just this month, the company announced that Digital Realty Trust, a provider of data center and interconnect solutions is using Denodo to modernize data sharing operations among crucial business-critical system – with tactical and strategic projects.

 

In specific, on a tactical basis, Digital Realty is replacing ETL with data virtualization to make data sharing more agile and efficient. To deliver on a broader strategic vision, the company is also basing a new data services fabric using data virtualization. 

“Denodo’s ability to rapidly deliver integrations across business systems, and then service those standardized views for internal applications and business users is significant,” said Paul Balas, Digital Realty’s vice president of business intelligence, said in a statement.

 

For Balas, these benefits come from Denodo’s data abstraction layer, which supports easier, highly-reliant and even real-time data sharing across multiple, distributed, heterogeneous repositories. The result: Integrated business data for analytics, apps, BI and processes, he explained.   Traditional tools, such as ETL and SOA are no longer capable of handling the needs of today’s digital environment as each requires too much specialized knowledge and is brittle to change.

 

“Data virtualization is a more agile data integration style that presents relevant, interrelated data, in real-time and in a consistent format irrespective of underlying database systems, structures and storage,” Shankar added.  [We] look forward to providing [Digital Realty] with a more agile data integration style that can deliver business-critical information in a timely fashion across the enterprise.”




back