Homegrown versus commercial data integration tools Finding the right balance

SearchDataManagement.com

Chipworks' Mark Kendall knows that a single, comprehensive view of customer data can be a competitive differentiator. Getting Chipworks' many legacy systems and newer applications, like Salesforce.com, to communicate is the challenge.

"The biggest problem with legacy systems is data disconnect," said Kendall, manager of information systems at Chipworks, a semiconductor research firm based in Ottawa, Canada.

To get his many data sources and applications talking, Kendall turned to a mix of homegrown and off-the-shelf customer data integration tools, a not uncommon, though potentially limiting, approach.

"The customer data integration [CDI] vendor products are a lot more sophisticated than what most people can build at home," said Philip Russom, a senior manager at The Data Warehousing Institute. "But CDI has a long history of hand-coding and building it internally."

CDI, in fact, predates master data management (MDM)and service-oriented architectures (SOAs), two important features for any comprehensive data integration effort, according to Russom, who recently authored a report on CDI.

The result is that most homegrown CDI tools have little in the way of data quality and data governance. Definitions of "customer" abound, Russom said, negating many of the benefits of sharing customer data enterprise-wide in the first place.

"In a complicated company with a variety of products and services, your customer may actually be a customer in many different business units," he said. In financial services, for example, a customer may be unprofitable from a retail banking perspective but may be a heavy investor in the markets. If the two business units can't communicate that fact, customer service could suffer.

Another drawback of internally built CDI tools is their lack of bidirectional communication, Russom said. Many homegrown CDI tools collect customer data from across the enterprise and deposit it in an operational data store, where users can view customer data via a custom search function. But, once collected, the data is rarely pushed back out to the applications from where it came, Russom said, limiting its effectiveness.

One way to get bidirectional data communication flowing is to implement Web services, according to Russom: "Instead of that sales guy going to a search engine and poking around in the operational data store, imagine a service coming out of the operational data store that has the information cleaned up and presented just the way sales needs it right there in his CRM [customer relationship management] or SFA [sales force automation] application."

Unfortunately, building such comprehensive CDI tools is outside the reach of most IT organizations, Russom said. In those cases, he suggests that companies look to vendor tools.

"If people want some of the forward-looking requirements – in particular, bidirectional, master data management, SOA -- it would kill them to build that into their homegrown solutions," he said. "So the more you need these forward-looking requirements, the better off you would be buying a solution from a vendor."

Chipworks' Kendall notes, however, that buying commercial CDI software doesn't always get you the most for your money.

"Sometimes, with a vendor CDI tool, you end up with 'full-feature functionality' of which you're only using 5%, and you're paying for a lot for things you don't need," he said. "In cases like that, we look into custom development."

On the other hand, "we're not looking to reinvent the wheel," Kendall added, which has led Chipworks to its hybrid approach -- combining both homegrown and vendor CDI tools.

Chipworks uses Informatica's On Demand Data Loader Service, for instance, to integrate Salesforce.com data with its internal data sources, including SQL Server, but the company employs internally built integration tools for sharing customer data with other applications.

"We take a fairly detailed approach to it in terms of sourcing vendors and seeing what's available, and also examining in-house development," Kendall said. "Basically, the numbers tell us what we're going to do."

Read more....