Data Warehouses Sociotechnical Factors

Data Warehouses Sociotechnical Factors

Sociotechnical Factors are significantly new for most IT organizations for warehousing project failures.

In recent years it has been found that the most common set of factors contributing to data warehousing project failure are not design factors or technology factors or procedural factors, but sociotechnical factors: people and politics. The sociotechnical factors are largely new; this area was pretty much submerged in classical OLTP design. People and politics were less of an issue, or no issue at all. The target user community held little organizational power, was rarely seriously consulted during design, and could in most cases be compelled to use the system once it was deployed (though there are those interestingly complex unionized situations). And the project’s sponsor was almost invariably after a rock-solid, measurable business objective: process or task routinization, cost containment, workforce reduction.

The most painful, indirect and ultimately revelatory discussions with the clients in recent years are centered around the politics of data warehousing. Sometimes these conversations begin under the heading of "engaging with the business unit"; other times they are framed by the question "How do we cost-justify a data warehouse?" or "How do we get management buy-in on a data warehouse?" Infrequently, they are frank enough that other, more central and more overtly political topics surface: how do we get the business units’ IT organizations (or end-user communities) to trust us?

Politics In Data Warehousing

Data warehousing projects are always potentially political because:

• they cross organizational treaty lines
• they change both the terms of data ownership and data access, and expose the often-checkered history of data management in the IT organization
• they affect the work practices of highly autonomous and powerful user communities in the firm.

Any sensible data warehouse design is a part of a larger architectural model designed to deliver data from the points of capture (inside or outside the firm) to the points of use, probably transforming the data elements in the process. A warehouse, in other words, is the key data consolidation and pumping station in a complex data distribution system that begins with the firm’s production applications and external data syndicates, wholesalers and enrichers of various sorts, and ends on the intelligent desktops of managers, analysts, customer care personnel and the like. That network always crosses treaty lines: invisible boundaries within the firm that mark both "turf" and "domains of control".
Some of these treaty lines are functional (with the inevitable tensions that are an in-built feature of the functional organization as an organizational form), but the most insidious treaty lines are almost never functional. Consider, for example, the near-invisible treaty line that is drawn across the backplane of every PC in the modern corporation. Much like the residence jack in the American telephone system, the desktop treaty line marks a boundary between the "provider" and the "consumer" and, just as we are free to choose our telephones, PC users feel themselves imperatively to be in rightful possession of personal computers, the tool configuration of which is a private affair. When a warehousing project mandates toolsets or impinges on the desktop in other ways, the treaty line is crossed, and warning bells often ring out.

If there is any rule that does apply across organizations regardless of their market focus or structure, it is this: power accrues to those who:

• gather data
• control access to that data.

The irony in data warehouse projects is that, all too often, these laws are working against the very organization they used to work for: the corporate IS organization. As often as not, the data we can’t get for a data warehouse is the data controlled by semi-autonomous divisional or departmental IS functions that came into being because the historically stingy policies of the c orporate IS organization with respect to data access hamstrung a division, department or business unit so badly that they built their own IS function to regain the autonomy they needed for marketplace effectiveness. The dangers of centralized data control constitute, at some fundamental level, the raison d’etre of too many distributed IS organizations, and as a result their willingness to collaborate with the adversary is minimal at best.

But the real political problem with data warehousing is not the loss of data ownership that such projects imply, for every organization asked to contribute to the warehouse, a loss of control over access to the raw data itself, something frightening for any group with:

• dirty data
• ambiguous data
• unflattering data

which is pretty much every commercial organization operating today. In as much as divisions, departments and business units have gently cooked their dirty, ambiguous and unflattering data for years in the interest of keeping things clear and simple, these organizations are understandably leery about exposing the uncooked data to inspection by other groups that may have a vested interest or historical reason to point out the data quality issues.

One of the things knowledge workers do, early and often, in a mostly unscrutinized way, is make decisions. Small and large, important and inconsequential, decisions get made every day about all kinds of things. And the plain fact seems to be that most of these decisions are data-free or data-poor, made based on "experience" or "gut feel" or some other intangible, unmeasurable quality that is decidely human, decidely part of these magic professional disciplines and decidely not something a computer can frame, direct or do.

The work practice of decision-making has been done historically outside the IT infrastructure of the firm. Data warehousing projects threaten this long-standing practice. And they create, in knowledge workers, what Thorsten Veblen called, in another context,"the conscious withdrawal of efficiency": passive-aggressive behavior on the part of knowledge worker communities that includes:

• an unwillingness to participate in requirements gathering, schema design activities, and pilots
• failure to use deployed warehouses and marts
• endless, and pointless, micro-analysis of the "quality" or "real meaning" of the data provided by warehouses and marts.

Information technology is, for better or for worse, social these days. The good old days of batch and online transaction processing systems design and deployment are gone; we buy those things now, from independent software vendors. The systems we have to build – decision support systems, computer-supported collaborative work environments, workflow systems, intranets, extranets, whathaveyou-nets – are all deeply and inextricably social applications of IT: computing applied to groups of people with power, status and a network of relationships.

That means, for better or for worse, that politics is an integral part of IT projects from here on in. Or out, depending on your perspective. And that, in turn, means IT professionals have no choice but to get into organizational politics, understand the forms, shapes and paths organizational politics takes, and become astute at navigating in a political environment. Not because politics is cool, or fun, but because politics is a feature of the landscape: the beast standing between us and the gate marked "successful project conclusion."