I was discussing the challenges of ensuring that data is protected and proven to to have integrity and that as the data moves through systems that there isn’t data loss. This sort of thing starts at the simplest level with data validation and with the most advanced and greatest investment you have some end to end reconciliation framework.

Obviously this thinking doesn’t work in every environment, for example complex event processing (CEP) your going to just accept the data coming through and if it’s incomplete or data has been lost along the way you just accept it as it is – these conditions will create outliers which will get smoothed out in trends. It is possible you will have created he gaps by dropping data slow to arrive. But for the majority of your run of the mill solutions such as accounting, HR and so on the thinking stands up.

To communicate the idea effectively to senior management on the risks of just focusing on functional delivery and whether there is maturity in the delivery capability we hit on the idea of using a variant of Maslow’s triangle of needs – something I think everyone gets. You can see our representation here:

The interesting thing is that you could look at the triangle and suggest that typically the more pressed a project is on factors such as volume of functionality, cost and/or time the more likely a project will remain at the bottom of the triangle. But as the width of the triangle at the point of the capabilities realised also reflects on the operational costs. So if you’re at the bottom of the triangle then you’re likely to incur more costs dealing with data issues as the means to detect and then resolve are a lot more restricted.

With frameworks such as those in Oracle’s SOA Suite and AIA it should make it easier to move up atleast part of the triangle, although full end to end reconciliation is more likely to demand more data centric tools, as you probably want to perform by doing batch like assessments.