Push data:

The first solution we had considered was pushing data from one layer to another when ready up to the data storage at the end. If performed very puristicly, the data extractors wouldn’t so much be extractors but API’s that are able to receive data from the appropriate sources. A list of advantages (green) and disadvantages (red) are listed in this section.

(Dis) advantages

A big advantage of using a puristic push-technique is the fact that you essentially lay the control of when to push data to the data extractors to the tool the data comes from. The stakeholders have reported that these systems are critical and must in now way be harmed by a system such as this one that is not.

By laying the control at the critical applications, they can decide when to push the data. They could wait for a moment in which they’re less busy and do it then. This is a very easy method to ensure the processes are not run at an inconvenient time.

The flipside of the coin is that this solution offers no possibility to decide when to receive data from what sources, troubling the data coupling process later on. This would essentially mean that all data should we coupled at database level and not programmatically.

As stated before, this solution is not possible for all application types. SVN for example offers possibilities to implement post-commit hooks but another system may not. This could seriously limit the possible data-providing systems we can work with.

Pull data:

(Dis) advantages

This strategy offers the possibility to have complete control over when to receive data from what sources. The advantage of this strategy is the fact that more data can be pre-coupled before persisted to the final database.

Another big advantage is the fact that unlike with the push-strategy a pull-strategy is possible with every application so long as they have a way to retrieve data from it such as an API or a log file. This means that the amount of possible applications that can connect with our application is much larger.

A disadvantage of the pull-strategy is the fact that deciding when to retrieve data from the critical systems is not at all trivial, sincethe data extractors some way to analyse whether or not the external critical system is quite busy at the moment. This could be overcome with some monitoring software however.