Reasons not to use Import and Export and to get a direct Connection

From
Jump to: navigation, search

Reasons not to use CSV Export - Import and to get an appropriate API style connection


  1. Security of data
  • Exporting data means it will be left on a server unsecured for a period of time this means during a network breach that data is there to be stolen. Equally in CSV format it is readable by anyone that has access
  1. Identifying Failures
  • When pulling data from an API it is clear in the UI that a failure has occurred and can be reported. *Picking up a file from a share means the same file could be imported for weeks before anyone notices
  1. Troubleshooting
  • When a failure occurs there are 3 systems involved, each having system resources and permissions to be investigated to confirm where and what is causing the failure.
  1. Frequency
  • Pulling by API means the time / data / frequency of collecting the data is determined by the calling system not by the exporting system (ie. once a day / week / month)
  1. Cost
  • Exporting to storage means paying for that data transfer and storage, importing is another cost for transferring the data.
  • Via API it is one pull / no storage
  1. Data accuracy
  • Exporting the data at 01:00 and importing it at 11:00 means it is 10 hours old

API connection at 11:00 means it is 0 minutes old.

  1. Management
  • Managing an export process and a separate import process is managing 2 processes
  • Managing an API pull is 1 process
  1. Data compatibility
  • Exports to CSV can be limited in format and can be corrupted, which may not necessarily be identified promptly in the import process, which means another whole day before another export is available. API import breaking half way through is more likely to be flagged, and can be monitored more accurately and immediately, and retried without asking other personnel to carry out a manual export.

While it may take time to get an API connection built, in the long run it can help streamline the data import process.

This is just a rough outline, and there are likely many exceptions, but I needed this recorded.