Data Exchange

Data Exchange¶

Data Exchange is a way of exchanging data between other MIKE INFO installations, and not with other software; e.g. from a specific (e.g. national) MIKE INFO installation to another installation (e.g. MIKE INFO master).

The Data Exchange functionality allows creation of an exchange file containing the data for selected modified entities (time series, documents, …), which can be shared with another installation (or other database at the same installation). The created file can then be loaded into the target database using the “Load data” option of the Data Exchange functionality.

The Data Exchange dialog contains a number of aids to verify or validate whether an entity is supposed to be Dumped or Loaded.

Besides the Entity Path and Type the dialog has columns representing

Key properties - Selected values for each type of entity describing it.
Entity Preview - a button allowing the user a peek of the entity with additional details before it is being selected for Load or Dump.
Validation – results of the DataQC validation script(s), see Creating DataQC scripts.

Furthermore, the user can Select/Deselect individual entities from the list to be Dumped or Loaded.

Below is a walk-through of the details of the Data Exchange dialog and functionality.

Figure 1 Data Exchange dialog

Dump or Load?¶

Once logged in the user can start Data Exchange from the Settings menu.

Figure 2 Starting Data Exchange in MIKE INFO

During startup of Data Exchange in MIKE INFO the user will get a warning of no spreadsheet is configured, see Setting up DataQC configuration.

Figure 3 Warning if DataQC is not configured

The Data Exchange dialog can be used for Dumping data from the database to disk or Load of data from disk to the database. Dumping entities means exporting data for selected entities from the database into an exchange file, while Load performs an import of entities from an exchange file into the database.

Figure 4 Data Exchange dialog

Dump¶

Selecting the entities for Dump is done based on a date criteria:

All
Since last dump (as noted in the database)
Since a user specified date

When a new Dump option is selected, the list will be filled with entities that are modified since the chosen date (or All if that is chosen). The selection of entities is furthermore controlled by the entity types listed in the INFO-CONFIG.txt file under the Exchange section, see Figure 5.

Figure 5 Exchange configuration in INFO-CONFIG-txt

The list in the dialog is populated with information about the entities. Besides the name (path) and type of the entity also up to five key properties is shown in columns DBprop1, DBprop2,…,DBprop5 (see more in Key Properties) and the list will also have “Preview” buttons and “Validation” fields for each entity. This will allow the user to inspect the details of an entity before Dump as well as inspect the Validation messages produced, see more in Key Properties, Validation and Entity Preview.

Once inspected and validated the user selects entities to be Dumped, specifies name and location of the exchange file and presses “Start”. The system will respond with a confirmation dialog (see Figure 6), summarizing the number of entities, number and selected entities in total and by type. From there Dump can be cancelled or completed.

Figure 6 Dump confirmation dialog

Dump is completed with each dumped entity being flagged in the list and with dialog giving a link to an Excel file with an overview of the exported entities, see Figure 7.

Figure 7 Dumping complete

Note that Dump of FeatureClass and Temporal Raster will include details of the entity and modified features and raster time steps, i.e. it is a partial export of data. So if a FeatureClass only changes name, no features will be Dumped. This is important to know when doing a Load, as loading an empty feature class and specifying “Overwrite” will then delete all features in the FeatureClass in the database.

Load¶

Load will allow import of Dumped entities into the database.

The user will select Load and select an exchange file to load. Upon selection, the list is filled with information of the entities, similar to that used by Dump. The list will for each entity in the exchange file be filled with details – name (path), type, up to five key properties (EXprop1, EXporp2,…Exprop5), Preview button and Validation field.

But in addition to displaying details of the entity in the exchange file, the dialog may also present details from the database, if an entity with the same path and type exists in the database. The built-in validation will then also compare the key properties and mark if they are different.

Likewise, the Preview offers display of the entity in the exchange file – as well as of an eventual version in the database, see more in Entity Preview.

Loading will insert new entities in the database. If a version of an entity already exists in the database, Load will normally replace the entity with the version from the exchange file. However, for complex entities like FeatureClass and Raster, where an entity consists of some common data and then a list of details (features or temporal raster steps) a decision must be made for how data should be imported. The tick-box “Overwrite” does that. Please note the same choice is made for all selected entities, so if different entities require different behavior it is advised to do multiple Dumps and/or re-run the Load multiple times for different entities. Table 1 will explain how “Overwrite” is interpreted for different type of entity.

Table 1 "Overwrite" behaviour during Load

Overwrite Entity type	Unticked	Ticked
Time series	Properties of the time series will be updated and time steps from the exchange file will replace time steps in the database for the period of data in the exchange file.	Same as unticked
FeatureClass	Properties of the feature class will be updated, including the list of attributes, and features from the exchange file will be added	As unticked, but all existing features will be replaced by features in the exchange file
Raster	Raster properties are updated. Regular rasters have all rasterbands replaced. For Temporal rasters existing timesteps not present in the exchange file will be kept	As unticked, but for temporal rasters all time steps will be replaced with data from the exchange file
Document	Existing entity will be replaced with data from the exchange file	Same as unticked
Script Storage	Existing entity will be replaced with data from the exchange file	Same as unticked
Spreadsheet	Existing entity will be replaced with data from the exchange file	Same as unticked

When all entities have been validated and/or Previewed, the user selects entities to import and presses Start. The Load finish with updating the “Loaded” column and displays a dialog with a link to an Excel files with a log of the Load process, see Figure 8.

Figure 8 Load finished

Key Properties¶

Hovering the mouse over a field with a property value will show what the value represents, see Figure 9 where the date in DBProp1 for a Script Storage means “Last update”, but for a Time series it will mean “First Timestep”

Figure 9 Example of fly-by text describing a cell

The possible properties for different entity types are listed in Table 2.

Table 2 Key properties by entity type

Entity type	DBProp1	DBProp2	DBProp3	DBProp4	DBProp5
Time series	First timestep	Last timestep	Number of timesteps	Variable	Unit
FeatureClass	Geometry type	Attribute count	Feature count
Raster	Width	Height	Cooridnate system	Number of Timesteps
Spreadsheet	Number of worksheets
Document	Author	Language	Copyright	Is public?
Script Storage	Last Update	Number of scripts

During Load five more columns are shown with the similar properties of the entities in the exchange file. If a version of the entity already exists in the database, the dialog will show the same properties for this. The built-in validation of the entities will compare the key properties and if different, show this in the Validation column and the Entity Preview will show the both entities

Validation¶

Validation is a system which allows simple out-of-the-box validation of key properties during Load of data. It will basically for entities already present in the database compare the key properties and if different mark the entity as DIFFERENT.

For both Dump and Load this can be combined with a more advanced Validation. This requires the user (or administrator) to develop validation scripts and configure these in a spreadsheet, see Quality Control.

Once configured each validation added to the list is matches against the configuration, and a series of scripts is being executed giving a combined validation result.

The Data Exchange Validation makes us of the so-called DataQC component of MIKE INFO.

The file “INFO-Config.txt” in the MIKE INFO installation folder (usually “C:\Program Files (x86)\DHI\2019\MIKE OPERATIONS\”) defines if a spreadsheet is configured in the database and should be used with DataQC during Data Exchange.

Figure 10 MIKE INFO configuration of DataQC spreadsheet

DataQC validation works in both directions:

Dumping will select modified entities based on a date criteria
- All
- Since last dump (as noted in the database)
- Since a user specified date
  
  When a new Dump option is selected, the list will be filled with entities that are modified since the chosen date.
  
  As the entities are entered into the list, they are matched against the validation spreadsheet rules and if match between rule Entity Path specification and the entity the sequence of scripts are called for the entity.
  
  Gradually the Validation cell will be filled and colored with the results of the validation. Possible results are:
one or more rules were validated and at least one returned ERROR
one or more rules were validated and at least one returned WARNING (and none flagged an error)
one or more rules were validated and validation passed OK
this reflects either that no rules were found to validate the entity or that no result was filled by the script(s)

The script may supplement the status with messages indicating the validation performed. These messages are available by clicking on the validation cell, see Figure 11.
Loading of data from an exchange file works in a similar fashion with respect to DataQC validation. The entities found in the exchange file are added to the grid and matched against the validation configuration spreadsheet. If matched, the scripts are called for the entity in the exchange file and the validation cell is filled.

Possible results are
one or more rules were validated and at least one returned ERROR
one or more rules were validated and at least one returned WARNING (and none flagged an error0029
no rules were found or their validation passed, but the key properties in the form are different
no rules were found or their validation passed, but the entity does not exist in the database
properties matched and rules validated OK
this reflects either that no rules were found to validate the entity or that no result was filled by the script(s)

Figure 11 Display of validation detail

See Quality Control for more detail on how to Data QC functions, how to make validation scripts and how to configure Validation.

Entity Preview¶

The Preview button for each entry in the list allows the user to have a peek of the entity and its details directly from the Data Exchange dialog.

In Dump mode the Preview will display details from the database. In Load mode it will show entities from the exchange file, but if a version of the entity (same path and type) already exists in the database, Preview will offer switching between the two. This will enable a possibility for a visual inspection and comparison of the details of an entity.

Figure 12 shows an example of a Preview of a time series in the database. The title of the dialog reflects the name of the entity. Two arrows allow moving up/down the list of entities for quick access to other entities without closing the dialog. If both database and exchange file version exist during Load, the user may also be able to switch between database version and exchange file version of the entity displayed. Different details may be available. Notice how this view show four tabs providing access to a plot of the time series, tabular view of the time series, Metadata, and Changelog.

Figure 12 Preview of a timeseries

As the entities are different in nature, the preview will look slightly different for different entity types as well. Figure 13 to Figure 18 show examples of the displays for the different data types.

Figure 13 Previews of timeseries

Figure 14 Previews of Document

Figure 15Previews of Script Storage

Figure 16 Previews of Spatial Layer

Figure 17 Previews of Spatial Layer - Raster

Figure 18 Previews of Spreadsheet

Using Data Exchange between installations¶

Data Exchange can be used to align content in databases between different installations, e.g. a master installation with a number of client installations each controlling their own sets of data.

Assuming all installations start out with the same database the flow of Data Exchange could be like this:

The clients Dump data into an exchange file of the entities they have modified, see Dump. The exchange file is transferred (e.g. emailed) to the master administrator and no new data is entered into the client databases until step 4.
The administrator Loads the exchange file data into the master database, using the Key properties, Validation and Preview as guide to ensure the data is OK, see Load.
The master Dumps all changed entities into an exchange file which is transferred to the clients, see Dump.
The clients Loads the master exchange file, using the Key properties, Validation and Preview as guide to ensure the data is OK (which I should be), see Load.
All databases are now synchronized with changes from all other installations.

A similar procedure can also be used between two client installations or two databases in the same installation.