MyRapidi
MyRapidi

Search our Wiki

Source Control

  • If you want to transfer only the records that have been changed since the last transfer, you can make use of the feature, Source Control.
  • The Source Control field is a field - a number in succession - that must be updated whenever a record is inserted and/or updated. The field is created in the table from where the changed data is to be transferred.
  • If the table that you want to apply Source Control on already contains a field, which is a number in succession and is updated whenever records are either inserted or modified, no additional programming is required. This could be a field of type Timestamp in MS-SQL-based systems.
  • If the table does not contain such a field with the above-mentioned capabilities, you need to make additional programming in the table.
    • First, you need to create a new field in the table. This field can be of the type Timestamp or BigInteger or Integer.
    • If you use Timestamp as a Source Control field nothing further is required.
    • But if you use an Integer as a Source Control field, you need to add code, that will update the new field whenever changes are made to records. For example, applying Source Control as an integer in a Microsoft Dynamics NAV table requires code, that will update the new field whenever changes are made to records.

How Source Control works:

  • When you run a Transfer, where the table you transfer data from uses Source Control, Rapidi will first refer to the Runtime Information Register (RTI) in order to see if the Transfer has been run with Source Control before (within the Runtime Information Register, Rapidi keeps track on the last used Source Control number/timestamp).
  • If the Transfer has not been run with Source Control before and no changes have been made since Source Control has been enabled, Rapidi will not transfer any records before changes have been made (this is only true for Integer/BigInteger fields and where the initial value is 0).
  • If the Transfer has been run with Source Control before (meaning if changes have been made), it searches for the last used Source Control number/timestamp and uses this as a filter, when reading in the table data is transferred from. All the records that contain a higher/newer Source Control number/timestamp than this filter, will be read and transferred. After a successful transfer, Rapidi stores the new last used Source Control number/timestamp in the Runtime Information Register.

Important:

When using Source Control, it is not possible to perform "Delete" in the Transfer. The reason for this is to prevent having all non-modified records in the destination Data Source deleted. You can use Actions in order to delete data.

Source Control on a table that already contains a Source Control field

  • In the Transfer, where you have chosen the requested table as source, you choose "Source Control".
  • In the field "Source Control Field", you can specify the field that should act as a Source Control.

Source Control on a table that does not already contain a Source Control field

  • In the table that should use Source Control, you create a "Source Control" field. The Data Type should be either a Timestamp or a Big Integer/Integer. If you use a BigInteger, additional programming might be required so that each time a record is either modified or inserted it is assigned the next Source Control number.
  • Since the design has been changed by creating a new field, you must re-run 'read design'. You can run Read Design on the Connection page.
  • In the Transfer, where you have chosen the requested table as source, you choose "Source Control".
  • In the field "Source Control Field", you can specify the field that should act as a Source Control.

Source Control Field:

Specify the field that should act as a Source Control.

Destination Control Field:

This feature will update a field in the Destination database/table so that this field can be used as a "Source Control" field (be used to identify which records were updated). You can use this feature if you have some process that needs to handle each updated or added record in the Destination table. This could for example be the case if you transfer Sales Orders into the Destination and want to print these Sales Orders or notify someone that a new order is available.

The "Destination Control Field" now supports many different field types. The field selected can be any of the following types: Integer, BigInteger, Text, Date, Time and DateTime (before we only supported Integer and BigInteger types). When the Transfer runs, we automatically detect the field type of the selected field, and depending on this field type we generate a value to be used to update this field in the destination:

  • Date, Time, DateTime and Text fields: We insert the current datetime in the destination field.
  • Integer, BigInteger: We find the Last record (record with the highest value in the Destination Control Field) and increment this value by 1.

We ensure that the value of the "Destination Control Field" itself, is not taken into consideration when we compare the existing record in the destination with the record coming from the source system. In this way, only records that are really different will be updated and also marked as updated in the Destination.

We have some limitations with Integer and BigInteger field type support as not all systems support getting the highest value across all records in the system. Please contact us or use DateTime or Text fields instead if this is not working for you.

Please note that you should upgrade both the central service and the RapidiConnectors to version 3.2.92i or later to use the "Destination Control Field" feature.

Commit Per:

  • In situations, where you have to transfer a large amount of data or where slow and unstable connections are in use, the field "Commit Per" can be very useful.
  • If you enter a number here, Rapidi will do a commit (including storing RTI numbers) each time it has transferred this many records, then continue with the next records.
  • Commit Per should normally only be used in combination with the Source Control feature. In this way you ensure progress in the data transfer if an error happens after a number of records have been transferred as the transfer can resume at the last stored RTI source control value.
  • You should not use "Commit Per" when reading from Salesforce.com using the DBSourceControl feature as we cannot then guarantee that all records are transferred in case of an error.

Transferred Field:

  • The field "Transferred Field" makes you able to write a timestamp (date/time, text, or boolean true value) back into the source table on each transferred record.
  • The field must contain the name of the field in the Source Table that should be updated when a record is read.

Store new ID Field:

  • A facility to store the ID of a newly created record.
  • The ID is stored in a field in the source table (the field indicated in "Store New ID Field").
  • The link (key fields) needs to be set up in a special way:
    • the left side must be a unique key field to the source table (normally the primary key)
    • the right side must be the name of the ID field to get from the destination table (from the newly created record).
  • Disable Dest. Lookup must be checked.
  • Only Add must be enabled for this transfer - no update or Delete !

Source Mirror:

Allows you to specify the Connection that has information on how to connect to the Mirror database.

Please find more information on the Mirror Technology here

Read more:

HowTo Implement Source Control in MS Dynamics NAV

Source Control with Salesforce.com