Overview
With data flowing from multiple systems with different transformations and different mapping rules, it is likely that from time to time that an attempt to duplicate data will occur. We call these events collisions. Typically, an external system will detect a collision and reject it. For example, if we try to send a customer to an external system but that customer’s email address is already in use, then the external system may reject that API call. The collision handling system helps us define a way to gracefully detect and recover from these collision events.
Collision Handling Methods
In iPaaS, we define four ways of handling a data collision.
Error: This is the simplest (and default) handling method. When the API call is rejected by the external system, simply throw that error message as an exception. The user will be responsible for seeing this error message and correcting the data accordingly.
Remap and Link: Link the new data to the existing data (by using the existing data’s external id) then fire a new update hook to re-run your mappings on the now-linked data. This is a useful option if you have different mapping behavior for adds and updates.
Update and Link: Link the new data to the existing data and convert the ModelCreateAsync call to a ModelUpdateAsync call. This is the most common form of data linking and should be the default for linking data that can be safely handled without user intervention.
Update and No Link: Find the id of the existing data, convert the ModelCreateAsync call to a ModelUpdateAsync call, but do not permanently link the data. This is useful if there are two sources of data in a single system that are both feeding into a single record in iPaaS. For example, if you have a system where the same email address has two customer accounts, such as one personal account and one account as an organizational buyer, you might want updates from both customers, but iPaaS won’t let you link to them both.
Users will specify which of the above methods they would like to use in their mapping collections:
Adding Collision Handling to Your Integration
As an integrator, you will not need to know the specifics of which handling method was chosen or how it will be handled in iPaaS.com. That work is all handled as part of the platform. Your integration will be responsible for only two things: 1) detecting that the attempted API call was a collision, and 2) if the user is using collision handling, finding and returning the existing data in the external system that the new data is colliding with.
The calls to ModelCreateAsync and ModelUpdateAsync include a CollisionHandlerSettings object that will indicate to you if the user opted for collision handling. This model only has one field that is currently in use: CollisionHandlerMethod. (There are other fields defined in the Settings object for future features that have not yet been implemented.) If the CollisionHandlerMethod field is populated with any value other than 0, then you will need to return the existing data to iPaaS.
To return existing collision data, simply populate the CollisionData field of the normal ResponseObject and return that. No other fields are required in this ResponseObject.
Example
Below is an example of a simple implementation for handling a collision on a customer email address. This code is from the ModelCreateAsync method.
We first check for a collision by looking for specific text in the error message that is returned from the external system. We also check that the user has opted into collision handling. We then search for the data that caused the collision and return that in our ResponseObject.
try
{
response = await apiWrapper.Customer_POST(cpCustomer);
}
catch(Exception ex)
{
if (ex.Message.Contains("This is a duplicate email address") && collisionHandlerSettings.CollisionHandlerMethod.HasValue
&& collisionHandlerSettings.CollisionHandlerMethod != 0)
{
var customerEmailFilter = CallWrapper.QuickEqualityFilter("EMAIL_ADRS_1", cpCustomer.EMAIL_ADRS_1);
var matchingCustomer = await apiWrapper.Customers_GET(customerEmailFilter);
var retValCollisionCustomer = new ResponseObject();
if (matchingCustomer != null && matchingCustomer.Count > 0)
retValCollisionCustomer.CollisionData = matchingCustomer[0]; //The above GET call returns a list. We only want one entry
else
throw ex; //If we didn't get a match for some reason, just throw the normal error
return retValCollisionCustomer;
}
}
Handling Secondary Data
If a collision occurs on a data object that has children with Spaceport_SourceId’s assigned, you may need to migrate the assignments to the object you are returning in CollisionData. By migrating this value, you allow iPaaS to correctly assign external ids to your child data.
Below is an expanded version of the previous example that migrates the Spaceport_SourceId to any address matches. Note that we use the field ADRS_1 as the match for demonstration purposes only. You will likely need more data to match on than this.
try
{
response = await apiWrapper.Customer_POST(cpCustomer);
}
catch(Exception ex)
{
if (ex.Message.Contains("This is a duplicate email address") && collisionHandlerSettings.CollisionHandlerMethod.HasValue
&& collisionHandlerSettings.CollisionHandlerMethod != 0)
{
var customerEmailFilter = CallWrapper.QuickEqualityFilter("EMAIL_ADRS_1", cpCustomer.EMAIL_ADRS_1);
var matchingCustomers = await apiWrapper.Customers_GET(customerEmailFilter);
if (matchingCustomers == null && matchingCustomers.Count == 0)
throw ex; //If we didn't get a match for some reason, just throw the normal error
var matchingCustomerTemp = matchingCustomers[0]; //The above GET call returns a list. We only want one entry
//Reload the customer with all secondary data so that we can match the addresses
var matchingCustomer = await apiWrapper.Customer_GET_Extended(matchingCustomerTemp.CUST_NO);
//To handle child entires, we need to match the addresses. If we have a matching address, we can assign the Spaceport_SourceId and let iPaaS link from there.
foreach (var adrs in matchingCustomer.Addresses)
{
var matchCandidateAddress = cpCustomer.Addresses.Find(x => x.ADRS_1 == adrs.ADRS_1);
if(matchCandidateAddress != null)
adrs.Spaceport_SourceId = matchCandidateAddress.Spaceport_SourceId;
}
var retValCollisionCustomer = new ResponseObject();
retValCollisionCustomer.CollisionData = matchingCustomer;
return retValCollisionCustomer;
}
}