Skip to main content

Data Modeling

Defining data model fields, handling CRUD operations, and managing custom fields

Updated this week

This article explains how to define data model fields, handle CRUD operations, and manage custom fields.

Data Model Fields

Typically, the external API you are integrating with will name fields in snake_case. By convention, iPaaS.com field names should be in PascalCase for consistency and readability. You can support both naming conventions by naming your model property in PascalCase, then adding a JsonProperty tag in your API’s format. E.g.,:

[JsonProperty("first_name")]

public string FirstName { get; set; }

When defining a data model, include all available fields from the destination endpoint, not just the fields that you expect to use.

Sometimes it may be necessary to add fields to your data model that control behavior and do not map directly to an existing endpoint field. For example, if your external system has separate endpoints for billing addresses and shipping addresses, you may want to add a field that users can map to that indicates the address type. Generally, these control fields should be tagged [JsonIgnore] so that they are not sent to the external system.

Your external data structure might include an embedded collection. When possible, add Properties with a getter and setter that will assign the embedded values. For example, you might have an API endpoint that uses this format:
{
"sku": "ADM-TL2",
"all_prices": {
"msrp": 399.99,
"price": 379.99, "sale\_price": 339.98
}
}

You can create your data model to present a single set of fields that are accessible on the product model:

{
[JsonProperty("sku")]
public string Sku {get; set;}
[JsonProperty("all_prices_msrp")]
public decimal? AllPrices_MSRP { get { return (decimal?)GetAllPriceField("msrp"); } set { SetAllPriceField("msrp", value); } }
[JsonProperty("all_prices_price")]
public decimal? AllPrices_Price { get { return (decimal?)GetAllPriceField("price"); } set { SetAllPriceField("price", value); }
}
[JsonProperty("all_prices")]
public AllPriceModel AllPrices { get; set; }
//Define GetAllPriceField and SetAllPriceField methods to get and set values in the AllPrices model
}

Name your fields to include a prefix indicating the parent record name. E.g., in the example above, we used AllPrices_MSRP and AllPrices_Price. This will make it clear to users that the fields are related and contained within a child object.

For a one-to-many or array subcollection, this will not work. In that case, you may need to define a conversion function that can build and populate the required structure (while considering id requirements).

If you expose child fields using the method shown above, you should also expose the full model as well for advanced users who are familiar with the data structure.

Data Model CRUD Operations

iPaaS.com allows users to define collision handling preferences on a per mapping collection basis for mappings FROM iPaaS.com (mappings TO iPaaS.com will automatically have collision handling built in). Whether or not you support this, and if so, which fields you do support, will be a design decision based on the capabilities of the external API.

Some considerations that may affect your decision are:

  • As much as possible, honor the collision handling preferences specified in the mapping collection. It provides a better user experience and, in most cases, is easier to implement, since much of the work of handling the collision is included in the platform.

  • Any field that has a uniqueness requirement in the external system is a candidate for collision handling. This might include things like product SKU, customer email address, location name, etc.

  • Collision handling requires detectable error messages from the external system. For example, if an email address is rejected as a duplicate, the error message needs to indicate this in a way that you can parse.

  • Collision handling also requires the ability to return the existing data from the external system. This means that your external system must either be searchable on the collision field, or the collision error must indicate the id of the existing data. For example, if you collide on an existing customer email address, you will need to return the existing customer’s id. So that id must be either included in the error message that was returned to you by the external system, or the external system must have the ability to search by email address so you can find the id yourself.

  • Your external API may allow you to search on the duplicate candidate prior to your create/update call. However, barring any special considerations, it is generally better to initiate your create/update call without checking and handle the collision response, rather than trying to anticipate any collisions by checking uniqueness first. This will cut down on the total number of API calls you make.

  • There may be times when you want to handle collisions internally without resorting to the collision handling system. For example, if you are integrating with an email marketing system, you might not need the data you send to be tied to an iPaaS.com customer record, only to the customer’s email address. In that case, you might detect an email collision and immediately retry the call as an update rather than a create. If you do implement handling like this, ensure it is well documented.

  • Be cognizant of case-sensitivity. Some systems might enforce SKU uniqueness in a case-sensitive way, while enforcing email uniqueness in a case-insensitive way.

  • Full support for collision handling includes linking child data when possible. See the collision handling documentation for more details on this.

  • Sometimes a single operation may require multiple API calls. For example, a call to ModelCreateAsync for a product might require calls to product, inventory, variant, and price endpoints. In this case, ensure that you properly handle partially successful transfers. Throwing an error from a secondary API call could result in unlinked data from the primary API call. Instead, you should catch errors on the secondary calls, create a new ExpectedException, and attach any successfully created data to it. The following shows how to catch an exception on an inventory call while returning the successful product portion:
    var productResponse = await Product.Create();
    try
    {
    var inventoryResponse = Product.Inventory.Create();
    productResponse.Inventory = inventoryResponse;
    }
    catch(Exception ex)
    {
    throw new ExpectedException(ex.Message) { SavedExternalData = productResponse };
    }

Depending on the API calls involved and the data requirements, you may choose to delay throwing the ExpectedException until all secondary API calls are complete.

In some scenarios, you may want to delete the partially created record.

  • If your integration uses GraphQL, be cognizant of newlines and other characters that must be escaped. Embedded newlines are not supported in GraphQL and must be escaped. Either use the GraphQL for .Net package, which will handle this for you, or ensure that you are correctly escaping your newlines.

  • Typically, a POST or PUT call will return the updated data, but this is not always the case. It may just return the id of the record that is created, or may return nothing on updates. Our translation service expects an object that it can call GetPrimaryId on and that it can navigate to gather primary ids for children as well. You may choose to make a GET call from the external system to retrieve the full created record, but this may not be necessary. If you can use the ids returned to build a suitable object, then just return that. If you do make a GET call, ensure that you update the Spaceport_SourceId for any child objects.

  • If a ModelGetAsync call does not find any data in the external system, this should generally NOT be treated as an error. This likely indicates that we received a hook for deleted or inaccessible data. For example, if a user in an external system creates a customer, realizes they made a mistake, and immediately deletes it, we will get a customer/created hook for a customer record that no longer exists. In that case, we do not want to throw an error and retry the call later; the record will never exist. There may be cases where you do want to throw exceptions for missing data, but in general, you should not.

  • For data that is not specific to creating a given record, ensure that the data does not exist before creating it. This is often seen when sending product options and product option values from iPaaS.com. Many systems store them as universal records shared by all products, rather than a child record of a specific product. In that case, always check that the product option and/or product option value does not already exist before creating it. On the iPaaS.com side, ensure that your external id includes a product-specific segment to avoid duplicate external ids on shared records.

  • The PollRequest method accepts a string filter. Provide support for this feature if your external system allows additional filters on polling calls. Document the requirements and support available for your filter so that users know how to structure the filter properly. You may also use the filter to pass parameters to the integration. For example, if your transaction polling event can poll both orders and invoices, you may want to let users add a flag, such as “invoiceOnly=true” to the filter to indicate to the integration that only the invoice polling should be completed.

Custom Fields

iPaaS.com internal Custom Fields should use the most specific data type that fits the data being stored. For example, if you have NumberOfVisitors as a custom field on an admission item, it would be more appropriate to set this as type Integer rather than String, Long, Double, etc.

iPaaS.com internal Custom Fields are stored as a string, but may be parsed into the appropriate data type before they are saved. For example, if you send “1.00” to an integer custom field, it will be stored as “1”. The exception to this is datetime fields, which are validated but otherwise stored and returned as-is.

External Custom Field names will need to be in the format required by the destination system. This will supersede any iPaaS.com field naming conventions. E.g., Your external system may define fields in snake_case, so you would need to name your custom fields in snake_case, rather than the iPaaS.com-preferred casing.

Be aware of data types on custom fields in your external system. The SetValueCustomField method takes an object as the value. It is up to your integration to determine how to handle that type. If you cannot determine the type in the external system, you can use the data type defined in your custom field record in iPaaS. For example, if the external system allows custom fields to be defined as booleans, it is imperative that you send a boolean in your request (and not a string). You could require that users set the data type in iPaaS.com, then query that custom field record from SetValueCustomField. If you use the external custom field’s datatype, ensure that you include that requirement in your documentation.

Did this answer your question?