Pushing or Pulling an Entire Dataset to or from iPaaS.com

Background

This document outlines a method for synchronizing large datasets—either by pushing data to iPaaS.com or pulling data from iPaaS.com—by leveraging the Postman Runner feature. This technique is particularly useful when seeding an entire product catalog, migrating historical orders, or handling datasets that exceed the standard limits of the manual sync interface.

Understanding the Standard iPaaS.com Sync Process

Typically, iPaaS.com processes data synchronization as follows:

Webhook Notification: For incoming data, a webhook notifies iPaaS.com about new or updated records in the source system.
Queue Entry: The webhook triggers the creation of an entry in a processing queue.
Data Retrieval & Normalization: As the entry reaches the top of the queue, iPaaS.com makes a GET request to the source system API to retrieve the full record details. Based on the defined scope (e.g., CREATE, UPDATE, DELETE) and mapping configurations, the data is normalized into the iPaaS.com data models.
Outbound Processing: Once the data is created or updated within iPaaS.com, any connected systems listening for changes to that data type will have the corresponding record queued for outbound synchronization, following a similar process.

Polling Alternative: Systems using polling follow a similar pattern; the polling process identifies new or updated record IDs, which are then added to the processing queue.

Manual Sync: The manual sync page enables users to add records directly to the queue. Using Debug during a manual sync prioritizes the record in the queue and provides additional logging. However, the manual sync page typically has a limit (e.g., 3,000 IDs).

This Postman Runner method essentially automates the process of manually adding many records to the iPaaS.com queue via API calls, bypassing the standard interface limitations.

Before You Begin: Alternative Methods

Consider these alternatives before using the Postman Runner method, as they might be simpler for your specific needs:

Polling with /poll Scope: If you need to import data TO iPaaS.com and the integration uses polling (not all do), you can often trigger a full data pull using the manual sync feature with a /poll scope.

NOTE: Many polling integrations store a "last polled date." To ensure all data is captured, you may need to clear this date in the integration settings before initiating the /poll manual sync.

Initialization Feature: Check if the specific data type and direction support Initialization within iPaaS.com, which is designed for bulk data loading.

Step-by-Step Guide to Using Postman Runner for Bulk Sync

Step 1: Prepare Your Data

To move data TO iPaaS.com (importing into iPaaS.com):

Obtain a complete list of the unique identifiers (IDs) for the records you want to import from the source system. You may receive this information via an export feature or by querying the source system's API.

NOTE: Ensure these IDs are in the exact external ID format that iPaaS.com expects for this specific integration, as documented in the subscription's documentation.

To move data FROM iPaaS.com (exporting from iPaaS.com):

This is only necessary if you need to trigger a sync from iPaaS.com to a single specific target system, bypassing the standard outbound flow that follows an import. This updates any systems listening. However, you can turn off the FROM webhooks from other systems and have the same effect.

NOTE: Remember that TO iPaaS.com hooks will then result in FROM iPaaS.com hooks, so this is only relevant when you only need to sync from iPaaS.com to one system.

Use the iPaaS.com API to retrieve the internal iPaaS.com record IDs you need to push. You will likely need to parse these IDs from the JSON API response.

If the data type and direction support initialization, you can use that feature as well.

Step 2: Identify the Correct Scope

The scope tells iPaaS.com what action to perform (e.g., product/create, customer/update, order/delete).

The supported scopes for your integration and data type are found on the Inbound/Outbound Data Flows page within a subscription's settings or on the Manual Sync page in iPaaS.com.

NOTES:

You must use the exact scope name as it appears in iPaaS.com for that specific integration. Ensure accuracy in spelling and case. You can still get a 200 response from Postman with a wrong scope name.
Be aware that scopes, particularly for importing data (TO iPaaS.com), can vary significantly between different integrations. Always verify the correct scope for the specific integration you are working with.

Step 3: Create Your CSV File

Prepare a CSV file with at least one column, id.
(Optional): Include a second column, scope. This is useful if you need to perform different actions for different records. If all records use the same scope, you can omit this column and specify the scope directly in your Postman request setup later.

The column headers must match the variable names you use in Postman (e.g., id, scope).

Example CSV:

id,scope 
12345,product/create 
45671,customer/update 
88900,product/create

Step 4: Configuring Postman

Authentication

You need the Webhook API key associated with the iPaaS.com subscription you are working with. Find this key in iPaaS.com under: Subscription Management > Subscriptions. Find the relevant subscription in the list and click the Edit icon.

This key will be used as a Bearer Token for authentication in your Postman request header (Authorization: Bearer YOUR_API_KEY).

API Call Details

Use the following settings for the API call details.

Method: POST
URL: You will target the /v2/ipaas endpoint. The base URL depends on the environment:
- Staging: https://stagingapi.ipaas.com/v2/ipaas
- Production: https://api.ipaas.com/v2/ipaas
Body: Construct the request body according to the iPaaS.comAPI documentation (refer to the Swagger page).
Here is the Swagger request body to POST a new customer.

Request Body to POST a New Customer

{
  "id": "string",
  "customer_number": "string",
  "tracking_guid": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "first_name": "string",
  "last_name": "string",
  "email_address": "string",
  "company": "string",
  "categories": [
    {
      "id": "string",
      "name": "string"
    }
  ],
  "comment": "string",
  "addresses": [
    {
      "id": "string",
      "tracking_guid": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "type": "string",
      "phone_number": "string",
      "first_name": "string",
      "last_name": "string",
      "company": "string",
      "address_1": "string",
      "address_2": "string",
      "address_3": "string",
      "city": "string",
      "region": "string",
      "country": "string",
      "postal_code": "string",
      "is_primary_billing": true,
      "is_primary_shipping": true,
      "customer_id": 0,
      "custom_fields": [
        {
          "id": "string",
          "type": "string",
          "custom_field_name": "string",
          "value": "string",
          "tracking_guid": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
        }
      ],
      "external_ids": [
        {
          "system_id": 0,
          "external_id": "string",
          "internal_id": "string",
          "tracking_guid": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
        }
      ]
    }
  ],
  "relationships": [
    {
      "id": "string",
      "type": "string",
      "tracking_guid": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "start_date": "string",
      "end_date": "string",
      "is_primary": true,
      "related_to_id": "string",
      "custom_fields": [
        {
          "id": "string",
          "type": "string",
          "custom_field_name": "string",
          "value": "string",
          "tracking_guid": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
        }
      ],
      "external_ids": [
        {
          "system_id": 0,
          "external_id": "string",
          "internal_id": "string",
          "tracking_guid": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
        }
      ]
    }
  ],
  "custom_fields": [
    {
      "id": "string",
      "type": "string",
      "custom_field_name": "string",
      "value": "string",
      "tracking_guid": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
    }
  ],
  "external_ids": [
    {
      "system_id": 0,
      "external_id": "string",
      "internal_id": "string",
      "tracking_guid": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
    }
  ]
}

Use Postman variables {{id}} and {{scope}} (if using the scope column) as placeholders for the data from your CSV file. The exact structure will depend on the expected payload format for triggering syncs via this endpoint.

To create a call for a TO/FROM iPaaS.com webhook:

Ensure your POST request is saved within a Postman collection.
Click File > New Runner tab, or find the Runner button in Postman.
Drag the collection containing your prepared API request into the Runner window.
Select the specific request you configured.
Click Select File and upload the CSV file you created in Step 3: Create Your CSV File.
Preview the data to ensure it's parsed correctly.
Ensure the iteration count matches the number of data rows in your CSV.
Double-check that your request includes the correct Authorization header with the Bearer Token (Webhook API Key). If you set authentication at the collection level, it should be inherited.
Confirm that the variable names in your request body ({{id}}}, {{scope}}) match the headers in your CSV file.
(Optional): In Advanced Settings, add a delay between requests (e.g., 100-500 milliseconds). This helps prevent overwhelming the iPaaS.com queue and potentially impacting other processing tasks.
Click Start Run.

Important Considerations

Local Execution: The Postman Runner executes the requests from your local machine. Ensure your computer remains powered on and connected to the internet throughout the entire run.
Test Thoroughly: Always perform a test run with a small batch of IDs (e.g., 5-10 records) first. Verify in iPaaS.com that these records are queued and processed as expected before running your full dataset.
Monitor Queue: Keep an eye on the iPaaS.com events queue during and after the run to ensure records are being processed correctly and to gauge the impact on system performance.
Validation via Queue: Do not rely solely on a 200 OK HTTP status response from Postman to confirm success. The API endpoint is designed for speed and may accept the request (returning 200 OK) even if the provided scope is invalid or misspelled for that integration, only failing later during processing. You must validate that the sync operations worked correctly by checking the Events Queue within iPaaS.com after running your test batch and full sync.

Syncing a Large Dataset using iPaaS.com and Postman Runner

How to Add, Test, And Go Live with a New Subscription