Filtering Logically Deleted Records from API Responses

Overview

This document describes a strategy for filtering logically deleted records from API responses. In many systems, data is not permanently destroyed but is instead marked as "deleted" or "inactive" with a status flag. This practice, known as a soft or logical delete, preserves a complete data history for auditing and recovery.

The solution outlined here involves a post-processing step that inspects API response data. By checking for a specific flag (e.g., an IsActive: "False" extension field), this filter ensures that client applications receive only active, relevant data by default, simplifying application logic and improving data quality.

NOTE: This method marks the "deleted" records as inactive and Implementing Soft Deletes with Extension Fields in API Integrations displays only active records.

Problem/Challenge

When integrating with an API that uses logical deletes, the raw response often includes both active and inactive records. This presents several challenges:

Repetitive Filtering Logic: Every part of the application that fetches data must manually filter out the inactive records. This is inefficient and error-prone.
Cluttered Business Logic: Application code becomes cluttered with data-filtering concerns instead of focusing on its primary business functions.
Risk of Stale Data: If a developer forgets to add a filter in a new feature, the application may inadvertently display or process outdated, "deleted" information, resulting in incorrect results.

Solution Design

The proposed solution is a centralized filtering mechanism that intercepts and cleans the API response before it is passed to the rest of the application.

Intercept the Response: The raw data returned from the API call is captured by a handler or wrapper function.
Provide an Escape Hatch: The handler includes an optional parameter (e.g., includeDeleted: true) that allows the caller to bypass the filter when access to the complete, unfiltered dataset is required for auditing or administrative tasks.
Inspect the Data: The handler determines if the response is a single object or a collection of objects.
Apply Filtering Logic:
- For each object, it inspects the metadata for a predefined deletion flag (e.g., an extension field {"key": "IsActive", "value": "False"}).
- If an object is flagged as deleted, it is removed from the results. If the response was a single object, the handler returns null; if it were a collection, the object is removed from the list.
Return Clean Data: The handler returns the processed response, now containing only active records.

Benefits

Centralized Logic: The rules for identifying and filtering deleted records are defined in a single location, making them easy to maintain and preventing code duplication.
Improved Data Quality: Applications work with a clean, active dataset by default, reducing the risk of bugs caused by stale data.
Flexible Data Access: The ability to bypass the filter provides on-demand access to the complete historical record when needed.
Simplified Client Code: Application developers are freed from the responsibility of data filtering and can focus on core business logic.

Implementation Notes

Deletion Convention: The effectiveness of this pattern depends on a consistent convention for marking records as deleted. The example code looks for an extension field IsActive with a string value of False.
Performance Consideration: This filtering occurs after the full API response has been received over the network. It improves the performance and reliability of the consuming application by reducing its processing load, but it does not reduce network traffic or the load on the source API endpoint.
Handling Nulls: The implementation should be robust enough to handle null responses or objects that lack extension fields altogether, treating them as non-deleted by default.

Example implementation

/*
* Example implementation in IPaaSApiCallWrapper
*/

        /// <summary>
        /// Checks a list of extension fields to see if a record is marked as deleted.
        /// A record is considered deleted if it has an extension field with the key "IsActive" and the value "False".
        /// </summary>
        /// <param name="extensionFields">A list of key-value pairs representing the extension fields.</param>
        /// <returns>True if the fields indicate the record is deleted, otherwise false.</returns>
        private bool ExtensionFieldsIndicatesDeleted(List<KeyValuePair<string, string>> extensionFields)
        {
            // If there are no extension fields, the record cannot be marked as deleted. 
            if (extensionFields == null)
                return false;
            
            // Find the first extension field with the key "IsActive". 
            var isActiveIndicator = extensionFields.FirstOrDefault(x => x.Key == "IsActive");

            // If no "IsActive" key is found, the record is not considered deleted. 
            if (isActiveIndicator.Key != "IsActive")
                return false;
            
            // The record is considered deleted only if the "IsActive" value is "False". 
            return (isActiveIndicator.Value == "False");
        }

        /// <summary>
        /// Processes an API response to remove data that is marked as deleted.
        /// </summary>
        /// <param name="response">The API response object, which can be a single object or a list of objects.</param>
        /// <param name="includeDeleted">A flag to indicate whether to keep deleted data in the response.</param>
        /// <returns>The processed response with deleted data removed, unless `includeDeleted` is true.</returns>
        private object HandleDeletedData(object response, bool includeDeleted = false)
        {
            // If the caller has requested to include deleted data, or if the response is empty, do nothing and return the original response. 
            if (includeDeleted || response == null)
                return response;
            
            // Check if the response is a single object (not a list). 
            if (!(response is IList))
            {
                // Extract the extension fields from the object.
                var extensionFields = GetExtensionFields(response);
                
                // If the extension fields indicate this is a deleted entry, set the response to null to remove its payload. 
                if (ExtensionFieldsIndicatesDeleted(extensionFields))
                    response = null;
            }
            // Check if the response is a list of objects. We use IList because it allows for item removal. 
            // Note: The comment indicates that current endpoints use lists, so `IList` is a safe assumption. 
            else if (response is IList) 
            {
                // Determine the type of objects contained within the list.
                var singletonType = response.GetType().GetGenericArguments().Single(); 
                
                // Note: The implementation for list filtering is not shown in this snippet but would logically go here.
                // It would iterate through the list, check each item using `ExtensionFieldsIndicatesDeleted`,
                // and remove the items that are marked as deleted.
            }

            // Return the modified response. 
            return response;
        }

Implementing Soft Deletes with Extension Fields in API Integrations

Simplifying End-User Data Mapping for Nested API Requests (Acumatica Example)