Overview
This document outlines a method for implementing soft deletes in API integrations, specifically using extension fields. Soft deleting marks records as no longer active without permanently removing them from the database. This preserves data for auditing, data recovery, and analysis. The approach detailed here uses a flexible key-value metadata system, commonly found in modern APIs like BigCommerce, to add a "deleted" status to a record.
NOTE: This method displays active records and not the records marked as deleted by the method described in Filtering Logically Deleted Records from API Responses. |
Problem/Challenge
Using a "hard delete" to permanently remove data with no recovery options poses several challenges:
Permanent Data Loss: Hard deletes are irreversible, making it impossible to recover accidentally deleted records or analyze data.
Broken Audit Trails: Removing records creates gaps in audit trails, which is often unacceptable for compliance and security requirements.
No Rollback Option: There is no simple way to undelete a record that was removed by mistake.
Solution Design
The solution is to avoid calling the DELETE endpoint and instead use the API's extension field capabilities to flag a record as deleted.
Flag the Record: When a user deletes a record, the integration makes a
PUTorPOSTrequest to update that entity. It adds or modifies an extension field to signify its deleted state.Define a Convention: A consistent key-value pair is used as the deletion flag. Common conventions include:
A boolean flag:
{"key": "deleted", "value": "true"}A timestamp:
{"key": "deleted_at", "value": "2025-06-20T12:30:00Z"}
Filter on Retrieval: All application logic for retrieving data (e.g.,
GETrequests) must be updated to filter out records containing the deletion flag. These deleted records remain in the system but are hidden from standard views and operations.
This approach effectively creates a soft delete system without requiring any changes to the core API schema.
Benefits
Data Preservation: Maintains historical records, enabling data recovery and audit compliance without permanent data loss.
API Compatibility: Works with existing API endpoints without requiring modifications from the API provider.
Flexibility: The generic key-value structure supports other metadata needs beyond deletion tracking, such as versioning or status flags.
Integration Simplicity: Leverages existing API features, allowing you to implement soft deletes without complex workarounds.
Scalability: The pattern is scalable across different entity types and provides consistent behavior.
Rollback Capability: Soft-deleted records can be erestored by removing or modifying the deletion-related extension field.
Implementation Notes
JSON Serialization: When using typed languages like C# or Java, use JSON property attributes (e.g.,
Newtonsoft.Json's[JsonProperty("...")]) to ensure class properties map correctly to the API's JSON fields.Client-Side Filtering: It is critical to ensure all data retrieval logic is updated to consistently filter out records marked as deleted. Without this, soft-deleted records will still appear in user interfaces and reports as if they were active.
Namespace Organization: In larger projects, organize API model classes into logical namespaces (e.g.,
VendorName.Api.Model.EntityType) for clarity and maintainability.Type Safety: Using strongly-typed classes to represent API objects helps catch errors during development and ensures that data structures are handled correctly.
Example "soft delete" implementation
/*
* Example handling of delete implementation in BigCommerce integration, necessary to support delete workflows
*
* Example definition in iPaaS API models:
*/
using Newtonsoft.Json;
// Basic system utilities.
using System;
// Used for creating lists, such as a list of extension fields.
using System.Collections.Generic;
namespace BigCommerce.v3.Data.IPaaSApi.Model.Customer {
/// <summary>
/// Represents a generic key-value pair used for extension fields.
/// </summary>
public class GenericExtensionFieldResponse {
// Maps the "key" property in the JSON to the Key property in this class.
[JsonProperty("key")]
public string Key { get; set; }
// Maps the "value" property in the JSON to the Value property in this class.
[JsonProperty("value")]
public string Value { get; set; }
}
}
namespace BigCommerce.v3.Data.IPaaSApi.Model.Customer {
... // Other class definitions would be here.
/// <summary>
/// Represents a Company object response from the API.
/// </summary>
public class CompanyResponse {
// This property holds a list of extension fields associated with the company.
// The JSON property is named "extension_fields".
[JsonProperty("extension_fields")]
public List<GenericExtensionFieldResponse> ExtensionFields { get; set; }
}
}