API Integration Patterns

Every integration starts with a well-designed API surface. These are the patterns that keep APIs maintainable as they grow.

Resource-oriented design: Structure endpoints around business resources, not actions. Use standard HTTP verbs — GET for retrieval, POST for creation, PATCH for partial updates, DELETE for removal. Nest related resources logically: /api/v1/orders/{id}/items/ rather than /api/v1/get-order-items/.

Versioning strategy: URL-based versioning (/api/v1/, /api/v2/) for public APIs that external clients consume. Header-based versioning for internal APIs where URL aesthetics matter less. Always maintain backward compatibility within a version — deprecate, do not break.

Pagination and filtering: Cursor-based pagination for large datasets (more efficient than offset-based for datasets over 100,000 records). Filterable endpoints using query parameters with Django-filter integration. Always return total count and next/previous cursor in the response envelope.

Authentication patterns: JWT tokens for stateless API access with short-lived access tokens (15 minutes) and long-lived refresh tokens (7 days). API key authentication for service-to-service communication. OAuth 2.0 for third-party integrations. Django REST Framework's permission classes make it straightforward to enforce role-based access at the view level.

Response envelope pattern: Consistent response structure across all endpoints — always include status, data, and errors fields. Clients can rely on the same parsing logic regardless of the endpoint. Error responses include machine-readable codes alongside human-readable messages for easier debugging.

Webhooks are how external systems notify your application of events — Stripe sends payment confirmations, Shopify sends order updates, Telegram sends messages. A robust webhook receiver must handle unreliable delivery, duplicate events, and high throughput.

Receive-and-queue pattern: The webhook endpoint does minimal processing — it validates the signature, stores the raw payload in a queue (Redis or PostgreSQL), and returns a 200 response within 500ms. A background worker (Celery) processes the payload asynchronously. This pattern prevents webhook timeouts and ensures you never lose events even if processing fails temporarily.

Signature verification: Every webhook source provides a signing mechanism — Stripe uses HMAC-SHA256 with a webhook secret, Telegram uses a SHA-256 hash of the bot token. Always verify signatures before processing. Reject unsigned or incorrectly signed payloads with a 401 response. This prevents replay attacks and forged events.

Idempotency handling: Webhook providers retry failed deliveries, so you will receive duplicate events. Store a hash of each processed event ID in a database table or Redis set. Before processing, check if the event was already handled. This pattern ensures that processing an event twice does not create duplicate orders, double-charge customers, or send duplicate notifications.

Dead letter queue: Events that fail processing after maximum retries (typically 3-5 attempts with exponential backoff) are moved to a dead letter queue for manual inspection. An admin notification is sent via Telegram when the dead letter queue has items. This ensures no event is silently lost.

Multi-source webhook router: A single Django URL pattern (/webhooks/{source}/) routes incoming webhooks to source-specific handlers. Each handler validates the signature using the source's method, normalises the payload into an internal event format, and dispatches to the processing queue. Adding a new webhook source requires implementing one handler function — the router, queue, and error handling are shared infrastructure.

Keeping data consistent across multiple systems — your database, a CRM, an e-commerce platform, an accounting tool — requires a deliberate sync strategy. The wrong approach leads to data drift, conflicts, and debugging nightmares.

Event-driven sync (recommended): When data changes in one system, an event is emitted and all dependent systems update. Implementation: Django signals or explicit event dispatch on model save, processed by Celery workers that push changes to external systems via their APIs. Pros: near-real-time sync, minimal API calls, clear audit trail. Cons: requires event infrastructure, eventual consistency.

Polling sync (fallback): A scheduled task (Celery Beat) periodically queries external systems for changes. Compare timestamps or version numbers to detect what changed since the last sync. Use this when the external system does not support webhooks. Polling interval depends on freshness requirements — every 5 minutes for order data, every hour for inventory, daily for analytics.

Conflict resolution: When the same record is modified in two systems simultaneously, you need a conflict resolution strategy. Options: last-write-wins (simplest, acceptable for non-critical data), source-of-truth priority (one system always wins — typically the system closest to the user action), and merge with manual review (flag conflicts for human resolution). I default to source-of-truth priority with conflict logging.

Change detection: Use updated_at timestamps and version counters on all syncable models. Django's auto_now on DateTimeField provides timestamps automatically. For external systems, store the last known state hash and compare on each sync cycle. Only push changes when data actually differs — this reduces unnecessary API calls by 80-90%.

Sync monitoring: Every sync operation logs the source, destination, record count, success/failure status, and duration. A daily summary report flags any sync failures, data mismatches, or unusual patterns (e.g., sync volume doubled overnight). Alert on consecutive failures — if a sync fails 3 times in a row, something is structurally wrong, not transient.

External APIs fail. Networks have transient issues. Rate limits are hit. Robust error handling is the difference between a system that recovers gracefully and one that loses data.

Retry with exponential backoff: On transient errors (HTTP 429, 500, 502, 503, 504, connection timeout), retry with increasing delays: 1 second, 4 seconds, 16 seconds, 64 seconds. Cap at 5 retries. After exhausting retries, move the task to the dead letter queue and alert. Celery's built-in retry mechanism with autoretry_for and retry_backoff handles this pattern cleanly.

Circuit breaker pattern: If an external API fails repeatedly (e.g., 5 consecutive failures within 10 minutes), stop sending requests and enter a "circuit open" state. After a cooldown period (e.g., 5 minutes), send a single test request. If it succeeds, resume normal operation ("circuit closed"). If it fails, extend the cooldown. This prevents cascading failures and avoids wasting resources on a system that is down.

Graceful degradation: When a non-critical integration fails, the primary user action should still succeed. Example: if the CRM sync fails after creating an order, the order is created successfully, and the CRM sync is retried in the background. The customer never sees an error. Only critical path failures (e.g., payment processing) should block the user action.

Structured error logging: Log every external API call with request URL, method, headers (sanitised — no API keys), response status, response body (truncated to 1KB), and duration. Use structured logging (JSON format) for easy querying. Tag errors with the integration name and error category for aggregated monitoring. This logging makes debugging integration issues a matter of filtering logs rather than reproducing the problem.

Rate limit management: Track API rate limits per external service. Before making a request, check the remaining quota. If the quota is low, throttle requests by introducing delays or queuing non-urgent calls. Store rate limit counters in Redis with TTL matching the rate limit window. This prevents hitting rate limits in the first place, which is more efficient than handling 429 errors reactively.