The hardest import failures are often uncertain outcomes. A client sends a batch, the connection closes, and nobody knows whether the destination committed the records. Repeating the request may be necessary, but repeating it blindly can create duplicates or apply an update twice. A reliable batch workflow makes that uncertainty visible and gives both clients and operators a safe way to resolve it.
This guide proposes a catalog import with stable product identifiers and an asynchronous processing job. The design examples are patterns to evaluate in your own application, not a description of a hosted ImportAPI.com service. Start by defining what a repeated request means, then build retries and progress reporting around that agreement.
Distinguish a logical operation from an attempt
A logical import is the change the caller intends to make. An attempt is one effort to deliver or process that change. Network interruptions and worker restarts can create several attempts for one logical operation. Keep their identifiers distinct so the user sees one import with a history rather than several unrelated jobs.
For a catalog handoff, the logical operation might mean apply this source revision to this destination under this mapping. An attempt records when processing began, which stage it reached, and why it stopped. This separation gives support teams a coherent story and prevents infrastructure-level retries from automatically appearing as new business actions.
Define idempotency at the boundary
An idempotent operation is designed so repeating the same intended request does not repeatedly apply its effect. For a batch API, a caller-supplied request identifier can help the service recognize a retry. The identifier needs a defined scope, retention policy, and relationship to the request content. A token alone is not a complete implementation.
The AWS Builders' Library discussion of idempotent APIs explains the importance of expressing caller intent and handling repeated requests. Apply that idea deliberately: a repeated identifier with different content should not silently be treated as the original request. Define whether it produces a conflict, and preserve enough information to recognize the mismatch.
Keep request deduplication and record identity separate
Request deduplication asks whether the caller is repeating the same operation. Record identity asks whether a source product already exists in the destination. Both matter. A brand-new request may intentionally update existing products, while a retry of an earlier request should not create another set of those products.
Use stable external identifiers for records where the source can provide them. Decide how the destination handles a missing or conflicting identifier. Do not substitute a random identifier on every attempt if repeated processing should refer to the same record. When a source lacks stable identity, make that limitation explicit before promising safe automatic retries.
Make the job record a source of truth
A job record should report a meaningful stage and reconciled outcomes. Suggested fields include source revision, mapping version, total records evaluated, accepted records, rejected records, committed changes, and the most recent error category. Define these fields precisely. Accepted during validation is not necessarily the same as committed to the destination.
Avoid progress labels that imply more certainty than the system has. A completed upload means the bytes arrived, not that the destination has changed. A worker finishing its loop does not prove every requested write succeeded. Separate transport, validation, and commit states so an operator can determine which part of the workflow needs attention.
Choose atomic or partial behavior deliberately
An atomic batch aims to apply the intended set of changes together or not at all within the defined transaction boundary. A partial batch can accept some records while rejecting others. The right policy depends on the destination and its consistency requirements. A set of interdependent configuration records may need different handling from an independent product catalog.
For partial processing, retain record-level outcomes and a clear continuation strategy. A correction batch should identify which rejected records it replaces or retries. Do not force the caller to guess whether previously successful rows will be skipped, updated, or duplicated. The mapping guide explains how stable contracts make those decisions easier to express.
Retry only when the failure justifies it
Classify failures before retrying. A temporary service interruption may justify another attempt. A missing required product identifier needs correction, not repeated processing. An authorization failure should prompt investigation rather than an aggressive loop. Stable error categories help client software choose an appropriate response without parsing human-readable sentences.
Bound the retry policy
Use bounded retries and a delay policy appropriate to the system. Avoid synchronized retry bursts that add load during a recovery. Respect documented retry guidance from any downstream service. After the retry limit, leave a clear unresolved state and an operator path. Endless background attempts can hide persistent errors while consuming resources and delaying useful feedback.
Handle timeouts as unknown outcomes
A timeout describes what the caller observed, not necessarily what the destination did. After a connection closes, a worker might still finish a commit. Provide a status lookup by logical job or request identifier so the caller can reconcile the result before starting a new operation. This is particularly important when destination changes are not easily reversible.
Test the case explicitly. Interrupt the response after the server has accepted the request, then repeat the request with the same identifier. Confirm that the system returns or resolves the existing operation according to its contract. A happy-path demonstration cannot reveal whether the uncertain-outcome path creates duplicates.
Make commits and bookkeeping consistent
Consider what happens if records are written but the job's completion status is not. Conversely, consider a completion status being recorded before all intended changes are durable. Design the transaction boundary and recovery procedure so these disagreements can be detected and resolved. The exact mechanism depends on the destination and processing architecture.
Keep a durable relationship between the logical import and committed records or effects where that is appropriate. A reconciliation process can then inspect the actual destination instead of relying solely on a worker's last message. Do not claim exactly-once behavior unless the complete boundary and its failure assumptions have been established and tested.
Plan cancellation and correction
Cancellation is simple only before any effects occur. Once a partial batch has committed some records, stopping further work is different from undoing completed changes. Name those actions distinctly. An operator should know whether cancel means stop processing, roll back, or create a compensating operation.
For a catalog, a correction may be safer and clearer than attempting to reconstruct every prior state automatically. Preserve the source revision and outcome report so the correction can target the right records. Where rollback is required, establish what prior state must be retained and how concurrent updates affect that process before offering the action to users.
Test a realistic failure matrix
Use a small fixture batch and inject failures before validation, during processing, after a partial commit, and after completion but before the response reaches the caller. Add a repeated request with changed content and a corrected request with a new identifier. For each scenario, write down the expected job count and destination state.
Test the operator view too. Someone should be able to distinguish waiting, retrying, requiring correction, and completed without reading internal logs. Redact sensitive source values while preserving useful record references. A reliable workflow includes understandable recovery, not merely code that eventually succeeds in a controlled test.
Conclusion: make repeats and uncertainty explicit
Reliable batch imports depend on well-defined intent, stable identity, visible job states, and a tested recovery path. Separate the logical operation from its attempts, choose partial or atomic behavior deliberately, and treat timeouts as potentially uncertain outcomes. Retries become safer when they follow a contract rather than a guess.
Use the data migration workflow to connect these ideas to a complete project, and the developer reference to inspect a sample outcome shape. A small import that recovers predictably from an interrupted response is a better foundation than a fast pipeline whose repeated effects are difficult to explain.
