How to Prevent Duplicate DWS Records
Prevent duplicate DWS records with stable measurement IDs, idempotent processing, explicit acknowledgements, safe retries and remeasurement rules.
Sep 11,2026 | Jasper Lee
Direct answer: Prevent duplicate DWS records by assigning one stable measurement ID to one capture event, sending that same ID on every retry, and making the receiving system idempotent: if it has already accepted the ID, it returns the previous result without applying the update again. Add explicit acknowledgement states, preserve the original payload, separate re-delivery from a genuine remeasurement, and reconcile uncertain outcomes instead of generating a new ID.
Why duplicates happen
A duplicate does not necessarily mean the DWS measured twice. A sender may complete a transmission but fail to receive the acknowledgement, then retry. A consumer may process a message and restart before marking it complete. A network timeout can leave both sides uncertain. Microsoft documents these patterns for at-least-once messaging and recommends combining duplicate detection with idempotent processing.
In a DWS workflow, the risk is operational: the same observation might overwrite a newer value, create two shipment events, or detach data from the physical parcel.
Start with one stable identity
Use a unique measurement_id for the capture event. It should be created once and remain unchanged when the identical event is retried. The record should also carry the business object ID—such as parcel, license plate, shipment or SKU—and capture context such as station and timestamp.
| Identifier | Purpose | Retry behavior |
|---|---|---|
| measurement_id | Identifies one physical observation | Reuse for the same event |
| object_id | Identifies the parcel, SKU or handling unit | Do not use alone as the duplicate key |
| attempt_id | Optional transport diagnostic | May change per delivery attempt |
| supersedes_id | Links a genuine remeasurement to the prior observation | Create only for a new capture event |
Retry and remeasure are different
A retry resends the same observation because delivery or acknowledgement is uncertain. It keeps the same measurement ID and payload. A remeasurement creates a new physical observation, so it needs a new measurement ID plus an explicit relationship to the record it may replace. Treating every retry as a new measurement defeats duplicate detection; treating a real remeasurement as a retry can hide changed data.
Make the receiver idempotent
Before applying a write, the receiver checks whether the measurement ID was already processed. If so, it returns the stored outcome. If not, it validates and applies the operation, then records the ID and outcome in the same reliable transaction boundary where practical.
- Receive the message and validate its schema.
- Look up the measurement ID in a durable processed-event store.
- If accepted before, return the prior acknowledgement without another write.
- If new, apply business validation and persist the result.
- Record acceptance or rejection so the same event remains repeatable.
Broker duplicate detection can help, but it has a configured history window and does not replace receiver idempotency. Microsoft explicitly recommends idempotent consumers even when send-side duplicate detection is enabled.
Use precise acknowledgement states
| Status | Meaning | Sender action |
|---|---|---|
| received | Transport accepted the message | Do not assume the WMS record changed |
| accepted | Business validation passed and the intended write is durable | Stop retrying this event |
| duplicate | The same measurement ID already has a known outcome | Use the prior result |
| rejected | The message or business object is invalid | Route to correction; blind retry will not help |
| retryable_error | A temporary dependency prevented completion | Retry the same ID with backoff |
Protect against out-of-order updates
Idempotency stops the same event from applying twice, but it does not automatically stop an older observation from arriving after a newer one. Define version, capture time or supersession rules. A master-data update may require review before replacing an accepted value, while a shipment observation may be immutable evidence rather than a field to overwrite.
CONLIDA engineering note: reconcile the physical parcel
Technical deduplication is incomplete if the team cannot locate the parcel associated with the uncertain message. Keep a correlation key across DWS, WCS and WMS events so operations can answer: where is the item, which observation was accepted, and whether a new physical measurement is required?
Test these failure cases
- the sender times out after the receiver commits;
- the consumer restarts after writing but before acknowledging;
- the same message arrives twice concurrently;
- an old retry arrives after a genuine remeasurement;
- the WMS rejects the object ID or unit of measure;
- the duplicate-history window expires;
- the integration queue is replayed after recovery.
Implementation checklist
- Define one measurement ID and its creation point.
- Reuse it for every identical retry.
- Store processed IDs and outcomes durably.
- Make acceptance idempotent and concurrency-safe.
- Separate transport receipt from business acceptance.
- Give genuine remeasurement a new ID and supersession rule.
- Monitor duplicates, retries, rejects and unresolved outcomes.
Connecting the pattern to CONLIDA DWS
A CONLIDA DWS integration can provide captured physical data and identifiers, but the approved project interface must assign ID creation, persistence, acknowledgement and retry ownership. Exact protocols and fields depend on the selected configuration and receiving systems.
Design a retry-safe DWS integration.
Sources
Comment
