James Coholan
All writing

Field note 02 / Entitlements

Replaying Apple's missed notifications from their own signatures

Subscription access drifted from the App Store. The fix was to treat Apple's signed history as the source of truth and reconcile against it every day.

  1. 01

    Symptom

    Some subscribers' access did not match their App Store state. Apple's notification history showed notifications the webhook had rejected on every retry.

  2. 02

    Diagnosis

    • Apple retries a failed notification a limited number of times and then stops, so a missed event silently diverges access from reality.
    • A webhook cannot fix what it never accepted. The source of truth had to be Apple's own history, not our logs.
    • Some notifications also arrive before the app has linked the buyer's identity.
  3. 03

    Fix

    • Every signed payload is verified against Apple's root certificates before anything is written, and sandbox payloads never grant production access.
    • Notifications are deduplicated by UUID into a transactional ledger. Local processing and forwarding are tracked separately, so a retry finishes whichever half is missing.
    • A reconcile tool pulls Apple's notification history with an ES256-signed App Store Server API token and replays anything undelivered from its original signed payload.
    • The reconcile runs daily, and scheduled repair re-links purchases that arrived before identity mapping.
What I do now

Treat the provider's history as the source of truth

  1. Verify signatures before reading any business field.
  2. Make every handler idempotent on the provider's event ID.
  3. Split multi-step handling into parts that can each be retried.
  4. Reconcile against the provider's history on a schedule, not only on alerts.
  5. Replay original signed payloads instead of recreating events.
  6. Quarantine ambiguous identity evidence instead of guessing.