--- date: 2026-02-24 title: Custom Email Domain Sync And Recovery --- ## Problem Statement Custom email domains configured via AWS SES can get stuck in a `PENDING` state or fail validation silently. Currently, there is **no automated verification** -- users must manually click "Sync" in the UI to check domain status. If a domain fails to validate, the only option is to delete it and recreate it, which generates new DKIM keys and requires the user to update their DNS records. ### Current Pain Points 1. **No background sync** -- Domain verification status is never checked automatically; users must manually click "Sync" 2. **Stuck domains** -- Domains can remain in `PENDING` state indefinitely with no alerting or auto-recovery 3. **Failed recovery requires DNS changes** -- Deleting and recreating a domain generates new keys, forcing the user to update DNS records 4. **No visibility into failure duration** -- There's no tracking of how long a domain has been pending ## Proposed Solution ### 1. Hourly Background Sync Job Create a new cron job (`internal.sync-email-domains`) that runs every hour to automatically verify all `PENDING` email domains. **Job Definition:** `packages/lib/jobs/definitions/internal/sync-email-domains.ts` **Job Handler:** `packages/lib/jobs/definitions/internal/sync-email-domains.handler.ts` **Pattern:** Follow the existing `cleanup-rate-limits` cron job pattern: - `cron: '0 * * * *'` (every hour, on the hour) - Empty `z.object({})` schema (no payload needed) - Register in `packages/lib/jobs/client.ts` **Handler Logic:** 1. Query all `EmailDomain` records with `status: 'PENDING'` 2. For each domain, call `verifyEmailDomain(emailDomainId)` which: - Calls AWS SES `GetEmailIdentityCommand` to check current verification status - Updates DB status to `ACTIVE` if verified, keeps `PENDING` otherwise 3. Log results via `io.logger` (how many checked, how many transitioned to ACTIVE) 4. Process domains in batches to avoid overwhelming SES API rate limits 5. Add error handling per-domain so one failure doesn't stop the entire sweep ### 2. Schema Changes -- Track Pending Duration Add a `lastVerifiedAt` column to the `EmailDomain` model to track when verification was last attempted, enabling "stale domain" detection. **File:** `packages/prisma/schema.prisma` ```prisma model EmailDomain { // ... existing fields ... lastVerifiedAt DateTime? // Last time verification was checked against SES } ``` **Migration:** Create a new Prisma migration for this column addition. **Updates needed:** - `verify-email-domain.ts` -- Update `lastVerifiedAt` when verification is checked - The sync job handler -- Use `lastVerifiedAt` to avoid re-checking domains that were just verified ### 3. Domain Re-registration (Recovery) -- Delete & Recreate in SES Without Changing Keys Add a new "Re-register" action that deletes the SES identity and recreates it using the **same** DKIM key pair stored in the database, so the user's DNS records remain valid. #### 3a. New Service Function **File:** `packages/ee/server-only/lib/reregister-email-domain.ts` ```typescript export const reregisterEmailDomain = async (options: { emailDomainId: string }) => { // 1. Fetch the EmailDomain record (including encrypted privateKey) // 2. Decrypt the private key using DOCUMENSO_ENCRYPTION_KEY // 3. Call DeleteEmailIdentityCommand on SES (ignore NotFoundException) // 4. Call CreateEmailIdentityCommand with BYODKIM using the SAME selector + private key // 5. Update EmailDomain status back to PENDING, update lastVerifiedAt // 6. Return the updated domain }; ``` Key points: - Uses the existing encrypted `privateKey` from the DB -- no new key generation - Uses the existing `selector` -- DNS records stay the same - Deletes first, then recreates -- handles cases where SES state is corrupted - Resets status to `PENDING` since verification will need to re-occur - Uses `verifyDomainWithDKIM()` from `create-email-domain.ts` (may need to extract/export this helper) #### 3b. Admin TRPC Routes (Find, Get, Re-register) All email domain admin routes use `adminProcedure` -- requires system-level `Role.ADMIN`. **Find (list) route:** **File:** `packages/trpc/server/admin-router/find-email-domains.ts` **Types:** `packages/trpc/server/admin-router/find-email-domains.types.ts` - Query route: `admin.emailDomain.find` - Input: `{ query?: string, page?: number, perPage?: number, status?: EmailDomainStatus }` - Extends `ZFindSearchParamsSchema` with optional `status` filter - Returns standard `ZFindResultResponse` with email domain data including: id, domain, status, selector, createdAt, lastVerifiedAt, organisation name, email count - Prisma query filters by domain name (LIKE search on `query`), optional status, joins organisation for name, counts emails **Get (detail) route:** **File:** `packages/trpc/server/admin-router/get-email-domain.ts` **Types:** `packages/trpc/server/admin-router/get-email-domain.types.ts` - Query route: `admin.emailDomain.get` - Input: `{ emailDomainId: string }` - Returns full email domain detail: all fields (except privateKey), organisation info, list of associated emails, DNS records (generated from publicKey + selector) - Omits `privateKey` from response **Re-register (mutation) route:** **File:** `packages/trpc/server/admin-router/reregister-email-domain.ts` **Types:** `packages/trpc/server/admin-router/reregister-email-domain.types.ts` - Mutation route: `admin.emailDomain.reregister` - Input: `{ emailDomainId: string }` - Calls `reregisterEmailDomain()` - Rationale: Re-registration is a recovery/operational action that deletes and recreates an SES identity. This is a privileged operation that should only be performed by platform operators, not self-service by org admins. #### 3c. Register in Admin Router **File:** `packages/trpc/server/admin-router/router.ts` Add a new `emailDomain` namespace to the admin router: ```typescript emailDomain: { find: findEmailDomainsRoute, get: getEmailDomainRoute, reregister: reregisterEmailDomainRoute, }, ``` #### 3d. Admin Panel UI -- Email Domains Section **List page:** `apps/remix/app/routes/_authenticated+/admin+/email-domains._index.tsx` - New admin panel page at `/admin/email-domains` - Follow the existing admin documents list pattern (client-side TRPC data fetching) - Search input (debounced) filtering by domain name - Status filter dropdown (All / Pending / Active) - DataTable with columns: Domain, Organisation, Status (badge), Email Count, Created, Last Verified, Actions - Actions dropdown per row: View details, Re-register - Pagination via `DataTablePagination` **Detail page:** `apps/remix/app/routes/_authenticated+/admin+/email-domains.$id.tsx` - Shows full domain details: domain, selector, status, organisation, created date, last verified date - Shows DNS records (DKIM + SPF) with copy buttons (reuse `organisation-email-domain-records-dialog` pattern) - Table of associated organisation emails - "Re-register" button with confirmation dialog explaining the action (SES identity will be deleted and recreated with the same keys) - "Verify Now" button to manually trigger a verification check - Shows how long the domain has been pending (using `lastVerifiedAt` or `createdAt`) **Navigation:** Add menu item to admin sidebar in `_layout.tsx`: ```tsx ``` **Table component:** `apps/remix/app/components/tables/admin-email-domains-table.tsx` (optional -- can be inline in the route file like the documents page) #### 3e. Automatic Re-registration in Sync Job (Optional Enhancement) In the hourly sync job, after checking verification status, if a domain has been `PENDING` for more than 48 hours: - Automatically call `reregisterEmailDomain()` to attempt recovery - Log the auto-recovery attempt - This provides a self-healing mechanism without user intervention ## Implementation Plan ### Phase 1: Background Sync Job (Core) 1. Create `sync-email-domains.ts` job definition with hourly cron 2. Create `sync-email-domains.handler.ts` with batch verification logic 3. Register job in `packages/lib/jobs/client.ts` 4. Add error handling and logging ### Phase 2: Schema Enhancement 5. Add `lastVerifiedAt` column to `EmailDomain` model 6. Create Prisma migration 7. Update `verifyEmailDomain()` to set `lastVerifiedAt` on each check 8. Update sync job to use `lastVerifiedAt` for intelligent scheduling ### Phase 3: Admin Email Domains Panel 9. Create `find-email-domains` admin TRPC route + types (list/search with pagination and status filter) 10. Create `get-email-domain` admin TRPC route + types (detail view with org info, emails, DNS records) 11. Register find + get routes in admin router under `emailDomain` namespace 12. Create admin list page (`admin+/email-domains._index.tsx`) with search, status filter, DataTable 13. Create admin detail page (`admin+/email-domains.$id.tsx`) with domain info, emails table, DNS records 14. Add "Email Domains" menu item to admin sidebar (`_layout.tsx`) ### Phase 4: Re-registration Feature 15. Extract `verifyDomainWithDKIM()` as a shared helper (if not already exported) 16. Create `reregisterEmailDomain()` service function 17. Create `reregister-email-domain` admin TRPC mutation route + types 18. Register reregister route in admin router under `emailDomain.reregister` 19. Add "Re-register" button + confirmation dialog on admin detail page ### Phase 5: Auto-Recovery (Optional) 20. Add 48-hour stale detection logic to sync job 21. Auto-trigger re-registration for stale domains 22. Add logging/notifications for auto-recovery events ## Files to Create/Modify ### New Files - `packages/lib/jobs/definitions/internal/sync-email-domains.ts` - `packages/lib/jobs/definitions/internal/sync-email-domains.handler.ts` - `packages/ee/server-only/lib/reregister-email-domain.ts` - `packages/trpc/server/admin-router/find-email-domains.ts` - `packages/trpc/server/admin-router/find-email-domains.types.ts` - `packages/trpc/server/admin-router/get-email-domain.ts` - `packages/trpc/server/admin-router/get-email-domain.types.ts` - `packages/trpc/server/admin-router/reregister-email-domain.ts` - `packages/trpc/server/admin-router/reregister-email-domain.types.ts` - `apps/remix/app/routes/_authenticated+/admin+/email-domains._index.tsx` - `apps/remix/app/routes/_authenticated+/admin+/email-domains.$id.tsx` ### Modified Files - `packages/prisma/schema.prisma` -- Add `lastVerifiedAt` field - `packages/lib/jobs/client.ts` -- Register new sync job - `packages/ee/server-only/lib/verify-email-domain.ts` -- Update `lastVerifiedAt` - `packages/ee/server-only/lib/create-email-domain.ts` -- Export `verifyDomainWithDKIM` helper - `packages/trpc/server/admin-router/router.ts` -- Add `emailDomain.{find, get, reregister}` routes - `apps/remix/app/routes/_authenticated+/admin+/_layout.tsx` -- Add "Email Domains" nav item to sidebar - New Prisma migration file ## Technical Considerations 1. **SES API Rate Limits** -- AWS SES has rate limits on `GetEmailIdentityCommand`. The sync job should process domains in batches with small delays between calls (e.g., 5-10 per batch with 1s delay). 2. **Concurrency** -- The local job provider has deterministic deduplication via SHA-256 IDs, so multiple app instances won't run the same cron tick twice. 3. **Error Isolation** -- Each domain verification in the sync job should be wrapped in try/catch so one failing domain doesn't prevent others from being checked. 4. **Re-registration Safety** -- The re-register function should be idempotent. Deleting a non-existent SES identity should be handled gracefully (already done in `deleteEmailDomain`). 5. **Private Key Security** -- The private key is encrypted at rest and should only be decrypted transiently during re-registration. It should never be logged or exposed in API responses. 6. **Feature Gating** -- The sync job should only process domains belonging to organisations with active `emailDomains` claim flags. This prevents processing domains for orgs that have downgraded. 7. **Observability** -- Add structured logging to the sync job so operations teams can monitor domain verification health across all tenants.