chore(deps): upgrade to Rails 7.2.3.2, Ruby 3.4.10 and latest gems

Upgrades Rails from 7.1.6 to 7.2.3.2, Ruby from 3.4.6 to 3.4.10, and
brings the rest of the bundle up to date.

Two things were broken by Rails 7.2 and are fixed here:

* MigrationWaiter called `ActiveRecord::Base.connection.migration_context`,
  which moved to the connection pool in 7.2. This raised NoMethodError on
  boot for any deployment with `migration_waiter.enabled` set, and had no
  test covering it.
* `config.action_dispatch.show_exceptions = false` was deprecated in 7.1
  and removed in 7.2. Now `:none`.

`json` is pinned to < 3. Active Support 7.2 passes `quirks_mode:` to both
`JSON.generate` and `JSON.parse`, and json 3.0 removed that option, so an
unpinned json resolves to 3.x and makes every `to_json` raise - webhook
payloads included. Nothing else in the bundle constrains it. The pin can go
when we reach Rails 8.1, which no longer passes the option.

`dynamic_form` is dropped. It is unmaintained and from 1.3.1 requires Ruby
< 3.4, so it could never be updated again. Ten views used it, so
`error_messages_for` now lives in ErrorMessagesHelper and `f.error_messages`
in a form builder extension. The rendered markup is unchanged apart from a
heading and description the stylesheet had always hidden; the rules that
hid them are removed too.

`execjs` is unpinned. The `< 2.8` constraint dated from the Rails 6.0
upgrade and asset output is byte-identical without it.

Notable gem majors: puma 8, haml 7, secure_headers 7, sentry 7, jwt 3,
prometheus-client 5, highline 3, omniauth-rails_csrf_protection 2,
rspec-rails 8, shoulda-matchers 8, rubocop 1.91. secure_headers 7 changes
X-XSS-Protection from "1; mode=block" to "0", which is the correct modern
default and matches Rails' own.

`config.load_defaults` deliberately stays at 7.0. The pending 7.1 and 7.2
options are listed, commented out, in new_framework_defaults_7_1.rb and
_7_2.rb so they can be adopted in stages - several change the signed cookie
and cache formats and need a staged deploy. The vestigial
new_framework_defaults_7_0.rb is removed, as `load_defaults 7.0` already
applies all of it.

Adds tests for the areas this touched, which were mostly uncovered:
MigrationWaiter, the serialized columns that gate a defaults bump, JSON
encoding, HAML template compilation for all 103 templates, page rendering,
form error rendering and security headers. The assets Gemfile group now
includes :test so the view layer can be exercised at all.

.github/workflows/ci.yml

@@ -21,7 +21,7 @@ jobs:
     name: CI Image Build
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
       - uses: docker/setup-buildx-action@v2
@@ -37,13 +37,14 @@ jobs:
           cache-from: type=gha
           cache-to: type=gha,mode=max
           target: ci
+          platforms: linux/amd64
 
   test:
     name: Test Suite
     runs-on: ubuntu-latest
     needs: build
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
       - uses: docker/login-action@v2
@@ -51,10 +52,10 @@ jobs:
           registry: ghcr.io
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
-      - run: docker-compose pull
+      - run: docker compose pull
         env:
           POSTAL_IMAGE: ghcr.io/postalserver/postal:ci-${{ github.sha }}
-      - run: docker-compose run postal sh -c 'bundle exec rspec'
+      - run: docker compose run postal sh -c 'bundle exec rspec'
         env:
           POSTAL_IMAGE: ghcr.io/postalserver/postal:ci-${{ github.sha }}
 
@@ -62,30 +63,50 @@ jobs:
     name: Release (branch)
     runs-on: ubuntu-latest
     needs: [build]
-    if: startsWith(github.ref, 'refs/heads/')
+    if: >-
+      startsWith(github.ref, 'refs/heads/') &&
+      startsWith(github.ref, 'refs/heads/release-please-') != true &&
+      startsWith(github.ref, 'refs/heads/dependabot/') != true
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: docker/setup-qemu-action@v3
       - uses: docker/setup-buildx-action@v2
       - uses: docker/login-action@v2
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
-      - id: tag
+      - id: info
         run: |
-          TAG="${GITHUB_REF#refs/heads/}"
+          IMAGE=ghcr.io/postalserver/postal
-          if [ -z "$TAG" ]; then exit 1; fi
+
-          if [[ $TAG == "main" ]]; then TAG="latest"; else TAG="branch-${TAG}"; fi
+          REF="${GITHUB_REF#refs/heads/}"
-          echo "tag=${TAG}" >> $GITHUB_OUTPUT
+          if [ -z "$REF" ]; then exit 1; fi
+
+          VER="$(git describe --tags 2>/dev/null)"
+          echo "version=${VER}" >> "$GITHUB_OUTPUT"
+          echo "branch=${REF}" >> "$GITHUB_OUTPUT"
+
+          echo 'tags<<EOF' >> "$GITHUB_OUTPUT"
+          if [[ "$REF" == "main" ]]; then
+            echo "${IMAGE}:latest" >> "$GITHUB_OUTPUT"
+          else
+            echo "${IMAGE}:branch-${REF}" >> "$GITHUB_OUTPUT"
+          fi
+          echo 'EOF' >> "$GITHUB_OUTPUT"
       - uses: docker/build-push-action@v4
         with:
           push: true
-          tags: |
+          tags: ${{ steps.info.outputs.tags }}
-            ghcr.io/postalserver/postal:${{ steps.tag.outputs.tag }}
-            ghcr.io/postalserver/postal:commit-${{ github.sha }}
           cache-from: type=gha
           cache-to: type=gha,mode=max
           target: full
+          platforms: linux/amd64
+          build-args: |
+            VERSION=${{ steps.info.outputs.version }}
+            BRANCH=${{ steps.info.outputs.branch }}
 
   publish-image:
     name: Publish Image
@@ -93,7 +114,8 @@ jobs:
     needs: [build, test, release-please]
     if: ${{ needs.release-please.outputs.release_created }}
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+      - uses: docker/setup-qemu-action@v3
       - uses: docker/setup-buildx-action@v2
       - uses: docker/login-action@v2
         with:
@@ -111,3 +133,4 @@ jobs:
           target: full
           build-args: |
             VERSION=${{ needs.release-please.outputs.version }}
+          platforms: linux/amd64

.gitignore

@@ -29,5 +29,11 @@ vendor/bundle
 
 Procfile.local
 VERSION
+BRANCH
 
 .rubocop-https*
+.env*
+
+node_modules
+yarn.lock
+

.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "2.3.2"
+  ".": "3.3.7"
 }

.rubocop.yml

@@ -1,11 +1,12 @@
 AllCops:
-  TargetRubyVersion: 3.0
+  TargetRubyVersion: 3.4
   NewCops: enable
   Exclude:
     - "bin/*"
     - "db/schema.rb"
     # Fixes missing gem exception when running Rubocop on GitHub Actions.
     - "vendor/bundle/**/*"
+    - lib/tasks/auto_annotate_models.rake
 
 # Always use double quotes
 Style/StringLiterals:
@@ -124,6 +125,9 @@ Style/MultilineBlockChain:
   Exclude:
     - "spec/**/*.rb"
 
+Style/TrailingCommaInArrayLiteral:
+  EnforcedStyleForMultiline: consistent_comma
+
 Metrics/AbcSize:
   Enabled: false
 
@@ -131,7 +135,7 @@ Style/FrozenStringLiteralComment:
   Enabled: true
   SafeAutoCorrect: true
 
-Naming/PredicateName:
+Naming/PredicatePrefix:
   Enabled: false
 
 Layout/LineLength:

.ruby-version

@@ -1,2 +1 @@
-3.2.1
+3.4.10
-

CHANGELOG.md

@@ -2,6 +2,288 @@
 
 This file contains all the latest changes and updates to Postal.
 
+## [3.3.7](https://github.com/postalserver/postal/compare/3.3.6...3.3.7) (2026-06-03)
+
+
+### Bug Fixes
+
+* **http:** prevent SSRF in outbound webhook and HTTP endpoint requests ([11c9814](https://github.com/postalserver/postal/commit/11c9814474f956619da35e8385ef7fab9f304de0))
+* **message-db:** prevent SQL injection via condition keys (GHSA-x2hq-rfpg-3xr5) ([4314a6e](https://github.com/postalserver/postal/commit/4314a6ec1e2812daa67dd20effd1db1769c1f8e8))
+* **specs:** stub IPv6 support in address guard IPv6 literal spec ([029bfe0](https://github.com/postalserver/postal/commit/029bfe098d9b8c0b5cafc49eac33e767f5748cd3))
+
+
+### Miscellaneous Chores
+
+* **deps:** upgrade rack & rails ([0445e5c](https://github.com/postalserver/postal/commit/0445e5c509870dfe9c16366c53dee3fc02ad3904))
+* update security vulnerability reporting instructions ([8be1e27](https://github.com/postalserver/postal/commit/8be1e27fec489ab659ef5e909f705932028b1694))
+
+## [3.3.6](https://github.com/postalserver/postal/compare/3.3.5...3.3.6) (2026-04-28)
+
+
+### Bug Fixes
+
+* **messages:** sandbox rendered email HTML as extra XSS defence ([cad2aa6](https://github.com/postalserver/postal/commit/cad2aa6808519a3ff25215f09f4966d9fa3bb372))
+
+
+### Miscellaneous Chores
+
+* ignore node modules and yarn.lock ([b611d57](https://github.com/postalserver/postal/commit/b611d577af79b8e1e75b6d47fa04d1ba03e34eec))
+
+
+### Code Refactoring
+
+* **auth:** tighten return_to validation ([84f4e20](https://github.com/postalserver/postal/commit/84f4e20f05db2d11b0144f95960c956f8221e657))
+* **helpers:** escape interpolated values in select options ([9243524](https://github.com/postalserver/postal/commit/924352403553dcfcc569876ca76c219493fac9d6))
+* **tracking:** remove unused src image proxy ([dca7f90](https://github.com/postalserver/postal/commit/dca7f90b9046247c0d953567be35921167e79d87))
+
+## [3.3.5](https://github.com/postalserver/postal/compare/3.3.4...3.3.5) (2026-02-01)
+
+
+### Bug Fixes
+
+* **deliveries:** escape delivery details to prevent HTML injection ([11419f9](https://github.com/postalserver/postal/commit/11419f99140e13688a9613cab3ee03f8d3cbae45))
+* **health_server:** use rackup handler instead of rack handler ([7c47422](https://github.com/postalserver/postal/commit/7c47422c865e738c4d6af0fed1cca4405288341f))
+* oidc scopes are invalid when concatenated ([#3332](https://github.com/postalserver/postal/issues/3332)) ([9c5f96a](https://github.com/postalserver/postal/commit/9c5f96ae90cf06dcd5db776806865752f667bd95))
+* typo in process logging ([#3212](https://github.com/postalserver/postal/issues/3212)) ([b7e5232](https://github.com/postalserver/postal/commit/b7e5232e077b3c9b7a999dcb6676fba0ec61458e))
+* typo in the credentials page ([fd3c7cc](https://github.com/postalserver/postal/commit/fd3c7ccdf6dc4ee0a76c9523cbd735159e4b8000))
+* update url for v2 config ([#3225](https://github.com/postalserver/postal/issues/3225)) ([e00098b](https://github.com/postalserver/postal/commit/e00098b8003cf37f2708f536871b3ade377aed2d))
+
+
+### Documentation
+
+* **process.rb:** add help about time unit used by metric ([#3339](https://github.com/postalserver/postal/issues/3339)) ([f5325c4](https://github.com/postalserver/postal/commit/f5325c49ff1152ad53eaaec98717ad3412d379ae))
+
+
+### Miscellaneous Chores
+
+* **deps:** upgrade puma, net-imap and other deps ([c03c44b](https://github.com/postalserver/postal/commit/c03c44b442a29aa9881c1e1aae60bead9776a6b6))
+* **dockerfile:** reduce container size ([86de372](https://github.com/postalserver/postal/commit/86de372382bd62bdd5d1372254f8817b0360bd56))
+* remove version from docker-compose.yml ([c78000c](https://github.com/postalserver/postal/commit/c78000ca8f2998aa04648f465060768db6467de6))
+* upgrade resolv to 0.6.2 ([d00d978](https://github.com/postalserver/postal/commit/d00d978872a96369544303d08f6a9d11cdf56b62))
+* upgrade to rails 7.1 and ruby 3.4 ([#3457](https://github.com/postalserver/postal/issues/3457)) ([ab6d443](https://github.com/postalserver/postal/commit/ab6d4430baa33a05f1aa66e776cc2a5bcaa0ede8))
+* upgrade uri gem to 1.0.3 ([f193b8e](https://github.com/postalserver/postal/commit/f193b8e77fc096382ab7aaa6a2c29641b4cb12df))
+
+## [3.3.4](https://github.com/postalserver/postal/compare/3.3.3...3.3.4) (2024-06-20)
+
+
+### Bug Fixes
+
+* fix `postal version` command ([4fa88ac](https://github.com/postalserver/postal/commit/4fa88acea0dececd0eae485506a2ad8268fbea59))
+* fix issue running message pruning task ([3a33e53](https://github.com/postalserver/postal/commit/3a33e53d843584757bb00898746aa059d7616db4))
+* raise NotImplementedError when no call method on a scheduled task ([2b0919c](https://github.com/postalserver/postal/commit/2b0919c1454eabea93db96f50ecbd8e36bb89f1f))
+
+## [3.3.3](https://github.com/postalserver/postal/compare/3.3.2...3.3.3) (2024-04-18)
+
+
+### Bug Fixes
+
+* **legacy-api:** allow _expansions to be provided as true to return all expansions ([39f704c](https://github.com/postalserver/postal/commit/39f704c256fc3e71a1dc009acc77796a1efffead)), closes [#2932](https://github.com/postalserver/postal/issues/2932)
+
+## [3.3.2](https://github.com/postalserver/postal/compare/3.3.1...3.3.2) (2024-03-21)
+
+
+### Code Refactoring
+
+* **versioning:** improve how current version and branch is determined and set ([07c6b31](https://github.com/postalserver/postal/commit/07c6b317f2b9dc04b6a8c88df1e6aa9e54597504))
+
+## [3.3.1](https://github.com/postalserver/postal/compare/3.3.0...3.3.1) (2024-03-21)
+
+
+### Bug Fixes
+
+* **smtp-sender:** ensure relays without a host are excluded ([3a56ec8](https://github.com/postalserver/postal/commit/3a56ec8a74950e0162d98f1af5f58a67a82d6455))
+* **smtp-sender:** fixes `SMTPSender.smtp_relays` ([b3264b9](https://github.com/postalserver/postal/commit/b3264b942776e254d3c351c94c435d172a514e18))
+
+
+### Miscellaneous Chores
+
+* **container:** add the branch name to the container ([bee5098](https://github.com/postalserver/postal/commit/bee509832edc151d97fe5bfc48c4973452873fc8))
+* **github-actions:** don't generate commit- tags ([d65bbe0](https://github.com/postalserver/postal/commit/d65bbe0579037c5df962a18134bc007f5159d7e5))
+* **github-actions:** don't run for dependabot or release-please PRs and fetch whole repo ([adaf2b0](https://github.com/postalserver/postal/commit/adaf2b07502e9ed91290873ad8465051c6fd814f))
+* **github-actions:** include a version string on branch-*/latest images ([64bc7dc](https://github.com/postalserver/postal/commit/64bc7dcf7c0a8e006ab6eb6e8b4a52ad5e7e6528))
+* **ui:** display branch in footer if present ([1823617](https://github.com/postalserver/postal/commit/18236171ebc398c157f2e61b15c7df9f91205284))
+
+
+### Code Refactoring
+
+* remove moonrope but maintain legacy API actions ([#2889](https://github.com/postalserver/postal/issues/2889)) ([4d9654d](https://github.com/postalserver/postal/commit/4d9654dac47d59c760be96388d0421de74d3e6ac))
+
+## [3.3.0](https://github.com/postalserver/postal/compare/3.2.2...3.3.0) (2024-03-18)
+
+
+### Features
+
+* **prometheus:** add `postal_message_queue_latency` metric ([ee8d829](https://github.com/postalserver/postal/commit/ee8d829a854f91e476167869cafe35c2d37bb314))
+* **worker:** allow number of threads to be configured ([7e2accc](https://github.com/postalserver/postal/commit/7e2acccd1ebd80750a3ebdb96cb5c36b5263cc24))
+* **worker:** scale connection pool with worker threads ([ea542a0](https://github.com/postalserver/postal/commit/ea542a0694b3465b04fd3ebc439837df414deb1e))
+
+
+### Bug Fixes
+
+* **message-dequeuer:** ability to disable batching ([4fcb9e9](https://github.com/postalserver/postal/commit/4fcb9e9a2e34be5aa4bdf13f0529f40e564b72b4))
+
+
+### Miscellaneous Chores
+
+* **config-docs:** update docs for latest oidc defaults ([364eba6](https://github.com/postalserver/postal/commit/364eba6c5fce2f08a36489f42856ad5024a2062c))
+* **config-docs:** update proxy protocol to mention v1 ([45dd8aa](https://github.com/postalserver/postal/commit/45dd8aaac56f15481cb7bf9081401cb28dc1e707))
+
+## [3.2.2](https://github.com/postalserver/postal/compare/3.2.1...3.2.2) (2024-03-14)
+
+
+### Bug Fixes
+
+* don't use authentication on org & server deletion ([be45652](https://github.com/postalserver/postal/commit/be456523dd3aacb5c3eb45c9261da97ebffe603c))
+* **smtp-server:** fixes proxy protocol ([9240612](https://github.com/postalserver/postal/commit/92406129cfcf1a06499a6f5aa18c73f1d6195793))
+
+
+### Miscellaneous Chores
+
+* allow config location message to be suppressed ([f760cdb](https://github.com/postalserver/postal/commit/f760cdb5a1d53e9c30ee495d129cbf12603a3cbd))
+* hide further config messages ([1c67f72](https://github.com/postalserver/postal/commit/1c67f72209c93404d7024ce3d15f6f54f2d707c4))
+* suppress config location on default-dkim-record ([aa76aae](https://github.com/postalserver/postal/commit/aa76aae2322af41af1bd60cfe1d69a11ac76324e))
+
+
+### Tests
+
+* add tests for the legacy API ([3d208d6](https://github.com/postalserver/postal/commit/3d208d632f4fc8a4adbfdb2bf4b377271eae6692))
+
+## [3.2.1](https://github.com/postalserver/postal/compare/3.2.0...3.2.1) (2024-03-13)
+
+
+### Bug Fixes
+
+* fixes `postal default-dkim-record` ([58dddeb](https://github.com/postalserver/postal/commit/58dddebeb81dc6fab945d2b10a91588eddc471c2))
+
+## [3.2.0](https://github.com/postalserver/postal/compare/3.1.1...3.2.0) (2024-03-13)
+
+
+### Features
+
+* add sha256 signatures to outgoing http requests ([#2874](https://github.com/postalserver/postal/issues/2874)) ([96d7365](https://github.com/postalserver/postal/commit/96d73653d7cb4dde1fbe74ccb3596147ef8cd9ed))
+* automatically remove queued messages with stale locks ([#2872](https://github.com/postalserver/postal/issues/2872)) ([d84152e](https://github.com/postalserver/postal/commit/d84152eb5df6f963426d6ba8d02d39b3c146c8a5))
+* openid connect support ([#2873](https://github.com/postalserver/postal/issues/2873)) ([5ed94f6](https://github.com/postalserver/postal/commit/5ed94f6f855735aa00544b2574dfb9e65d559a38))
+
+
+### Bug Fixes
+
+* **smtp-server:** add additional information to cram-md5 log entries ([9982bb8](https://github.com/postalserver/postal/commit/9982bb8c31ee4885d188666e2e8afdc218528df7))
+
+
+### Styles
+
+* **rubocop:** Style/TrailingCommaInArrayLiteral ([4e13577](https://github.com/postalserver/postal/commit/4e13577891dc827244abc6bf6d9ab4ee45860556))
+
+
+### Miscellaneous Chores
+
+* regenerate config docs ([5d8213a](https://github.com/postalserver/postal/commit/5d8213a98735f07fdf1700c7d01597654f41dbd0))
+
+## [3.1.1](https://github.com/postalserver/postal/compare/3.1.0...3.1.1) (2024-03-08)
+
+
+### Bug Fixes
+
+* don't override paths in dockerfile ([9399e32](https://github.com/postalserver/postal/commit/9399e3223467cdacd010e70b58ad6093e128213d))
+
+
+### Tests
+
+* **smtp-sender:** add more tests for AUTH LOGIN ([22dcd49](https://github.com/postalserver/postal/commit/22dcd4901f188915cf4b3c758c6f2fc637a4e1e3))
+
+## [3.1.0](https://github.com/postalserver/postal/compare/3.0.2...3.1.0) (2024-03-06)
+
+
+### Features
+
+* configurable trusted proxies for web requests ([3785c99](https://github.com/postalserver/postal/commit/3785c998513c634d225b489ccb43e926ce3f270a))
+
+
+### Bug Fixes
+
+* **message-dequeuer:** ensure SMTP endpoints are sent to SMTP sender appropriately ([e2d642c](https://github.com/postalserver/postal/commit/e2d642c0cbf443550886d90abc3a6edf3e4bc4fc)), closes [#2853](https://github.com/postalserver/postal/issues/2853)
+* **smtp-server:** listen on all interfaces by default ([d1e5b68](https://github.com/postalserver/postal/commit/d1e5b68200ea4b9710cc8714afb3271bad1f4f66)), closes [#2852](https://github.com/postalserver/postal/issues/2852)
+* **smtp-server:** remove ::ffff: from the start of ipv4 addresses ([0dc7359](https://github.com/postalserver/postal/commit/0dc7359431001c9ef1222913f8d1344093397596))
+* **smtp-server:** reset ansi sequence after logging ([9bf6152](https://github.com/postalserver/postal/commit/9bf6152060ffb8b611b66818c1d1ac7c929b7ffe))
+* **ui:** fixes typo on queue page ([33513a7](https://github.com/postalserver/postal/commit/33513a77c0df24d832ab7ed5237d68e2b1bde887))
+* **web-server:** allow for trusted proxies not be set ([4e1deb2](https://github.com/postalserver/postal/commit/4e1deb2d2aeb61d9dddb3729916411c94e73c1c6))
+
+
+### Styles
+
+* **rubocop:** use _ when not using a variable in helm config exporter ([2c20ba6](https://github.com/postalserver/postal/commit/2c20ba65f64ccb0f8174e3f523dedb3806478782))
+
+## [3.0.2](https://github.com/postalserver/postal/compare/3.0.1...3.0.2) (2024-03-05)
+
+
+### Bug Fixes
+
+* default to listening on all addresses when using legacy config ([48f6494](https://github.com/postalserver/postal/commit/48f6494240eb0374d5f865425b362e6f306b2653))
+
+
+### Miscellaneous Chores
+
+* removing arm64 support until we can get a reasonable build pipeline sorted ([e8e44f5](https://github.com/postalserver/postal/commit/e8e44f54b09426c8a04e466bc037851a0833d124))
+
+## [3.0.1](https://github.com/postalserver/postal/compare/3.0.0...3.0.1) (2024-03-05)
+
+
+### Bug Fixes
+
+* fix issue with sending mail when smtp relays are configured ([6dd6e29](https://github.com/postalserver/postal/commit/6dd6e29929c70eaa8b9d3b33c184996b0b6abb82))
+
+## [3.0.0](https://github.com/postalserver/postal/compare/2.3.2...3.0.0) (2024-03-04)
+
+This version of Postal introduces a number of larger changes. Please be sure to follow the [upgrade guide](https://docs.postalserver.io/getting-started/upgrade-to-v3) when upgrading to Postal v3. Highlights include:
+
+* Removal of RabbitMQ dependency
+* Removal of `cron` and `requeuer` processes
+* Improved logging
+* Improved configuration
+* Adds prometheus metric exporters for workers and SMTP servers
+* Only accepted RFC-compliant end-of-DATA sequences (to avoid SMTP smuggling)
+
+### Features
+
+* add health server and prometheus metrics to worker ([a2eb70e](https://github.com/postalserver/postal/commit/a2eb70e))
+* add option to delay starting processes until all migrations are run ([1c5ff5a](https://github.com/postalserver/postal/commit/1c5ff5a))
+* add prometheus metrics to smtp server ([2e7b36c](https://github.com/postalserver/postal/commit/2e7b36c))
+* add prometheus metrics to worker ([bea7450](https://github.com/postalserver/postal/commit/bea7450))
+* more consistent logging ([044058d](https://github.com/postalserver/postal/commit/044058d))
+* new background work process ([dc8e895](https://github.com/postalserver/postal/commit/dc8e895))
+* new configuration system (and schema) (#2819) ([0163ac3](https://github.com/postalserver/postal/commit/0163ac3))
+* only accept RFC-compliant End-of-DATA sequence ([0140dc4](https://github.com/postalserver/postal/commit/0140dc4))
+* add "postal:update" task ([b35eea6](https://github.com/postalserver/postal/commit/b35eea6338f1888bfac2ed377d0a412680483e90))
+* add helm env var config exporter ([8938988](https://github.com/postalserver/postal/commit/893898835dcd9684ae3401549389b173a3feb1fb))
+* include list-unsubscribe-post header in dkim signatures ([e1bae60](https://github.com/postalserver/postal/commit/e1bae60768c4cf151d5a6a141985c78753dce02d)), closes [#2789](https://github.com/postalserver/postal/issues/2789) [#2788](https://github.com/postalserver/postal/issues/2788)
+* support for additional SMTP client options ([0daa667](https://github.com/postalserver/postal/commit/0daa667b55fd9b948da643d37ec438e341809369))
+
+
+### Bug Fixes
+
+* fixes potential issue if machine hostname cannot be determined ([0fcf778](https://github.com/postalserver/postal/commit/0fcf778))
+* raise an error if MX lookup times out during sending ([fadca88](https://github.com/postalserver/postal/commit/fadca88)), closes [#2833](https://github.com/postalserver/postal/issues/2833)
+* set correct component in health server log lines ([a7a9a18](https://github.com/postalserver/postal/commit/a7a9a18))
+* translate unicode domain names to Punycode for compatibility with DNS (#2823) ([be0df7b](https://github.com/postalserver/postal/commit/be0df7b))
+* remove user email verification ([e05f0b3](https://github.com/postalserver/postal/commit/e05f0b3616da8b962b763c48a2139882fd88047a))
+* unescape in URLs which are stored for tracking ([1da1182](https://github.com/postalserver/postal/commit/1da1182c23e9673d8f109d8ed29e80983cdccabf)), closes [#2568](https://github.com/postalserver/postal/issues/2568) [#907](https://github.com/postalserver/postal/issues/907) [#115](https://github.com/postalserver/postal/issues/115)
+* update wording about tracking domain cnames ([0d3eccb](https://github.com/postalserver/postal/commit/0d3eccb368630b4fd21bd858a7829ec00c35f153)), closes [#2808](https://github.com/postalserver/postal/issues/2808)
+
+
+### Documentation
+
+* add message_db.encoding to config docs ([0c1f925](https://github.com/postalserver/postal/commit/0c1f925))
+* add new repo readme welcome image ([afa1726](https://github.com/postalserver/postal/commit/afa1726))
+* add quick contributing instructions ([8d21adc](https://github.com/postalserver/postal/commit/8d21adc))
+* update SECURITY policy ([cfc1c9b](https://github.com/postalserver/postal/commit/cfc1c9b))
+* update docs for how IP address allocation works ([07eb152](https://github.com/postalserver/postal/commit/07eb152)), closes [#2209](https://github.com/postalserver/postal/issues/2209)
+
+### Miscellaneous Chores
+
+* upgrade bundler to 2.5.6 ([1ae8ef6](https://github.com/postalserver/postal/commit/1ae8ef6))
+* upgrade rails to 7.0 and other dependencies ([ecd09a2](https://github.com/postalserver/postal/commit/ecd09a2))
+* upgrade ruby to 3.2.2 and nodejs to 20.x ([72715fe](https://github.com/postalserver/postal/commit/72715fe))
+
 ## [2.3.2](https://github.com/postalserver/postal/compare/2.3.1...2.3.2) (2024-03-01)
 
 

CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing to Postal
+
+This doc explains how to go about running Postal in development to allow you to make contributions to the project.
+
+## Dependencies
+
+You will need a MySQL database server to get started. Postal needs to be able to make databases within that server whenever new mail servers are created so the permissions that you use should be suitable for that.
+
+You'll also need Ruby. Postal currently uses the Ruby version in `.ruby-version`. Install that using whichever version manager takes your fancy - rbenv, asdf, rvm etc.
+
+## Clone
+
+You'll need to clone the repository
+
+```
+git clone git@github.com:postalserver/postal
+```
+
+Once cloned, you can install the Ruby dependencies using bundler.
+
+```
+bundle install
+```
+
+## Configuration
+
+Configuration is handled using a config file. This lives in `config/postal/postal.yml`. An example configuration file is provided in `config/examples/development.yml`. This example is for development use only and not an example for production use.
+
+You'll also need a key for signing. You can generate one of these like this:
+
+```
+openssl genrsa -out config/postal/signing.key 2048
+```
+
+If you're running the tests (and you probably should be), you'll find an example file for test configuration in `config/examples/test.yml`. This should be placed in `config/postal/postal.test.yml` with the appropriate values.
+
+If you prefer, you can configure Postal using environment variables. These should be placed in `.env` or `.env.test` as apprpriate.
+
+## Running
+
+The neatest way to run postal is to ensure that `./bin` is your `$PATH` and then use one of the following commands.
+
+* `bin/dev` - will run all components of the application using Foreman
+* `bin/postal` - will run the Postal binary providing access to running individual components or other tools.
+
+## Database initialization
+
+Use the commands below to initialize your database and make your first user.
+
+```
+postal initialize
+postal make-user
+```

Dockerfile

@@ -1,22 +1,25 @@
-FROM ruby:3.2.1-bullseye AS base
+FROM ruby:3.4.10-slim-bookworm AS base
 
 SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 RUN apt-get update \
-  && apt-get install -y --no-install-recommends \
+  && apt-get install --no-install-recommends -y curl \
-  software-properties-common dirmngr apt-transport-https \
+  && apt-get clean \
-  && (curl -sL https://deb.nodesource.com/setup_14.x | bash -) \
   && rm -rf /var/lib/apt/lists/*
 
+RUN (curl -sL https://deb.nodesource.com/setup_20.x | bash -)
+
 # Install main dependencies
 RUN apt-get update && \
   apt-get install -y --no-install-recommends \
-  build-essential  \
+    build-essential  \
-  netcat \
+    netcat-openbsd \
-  curl \
+    libmariadb-dev \
-  libmariadb-dev \
+    libcap2-bin \
-  libcap2-bin \
+    nano \
-  nano \
+    libyaml-dev \
-  nodejs
+    nodejs \
+  && apt-get clean \
+  && rm -rf /var/lib/apt/lists/*
 
 RUN setcap 'cap_net_bind_service=+ep' /usr/local/bin/ruby
 
@@ -31,23 +34,25 @@ RUN mkdir -p /opt/postal/app /opt/postal/config
 WORKDIR /opt/postal/app
 
 # Install bundler
-RUN gem install bundler -v 2.4.9 --no-doc
+RUN gem install bundler -v 2.7.2 --no-doc
 
 # Install the latest and active gem dependencies and re-run
 # the appropriate commands to handle installs.
-COPY Gemfile Gemfile.lock ./
+COPY --chown=postal Gemfile Gemfile.lock ./
-RUN bundle config set force_ruby_platform true && bundle install -j 4
+RUN bundle install
 
 # Copy the application (and set permissions)
 COPY ./docker/wait-for.sh /docker-entrypoint.sh
 COPY --chown=postal . .
 
 # Export the version
-ARG VERSION=unspecified
+ARG VERSION
-RUN echo $VERSION > VERSION
+ARG BRANCH
+RUN if [ "$VERSION" != "" ]; then echo $VERSION > VERSION; fi \
+  && if [ "$BRANCH" != "" ]; then echo $BRANCH > BRANCH; fi
 
-# Set the path to the config
+# Set paths for when running in a container
-ENV POSTAL_CONFIG_ROOT=/config
+ENV POSTAL_CONFIG_FILE_PATH=/config/postal.yml
 
 # Set the CMD
 ENTRYPOINT [ "/docker-entrypoint.sh" ]
@@ -59,5 +64,5 @@ FROM base AS ci
 # full target - default if no --target option is given
 FROM base AS full
 
-RUN POSTAL_SKIP_CONFIG_CHECK=1 RAILS_GROUPS=assets bundle exec rake assets:precompile
+RUN RAILS_GROUPS=assets bundle exec rake assets:precompile
 RUN touch /opt/postal/app/public/assets/.prebuilt

Gemfile

@@ -1,57 +1,72 @@
 # frozen_string_literal: true
 
 source "https://rubygems.org"
+gem "abbrev"
 gem "authie"
 gem "autoprefixer-rails"
-gem "basic_ssl"
 gem "bcrypt"
-gem "bunny"
-gem "changey"
 gem "chronic"
-gem "clockwork"
+gem "domain_name"
-gem "dotenv-rails"
+gem "dotenv"
-gem "dynamic_form"
+gem "execjs"
-gem "encrypto_signo"
-gem "execjs", "~> 2.7", "< 2.8"
-gem "foreman"
 gem "gelf"
 gem "haml"
 gem "hashie"
 gem "highline", require: false
+# json 3.0 removed the `quirks_mode` option, which Active Support 7.2 still
+# passes in both its JSON encoder and decoder. Nothing else constrains json, so
+# without this pin Bundler resolves to 3.x and every `to_json` call raises
+# ArgumentError. Active Support dropped those calls in Rails 8.1, so this pin
+# can go when we move to 8.1.
+gem "json", "< 3"
 gem "jwt"
 gem "kaminari"
+gem "klogger-logger"
+gem "konfig-config", "~> 3.0"
+gem "logger"
 gem "mail"
-gem "moonrope"
+gem "mutex_m"
 gem "mysql2"
 gem "nifty-utils"
 gem "nilify_blanks"
 gem "nio4r"
+gem "ostruct"
+gem "prometheus-client"
 gem "puma"
-gem "rails", "= 6.1.7.6"
+gem "rackup"
-gem "resolv", "~> 0.2.1"
+gem "rails", "= 7.2.3.2"
+gem "resolv"
 gem "secure_headers"
 gem "sentry-rails"
-gem "sentry-ruby"
 gem "turbolinks", "~> 5"
+gem "webrick"
 
-group :development, :assets do
+group :oidc do
+  # These are gems which are needed for OpenID connect. They are only required by the application
+  # when OIDC is enabled in the Postal configuration.
+  gem "omniauth_openid_connect"
+  gem "omniauth-rails_csrf_protection"
+end
+
+group :development, :test, :assets do
   gem "coffee-rails", "~> 5.0"
   gem "jquery-rails"
   gem "sass-rails"
   gem "uglifier", ">= 1.3.0"
 end
 
-group :development, :test do
-  gem "byebug"
-end
-
 group :development do
   gem "annotate"
-  gem "database_cleaner", require: false
-  gem "factory_bot_rails", require: false
-  gem "rspec", require: false
-  gem "rspec-rails", require: false
   gem "rubocop"
   gem "rubocop-rails"
+end
+
+group :test do
+  gem "database_cleaner-active_record"
+  gem "factory_bot_rails"
+  gem "rspec"
+  gem "rspec-rails"
+  gem "shoulda-matchers"
   gem "timecop"
+  gem "webmock"
 end

Gemfile.lock

@@ -1,87 +1,100 @@
 GEM
   remote: https://rubygems.org/
   specs:
-    actioncable (6.1.7.6)
+    abbrev (0.1.2)
-      actionpack (= 6.1.7.6)
+    actioncable (7.2.3.2)
-      activesupport (= 6.1.7.6)
+      actionpack (= 7.2.3.2)
+      activesupport (= 7.2.3.2)
       nio4r (~> 2.0)
       websocket-driver (>= 0.6.1)
-    actionmailbox (6.1.7.6)
+      zeitwerk (~> 2.6)
-      actionpack (= 6.1.7.6)
+    actionmailbox (7.2.3.2)
-      activejob (= 6.1.7.6)
+      actionpack (= 7.2.3.2)
-      activerecord (= 6.1.7.6)
+      activejob (= 7.2.3.2)
-      activestorage (= 6.1.7.6)
+      activerecord (= 7.2.3.2)
-      activesupport (= 6.1.7.6)
+      activestorage (= 7.2.3.2)
-      mail (>= 2.7.1)
+      activesupport (= 7.2.3.2)
-    actionmailer (6.1.7.6)
+      mail (>= 2.8.0)
-      actionpack (= 6.1.7.6)
+    actionmailer (7.2.3.2)
-      actionview (= 6.1.7.6)
+      actionpack (= 7.2.3.2)
-      activejob (= 6.1.7.6)
+      actionview (= 7.2.3.2)
-      activesupport (= 6.1.7.6)
+      activejob (= 7.2.3.2)
-      mail (~> 2.5, >= 2.5.4)
+      activesupport (= 7.2.3.2)
-      rails-dom-testing (~> 2.0)
+      mail (>= 2.8.0)
-    actionpack (6.1.7.6)
+      rails-dom-testing (~> 2.2)
-      actionview (= 6.1.7.6)
+    actionpack (7.2.3.2)
-      activesupport (= 6.1.7.6)
+      actionview (= 7.2.3.2)
-      rack (~> 2.0, >= 2.0.9)
+      activesupport (= 7.2.3.2)
-      rack-test (>= 0.6.3)
+      cgi
-      rails-dom-testing (~> 2.0)
-      rails-html-sanitizer (~> 1.0, >= 1.2.0)
-    actiontext (6.1.7.6)
-      actionpack (= 6.1.7.6)
-      activerecord (= 6.1.7.6)
-      activestorage (= 6.1.7.6)
-      activesupport (= 6.1.7.6)
       nokogiri (>= 1.8.5)
-    actionview (6.1.7.6)
+      racc
-      activesupport (= 6.1.7.6)
+      rack (>= 2.2.4, < 3.3)
+      rack-session (>= 1.0.1)
+      rack-test (>= 0.6.3)
+      rails-dom-testing (~> 2.2)
+      rails-html-sanitizer (~> 1.6)
+      useragent (~> 0.16)
+    actiontext (7.2.3.2)
+      actionpack (= 7.2.3.2)
+      activerecord (= 7.2.3.2)
+      activestorage (= 7.2.3.2)
+      activesupport (= 7.2.3.2)
+      globalid (>= 0.6.0)
+      nokogiri (>= 1.8.5)
+    actionview (7.2.3.2)
+      activesupport (= 7.2.3.2)
       builder (~> 3.1)
-      erubi (~> 1.4)
+      cgi
-      rails-dom-testing (~> 2.0)
+      erubi (~> 1.11)
-      rails-html-sanitizer (~> 1.1, >= 1.2.0)
+      rails-dom-testing (~> 2.2)
-    activejob (6.1.7.6)
+      rails-html-sanitizer (~> 1.6)
-      activesupport (= 6.1.7.6)
+    activejob (7.2.3.2)
+      activesupport (= 7.2.3.2)
       globalid (>= 0.3.6)
-    activemodel (6.1.7.6)
+    activemodel (7.2.3.2)
-      activesupport (= 6.1.7.6)
+      activesupport (= 7.2.3.2)
-    activerecord (6.1.7.6)
+    activerecord (7.2.3.2)
-      activemodel (= 6.1.7.6)
+      activemodel (= 7.2.3.2)
-      activesupport (= 6.1.7.6)
+      activesupport (= 7.2.3.2)
-    activestorage (6.1.7.6)
+      timeout (>= 0.4.0)
-      actionpack (= 6.1.7.6)
+    activestorage (7.2.3.2)
-      activejob (= 6.1.7.6)
+      actionpack (= 7.2.3.2)
-      activerecord (= 6.1.7.6)
+      activejob (= 7.2.3.2)
-      activesupport (= 6.1.7.6)
+      activerecord (= 7.2.3.2)
+      activesupport (= 7.2.3.2)
       marcel (~> 1.0)
-      mini_mime (>= 1.1.0)
+    activesupport (7.2.3.2)
-    activesupport (6.1.7.6)
+      base64
-      concurrent-ruby (~> 1.0, >= 1.0.2)
+      benchmark (>= 0.3)
+      bigdecimal
+      concurrent-ruby (~> 1.0, >= 1.3.1)
+      connection_pool (>= 2.2.5)
+      drb
       i18n (>= 1.6, < 2)
-      minitest (>= 5.1)
+      logger (>= 1.4.2)
-      tzinfo (~> 2.0)
+      minitest (>= 5.1, < 6)
-      zeitwerk (~> 2.3)
+      securerandom (>= 0.3)
-    amq-protocol (2.3.2)
+      tzinfo (~> 2.0, >= 2.0.5)
+    addressable (2.9.0)
+      public_suffix (>= 2.0.2, < 8.0)
+    aes_key_wrap (1.1.0)
     annotate (3.2.0)
       activerecord (>= 3.2, < 8.0)
       rake (>= 10.4, < 14.0)
-    ast (2.4.2)
+    ast (2.4.3)
-    authie (3.4.0)
+    attr_required (1.0.2)
-      secure_random_string
+    authie (5.0.0)
-    autoprefixer-rails (10.4.13.0)
+      activerecord (>= 6.1, < 9.0)
+    autoprefixer-rails (10.4.21.0)
       execjs (~> 2)
-    basic_ssl (1.0.3)
+    base64 (0.3.0)
-    bcrypt (3.1.18)
+    bcrypt (3.1.22)
-    builder (3.2.4)
+    benchmark (0.5.0)
-    bunny (2.20.3)
+    bigdecimal (4.1.3)
-      amq-protocol (~> 2.3, >= 2.3.1)
+    bindata (3.0.0)
-      sorted_set (~> 1, >= 1.0.2)
+    builder (3.3.0)
-    byebug (11.1.3)
+    cgi (0.5.2)
-    changey (1.1.0)
-      activerecord (>= 4.2, < 7)
     chronic (0.10.2)
-    clockwork (3.0.2)
-      activesupport
-      tzinfo
     coffee-rails (5.0.0)
       coffee-script (>= 2.2.0)
       railties (>= 5.2.0)
@@ -89,52 +102,75 @@ GEM
       coffee-script-source
       execjs
     coffee-script-source (1.12.2)
-    concurrent-ruby (1.2.3)
+    concurrent-ruby (1.3.8)
-    crass (1.0.6)
+    connection_pool (3.0.2)
-    database_cleaner (2.0.2)
+    crack (1.0.1)
-      database_cleaner-active_record (>= 2, < 3)
+      bigdecimal
-    database_cleaner-active_record (2.1.0)
+      rexml
+    crass (1.0.7)
+    database_cleaner-active_record (2.2.2)
       activerecord (>= 5.a)
-      database_cleaner-core (~> 2.0.0)
+      database_cleaner-core (~> 2.0)
-    database_cleaner-core (2.0.1)
+    database_cleaner-core (2.1.0)
-    date (3.3.3)
+    date (3.5.1)
-    deep_merge (1.2.2)
+    diff-lcs (1.6.2)
-    diff-lcs (1.5.0)
+    domain_name (0.6.20260907)
-    dotenv (2.8.1)
+    dotenv (3.2.0)
-    dotenv-rails (2.8.1)
+    drb (2.2.3)
-      dotenv (= 2.8.1)
+    erb (6.0.7)
-      railties (>= 3.2)
+    erubi (1.13.1)
-    dynamic_form (1.3.1)
+    execjs (2.10.2)
-      actionview (> 5.2.0)
+      json (>= 2)
-      activemodel (> 5.2.0)
+    factory_bot (6.6.0)
-    encrypto_signo (1.0.0)
+      activesupport (>= 6.1.0)
-    erubi (1.12.0)
+    factory_bot_rails (6.5.1)
-    execjs (2.7.0)
+      factory_bot (~> 6.5)
-    factory_bot (6.4.6)
+      railties (>= 6.1.0)
-      activesupport (>= 5.0.0)
+    faraday (2.14.3)
-    factory_bot_rails (6.4.3)
+      faraday-net_http (>= 2.0, < 3.5)
-      factory_bot (~> 6.4)
+      json
-      railties (>= 5.0.0)
+      logger
+    faraday-follow_redirects (0.5.0)
+      faraday (>= 1, < 3)
+    faraday-net_http (3.4.4)
+      net-http (~> 0.5)
     ffi (1.15.5)
-    foreman (0.87.2)
     gelf (3.1.0)
       json
-    globalid (1.2.1)
+    globalid (1.4.0)
       activesupport (>= 6.1)
-    haml (6.1.1)
+    haml (7.5.1)
+      prism (>= 1.1.0)
       temple (>= 0.8.2)
       thor
       tilt
-    hashie (5.0.0)
+    hashdiff (1.2.1)
-    highline (2.1.0)
+    hashie (5.1.0)
-    i18n (1.14.1)
+      logger
+    highline (3.1.2)
+      reline
+    i18n (1.15.2)
       concurrent-ruby (~> 1.0)
-    jquery-rails (4.5.1)
+    io-console (0.9.2)
+    irb (1.18.0)
+      pp (>= 0.6.0)
+      prism (>= 1.3.0)
+      rdoc (>= 4.0.0)
+      reline (>= 0.4.2)
+    jquery-rails (4.6.1)
       rails-dom-testing (>= 1, < 3)
       railties (>= 4.2.0)
       thor (>= 0.14, < 2.0)
-    json (2.6.3)
+    json (2.21.2)
-    jwt (2.7.0)
+    json-jwt (1.17.2)
+      activesupport (>= 4.2)
+      aes_key_wrap
+      base64
+      bindata
+      faraday (~> 2.0)
+      faraday-follow_redirects
+    jwt (3.3.0)
+      base64
     kaminari (1.2.2)
       activesupport (>= 4.1.0)
       kaminari-actionview (= 1.2.2)
@@ -147,125 +183,200 @@ GEM
       activerecord
       kaminari-core (= 1.2.2)
     kaminari-core (1.2.2)
-    loofah (2.22.0)
+    klogger-logger (1.5.0)
+      concurrent-ruby (>= 1.0, < 2.0)
+      json
+      rouge (>= 3.30, < 5.0)
+    konfig-config (3.0.0)
+      hashie
+    language_server-protocol (3.17.0.6)
+    lint_roller (1.1.0)
+    logger (1.7.0)
+    loofah (2.25.2)
       crass (~> 1.0.2)
       nokogiri (>= 1.12.0)
-    mail (2.8.1)
+    mail (2.9.1)
+      logger
       mini_mime (>= 0.1.1)
       net-imap
       net-pop
       net-smtp
-    marcel (1.0.2)
+    marcel (1.2.1)
-    method_source (1.0.0)
+    mini_mime (1.1.5)
-    mini_mime (1.1.2)
+    minitest (5.27.0)
-    mini_portile2 (2.8.5)
+    mutex_m (0.3.0)
-    minitest (5.22.2)
+    mysql2 (0.5.7)
-    moonrope (2.0.2)
+      bigdecimal
-      deep_merge (~> 1.0)
+    net-http (0.9.1)
-      json
+      uri (>= 0.11.1)
-      rack (>= 1.4)
+    net-imap (0.6.7)
-    mysql2 (0.5.5)
-    net-imap (0.3.4)
       date
       net-protocol
     net-pop (0.1.2)
       net-protocol
-    net-protocol (0.2.1)
+    net-protocol (0.3.0)
       timeout
-    net-smtp (0.3.3)
+    net-smtp (0.5.1)
       net-protocol
     nifty-utils (1.1.7)
     nilify_blanks (1.4.0)
       activerecord (>= 4.0.0)
       activesupport (>= 4.0.0)
-    nio4r (2.7.0)
+    nio4r (2.7.5)
-    nokogiri (1.16.2)
+    nokogiri (1.19.4-aarch64-linux-gnu)
-      mini_portile2 (~> 2.8.2)
       racc (~> 1.4)
-    nokogiri (1.16.2-arm64-darwin)
+    nokogiri (1.19.4-arm64-darwin)
       racc (~> 1.4)
-    nokogiri (1.16.2-x86_64-linux)
+    nokogiri (1.19.4-x86_64-darwin)
       racc (~> 1.4)
-    parallel (1.22.1)
+    nokogiri (1.19.4-x86_64-linux-gnu)
-    parser (3.2.1.1)
+      racc (~> 1.4)
+    omniauth (2.1.4)
+      hashie (>= 3.4.6)
+      logger
+      rack (>= 2.2.3)
+      rack-protection
+    omniauth-rails_csrf_protection (2.0.1)
+      actionpack (>= 4.2)
+      omniauth (~> 2.0)
+    omniauth_openid_connect (0.8.0)
+      omniauth (>= 1.9, < 3)
+      openid_connect (~> 2.2)
+    openid_connect (2.5.0)
+      activemodel
+      attr_required (>= 1.0.0)
+      faraday (~> 2.0)
+      faraday-follow_redirects
+      json-jwt (>= 1.16)
+      mail
+      rack-oauth2 (~> 2.2)
+      swd (~> 2.0)
+      tzinfo
+      validate_url
+      webfinger (~> 2.0)
+    ostruct (0.6.3)
+    parallel (2.2.0)
+    parser (3.3.12.0)
       ast (~> 2.4.1)
-    puma (6.4.2)
+      racc
+    pp (0.6.4)
+      prettyprint
+    prettyprint (0.2.0)
+    prism (1.9.0)
+    prometheus-client (5.0.0)
+      base64
+    public_suffix (7.0.5)
+    puma (8.0.2)
       nio4r (~> 2.0)
-    racc (1.7.3)
+    racc (1.8.1)
-    rack (2.2.8)
+    rack (3.2.7)
-    rack-test (2.1.0)
+    rack-oauth2 (2.3.0)
+      activesupport
+      attr_required
+      faraday (~> 2.0)
+      faraday-follow_redirects
+      json-jwt (>= 1.11.0)
+      rack (>= 2.1.0)
+    rack-protection (4.2.1)
+      base64 (>= 0.1.0)
+      logger (>= 1.6.0)
+      rack (>= 3.0.0, < 4)
+    rack-session (2.1.2)
+      base64 (>= 0.1.0)
+      rack (>= 3.0.0)
+    rack-test (2.2.0)
       rack (>= 1.3)
-    rails (6.1.7.6)
+    rackup (2.3.1)
-      actioncable (= 6.1.7.6)
+      rack (>= 3)
-      actionmailbox (= 6.1.7.6)
+    rails (7.2.3.2)
-      actionmailer (= 6.1.7.6)
+      actioncable (= 7.2.3.2)
-      actionpack (= 6.1.7.6)
+      actionmailbox (= 7.2.3.2)
-      actiontext (= 6.1.7.6)
+      actionmailer (= 7.2.3.2)
-      actionview (= 6.1.7.6)
+      actionpack (= 7.2.3.2)
-      activejob (= 6.1.7.6)
+      actiontext (= 7.2.3.2)
-      activemodel (= 6.1.7.6)
+      actionview (= 7.2.3.2)
-      activerecord (= 6.1.7.6)
+      activejob (= 7.2.3.2)
-      activestorage (= 6.1.7.6)
+      activemodel (= 7.2.3.2)
-      activesupport (= 6.1.7.6)
+      activerecord (= 7.2.3.2)
+      activestorage (= 7.2.3.2)
+      activesupport (= 7.2.3.2)
       bundler (>= 1.15.0)
-      railties (= 6.1.7.6)
+      railties (= 7.2.3.2)
-      sprockets-rails (>= 2.0.0)
+    rails-dom-testing (2.3.0)
-    rails-dom-testing (2.2.0)
       activesupport (>= 5.0.0)
       minitest
       nokogiri (>= 1.6)
-    rails-html-sanitizer (1.6.0)
+    rails-html-sanitizer (1.7.1)
-      loofah (~> 2.21)
+      loofah (~> 2.25, >= 2.25.2)
-      nokogiri (~> 1.14)
+      nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0)
-    railties (6.1.7.6)
+    railties (7.2.3.2)
-      actionpack (= 6.1.7.6)
+      actionpack (= 7.2.3.2)
-      activesupport (= 6.1.7.6)
+      activesupport (= 7.2.3.2)
-      method_source
+      cgi
+      irb (~> 1.13)
+      rackup (>= 1.0.0)
       rake (>= 12.2)
-      thor (~> 1.0)
+      thor (~> 1.0, >= 1.2.2)
+      tsort (>= 0.2)
+      zeitwerk (~> 2.6)
     rainbow (3.1.1)
-    rake (13.1.0)
+    rake (13.4.2)
-    rbtree (0.4.6)
+    rbs (4.2.0)
-    regexp_parser (2.7.0)
+      logger
-    resolv (0.2.2)
+      prism (>= 1.6.0)
-    rexml (3.2.5)
+      tsort
-    rspec (3.12.0)
+    rdoc (8.0.0)
-      rspec-core (~> 3.12.0)
+      erb
-      rspec-expectations (~> 3.12.0)
+      prism (>= 1.6.0)
-      rspec-mocks (~> 3.12.0)
+      rbs (>= 4.0.0)
-    rspec-core (3.12.1)
+      tsort
-      rspec-support (~> 3.12.0)
+    regexp_parser (2.12.0)
-    rspec-expectations (3.12.2)
+    reline (0.7.0)
+      io-console (~> 0.5)
+    resolv (0.7.2)
+    rexml (3.4.4)
+    rouge (4.7.0)
+    rspec (3.13.2)
+      rspec-core (~> 3.13.0)
+      rspec-expectations (~> 3.13.0)
+      rspec-mocks (~> 3.13.0)
+    rspec-core (3.13.6)
+      rspec-support (~> 3.13.0)
+    rspec-expectations (3.13.5)
       diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.12.0)
+      rspec-support (~> 3.13.0)
-    rspec-mocks (3.12.4)
+    rspec-mocks (3.13.8)
       diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.12.0)
+      rspec-support (~> 3.13.0)
-    rspec-rails (6.1.1)
+    rspec-rails (8.0.4)
-      actionpack (>= 6.1)
+      actionpack (>= 7.2)
-      activesupport (>= 6.1)
+      activesupport (>= 7.2)
-      railties (>= 6.1)
+      railties (>= 7.2)
-      rspec-core (~> 3.12)
+      rspec-core (>= 3.13.0, < 5.0.0)
-      rspec-expectations (~> 3.12)
+      rspec-expectations (>= 3.13.0, < 5.0.0)
-      rspec-mocks (~> 3.12)
+      rspec-mocks (>= 3.13.0, < 5.0.0)
-      rspec-support (~> 3.12)
+      rspec-support (>= 3.13.0, < 5.0.0)
-    rspec-support (3.12.0)
+    rspec-support (3.13.7)
-    rubocop (1.48.1)
+    rubocop (1.91.0)
-      json (~> 2.3)
+      json (>= 2.3)
-      parallel (~> 1.10)
+      language_server-protocol (~> 3.17.0.2)
-      parser (>= 3.2.0.0)
+      lint_roller (~> 1.1.0)
+      parallel (>= 1.10)
+      parser (>= 3.3.0.2)
       rainbow (>= 2.2.2, < 4.0)
-      regexp_parser (>= 1.8, < 3.0)
+      regexp_parser (>= 2.9.3, < 3.0)
-      rexml (>= 3.2.5, < 4.0)
+      rubocop-ast (>= 1.49.0, < 2.0)
-      rubocop-ast (>= 1.26.0, < 2.0)
       ruby-progressbar (~> 1.7)
-      unicode-display_width (>= 2.4.0, < 3.0)
+      unicode-display_width (>= 2.4.0, < 4.0)
-    rubocop-ast (1.27.0)
+    rubocop-ast (1.50.0)
-      parser (>= 3.2.1.0)
+      parser (>= 3.3.7.2)
-    rubocop-rails (2.18.0)
+      prism (~> 1.7)
+    rubocop-rails (2.37.0)
       activesupport (>= 4.2.0)
+      lint_roller (~> 1.1)
       rack (>= 1.1)
-      rubocop (>= 1.33.0, < 2.0)
+      rubocop (>= 1.89.0, < 2.0)
+      rubocop-ast (>= 1.44.0, < 2.0)
     ruby-progressbar (1.13.0)
     sass-rails (6.0.0)
       sassc-rails (~> 2.1, >= 2.1.1)
@@ -277,83 +388,111 @@ GEM
       sprockets (> 3.0)
       sprockets-rails
       tilt
-    secure_headers (6.5.0)
+    secure_headers (7.3.0)
-    secure_random_string (1.0.0)
+      cgi (>= 0.1)
-    sentry-rails (5.8.0)
+    securerandom (0.4.1)
-      railties (>= 5.0)
+    sentry-rails (7.0.0)
-      sentry-ruby (~> 5.8.0)
+      railties (>= 5.2.0)
-    sentry-ruby (5.8.0)
+      sentry-ruby (~> 7.0.0)
+    sentry-ruby (7.0.0)
+      bigdecimal
       concurrent-ruby (~> 1.0, >= 1.0.2)
-    set (1.0.3)
+      logger
-    sorted_set (1.0.3)
+    shoulda-matchers (8.0.1)
-      rbtree
+      activesupport (>= 7.2)
-      set (~> 1.0)
+    sprockets (4.4.1)
-    sprockets (4.2.0)
+      concurrent-ruby (~> 1.1)
-      concurrent-ruby (~> 1.0)
+      logger
       rack (>= 2.2.4, < 4)
-    sprockets-rails (3.4.2)
+    sprockets-rails (3.5.2)
-      actionpack (>= 5.2)
+      actionpack (>= 6.1)
-      activesupport (>= 5.2)
+      activesupport (>= 6.1)
       sprockets (>= 3.0.0)
-    temple (0.10.0)
+    swd (2.0.3)
-    thor (1.3.0)
+      activesupport (>= 3)
-    tilt (2.1.0)
+      attr_required (>= 0.0.5)
-    timecop (0.9.8)
+      faraday (~> 2.0)
-    timeout (0.3.2)
+      faraday-follow_redirects
+    temple (0.10.7)
+    thor (1.5.0)
+    tilt (2.9.0)
+    timecop (0.9.11)
+    timeout (0.6.1)
+    tsort (0.2.0)
     turbolinks (5.2.1)
       turbolinks-source (~> 5.2)
     turbolinks-source (5.2.0)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    uglifier (4.2.0)
+    uglifier (4.2.1)
       execjs (>= 0.3.0, < 3)
-    unicode-display_width (2.4.2)
+    unicode-display_width (3.2.0)
-    websocket-driver (0.7.6)
+      unicode-emoji (~> 4.1)
+    unicode-emoji (4.2.0)
+    uri (1.1.1)
+    useragent (0.16.11)
+    validate_url (1.0.15)
+      activemodel (>= 3.0.0)
+      public_suffix
+    webfinger (2.1.3)
+      activesupport
+      faraday (~> 2.0)
+      faraday-follow_redirects
+    webmock (3.26.4)
+      addressable (>= 2.8.0)
+      crack (>= 0.3.2)
+      hashdiff (>= 0.4.0, < 2.0.0)
+    webrick (1.9.2)
+    websocket-driver (0.8.2)
+      base64
       websocket-extensions (>= 0.1.0)
     websocket-extensions (0.1.5)
-    zeitwerk (2.6.13)
+    zeitwerk (2.8.3)
 
 PLATFORMS
-  arm64-darwin-22
+  aarch64-linux
-  arm64-darwin-23
+  arm64-darwin
-  ruby
+  x86_64-darwin
   x86_64-linux
 
 DEPENDENCIES
+  abbrev
   annotate
   authie
   autoprefixer-rails
-  basic_ssl
   bcrypt
-  bunny
-  byebug
-  changey
   chronic
-  clockwork
   coffee-rails (~> 5.0)
-  database_cleaner
+  database_cleaner-active_record
-  dotenv-rails
+  domain_name
-  dynamic_form
+  dotenv
-  encrypto_signo
+  execjs
-  execjs (~> 2.7, < 2.8)
   factory_bot_rails
-  foreman
   gelf
   haml
   hashie
   highline
   jquery-rails
+  json (< 3)
   jwt
   kaminari
+  klogger-logger
+  konfig-config (~> 3.0)
+  logger
   mail
-  moonrope
+  mutex_m
   mysql2
   nifty-utils
   nilify_blanks
   nio4r
+  omniauth-rails_csrf_protection
+  omniauth_openid_connect
+  ostruct
+  prometheus-client
   puma
-  rails (= 6.1.7.6)
+  rackup
-  resolv (~> 0.2.1)
+  rails (= 7.2.3.2)
+  resolv
   rspec
   rspec-rails
   rubocop
@@ -361,10 +500,12 @@ DEPENDENCIES
   sass-rails
   secure_headers
   sentry-rails
-  sentry-ruby
+  shoulda-matchers
   timecop
   turbolinks (~> 5)
   uglifier (>= 1.3.0)
+  webmock
+  webrick
 
 BUNDLED WITH
-   2.4.9
+   2.7.2

MIT-LICENCE

@@ -1,4 +1,6 @@
-Copyright 2017 aTech Media Limited.
+Copyright 2017-2024 Krystal Hosting Ltd
+Copyright 2024 Adam Cooke
+Copyright 2024 Charlie Smurthwaite
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

Procfile.dev

@@ -1,5 +1,3 @@
-web: bundle exec puma -C config/puma.rb
+web: unset PORT; bundle exec puma -C config/puma.rb
 worker: bundle exec ruby script/worker.rb
-cron: bundle exec rake postal:cron
+smtp: unset PORT; bundle exec ruby script/smtp_server.rb
-smtp: bundle exec rake postal:smtp_server
-requeuer: bundle exec rake postal:requeuer

README.md

@@ -1,10 +1,9 @@
-<img src="https://share.adam.ac/21/logo-g3hWemROTemCYbWlmL.svg" width="250" />
+![GitHub Header](https://github.com/postalserver/.github/assets/4765/7a63c35d-2f47-412f-a6b3-aebc92a55310)
 
-**Postal** is a complete and fully featured mail server for use by websites & web servers. Think Sendgrid, Mailgun or Postmark but open source and ready for you to run on your own servers. Postal is developed by [Krystal](https://k.io) to serve its own mail processing requirements and we have since decided that it should be released as an open source project for the community.
+**Postal** is a complete and fully featured mail server for use by websites & web servers. Think Sendgrid, Mailgun or Postmark but open source and ready for you to run on your own servers. 
-
-![Screenshot](https://share.adam.ac/17/k4lA5OuPlU2.png)
 
 * [Documentation](https://docs.postalserver.io)
-* [Installation Instructions](https://docs.postalserver.io/install/prerequisites)
+* [Installation Instructions](https://docs.postalserver.io/getting-started)
 * [FAQs](https://docs.postalserver.io/welcome/faqs) & [Features](https://docs.postalserver.io/welcome/feature-list)
-* [Ask for help](https://github.com/postalserver/postal/discussions)
+* [Discussions](https://github.com/postalserver/postal/discussions) - ask for help or request a feature
+* [Join us on Discord](https://discord.postalserver.io)

SECURITY.md

@@ -2,14 +2,14 @@
 
 ## Supported Versions
 
-We only support updates to the 2.x versions of Postal.
+We only support updates to the 3.x versions of Postal.
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 2.x.x   | :white_check_mark: |
+| 3.x.x   | :white_check_mark: |
-| < 2.0   | :x:                |
+| < 3.0   | :x:                |
 
 ## Reporting a Vulnerability
 
-If you discover a vulnerability in Postal, please do not post an issue on GitHub. Instead you should send an
+If you discover a vulnerability in Postal, please do not post an issue on GitHub. Please,
-e-mail to security@postalserver.io with details. We will get back to you directly.
+instead, [create a new security advisory through GitHub](https://github.com/postalserver/postal/security/advisories/new).

api/authenticator.rb

@@ -1,31 +0,0 @@
-# frozen_string_literal: true
-
-authenticator :server do
-  friendly_name "Server Authenticator"
-  header "X-Server-API-Key", "The API token for a server that you wish to authenticate with.", example: "f29a45f0d4e1744ebaee"
-  error "InvalidServerAPIKey", "The API token provided in X-Server-API-Key was not valid.", attributes: { token: "The token that was looked up" }
-  error "ServerSuspended", "The mail server has been suspended"
-  lookup do
-    if key = request.headers["X-Server-API-Key"]
-      if credential = Credential.where(type: "API", key: key).first
-        if credential.server.suspended?
-          error "ServerSuspended"
-        else
-          credential.use
-          credential
-        end
-      else
-        error "InvalidServerAPIKey", token: key
-      end
-    end
-  end
-  rule :default, "AccessDenied", "Must be authenticated as a server." do
-    identity.is_a?(Credential)
-  end
-end
-
-authenticator :anonymous do
-  rule :default, "MustNotBeAuthenticated", "Must not be authenticated." do
-    identity.nil?
-  end
-end

api/controllers/messages_api_controller.rb

@@ -1,41 +0,0 @@
-# frozen_string_literal: true
-
-controller :messages do
-  friendly_name "Messages API"
-  description "This API allows you to access message details"
-  authenticator :server
-
-  action :message do
-    title "Return message details"
-    description "Returns all details about a message"
-    param :id, "The ID of the message", type: Integer, required: true
-    returns Hash, structure: :message, structure_opts: { paramable: { expansions: false } }
-    error "MessageNotFound", "No message found matching provided ID", attributes: { id: "The ID of the message" }
-    action do
-      begin
-        message = identity.server.message(params.id)
-      rescue Postal::MessageDB::Message::NotFound
-        error "MessageNotFound", id: params.id
-      end
-      structure :message, message, return: true
-    end
-  end
-
-  action :deliveries do
-    title "Return deliveries for a message"
-    description "Returns an array of deliveries which have been attempted for this message"
-    param :id, "The ID of the message", type: Integer, required: true
-    returns Array, structure: :delivery, structure_opts: { full: true }
-    error "MessageNotFound", "No message found matching provided ID", attributes: { id: "The ID of the message" }
-    action do
-      begin
-        message = identity.server.message(params.id)
-      rescue Postal::MessageDB::Message::NotFound
-        error "MessageNotFound", id: params.id
-      end
-      message.deliveries.map do |d|
-        structure :delivery, d
-      end
-    end
-  end
-end

api/controllers/send_api_controller.rb

@@ -1,112 +0,0 @@
-# frozen_string_literal: true
-
-controller :send do
-  friendly_name "Send API"
-  description "This API allows you to send messages"
-  authenticator :server
-
-  action :message do
-    title "Send a message"
-    description "This action allows you to send a message by providing the appropriate options"
-    # Acceptable Parameters
-    param :to, "The e-mail addresses of the recipients (max 50)", type: Array
-    param :cc, "The e-mail addresses of any CC contacts (max 50)", type: Array
-    param :bcc, "The e-mail addresses of any BCC contacts (max 50)", type: Array
-    param :from, "The e-mail address for the From header", type: String
-    param :sender, "The e-mail address for the Sender header", type: String
-    param :subject, "The subject of the e-mail", type: String
-    param :tag, "The tag of the e-mail", type: String
-    param :reply_to, "Set the reply-to address for the mail", type: String
-    param :plain_body, "The plain text body of the e-mail", type: String
-    param :html_body, "The HTML body of the e-mail", type: String
-    param :attachments, "An array of attachments for this e-mail", type: Array
-    param :headers, "A hash of additional headers", type: Hash
-    param :bounce, "Is this message a bounce?", type: :boolean
-    # Errors
-    error "ValidationError", "The provided data was not sufficient to send an email", attributes: { errors: "A hash of error details" }
-    error "NoRecipients", "There are no recipients defined to receive this message"
-    error "NoContent", "There is no content defined for this e-mail"
-    error "TooManyToAddresses", "The maximum number of To addresses has been reached (maximum 50)"
-    error "TooManyCCAddresses", "The maximum number of CC addresses has been reached (maximum 50)"
-    error "TooManyBCCAddresses", "The maximum number of BCC addresses has been reached (maximum 50)"
-    error "FromAddressMissing", "The From address is missing and is required"
-    error "UnauthenticatedFromAddress", "The From address is not authorised to send mail from this server"
-    error "AttachmentMissingName", "An attachment is missing a name"
-    error "AttachmentMissingData", "An attachment is missing data"
-    # Return
-    returns Hash
-    # Action
-    action do
-      attributes = {}
-      attributes[:to] = params.to
-      attributes[:cc] = params.cc
-      attributes[:bcc] = params.bcc
-      attributes[:from] = params.from
-      attributes[:sender] = params.sender
-      attributes[:subject] = params.subject
-      attributes[:reply_to] = params.reply_to
-      attributes[:plain_body] = params.plain_body
-      attributes[:html_body] = params.html_body
-      attributes[:bounce] = params.bounce ? true : false
-      attributes[:tag] = params.tag
-      attributes[:custom_headers] = params.headers
-      attributes[:attachments] = []
-      (params.attachments || []).each do |attachment|
-        next unless attachment.is_a?(Hash)
-
-        attributes[:attachments] << { name: attachment["name"], content_type: attachment["content_type"], data: attachment["data"], base64: true }
-      end
-      message = OutgoingMessagePrototype.new(identity.server, request.ip, "api", attributes)
-      message.credential = identity
-      if message.valid?
-        result = message.create_messages
-        { message_id: message.message_id, messages: result }
-      else
-        error message.errors.first
-      end
-    end
-  end
-
-  action :raw do
-    title "Send a raw RFC2822 message"
-    description "This action allows you to send us a raw RFC2822 formatted message along with the recipients that it should be sent to. This is similar to sending a message through our SMTP service."
-    param :mail_from, "The address that should be logged as sending the message", type: String, required: true
-    param :rcpt_to, "The addresses this message should be sent to", type: Array, required: true
-    param :data, "A base64 encoded RFC2822 message to send", type: String, required: true
-    param :bounce, "Is this message a bounce?", type: :boolean
-    returns Hash
-    error "UnauthenticatedFromAddress", "The From address is not authorised to send mail from this server"
-    action do
-      # Decode the raw message
-      raw_message = Base64.decode64(params.data)
-
-      # Parse through mail to get the from/sender headers
-      mail = Mail.new(raw_message.split("\r\n\r\n", 2).first)
-      from_headers = { "from" => mail.from, "sender" => mail.sender }
-      authenticated_domain = identity.server.find_authenticated_domain_from_headers(from_headers)
-
-      # If we're not authenticated, don't continue
-      if authenticated_domain.nil?
-        error "UnauthenticatedFromAddress"
-      end
-
-      # Store the result ready to return
-      result = { message_id: nil, messages: {} }
-      params.rcpt_to.uniq.each do |rcpt_to|
-        message = identity.server.message_db.new_message
-        message.rcpt_to = rcpt_to
-        message.mail_from = params.mail_from
-        message.raw_message = raw_message
-        message.received_with_ssl = true
-        message.scope = "outgoing"
-        message.domain_id = authenticated_domain.id
-        message.credential_id = identity.id
-        message.bounce = params.bounce
-        message.save
-        result[:message_id] = message.message_id if result[:message_id].nil?
-        result[:messages][rcpt_to] = { id: message.id, token: message.token }
-      end
-      result
-    end
-  end
-end

api/structures/delivery_api_structure.rb

@@ -1,12 +0,0 @@
-# frozen_string_literal: true
-
-structure :delivery do
-  basic :id
-  basic :status
-  basic :details
-  basic :output, value: proc { o.output&.strip }
-  basic :sent_with_ssl, value: proc { o.sent_with_ssl }
-  basic :log_id
-  basic :time, value: proc { o.time&.to_f }
-  basic :timestamp, value: proc { o.timestamp.to_f }
-end

api/structures/message_api_structure.rb

@@ -1,68 +0,0 @@
-# frozen_string_literal: true
-
-structure :message do
-  basic :id
-  basic :token
-
-  expansion(:status) do
-    {
-      status: o.status,
-      last_delivery_attempt: o.last_delivery_attempt&.to_f,
-      held: o.held,
-      hold_expiry: o.hold_expiry&.to_f
-    }
-  end
-
-  expansion(:details) do
-    {
-      rcpt_to: o.rcpt_to,
-      mail_from: o.mail_from,
-      subject: o.subject,
-      message_id: o.message_id,
-      timestamp: o.timestamp.to_f,
-      direction: o.scope,
-      size: o.size,
-      bounce: o.bounce,
-      bounce_for_id: o.bounce_for_id,
-      tag: o.tag,
-      received_with_ssl: o.received_with_ssl
-    }
-  end
-
-  expansion(:inspection) do
-    {
-      inspected: o.inspected,
-      spam: o.spam,
-      spam_score: o.spam_score.to_f,
-      threat: o.threat,
-      threat_details: o.threat_details
-    }
-  end
-
-  expansion(:plain_body) { o.plain_body }
-
-  expansion(:html_body) { o.html_body }
-
-  expansion(:attachments) do
-    o.attachments.map do |attachment|
-      {
-        filename: attachment.filename.to_s,
-        content_type: attachment.mime_type,
-        data: Base64.encode64(attachment.body.to_s),
-        size: attachment.body.to_s.bytesize,
-        hash: Digest::SHA1.hexdigest(attachment.body.to_s)
-      }
-    end
-  end
-
-  expansion(:headers) { o.headers }
-
-  expansion(:raw_message) { Base64.encode64(o.raw_message) }
-
-  expansion(:activity_entries) do
-    {
-      loads: o.loads,
-      clicks: o.clicks
-    }
-  end
-end

app/assets/stylesheets/application/components/_error_explanation.scss

@@ -9,14 +9,6 @@
   line-height:1.5;
 }
 
-.errorExplanation h2 {
-  display:none;
-}
-
-.errorExplanation p {
-  display:none;
-}
-
 .errorExplanation ul li {
   list-style:disc;
   margin-left:20px;

app/assets/stylesheets/application/components/_login_form.scss

@@ -1,22 +1,32 @@
-.loginForm {
+.loginForm {}
-
-}
 
 
 .loginForm__input {
-  margin-bottom:15px;
+  margin-bottom: 15px;
 }
 
 .loginForm__submit {
-  display:flex;
+  display: flex;
-  justify-content:space-between;
+  justify-content: space-between;
-  align-items:center;
+  align-items: center;
 }
 
 
 .loginForm__links {
-  font-size:12px;
+  font-size: 12px;
-  color:#999;
+  color: #999;
   text-decoration: underline;
-  line-height:1.7;
+  line-height: 1.7;
+}
+
+.loginForm__divider {
+  margin-top: 25px;
+  margin-bottom: 25px;
+  border-top: 1px solid #e4e8ef;
+}
+
+.loginForm__localTitle {
+  text-align: center;
+  margin-bottom: 15px;
+  color: #999;
 }

app/assets/stylesheets/application/components/_user_list.scss

@@ -1,74 +1,67 @@
 .userList {
-  border-radius:4px;
+  border-radius: 4px;
-  color:$darkBlue;
+  color: $darkBlue;
-  overflow:hidden;
+  overflow: hidden;
-  box-shadow:0 0 10px rgba(0,0,0,0.2);
+  box-shadow: 0 0 10px rgba(0, 0, 0, 0.2);
 }
 
 .userList__item {
-  display:block;
+  display: block;
-  background:#fff;
+  background: #fff;
-  padding:15px;
+  padding: 15px;
-  display:flex;
+  display: flex;
   align-items: center;
 }
+
 .userList__item:nth-child(even) {
-  background:none;
+  background: none;
 }
 
-.userList__item + .userList__item {
+.userList__item+.userList__item {
-  border-top:1px solid lighten(#ccd4e0, 10%);
+  border-top: 1px solid lighten(#ccd4e0, 10%);
-}
-
-.userList__avatar {
-  width:50px;
-  height:50px;
-  border-radius:50%;
-  background:#fff;
-  border:2px solid #efefef;
-  padding:3px;
-  flex: 0 0 auto;
 }
 
 .userList__details {
   flex: 1 1 auto;
-  margin:0 25px;
+  margin: 0 0;
 }
 
 
 .userList__actions {
   flex: 0 0 auto;
-  width:180px;
+  width: 120px;
-  font-size:12px;
+  font-size: 12px;
-  line-height:1.5;
+  line-height: 1.5;
-  color:#999;
+  color: #999;
-  text-decoration: underline;
+
+  a {
+    text-decoration: underline;
+  }
 }
 
 .userList__name {
-  font-weight:600;
+  font-weight: 600;
-  font-size:16px;
+  font-size: 16px;
-  margin-bottom:3px;
+  margin-bottom: 3px;
 }
 
 .userList__owner {
-  vertical-align:2px;
+  vertical-align: 2px;
-  margin-left:5px;
+  margin-left: 5px;
-  background-color:$orange;
+  background-color: $orange;
 }
 
 .userList__pending {
-  vertical-align:2px;
+  vertical-align: 2px;
-  margin-left:5px;
+  margin-left: 5px;
-  background-color:#ccc;
+  background-color: #ccc;
 }
 
-.userList__admin {
+.userList__tag {
-  vertical-align:2px;
+  vertical-align: 2px;
-  margin-left:5px;
+  margin-left: 3px;
-  background-color:$blue;
 }
 
 .userList__revoke {
-  color:$red;
+  color: $red;
 }

app/assets/stylesheets/application/elements/_button.scss

@@ -1,98 +1,117 @@
 .button {
-  display:inline-block;
+  display: inline-block;
-  font:inherit;
+  font: inherit;
-  border-radius:4px;
+  border-radius: 4px;
-  appearance:none;
+  appearance: none;
-  background:$blue;
+  background: $blue;
-  color:#fff;
+  color: #fff;
-  font-size:14px !important;
+  font-size: 14px !important;
-  margin:0;
+  margin: 0;
-  vertical-align:top;
+  vertical-align: top;
-  padding:6px 15px;
+  padding: 6px 15px;
-  border:2px solid transparent;
+  border: 2px solid transparent;
-  border-bottom:2px solid darken($blue, 20%);
+  border-bottom: 2px solid darken($blue, 20%);
+
   &:active {
-    background-color:darken($blue, 15%);
+    background-color: darken($blue, 15%);
   }
+
   &:focus {
-    border-color:darken($blue, 15%);
+    border-color: darken($blue, 15%);
-    background-color:lighten($blue, 5%);
+    background-color: lighten($blue, 5%);
   }
+
   &.is-spinning {
-    color:transparent;
+    color: transparent;
-    background-repeat:no-repeat;
+    background-repeat: no-repeat;
     background-position: center center;
-    background-size:25px;
+    background-size: 25px;
-    background-image:image-url('button-spinner.gif');
+    background-image: image-url('button-spinner.gif');
   }
 }
 
 .button--small {
-  font-size:12px !important;
+  font-size: 12px !important;
-  padding:3px 10px;
+  padding: 3px 10px;
-  border-width:1px;
+  border-width: 1px;
 }
 
 .button--positive {
-  background-color:$green;
+  background-color: $green;
-  border-bottom-color:darken($green, 15%);
+  border-bottom-color: darken($green, 15%);
+
   &:active {
-    background-color:darken($green, 15%);
+    background-color: darken($green, 15%);
   }
+
   &:focus {
-    border-color:darken($green, 15%);
+    border-color: darken($green, 15%);
-    background-color:lighten($green, 5%);
+    background-color: lighten($green, 5%);
   }
+
   &.is-spinning {
-    background-image:image-url('button-spinner-positive.gif');
+    background-image: image-url('button-spinner-positive.gif');
   }
 
 }
 
 
 .button--neutral {
-  background-color:#ccc;
+  background-color: #ccc;
-  border-bottom-color:darken(#ccc, 15%);
+  border-bottom-color: darken(#ccc, 15%);
+
   &:active {
-    background-color:darken(#ccc, 15%);
+    background-color: darken(#ccc, 15%);
   }
+
   &:focus {
-    border-color:darken(#ccc, 15%);
+    border-color: darken(#ccc, 15%);
-    background-color:lighten(#ccc, 5%);
+    background-color: lighten(#ccc, 5%);
   }
+
   &.is-spinning {
-    background-image:image-url('button-spinner-neutral.gif');
+    background-image: image-url('button-spinner-neutral.gif');
   }
 }
 
 .button--danger {
-  background-color:$red;
+  background-color: $red;
-  border-bottom-color:darken($red, 15%);
+  border-bottom-color: darken($red, 15%);
+
   &:active {
-    background-color:darken($red, 15%);
+    background-color: darken($red, 15%);
   }
+
   &:focus {
-    border-color:darken($red, 15%);
+    border-color: darken($red, 15%);
-    background-color:lighten($red, 5%);
+    background-color: lighten($red, 5%);
   }
+
   &.is-spinning {
-    background-image:image-url('button-spinner-danger.gif');
+    background-image: image-url('button-spinner-danger.gif');
   }
 }
 
 .button--dark {
-  background-color:$darkBlue;
+  background-color: $darkBlue;
-  border-bottom-color:darken($darkBlue, 15%);
+  border-bottom-color: darken($darkBlue, 15%);
+
   &:active {
-    background-color:darken($darkBlue, 15%);
+    background-color: darken($darkBlue, 15%);
   }
+
   &:focus {
-    border-color:darken($darkBlue, 15%);
+    border-color: darken($darkBlue, 15%);
-    background-color:lighten($darkBlue, 5%);
+    background-color: lighten($darkBlue, 5%);
   }
+
   &.is-spinning {
-    background-image:image-url('button-spinner-dark.gif');
+    background-image: image-url('button-spinner-dark.gif');
   }
 
 }
 
+.button--full {
+  width: 100%;
+  text-align: center;
+}

app/assets/stylesheets/application/elements/_label.scss

@@ -1,136 +1,137 @@
 .label {
-  display:inline-block;
+  display: inline-block;
-  background:#000;
+  background: #000;
-  color:#fff;
+  color: #fff;
-  font-size:9px;
+  font-size: 9px;
   text-transform: uppercase;
-  border-radius:40px;
+  border-radius: 40px;
-  padding:2px 6px;
+  padding: 2px 6px;
-  line-height:0.9;
+  line-height: 0.9;
 }
 
 .label--green {
-  background-color:$green;
+  background-color: $green;
 }
 
 .label--red {
-  background-color:$red;
+  background-color: $red;
 }
 
 .label--orange {
-  background-color:$orange;
+  background-color: $orange;
 }
 
 .label--blue {
-  background-color:$blue;
+  background-color: $blue;
 }
 
 .label--grey {
-  background-color:#999;
+  background-color: #999;
 }
 
 .label--turquoise {
-  background-color:$blue;
+  background-color: $blue;
 }
 
 .label--purple {
-  background-color:$purple;
+  background-color: $purple;
 }
 
 .label--large {
-  font-size:11px;
+  font-size: 11px;
-  padding:4px 10px;
+  padding: 4px 10px;
 }
 
 .label--serverStatus-live {
-  background-color:$green;
+  background-color: $green;
 }
 
 .label--serverStatus-development {
-  background-color:#636363;
+  background-color: #636363;
 }
 
 .label--serverStatus-suspended {
-  background-color:$red;
+  background-color: $red;
 }
 
 .label--messageStatus-pending {
-  background-color:$subBlue;
+  background-color: $subBlue;
 }
 
 .label--messageStatus-held {
-  background-color:#aaa;
+  background-color: #aaa;
 }
 
 .label--messageStatus-processed {
-  background-color:$green;
+  background-color: $green;
 }
 
 .label--messageStatus-sent {
-  background-color:$green;
+  background-color: $green;
 }
 
 .label--messageStatus-hard_fail {
-  background-color:$red;
+  background-color: $red;
 }
 
 .label--messageStatus-soft_fail {
-  background-color:$orange;
+  background-color: $orange;
 }
 
 .label--messageStatus-bounced {
-  background-color:$red;
+  background-color: $red;
 }
 
 .label--messageStatus-hold_cancelled {
-  background-color:#ccc;
+  background-color: #ccc;
 }
 
 
 .label--credentialType-api {
-  background-color:$blue;
+  background-color: $blue;
 }
 
 .label--credentialType-smtp {
-  background-color:$turquoise;
+  background-color: $turquoise;
 }
 
 .label--credentialType-smtp_ip {
-  background-color:$orange;
+  background-color: $orange;
 }
 
 .label--spamStatus-not_checked {
-  background:#aaa;
+  background: #aaa;
 }
 
 .label--spamStatus-spam {
-  background:$orange;
+  background: $orange;
 }
 
 .label--spamStatus-not_spam {
-  background:$turquoise;
+  background: $turquoise;
 }
 
 .label--http-status-2 {
-  background-color:$green;
+  background-color: $green;
 }
 
 .label--http-status-3 {
-  background-color:$orange;
+  background-color: $orange;
 }
 
 .label--http-status-4,
 .label--http-status-5 {
-  background-color:$red;
+  background-color: $red;
 }
 
 .domainList__ssl {
-  color:$green;
+  color: $green;
+
   &:hover {
-    text-decoration:underline;
+    text-decoration: underline;
   }
 }
 
 .domainList__ssl--disabled {
-  color:#999;
+  color: #999;
 }

app/controllers/application_controller.rb

@@ -62,10 +62,13 @@ class ApplicationController < ActionController::Base
   end
 
   def url_with_return_to(url)
-    if params[:return_to].blank? || !params[:return_to].starts_with?("/")
+    return_to = params[:return_to]
+    if return_to.blank? ||
+       !return_to.start_with?("/") ||
+       return_to.start_with?("//", "/\\")
       url_for(url)
     else
-      params[:return_to]
+      return_to
     end
   end
 
@@ -109,7 +112,8 @@ class ApplicationController < ActionController::Base
       auth_session.invalidate!
       reset_session
     end
-    Authie::Session.start(self, user: user)
+
+    create_auth_session(user)
     @current_user = user
   end
 

app/controllers/domains_controller.rb

@@ -71,7 +71,7 @@ class DomainsController < ApplicationController
     when "Email"
       if params[:code]
         if @domain.verification_token == params[:code].to_s.strip
-          @domain.verify
+          @domain.mark_as_verified
           redirect_to_with_json [:setup, organization, @server, @domain], notice: "#{@domain.name} has been verified successfully. You now need to configure your DNS records."
         else
           respond_to do |wants|

app/controllers/legacy_api/base_controller.rb

@@ -0,0 +1,133 @@
+# frozen_string_literal: true
+
+module LegacyAPI
+  # The Legacy API is the Postal v1 API which existed from the start with main
+  # aim of allowing e-mails to sent over HTTP rather than SMTP. The API itself
+  # did not feature much functionality. This API was implemented using Moonrope
+  # which was a self documenting API tool, however, is now no longer maintained.
+  # In light of that, these controllers now implement the same functionality as
+  # the original Moonrope API without the actual requirement to use any of the
+  # Moonrope components.
+  #
+  # Important things to note about the API:
+  #
+  #   * Moonrope allow params to be provided as JSON in the body of the request
+  #     along with the application/json content type. It also allowed for params
+  #     to be sent in the 'params' parameter when using the
+  #     application/x-www-form-urlencoded content type. Both methods are supported.
+  #
+  #   * Authentication is performed using a X-Server-API-Key variable.
+  #
+  #   * The method used to make the request is not important. Most clients use POST
+  #     but other methods should be supported. The routing for this legacvy
+  #     API supports GET, POST, PUT and PATCH.
+  #
+  #   * The status code for responses will always be 200 OK. The actual status of
+  #     a request is determined by the value of the 'status' attribute in the
+  #     returned JSON.
+  class BaseController < ActionController::Base
+
+    skip_before_action :set_browser_id
+    skip_before_action :verify_authenticity_token
+
+    before_action :start_timer
+    before_action :authenticate_as_server
+
+    private
+
+    # The Moonrope API spec allows for parameters to be provided in the body
+    # along with the application/json content type or they can be provided,
+    # as JSON, in the 'params' parameter when used with the
+    # application/x-www-form-urlencoded content type. This legacy API needs
+    # support both options for maximum compatibility.
+    #
+    # @return [Hash]
+    def api_params
+      if request.headers["content-type"] =~ /\Aapplication\/json/
+        return params.to_unsafe_hash
+      end
+
+      if params["params"].present?
+        return JSON.parse(params["params"])
+      end
+
+      {}
+    end
+
+    # The API returns a length of time to complete a request. We'll start
+    # a timer when the request starts and then use this method to calculate
+    # the time taken to complete the request.
+    #
+    # @return [void]
+    def start_timer
+      @start_time = Time.now.to_f
+    end
+
+    # The only method available to authenticate to the legacy API is using a
+    # credential from the server itself. This method will attempt to find
+    # that credential from the X-Server-API-Key header and will set the
+    # current_credential instance variable if a token is valid. Otherwise it
+    # will render an error to halt execution.
+    #
+    # @return [void]
+    def authenticate_as_server
+      key = request.headers["X-Server-API-Key"]
+      if key.blank?
+        render_error "AccessDenied",
+                     message: "Must be authenticated as a server."
+        return
+      end
+
+      credential = Credential.where(type: "API", key: key).first
+      if credential.nil?
+        render_error "InvalidServerAPIKey",
+                     message: "The API token provided in X-Server-API-Key was not valid.",
+                     token: key
+        return
+      end
+
+      if credential.server.suspended?
+        render_error "ServerSuspended"
+        return
+      end
+
+      credential.use
+      @current_credential = credential
+    end
+
+    # Render a successful response to the client
+    #
+    # @param [Hash] data
+    # @return [void]
+    def render_success(data)
+      render json: { status: "success",
+                     time: (Time.now.to_f - @start_time).round(3),
+                     flags: {},
+                     data: data }
+    end
+
+    # Render an error response to the client
+    #
+    # @param [String] code
+    # @param [Hash] data
+    # @return [void]
+    def render_error(code, data = {})
+      render json: { status: "error",
+                     time: (Time.now.to_f - @start_time).round(3),
+                     flags: {},
+                     data: data.merge(code: code) }
+    end
+
+    # Render a parameter error response to the client
+    #
+    # @param [String] message
+    # @return [void]
+    def render_parameter_error(message)
+      render json: { status: "parameter-error",
+                     time: (Time.now.to_f - @start_time).round(3),
+                     flags: {},
+                     data: { message: message } }
+    end
+
+  end
+end

app/controllers/legacy_api/messages_controller.rb

@@ -0,0 +1,166 @@
+# frozen_string_literal: true
+
+module LegacyAPI
+  class MessagesController < BaseController
+
+    # Returns details about a given message
+    #
+    #   URL:            /api/v1/messages/message
+    #
+    #   Parameters:     id              => REQ: The ID of the message
+    #                   _expansions     => An array of types of details t
+    #                                      to return
+    #
+    #   Response:       A hash containing message information
+    #                   OR an error if the message does not exist.
+    #
+    def message
+      message = find_message
+      return if performed?
+
+      message_hash = { id: message.id, token: message.token }
+      expansions = api_params["_expansions"]
+
+      if expansions == true || (expansions.is_a?(Array) && expansions.include?("status"))
+        message_hash[:status] = {
+          status: message.status,
+          last_delivery_attempt: message.last_delivery_attempt&.to_f,
+          held: message.held,
+          hold_expiry: message.hold_expiry&.to_f
+        }
+      end
+
+      if expansions == true || (expansions.is_a?(Array) && expansions.include?("details"))
+        message_hash[:details] = {
+          rcpt_to: message.rcpt_to,
+          mail_from: message.mail_from,
+          subject: message.subject,
+          message_id: message.message_id,
+          timestamp: message.timestamp.to_f,
+          direction: message.scope,
+          size: message.size,
+          bounce: message.bounce,
+          bounce_for_id: message.bounce_for_id,
+          tag: message.tag,
+          received_with_ssl: message.received_with_ssl
+        }
+      end
+
+      if expansions == true || (expansions.is_a?(Array) && expansions.include?("inspection"))
+        message_hash[:inspection] = {
+          inspected: message.inspected,
+          spam: message.spam,
+          spam_score: message.spam_score.to_f,
+          threat: message.threat,
+          threat_details: message.threat_details
+        }
+      end
+
+      if expansions == true || (expansions.is_a?(Array) && expansions.include?("plain_body"))
+        message_hash[:plain_body] = message.plain_body
+      end
+
+      if expansions == true || (expansions.is_a?(Array) && expansions.include?("html_body"))
+        message_hash[:html_body] = message.html_body
+      end
+
+      if expansions == true || (expansions.is_a?(Array) && expansions.include?("attachments"))
+        message_hash[:attachments] = message.attachments.map do |attachment|
+          {
+            filename: attachment.filename.to_s,
+            content_type: attachment.mime_type,
+            data: Base64.encode64(attachment.body.to_s),
+            size: attachment.body.to_s.bytesize,
+            hash: Digest::SHA1.hexdigest(attachment.body.to_s)
+          }
+        end
+      end
+
+      if expansions == true || (expansions.is_a?(Array) && expansions.include?("headers"))
+        message_hash[:headers] = message.headers
+      end
+
+      if expansions == true || (expansions.is_a?(Array) && expansions.include?("raw_message"))
+        message_hash[:raw_message] = Base64.encode64(message.raw_message)
+      end
+
+      if expansions == true || (expansions.is_a?(Array) && expansions.include?("activity_entries"))
+        message_hash[:activity_entries] = {
+          loads: message.loads,
+          clicks: message.clicks
+        }
+      end
+
+      render_success message_hash
+    rescue Postal::MessageDB::Message::NotFound
+      render_error "MessageNotFound",
+                   message: "No message found matching provided ID",
+                   id: api_params["id"]
+    end
+
+    # Returns all the deliveries for a given message
+    #
+    #   URL:            /api/v1/messages/deliveries
+    #
+    #   Parameters:     id              => REQ: The ID of the message
+    #
+    #   Response:       A array of hashes containing delivery information
+    #                   OR an error if the message does not exist.
+    #
+    def deliveries
+      message = find_message
+      return if performed?
+
+      deliveries = message.deliveries.map do |d|
+        {
+          id: d.id,
+          status: d.status,
+          details: d.details,
+          output: d.output&.strip,
+          sent_with_ssl: d.sent_with_ssl,
+          log_id: d.log_id,
+          time: d.time&.to_f,
+          timestamp: d.timestamp.to_f
+        }
+      end
+      render_success deliveries
+    rescue Postal::MessageDB::Message::NotFound
+      render_error "MessageNotFound",
+                   message: "No message found matching provided ID",
+                   id: api_params["id"]
+    end
+
+    private
+
+    # Look up the message referenced by the request's `id` parameter.
+    #
+    # The legacy API only ever identifies a message by its integer ID. The
+    # request body is parsed as JSON, so without validation a JSON object or
+    # array supplied for `id` would arrive as a Ruby Hash/Array and be passed
+    # straight through to the message database as a raw set of SQL conditions.
+    # We therefore reject anything that is not a simple scalar before it can
+    # reach the database and coerce the value to an integer ID.
+    #
+    # Renders an error and returns nil when the parameter is missing or is not
+    # a scalar; otherwise returns the matched message (raising NotFound when no
+    # message matches, which the actions rescue).
+    #
+    # @return [Postal::MessageDB::Message, nil]
+    def find_message
+      id = api_params["id"]
+
+      if id.blank?
+        render_parameter_error "`id` parameter is required but is missing"
+        return
+      end
+
+      unless id.is_a?(String) || id.is_a?(Integer)
+        render_parameter_error "`id` parameter must be a string or integer"
+        return
+      end
+
+      @current_credential.server.message(id.to_i)
+    end
+
+  end
+end

app/controllers/legacy_api/send_controller.rb

@@ -0,0 +1,135 @@
+# frozen_string_literal: true
+
+module LegacyAPI
+  class SendController < BaseController
+
+    ERROR_MESSAGES = {
+      "NoRecipients" => "There are no recipients defined to receive this message",
+      "NoContent" => "There is no content defined for this e-mail",
+      "TooManyToAddresses" => "The maximum number of To addresses has been reached (maximum 50)",
+      "TooManyCCAddresses" => "The maximum number of CC addresses has been reached (maximum 50)",
+      "TooManyBCCAddresses" => "The maximum number of BCC addresses has been reached (maximum 50)",
+      "FromAddressMissing" => "The From address is missing and is required",
+      "UnauthenticatedFromAddress" => "The From address is not authorised to send mail from this server",
+      "AttachmentMissingName" => "An attachment is missing a name",
+      "AttachmentMissingData" => "An attachment is missing data"
+    }.freeze
+
+    # Send a message with the given options
+    #
+    #   URL:            /api/v1/send/message
+    #
+    #   Parameters:     to              => REQ: An array of emails addresses
+    #                   cc              => An array of email addresses to CC
+    #                   bcc             => An array of email addresses to BCC
+    #                   from            => The name/email to send the email from
+    #                   sender          => The name/email of the 'Sender'
+    #                   reply_to        => The name/email of the 'Reply-to'
+    #                   plain_body      => The plain body
+    #                   html_body       => The HTML body
+    #                   bounce          => Is this message a bounce?
+    #                   tag             => A custom tag to add to the message
+    #                   custom_headers  => A hash of custom headers
+    #                   attachments     => An array of attachments
+    #                                      (name, content_type and data (base64))
+    #
+    #   Response:       A array of hashes containing message information
+    #                   OR an error if there is an issue sending the message
+    #
+    def message
+      attributes = {}
+      attributes[:to] = api_params["to"]
+      attributes[:cc] = api_params["cc"]
+      attributes[:bcc] = api_params["bcc"]
+      attributes[:from] = api_params["from"]
+      attributes[:sender] = api_params["sender"]
+      attributes[:subject] = api_params["subject"]
+      attributes[:reply_to] = api_params["reply_to"]
+      attributes[:plain_body] = api_params["plain_body"]
+      attributes[:html_body] = api_params["html_body"]
+      attributes[:bounce] = api_params["bounce"] ? true : false
+      attributes[:tag] = api_params["tag"]
+      attributes[:custom_headers] = api_params["headers"] if api_params["headers"]
+      attributes[:attachments] = []
+
+      (api_params["attachments"] || []).each do |attachment|
+        next unless attachment.is_a?(Hash)
+
+        attributes[:attachments] << { name: attachment["name"], content_type: attachment["content_type"], data: attachment["data"], base64: true }
+      end
+
+      message = OutgoingMessagePrototype.new(@current_credential.server, request.ip, "api", attributes)
+      message.credential = @current_credential
+      if message.valid?
+        result = message.create_messages
+        render_success message_id: message.message_id, messages: result
+      else
+        render_error message.errors.first, message: ERROR_MESSAGES[message.errors.first]
+      end
+    end
+
+    # Send a message by providing a raw message
+    #
+    #   URL:            /api/v1/send/raw
+    #
+    #   Parameters:     rcpt_to         => REQ: An array of email addresses to send
+    #                                      the message to
+    #                   mail_from       => REQ: the address to send the email from
+    #                   data            => REQ: base64-encoded mail data
+    #
+    #   Response:       A array of hashes containing message information
+    #                   OR an error if there is an issue sending the message
+    #
+    def raw
+      unless api_params["rcpt_to"].is_a?(Array)
+        render_parameter_error "`rcpt_to` parameter is required but is missing"
+        return
+      end
+
+      if api_params["mail_from"].blank?
+        render_parameter_error "`mail_from` parameter is required but is missing"
+        return
+      end
+
+      if api_params["data"].blank?
+        render_parameter_error "`data` parameter is required but is missing"
+        return
+      end
+
+      # Decode the raw message
+      raw_message = Base64.decode64(api_params["data"])
+
+      # Parse through mail to get the from/sender headers
+      mail = Mail.new(raw_message.split("\r\n\r\n", 2).first)
+      from_headers = { "from" => mail.from, "sender" => mail.sender }
+      authenticated_domain = @current_credential.server.find_authenticated_domain_from_headers(from_headers)
+
+      # If we're not authenticated, don't continue
+      if authenticated_domain.nil?
+        render_error "UnauthenticatedFromAddress"
+        return
+      end
+
+      # Store the result ready to return
+      result = { message_id: nil, messages: {} }
+      if api_params["rcpt_to"].is_a?(Array)
+        api_params["rcpt_to"].uniq.each do |rcpt_to|
+          message = @current_credential.server.message_db.new_message
+          message.rcpt_to = rcpt_to
+          message.mail_from = api_params["mail_from"]
+          message.raw_message = raw_message
+          message.received_with_ssl = true
+          message.scope = "outgoing"
+          message.domain_id = authenticated_domain.id
+          message.credential_id = @current_credential.id
+          message.bounce = api_params["bounce"] ? true : false
+          message.save
+          result[:message_id] = message.message_id if result[:message_id].nil?
+          result[:messages][rcpt_to] = { id: message.id, token: message.token }
+        end
+      end
+      render_success result
+    end
+
+  end
+end

app/controllers/messages_controller.rb

@@ -19,7 +19,7 @@ class MessagesController < ApplicationController
         @message.from = "test@#{domain.name}"
       end
     end
-    @message.subject = "Test Message at #{Time.zone.now.to_s(:long)}"
+    @message.subject = "Test Message at #{Time.zone.now.to_fs(:long)}"
     @message.plain_body = "This is a message to test the delivery of messages through Postal."
   end
 
@@ -89,6 +89,18 @@ class MessagesController < ApplicationController
   end
 
   def html_raw
+    override_content_security_policy_directives(
+      default_src: %w('none'),
+      script_src: %w('none'),
+      style_src: %w('unsafe-inline'),
+      img_src: %w(* data:),
+      font_src: %w(*),
+      frame_ancestors: %w('self'),
+      form_action: %w('none'),
+      base_uri: %w('none')
+    )
+    response.headers["X-Content-Type-Options"] = "nosniff"
+    response.headers["Referrer-Policy"] = "no-referrer"
     render html: @message.html_body_without_tracking_image.html_safe
   end
 
@@ -116,7 +128,7 @@ class MessagesController < ApplicationController
   def retry
     if @message.raw_message?
       if @message.queued_message
-        @message.queued_message.queue!
+        @message.queued_message.retry_now
         flash[:notice] = "This message will be retried shortly."
       elsif @message.held?
         @message.add_to_message_queue(manual: true)
@@ -161,7 +173,7 @@ class MessagesController < ApplicationController
 
       if @query = (params[:query] || session["msg_query_#{@server.id}_#{scope}"]).presence
         session["msg_query_#{@server.id}_#{scope}"] = @query
-        qs = Postal::QueryString.new(@query)
+        qs = QueryString.new(@query)
         if qs.empty?
           flash.now[:alert] = "It doesn't appear you entered anything to filter on. Please double check your query."
         else

app/controllers/organizations_controller.rb

@@ -43,10 +43,11 @@ class OrganizationsController < ApplicationController
   end
 
   def destroy
-    unless current_user.authenticate(params[:password])
+    if params[:confirm_text].blank? || params[:confirm_text].downcase.strip != organization.name.downcase.strip
       respond_to do |wants|
-        wants.html { redirect_to organization_delete_path(@organization), alert: "The password you entered was not valid. Please check and try again." }
+        alert_text = "The text you entered does not match the organization name. Please check and try again."
-        wants.json { render json: { alert: "The password you entered was invalid. Please check and try again." } }
+        wants.html { redirect_to organization_delete_path(@organization), alert: alert_text }
+        wants.json { render json: { alert: alert_text } }
       end
       return
     end

app/controllers/servers_controller.rb

@@ -45,8 +45,16 @@ class ServersController < ApplicationController
     extra_params = [:spam_threshold, :spam_failure_threshold, :postmaster_address]
 
     if current_user.admin?
-      extra_params += [:send_limit, :allow_sender, :privacy_mode, :log_smtp_data, :outbound_spam_threshold,
+      extra_params += [
-                       :message_retention_days, :raw_message_retention_days, :raw_message_retention_size]
+        :send_limit,
+        :allow_sender,
+        :privacy_mode,
+        :log_smtp_data,
+        :outbound_spam_threshold,
+        :message_retention_days,
+        :raw_message_retention_days,
+        :raw_message_retention_size,
+      ]
     end
 
     if @server.update(safe_params(*extra_params))
@@ -57,23 +65,21 @@ class ServersController < ApplicationController
   end
 
   def destroy
-    unless current_user.authenticate(params[:password])
+    if params[:confirm_text].blank? || params[:confirm_text].downcase.strip != @server.name.downcase.strip
       respond_to do |wants|
-        wants.html do
+        alert_text = "The text you entered does not match the server name. Please check and try again."
-          redirect_to [:delete, organization, @server], alert: "The password you entered was not valid. Please check and try again."
+        wants.html { redirect_to organization_delete_path(@organization), alert: alert_text }
-        end
+        wants.json { render json: { alert: alert_text } }
-        wants.json do
-          render json: { alert: "The password you entere was invalid. Please check and try again" }
-        end
       end
       return
     end
+
     @server.soft_destroy
     redirect_to_with_json organization_root_path(organization), notice: "#{@server.name} has been deleted successfully"
   end
 
   def queue
-    @messages = @server.queued_messages.order(id: :desc).page(params[:page])
+    @messages = @server.queued_messages.order(id: :desc).page(params[:page]).includes(:ip_address)
     @messages_with_message = @messages.include_message
   end
 

app/controllers/sessions_controller.rb

@@ -4,7 +4,8 @@ class SessionsController < ApplicationController
 
   layout "sub"
 
-  skip_before_action :login_required, only: [:new, :create, :create_with_token, :begin_password_reset, :finish_password_reset, :ip, :raise_error]
+  before_action :require_local_authentication, only: [:create, :begin_password_reset, :finish_password_reset]
+  skip_before_action :login_required, only: [:new, :create, :begin_password_reset, :finish_password_reset, :ip, :raise_error, :create_from_oidc, :oauth_failure]
 
   def create
     login(User.authenticate(params[:email_address], params[:password]))
@@ -15,18 +16,6 @@ class SessionsController < ApplicationController
     render "new"
   end
 
-  def create_with_token
-    result = JWT.decode(params[:token], Postal.signing_key.to_s, "HS256")[0]
-    if result["timestamp"] > 1.minute.ago.to_f
-      login(User.find(result["user"].to_i))
-      redirect_to root_path
-    else
-      destroy
-    end
-  rescue JWT::VerificationError
-    destroy
-  end
-
   def destroy
     auth_session.invalidate! if logged_in?
     reset_session
@@ -41,12 +30,16 @@ class SessionsController < ApplicationController
   def begin_password_reset
     return unless request.post?
 
-    if user = User.where(email_address: params[:email_address]).first
+    user_scope = Postal::Config.oidc.enabled? ? User.with_password : User
-      user.begin_password_reset(params[:return_to])
+    user = user_scope.find_by(email_address: params[:email_address])
-      redirect_to login_path(return_to: params[:return_to]), notice: "Please check your e-mail and click the link in the e-mail we've sent you."
+
-    else
+    if user.nil?
-      redirect_to login_reset_path(return_to: params[:return_to]), alert: "No user exists with that e-mail address. Please check and try again."
+      redirect_to login_reset_path(return_to: params[:return_to]), alert: "No local user exists with that e-mail address. Please check and try again."
+      return
     end
+
+    user.begin_password_reset(params[:return_to])
+    redirect_to login_path(return_to: params[:return_to]), notice: "Please check your e-mail and click the link in the e-mail we've sent you."
   end
 
   def finish_password_reset
@@ -61,6 +54,7 @@ class SessionsController < ApplicationController
       flash.now[:alert] = "You must enter a new password"
       return
     end
+
     @user.password = params[:password]
     @user.password_confirmation = params[:password_confirmation]
     return unless @user.save
@@ -73,4 +67,33 @@ class SessionsController < ApplicationController
     render plain: "ip: #{request.ip} remote ip: #{request.remote_ip}"
   end
 
+  def create_from_oidc
+    unless Postal::Config.oidc.enabled?
+      raise Postal::Error, "OIDC cannot be used unless enabled in the configuration"
+    end
+
+    auth = request.env["omniauth.auth"]
+    user = User.find_from_oidc(auth.extra.raw_info, logger: Postal.logger)
+    if user.nil?
+      redirect_to login_path, alert: "No user was found matching your identity. Please contact your administrator."
+      return
+    end
+
+    login(user)
+    flash[:remember_login] = true
+    redirect_to_with_return_to root_path
+  end
+
+  def oauth_failure
+    redirect_to login_path, alert: "An issue occurred while logging you in with OpenID. Please try again later or contact your administrator."
+  end
+
+  private
+
+  def require_local_authentication
+    return if Postal::Config.oidc.local_authentication_enabled?
+
+    redirect_to login_path, alert: "Local authentication is not enabled"
+  end
+
 end

app/controllers/user_controller.rb

@@ -28,69 +28,35 @@ class UserController < ApplicationController
     end
   end
 
-  def join
-    if @invite = UserInvite.where(uuid: params[:token]).where("expires_at > ?", Time.now).first
-      if logged_in?
-        if request.post?
-          @invite.accept(current_user)
-          redirect_to_with_json root_path(nrd: 1), notice: "Invitation has been accepted successfully. You now have access to this organization."
-        elsif request.delete?
-          @invite.reject
-          redirect_to_with_json root_path(nrd: 1), notice: "Invitation has been rejected successfully."
-        else
-          @organizations = @invite.organizations.order(:name).to_a
-        end
-      else
-        redirect_to new_signup_path(params[:token])
-      end
-    else
-      redirect_to_with_json root_path(nrd: 1), alert: "The invite URL you have has expired. Please ask the person who invited you to re-send your invitation."
-    end
-  end
-
   def update
     @user = User.find(current_user.id)
-    @user.attributes = params.require(:user).permit(:first_name, :last_name, :time_zone, :email_address, :password, :password_confirmation)
+    safe_params = [:first_name, :last_name, :time_zone, :email_address]
 
-    if @user.authenticate_with_previous_password_first(params[:password])
+    if @user.password? && Postal::Config.oidc.local_authentication_enabled?
-      @password_correct = true
+      safe_params += [:password, :password_confirmation]
-    else
+      if @user.authenticate_with_previous_password_first(params[:password])
-      respond_to do |wants|
+        @password_correct = true
-        wants.html do
+      else
-          flash.now[:alert] = "The current password you have entered is incorrect. Please check and try again."
+        respond_to do |wants|
-          render "edit"
+          wants.html do
-        end
+            flash.now[:alert] = "The current password you have entered is incorrect. Please check and try again."
-        wants.json do
+            render "edit"
-          render json: { alert: "The current password you've entered is incorrect. Please check and try again" }
+          end
+          wants.json do
+            render json: { alert: "The current password you've entered is incorrect. Please check and try again" }
+          end
         end
+        return
       end
-      return
     end
 
-    email_changed = @user.email_address_changed?
+    @user.attributes = params.require(:user).permit(safe_params)
 
     if @user.save
-      if email_changed
+      redirect_to_with_json settings_path, notice: "Your settings have been updated successfully."
-        redirect_to_with_json verify_path(return_to: settings_path),
-                              notice: "Your settings have been updated successfully. As you've changed, your e-mail " \
-                                      "address you'll need to verify it before you can continue."
-      else
-        redirect_to_with_json settings_path, notice: "Your settings have been updated successfully."
-      end
     else
       render_form_errors "edit", @user
     end
   end
 
-  def verify
-    return unless request.post?
-
-    if params[:code].to_s.strip == current_user.email_verification_token.to_s || (Rails.env.development? && params[:code].to_s.strip == "123456")
-      current_user.verify!
-      redirect_to_with_json [:return_to, root_path], notice: "Thanks - your e-mail address has been verified successfully."
-    else
-      flash_now :alert, "The code you've entered isn't correct. Please check and try again."
-    end
-  end
-
 end

app/controllers/well_known_controller.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+class WellKnownController < ApplicationController
+
+  layout false
+
+  skip_before_action :set_browser_id
+  skip_before_action :login_required
+  skip_before_action :set_timezone
+
+  def jwks
+    render json: JWT::JWK::Set.new(Postal.signer.jwk).export.to_json
+  end
+
+end

app/helpers/application_helper.rb

@@ -3,6 +3,7 @@
 module ApplicationHelper
 
   def format_delivery_details(server, text)
+    text = h(text)
     text.gsub!(/<msg:(\d+)>/) do
       id = ::Regexp.last_match(1).to_i
       link_to("message ##{id}", organization_server_message_path(server.organization, server, id), class: "u-link")
@@ -32,7 +33,7 @@ module ApplicationHelper
         s << "<optgroup label='Server Domains'>"
         server_domains.each do |domain|
           selected = domain == selected_domain ? "selected='selected'" : ""
-          s << "<option value='#{domain.id}' #{selected}>#{domain.name}</option>"
+          s << "<option value='#{h(domain.id)}' #{selected}>#{h(domain.name)}</option>"
         end
         s << "</optgroup>"
       end
@@ -42,7 +43,7 @@ module ApplicationHelper
         s << "<optgroup label='Organization Domains'>"
         organization_domains.each do |domain|
           selected = domain == selected_domain ? "selected='selected'" : ""
-          s << "<option value='#{domain.id}' #{selected}>#{domain.name}</option>"
+          s << "<option value='#{h(domain.id)}' #{selected}>#{h(domain.name)}</option>"
         end
         s << "</optgroup>"
       end
@@ -59,7 +60,7 @@ module ApplicationHelper
         http_endpoints.each do |endpoint|
           value = "#{endpoint.class}##{endpoint.uuid}"
           selected = value == selected_value ? "selected='selected'" : ""
-          s << "<option value='#{value}' #{selected}>#{endpoint.description}</option>"
+          s << "<option value='#{h(value)}' #{selected}>#{h(endpoint.description)}</option>"
         end
         s << "</optgroup>"
       end
@@ -70,7 +71,7 @@ module ApplicationHelper
         smtp_endpoints.each do |endpoint|
           value = "#{endpoint.class}##{endpoint.uuid}"
           selected = value == selected_value ? "selected='selected'" : ""
-          s << "<option value='#{value}' #{selected}>#{endpoint.description}</option>"
+          s << "<option value='#{h(value)}' #{selected}>#{h(endpoint.description)}</option>"
         end
         s << "</optgroup>"
       end
@@ -81,7 +82,7 @@ module ApplicationHelper
         address_endpoints.each do |endpoint|
           value = "#{endpoint.class}##{endpoint.uuid}"
           selected = value == selected_value ? "selected='selected'" : ""
-          s << "<option value='#{value}' #{selected}>#{endpoint.address}</option>"
+          s << "<option value='#{h(value)}' #{selected}>#{h(endpoint.address)}</option>"
         end
         s << "</optgroup>"
       end
@@ -93,11 +94,18 @@ module ApplicationHelper
 
           selected = (selected_value == mode ? "selected='selected'" : "")
           text = t("route_modes.#{mode.underscore}")
-          s << "<option value='#{mode}' #{selected}>#{text}</option>"
+          s << "<option value='#{h(mode)}' #{selected}>#{h(text)}</option>"
         end
         s << "</optgroup>"
       end
     end.html_safe
   end
 
+  def postal_version_string
+    string = Postal.version
+    string += " (#{Postal.branch})" if Postal.branch &&
+                                       Postal.branch != "main"
+    string
+  end
+
 end

app/helpers/error_messages_helper.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module ErrorMessagesHelper
+
+  # Renders a record's validation errors.
+  #
+  # This replaces the helper of the same name from the dynamic_form gem, which
+  # is unmaintained and cannot be installed on Ruby 3.4. The markup matches what
+  # that gem produced, minus the heading and description which the
+  # errorExplanation styles have always hidden, and matches the structure
+  # ajax.coffee builds when it renders errors from a remote form.
+  def error_messages_for(record)
+    record = record.to_model if record.respond_to?(:to_model)
+    return "".html_safe if record.nil? || record.errors.empty?
+
+    items = safe_join(record.errors.full_messages.map { |message| tag.li(message) })
+    tag.div(tag.ul(items), id: "errorExplanation", class: "errorExplanation")
+  end
+
+end

app/jobs/action_deletion_job.rb

@@ -1,16 +0,0 @@
-# frozen_string_literal: true
-
-class ActionDeletionJob < Postal::Job
-
-  def perform
-    object = params["type"].constantize.deleted.find_by_id(params["id"])
-    if object
-      log "Deleting #{params['type']}##{params['id']}"
-      object.destroy
-      log "Deleted #{params['type']}##{params['id']}"
-    else
-      log "Couldn't find deleted object #{params['type']}##{params['id']}"
-    end
-  end
-
-end

app/jobs/action_deletions_job.rb

@@ -1,17 +0,0 @@
-# frozen_string_literal: true
-
-class ActionDeletionsJob < Postal::Job
-
-  def perform
-    Organization.deleted.each do |org|
-      log "Permanently removing organization #{org.id} (#{org.permalink})"
-      org.destroy
-    end
-
-    Server.deleted.each do |server|
-      log "Permanently removing server #{server.id} (#{server.full_permalink})"
-      server.destroy
-    end
-  end
-
-end

app/jobs/prune_suppression_lists_job.rb

@@ -1,12 +0,0 @@
-# frozen_string_literal: true
-
-class PruneSuppressionListsJob < Postal::Job
-
-  def perform
-    Server.all.each do |s|
-      log "Pruning suppression lists for server #{s.id}"
-      s.message_db.suppression_list.prune
-    end
-  end
-
-end

app/jobs/prune_webhook_requests_job.rb

@@ -1,12 +0,0 @@
-# frozen_string_literal: true
-
-class PruneWebhookRequestsJob < Postal::Job
-
-  def perform
-    Server.all.each do |s|
-      log "Pruning webhook requests for server #{s.id}"
-      s.message_db.webhooks.prune
-    end
-  end
-
-end

app/jobs/requeue_webhooks_job.rb

@@ -1,9 +0,0 @@
-# frozen_string_literal: true
-
-class RequeueWebhooksJob < Postal::Job
-
-  def perform
-    WebhookRequest.requeue_all
-  end
-
-end

app/jobs/send_notifications_job.rb

@@ -1,9 +0,0 @@
-# frozen_string_literal: true
-
-class SendNotificationsJob < Postal::Job
-
-  def perform
-    Server.send_send_limit_notifications
-  end
-
-end

app/jobs/send_webhook_job.rb

@@ -1,29 +0,0 @@
-# frozen_string_literal: true
-
-class SendWebhookJob < Postal::Job
-
-  def perform
-    if server = Server.find(params["server_id"])
-      new_items = {}
-      params["payload"]&.each do |key, value|
-        next unless key.to_s =~ /\A_(\w+)/
-
-        begin
-          new_items[::Regexp.last_match(1)] = server.message_db.message(value.to_i).webhook_hash
-        rescue Postal::MessageDB::Message::NotFound
-          # No message found, don't do any replacement
-        end
-      end
-
-      new_items.each do |key, value|
-        params["payload"].delete("_#{key}")
-        params["payload"][key] = value
-      end
-
-      WebhookRequest.trigger(server, params["event"], params["payload"])
-    else
-      log "Couldn't find server with ID #{params['server_id']}"
-    end
-  end
-
-end

app/jobs/sleep_job.rb

@@ -1,9 +0,0 @@
-# frozen_string_literal: true
-
-class SleepJob < Postal::Job
-
-  def perform
-    sleep 5
-  end
-
-end

app/jobs/tidy_raw_messages_job.rb

@@ -1,8 +0,0 @@
-# frozen_string_literal: true
-
-class TidyRawMessagesJob < Postal::Job
-
-  def perform
-  end
-
-end

app/jobs/unqueue_message_job.rb

@@ -1,468 +0,0 @@
-# frozen_string_literal: true
-
-class UnqueueMessageJob < Postal::Job
-
-  # rubocop:disable Layout/LineLength
-  def perform
-    if original_message = QueuedMessage.find_by_id(params["id"])
-      if original_message.acquire_lock
-
-        log "Lock acquired for queued message #{original_message.id}"
-
-        begin
-          original_message.message
-        rescue Postal::MessageDB::Message::NotFound
-          log "Unqueue #{original_message.id} because backend message has been removed."
-          original_message.destroy
-          return
-        end
-
-        unless original_message.retriable?
-          log "Skipping because retry after isn't reached"
-          original_message.unlock
-          return
-        end
-
-        begin
-          other_messages = original_message.batchable_messages(100)
-          log "Found #{other_messages.size} associated messages to process at the same time (batch key: #{original_message.batch_key})"
-        rescue StandardError
-          original_message.unlock
-          raise
-        end
-
-        ([original_message] + other_messages).each do |queued_message|
-          log_prefix = "[#{queued_message.server_id}::#{queued_message.message_id} #{queued_message.id}]"
-          begin
-            log "#{log_prefix} Got queued message with exclusive lock"
-
-            begin
-              queued_message.message
-            rescue Postal::MessageDB::Message::NotFound
-              log "#{log_prefix} Unqueueing #{queued_message.id} because backend message has been removed"
-              queued_message.destroy
-              next
-            end
-
-            #
-            # If the server is suspended, hold all messages
-            #
-            if queued_message.server.suspended?
-              log "#{log_prefix} Server is suspended. Holding message."
-              queued_message.message.create_delivery("Held", details: "Mail server has been suspended. No e-mails can be processed at present. Contact support for assistance.")
-              queued_message.destroy
-              next
-            end
-
-            # We might not be able to send this any more, check the attempts
-            if queued_message.attempts >= Postal.config.general.maximum_delivery_attempts
-              details = "Maximum number of delivery attempts (#{queued_message.attempts}) has been reached."
-              if queued_message.message.scope == "incoming"
-                # Send bounces to incoming e-mails when they are hard failed
-                if bounce_id = queued_message.send_bounce
-                  details += " Bounce sent to sender (see message <msg:#{bounce_id}>)"
-                end
-              elsif queued_message.message.scope == "outgoing"
-                # Add the recipient to the suppression list
-                if queued_message.server.message_db.suppression_list.add(:recipient, queued_message.message.rcpt_to, reason: "too many soft fails")
-                  log "Added #{queued_message.message.rcpt_to} to suppression list because maximum attempts has been reached"
-                  details += " Added #{queued_message.message.rcpt_to} to suppression list because delivery has failed #{queued_message.attempts} times."
-                end
-              end
-              queued_message.message.create_delivery("HardFail", details: details)
-              queued_message.destroy
-              log "#{log_prefix} Message has reached maximum number of attempts. Hard failing."
-              next
-            end
-
-            # If the raw message has been removed (removed by retention)
-            unless queued_message.message.raw_message?
-              log "#{log_prefix} Raw message has been removed. Not sending."
-              queued_message.message.create_delivery("HardFail", details: "Raw message has been removed. Cannot send message.")
-              queued_message.destroy
-              next
-            end
-
-            #
-            #  Handle Incoming Messages
-            #
-            if queued_message.message.scope == "incoming"
-              #
-              # If this is a bounce, we need to handle it as such
-              #
-              if queued_message.message.bounce
-                log "#{log_prefix} Message is a bounce"
-                original_messages = queued_message.message.original_messages
-                unless original_messages.empty?
-                  queued_message.message.original_messages.each do |orig_msg|
-                    queued_message.message.update(bounce_for_id: orig_msg.id, domain_id: orig_msg.domain_id)
-                    queued_message.message.create_delivery("Processed", details: "This has been detected as a bounce message for <msg:#{orig_msg.id}>.")
-                    orig_msg.bounce!(queued_message.message)
-                    log "#{log_prefix} Bounce linked with message #{orig_msg.id}"
-                  end
-                  queued_message.destroy
-                  next
-                end
-
-                # This message was sent to the return path but hasn't been matched
-                #  to an original message. If we have a route for this, route it
-                #  otherwise we'll drop at this point.
-                if queued_message.message.route_id.nil?
-                  log "#{log_prefix} No source messages found. Hard failing."
-                  queued_message.message.create_delivery("HardFail", details: "This message was a bounce but we couldn't link it with any outgoing message and there was no route for it.")
-                  queued_message.destroy
-                  next
-                end
-              end
-
-              #
-              # Update live stats
-              #
-              queued_message.message.database.live_stats.increment(queued_message.message.scope)
-
-              #
-              # Inspect incoming messages
-              #
-              unless queued_message.message.inspected
-                log "#{log_prefix} Inspecting message"
-                queued_message.message.inspect_message
-                if queued_message.message.inspected
-                  is_spam = queued_message.message.spam_score > queued_message.server.spam_threshold
-                  queued_message.message.update(spam: true) if is_spam
-                  queued_message.message.append_headers(
-                    "X-Postal-Spam: #{queued_message.message.spam ? 'yes' : 'no'}",
-                    "X-Postal-Spam-Threshold: #{queued_message.server.spam_threshold}",
-                    "X-Postal-Spam-Score: #{queued_message.message.spam_score}",
-                    "X-Postal-Threat: #{queued_message.message.threat ? 'yes' : 'no'}"
-                  )
-                  log "#{log_prefix} Message inspected successfully. Headers added."
-                end
-              end
-
-              #
-              # If this message has a SPAM score higher than is permitted
-              #
-              if queued_message.message.spam_score >= queued_message.server.spam_failure_threshold
-                log "#{log_prefix} Message has a spam score higher than the server's maxmimum. Hard failing."
-                queued_message.message.create_delivery("HardFail", details: "Message's spam score is higher than the failure threshold for this server. Threshold is currently #{queued_message.server.spam_failure_threshold}.")
-                queued_message.destroy
-                next
-              end
-
-              # If the server is in development mode, hold it
-              if queued_message.server.mode == "Development" && !queued_message.manual?
-                log "Server is in development mode so holding."
-                queued_message.message.create_delivery("Held", details: "Server is in development mode.")
-                queued_message.destroy
-                log "#{log_prefix} Server is in development mode. Holding."
-                next
-              end
-
-              #
-              # Find out what sort of message we're supposed to be sending and dispatch this request over to
-              # the sender.
-              #
-              if route = queued_message.message.route
-
-                # If the route says we're holding quananteed mail and this is spam, we'll hold this
-                if route.spam_mode == "Quarantine" && queued_message.message.spam && !queued_message.manual?
-                  queued_message.message.create_delivery("Held", details: "Message placed into quarantine.")
-                  queued_message.destroy
-                  log "#{log_prefix} Route says to quarantine spam message. Holding."
-                  next
-                end
-
-                # If the route says we're holding quananteed mail and this is spam, we'll hold this
-                if route.spam_mode == "Fail" && queued_message.message.spam && !queued_message.manual?
-                  queued_message.message.create_delivery("HardFail", details: "Message is spam and the route specifies it should be failed.")
-                  queued_message.destroy
-                  log "#{log_prefix} Route says to fail spam message. Hard failing."
-                  next
-                end
-
-                #
-                # Messages that should be blindly accepted are blindly accepted
-                #
-                if route.mode == "Accept"
-                  queued_message.message.create_delivery("Processed", details: "Message has been accepted but not sent to any endpoints.")
-                  queued_message.destroy
-                  log "#{log_prefix} Route says to accept without endpoint. Marking as processed."
-                  next
-                end
-
-                #
-                # Messages that should be accepted and held should be held
-                #
-                if route.mode == "Hold"
-                  log "#{log_prefix} Route says to hold message."
-                  if queued_message.manual?
-                    log "#{log_prefix} Message was queued manually. Marking as processed."
-                    queued_message.message.create_delivery("Processed", details: "Message has been processed.")
-                  else
-                    log "#{log_prefix} Message was not queued manually. Holding."
-                    queued_message.message.create_delivery("Held", details: "Message has been accepted but not sent to any endpoints.")
-                  end
-                  queued_message.destroy
-                  next
-                end
-
-                #
-                # Messages that should be bounced should be bounced (or rejected if they got this far)
-                #
-                if route.mode == "Bounce" || route.mode == "Reject"
-                  if id = queued_message.send_bounce
-                    queued_message.message.create_delivery("HardFail", details: "Message has been bounced because the route asks for this. See message <msg:#{id}>")
-                    log "#{log_prefix} Route says to bounce. Hard failing and sent bounce (#{id})."
-                  end
-                  queued_message.destroy
-                  next
-                end
-
-                if @fixed_result
-                  result = @fixed_result
-                else
-                  case queued_message.message.endpoint
-                  when SMTPEndpoint
-                    sender = cached_sender(Postal::SMTPSender, queued_message.message.recipient_domain, nil, servers: [queued_message.message.endpoint])
-                  when HTTPEndpoint
-                    sender = cached_sender(Postal::HTTPSender, queued_message.message.endpoint)
-                  when AddressEndpoint
-                    sender = cached_sender(Postal::SMTPSender, queued_message.message.endpoint.domain, nil, force_rcpt_to: queued_message.message.endpoint.address)
-                  else
-                    log "#{log_prefix} Invalid endpoint for route (#{queued_message.message.endpoint_type})"
-                    queued_message.message.create_delivery("HardFail", details: "Invalid endpoint for route.")
-                    queued_message.destroy
-                    next
-                  end
-                  result = sender.send_message(queued_message.message)
-                  if result.connect_error
-                    @fixed_result = result
-                  end
-                end
-
-                # Log the result
-                log_details = result.details
-                if result.type == "HardFail" && result.suppress_bounce
-                  # The delivery hard failed, but requested that no bounce be sent
-                  log "#{log_prefix} Suppressing bounce message after hard fail"
-                elsif result.type == "HardFail" && queued_message.message.send_bounces?
-                  # If the message is a hard fail, send a bounce message for this message.
-                  log "#{log_prefix} Sending a bounce because message hard failed"
-                  if bounce_id = queued_message.send_bounce
-                    log_details += ". " unless log_details =~ /\.\z/
-                    log_details += " Sent bounce message to sender (see message <msg:#{bounce_id}>)"
-                  end
-                end
-
-                queued_message.message.create_delivery(result.type, details: log_details, output: result.output&.strip, sent_with_ssl: result.secure, log_id: result.log_id, time: result.time)
-
-                if result.retry
-                  log "#{log_prefix} Message requeued for trying later."
-                  queued_message.retry_later(result.retry.is_a?(Integer) ? result.retry : nil)
-                  queued_message.allocate_ip_address
-                  queued_message.update_column(:ip_address_id, queued_message.ip_address&.id)
-                else
-                  log "#{log_prefix} Message processing completed."
-                  queued_message.message.endpoint.mark_as_used
-                  queued_message.destroy
-                end
-              else
-                log "#{log_prefix} No route and/or endpoint available for processing. Hard failing."
-                queued_message.message.create_delivery("HardFail", details: "Message does not have a route and/or endpoint available for delivery.")
-                queued_message.destroy
-                next
-              end
-            end
-
-            #
-            # Handle Outgoing Messages
-            #
-            if queued_message.message.scope == "outgoing"
-              if queued_message.message.domain.nil?
-                log "#{log_prefix} Message has no domain. Hard failing."
-                queued_message.message.create_delivery("HardFail", details: "Message's domain no longer exist")
-                queued_message.destroy
-                next
-              end
-
-              #
-              # If there's no to address, we can't do much. Fail it.
-              #
-              if queued_message.message.rcpt_to.blank?
-                log "#{log_prefix} Message has no to address. Hard failing."
-                queued_message.message.create_delivery("HardFail", details: "Message doesn't have an RCPT to")
-                queued_message.destroy
-                next
-              end
-
-              # Extract a tag and add it to the message if one doesn't exist
-              if queued_message.message.tag.nil? && tag = queued_message.message.headers["x-postal-tag"]
-                log "#{log_prefix} Added tag #{tag.last}"
-                queued_message.message.update(tag: tag.last)
-              end
-
-              #
-              # If the credentials for this message is marked as holding and this isn't manual, hold it
-              #
-              if !queued_message.manual? && queued_message.message.credential && queued_message.message.credential.hold?
-                log "#{log_prefix} Credential wants us to hold messages. Holding."
-                queued_message.message.create_delivery("Held", details: "Credential is configured to hold all messages authenticated by it.")
-                queued_message.destroy
-                next
-              end
-
-              #
-              # If the recipient is on the suppression list and this isn't a manual queueing block sending
-              #
-              if !queued_message.manual? && sl = queued_message.server.message_db.suppression_list.get(:recipient, queued_message.message.rcpt_to)
-                log "#{log_prefix} Recipient is on the suppression list. Holding."
-                queued_message.message.create_delivery("Held", details: "Recipient (#{queued_message.message.rcpt_to}) is on the suppression list (reason: #{sl['reason']})")
-                queued_message.destroy
-                next
-              end
-
-              # Parse the content of the message as appropriate
-              if queued_message.message.should_parse?
-                log "#{log_prefix} Parsing message content as it hasn't been parsed before"
-                queued_message.message.parse_content
-              end
-
-              # Inspect outgoing messages when there's a threshold set for the server
-              if !queued_message.message.inspected && queued_message.server.outbound_spam_threshold
-                log "#{log_prefix} Inspecting message"
-                queued_message.message.inspect_message
-                if queued_message.message.inspected
-                  if queued_message.message.spam_score >= queued_message.server.outbound_spam_threshold
-                    queued_message.message.update(spam: true)
-                  end
-                  log "#{log_prefix} Message inspected successfully"
-                end
-              end
-
-              if queued_message.message.spam
-                queued_message.message.create_delivery("HardFail", details: "Message is likely spam. Threshold is #{queued_message.server.outbound_spam_threshold} and the message scored #{queued_message.message.spam_score}.")
-                queued_message.destroy
-                log "#{log_prefix} Message is spam (#{queued_message.message.spam_score}). Hard failing."
-                next
-              end
-
-              # Add outgoing headers
-              unless queued_message.message.has_outgoing_headers?
-                queued_message.message.add_outgoing_headers
-              end
-
-              # Check send limits
-              if queued_message.server.send_limit_exceeded?
-                # If we're over the limit, we're going to be holding this message
-                queued_message.server.update_columns(send_limit_exceeded_at: Time.now, send_limit_approaching_at: nil)
-                queued_message.message.create_delivery("Held", details: "Message held because send limit (#{queued_message.server.send_limit}) has been reached.")
-                queued_message.destroy
-                log "#{log_prefix} Server send limit has been exceeded. Holding."
-                next
-              elsif queued_message.server.send_limit_approaching?
-                # If we're approaching the limit, just say we are but continue to process the message
-                queued_message.server.update_columns(send_limit_approaching_at: Time.now, send_limit_exceeded_at: nil)
-              else
-                queued_message.server.update_columns(send_limit_approaching_at: nil, send_limit_exceeded_at: nil)
-              end
-
-              # Update the live stats for this message.
-              queued_message.message.database.live_stats.increment(queued_message.message.scope)
-
-              # If the server is in development mode, hold it
-              if queued_message.server.mode == "Development" && !queued_message.manual?
-                log "Server is in development mode so holding."
-                queued_message.message.create_delivery("Held", details: "Server is in development mode.")
-                queued_message.destroy
-                log "#{log_prefix} Server is in development mode. Holding."
-                next
-              end
-
-              # Send the outgoing message to the SMTP sender
-
-              if @fixed_result
-                result = @fixed_result
-              else
-                sender = cached_sender(Postal::SMTPSender, queued_message.message.recipient_domain, queued_message.ip_address)
-                result = sender.send_message(queued_message.message)
-                if result.connect_error
-                  @fixed_result = result
-                end
-              end
-
-              #
-              # If the message has been hard failed, check to see how many other recent hard fails we've had for the address
-              # and if there are more than 2, suppress the address for 30 days.
-              #
-              if result.type == "HardFail"
-                recent_hard_fails = queued_message.server.message_db.select(:messages, where: { rcpt_to: queued_message.message.rcpt_to, status: "HardFail", timestamp: { greater_than: 24.hours.ago.to_f } }, count: true)
-                if recent_hard_fails >= 1 && queued_message.server.message_db.suppression_list.add(:recipient, queued_message.message.rcpt_to, reason: "too many hard fails")
-                  log "#{log_prefix} Added #{queued_message.message.rcpt_to} to suppression list because #{recent_hard_fails} hard fails in 24 hours"
-                  result.details += "." if result.details =~ /\.\z/
-                  result.details += " Recipient added to suppression list (too many hard fails)."
-                end
-              end
-
-              #
-              # If a message is sent successfully, remove the users from the suppression list
-              #
-              if result.type == "Sent" && queued_message.server.message_db.suppression_list.remove(:recipient, queued_message.message.rcpt_to)
-                log "#{log_prefix} Removed #{queued_message.message.rcpt_to} from suppression list because success"
-                result.details += "." if result.details =~ /\.\z/
-                result.details += " Recipient removed from suppression list."
-              end
-
-              # Log the result
-              queued_message.message.create_delivery(result.type, details: result.details, output: result.output, sent_with_ssl: result.secure, log_id: result.log_id, time: result.time)
-              if result.retry
-                log "#{log_prefix} Message requeued for trying later."
-                queued_message.retry_later(result.retry.is_a?(Integer) ? result.retry : nil)
-              else
-                log "#{log_prefix} Processing complete"
-                queued_message.destroy
-              end
-            end
-          rescue StandardError => e
-            log "#{log_prefix} Internal error: #{e.class}: #{e.message}"
-            e.backtrace.each { |line| log("#{log_prefix} #{line}") }
-            queued_message.retry_later
-            log "#{log_prefix} Queued message was unlocked"
-            if defined?(Sentry)
-              Sentry.capture_exception(e, extra: { job_id: self.id, server_id: queued_message.server_id, message_id: queued_message.message_id })
-            end
-            queued_message.message&.create_delivery("Error",
-                                                    details: "An internal error occurred while sending " \
-                                                             "this message. This message will be retried " \
-                                                             "automatically.",
-                                                    output: "#{e.class}: #{e.message}", log_id: "J-#{self.id}")
-          end
-        end
-
-      else
-        log "Couldn't get lock for message #{params['id']}. I won't do this."
-      end
-    else
-      log "No queued message with ID #{params['id']} was available for processing."
-    end
-  ensure
-    begin
-      @sender&.finish
-    rescue StandardError
-      nil
-    end
-  end
-  # rubocop:enable Layout/LineLength
-
-  private
-
-  # rubocop:disable Naming/MemoizedInstanceVariableName
-  def cached_sender(klass, *args)
-    @sender ||= begin
-      sender = klass.new(*args)
-      sender.start
-      sender
-    end
-  end
-  # rubocop:enable Naming/MemoizedInstanceVariableName
-
-end

app/jobs/webhook_delivery_job.rb

@@ -1,17 +0,0 @@
-# frozen_string_literal: true
-
-class WebhookDeliveryJob < Postal::Job
-
-  def perform
-    if webhook_request = WebhookRequest.find_by_id(params["id"])
-      if webhook_request.deliver
-        log "Succesfully delivered"
-      else
-        log "Delivery failed"
-      end
-    else
-      log "No webhook request found with ID '#{params['id']}'"
-    end
-  end
-
-end

app/lib/dkim_header.rb

@@ -0,0 +1,130 @@
+# frozen_string_literal: true
+
+class DKIMHeader
+
+  def initialize(domain, message)
+    if domain && domain.dkim_status == "OK"
+      @domain_name = domain.name
+      @dkim_key = domain.dkim_key
+      @dkim_identifier = domain.dkim_identifier
+    else
+      @domain_name = Postal::Config.dns.return_path_domain
+      @dkim_key = Postal.signer.private_key
+      @dkim_identifier = Postal::Config.dns.dkim_identifier
+    end
+    @domain = domain
+    @message = message
+    @raw_headers, @raw_body = @message.gsub(/\r?\n/, "\r\n").split(/\r\n\r\n/, 2)
+  end
+
+  def dkim_header
+    "DKIM-Signature: v=1; " + dkim_properties.join("\r\n\t") + signature.scan(/.{1,72}/).join("\r\n\t")
+  end
+
+  private
+
+  def headers
+    @headers ||= @raw_headers.to_s.gsub(/\r?\n\s/, " ").split(/\r?\n/)
+  end
+
+  def header_names
+    normalized_headers.map { |h| h.split(":")[0].strip }
+  end
+
+  def normalized_headers
+    [].tap do |new_headers|
+      dkim_headers = headers.select do |h|
+        h.match(/
+          ^(
+            from|sender|reply-to|subject|date|message-id|to|cc|mime-version|content-type|content-transfer-encoding|
+            resent-to|resent-cc|resent-from|resent-sender|resent-message-id|in-reply-to|references|list-id|list-help|
+            list-owner|list-unsubscribe|list-unsubscribe-post|list-subscribe|list-post
+          ):/ix)
+      end
+      dkim_headers.each do |h|
+        new_headers << normalize_header(h)
+      end
+    end
+  end
+
+  def normalize_header(content)
+    content = content.dup
+
+    # From the DKIM RFC6376
+    # https://datatracker.ietf.org/doc/html/rfc6376#section-3.4.2
+
+    # Split the key and value.
+    key, value = content.split(":", 2)
+
+    # Convert all header field names (not the header field values) to
+    # lowercase.  For example, convert "SUBJect: AbC" to "subject: AbC".
+    key.downcase!
+
+    # Unfold all header field continuation lines as described in [RFC5322]
+    value.gsub!(/\r?\n[ \t]+/, " ")
+
+    # Convert all sequences of one or more WSP characters to a single SP character.
+    value.gsub!(/[ \t]+/, " ")
+
+    # Delete all WSP characters at the end of each unfolded header field value.
+    value.gsub!(/[ \t]*\z/, "")
+
+    # Delete any WSP characters remaining after the colon separating the header field name from the header field value.
+    value.gsub!(/\A[ \t]*/, "")
+
+    # Join together
+    key + ":" + value
+  end
+
+  def normalized_body
+    @normalized_body ||= begin
+      content = @raw_body.dup
+
+      # From the DKIM RFC6376
+      # https://datatracker.ietf.org/doc/html/rfc6376#section-3.4.4
+
+      # a. Reduce whitespace
+      #
+      # * Reduce all sequences of WSP within a line to a single SP character.
+      content.gsub!(/[ \t]+/, " ")
+
+      # * Ignore all whitespace at the end of lines.  Implementations MUST NOT
+      #   remove the CRLF at the end of the line.
+      content.gsub!(/ \r\n/, "\r\n")
+
+      # b. Ignore all empty lines at the end of the message body.
+      content.gsub!(/[ \r\n]*\z/, "")
+
+      content += "\r\n"
+      content
+    end
+  end
+
+  def body_hash
+    @body_hash ||= Base64.encode64(Digest::SHA256.digest(normalized_body)).strip
+  end
+
+  def dkim_properties
+    @dkim_properties ||= [].tap do |header|
+      header << "a=rsa-sha256; c=relaxed/relaxed;"
+      header << "d=#{@domain_name};"
+      header << "s=#{@dkim_identifier}; t=#{Time.now.utc.to_i};"
+      header << "bh=#{body_hash};"
+      header << "h=#{header_names.join(':')};"
+      header << "b="
+    end
+  end
+
+  def dkim_header_for_signing
+    "dkim-signature:v=1; #{dkim_properties.join(' ')}"
+  end
+
+  def signable_header_string
+    (normalized_headers + [dkim_header_for_signing]).join("\r\n")
+  end
+
+  def signature
+    Base64.encode64(@dkim_key.sign(OpenSSL::Digest.new("SHA256"), signable_header_string)).gsub("\n", "")
+  end
+
+end

app/lib/dns_resolver.rb

@@ -0,0 +1,163 @@
+# frozen_string_literal: true
+
+require "resolv"
+
+class DNSResolver
+
+  class LocalResolversUnavailableError < StandardError
+  end
+
+  attr_reader :nameservers
+  attr_reader :timeout
+
+  def initialize(nameservers)
+    @nameservers = nameservers
+  end
+
+  # Return all A records for the given name
+  #
+  # @param [String] name
+  # @return [Array<String>]
+  def a(name, **options)
+    get_resources(name, Resolv::DNS::Resource::IN::A, **options).map do |s|
+      s.address.to_s
+    end
+  end
+
+  # Return all AAAA records for the given name
+  #
+  # @param [String] name
+  # @return [Array<String>]
+  def aaaa(name, **options)
+    get_resources(name, Resolv::DNS::Resource::IN::AAAA, **options).map do |s|
+      s.address.to_s
+    end
+  end
+
+  # Return all TXT records for the given name
+  #
+  # @param [String] name
+  # @return [Array<String>]
+  def txt(name, **options)
+    get_resources(name, Resolv::DNS::Resource::IN::TXT, **options).map do |s|
+      s.data.to_s.strip
+    end
+  end
+
+  # Return all CNAME records for the given name
+  #
+  # @param [String] name
+  # @return [Array<String>]
+  def cname(name, **options)
+    get_resources(name, Resolv::DNS::Resource::IN::CNAME, **options).map do |s|
+      s.name.to_s.downcase
+    end
+  end
+
+  # Return all MX records for the given name
+  #
+  # @param [String] name
+  # @return [Array<Array<Integer, String>>]
+  def mx(name, **options)
+    records = get_resources(name, Resolv::DNS::Resource::IN::MX, **options).map do |m|
+      [m.preference.to_i, m.exchange.to_s]
+    end
+    records.sort do |a, b|
+      if a[0] == b[0]
+        [-1, 1].sample
+      else
+        a[0] <=> b[0]
+      end
+    end
+  end
+
+  # Return the effective nameserver names for a given domain name.
+  #
+  # @param [String] name
+  # @return [Array<String>]
+  def effective_ns(name, **options)
+    records = []
+    parts = name.split(".")
+    (parts.size - 1).times do |n|
+      d = parts[n, parts.size - n + 1].join(".")
+
+      records = get_resources(d, Resolv::DNS::Resource::IN::NS, **options).map do |s|
+        s.name.to_s
+      end
+
+      break if records.present?
+    end
+
+    records
+  end
+
+  # Return the hostname for a given IP address.
+  # Returns the IP address itself if no hostname can be determined.
+  #
+  # @param [String] ip_address
+  # @return [String]
+  def ip_to_hostname(ip_address, **options)
+    dns(**options) do |dns|
+      dns.getname(ip_address)&.to_s
+    end
+  rescue Resolv::ResolvError => e
+    raise if e.message =~ /timeout/ && options[:raise_timeout_errors]
+
+    ip_address
+  end
+
+  private
+
+  def dns(raise_timeout_errors: false)
+    Resolv::DNS.open(nameserver: @nameservers,
+                     raise_timeout_errors: raise_timeout_errors) do |dns|
+      dns.timeouts = [
+        Postal::Config.dns.timeout,
+        Postal::Config.dns.timeout / 2,
+        Postal::Config.dns.timeout / 2,
+      ]
+      yield dns
+    end
+  end
+
+  def get_resources(name, type, **options)
+    encoded_name = DomainName::Punycode.encode_hostname(name)
+    dns(**options) do |dns|
+      dns.getresources(encoded_name, type)
+    end
+  end
+
+  class << self
+
+    # Return a resolver which will use the nameservers for the given domain
+    #
+    # @param [String] name
+    # @return [DNSResolver]
+    def for_domain(name)
+      nameservers = local.effective_ns(name)
+      ips = nameservers.map do |ns|
+        local.a(ns)
+      end.flatten.uniq
+      new(ips)
+    end
+
+    # Return a local resolver to use for lookups
+    #
+    # @return [DNSResolver]
+    def local
+      @local ||= begin
+        resolv_conf_path = Postal::Config.dns.resolv_conf_path
+        raise LocalResolversUnavailableError, "No resolver config found at #{resolv_conf_path}" unless File.file?(resolv_conf_path)
+
+        resolv_conf = Resolv::DNS::Config.parse_resolv_conf(resolv_conf_path)
+        if resolv_conf.nil? || resolv_conf[:nameserver].nil? || resolv_conf[:nameserver].empty?
+          raise LocalResolversUnavailableError, "Could not find nameservers in #{resolv_conf_path}"
+        end
+
+        new(resolv_conf[:nameserver])
+      end
+    end
+
+  end
+
+end

app/lib/message_dequeuer.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+module MessageDequeuer
+
+  class << self
+
+    def process(message, logger:)
+      processor = InitialProcessor.new(message, logger: logger)
+      processor.process
+    end
+
+  end
+
+end

app/lib/message_dequeuer/base.rb

@@ -0,0 +1,108 @@
+# frozen_string_literal: true
+
+module MessageDequeuer
+  class Base
+
+    class StopProcessing < StandardError
+    end
+
+    attr_reader :queued_message
+    attr_reader :logger
+    attr_reader :state
+
+    def initialize(queued_message, logger:, state: nil)
+      @queued_message = queued_message
+      @logger = logger
+      @state = state || State.new
+    end
+
+    def process
+      raise NotImplemented
+    end
+
+    class << self
+
+      def process(message, **kwargs)
+        new(message, **kwargs).process
+      end
+
+    end
+
+    private
+
+    def stop_processing
+      raise StopProcessing
+    end
+
+    def catch_stops
+      yield if block_given?
+      true
+    rescue StopProcessing
+      false
+    end
+
+    def remove_from_queue
+      @queued_message.destroy
+    end
+
+    def create_delivery(type, **kwargs)
+      @queued_message.message.create_delivery(type, **kwargs)
+    end
+
+    def log(text, **tags)
+      logger.info text, **tags
+    end
+
+    def increment_live_stats
+      queued_message.message.database.live_stats.increment(queued_message.message.scope)
+    end
+
+    def hold_if_server_development_mode
+      return if queued_message.manual?
+      return unless queued_message.server.mode == "Development"
+
+      log "server is in development mode, holding"
+      create_delivery "Held", details: "Server is in development mode."
+      remove_from_queue
+      stop_processing
+    end
+
+    def log_sender_result
+      log_details = @result.details
+
+      if @additional_delivery_details
+        log_details += "." unless log_details =~ /\.\z/
+        log_details += " "
+        log_details += @additional_delivery_details
+      end
+
+      create_delivery @result.type, details: log_details,
+                                    output: @result.output&.strip,
+                                    sent_with_ssl: @result.secure,
+                                    log_id: @result.log_id,
+                                    time: @result.time
+    end
+
+    def handle_exception(exception)
+      log "internal error: #{exception.class}: #{exception.message}"
+      exception.backtrace.each { |line| log(line) }
+
+      queued_message.retry_later unless queued_message.destroyed?
+      log "message requeued for trying later, at #{queued_message.retry_after}"
+
+      if defined?(Sentry)
+        Sentry.capture_exception(exception, extra: {
+          server_id: queued_message.server_id,
+          queued_message_id: queued_message.message_id
+        })
+      end
+
+      queued_message.message&.create_delivery("Error",
+                                              details: "An internal error occurred while sending " \
+                                                       "this message. This message will be retried " \
+                                                       "automatically.",
+                                              output: "#{exception.class}: #{exception.message}")
+    end
+
+  end
+end

app/lib/message_dequeuer/incoming_message_processor.rb

@@ -0,0 +1,215 @@
+# frozen_string_literal: true
+
+module MessageDequeuer
+  class IncomingMessageProcessor < Base
+
+    attr_reader :route
+
+    def process
+      log "message is incoming"
+
+      catch_stops do
+        handle_bounces
+        increment_live_stats
+        inspect_message
+        fail_if_spam
+        hold_if_server_development_mode
+        find_route
+        hold_or_reject_spam
+        accept_mail_without_endpoints
+        hold_messages
+        bounce_messages
+        send_message_to_sender
+        send_bounce_on_hard_fail
+        log_sender_result
+        finish_processing
+      end
+    rescue StandardError => e
+      handle_exception(e)
+    end
+
+    private
+
+    def handle_bounces
+      return unless queued_message.message.bounce
+
+      log "message is a bounce"
+      original_messages = queued_message.message.original_messages
+      unless original_messages.empty?
+        queued_message.message.original_messages.each do |orig_msg|
+          queued_message.message.update(bounce_for_id: orig_msg.id, domain_id: orig_msg.domain_id)
+          create_delivery "Processed", details: "This has been detected as a bounce message for <msg:#{orig_msg.id}>."
+          orig_msg.bounce!(queued_message.message)
+          log "bounce linked with message #{orig_msg.id}"
+        end
+        remove_from_queue
+        stop_processing
+      end
+
+      # This message was sent to the return path but hasn't been matched
+      # to an original message. If we have a route for this, route it
+      # otherwise we'll drop at this point.
+      return unless queued_message.message.route_id.nil?
+
+      log "no source messages found, hard failing"
+      create_delivery "HardFail", details: "This message was a bounce but we couldn't link it with any outgoing message and there was no route for it."
+      remove_from_queue
+      stop_processing
+    end
+
+    def inspect_message
+      return if queued_message.message.inspected
+
+      log "inspecting message"
+      queued_message.message.inspect_message
+      return unless queued_message.message.inspected
+
+      is_spam = queued_message.message.spam_score > queued_message.server.spam_threshold
+      if is_spam
+        queued_message.message.update(spam: true)
+        log "message is spam (scored #{queued_message.message.spam_score}, threshold is #{queued_message.server.spam_threshold})"
+      end
+
+      queued_message.message.append_headers(
+        "X-Postal-Spam: #{queued_message.message.spam ? 'yes' : 'no'}",
+        "X-Postal-Spam-Threshold: #{queued_message.server.spam_threshold}",
+        "X-Postal-Spam-Score: #{queued_message.message.spam_score}",
+        "X-Postal-Threat: #{queued_message.message.threat ? 'yes' : 'no'}"
+      )
+      log "message inspected, headers added", spam: queued_message.message.spam?, spam_score: queued_message.message.spam_score, threat: queued_message.message.threat?
+    end
+
+    def fail_if_spam
+      return if queued_message.message.spam_score < queued_message.server.spam_failure_threshold
+
+      log "message has a spam score higher than the server's maxmimum, hard failing", server_threshold: queued_message.server.spam_failure_threshold
+      create_delivery "HardFail",
+                      details: "Message's spam score is higher than the failure threshold for this server. " \
+                               "Threshold is currently #{queued_message.server.spam_failure_threshold}."
+      remove_from_queue
+      stop_processing
+    end
+
+    def find_route
+      @route = queued_message.message.route
+      return if @route
+
+      log "no route and/or endpoint available for processing, hard failing"
+      create_delivery "HardFail", details: "Message does not have a route and/or endpoint available for delivery."
+      remove_from_queue
+      stop_processing
+    end
+
+    def hold_or_reject_spam
+      return unless queued_message.message.spam
+      return if queued_message.manual?
+
+      case @route.spam_mode
+      when "Quarantine"
+        log "message is spam and route says to quarantine spam message, holding"
+        create_delivery "Held", details: "Message placed into quarantine."
+      when "Fail"
+        log "message is spam and route says to fail spam message, hard failing"
+        create_delivery "HardFail", details: "Message is spam and the route specifies it should be failed."
+      else
+        return
+      end
+
+      remove_from_queue
+      stop_processing
+    end
+
+    def accept_mail_without_endpoints
+      return unless @route.mode == "Accept"
+
+      log "route says to accept without endpoint, marking as processed"
+      create_delivery "Processed", details: "Message has been accepted but not sent to any endpoints."
+      remove_from_queue
+      stop_processing
+    end
+
+    def hold_messages
+      return unless @route.mode == "Hold"
+
+      if queued_message.manual?
+        log "route says to hold and message was queued manually, marking as processed"
+        create_delivery "Processed", details: "Message has been processed."
+      else
+        log "route says to hold, marking as held"
+        create_delivery "Held", details: "Message has been accepted but not sent to any endpoints."
+      end
+
+      remove_from_queue
+      stop_processing
+    end
+
+    def bounce_messages
+      return unless route.mode == "Bounce" || route.mode == "Reject"
+
+      log "route says to bounce, hard failing and sending bounce"
+
+      if id = queued_message.send_bounce
+        log "bounce sent with id #{id}"
+        create_delivery "HardFail", details: "Message has been bounced because the route asks for this. See message <msg:#{id}>"
+      end
+
+      remove_from_queue
+      stop_processing
+    end
+
+    def send_message_to_sender
+      @result = @state.send_result
+      return if @result
+
+      case queued_message.message.endpoint
+      when SMTPEndpoint
+        sender = @state.sender_for(SMTPSender, queued_message.message.recipient_domain, nil, servers: [queued_message.message.endpoint.to_smtp_client_server])
+      when HTTPEndpoint
+        sender = @state.sender_for(HTTPSender, queued_message.message.endpoint)
+      when AddressEndpoint
+        sender = @state.sender_for(SMTPSender, queued_message.message.endpoint.domain, nil, rcpt_to: queued_message.message.endpoint.address)
+      else
+        log "invalid endpoint for route (#{queued_message.message.endpoint_type})"
+        create_delivery "HardFail", details: "Invalid endpoint for route."
+        remove_from_queue
+        stop_processing
+      end
+
+      @result = sender.send_message(queued_message.message)
+      return unless @result.connect_error
+
+      @state.send_result = @result
+    end
+
+    def send_bounce_on_hard_fail
+      return unless @result.type == "HardFail"
+
+      if @result.suppress_bounce
+        log "suppressing bounce message after hard fail"
+        return
+      end
+
+      return unless queued_message.message.send_bounces?
+
+      log "sending a bounce because message hard failed"
+      return unless bounce_id = queued_message.send_bounce
+
+      @additional_delivery_details = "Sent bounce message to sender (see message <msg:#{bounce_id}>)"
+    end
+
+    def finish_processing
+      if @result.retry
+        queued_message.retry_later(@result.retry.is_a?(Integer) ? @result.retry : nil)
+        log "message requeued for trying later, at #{queued_message.retry_after}"
+        queued_message.allocate_ip_address
+        queued_message.update_column(:ip_address_id, queued_message.ip_address&.id)
+        stop_processing
+      end
+
+      log "message processing completed"
+      queued_message.message.endpoint.mark_as_used
+      remove_from_queue
+    end
+
+  end
+end

app/lib/message_dequeuer/initial_processor.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+module MessageDequeuer
+  class InitialProcessor < Base
+
+    include HasPrometheusMetrics
+
+    attr_accessor :send_result
+
+    def process
+      logger.tagged(original_queued_message: @queued_message.id) do
+        logger.info "starting message unqueue"
+        begin
+          catch_stops do
+            increment_dequeue_metric
+            check_message_exists
+            check_message_is_ready
+            find_other_messages_for_batch
+
+            # Process the original message and then all of those
+            # found for batching.
+            process_message(@queued_message)
+            @other_messages&.each { |message| process_message(message) }
+          end
+        ensure
+          @state.finished
+        end
+        logger.info "finished message unqueue"
+      end
+    end
+
+    private
+
+    def increment_dequeue_metric
+      time_in_queue = Time.now.to_f - @queued_message.created_at.to_f
+      log "queue latency is #{time_in_queue}s"
+      observe_prometheus_histogram :postal_message_queue_latency,
+                                   time_in_queue
+    end
+
+    def check_message_exists
+      return if @queued_message.message
+
+      log "unqueue because backend message has been removed."
+      remove_from_queue
+      stop_processing
+    end
+
+    def check_message_is_ready
+      return if @queued_message.ready?
+
+      log "skipping because message isn't ready for processing"
+      @queued_message.unlock
+      stop_processing
+    end
+
+    def find_other_messages_for_batch
+      return unless Postal::Config.postal.batch_queued_messages?
+
+      @other_messages = @queued_message.batchable_messages(100)
+      log "found #{@other_messages.size} associated messages to process at the same time", batch_key: @queued_message.batch_key
+    rescue StandardError
+      @queued_message.unlock
+      raise
+    end
+
+    def process_message(queued_message)
+      logger.tagged(queued_message: queued_message.id) do
+        SingleMessageProcessor.process(queued_message, logger: @logger, state: @state)
+      end
+    end
+
+  end
+end

app/lib/message_dequeuer/outgoing_message_processor.rb

@@ -0,0 +1,190 @@
+# frozen_string_literal: true
+
+module MessageDequeuer
+  class OutgoingMessageProcessor < Base
+
+    def process
+      catch_stops do
+        check_domain
+        check_rcpt_to
+        add_tag
+        hold_if_credential_is_set_to_hold
+        hold_if_recipient_on_suppression_list
+        parse_content
+        inspect_message
+        fail_if_spam
+        add_outgoing_headers
+        check_send_limits
+        increment_live_stats
+        hold_if_server_development_mode
+        send_message_to_sender
+        add_recipient_to_suppression_list_on_too_many_hard_fails
+        remove_recipient_from_suppression_list_on_success
+        log_sender_result
+        finish_processing
+      end
+    rescue StandardError => e
+      handle_exception(e)
+    end
+
+    private
+
+    def check_domain
+      return if queued_message.message.domain
+
+      log "message has no domain, hard failing"
+      create_delivery "HardFail", details: "Message's domain no longer exist"
+      remove_from_queue
+      stop_processing
+    end
+
+    def check_rcpt_to
+      return unless queued_message.message.rcpt_to.blank?
+
+      log "message has no 'to' address, hard failing"
+      create_delivery "HardFail", details: "Message doesn't have an RCPT to"
+      remove_from_queue
+      stop_processing
+    end
+
+    def add_tag
+      return if queued_message.message.tag
+      return unless tag = queued_message.message.headers["x-postal-tag"]
+
+      log "added tag: #{tag.last}"
+      queued_message.message.update(tag: tag.last)
+    end
+
+    def hold_if_credential_is_set_to_hold
+      return if queued_message.manual?
+      return if queued_message.message.credential.nil?
+      return unless queued_message.message.credential.hold?
+
+      log "credential wants us to hold messages, holding"
+      create_delivery "Held", details: "Credential is configured to hold all messages authenticated by it."
+      remove_from_queue
+      stop_processing
+    end
+
+    def hold_if_recipient_on_suppression_list
+      return if queued_message.manual?
+      return unless sl = queued_message.server.message_db.suppression_list.get(:recipient, queued_message.message.rcpt_to)
+
+      log "recipient is on the suppression list, holding"
+      create_delivery "Held", details: "Recipient (#{queued_message.message.rcpt_to}) is on the suppression list (reason: #{sl['reason']})"
+      remove_from_queue
+      stop_processing
+    end
+
+    def parse_content
+      return unless queued_message.message.should_parse?
+
+      log "parsing message content as it hasn't been parsed before"
+      queued_message.message.parse_content
+    end
+
+    def inspect_message
+      return if queued_message.message.inspected
+      return unless queued_message.server.outbound_spam_threshold
+
+      log "inspecting message"
+      queued_message.message.inspect_message
+      return unless queued_message.message.inspected
+
+      if queued_message.message.spam_score >= queued_message.server.outbound_spam_threshold
+        queued_message.message.update(spam: true)
+      end
+
+      log "message inspected successfully", spam: queued_message.message.spam?, spam_score: queued_message.message.spam_score
+    end
+
+    def fail_if_spam
+      return unless queued_message.message.spam
+
+      log "message is spam (#{queued_message.message.spam_score}), hard failing", server_threshold: queued_message.server.outbound_spam_threshold
+      create_delivery "HardFail",
+                      details: "Message is likely spam. Threshold is #{queued_message.server.outbound_spam_threshold} and " \
+                               "the message scored #{queued_message.message.spam_score}."
+      remove_from_queue
+      stop_processing
+    end
+
+    def add_outgoing_headers
+      return if queued_message.message.has_outgoing_headers?
+
+      queued_message.message.add_outgoing_headers
+    end
+
+    def check_send_limits
+      if queued_message.server.send_limit_exceeded?
+        # If we're over the limit, we're going to be holding this message
+        log "server send limit has been exceeded, holding", send_limit: queued_message.server.send_limit
+        queued_message.server.update_columns(send_limit_exceeded_at: Time.now, send_limit_approaching_at: nil)
+        create_delivery "Held", details: "Message held because send limit (#{queued_message.server.send_limit}) has been reached."
+        remove_from_queue
+        stop_processing
+      elsif queued_message.server.send_limit_approaching?
+        # If we're approaching the limit, just say we are but continue to process the message
+        queued_message.server.update_columns(send_limit_approaching_at: Time.now, send_limit_exceeded_at: nil)
+      else
+        queued_message.server.update_columns(send_limit_approaching_at: nil, send_limit_exceeded_at: nil)
+      end
+    end
+
+    def send_message_to_sender
+      @result = @state.send_result
+      return if @result
+
+      sender = @state.sender_for(SMTPSender,
+                                 queued_message.message.recipient_domain,
+                                 queued_message.ip_address)
+
+      @result = sender.send_message(queued_message.message)
+      return unless @result.connect_error
+
+      @state.send_result = @result
+    end
+
+    def add_recipient_to_suppression_list_on_too_many_hard_fails
+      return unless @result.type == "HardFail"
+
+      recent_hard_fails = queued_message.server.message_db.select(:messages,
+                                                                  where: {
+                                                                    rcpt_to: queued_message.message.rcpt_to,
+                                                                    status: "HardFail",
+                                                                    timestamp: { greater_than: 24.hours.ago.to_f }
+                                                                  },
+                                                                  count: true)
+      return if recent_hard_fails < 1
+
+      added = queued_message.server.message_db.suppression_list.add(:recipient, queued_message.message.rcpt_to,
+                                                                    reason: "too many hard fails")
+      return unless added
+
+      log "Added #{queued_message.message.rcpt_to} to suppression list because #{recent_hard_fails} hard fails in 24 hours"
+      @additional_delivery_details = "Recipient added to suppression list (too many hard fails)"
+    end
+
+    def remove_recipient_from_suppression_list_on_success
+      return unless @result.type == "Sent"
+
+      removed = queued_message.server.message_db.suppression_list.remove(:recipient, queued_message.message.rcpt_to)
+      return unless removed
+
+      log "removed #{queued_message.message.rcpt_to} from suppression list"
+      @additional_delivery_details = "Recipient removed from suppression list"
+    end
+
+    def finish_processing
+      if @result.retry
+        queued_message.retry_later(@result.retry.is_a?(Integer) ? @result.retry : nil)
+        log "message requeued for trying later", retry_after: queued_message.retry_after
+        stop_processing
+      end
+
+      log "message processing complete"
+      remove_from_queue
+    end
+
+  end
+end

app/lib/message_dequeuer/single_message_processor.rb

@@ -0,0 +1,83 @@
+# frozen_string_literal: true
+
+module MessageDequeuer
+  class SingleMessageProcessor < Base
+
+    def process
+      catch_stops do
+        check_message_exists
+        check_server_suspension
+        check_delivery_attempts
+        check_raw_message_exists
+
+        processor = nil
+        case queued_message.message.scope
+        when "incoming"
+          processor = IncomingMessageProcessor
+        when "outgoing"
+          processor = OutgoingMessageProcessor
+        else
+          create_delivery "HardFail", details: "Scope #{queued_message.message.scope} is not valid"
+          remove_from_queue
+          stop_processing
+        end
+
+        processor.process(queued_message, logger: @logger, state: @state)
+      end
+    rescue StandardError => e
+      handle_exception(e)
+    end
+
+    private
+
+    def check_message_exists
+      return if queued_message.message
+
+      log "unqueueing because backend message has been removed"
+      remove_from_queue
+      stop_processing
+    end
+
+    def check_server_suspension
+      return unless queued_message.server.suspended?
+
+      log "server is suspended, holding message"
+      create_delivery "Held", details: "Mail server has been suspended. No e-mails can be processed at present. Contact support for assistance."
+      remove_from_queue
+      stop_processing
+    end
+
+    def check_delivery_attempts
+      return if queued_message.attempts < Postal::Config.postal.default_maximum_delivery_attempts
+
+      details = "Maximum number of delivery attempts (#{queued_message.attempts}) has been reached."
+      if queued_message.message.scope == "incoming"
+        # Send bounces to incoming e-mails when they are hard failed
+        if bounce_id = queued_message.send_bounce
+          details += " Bounce sent to sender (see message <msg:#{bounce_id}>)"
+        end
+      elsif queued_message.message.scope == "outgoing"
+        # Add the recipient to the suppression list
+        if queued_message.server.message_db.suppression_list.add(:recipient, queued_message.message.rcpt_to, reason: "too many soft fails")
+          log "added #{queued_message.message.rcpt_to} to suppression list because maximum attempts has been reached"
+          details += " Added #{queued_message.message.rcpt_to} to suppression list because delivery has failed #{queued_message.attempts} times."
+        end
+      end
+
+      log "message has reached maximum number of attempts, hard failing"
+      create_delivery "HardFail", details: details
+      remove_from_queue
+      stop_processing
+    end
+
+    def check_raw_message_exists
+      return if queued_message.message.raw_message?
+
+      log "raw message has been removed, not sending"
+      create_delivery "HardFail", details: "Raw message has been removed. Cannot send message."
+      remove_from_queue
+      stop_processing
+    end
+
+  end
+end

app/lib/message_dequeuer/state.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module MessageDequeuer
+  class State
+
+    attr_accessor :send_result
+
+    def sender_for(klass, *args, **kwargs)
+      @cached_senders ||= {}
+      @cached_senders[[klass, args, kwargs]] ||= begin
+        klass_instance = klass.new(*args, **kwargs)
+        klass_instance.start
+        klass_instance
+      end
+    end
+
+    def finished
+      @cached_senders&.each_value do |sender|
+        sender.finish
+      rescue StandardError
+        false
+      end
+    end
+
+  end
+end

app/lib/query_string.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+class QueryString
+
+  def initialize(string)
+    @string = string.strip + " "
+  end
+
+  def [](value)
+    hash[value.to_s]
+  end
+
+  delegate :empty?, to: :hash
+
+  def hash
+    @hash ||= @string.scan(/([a-z]+):\s*(?:(\d{2,4}-\d{2}-\d{2}\s\d{2}:\d{2})|"(.*?)"|(.*?))(\s|\z)/).each_with_object({}) do |(key, date, string_with_spaces, value), hash|
+      if date
+        actual_value = date
+      elsif string_with_spaces
+        actual_value = string_with_spaces
+      elsif value == "[blank]"
+        actual_value = nil
+      else
+        actual_value = value
+      end
+
+      if hash.keys.include?(key.to_s)
+        hash[key.to_s] = [hash[key.to_s]].flatten
+        hash[key.to_s] << actual_value
+      else
+        hash[key.to_s] = actual_value
+      end
+    end
+  end
+
+end

app/lib/received_header.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+class ReceivedHeader
+
+  OUR_HOSTNAMES = {
+    smtp: Postal::Config.postal.smtp_hostname,
+    http: Postal::Config.postal.web_hostname
+  }.freeze
+
+  class << self
+
+    def generate(server, helo, ip_address, method)
+      our_hostname = OUR_HOSTNAMES[method]
+      if our_hostname.nil?
+        raise Error, "`method` is invalid (must be one of #{OUR_HOSTNAMES.join(', ')})"
+      end
+
+      header = "by #{our_hostname} with #{method.to_s.upcase}; #{Time.now.utc.rfc2822}"
+
+      if server.nil? || server.privacy_mode == false
+        hostname = DNSResolver.local.ip_to_hostname(ip_address)
+        header = "from #{helo} (#{hostname} [#{ip_address}]) #{header}"
+      end
+
+      header
+    end
+
+  end
+
+end

app/lib/reply_separator.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+class ReplySeparator
+
+  RULES = [
+    /^-{2,10} $.*/m,
+    /^>*\s*----- ?Original Message ?-----.*/m,
+    /^>*\s*From:[^\r\n]*[\r\n]+Sent:.*/m,
+    /^>*\s*From:[^\r\n]*[\r\n]+Date:.*/m,
+    /^>*\s*-----Urspr.ngliche Nachricht----- .*/m,
+    /^>*\s*Le[^\r\n]{10,200}a .crit ?:\s*$.*/,
+    /^>*\s*__________________.*/m,
+    /^>*\s*On.{10,200}wrote:\s*$.*/m,
+    /^>*\s*Sent from my.*/m,
+    /^>*\s*=== Please reply above this line ===.*/m,
+    /(^>.*\n?){10,}/,
+  ].freeze
+
+  def self.separate(text)
+    return "" unless text.is_a?(String)
+
+    text = text.gsub("\r", "")
+    stripped = String.new
+    RULES.each do |rule|
+      text.gsub!(rule) do
+        stripped = ::Regexp.last_match(0).to_s + "\n" + stripped
+        ""
+      end
+    end
+    stripped = stripped.strip
+    [text.strip, stripped.presence]
+  end
+
+end

app/lib/smtp_client/endpoint.rb

@@ -0,0 +1,169 @@
+# frozen_string_literal: true
+
+module SMTPClient
+  class Endpoint
+
+    class SMTPSessionNotStartedError < StandardError
+    end
+
+    attr_reader :server
+    attr_reader :ip_address
+    attr_accessor :smtp_client
+
+    # @param server [Server] the server that this IP address is for
+    # @param ip_address [String] the IP address
+    def initialize(server, ip_address)
+      @server = server
+      @ip_address = ip_address
+    end
+
+    # Return a description of this server with its IP address
+    #
+    # @return [String]
+    def description
+      "#{@ip_address}:#{@server.port} (#{@server.hostname})"
+    end
+
+    # Return a string representation of this server
+    #
+    # @return [String]
+    def to_s
+      description
+    end
+
+    # Return true if this is an IPv6 address
+    #
+    # @return [Boolean]
+    def ipv6?
+      @ip_address.include?(":")
+    end
+
+    # Return true if this is an IPv4 address
+    #
+    # @return [Boolean]
+    def ipv4?
+      !ipv6?
+    end
+
+    # Start a new SMTP session and store the client with this server for future use as needed
+    #
+    # @param source_ip_address [IPAddress] the IP address to use as the source address for the connection
+    # @param allow_ssl [Boolean] whether to allow SSL for this connection, if false SSL mode is ignored
+    #
+    # @return [Net::SMTP]
+    def start_smtp_session(source_ip_address: nil, allow_ssl: true)
+      @smtp_client = Net::SMTP.new(@ip_address, @server.port)
+      @smtp_client.open_timeout = Postal::Config.smtp_client.open_timeout
+      @smtp_client.read_timeout = Postal::Config.smtp_client.read_timeout
+      @smtp_client.tls_hostname = @server.hostname
+
+      if source_ip_address
+        @source_ip_address = source_ip_address
+      end
+
+      if @source_ip_address
+        @smtp_client.source_address = ipv6? ? @source_ip_address.ipv6 : @source_ip_address.ipv4
+      end
+
+      if allow_ssl
+        case @server.ssl_mode
+        when SSLModes::AUTO
+          @smtp_client.enable_starttls_auto(self.class.ssl_context_without_verify)
+        when SSLModes::STARTTLS
+          @smtp_client.enable_starttls(self.class.ssl_context_with_verify)
+        when SSLModes::TLS
+          @smtp_client.enable_tls(self.class.ssl_context_with_verify)
+        else
+          @smtp_client.disable_starttls
+          @smtp_client.disable_tls
+        end
+      else
+        @smtp_client.disable_starttls
+        @smtp_client.disable_tls
+      end
+
+      @smtp_client.start(@source_ip_address ? @source_ip_address.hostname : self.class.default_helo_hostname)
+
+      @smtp_client
+    end
+
+    # Send a message to the current SMTP session (or create one if there isn't one for this endpoint).
+    # If sending messsage encouters some connection errors, retry again after re-establishing the SMTP
+    # session.
+    #
+    # @param raw_message [String] the raw message to send
+    # @param mail_from [String] the MAIL FROM address
+    # @param rcpt_to [String] the RCPT TO address
+    # @param retry_on_connection_error [Boolean] whether to retry the connection if there is a connection error
+    #
+    # @return [void]
+    def send_message(raw_message, mail_from, rcpt_to, retry_on_connection_error: true)
+      raise SMTPSessionNotStartedError if @smtp_client.nil? || (@smtp_client && !@smtp_client.started?)
+
+      @smtp_client.rset_errors
+      @smtp_client.send_message(raw_message, mail_from, [rcpt_to])
+    rescue Errno::ECONNRESET, Errno::EPIPE, OpenSSL::SSL::SSLError
+      if retry_on_connection_error
+        finish_smtp_session
+        start_smtp_session
+        return send_message(raw_message, mail_from, rcpt_to, retry_on_connection_error: false)
+      end
+
+      raise
+    end
+
+    # Reset the current SMTP session for this server if possible otherwise
+    # finish the session
+    #
+    # @return [void]
+    def reset_smtp_session
+      @smtp_client&.rset
+    rescue StandardError
+      finish_smtp_session
+    end
+
+    # Finish the current SMTP session for this server if possible.
+    #
+    # @return [void]
+    def finish_smtp_session
+      @smtp_client&.finish
+    rescue StandardError
+      nil
+    ensure
+      @smtp_client = nil
+    end
+
+    class << self
+
+      # Return the default HELO hostname to present to SMTP servers that
+      # we connect to
+      #
+      # @return [String]
+      def default_helo_hostname
+        Postal::Config.dns.helo_hostname ||
+          Postal::Config.postal.smtp_hostname ||
+          "localhost"
+      end
+
+      def ssl_context_with_verify
+        @ssl_context_with_verify ||= begin
+          c = OpenSSL::SSL::SSLContext.new
+          c.verify_mode = OpenSSL::SSL::VERIFY_PEER
+          c.cert_store = OpenSSL::X509::Store.new
+          c.cert_store.set_default_paths
+          c
+        end
+      end
+
+      def ssl_context_without_verify
+        @ssl_context_without_verify ||= begin
+          c = OpenSSL::SSL::SSLContext.new
+          c.verify_mode = OpenSSL::SSL::VERIFY_NONE
+          c
+        end
+      end
+
+    end
+
+  end
+end

app/lib/smtp_client/server.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module SMTPClient
+  class Server
+
+    attr_reader :hostname
+    attr_reader :port
+    attr_accessor :ssl_mode
+
+    def initialize(hostname, port: 25, ssl_mode: SSLModes::AUTO)
+      @hostname = hostname
+      @port = port
+      @ssl_mode = ssl_mode
+    end
+
+    # Return all IP addresses for this server by resolving its hostname.
+    # IPv6 addresses will be returned first.
+    #
+    # @return [Array<SMTPClient::Endpoint>]
+    def endpoints
+      ips = []
+
+      DNSResolver.local.aaaa(@hostname).each do |ip|
+        ips << Endpoint.new(self, ip)
+      end
+
+      DNSResolver.local.a(@hostname).each do |ip|
+        ips << Endpoint.new(self, ip)
+      end
+
+      ips
+    end
+
+  end
+end

app/lib/smtp_client/ssl_modes.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module SMTPClient
+  module SSLModes
+
+    AUTO = "Auto"
+    STARTTLS = "STARTLS"
+    TLS = "TLS"
+    NONE = "None"
+
+  end
+end

app/lib/smtp_server/client.rb

@@ -0,0 +1,597 @@
+# frozen_string_literal: true
+
+module SMTPServer
+  class Client
+
+    extend HasPrometheusMetrics
+    include HasPrometheusMetrics
+
+    CRAM_MD5_DIGEST = OpenSSL::Digest.new("md5")
+    LOG_REDACTION_STRING = "[redacted]"
+
+    attr_reader :logging_enabled
+    attr_reader :credential
+    attr_reader :ip_address
+    attr_reader :recipients
+    attr_reader :headers
+    attr_reader :state
+    attr_reader :helo_name
+
+    def initialize(ip_address)
+      @logging_enabled = true
+      @ip_address = ip_address
+
+      @cr_present = false
+      @previous_cr_present = nil
+
+      if @ip_address
+        check_ip_address
+        @state = :welcome
+      else
+        @state = :preauth
+      end
+      transaction_reset
+    end
+
+    def check_ip_address
+      return unless @ip_address &&
+                    Postal::Config.smtp_server.log_ip_address_exclusion_matcher &&
+                    @ip_address =~ Regexp.new(Postal::Config.smtp_server.log_ip_address_exclusion_matcher)
+
+      @logging_enabled = false
+    end
+
+    def transaction_reset
+      @recipients = []
+      @mail_from = nil
+      @data = nil
+      @headers = nil
+    end
+
+    def trace_id
+      @trace_id ||= SecureRandom.alphanumeric(8).upcase
+    end
+
+    def handle(data)
+      if data[-1] == "\r"
+        @cr_present = true
+        data = data.chop # remove last character (\r)
+      else
+        # This doesn't use `logger` because that will be nil when logging is disabled
+        # and we always want to log this.
+        Postal.logger&.warn("Detected line with invalid line ending (missing <CR>)", trace_id: trace_id)
+        @cr_present = false
+      end
+
+      if @state == :preauth
+        return proxy(data)
+      end
+
+      logger&.debug "\e[32m<= #{sanitize_input_for_log(data.strip)}\e[0m"
+      if @proc
+        @proc.call(data)
+      else
+        handle_command(data)
+      end
+    ensure
+      @previous_cr_present = @cr_present
+    end
+
+    def finished?
+      @finished || false
+    end
+
+    def start_tls?
+      @start_tls || false
+    end
+
+    attr_writer :start_tls
+
+    def handle_command(data)
+      case data
+      when /^QUIT/i           then quit
+      when /^STARTTLS/i       then starttls
+      when /^EHLO/i           then ehlo(data)
+      when /^HELO/i           then helo(data)
+      when /^RSET/i           then rset
+      when /^NOOP/i           then noop
+      when /^AUTH PLAIN/i     then auth_plain(data)
+      when /^AUTH LOGIN/i     then auth_login(data)
+      when /^AUTH CRAM-MD5/i  then auth_cram_md5(data)
+      when /^MAIL FROM/i      then mail_from(data)
+      when /^RCPT TO/i        then rcpt_to(data)
+      when /^DATA/i           then data(data)
+      else
+        increment_error_count("invalid-command")
+        "502 Invalid/unsupported command"
+      end
+    end
+
+    def logger
+      return nil unless @logging_enabled
+
+      @logger ||= Postal.logger.create_tagged_logger(trace_id: trace_id)
+    end
+
+    private
+
+    def proxy(data)
+      # inet-protocol, client-ip, proxy-ip, client-port, proxy-port
+      if m = data.match(/\APROXY (.+) (.+) (.+) (.+) (.+)\z/)
+        @ip_address = m[2]
+        check_ip_address
+        @state = :welcome
+        logger&.debug "\e[35mClient identified as #{@ip_address}\e[0m"
+        increment_command_count("PROXY")
+        return "220 #{Postal::Config.postal.smtp_hostname} ESMTP Postal/#{trace_id}"
+      end
+
+      @finished = true
+      increment_error_count("proxy-error")
+      "502 Proxy Error"
+    end
+
+    def quit
+      @finished = true
+      "221 Closing Connection"
+    end
+
+    def starttls
+      if Postal::Config.smtp_server.tls_enabled?
+        @start_tls = true
+        @tls = true
+        increment_command_count("STARTLS")
+        "220 Ready to start TLS"
+      else
+        increment_error_count("tls-unavailable")
+        "502 TLS not available"
+      end
+    end
+
+    def ehlo(data)
+      @helo_name = data.strip.split(" ", 2)[1]
+      transaction_reset
+      @state = :welcomed
+      increment_command_count("EHLO")
+      [
+        "250-My capabilities are",
+        Postal::Config.smtp_server.tls_enabled? && !@tls ? "250-STARTTLS" : nil,
+        "250 AUTH CRAM-MD5 PLAIN LOGIN",
+      ].compact
+    end
+
+    def helo(data)
+      @helo_name = data.strip.split(" ", 2)[1]
+      transaction_reset
+      @state = :welcomed
+      increment_command_count("HELO")
+      "250 #{Postal::Config.postal.smtp_hostname}"
+    end
+
+    def rset
+      transaction_reset
+      @state = :welcomed
+      increment_command_count("RSET")
+      "250 OK"
+    end
+
+    def noop
+      "250 OK"
+    end
+
+    def auth_plain(data)
+      increment_command_count("AUTH PLAIN")
+
+      handler = proc do |idata|
+        @proc = nil
+        idata = Base64.decode64(idata)
+        parts = idata.split("\0")
+        username = parts[-2]
+        password = parts[-1]
+        unless username && password
+          increment_error_count("missing-credentials")
+          next "535 Authenticated failed - protocol error"
+        end
+
+        authenticate(password)
+      end
+
+      data = data.gsub(/AUTH PLAIN ?/i, "")
+      if data.strip == ""
+        @proc = handler
+        @password_expected_next = true
+        "334"
+      else
+        handler.call(data)
+      end
+    end
+
+    def auth_login(data)
+      increment_command_count("AUTH LOGIN")
+
+      password_handler = proc do |idata|
+        @proc = nil
+        password = Base64.decode64(idata)
+        authenticate(password)
+      end
+
+      username_handler = proc do
+        @proc = password_handler
+        @password_expected_next = true
+        "334 UGFzc3dvcmQ6" # "Password:"
+      end
+
+      data = data.gsub(/AUTH LOGIN ?/i, "")
+      if data.strip == ""
+        @proc = username_handler
+        "334 VXNlcm5hbWU6" # "Username:"
+      else
+        username_handler.call(nil)
+      end
+    end
+
+    def authenticate(password)
+      if @credential = Credential.where(type: "SMTP", key: password).first
+        @credential.use
+        "235 Granted for #{@credential.server.organization.permalink}/#{@credential.server.permalink}"
+      else
+        logger&.warn "Authentication failure for #{@ip_address}"
+        increment_error_count("invalid-credentials")
+        "535 Invalid credential"
+      end
+    end
+
+    def auth_cram_md5(data)
+      increment_command_count("AUTH CRAM-MD5")
+
+      challenge = Digest::SHA1.hexdigest(Time.now.to_i.to_s + rand(100_000).to_s)
+      challenge = "<#{challenge[0, 20]}@#{Postal::Config.postal.smtp_hostname}>"
+
+      handler = proc do |idata|
+        @proc = nil
+        username, password = Base64.decode64(idata).split(" ", 2).map { |a| a.chomp }
+        org_permlink, server_permalink = username.split(/[\/_]/, 2)
+        server = ::Server.includes(:organization).where(organizations: { permalink: org_permlink }, permalink: server_permalink).first
+        if server.nil?
+          logger&.warn "Authentication failure for #{@ip_address} (no server found matching #{username})"
+          increment_error_count("invalid-credentials")
+          next "535 Denied"
+        end
+
+        grant = nil
+        server.credentials.where(type: "SMTP").each do |credential|
+          correct_response = OpenSSL::HMAC.hexdigest(CRAM_MD5_DIGEST, credential.key, challenge)
+          next unless password == correct_response
+
+          @credential = credential
+          @credential.use
+          logger&.debug "Authenticated with with credential #{credential.id}"
+          grant = "235 Granted for #{credential.server.organization.permalink}/#{credential.server.permalink}"
+          break
+        end
+
+        if grant.nil?
+          logger&.warn "Authentication failure for #{@ip_address} (invalid credential)"
+          increment_error_count("invalid-credentials")
+          next "535 Denied"
+        end
+
+        grant
+      end
+
+      @proc = handler
+      "334 " + Base64.encode64(challenge).gsub(/[\r\n]/, "")
+    end
+
+    def mail_from(data)
+      unless in_state(:welcomed, :mail_from_received)
+        increment_error_count("mail-from-out-of-order")
+        return "503 EHLO/HELO first please"
+      end
+
+      @state = :mail_from_received
+      transaction_reset
+      if data =~ /AUTH=/
+        # Discard AUTH= parameter and anything that follows.
+        # We don't need this parameter as we don't trust any client to set it
+        mail_from_line = data.sub(/ *AUTH=.*/, "")
+      else
+        mail_from_line = data
+      end
+      @mail_from = mail_from_line.gsub(/MAIL FROM\s*:\s*/i, "").gsub(/.*</, "").gsub(/>.*/, "").strip
+      "250 OK"
+    end
+
+    def rcpt_to(data)
+      unless in_state(:mail_from_received, :rcpt_to_received)
+        increment_error_count("rcpt-to-out-of-order")
+        return "503 EHLO/HELO and MAIL FROM first please"
+      end
+
+      rcpt_to = data.gsub(/RCPT TO\s*:\s*/i, "").gsub(/.*</, "").gsub(/>.*/, "").strip
+
+      if rcpt_to.blank?
+        increment_error_count("empty-rcpt-to")
+        return "501 RCPT TO should not be empty"
+      end
+
+      uname, domain = rcpt_to.split("@", 2)
+
+      if domain.blank?
+        increment_error_count("invalid-rcpt-to")
+        return "501 Invalid RCPT TO"
+      end
+
+      uname, tag = uname.split("+", 2)
+
+      if domain == Postal::Config.dns.return_path_domain || domain =~ /\A#{Regexp.escape(Postal::Config.dns.custom_return_path_prefix)}\./
+        # This is a return path
+        @state = :rcpt_to_received
+        if server = ::Server.where(token: uname).first
+          if server.suspended?
+            increment_error_count("server-suspended")
+            "535 Mail server has been suspended"
+          else
+            logger&.debug "Added bounce on server #{server.id}"
+            @recipients << [:bounce, rcpt_to, server]
+            "250 OK"
+          end
+        else
+          increment_error_count("invalid-server-token")
+          "550 Invalid server token"
+        end
+
+      elsif domain == Postal::Config.dns.route_domain
+        # This is an email direct to a route. This isn't actually supported yet.
+        @state = :rcpt_to_received
+        if route = Route.where(token: uname).first
+          if route.server.suspended?
+            increment_error_count("server-suspended")
+            "535 Mail server has been suspended"
+          elsif route.mode == "Reject"
+            increment_error_count("route-rejected")
+            "550 Route does not accept incoming messages"
+          else
+            logger&.debug "Added route #{route.id} to recipients (tag: #{tag.inspect})"
+            actual_rcpt_to = "#{route.name}#{tag ? "+#{tag}" : ''}@#{route.domain.name}"
+            @recipients << [:route, actual_rcpt_to, route.server, { route: route }]
+            "250 OK"
+          end
+        else
+          "550 Invalid route token"
+        end
+
+      elsif @credential
+        # This is outgoing mail for an authenticated user
+        @state = :rcpt_to_received
+        if @credential.server.suspended?
+          increment_error_count("server-suspended")
+          "535 Mail server has been suspended"
+        else
+          logger&.debug "Added external address '#{rcpt_to}'"
+          @recipients << [:credential, rcpt_to, @credential.server]
+          "250 OK"
+        end
+
+      elsif uname && domain && route = Route.find_by_name_and_domain(uname, domain)
+        # This is incoming mail for a route
+        @state = :rcpt_to_received
+        if route.server.suspended?
+          increment_error_count("server-suspended")
+          "535 Mail server has been suspended"
+        elsif route.mode == "Reject"
+          increment_error_count("route-rejection")
+          "550 Route does not accept incoming messages"
+        else
+          logger&.debug "Added route #{route.id} to recipients (tag: #{tag.inspect})"
+          @recipients << [:route, rcpt_to, route.server, { route: route }]
+          "250 OK"
+        end
+
+      else
+        # User is trying to relay but is not authenticated. Try to authenticate by IP address
+        @credential = Credential.where(type: "SMTP-IP").all.sort_by { |c| c.ipaddr&.prefix || 0 }.reverse.find do |credential|
+          credential.ipaddr.include?(@ip_address) || (credential.ipaddr.ipv4? && credential.ipaddr.ipv4_mapped.include?(@ip_address))
+        end
+
+        if @credential
+          # Retry with credential
+          @credential.use
+          rcpt_to(data)
+        else
+          increment_error_count("authentication-required")
+          logger&.warn "Authentication failure for #{@ip_address}"
+          "530 Authentication required"
+        end
+      end
+    end
+
+    def data(_data)
+      unless in_state(:rcpt_to_received)
+        increment_error_count("data-out-of-order")
+        return "503 HELO/EHLO, MAIL FROM and RCPT TO before sending data"
+      end
+
+      @data = String.new.force_encoding("BINARY")
+      @headers = {}
+      @receiving_headers = true
+
+      received_header = ReceivedHeader.generate(@credential&.server, @helo_name, @ip_address, :smtp)
+                                      .force_encoding("BINARY")
+
+      @data << "Received: #{received_header}\r\n"
+      @headers["received"] = [received_header]
+
+      handler = proc do |idata|
+        if idata == "." && @cr_present && @previous_cr_present
+          @logging_enabled = true
+          @proc = nil
+          finished
+        else
+          idata = idata.to_s.sub(/\A\.\./, ".")
+
+          if @credential&.server&.log_smtp_data?
+            # We want to log if enabled
+          else
+            logger&.debug "Not logging further message data."
+            @logging_enabled = false
+          end
+
+          if @receiving_headers
+            if idata&.length&.zero?
+              @receiving_headers = false
+            elsif idata.to_s =~ /^\s/
+              # This is a continuation of a header
+              if @header_key && @headers[@header_key.downcase] && @headers[@header_key.downcase].last
+                @headers[@header_key.downcase].last << idata.to_s
+              end
+            else
+              @header_key, value = idata.split(/:\s*/, 2)
+              @headers[@header_key.downcase] ||= []
+              @headers[@header_key.downcase] << value
+            end
+          end
+          @data << idata
+          @data << "\r\n"
+          nil
+        end
+      end
+
+      @proc = handler
+      "354 Go ahead"
+    end
+
+    def finished
+      if @data.bytesize > Postal::Config.smtp_server.max_message_size.megabytes.to_i
+        transaction_reset
+        @state = :welcomed
+        increment_error_count("message-too-large")
+        return format("552 Message too large (maximum size %dMB)", Postal::Config.smtp_server.max_message_size)
+      end
+
+      if @headers["received"].grep(/by #{Postal::Config.postal.smtp_hostname}/).count > 4
+        transaction_reset
+        @state = :welcomed
+        increment_error_count("loop-detected")
+        return "550 Loop detected"
+      end
+
+      authenticated_domain = nil
+      if @credential
+        authenticated_domain = @credential.server.find_authenticated_domain_from_headers(@headers)
+        if authenticated_domain.nil?
+          transaction_reset
+          @state = :welcomed
+          increment_error_count("from-name-invalid")
+          return "530 From/Sender name is not valid"
+        end
+      end
+
+      @recipients.each do |recipient|
+        type, rcpt_to, server, options = recipient
+
+        case type
+        when :credential
+          increment_message_count("outgoing")
+
+          # Outgoing messages are just inserted
+          message = server.message_db.new_message
+          message.rcpt_to = rcpt_to
+          message.mail_from = @mail_from
+          message.raw_message = @data
+          message.received_with_ssl = @tls
+          message.scope = "outgoing"
+          message.domain_id = authenticated_domain&.id
+          message.credential_id = @credential.id
+          message.save
+
+        when :bounce
+          increment_message_count("bounce")
+          if rp_route = server.routes.where(name: "__returnpath__").first
+            # If there's a return path route, we can use this to create the message
+            rp_route.create_messages do |msg|
+              msg.rcpt_to = rcpt_to
+              msg.mail_from = @mail_from
+              msg.raw_message = @data
+              msg.received_with_ssl = @tls
+              msg.bounce = 1
+            end
+          else
+            # There's no return path route, we just need to insert the mesage
+            # without going through the route.
+            message = server.message_db.new_message
+            message.rcpt_to = rcpt_to
+            message.mail_from = @mail_from
+            message.raw_message = @data
+            message.received_with_ssl = @tls
+            message.scope = "incoming"
+            message.bounce = 1
+            message.save
+          end
+        when :route
+          increment_message_count("incoming")
+          options[:route].create_messages do |msg|
+            msg.rcpt_to = rcpt_to
+            msg.mail_from = @mail_from
+            msg.raw_message = @data
+            msg.received_with_ssl = @tls
+          end
+        end
+      end
+      transaction_reset
+      @state = :welcomed
+      "250 OK"
+    end
+
+    def in_state(*states)
+      states.include?(@state)
+    end
+
+    def sanitize_input_for_log(data)
+      if @password_expected_next
+        @password_expected_next = false
+        if data =~ /\A[a-z0-9]{3,}=*\z/i
+          return LOG_REDACTION_STRING
+        end
+      end
+
+      data = data.dup
+      data.gsub!(/(.*AUTH \w+) (.*)\z/i) { "#{::Regexp.last_match(1)} #{LOG_REDACTION_STRING}" }
+      data
+    end
+
+    def increment_error_count(error)
+      increment_prometheus_counter :postal_smtp_server_client_errors, labels: { error: error }
+    end
+
+    def increment_command_count(command)
+      increment_prometheus_counter :postal_smtp_server_commands_total, labels: { command: command }
+    end
+
+    def increment_message_count(type)
+      increment_prometheus_counter :postal_smtp_server_messages_total, labels: {
+        type: type,
+        tls: @tls ? "yes" : "no"
+      }
+    end
+
+    class << self
+
+      def register_prometheus_metrics
+        register_prometheus_counter :postal_smtp_server_commands_total,
+                                    docstring: "The number of key commands received by the server",
+                                    labels: [:command]
+
+        register_prometheus_counter :postal_smtp_server_client_errors,
+                                    docstring: "The number of errors sent to a client",
+                                    labels: [:error]
+
+        register_prometheus_counter :postal_smtp_server_messages_total,
+                                    docstring: "The number of messages accepted by the SMTP server",
+                                    labels: [:type, :tls]
+      end
+
+    end
+
+  end
+end

app/lib/smtp_server/server.rb

@@ -0,0 +1,317 @@
+# frozen_string_literal: true
+
+require "ipaddr"
+require "nio"
+
+module SMTPServer
+  class Server
+
+    include HasPrometheusMetrics
+
+    class << self
+
+      def tls_private_key
+        @tls_private_key ||= OpenSSL::PKey.read(File.read(Postal::Config.smtp_server.tls_private_key_path))
+      end
+
+      def tls_certificates
+        @tls_certificates ||= begin
+          data = File.read(Postal::Config.smtp_server.tls_certificate_path)
+          certs = data.scan(/-----BEGIN CERTIFICATE-----.+?-----END CERTIFICATE-----/m)
+          certs.map do |c|
+            OpenSSL::X509::Certificate.new(c)
+          end.freeze
+        end
+      end
+
+    end
+
+    def initialize(options = {})
+      @options = options
+      @options[:debug] ||= false
+      register_prometheus_metrics
+      prepare_environment
+    end
+
+    def run
+      logger.tagged(component: "smtp-server") do
+        listen
+        run_event_loop
+      end
+    end
+
+    private
+
+    def prepare_environment
+      $\ = "\r\n"
+      BasicSocket.do_not_reverse_lookup = true
+
+      trap("TERM") do
+        $stdout.puts "Received TERM signal, shutting down."
+        unlisten
+      end
+
+      trap("INT") do
+        $stdout.puts "Received INT signal, shutting down."
+        unlisten
+      end
+    end
+
+    def ssl_context
+      @ssl_context ||= begin
+        ssl_context      = OpenSSL::SSL::SSLContext.new
+        ssl_context.cert = self.class.tls_certificates[0]
+        ssl_context.extra_chain_cert = self.class.tls_certificates[1..]
+        ssl_context.key = self.class.tls_private_key
+        ssl_context.ssl_version = Postal::Config.smtp_server.ssl_version if Postal::Config.smtp_server.ssl_version
+        ssl_context.ciphers = Postal::Config.smtp_server.tls_ciphers if Postal::Config.smtp_server.tls_ciphers
+        ssl_context
+      end
+    end
+
+    def listen
+      bind_address = ENV.fetch("BIND_ADDRESS", Postal::Config.smtp_server.default_bind_address)
+      port = ENV.fetch("PORT", Postal::Config.smtp_server.default_port)
+
+      @server = TCPServer.open(bind_address, port)
+      @server.autoclose = false
+      @server.close_on_exec = false
+      if defined?(Socket::SOL_SOCKET) && defined?(Socket::SO_KEEPALIVE)
+        @server.setsockopt(Socket::SOL_SOCKET, Socket::SO_KEEPALIVE, true)
+      end
+      if defined?(Socket::SOL_TCP) && defined?(Socket::TCP_KEEPIDLE) && defined?(Socket::TCP_KEEPINTVL) && defined?(Socket::TCP_KEEPCNT)
+        @server.setsockopt(Socket::SOL_TCP, Socket::TCP_KEEPIDLE, 50)
+        @server.setsockopt(Socket::SOL_TCP, Socket::TCP_KEEPINTVL, 10)
+        @server.setsockopt(Socket::SOL_TCP, Socket::TCP_KEEPCNT, 5)
+      end
+
+      logger.info "Listening on #{bind_address}:#{port}"
+    end
+
+    def unlisten
+      # Instruct the nio loop to unlisten and wake it
+      @unlisten = true
+      @io_selector.wakeup
+    end
+
+    def run_event_loop
+      # Set up an instance of nio4r to monitor for connections and data
+      @io_selector = NIO::Selector.new
+      # Register the SMTP listener
+      @io_selector.register(@server, :r)
+      # Create a hash to contain a buffer for each client.
+      buffers = Hash.new { |h, k| h[k] = String.new.force_encoding("BINARY") }
+      loop do
+        # Wait for an event to occur
+        @io_selector.select do |monitor|
+          # Get the IO from the nio monitor
+          io = monitor.io
+          # Is this event an incoming connection?
+          if io.is_a?(TCPServer)
+            begin
+              # Accept the connection
+              new_io = io.accept
+              increment_prometheus_counter :postal_smtp_server_connections_total
+              # Get the client's IP address and strip `::ffff:` for consistency.
+              client_ip_address = new_io.remote_address.ip_address.sub(/\A::ffff:/, "")
+              if Postal::Config.smtp_server.proxy_protocol?
+                # If we are using the haproxy proxy protocol, we will be sent the
+                # client's IP later. Delay the welcome process.
+                client = Client.new(nil)
+                if Postal::Config.smtp_server.log_connections?
+                  client.logger&.debug "Connection opened from #{client_ip_address}"
+                end
+              else
+                # We're not using the proxy protocol so we already know the client's IP
+                client = Client.new(client_ip_address)
+                if Postal::Config.smtp_server.log_connections?
+                  client.logger&.debug "Connection opened from #{client_ip_address}"
+                end
+                # We know who the client is, welcome them.
+                client.logger&.debug "Client identified as #{client_ip_address}"
+                new_io.print("220 #{Postal::Config.postal.smtp_hostname} ESMTP Postal/#{client.trace_id}")
+              end
+              # Register the client and its socket with nio4r
+              monitor = @io_selector.register(new_io, :r)
+              monitor.value = client
+            rescue StandardError => e
+              # If something goes wrong, log as appropriate and disconnect the client
+              if defined?(Sentry)
+                Sentry.capture_exception(e, extra: { trace_id: begin
+                  client.trace_id
+                rescue StandardError
+                  nil
+                end })
+              end
+              logger.error "An error occurred while accepting a new client."
+              logger.error "#{e.class}: #{e.message}"
+              e.backtrace.each do |line|
+                logger.error line
+              end
+              increment_prometheus_counter :postal_smtp_server_exceptions_total,
+                                           labels: { error: e.class.to_s, type: "client-accept" }
+              begin
+                new_io.close
+              rescue StandardError
+                nil
+              end
+            end
+          else
+            # This event is not an incoming connection so it must be data from a client
+            begin
+              # Get the client from the nio monitor
+              client = monitor.value
+              # For now we assume the connection isn't closed
+              eof = false
+              # Is the client negotiating a TLS handshake?
+              if client.start_tls?
+                begin
+                  # Can we accept the TLS connection at this time?
+                  io.accept_nonblock
+                  # Increment prometheus
+                  increment_prometheus_counter :postal_smtp_server_tls_connections_total
+                  # We were able to accept the connection, the client is no longer handshaking
+                  client.start_tls = false
+                rescue IO::WaitReadable, IO::WaitWritable => e
+                  # Could not accept without blocking
+                  # We will try again later
+                  next
+                rescue OpenSSL::SSL::SSLError => e
+                  client.logger&.debug "SSL Negotiation Failed: #{e.message}"
+                  eof = true
+                end
+              else
+                # The client is not negotiating a TLS handshake at this time
+                begin
+                  # Read 10kiB of data at a time from the socket.
+                  buffers[io] << io.readpartial(10_240)
+
+                  # There is an extra step for SSL sockets
+                  if io.is_a?(OpenSSL::SSL::SSLSocket)
+                    buffers[io] << io.readpartial(10_240) while io.pending.positive?
+                  end
+                rescue EOFError, Errno::ECONNRESET, Errno::ETIMEDOUT
+                  # Client went away
+                  eof = true
+                end
+
+                # We line buffer, so look to see if we have received a newline
+                # and keep doing so until all buffered lines have been processed.
+                while buffers[io].index("\n")
+                  # Extract the line
+                  line, buffers[io] = buffers[io].split("\n", 2)
+                  # Send the received line to the client object for processing
+                  result = client.handle(line)
+                  # If the client object returned some data, write it back to the client
+                  next if result.nil?
+
+                  result = [result] unless result.is_a?(Array)
+                  result.compact.each do |iline|
+                    client.logger&.debug "\e[34m=> #{iline.strip}\e[0m"
+                    begin
+                      io.write(iline.to_s + "\r\n")
+                      io.flush
+                    rescue Errno::ECONNRESET
+                      # Client disconnected before we could write response
+                      eof = true
+                    end
+                  end
+                end
+
+                # Did the client request STARTTLS?
+                if !eof && client.start_tls?
+                  # Deregister the unencrypted IO
+                  @io_selector.deregister(io)
+                  buffers.delete(io)
+                  io = OpenSSL::SSL::SSLSocket.new(io, ssl_context)
+                  # Close the underlying IO when the TLS socket is closed
+                  io.sync_close = true
+                  # Register the new TLS socket with nio
+                  monitor = @io_selector.register(io, :r)
+                  monitor.value = client
+                end
+              end
+
+              # Has the client requested we close the connection?
+              if client.finished? || eof
+                client.logger&.debug "Connection closed"
+                # Deregister the socket and close it
+                @io_selector.deregister(io)
+                buffers.delete(io)
+                io.close
+                # If we have no more clients or listeners left, exit the process
+                if @io_selector.empty?
+                  Process.exit(0)
+                end
+              end
+            rescue StandardError => e
+              # Something went wrong, log as appropriate
+              client_id = client ? client.trace_id : "------"
+              if defined?(Sentry)
+                Sentry.capture_exception(e, extra: { trace_id: begin
+                  client&.trace_id
+                rescue StandardError
+                  nil
+                end })
+              end
+              logger.error "An error occurred while processing data from a client.", trace_id: client_id
+              logger.error "#{e.class}: #{e.message}", trace_id: client_id
+              e.backtrace.each do |iline|
+                logger.error iline, trace_id: client_id
+              end
+
+              increment_prometheus_counter :postal_smtp_server_exceptions_total,
+                                           labels: { error: e.class.to_s, type: "data" }
+
+              # Close all IO and forget this client
+              begin
+                @io_selector.deregister(io)
+              rescue StandardError
+                nil
+              end
+              buffers.delete(io)
+              begin
+                io.close
+              rescue StandardError
+                nil
+              end
+              if @io_selector.empty?
+                Process.exit(0)
+              end
+            end
+          end
+        end
+        # If unlisten has been called, stop listening
+        next unless @unlisten
+
+        @io_selector.deregister(@server)
+        @server.close
+        # If there's nothing left to do, shut down the process
+        if @io_selector.empty?
+          Process.exit(0)
+        end
+        # Clear the request
+        @unlisten = false
+      end
+    end
+
+    def logger
+      Postal.logger
+    end
+
+    def register_prometheus_metrics
+      register_prometheus_counter :postal_smtp_server_connections_total,
+                                  docstring: "The number of connections made to the Postal SMTP server."
+
+      register_prometheus_counter :postal_smtp_server_exceptions_total,
+                                  docstring: "The number of server exceptions encountered by the SMTP server",
+                                  labels: [:type, :error]
+
+      register_prometheus_counter :postal_smtp_server_tls_connections_total,
+                                  docstring: "The number of successfuly TLS connections established"
+
+      Client.register_prometheus_metrics
+    end
+
+  end
+end

app/lib/worker/jobs/base_job.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module Worker
+  module Jobs
+    class BaseJob
+
+      def initialize(logger:)
+        @logger = logger
+      end
+
+      def call
+        # Override me.
+      end
+
+      def work_completed?
+        @work_completed == true
+      end
+
+      private
+
+      def work_completed!
+        @work_completed = true
+      end
+
+      attr_reader :logger
+
+    end
+  end
+end

app/lib/worker/jobs/process_queued_messages_job.rb

@@ -0,0 +1,73 @@
+# frozen_string_literal: true
+
+module Worker
+  module Jobs
+    class ProcessQueuedMessagesJob < BaseJob
+
+      def call
+        @lock_time = Time.current
+        @locker = Postal.locker_name_with_suffix(SecureRandom.hex(8))
+
+        find_ip_addresses
+        lock_message_for_processing
+        obtain_locked_messages
+        process_messages
+        @messages_to_process
+      end
+
+      private
+
+      # Returns an array of IP address IDs that are present on the host that is
+      # running this job.
+      #
+      # @return [Array<Integer>]
+      def find_ip_addresses
+        ip_addresses = { 4 => [], 6 => [] }
+        Socket.ip_address_list.each do |address|
+          next if local_ip?(address.ip_address)
+
+          ip_addresses[address.ipv4? ? 4 : 6] << address.ip_address
+        end
+        @ip_addresses = IPAddress.where(ipv4: ip_addresses[4]).or(IPAddress.where(ipv6: ip_addresses[6])).pluck(:id)
+      end
+
+      # Is the given IP address a local address?
+      #
+      # @param [String] ip
+      # @return [Boolean]
+      def local_ip?(ip)
+        !!(ip =~ /\A(127\.|fe80:|::)/)
+      end
+
+      # Obtain a queued message from the database for processing
+      #
+      # @return [void]
+      def lock_message_for_processing
+        QueuedMessage.where(ip_address_id: [nil, @ip_addresses])
+                     .where(locked_by: nil, locked_at: nil)
+                     .ready_with_delayed_retry
+                     .limit(1)
+                     .update_all(locked_by: @locker, locked_at: @lock_time)
+      end
+
+      # Get a full list of all messages which we can process (i.e. those which have just
+      # been locked by us for processing)
+      #
+      # @return [void]
+      def obtain_locked_messages
+        @messages_to_process = QueuedMessage.where(locked_by: @locker, locked_at: @lock_time)
+      end
+
+      # Process the messages we obtained from the database
+      #
+      # @return [void]
+      def process_messages
+        @messages_to_process.each do |message|
+          work_completed!
+          MessageDequeuer.process(message, logger: logger)
+        end
+      end
+
+    end
+  end
+end

app/lib/worker/jobs/process_webhook_requests_job.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+module Worker
+  module Jobs
+    class ProcessWebhookRequestsJob < BaseJob
+
+      def call
+        @lock_time = Time.current
+        @locker = Postal.locker_name_with_suffix(SecureRandom.hex(8))
+
+        lock_request_for_processing
+        obtain_locked_requests
+        process_requests
+      end
+
+      private
+
+      # Obtain a webhook request from the database for processing
+      #
+      # @return [void]
+      def lock_request_for_processing
+        WebhookRequest.unlocked
+                      .ready
+                      .limit(1)
+                      .update_all(locked_by: @locker, locked_at: @lock_time)
+      end
+
+      # Get a full list of all webhooks which we can process (i.e. those which have just
+      # been locked by us for processing)
+      #
+      # @return [void]
+      def obtain_locked_requests
+        @requests_to_process = WebhookRequest.where(locked_by: @locker, locked_at: @lock_time)
+      end
+
+      # Process the webhook requests we obtained from the database
+      #
+      # @return [void]
+      def process_requests
+        @requests_to_process.each do |request|
+          work_completed!
+
+          WebhookDeliveryService.new(webhook_request: request).call
+        end
+      end
+
+    end
+  end
+end

app/lib/worker/process.rb

@@ -0,0 +1,317 @@
+# frozen_string_literal: true
+
+module Worker
+  # The Postal Worker process is responsible for handling all background tasks. This includes processing of all
+  # messages, webhooks and other administrative tasks. There are two main types of background work which is completed,
+  # jobs and scheduled tasks.
+  #
+  # The 'Jobs' here allow for the continuous monitoring of a database table (or queue) and processing of any new items
+  # which may appear in that. The polling takes place every 5 seconds by default and the work is able to run multiple
+  # threads to look for and process this work.
+  #
+  # Scheduled Tasks allow for code to be executed on a ROUGH schedule. This is used for administrative tasks.  A single
+  # thread will run within each worker process and attempt to acquire the 'tasks' role. If successful it will run all
+  # tasks which are due to be run. The tasks are then scheduled to run again at a future time. Workers which are not
+  # successful in acquiring the role will not run any tasks but will still attempt to acquire a lock in case the current
+  # acquiree disappears.
+  #
+  # The worker process will run until it receives a TERM or INT signal. It will then attempt to gracefully shut down
+  # after it has completed any outstanding jobs which are already inflight.
+  class Process
+
+    include HasPrometheusMetrics
+
+    # An array of job classes that should be processed each time the worker ticks.
+    #
+    # @return [Array<Class>]
+    JOBS = [
+      Jobs::ProcessQueuedMessagesJob,
+      Jobs::ProcessWebhookRequestsJob,
+    ].freeze
+
+    # An array of tasks that should be processed
+    #
+    # @return [Array<Class>]
+    TASKS = [
+      ActionDeletionsScheduledTask,
+      CheckAllDNSScheduledTask,
+      CleanupAuthieSessionsScheduledTask,
+      ExpireHeldMessagesScheduledTask,
+      ProcessMessageRetentionScheduledTask,
+      PruneSuppressionListsScheduledTask,
+      PruneWebhookRequestsScheduledTask,
+      SendNotificationsScheduledTask,
+      TidyQueuedMessagesTask,
+    ].freeze
+
+    # @param [Integer] thread_count The number of worker threads to run in this process
+    def initialize(thread_count: Postal::Config.worker.threads,
+                   work_sleep_time: 5,
+                   task_sleep_time: 60)
+      @thread_count = thread_count
+      @exit_pipe_read, @exit_pipe_write = IO.pipe
+      @work_sleep_time = work_sleep_time
+      @task_sleep_time = task_sleep_time
+      @threads = []
+
+      setup_prometheus
+    end
+
+    def run
+      logger.tagged(component: "worker") do
+        setup_traps
+        ensure_connection_pool_size_is_suitable
+        start_work_threads
+        start_tasks_thread
+        wait_for_threads
+      end
+    end
+
+    private
+
+    # Install signal traps to allow for graceful shutdown
+    #
+    # @return [void]
+    def setup_traps
+      trap("INT") { receive_signal("INT") }
+      trap("TERM") { receive_signal("TERM") }
+    end
+
+    # Receive a signal and set the shutdown flag
+    #
+    # @param [String] signal The signal that was received z
+    # @return [void]
+    def receive_signal(signal)
+      puts "Received #{signal} signal. Stopping when able."
+      @shutdown = true
+      @exit_pipe_write.close
+    end
+
+    # Wait for the period of time and return true or false if shutdown has been requested. If the shutdown is
+    # requested during the wait, it will return immediately otherwise it will return false when it has finished
+    # waiting for the period of time.
+    #
+    # @param [Integer] wait_time The time to wait for
+    # @return [Boolean]
+    def shutdown_after_wait?(wait_time)
+      @exit_pipe_read.wait_readable(wait_time) ? true : false
+    end
+
+    # Ensure that the connection pool is big enough for the number of threads
+    # configured.
+    #
+    # @return [void]
+    def ensure_connection_pool_size_is_suitable
+      current_pool_size = ActiveRecord::Base.connection_pool.size
+      desired_pool_size = @thread_count + 3
+
+      return if current_pool_size >= desired_pool_size
+
+      logger.warn "number of worker threads (#{@thread_count}) is more  " \
+                  "than the db connection pool size (#{current_pool_size}+3), " \
+                  "increasing connection pool size to #{desired_pool_size}"
+
+      Postal.change_database_connection_pool_size(desired_pool_size)
+    end
+
+    # Wait for all threads to complete
+    #
+    # @return [void]
+    def wait_for_threads
+      @threads.each(&:join)
+    end
+
+    # Start the worker threads
+    #
+    # @return [void]
+    def start_work_threads
+      logger.info "starting #{@thread_count} work threads"
+      @thread_count.times do |index|
+        start_work_thread(index)
+      end
+    end
+
+    # Start a worker thread
+    #
+    # @return [void]
+    def start_work_thread(index)
+      @threads << Thread.new do
+        logger.tagged(component: "worker", thread: "work#{index}") do
+          logger.info "started work thread #{index}"
+          loop do
+            work_completed = work(index)
+
+            if shutdown_after_wait?(work_completed ? 0 : @work_sleep_time)
+              break
+            end
+          end
+
+          logger.info "stopping work thread #{index}"
+        end
+      end
+    end
+
+    # Actually perform the work for this tick. This will call each job which has been registered.
+    #
+    # @return [Boolean] Whether any work was completed in this job or not
+    def work(thread)
+      completed_work = 0
+      ActiveRecord::Base.connection_pool.with_connection do
+        JOBS.each do |job_class|
+          capture_errors do
+            job = job_class.new(logger: logger)
+
+            time = Benchmark.realtime { job.call }
+
+            observe_prometheus_histogram :postal_worker_job_runtime,
+                                         time,
+                                         labels: {
+                                          thread: thread,
+                                          job: job_class.to_s.split("::").last
+                                         }
+
+            if job.work_completed?
+              completed_work += 1
+              increment_prometheus_counter :postal_worker_job_executions,
+                                           labels: {
+                                              thread: thread,
+                                              job: job_class.to_s.split("::").last
+                                           }
+            end
+          end
+        end
+      end
+      completed_work.positive?
+    end
+
+    # Start the tasks thread
+    #
+    # @return [void]
+    def start_tasks_thread
+      logger.info "starting tasks thread"
+      @threads << Thread.new do
+        logger.tagged(component: "worker", thread: "tasks") do
+          loop do
+            run_tasks
+
+            if shutdown_after_wait?(@task_sleep_time)
+              break
+            end
+          end
+
+          logger.info "stopping tasks thread"
+          ActiveRecord::Base.connection_pool.with_connection do
+            if WorkerRole.release(:tasks)
+              logger.info "released tasks role"
+            end
+          end
+        end
+      end
+    end
+
+    # Run the tasks. This will attempt to acquire the tasks role and if successful it will all the registered
+    # tasks if they are due to be run.
+    #
+    # @return [void]
+    def run_tasks
+      role_acquisition_status = ActiveRecord::Base.connection_pool.with_connection do
+        WorkerRole.acquire(:tasks)
+      end
+
+      case role_acquisition_status
+      when :stolen
+        logger.info "acquired task role by stealing it from a lazy worker"
+      when :created
+        logger.info "acquired task role by creating it"
+      when :renewed
+        logger.debug "acquired task role by renewing it"
+      else
+        logger.debug "could not acquire task role, not doing anything"
+        return false
+      end
+
+      ActiveRecord::Base.connection_pool.with_connection do
+        TASKS.each { |task| run_task(task) }
+      end
+    end
+
+    # Run a single task
+    #
+    # @param [Class] task The task to run
+    # @return [void]
+    def run_task(task)
+      logger.tagged task: task do
+        scheduled_task = ScheduledTask.find_by(name: task.to_s)
+        if scheduled_task.nil?
+          logger.info "no existing task object, creating it now"
+          scheduled_task = ScheduledTask.create!(name: task.to_s, next_run_after: task.next_run_after)
+        end
+
+        next unless scheduled_task.next_run_after < Time.current
+
+        logger.info "running task"
+
+        time = 0
+        capture_errors do
+          time = Benchmark.realtime do
+            task.new(logger: logger).call
+          end
+
+          observe_prometheus_histogram :postal_worker_task_runtime,
+                                       time,
+                                       labels: {
+                                        task: task.to_s.split("::").last
+                                       }
+        end
+
+        next_run_after = task.next_run_after
+        logger.info "scheduling task to next run at #{next_run_after}"
+        scheduled_task.update!(next_run_after: next_run_after)
+      end
+    end
+
+    # Return the logger
+    #
+    # @return [Klogger::Logger]
+    def logger
+      Postal.logger
+    end
+
+    # Capture exceptions and handle this as appropriate.
+    #
+    # @yield The block of code to run
+    # @return [void]
+    def capture_errors
+      yield
+    rescue StandardError => e
+      logger.error "#{e.class} (#{e.message})"
+      e.backtrace.each { |line| logger.error line }
+      Sentry.capture_exception(e) if defined?(Sentry)
+
+      increment_prometheus_counter :postal_worker_errors,
+                                   labels: { error: e.class.to_s }
+    end
+
+    def setup_prometheus
+      register_prometheus_counter :postal_worker_job_executions,
+                                  docstring: "The number of jobs worked by a worker where work was completed",
+                                  labels: [:thread, :job]
+
+      register_prometheus_histogram :postal_worker_job_runtime,
+                                    docstring: "The time taken to process jobs (in seconds)",
+                                    labels: [:thread, :job]
+
+      register_prometheus_counter :postal_worker_errors,
+                                  docstring: "The number of errors encountered while processing jobs",
+                                  labels: [:error]
+
+      register_prometheus_histogram :postal_worker_task_runtime,
+                                    docstring: "The time taken to process tasks (in seconds)",
+                                    labels: [:task]
+
+      register_prometheus_histogram :postal_message_queue_latency,
+                                    docstring: "The length of time between a message being queued and being dequeued (in seconds)"
+    end
+
+  end
+end

app/mailers/app_mailer.rb

@@ -2,22 +2,6 @@
 
 class AppMailer < ApplicationMailer
 
-  def verify_email_address(user)
-    @user = user
-    mail to: @user.email_address, subject: "Verify your new e-mail address"
-  end
-
-  def new_user(user)
-    @user = user
-    mail to: @user.email_address, subject: "Welcome to Postal"
-  end
-
-  def user_invite(user_invite, organization)
-    @user_invite = user_invite
-    @organization = organization
-    mail to: @user_invite.email_address, subject: "Access the #{organization.name} organization on Postal"
-  end
-
   def verify_domain(domain, email_address, user)
     @domain = domain
     @email_address = email_address

app/mailers/application_mailer.rb

@@ -2,7 +2,7 @@
 
 class ApplicationMailer < ActionMailer::Base
 
-  default from: "#{Postal.smtp_from_name} <#{Postal.smtp_from_address}>"
+  default from: "#{Postal::Config.smtp.from_name} <#{Postal::Config.smtp.from_address}>"
   layout false
 
 end

app/models/bounce_message.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+class BounceMessage
+
+  def initialize(server, message)
+    @server = server
+    @message = message
+  end
+
+  def raw_message
+    mail = Mail.new
+    mail.to = @message.mail_from
+    mail.from = "Mail Delivery Service <#{@message.route.description}>"
+    mail.subject = "Mail Delivery Failed (#{@message.subject})"
+    mail.text_part = body
+    mail.attachments["Original Message.eml"] = { mime_type: "message/rfc822", encoding: "quoted-printable", content: @message.raw_message }
+    mail.message_id = "<#{SecureRandom.uuid}@#{Postal::Config.dns.return_path_domain}>"
+    mail.to_s
+  end
+
+  def queue
+    message = @server.message_db.new_message
+    message.scope = "outgoing"
+    message.rcpt_to = @message.mail_from
+    message.mail_from = @message.route.description
+    message.domain_id = @message.domain&.id
+    message.raw_message = raw_message
+    message.bounce = true
+    message.bounce_for_id = @message.id
+    message.save
+    message.id
+  end
+
+  def postmaster_address
+    @server.postmaster_address || "postmaster@#{@message.domain&.name || Postal::Config.postal.web_hostname}"
+  end
+
+  private
+
+  def body
+    <<~BODY
+      This is the mail delivery service responsible for delivering mail to #{@message.route.description}.
+
+      The message you've sent cannot be delivered. Your original message is attached to this message.
+
+      For further assistance please contact #{postmaster_address}. Please include the details below to help us identify the issue.
+
+      Message Token: #{@message.token}@#{@server.token}
+      Orginal Message ID: #{@message.message_id}
+      Mail from: #{@message.mail_from}
+      Rcpt To: #{@message.rcpt_to}
+    BODY
+  end
+
+end

app/models/concerns/has_authentication.rb

@@ -5,21 +5,20 @@ module HasAuthentication
   extend ActiveSupport::Concern
 
   included do
-    has_secure_password
+    has_secure_password validations: false
 
     validates :password, length: { minimum: 8, allow_blank: true }
+    validates :password, confirmation: { allow_blank: true }
+    validate :validate_password_presence
 
-    when_attribute :password_digest, changes_to: :anything do
+    before_save :clear_password_reset_token_on_password_change
-      before_save do
+
-        self.password_reset_token = nil
+    scope :with_password, -> { where.not(password_digest: nil) }
-        self.password_reset_token_valid_until = nil
-      end
-    end
   end
 
   class_methods do
     def authenticate(email_address, password)
-      user = where(email_address: email_address).first
+      user = find_by(email_address: email_address)
       raise Postal::Errors::AuthenticationError, "InvalidEmailAddress" if user.nil?
       raise Postal::Errors::AuthenticationError, "InvalidPassword" unless user.authenticate(password)
 
@@ -36,12 +35,31 @@ module HasAuthentication
   end
 
   def begin_password_reset(return_to = nil)
-    self.password_reset_token = Nifty::Utils::RandomString.generate(length: 24)
+    if Postal::Config.oidc.enabled? && (oidc_uid.present? || password_digest.blank?)
+      raise Postal::Error, "User has OIDC enabled, password resets are not supported"
+    end
+
+    self.password_reset_token = SecureRandom.alphanumeric(24)
     self.password_reset_token_valid_until = 1.day.from_now
     save!
     AppMailer.password_reset(self, return_to).deliver
   end
 
+  private
+
+  def clear_password_reset_token_on_password_change
+    return unless password_digest_changed?
+
+    self.password_reset_token = nil
+    self.password_reset_token_valid_until = nil
+  end
+
+  def validate_password_presence
+    return if password_digest.present? || Postal::Config.oidc.enabled?
+
+    errors.add :password, :blank
+  end
+
 end
 
 # -*- SkipSchemaAnnotations

app/models/concerns/has_dns_checks.rb

@@ -43,16 +43,16 @@ module HasDNSChecks
   #
 
   def check_spf_record
-    result = resolver.getresources(name, Resolv::DNS::Resource::IN::TXT)
+    result = resolver.txt(name)
-    spf_records = result.map(&:data).grep(/\Av=spf1/)
+    spf_records = result.grep(/\Av=spf1/)
     if spf_records.empty?
       self.spf_status = "Missing"
       self.spf_error = "No SPF record exists for this domain"
     else
-      suitable_spf_records = spf_records.grep(/include:\s*#{Regexp.escape(Postal.config.dns.spf_include)}/)
+      suitable_spf_records = spf_records.grep(/include:\s*#{Regexp.escape(Postal::Config.dns.spf_include)}/)
       if suitable_spf_records.empty?
         self.spf_status = "Invalid"
-        self.spf_error = "An SPF record exists but it doesn't include #{Postal.config.dns.spf_include}"
+        self.spf_error = "An SPF record exists but it doesn't include #{Postal::Config.dns.spf_include}"
         false
       else
         self.spf_status = "OK"
@@ -73,8 +73,7 @@ module HasDNSChecks
 
   def check_dkim_record
     domain = "#{dkim_record_name}.#{name}"
-    result = resolver.getresources(domain, Resolv::DNS::Resource::IN::TXT)
+    records = resolver.txt(domain)
-    records = result.map(&:data)
     if records.empty?
       self.dkim_status = "Missing"
       self.dkim_error = "No TXT records were returned for #{domain}"
@@ -104,17 +103,16 @@ module HasDNSChecks
   #
 
   def check_mx_records
-    result = resolver.getresources(name, Resolv::DNS::Resource::IN::MX)
+    records = resolver.mx(name).map(&:last)
-    records = result.map(&:exchange)
     if records.empty?
       self.mx_status = "Missing"
       self.mx_error = "There are no MX records for #{name}"
     else
-      missing_records = Postal.config.dns.mx_records.dup - records.map { |r| r.to_s.downcase }
+      missing_records = Postal::Config.dns.mx_records.dup - records.map { |r| r.to_s.downcase }
       if missing_records.empty?
         self.mx_status = "OK"
         self.mx_error = nil
-      elsif missing_records.size == Postal.config.dns.mx_records.size
+      elsif missing_records.size == Postal::Config.dns.mx_records.size
         self.mx_status = "Missing"
         self.mx_error = "You have MX records but none of them point to us."
       else
@@ -134,17 +132,16 @@ module HasDNSChecks
   #
 
   def check_return_path_record
-    result = resolver.getresources(return_path_domain, Resolv::DNS::Resource::IN::CNAME)
+    records = resolver.cname(return_path_domain)
-    records = result.map { |r| r.name.to_s.downcase }
     if records.empty?
       self.return_path_status = "Missing"
       self.return_path_error = "There is no return path record at #{return_path_domain}"
-    elsif records.size == 1 && records.first == Postal.config.dns.return_path
+    elsif records.size == 1 && records.first == Postal::Config.dns.return_path_domain
       self.return_path_status = "OK"
       self.return_path_error = nil
     else
       self.return_path_status = "Invalid"
-      self.return_path_error = "There is a CNAME record at #{return_path_domain} but it points to #{records.first} which is incorrect. It should point to #{Postal.config.dns.return_path}."
+      self.return_path_error = "There is a CNAME record at #{return_path_domain} but it points to #{records.first} which is incorrect. It should point to #{Postal::Config.dns.return_path_domain}."
     end
   end
 

app/models/concerns/has_locking.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+# This concern provides functionality for locking items along with additional functionality to handle
+# the concept of retrying items after a certain period of time. The following database columns are
+# required on the model
+#
+# * locked_by - A string column to store the name of the process that has locked the item
+# * locked_at - A datetime column to store the time the item was locked
+# * retry_after - A datetime column to store the time after which the item should be retried
+# * attempts - An integer column to store the number of attempts that have been made to process the item
+#
+# 'ready' means that it's ready to be processed.
+module HasLocking
+
+  extend ActiveSupport::Concern
+
+  included do
+    scope :unlocked, -> { where(locked_at: nil) }
+    scope :ready, -> { where("retry_after IS NULL OR retry_after < ?", Time.now) }
+  end
+
+  def ready?
+    retry_after.nil? || retry_after < Time.now
+  end
+
+  def unlock
+    self.locked_by = nil
+    self.locked_at = nil
+    update_columns(locked_by: nil, locked_at: nil)
+  end
+
+  def locked?
+    locked_at.present?
+  end
+
+  def retry_later(time = nil)
+    retry_time = time || calculate_retry_time(attempts, 5.minutes)
+    self.locked_by = nil
+    self.locked_at = nil
+    update_columns(locked_by: nil, locked_at: nil, retry_after: Time.now + retry_time, attempts: attempts + 1)
+  end
+
+  def calculate_retry_time(attempts, initial_period)
+    (1.3**attempts) * initial_period
+  end
+
+end

app/models/concerns/has_message.rb

@@ -7,7 +7,11 @@ module HasMessage
   end
 
   def message
-    @message ||= server.message_db.message(message_id)
+    return @message if instance_variable_defined?("@message")
+
+    @message = server.message_db.message(message_id)
+  rescue Postal::MessageDB::Message::NotFound
+    @message = nil
   end
 
   def message=(message)

app/models/concerns/has_soft_destroy.rb

@@ -14,7 +14,6 @@ module HasSoftDestroy
     run_callbacks :soft_destroy do
       self.deleted_at = Time.now
       save!
-      ActionDeletionJob.queue(:main, type: self.class.name, id: id)
     end
   end
 

app/models/credential.rb

@@ -31,7 +31,7 @@ class Credential < ApplicationRecord
   validate :validate_key_cannot_be_changed
   validate :validate_key_for_smtp_ip
 
-  serialize :options, Hash
+  serialize :options, type: Hash, coder: YAML
 
   before_validation :generate_key
 
@@ -39,7 +39,7 @@ class Credential < ApplicationRecord
     return if type == "SMTP-IP"
     return if persisted?
 
-    self.key = SecureRandomString.new(24)
+    self.key = SecureRandom.alphanumeric(24)
   end
 
   def to_param

app/models/domain.rb

@@ -61,23 +61,15 @@ class Domain < ApplicationRecord
 
   scope :verified, -> { where.not(verified_at: nil) }
 
-  when_attribute :verification_method, changes_to: :anything do
+  before_save :update_verification_token_on_method_change
-    before_save do
-      if verification_method == "DNS"
-        self.verification_token = Nifty::Utils::RandomString.generate(length: 32)
-      elsif verification_method == "Email"
-        self.verification_token = rand(999_999).to_s.ljust(6, "0")
-      else
-        self.verification_token = nil
-      end
-    end
-  end
 
   def verified?
     verified_at.present?
   end
 
-  def verify
+  def mark_as_verified
+    return false if verified?
+
     self.verified_at = Time.now
     save!
   end
@@ -94,6 +86,8 @@ class Domain < ApplicationRecord
   end
 
   def dkim_key
+    return nil unless dkim_private_key
+
     @dkim_key ||= OpenSSL::PKey::RSA.new(dkim_private_key)
   end
 
@@ -110,67 +104,72 @@ class Domain < ApplicationRecord
   end
 
   def spf_record
-    "v=spf1 a mx include:#{Postal.config.dns.spf_include} ~all"
+    "v=spf1 a mx include:#{Postal::Config.dns.spf_include} ~all"
   end
 
   def dkim_record
+    return if dkim_key.nil?
+
     public_key = dkim_key.public_key.to_s.gsub(/-+[A-Z ]+-+\n/, "").gsub(/\n/, "")
     "v=DKIM1; t=s; h=sha256; p=#{public_key};"
   end
 
   def dkim_identifier
-    Postal.config.dns.dkim_identifier + "-#{dkim_identifier_string}"
+    return nil unless dkim_identifier_string
+
+    Postal::Config.dns.dkim_identifier + "-#{dkim_identifier_string}"
   end
 
   def dkim_record_name
-    "#{dkim_identifier}._domainkey"
+    identifier = dkim_identifier
+    return if identifier.nil?
+
+    "#{identifier}._domainkey"
   end
 
   def return_path_domain
-    "#{Postal.config.dns.custom_return_path_prefix}.#{name}"
+    "#{Postal::Config.dns.custom_return_path_prefix}.#{name}"
-  end
-
-  def nameservers
-    @nameservers ||= get_nameservers
   end
 
+  # Returns a DNSResolver instance that can be used to perform DNS lookups needed for
+  # the verification and DNS checking for this domain.
+  #
+  # @return [DNSResolver]
   def resolver
-    @resolver ||= Postal.config.general.use_local_ns_for_domains? ? Resolv::DNS.new : Resolv::DNS.new(nameserver: nameservers)
+    return DNSResolver.local if Postal::Config.postal.use_local_ns_for_domain_verification?
+
+    @resolver ||= DNSResolver.for_domain(name)
   end
 
   def dns_verification_string
-    "#{Postal.config.dns.domain_verify_prefix} #{verification_token}"
+    "#{Postal::Config.dns.domain_verify_prefix} #{verification_token}"
   end
 
   def verify_with_dns
     return false unless verification_method == "DNS"
 
-    result = resolver.getresources(name, Resolv::DNS::Resource::IN::TXT)
+    result = resolver.txt(name)
-    if result.map { |d| d.data.to_s.strip }.include?(dns_verification_string)
+
+    if result.include?(dns_verification_string)
       self.verified_at = Time.now
-      save
+      return save
-    else
-      false
     end
+
+    false
   end
 
   private
 
-  def get_nameservers
+  def update_verification_token_on_method_change
-    local_resolver = Resolv::DNS.new
+    return unless verification_method_changed?
-    ns_records = []
+
-    parts = name.split(".")
+    if verification_method == "DNS"
-    (parts.size - 1).times do |n|
+      self.verification_token = SecureRandom.alphanumeric(32)
-      d = parts[n, parts.size - n + 1].join(".")
+    elsif verification_method == "Email"
-      ns_records = local_resolver.getresources(d, Resolv::DNS::Resource::IN::NS).map { |s| s.name.to_s }
+      self.verification_token = rand(999_999).to_s.ljust(6, "0")
-      break if ns_records.present?
+    else
+      self.verification_token = nil
     end
-    return [] if ns_records.blank?
-
-    ns_records = ns_records.map { |r| local_resolver.getresources(r, Resolv::DNS::Resource::IN::A).map { |s| s.address.to_s } }.flatten
-    return [] if ns_records.blank?
-
-    ns_records
   end
 
 end

app/models/http_endpoint.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "uri"
+
 # == Schema Information
 #
 # Table name: http_endpoints
@@ -38,6 +40,7 @@ class HTTPEndpoint < ApplicationRecord
 
   validates :name, presence: true
   validates :url, presence: true
+  validate :url_must_be_http_or_https
   validates :encoding, inclusion: { in: ENCODINGS }
   validates :format, inclusion: { in: FORMATS }
   validates :timeout, numericality: { greater_than_or_equal_to: 5, less_than_or_equal_to: 60 }
@@ -56,4 +59,17 @@ class HTTPEndpoint < ApplicationRecord
     routes.each { |r| r.update(endpoint: nil, mode: "Reject") }
   end
 
+  private
+
+  def url_must_be_http_or_https
+    return if url.blank?
+
+    uri = URI.parse(url)
+    return if uri.is_a?(URI::HTTP) && uri.host.present?
+
+    errors.add(:url, "must be an HTTP or HTTPS URL")
+  rescue URI::InvalidURIError
+    errors.add(:url, "must be a valid HTTP or HTTPS URL")
+  end
+
 end

app/models/incoming_message_prototype.rb

@@ -88,14 +88,14 @@ class IncomingMessagePrototype
       mail.from = @from
       mail.subject = @subject
       mail.text_part = @plain_body
-      mail.message_id = "<#{SecureRandom.uuid}@#{Postal.config.dns.return_path}>"
+      mail.message_id = "<#{SecureRandom.uuid}@#{Postal::Config.dns.return_path_domain}>"
       attachments.each do |attachment|
         mail.attachments[attachment[:name]] = {
           mime_type: attachment[:content_type],
           content: attachment[:data]
         }
       end
-      mail.header["Received"] = Postal::ReceivedHeader.generate(@server, @source_type, @ip, :http)
+      mail.header["Received"] = ReceivedHeader.generate(@server, @source_type, @ip, :http)
       mail.to_s
     end
   end

app/models/outgoing_message_prototype.rb

@@ -25,7 +25,7 @@ class OutgoingMessagePrototype
     @source_type = source_type
     @custom_headers = {}
     @attachments = []
-    @message_id = "#{SecureRandom.uuid}@#{Postal.config.dns.return_path}"
+    @message_id = "#{SecureRandom.uuid}@#{Postal::Config.dns.return_path_domain}"
     attributes.each do |key, value|
       instance_variable_set("@#{key}", value)
     end
@@ -99,7 +99,7 @@ class OutgoingMessagePrototype
       {
         name: attachment[:name],
         content_type: attachment[:content_type] || "application/octet-stream",
-        data: attachment[:base64] ? Base64.decode64(attachment[:data]) : attachment[:data]
+        data: attachment[:base64] && attachment[:data] ? Base64.decode64(attachment[:data]) : attachment[:data]
       }
     end
   end
@@ -177,7 +177,7 @@ class OutgoingMessagePrototype
           content: attachment[:data]
         }
       end
-      mail.header["Received"] = Postal::ReceivedHeader.generate(@server, @source_type, @ip, :http)
+      mail.header["Received"] = ReceivedHeader.generate(@server, @source_type, @ip, :http)
       mail.message_id = "<#{@message_id}>"
       mail.to_s
     end

app/models/queued_message.rb

@@ -29,81 +29,36 @@
 class QueuedMessage < ApplicationRecord
 
   include HasMessage
+  include HasLocking
 
   belongs_to :server
   belongs_to :ip_address, optional: true
-  belongs_to :user, optional: true
 
   before_create :allocate_ip_address
-  after_commit :queue, on: :create
 
-  scope :unlocked, -> { where(locked_at: nil) }
+  scope :ready_with_delayed_retry, -> { where("retry_after IS NULL OR retry_after < ?", 30.seconds.ago) }
-  scope :retriable, -> { where("retry_after IS NULL OR retry_after < ?", Time.now) }
+  scope :with_stale_lock, -> { where("locked_at IS NOT NULL AND locked_at < ?", Postal::Config.postal.queued_message_lock_stale_days.days.ago) }
-  scope :requeueable, -> { where("retry_after IS NULL OR retry_after < ?", 30.seconds.ago) }
 
-  def retriable?
+  def retry_now
-    retry_after.nil? || retry_after < Time.now
+    update!(retry_after: nil)
-  end
-
-  def queue
-    UnqueueMessageJob.queue(queue_name, id: id)
-  end
-
-  def queue!
-    update_column(:retry_after, nil)
-    queue
-  end
-
-  def queue_name
-    ip_address ? :"outgoing-#{ip_address.id}" : :main
   end
 
   def send_bounce
     return unless message.send_bounces?
 
-    Postal::BounceMessage.new(server, message).queue
+    BounceMessage.new(server, message).queue
   end
 
   def allocate_ip_address
-    return unless Postal.ip_pools? && message && pool = server.ip_pool_for_message(message)
+    return unless Postal.ip_pools?
+    return if message.nil?
+
+    pool = server.ip_pool_for_message(message)
+    return if pool.nil?
 
     self.ip_address = pool.ip_addresses.select_by_priority
   end
 
-  def acquire_lock
-    time = Time.now
-    locker = Postal.locker_name
-    rows = self.class.where(id: id, locked_by: nil, locked_at: nil).update_all(locked_by: locker, locked_at: time)
-    if rows == 1
-      self.locked_by = locker
-      self.locked_at = time
-      true
-    else
-      false
-    end
-  end
-
-  def retry_later(time = nil)
-    retry_time = time || self.class.calculate_retry_time(attempts, 5.minutes)
-    self.locked_by = nil
-    self.locked_at = nil
-    update_columns(locked_by: nil, locked_at: nil, retry_after: Time.now + retry_time, attempts: attempts + 1)
-  end
-
-  def unlock
-    self.locked_by = nil
-    self.locked_at = nil
-    update_columns(locked_by: nil, locked_at: nil)
-  end
-
-  def self.calculate_retry_time(attempts, initial_period)
-    (1.3**attempts) * initial_period
-  end
-
-  def locked?
-    locked_at.present?
-  end
-
   def batchable_messages(limit = 10)
     unless locked?
       raise Postal::Error, "Must lock current message before locking any friends"
@@ -114,13 +69,9 @@ class QueuedMessage < ApplicationRecord
     else
       time = Time.now
       locker = Postal.locker_name
-      self.class.retriable.where(batch_key: batch_key, ip_address_id: ip_address_id, locked_by: nil, locked_at: nil).limit(limit).update_all(locked_by: locker, locked_at: time)
+      self.class.ready.where(batch_key: batch_key, ip_address_id: ip_address_id, locked_by: nil, locked_at: nil).limit(limit).update_all(locked_by: locker, locked_at: time)
       QueuedMessage.where(batch_key: batch_key, ip_address_id: ip_address_id, locked_by: locker, locked_at: time).where.not(id: id)
     end
   end
 
-  def self.requeue_all
-    unlocked.requeueable.each(&:queue)
-  end
-
 end

app/models/route.rb

@@ -89,7 +89,7 @@ class Route < ApplicationRecord
   end
 
   def forward_address
-    @forward_address ||= "#{token}@#{Postal.config.dns.route_domain}"
+    @forward_address ||= "#{token}@#{Postal::Config.dns.route_domain}"
   end
 
   def wildcard?
@@ -128,7 +128,7 @@ class Route < ApplicationRecord
 
   #
   # This message will create a suitable number of message objects for messages that
-  #  are destined for this route. It receives a block which can set the message content
+  # are destined for this route. It receives a block which can set the message content
   # but most information is specified already.
   #
   # Returns an array of created messages.

app/models/scheduled_task.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+# == Schema Information
+#
+# Table name: scheduled_tasks
+#
+#  id             :bigint           not null, primary key
+#  name           :string(255)
+#  next_run_after :datetime
+#
+# Indexes
+#
+#  index_scheduled_tasks_on_name  (name) UNIQUE
+#
+class ScheduledTask < ApplicationRecord
+end

app/models/server.rb

@@ -71,8 +71,8 @@ class Server < ApplicationRecord
   default_value :raw_message_retention_days, -> { 30 }
   default_value :raw_message_retention_size, -> { 2048 }
   default_value :message_retention_days, -> { 60 }
-  default_value :spam_threshold, -> { Postal.config.general.default_spam_threshold }
+  default_value :spam_threshold, -> { Postal::Config.postal.default_spam_threshold }
-  default_value :spam_failure_threshold, -> { Postal.config.general.default_spam_failure_threshold }
+  default_value :spam_failure_threshold, -> { Postal::Config.postal.default_spam_failure_threshold }
 
   validates :name, presence: true, uniqueness: { scope: :organization_id, case_sensitive: false }
   validates :mode, inclusion: { in: MODES }
@@ -192,37 +192,33 @@ class Server < ApplicationRecord
   end
 
   def send_limit_approaching?
-    send_limit && (send_volume >= send_limit * 0.90)
+    return false unless send_limit
+
+    (send_volume >= send_limit * 0.90)
   end
 
   def send_limit_exceeded?
-    send_limit && send_volume >= send_limit
+    return false unless send_limit
+
+    send_volume >= send_limit
   end
 
   def send_limit_warning(type)
-    AppMailer.send("server_send_limit_#{type}", self).deliver
+    if organization.notification_addresses.present?
+      AppMailer.send("server_send_limit_#{type}", self).deliver
+    end
+
     update_column("send_limit_#{type}_notified_at", Time.now)
     WebhookRequest.trigger(self, "SendLimit#{type.to_s.capitalize}", server: webhook_hash, volume: send_volume, limit: send_limit)
   end
 
   def queue_size
-    @queue_size ||= queued_messages.retriable.count
+    @queue_size ||= queued_messages.ready.count
-  end
-
-  def stats
-    {
-      queue: queue_size,
-      held: held_messages,
-      bounce_rate: bounce_rate,
-      message_rate: message_rate,
-      throughput: throughput_stats,
-      size: message_db.total_size
-    }
   end
 
   # Return the domain which can be used to authenticate emails sent from the given e-mail address.
   #
-  #  @param address [String] an e-mail address
+  # @param address [String] an e-mail address
   # @return [Domain, nil] the domain to use for authentication
   def authenticated_domain_for_address(address)
     return nil if address.blank?
@@ -274,7 +270,10 @@ class Server < ApplicationRecord
     self.suspended_at = Time.now
     self.suspension_reason = reason
     save!
-    AppMailer.server_suspended(self).deliver
+    if organization.notification_addresses.present?
+      AppMailer.server_suspended(self).deliver
+    end
+    true
   end
 
   def unsuspend
@@ -283,12 +282,6 @@ class Server < ApplicationRecord
     save!
   end
 
-  def validate_ip_pool_belongs_to_organization
-    return unless ip_pool && ip_pool_id_changed? && !organization.ip_pools.include?(ip_pool)
-
-    errors.add :ip_pool_id, "must belong to the organization"
-  end
-
   def ip_pool_for_message(message)
     return unless message.scope == "outgoing"
 
@@ -300,46 +293,48 @@ class Server < ApplicationRecord
         end
       end
     end
+
     ip_pool
   end
 
-  def self.triggered_send_limit(type)
+  private
-    servers = where("send_limit_#{type}_at IS NOT NULL AND send_limit_#{type}_at > ?", 3.minutes.ago)
+
-    servers.where("send_limit_#{type}_notified_at IS NULL OR send_limit_#{type}_notified_at < ?", 1.hour.ago)
+  def validate_ip_pool_belongs_to_organization
+    return unless ip_pool && ip_pool_id_changed? && !organization.ip_pools.include?(ip_pool)
+
+    errors.add :ip_pool_id, "must belong to the organization"
   end
 
-  def self.send_send_limit_notifications
+  class << self
-    [:approaching, :exceeded].each_with_object({}) do |type, hash|
-      hash[type] = 0
-      servers = triggered_send_limit(type)
-      next if servers.empty?
 
-      servers.each do |server|
+    def triggered_send_limit(type)
-        hash[type] += 1
+      servers = where("send_limit_#{type}_at IS NOT NULL AND send_limit_#{type}_at > ?", 3.minutes.ago)
-        server.send_limit_warning(type)
+      servers.where("send_limit_#{type}_notified_at IS NULL OR send_limit_#{type}_notified_at < ?", 1.hour.ago)
-      end
-    end
-  end
-
-  def self.[](id, extra = nil)
-    server = nil
-    if id.is_a?(String)
-      if id =~ /\A(\w+)\/(\w+)\z/
-        server = includes(:organization).where(organizations: { permalink: ::Regexp.last_match(1) }, permalink: ::Regexp.last_match(2)).first
-      end
-    else
-      server = where(id: id).first
     end
 
-    if extra
+    def send_send_limit_notifications
-      if extra.is_a?(String)
+      [:approaching, :exceeded].each_with_object({}) do |type, hash|
-        server.domains.where(name: extra.to_s).first
+        hash[type] = 0
+        servers = triggered_send_limit(type)
+        next if servers.empty?
+
+        servers.each do |server|
+          hash[type] += 1
+          server.send_limit_warning(type)
+        end
+      end
+    end
+
+    def [](id, extra = nil)
+      if id.is_a?(String) && id =~ /\A(\w+)\/(\w+)\z/
+        joins(:organization).where(
+          organizations: { permalink: ::Regexp.last_match(1) }, permalink: ::Regexp.last_match(2)
+        ).first
       else
-        server.message(extra.to_i)
+        find_by(id: id.to_i)
       end
-    else
-      server
     end
+
   end
 
 end

app/models/smtp_endpoint.rb

@@ -47,4 +47,8 @@ class SMTPEndpoint < ApplicationRecord
     routes.each { |r| r.update(endpoint: nil, mode: "Reject") }
   end
 
+  def to_smtp_client_server
+    SMTPClient::Server.new(hostname, port: port || 25, ssl_mode: ssl_mode)
+  end
+
 end

app/models/track_domain.rb

@@ -54,17 +54,16 @@ class TrackDomain < ApplicationRecord
   end
 
   def check_dns
-    result = domain.resolver.getresources(full_name, Resolv::DNS::Resource::IN::CNAME)
+    records = domain.resolver.cname(full_name)
-    records = result.map { |r| r.name.to_s.downcase }
     if records.empty?
       self.dns_status = "Missing"
       self.dns_error = "There is no record at #{full_name}"
-    elsif records.size == 1 && records.first == Postal.config.dns.track_domain
+    elsif records.size == 1 && records.first == Postal::Config.dns.track_domain
       self.dns_status = "OK"
       self.dns_error = nil
     else
       self.dns_status = "Invalid"
-      self.dns_error = "There is a CNAME record at #{full_name} but it points to #{records.first} which is incorrect. It should point to #{Postal.config.dns.track_domain}."
+      self.dns_error = "There is a CNAME record at #{full_name} but it points to #{records.first} which is incorrect. It should point to #{Postal::Config.dns.track_domain}."
     end
     self.dns_checked_at = Time.now
     save!

app/models/user.rb

@@ -5,19 +5,21 @@
 # Table name: users
 #
 #  id                               :integer          not null, primary key
-#  uuid                             :string(255)
+#  admin                            :boolean          default(FALSE)
-#  first_name                       :string(255)
-#  last_name                        :string(255)
 #  email_address                    :string(255)
-#  password_digest                  :string(255)
-#  time_zone                        :string(255)
 #  email_verification_token         :string(255)
 #  email_verified_at                :datetime
-#  created_at                       :datetime
+#  first_name                       :string(255)
-#  updated_at                       :datetime
+#  last_name                        :string(255)
+#  oidc_issuer                      :string(255)
+#  oidc_uid                         :string(255)
+#  password_digest                  :string(255)
 #  password_reset_token             :string(255)
 #  password_reset_token_valid_until :datetime
-#  admin                            :boolean          default(FALSE)
+#  time_zone                        :string(255)
+#  uuid                             :string(255)
+#  created_at                       :datetime
+#  updated_at                       :datetime
 #
 # Indexes
 #
@@ -28,13 +30,11 @@
 class User < ApplicationRecord
 
   include HasUUID
-
   include HasAuthentication
 
   validates :first_name, presence: true
   validates :last_name, presence: true
   validates :email_address, presence: true, uniqueness: { case_sensitive: false }, format: { with: /@/, allow_blank: true }
-  validates :time_zone, presence: true
 
   default_value :time_zone, -> { "UTC" }
 
@@ -53,24 +53,85 @@ class User < ApplicationRecord
     "#{first_name} #{last_name}"
   end
 
+  def password?
+    password_digest.present?
+  end
+
+  def oidc?
+    oidc_uid.present?
+  end
+
   def to_param
     uuid
   end
 
-  def md5_for_gravatar
-    @md5_for_gravatar ||= Digest::MD5.hexdigest(email_address.to_s.downcase)
-  end
-
-  def avatar_url
-    @avatar_url ||= email_address ? "https://secure.gravatar.com/avatar/#{md5_for_gravatar}?rating=PG&size=120&d=mm" : nil
-  end
-
   def email_tag
     "#{name} <#{email_address}>"
   end
 
-  def self.[](email)
+  class << self
-    where(email_address: email).first
+
+    # Lookup a user by email address
+    #
+    # @param email [String] the email address
+    #
+    # @return [User, nil] the user
+    def [](email)
+      find_by(email_address: email)
+    end
+
+    # Find a user based on an OIDC authentication hash
+    #
+    # @param auth [Hash] the authentication hash
+    # @param logger [Logger] a logger to log debug information to
+    #
+    # @return [User, nil] the user
+    def find_from_oidc(auth, logger: nil)
+      config = Postal::Config.oidc
+
+      uid = auth[config.uid_field]
+      oidc_name = auth[config.name_field]
+      oidc_email_address = auth[config.email_address_field]
+
+      logger&.debug "got auth details from issuer: #{auth.inspect}"
+
+      # look for an existing user with the same UID and OIDC issuer. If we find one,
+      # this is the user we'll want to use.
+      user = where(oidc_uid: uid, oidc_issuer: config.issuer).first
+
+      if user
+        logger&.debug "found user with UID #{uid} for issuer #{config.issuer} (user ID: #{user.id})"
+      else
+        logger&.debug "no user with UID #{uid} for issuer #{config.issuer}"
+      end
+
+      # if we don't have an existing user, we will look for users which have no OIDC
+      # credentials but with a matching e-mail address.
+      if user.nil? && oidc_email_address.present?
+        user = where(oidc_uid: nil, email_address: oidc_email_address).first
+        if user
+          logger&.debug "found user with e-mail address #{oidc_email_address} (user ID: #{user.id})"
+        else
+          logger&.debug "no user with e-mail address #{oidc_email_address}"
+        end
+      end
+
+      # now, if we still don't have a user, we're not going to create one so we'll just
+      # return nil (we might auto create users in the future but not right now)
+      return if user.nil?
+
+      # otherwise, let's update our user as appropriate
+      user.oidc_uid = uid
+      user.oidc_issuer = config.issuer
+      user.email_address = oidc_email_address if oidc_email_address.present?
+      user.first_name, user.last_name = oidc_name.split(/\s+/, 2) if oidc_name.present?
+      user.password = nil
+      user.save!
+
+      # return the user
+      user
+    end
+
   end
 
 end

app/models/webhook.rb

@@ -35,12 +35,7 @@ class Webhook < ApplicationRecord
   scope :enabled, -> { where(enabled: true) }
 
   after_save :save_events
-
+  after_save :destroy_events_when_all_events_enabled
-  when_attribute :all_events, changes_to: true do
-    after_save do
-      webhook_events.destroy_all
-    end
-  end
 
   def events
     @events ||= webhook_events.map(&:event)
@@ -50,13 +45,22 @@ class Webhook < ApplicationRecord
     @events = value.map(&:to_s).select(&:present?)
   end
 
+  private
+
   def save_events
     return unless @events
 
     @events.each do |event|
       webhook_events.where(event: event).first_or_create!
     end
+
     webhook_events.where.not(event: @events).destroy_all
   end
 
+  def destroy_events_when_all_events_enabled
+    return unless all_events
+
+    webhook_events.destroy_all
+  end
+
 end

app/models/webhook_request.rb

@@ -5,23 +5,28 @@
 # Table name: webhook_requests
 #
 #  id          :integer          not null, primary key
+#  attempts    :integer          default(0)
+#  error       :text(65535)
+#  event       :string(255)
+#  locked_at   :datetime
+#  locked_by   :string(255)
+#  payload     :text(65535)
+#  retry_after :datetime
+#  url         :string(255)
+#  uuid        :string(255)
+#  created_at  :datetime
 #  server_id   :integer
 #  webhook_id  :integer
-#  url         :string(255)
+#
-#  event       :string(255)
+# Indexes
-#  uuid        :string(255)
+#
-#  payload     :text(65535)
+#  index_webhook_requests_on_locked_by  (locked_by)
-#  attempts    :integer          default(0)
-#  retry_after :datetime
-#  error       :text(65535)
-#  created_at  :datetime
 #
 
 class WebhookRequest < ApplicationRecord
 
   include HasUUID
-
+  include HasLocking
-  RETRIES = { 1 => 2.minutes, 2 => 3.minutes, 3 => 6.minutes, 4 => 10.minutes, 5 => 15.minutes }.freeze
 
   belongs_to :server
   belongs_to :webhook, optional: true
@@ -29,66 +34,21 @@ class WebhookRequest < ApplicationRecord
   validates :url, presence: true
   validates :event, presence: true
 
-  serialize :payload, Hash
+  serialize :payload, type: Hash, coder: YAML
 
-  after_commit :queue, on: :create
+  class << self
 
-  def self.trigger(server, event, payload = {})
+    def trigger(server, event, payload = {})
-    unless server.is_a?(Server)
+      unless server.is_a?(Server)
-      server = Server.find(server.to_i)
+        server = Server.find(server.to_i)
-    end
+      end
-
+
-    webhooks = server.webhooks.enabled.includes(:webhook_events).references(:webhook_events).where("webhooks.all_events = ? OR webhook_events.event = ?", true, event)
+      webhooks = server.webhooks.enabled.includes(:webhook_events).references(:webhook_events).where("webhooks.all_events = ? OR webhook_events.event = ?", true, event)
-    webhooks.each do |webhook|
+      webhooks.each do |webhook|
-      server.webhook_requests.create!(event: event, payload: payload, webhook: webhook, url: webhook.url)
+        server.webhook_requests.create!(event: event, payload: payload, webhook: webhook, url: webhook.url)
-    end
-  end
-
-  def self.requeue_all
-    where("retry_after < ?", Time.now).find_each(&:queue)
-  end
-
-  def queue
-    WebhookDeliveryJob.queue(:main, id: id)
-  end
-
-  def deliver
-    logger = Postal.logger_for(:webhooks)
-    payload = { event: event, timestamp: created_at.to_f, payload: self.payload, uuid: uuid }.to_json
-    logger.info "[#{id}] Sending webhook request to `#{url}`"
-    result = Postal::HTTP.post(url, sign: true, json: payload, timeout: 5)
-    self.attempts += 1
-    self.retry_after = RETRIES[self.attempts]&.from_now
-    server.message_db.webhooks.record(
-      event: event,
-      url: url,
-      webhook_id: webhook_id,
-      attempt: self.attempts,
-      timestamp: Time.now.to_f,
-      payload: self.payload.to_json,
-      uuid: uuid,
-      status_code: result[:code],
-      body: result[:body],
-      will_retry: (retry_after ? 0 : 1)
-    )
-
-    if result[:code] >= 200 && result[:code] < 300
-      logger.info "[#{id}]   -> Received #{result[:code]} status code. That's OK."
-      destroy
-      webhook&.update_column(:last_used_at, Time.now)
-      true
-    else
-      logger.error "[#{id}]   -> Received #{result[:code]} status code. That's not OK."
-      self.error = "Couldn't send to URL. Code received was #{result[:code]}"
-      if retry_after
-        logger.info "[#{id}]   -> Will retry #{retry_after} (this was attempt #{self.attempts})"
-        save
-      else
-        logger.info "[#{id}]   -> Have tried #{self.attempts} times. Giving up."
-        destroy
       end
-      false
     end
+
   end
 
 end

app/models/worker_role.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+# == Schema Information
+#
+# Table name: worker_roles
+#
+#  id          :bigint           not null, primary key
+#  acquired_at :datetime
+#  role        :string(255)
+#  worker      :string(255)
+#
+# Indexes
+#
+#  index_worker_roles_on_role  (role) UNIQUE
+#
+class WorkerRole < ApplicationRecord
+
+  class << self
+
+    # Acquire or renew a lock for the given role.
+    #
+    # @param role [String] The name of the role to acquire
+    # @return [Symbol, false] True if the lock was acquired or renewed, false otherwise
+    def acquire(role)
+      # update our existing lock if we already have one
+      updates = where(role: role, worker: Postal.locker_name).update_all(acquired_at: Time.current)
+      return :renewed if updates.positive?
+
+      # attempt to steal a role from another worker
+      updates = where(role: role).where("acquired_at is null OR acquired_at < ?", 5.minutes.ago)
+                                 .update_all(acquired_at: Time.current, worker: Postal.locker_name)
+      return :stolen if updates.positive?
+
+      # attempt to create a new role for this worker
+      begin
+        create!(role: role, worker: Postal.locker_name, acquired_at: Time.current)
+        :created
+      rescue ActiveRecord::RecordNotUnique, ActiveRecord::RecordInvalid
+        false
+      end
+    end
+
+    # Release a lock for the given role for the current process.
+    #
+    # @param role [String] The name of the role to release
+    # @return [Boolean] True if the lock was released, false otherwise
+    def release(role)
+      updates = where(role: role, worker: Postal.locker_name).delete_all
+      updates.positive?
+    end
+
+  end
+
+end

app/scheduled_tasks/action_deletions_scheduled_task.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+class ActionDeletionsScheduledTask < ApplicationScheduledTask
+
+  def call
+    Organization.deleted.each do |org|
+      logger.info "permanently removing organization #{org.id} (#{org.permalink})"
+      org.destroy
+    end
+
+    Server.deleted.each do |server|
+      logger.info "permanently removing server #{server.id} (#{server.full_permalink})"
+      server.destroy
+    end
+  end
+
+end

app/scheduled_tasks/application_scheduled_task.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+class ApplicationScheduledTask
+
+  def initialize(logger:)
+    @logger = logger
+  end
+
+  def call
+    raise NotImplementedError
+  end
+
+  attr_reader :logger
+
+  class << self
+
+    def next_run_after
+      quarter_past_each_hour
+    end
+
+    private
+
+    def quarter_past_each_hour
+      time = Time.current
+      time = time.change(min: 15, sec: 0)
+      time += 1.hour if time < Time.current
+      time
+    end
+
+    def quarter_to_each_hour
+      time = Time.current
+      time = time.change(min: 45, sec: 0)
+      time += 1.hour if time < Time.current
+      time
+    end
+
+    def three_am
+      time = Time.current
+      time = time.change(hour: 3, min: 0, sec: 0)
+      time += 1.day if time < Time.current
+      time
+    end
+
+  end
+
+end

app/scheduled_tasks/check_all_dns_scheduled_task.rb

@@ -1,15 +1,15 @@
 # frozen_string_literal: true
 
-class CheckAllDNSJob < Postal::Job
+class CheckAllDNSScheduledTask < ApplicationScheduledTask
 
-  def perform
+  def call
     Domain.where.not(dns_checked_at: nil).where("dns_checked_at <= ?", 1.hour.ago).each do |domain|
-      log "Checking DNS for domain: #{domain.name}"
+      logger.info "checking DNS for domain: #{domain.name}"
       domain.check_dns(:auto)
     end
 
     TrackDomain.where("dns_checked_at IS NULL OR dns_checked_at <= ?", 1.hour.ago).includes(:domain).each do |domain|
-      log "Checking DNS for track domain: #{domain.full_name}"
+      logger.info "checking DNS for track domain: #{domain.full_name}"
       domain.check_dns
     end
   end

app/scheduled_tasks/cleanup_authie_sessions_scheduled_task.rb

@@ -2,9 +2,9 @@
 
 require "authie/session"
 
-class CleanupAuthieSessionsJob < Postal::Job
+class CleanupAuthieSessionsScheduledTask < ApplicationScheduledTask
 
-  def perform
+  def call
     Authie::Session.cleanup
   end
 

app/scheduled_tasks/expire_held_messages_scheduled_task.rb

@@ -1,8 +1,8 @@
 # frozen_string_literal: true
 
-class ExpireHeldMessagesJob < Postal::Job
+class ExpireHeldMessagesScheduledTask < ApplicationScheduledTask
 
-  def perform
+  def call
     Server.all.each do |server|
       messages = server.message_db.messages(where: {
         status: "Held",

app/scheduled_tasks/process_message_retention_scheduled_task.rb

@@ -1,25 +1,29 @@
 # frozen_string_literal: true
 
-class ProcessMessageRetentionJob < Postal::Job
+class ProcessMessageRetentionScheduledTask < ApplicationScheduledTask
 
-  def perform
+  def call
     Server.all.each do |server|
       if server.raw_message_retention_days
         # If the server has a maximum number of retained raw messages, remove any that are older than this
-        log "Tidying raw messages (by days) for #{server.permalink} (ID: #{server.id}). Keeping #{server.raw_message_retention_days} days."
+        logger.info "Tidying raw messages (by days) for #{server.permalink} (ID: #{server.id}). Keeping #{server.raw_message_retention_days} days."
         server.message_db.provisioner.remove_raw_tables_older_than(server.raw_message_retention_days)
       end
 
       if server.raw_message_retention_size
-        log "Tidying raw messages (by size) for #{server.permalink} (ID: #{server.id}). Keeping #{server.raw_message_retention_size} MB of data."
+        logger.info "Tidying raw messages (by size) for #{server.permalink} (ID: #{server.id}). Keeping #{server.raw_message_retention_size} MB of data."
         server.message_db.provisioner.remove_raw_tables_until_less_than_size(server.raw_message_retention_size * 1024 * 1024)
       end
 
       if server.message_retention_days
-        log "Tidying messages for #{server.permalink} (ID: #{server.id}). Keeping #{server.message_retention_days} days."
+        logger.info "Tidying messages for #{server.permalink} (ID: #{server.id}). Keeping #{server.message_retention_days} days."
         server.message_db.provisioner.remove_messages(server.message_retention_days)
       end
     end
   end
 
+  def self.next_run_after
+    three_am
+  end
+
 end

app/scheduled_tasks/prune_suppression_lists_scheduled_task.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+class PruneSuppressionListsScheduledTask < ApplicationScheduledTask
+
+  def call
+    Server.all.each do |s|
+      logger.info "Pruning suppression lists for server #{s.id}"
+      s.message_db.suppression_list.prune
+    end
+  end
+
+  def self.next_run_after
+    three_am
+  end
+
+end

app/scheduled_tasks/prune_webhook_requests_scheduled_task.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+class PruneWebhookRequestsScheduledTask < ApplicationScheduledTask
+
+  def call
+    Server.all.each do |s|
+      logger.info "Pruning webhook requests for server #{s.id}"
+      s.message_db.webhooks.prune
+    end
+  end
+
+  def self.next_run_after
+    quarter_to_each_hour
+  end
+
+end