Reacher
  • Welcome to Reacher
  • Getting Started
    • Verify your 1st email
    • Understanding "is_reachable"
  • Self-Hosting
    • SaaS vs Self-Host
    • Install Reacher in 20min
    • Scaling for Production
      • Manage scaling yourself
      • Option 1: RabbitMQ-based Queue Architecture
    • Licensing
      • Commercial License Trial
    • Proxies
      • Multiple Proxies
    • Reacher Configuration
    • Debugging Reacher
  • Advanced
    • OpenAPI
      • /v0/check_email
      • /v1/check_email
      • /v1/bulk
    • Run your own Proxy
    • Migrations
      • Reacher Configuration (v0.10)
      • Migrating from 0.7 to 0.10
      • Bulk Verification (v0.7)
      • Docker Environment Variables (v0.7)
Powered by GitBook
On this page
  • Worker Configuration
  • Understanding the architecture with Docker Compose
  • More questions?
  1. Self-Hosting
  2. Scaling for Production

Option 1: RabbitMQ-based Queue Architecture

PreviousManage scaling yourselfNextLicensing

Last updated 13 days ago

Reacher includes an optional, opinionated queue-based architecture designed to scale efficiently and handle high email verification volumes. This architecture comprises 4 main components and is highly configurable to meet specific business needs.

Component
Description
Docker image

HTTP server

Accepts incoming email verification requests, and post them into the queue.

reacherhq/commercial-license-trial

Reacher uses a reliable, mature and open-source queue implementation.

rabbitmq:4.0-management

Workers

One or more consumers of the queue, which perform the actual email verification task.

reacherhq/commercial-license-trial

Storage

A place to store all results, currently only PostgresDB is supported.

postgres:14

Note that Reacher provides the same Docker image reacherhq/commercial-license-trial which can act as both a Worker and a HTTP server.

With this architecture, it's possible to horizontally scale the number of workers.

Worker Configuration

Enabling the Architecture

To enable the worker-based architecture, configure the following parameters in your deployment. The parameters are given in their backend_config.toml format (e.g. worker.enable) as well as in the environment variable format (e.g. RCH__WORKER__ENABLE):

  • worker.enable or RCH__WORKER__ENABLE: Set to true to activate the worker role.

  • worker.rabbitmq.url or RCH__WORKER__RABBITMQ__URL: URL of the RabbitMQ instance for task queuing.

  • postgres.storage.postgres.db_url or RCH__STORAGE__POSTGRES__DB_URL: URL of a PostgreSQL database to store verification results.

Using Proxies for Workers

Since spawning workers on cloud providers doesn't guarantee a reputable IP assigned to the worker, we configure all workers to use a proxy.

  • proxy.{host,port} or RCH__PROXY__{HOST,PORT}: Set a proxy to route all SMTP requests through. You can optionally pass in username and password if required.

The Dockerfile provided in the Commercial License Trial already has these parameters set up.

Optional Concurrency and Throttling Parameters

You may also configure the following parameters:

  • Concurrency:

    • worker.rabbitmq.concurrency: Set to 5. Each worker processes up to 5 emails concurrently.

  • Throttling:

    • throttle.max_requests_per_minute: Set to 60. Limits request spikes to prevent SMTP server flags.

    • throttle.max_requests_per_day: Set to 10,000.

The Dockerfile provided in the Commercial License Trial already has these parameters set up.

Understanding the architecture with Docker Compose

More questions?

  • deploying on Kubernetes (Ansible playbook, Pulumi)

  • more specialized workers (e.g. some workers doing headless verification only, others doing SMTP only)

We do not recommend using Docker Compose for a high-volume production setup. However, for understanding or learning the architecture, this file can be useful.

Contact if you have more questions about this architecture, such as:

docker_compose.yaml
RabbitMQ
amaury@reacher.email
Reacher queue architecture