Option 2: RabbitMQ-based Queue Architecture

Reacher includes an optional, opinionated queue-based architecture designed to scale efficiently and handle high email verification volumes. This architecture comprises 4 main components and is highly configurable to meet specific business needs.

Component	Description	Docker image
HTTP server	Accepts incoming email verification requests, and post them into the queue.	reacherhq/commercial-license-trial
RabbitMQ	Reacher uses a reliable, mature and open-source queue implementation.	rabbitmq:4.0-management
Workers	One or more consumers of the queue, which perform the actual email verification task.	reacherhq/commercial-license-trial
Storage	A place to store all results, currently only PostgresDB is supported.	postgres:14

Note that Reacher provides the same Docker image reacherhq/commercial-license-trial which can act as both a Worker and a HTTP server.

Reacher queue architecture

With this architecture, it's possible to horizontally scale the number of workers. However, to prevent spawning too many workers at once resulting in blacklisted IPs, we need to configure some concurrency and throttling parameters below.

Worker Configuration

Enabling the Architecture

To enable the worker-based architecture, configure the following parameters in your deployment:

• worker.enable: Set to true to activate the worker role.

• worker.rabbitmq.url: URL of the RabbitMQ instance for task queuing.

• postgres.db_url: URL of a PostgreSQL database to store verification results.

Using Proxies for Workers

Since spawning workers on cloud providers doesn't guarantee a reputable IP assigned to the worker, we configure all workers to use a proxy.

• proxy.{host,port}: Set a proxy to route all SMTP requests through. You can optionally pass in username and password if required.

The Dockerfile provided in the Commercial License Trial already has these parameters set up.

Proxy Recommendation:

• Use one proxy IP per 10,000 email verifications per day.

• Purchase additional proxy IPs as needed to scale.

• Spawn as many workers as IPs you purchased.

Concurrency and Throttling Parameters

To prevent IP blacklisting, configure the following parameters:

• Concurrency:

  • worker.rabbitmq.concurrency: Set to 5. Each worker processes up to 5 emails concurrently, which means each proxy IP address is perform maximum 5 email verifications at any give time.

• Throttling:

  • throttle.max_requests_per_minute: Set to 60. Limits request spikes to prevent SMTP server flags.

  • throttle.max_requests_per_day: Set to 10,000. Aligns with the recommended verifications per proxy IP.

The Dockerfile provided in the Commercial License Trial already has these parameters set up.

Scaling Example

For scaling, buy additional proxy IPs and spawn more workers accordingly. The "rule of thumb" is 10,000 verifications per IP per day.

For example, if your target is 10 million verifications per month:

1. Daily volume = 10,000,000 emails / 30 days = 333,000 emails/day.

2. Number of required IPs = 333,000 emails/day / 10,000 emails/IP = 33 or 34 IPs.

We suggest deploying one Reacher instance per purchased IP. The settings outlined above ensure fair email verification usage as perceived by external email providers. For advanced applications, these values can be adjusted to optimize performance.

Understanding the architecture with Docker Compose

We do not recommend using Docker Compose for a high-volume production setup. However, for understanding or learning the architecture, this docker_compose.yaml file can be useful.

More questions?

Contact [e-mail] you have more questions about this architecture, such as:

• deploying on Kubernetes (Ansible playbook, Pulumi)

• more specialized workers (e.g. some workers doing headless verification only, others doing SMTP only)