Option 2: RabbitMQ-based Queue Architecture
Last updated
Last updated
Reacher includes an optional, opinionated queue-based architecture designed to scale efficiently and handle high email verification volumes. This architecture comprises 4 main components and is highly configurable to meet specific business needs.
HTTP server
Accepts incoming email verification requests, and post them into the queue.
reacherhq/commercial-license-trial
Reacher uses a reliable, mature and open-source queue implementation.
rabbitmq:4.0-management
Workers
One or more consumers of the queue, which perform the actual email verification task.
reacherhq/commercial-license-trial
Storage
A place to store all results, currently only PostgresDB is supported.
postgres:14
Note that Reacher provides the same Docker image reacherhq/commercial-license-trial which can act as both a Worker and a HTTP server.
With this architecture, it's possible to horizontally scale the number of workers. However, to prevent spawning too many workers at once resulting in blacklisted IPs, we need to configure some concurrency and throttling parameters below.
To enable the worker-based architecture, configure the following parameters in your deployment:
worker.enable: Set to true to activate the worker role.
worker.rabbitmq.url: URL of the RabbitMQ instance for task queuing.
postgres.db_url: URL of a PostgreSQL database to store verification results.
Since spawning workers on cloud providers doesn't guarantee a reputable IP assigned to the worker, we configure all workers to use a proxy.
proxy.{host,port}: Set a proxy to route all SMTP requests through. You can optionally pass in username and password if required.
The Dockerfile provided in the Commercial License Trial already has these parameters set up.
Proxy Recommendation:
Use one proxy IP per 10,000 email verifications per day.
Purchase additional proxy IPs as needed to scale.
Spawn as many workers as IPs you purchased.
To prevent IP blacklisting, configure the following parameters:
Concurrency:
worker.rabbitmq.concurrency: Set to 5. Each worker processes up to 5 emails concurrently, which means each proxy IP address is perform maximum 5 email verifications at any give time.
Throttling:
throttle.max_requests_per_minute: Set to 60. Limits request spikes to prevent SMTP server flags.
throttle.max_requests_per_day: Set to 10,000. Aligns with the recommended verifications per proxy IP.
The Dockerfile provided in the Commercial License Trial already has these parameters set up.
For scaling, buy additional proxy IPs and spawn more workers accordingly. The "rule of thumb" is 10,000 verifications per IP per day.
For example, if your target is 10 million verifications per month:
Daily volume = 10,000,000 emails / 30 days = 333,000 emails/day.
Number of required IPs = 333,000 emails/day / 10,000 emails/IP = 33 or 34 IPs.
We suggest deploying one Reacher instance per purchased IP. The settings outlined above ensure fair email verification usage as perceived by external email providers. For advanced applications, these values can be adjusted to optimize performance.
We do not recommend using Docker Compose for a high-volume production setup. However, for understanding or learning the architecture, this docker_compose.yaml file can be useful.
deploying on Kubernetes (Ansible playbook, Pulumi)
more specialized workers (e.g. some workers doing headless verification only, others doing SMTP only)
Contact if you have more questions about this architecture, such as:
