Option 1: RabbitMQ-based Queue Architecture
Last updated
Last updated
Reacher includes an optional, opinionated queue-based architecture designed to scale efficiently and handle high email verification volumes. This architecture comprises 4 main components and is highly configurable to meet specific business needs.
HTTP server
Accepts incoming email verification requests, and post them into the queue.
reacherhq/commercial-license-trial
Reacher uses a reliable, mature and open-source queue implementation.
rabbitmq:4.0-management
Workers
One or more consumers of the queue, which perform the actual email verification task.
reacherhq/commercial-license-trial
Storage
A place to store all results, currently only PostgresDB is supported.
postgres:14
Note that Reacher provides the same Docker image reacherhq/commercial-license-trial
which can act as both a Worker and a HTTP server.
With this architecture, it's possible to horizontally scale the number of workers.
To enable the worker-based architecture, configure the following parameters in your deployment. The parameters are given in their backend_config.toml
format (e.g. worker.enable
) as well as in the environment variable format (e.g. RCH__WORKER__ENABLE
):
worker.enable
or RCH__WORKER__ENABLE
: Set to true
to activate the worker role.
worker.rabbitmq.url
or RCH__WORKER__RABBITMQ__URL
: URL of the RabbitMQ instance for task queuing.
postgres.storage.postgres.db_url
or RCH__STORAGE__POSTGRES__DB_URL
: URL of a PostgreSQL database to store verification results.
Since spawning workers on cloud providers doesn't guarantee a reputable IP assigned to the worker, we configure all workers to use a proxy.
proxy.{host,port}
or RCH__PROXY__{HOST,PORT}
: Set a proxy to route all SMTP requests through. You can optionally pass in username
and password
if required.
You may also configure the following parameters:
Concurrency:
worker.rabbitmq.concurrency
: Set to 5
. Each worker processes up to 5 emails concurrently.
Throttling:
throttle.max_requests_per_minute
: Set to 60
. Limits request spikes to prevent SMTP server flags.
throttle.max_requests_per_day
: Set to 10,000
.
deploying on Kubernetes (Ansible playbook, Pulumi)
more specialized workers (e.g. some workers doing headless verification only, others doing SMTP only)
We do not recommend using Docker Compose for a high-volume production setup. However, for understanding or learning the architecture, this file can be useful.
Contact if you have more questions about this architecture, such as: