WHMCS MailBaby SMTP Settings: Reliable Notifications with DKIM

WHMCS is the one application on a hosting company’s infrastructure whose mail must arrive: invoices, password resets, ticket replies and suspension warnings all go through it, and a billing domain that lands in spam costs money directly. There are two sensible ways to route WHMCS mail through MailBaby. Either WHMCS speaks SMTP to the relay itself, or it hands mail to the local MTA which is already relaying everything. This guide covers both, then the DKIM configuration that decides whether recipients see your billing domain or “via mailbaby.net”.

Short answer: In WHMCS » General Settings » Mail, set Mail Type to SMTP, host relay.mailbaby.net, port 25, SSL Type TLS (STARTTLS, not SSL), username mbXXXXX with no domain, and a From address on a domain authorised for your MailBaby account. If WHMCS runs on a cPanel or DirectAdmin server that already relays through MailBaby, choose PHP Mail instead so Exim applies the domain’s DKIM signature; in direct SMTP mode nothing signs the message and MailBaby’s transport signature will not align with a quarantine or reject DMARC policy on your billing domain.

Option one: WHMCS direct to the relay

In WHMCS » System Settings » General Settings » Mail, choose the SMTP mail provider and enter:

• Mail Type: SMTP
• SMTP Host: relay.mailbaby.net
• SMTP Port: 25
• SMTP SSL Type: TLS (this is STARTTLS in WHMCS terminology; do not choose SSL, which expects an implicit-TLS port)
• SMTP Username: mb12345 (the bare MailBaby account name with no domain)
• SMTP Password: the relay password from the InterServer portal

Set the System Emails From Email to an address on a domain that is authorised for your MailBaby account, for example billing@example.com. WHMCS uses that address as the envelope sender, and MailBaby rejects senders whose domain is not authorised. Publish either include:spf-c.mailbaby.net in the domain’s SPF or the _mailbaby.example.com TXT record; see the SPF guide.

This option is best when WHMCS runs on a server that does not otherwise send mail, such as a dedicated billing VM, or when you want billing mail on a separate MailBaby account for clean logs.

Option two: WHMCS through the local MTA

If WHMCS lives on a cPanel or DirectAdmin server already relaying through MailBaby, choose PHP Mail in the Mail settings and let Exim route it. Exim adds the X-AuthUser and source headers cPanel relies on, DKIM is applied by the transport, and you have one relay configuration to maintain. The Exim side is covered in the cPanel guide and the DirectAdmin guide.

The PHP Mail path has one weakness: the envelope sender defaults to the web server or account user, so set the system’s From Email in WHMCS and confirm in the Exim log that the <= line shows billing@example.com rather than user@hostname. If it does not, add -f billing@example.com to the sendmail_path in the account’s PHP configuration or switch to option one.

DKIM for the billing domain

MailBaby passes your DKIM signature through untouched and signs with its own transport domain only when no signature is present. A missing signature is why some providers see “via mailbaby.net” next to their invoice sender in Gmail; the mechanism is explained in the transport signing guide.

With option two, DKIM is the panel’s job. On cPanel:

whmapi1 install_dkim_private_key domain=example.com
whmapi1 enable_dkim domain=example.com
whmapi1 fetch_dkim_private_key domain=example.com | head -3

With option one, WHMCS has no DKIM signer of its own for SMTP mode, so the signature must be applied by the relay. MailBaby’s transport signature covers the message with its own domain, which passes DKIM but not DMARC alignment for example.com. If your billing domain publishes a DMARC policy of quarantine or reject, use option two so the panel signs, or run WHMCS behind a local Postfix with OpenDKIM as in the Postfix guide and point WHMCS at localhost:25. That gives you aligned DKIM and a MailBaby relay with a single hop added.

Throughput and cron

WHMCS sends invoice reminders and mass mails from its daily cron in bursts. MailBaby caps a sender address at 6,000 messages per hour and discards the excess. A WHMCS install with more than a few thousand clients should stagger the cron: enable Automation Settings » Invoice Generation » Throttle or split the mass mail into batches. If billing mail is routed through cPanel, per-domain hourly limits in Tweak Settings apply to it too; give the billing account a higher limit than shared customers with whmapi1 modifyacct user=billing MAX_EMAIL_PER_HOUR=2000.

Verify

WHMCS records every outbound message in Utilities » Logs » Email Message Log. Send a test from General Settings » Mail » Test Email Settings, then confirm the entry appears and that the received message’s Authentication-Results: shows spf=pass and dkim=pass header.d=example.com. If header.d= shows a mailbaby.net domain, DKIM is being applied by the relay rather than by you, and DMARC alignment will fail. Check the InterServer portal log to confirm a low score. The common pitfall is choosing SSL rather than TLS in the SMTP SSL Type field; WHMCS then attempts an implicit-TLS handshake on port 25 and reports a connection timeout that looks like a firewall problem.

WHMCS MailBaby SMTP at a glance

Official documentation: RFC 6376 (DKIM), cPanel & WHM documentation, RFC 5321 (SMTP).

Related guides: MailBaby with Plesk and Webuzo: outbound relay configuration · What is MailBaby? Outbound SMTP relay pricing, limits and how it works · How MailBaby handles feedback loops and abuse reports, and what to do when you get one.

Frequently asked questions