SMTP TLS security

Why MTA-STS and TLS-RPT matter even if you already terminate TLS at a mail gateway

Mail security appliances and cloud email filters can require TLS after mail reaches your edge. MTA-STS and TLS-RPT cover the public hop from the sending mail server to the MX you publish, and they give you reports when that hop fails.

How inbound email TLS is actually checked

When another organization sends mail to you, their Mail Transfer Agent looks up your MX records, connects on port 25, and usually offers STARTTLS. That handshake is often opportunistic: if TLS negotiation fails, many senders still deliver in cleartext unless a published policy tells them not to.

Your appliance or Microsoft 365 / Google Workspace tenant may insist on TLS for the next hop into the mailbox service. That protects mail after it arrives at your edge. It does not, by itself, publish a sender-visible rule for the first internet hop to your MX.

Sender MTA → your MX

Public SMTP delivery. MTA-STS is designed for this segment.

Your MX / appliance → mailbox

Internal or vendor TLS policy. Useful, but not a substitute for STS.

TLS-RPT reports

Aggregate JSON about certificate, policy, and STARTTLS failures on the public hop.

What happens on the public SMTP hop

Understanding this path is the key difference between gateway TLS settings and sender-visible transport security.

  1. Step 1

    Sender looks up MX and MTA-STS identity

    DNS returns your MX hosts plus an optional _mta-sts TXT record that tells supporting senders a transport policy exists.

  2. Step 2

    Sender fetches the HTTPS policy file

    The policy is loaded from https://mta-sts.<domain>/.well-known/mta-sts.txt. That host must present a valid SSL certificate or the policy may be ignored.

  3. Step 3

    Sender connects to your published MX over SMTP

    Port 25 STARTTLS is negotiated against the MX names listed in the policy. This is the public hop MTA-STS is designed to harden.

  4. Step 4

    TLS-RPT reports what happened

    Supporting reporters send aggregate JSON to your rua destination so certificate, policy, and STARTTLS failures are visible before you enable enforce.

Why an appliance TLS check is not enough

Security products often advertise “require TLS” or “reject cleartext.” That usually means the appliance rejects clients that connect to it without encryption. Attackers who sit on the public path between the sender and your published MX can still attempt a STARTTLS downgrade before the message ever reaches that appliance.

Without MTA-STS

Supporting senders treat TLS as best effort. Certificate mistakes, missing STARTTLS, or a downgrade attack can still result in cleartext delivery, and you may never see a report.

With MTA-STS + TLS-RPT

Senders that support RFC 8461 can require TLS to your published MX list, and RFC 8460 reports show policy and certificate failures so you can fix them before enforce mode.

MTA-STS modes and what to publish

Policy mode lives in the HTTPS file at mta-sts.<domain>/.well-known/mta-sts.txt. DNS only advertises that a policy exists.

none

STS is effectively off. Useful only while you are preparing DNS and the policy host.

testing (reporting)

Senders try to follow the policy and can report failures, but they still deliver if TLS fails. Use this while validating MX lists and certificates.

enforce

Supporting senders refuse delivery when they cannot complete valid TLS to a policy MX. Move here after TLS-RPT shows a clean reporting window.

Policy host certificate

The hostname mta-sts.<domain> must present a trusted HTTPS certificate. Email Security Monitor’s public domain check reports MTA-STS mode plus the policy-host SSL expiry so you can catch an expired STS certificate before senders stop loading your policy.

How TLS-RPT handles failures

Publish a TXT record at _smtp._tls.<domain> with a rua= destination. Supporting reporters periodically send compressed JSON about TLS session successes and failures for your domain, including result types tied to certificate validation, policy mismatch, and STARTTLS negotiation problems.

Those reports are the operational feedback loop for transport security, similar to what DMARC aggregate XML provides for authentication. They tell you whether inbound SMTP TLS is healthy across the senders that matter, not only whether your appliance is configured to require TLS locally.

Frequently asked questions

What is MTA-STS and how does it protect inbound email?

MTA-STS (Mail Transfer Agent Strict Transport Security) publishes a DNS TXT identity and an HTTPS policy that tell sending mail servers to use TLS when delivering mail to your domain. Without it, SMTP often falls back to opportunistic STARTTLS, which can be downgraded to cleartext by a network attacker.

If my mail security appliance already requires TLS, do I still need MTA-STS?

Yes. An inbound appliance or cloud email security service can enforce TLS after mail reaches your edge. MTA-STS covers the public internet hop from the sender's MTA to the MX hostname you advertise. Those are different segments of the delivery path, and appliance TLS settings do not publish a sender-visible policy for the first hop.

What do MTA-STS modes enforce, testing, and none mean?

Mode none disables STS behavior. Testing (often called reporting mode) asks senders to follow the policy and report failures without blocking delivery when TLS fails. Enforce tells supporting senders to refuse delivery when they cannot complete a valid TLS connection to the published MX hosts.

What is TLS-RPT used for?

TLS-RPT (SMTP TLS Reporting) publishes a rua mailto or https destination for TLS failure reports. Providers that support RFC 8460 can send you aggregate JSON about certificate problems, policy mismatches, and STARTTLS failures so you can fix transport issues before moving MTA-STS to enforce.

Why does the policy host certificate on mta-sts.example.com matter?

Senders fetch https://mta-sts.example.com/.well-known/mta-sts.txt to learn your policy. If that hostname has an expired, mismatched, or unreachable certificate, senders may treat the policy as unavailable and fall back to opportunistic TLS even when your DNS TXT identity exists.

How does inbound SMTP TLS get checked in practice?

The sending MTA looks up your MX, connects on port 25, and negotiates STARTTLS. With MTA-STS, that sender also fetches your HTTPS policy and validates the certificate for the published MX hosts. TLS-RPT then reports successes and failures so you can see whether the public hop is healthy across real senders.

Check MTA-STS and TLS-RPT on your domain

Run the free public check for SPF, DMARC, DKIM, MTA-STS mode, TLS-RPT, and the mta-sts policy-host certificate, then monitor the same posture on a schedule.