Understanding the Exchange transport pipeline with EXAMPLES !

This article aims to get a view of all the scenarios how SMTP messages travel through an Exchange organization. We'll cover all the variations, from having one server in the company only, all the way to many sites, with multiple Edge servers.

Single Exchange server

With a single server in the mix we have a simple job. The only two scenarios are there is receiving/sending external messages, and doing so between two mailboxes in the same server.

1. External, to single MB server

Message: Info@opentechtips.com -> Bob@alwayshotcafe.com

One of the most common scenarios. An external email arrives at the MAIL.alwayshotcafe.com server, it is received by the frontend transport service ( Default Frontend MAIL receive connector) on port 25/tcp (*Hop3). It is then passed over to the transport service (*Hop4), and then to the mailbox transport delivery service (*Hop5) that writes the message into the database.

2. Internal, to the same MB server

Message: Bob (Bob@alwayshotcafe.com) -> Kim (Kim@alwayshotcafe.com)

A very simple scenario: a mailbox user sends a message to another user on the same mailbox server. Here, after the message is received by the server it is passed to the mailbox transport delivery service by the transport service making it a three-hop transaction (if the first hop of Outlook sending the message is counted as hop 1)

Four AD sites, with a Hub site, with two Edge servers

In a simple environment only one single Mailbox server is used. There's not much to that scenario, both incoming and outgoing messages are passed to and from the internet through port 25/tcp SMTP port. However, things get interesting when multiple servers are involved. In our setup we have an EDGE server in the perimeter network, behind that four AD sites, each with an individual Exchange server, as shown in the following diagram

The site links costs in our system are as follows:

PS C:\> Get-AdSiteLink | Select Name, ADCost, ExchangeCost

Name             ADCost ExchangeCost
----             ------ ------------
NY-California       100           25
NY-Texas            100           50
California-Miami    100           25
Texas-Miami         100           50


PS C:\>

3. Route of Incoming SMTP from the Internet - no HUB sites

Message: Info@opentechtips.com -> John.Doe@alwayshotcafe.com Summary: with no hub sites, the mail is received by the EDGE server, forwarded to the site it has a subscription with (NY in our case), then directly delivered to the destination site (Miami).

PS C:\> Get-ADSite | Select name,*hub*

Name       HubSiteEnabled
----       --------------
NY                  False
California          False
Texas               False
Miami               False

Details:

As expected, SMTP messages arrive on port 25/tcp at the EDGE server through receive connector: "Default internal receive connector EDGE".  Then they are forwarded by the transport service also to port 25/tcp to the receiving frontend transport service on MB1, as this site has a subscription with the EDGE server. Receiving connector name at this hop: Default Frontend MB1. From there the message is forwarded to MB4 straight, using port 2525/tcp and receive connector "Default MB4" on the destination server.

4. Route of Incoming SMTP from the Internet - high cost HUB site

Summary: same as 3. After received by EDGE, the message is sent to MB1 (subscribed AD site), then straight to MB4 destination server.

PS C:\> Get-ADSite | Select name,*hub*

Name       HubSiteEnabled
----       --------------
NY                  False
California          False
Texas                True
Miami               False


PS C:\>

5. Route of Incoming SMTP from the Internet - lowest cost HUB site

Message: Info@opentechtips.com -> John.Doe@alwayshotcafe.com

Summary: with a hub site on the lowest cost route, the SMTP message is routed through the HUB site (California) on it's way to the destination (Miami).

PS C:\> Get-ADSite | Select name,*hub*

Name       HubSiteEnabled
----       --------------
NY                  False
California           True
Texas               False
Miami               False


PS C:\>

Details:

1. As expected SMTP messages arrive on port 25/tcp on the EDGE server through receive connector: "Default internal receive connector EDGE"
2. It is then forwarded by the transport service also on port 25/tcp to the frontend transport service MB1, that has a subscription with the EDGE server. Receiving connector: Default Frontend MB1
3. Message is routed to MB2, using port 2525/tcp
4. From the HUB site, the message reaches it's final destination MB4 on port 2525/tcp, through the "Default MB4" receive connector.

6. Outbound routing through equal-cost Edge servers

Message: John.Doe@alwayshotcafe.com -> Info@opentechtips.com

Summary: With multiple Edge servers available in the system - assuming that all servers can use all Edge servers to send emails out from the organization - the mailbox server will search for the lowest cost route to the internet. We have two Edge servers, EDGE and EGDE2. To reach the internet from EDGE, first the message needs to travel from MB4 to MB2, then from MB2 to MB1 and finally from MB1 to the EDGE server that will submit the message to the recipient. Adding that together it's 25 + 25 + 100 = 150.

Now, submitting through EDGE2 would cost only 100 as MB4 is directly adjacent with EDGE2, see the illustration below.

This means the system will use EDGE2 to send John's email out.

Details: the frontend transport service on MB4 submits the SMTP message to EDGE on port 25/tcp, then EDGE sends out the message to the next hop the MX record of the recipient dictates.

7. When EDGE2 cost is higher than cumulative cost through EDGE

Message: John.Doe@alwayshotcafe.com -> Info@opentechtips.com

Summary: in this scenario the message from MB4 can also take two routes: one through EDGE, or EDGE2. See which is path costs less: through EDGE2 it's only one hop from MB4, but the cost of that link is 90. Through EDGE the message first need to be passed through MB2, then MB1, only then it reaches EDGE. Adding the link costs of 25 + 25 + 25 will return a cumulative cost of only 75, which is lower than through EDGE2, so the message will be forwarded following the longer route.

It begs for the question: how is it possible that even though this route is longer, it's still the preferred path out? Well, in real life connection speed between sites, physical locations differ. In this case we assume that the connection between MB4 and EDGE2 is a very slow connection, hence the higher cost. The other connection links have lower cost values associated, as they have faster network connection between each other. Note, that these cost values are set by administrators, not a representation of the actual link speed. In general, as a guidance see the cost and network speed table to decide what cost you assign to your links in a production environment. I include both the Cisco and Microsoft approach, you decide which one you prefer. ( * https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/determining-the-cost)

Details: reverse of 5.

8. Intra email from MB1 to MB4 through a HUB site

Message:  Alfred.Franco@alwayshotcafe.com -> John.Doe@alwayshotcafe.com

Summary: the message starts from MB1, then received by MB2, and passed to MB4. Again, you see MB4 twice, in hop 3 and 4. This is because (like in earlier examples) first the message is received by the transport service, then it is passed to the mailbox transport delivery service that can write the message in the mailbox database.  

9. Intra, one hop only

Message:  Alice@alwayshotcafe.com -> John.Doe@alwayshotcafe.com

Summary: Same as 8., just omitting the hop through the HUB site.