The dangerous part of send(message) is how much meaning we quietly pack into the word “send.”
"Why send(message) is not one operation. Explore transient vs. persistent communication, synchronous vs. asynchronous messaging, queues, publish/subscribe, message brokers, MPI, collective communication, delivery guarantees, ordering, and the contracts hidden behind distributed communication."
Two processes need to communicate.
One has some data.
The other needs it.
So we do the obvious thing: send(message)
There. ✨Sent✨
Except distributed systems have a habit of taking perfectly ordinary verbs and asking uncomfortable follow-up questions.
Where is the message now?
Is it still in the sender’s memory?
Has the operating system accepted it?
Has it crossed the network?
Has the receiving machine stored it?
Has the receiving process seen it?
Has the receiving process processed it?
Can the sender disappear now?
Can the receiver be offline?
And when send() returns, which one of those things are we actually claiming happened?
That is the trap.
“Send” sounds like one operation because human communication lets us get away with vague semantics.
Distributed communication cannot.
The system eventually has to define exactly what survives, who waits, where the message lives, and what counts as success.
And once we start answering those questions, send(message) turns out to be several very different contracts wearing the same verb.
First Question: How Long Should the Message Stay Alive?
Suppose Alice sends a message to Bob.
Bob is not running.
What should happen?
One perfectly reasonable answer is:
Then the message cannot be delivered.
This is transient communication.
The communication system keeps the message only while the communicating applications and supporting communication infrastructure are active enough to handle it. If the receiver needs to be present when the message arrives, absence matters.
Transport-level communication often has this flavour.
The sender and receiver are participating in the same live conversation.
That works beautifully when both sides are expected to exist at roughly the same time.
Then consider email.
Alice sends a message at 2 p.m.
Bob opens his mail at 9 p.m.
Nobody finds this surprising.
We quietly introduced a completely different contract.
The communication system takes responsibility for preserving the message until Bob can receive it.
That is persistent communication.
The sender does not have to remain running after submitting the message.
The receiver does not need to be running when the sender submits it.
Somewhere between them, the infrastructure remembers.
The distinction is not really about messaging syntax.
It is about who owns the message while the receiver is absent.
Transient communication says:
I will carry this while the conversation exists.
Persistent communication says:
Give it to me. I will keep it until the other side can deal with it.
Same word.
Very different promise.
Second Question: When Is the Sender Allowed to Continue?
Now suppose Bob is available.
Alice sends the message.
Can Alice immediately continue doing other work?
That gives us another axis entirely.
With asynchronous communication, the sender submits the message for transmission and continues.
The message may still be sitting in a local buffer.
It may be inside an intermediate communication server.
It may not be anywhere near Bob yet.
Alice has handed off responsibility and moved on.
With synchronous communication, Alice waits.
But even this turns out not to have one obvious meaning.
What exactly is she waiting for?
Perhaps she waits until the message reaches a buffer on Bob’s machine.
That is receipt-based synchronization.
Or perhaps she waits until the message is actually delivered to Bob’s application.
That is stronger.
Or perhaps she waits until Bob processes the message and sends something back.
That is response-based synchronization.
Now the innocent verb send has become a ladder of guarantees.
Submitted.
Buffered.
Received by host.
Delivered to application.
Processed.
Responded to.
Each level tells Alice more.
Each level can also make Alice wait longer.
And this is the part that is easy to miss:
Persistent vs. Transient and Synchronous vs. Asynchronous are not the same question.
One asks:
How long can the communication system preserve the message?
The other asks:
How long must the sender wait?
Those two choices can be combined.
A message can be persistent and asynchronous, like the intuitive model of email.
Persistent and synchronous communication can wait until durable storage exists at the receiving side without requiring the receiving application itself to be running.
Transient asynchronous communication can let the sender continue immediately while relying only on temporary buffering.
Transient synchronous communication can wait for receipt, delivery, or even a response while still assuming the live communication context exists.
We did not make messaging complicated for fun.
We discovered that “sent” was hiding multiple independent decisions.
The Queue Says, “I’ll Hold It”
Once persistence matters, a useful character enters the story.
The message queue.
Instead of requiring Alice and Bob to meet directly, Alice puts a message into a queue.
Bob retrieves it later.
The queue sits between them like a patient intermediary saying:
You two do not actually need to coordinate your calendars.
Applications communicate by placing messages into queues and removing them when ready.
That produces loosely coupled communication.
Alice does not necessarily need Bob to be active.
Bob does not necessarily need Alice to still exist.
Neither side has to wait for the other to reach the exact same point in its lifecycle.
AWS describes SQS in essentially this role: a durable queue that helps decouple distributed components, allowing producers and consumers to operate without requiring them to be concurrently available.
That is a profound change in ownership.
In direct transient communication, Alice and Bob are responsible for meeting each other.
With persistent messaging, the queue accepts part of that responsibility.
And because it does, the interface itself changes.
Instead of one vague send, queue systems expose operations with much clearer meanings.
Put appends a message.
Get waits until a message exists and removes it.
Poll checks without waiting.
Notify arranges for a handler to be called when a message becomes available.
The verbs are more precise because the state of the communication is now explicit.
The message can exist independently of both applications.
Queued Does Not Mean Processed
This creates another seductive assumption.
If Alice successfully puts the message in the queue, perhaps the job is done.
Not quite.
What has actually happened?
The queue accepted responsibility for the message.
That is valuable.
It is not the same thing as Bob processing it.
Bob may not read the queue immediately.
Bob may fail while processing.
The message may remain waiting.
A queue can therefore give the sender confidence about handoff without automatically giving confidence about application completion.
Modern queue systems make this distinction visible. In Amazon SQS, for example, a message can remain stored until a consumer retrieves it; while being processed it can remain hidden for a visibility period, and the consumer deletes it only after successful processing. (docs.aws.amazon.com)
That lifecycle exists because these are different events:
The communication system has the message.
and:
The application has completed the work represented by the message.
Again, “sent” was too vague.
The queue forces us to become honest.
Then We Stop Naming the Receiver
Queues decouple sender and receiver in time.
But Alice may still be sending work toward a particular destination.
What if she does not know who should receive the information?
Suppose Alice produces an event:
Order 781 was completed.
Several systems may care.
Billing.
Analytics.
Inventory.
Notifications.
Perhaps systems we have not even built yet.
Making Alice explicitly call every consumer creates another dependency problem.
So we change the communication model again.
Alice becomes a publisher.
She publishes the event, usually to a topic.
Interested systems become subscribers.
The publish/subscribe infrastructure determines who should receive which messages.
Now communication has been decoupled in several dimensions.
The publisher and subscriber do not necessarily need to be online at the same time.
They do not necessarily need to know each other’s addresses.
And the publisher does not need to stop and wait for every subscriber to finish processing the event.
AWS’s SNS documentation describes this fan-out model directly: publishers send through a topic, which distributes messages to multiple subscribers; combined with queues, messages can also be persisted for later asynchronous processing.
The publisher says:
This happened.
Not:
Bob, please execute method X.
That is a different relationship entirely.
The components are becoming less aware of one another.
Which is excellent for loose coupling.
It also means the communication infrastructure is taking on more responsibility.
Somebody Still Has to Understand the Message
There is another hidden assumption.
Alice publishes:
{ customer: 17, total: 4500 }
Bob expects:
{ customerId: 17, amount: 4500 }
The message arrived perfectly.
Congratulations. Nobody understands each other.
Transport success does not imply semantic compatibility.
Applications still need agreed message formats, schemas, commands, and meanings.
One option is standardization.
Make every participant agree on the same format.
That works until integrating independently developed systems becomes painful.
Another option is a message broker.
The broker sits between applications and can transform incoming messages into representations understood by destination applications.
Now the intermediary is no longer merely storing and forwarding.
It may be translating.
Routing.
Filtering.
Mediating between different expectations.
The system has bought more decoupling by moving more responsibility into the communication layer.
There is always a bill somewhere.
A Message Bus Is Not a Conveyor Belt
Once many applications communicate through shared infrastructure, we can go further and define a message bus.
Applications agree on message schemas.
They agree on command messages.
They use shared infrastructure for sending those messages to recipients.
This feels pleasantly orderly.
Then comes another assumption worth losing:
If I sent A and then B, surely everyone receives A and then B.
Not necessarily.
Routing can change.
Messages can be buffered.
Internal optimization can alter paths.
Different transports may behave differently.
Some messaging infrastructures explicitly provide ordering guarantees, while others do not or provide them only under particular configurations. For example, AWS distinguishes standard queues, which can deliver messages more than once and occasionally out of order, from FIFO queues with stronger ordering guarantees. (docs.aws.amazon.com)
This is another reminder that ordering is a guarantee, not a vibe.
If application correctness depends on order, the communication contract must actually provide that order or the application has to reconstruct it.
By now, our simple message has acquired quite a résumé.
Persistence.
Synchronization.
Delivery guarantees.
Destination independence.
Schemas.
Routing.
Ordering.
And we still have not reached the strangest version of send.
MPI Wants Speed More Than Comfort
Message queues are excellent when components need loose coupling and persistent asynchronous communication.
But consider a very different environment.
A parallel scientific program runs across hundreds or thousands of processors connected by a high-speed network.
The processes are alive together.
They are cooperating on one computation.
Milliseconds of messaging middleware overhead may be deeply unwelcome.
Persistence after one process disappears is not necessarily the primary goal.
What matters is fast, explicit communication between parallel processes.
Enter the Message-Passing Interface, or MPI.
MPI is designed for distributed-memory parallel applications and provides standardized communication primitives that can be used across different hardware platforms.
It is deliberately more explicit.
Processes have ranks.
They belong to communicators.
Messages have destinations and tags.
The programmer controls how data moves.
This gives MPI performance and flexibility.
It also gives the programmer enough rope to construct some wonderfully sophisticated deadlocks.
The abstraction is no longer trying to pretend that communication is someone else’s problem.
MPI looks at the programmer and says:
You wanted control. Here is control.
MPI_Send Still Does Not Mean What You Think
Consider: MPI_Send(...)
It is a blocking send.
Easy.
So it must block until the receiver receives the message.
Right?
Not necessarily.
The MPI standard says that a blocking standard-mode send does not return until the send buffer can safely be reused. The implementation may accomplish that by copying the message into a temporary system buffer, or by transferring it into the matching receive. Whether buffering is used in standard mode is left to MPI.
That means MPI_Send can return even though the destination process has not yet executed its matching receive.
Now make the message larger.
The MPI implementation may decide that buffering it is too expensive.
The same call may now have to wait for the receiver.
The source code still says: MPI_Send(...)
The observed behaviour changed because the implementation’s buffering strategy changed.
This is why MPI programs that secretly depend on “small messages are buffered, therefore my send won’t wait” are living on borrowed time.
A different MPI implementation can expose the assumption.
A different buffer threshold can expose it.
A different message size can expose it.
The code did not become wrong because the message grew.
The code was already relying on a guarantee MPI never made.
That is an uncomfortable but useful distinction.
MPI Has More Than One Kind of Send for a Reason
Once we understand the ambiguity, MPI’s multiple send primitives stop looking like API clutter.
A buffered send explicitly uses buffering.
A synchronous send waits until the corresponding receive has begun.
A standard send leaves more of the buffering decision to the implementation.
Nonblocking operations allow communication to begin while the process continues with other work, with completion handled separately.
The MPI standard deliberately provides these communication modes because buffering changes both performance and synchronization behaviour; standard mode may buffer, while synchronous mode lets the programmer require stronger coordination with the matching receive. (mpi-forum.org)
These are not several spellings of the same operation.
They express different contracts.
And once again, the right question is not:
Which send function is coolest?
It is:
What must be true before this process is allowed to continue?
That is the recurring question underneath almost every communication abstraction we have touched.
Receiving Requires Its Own Contract
Sending is only half the story.
An MPI receiver must decide what messages it is willing to accept.
Messages can be matched using a source, a tag, and a communicator.
The communicator defines the group of processes participating in that communication universe.
The process rank identifies participants inside that group.
Tags distinguish messages that might otherwise look similar.
A receive may accept a message from a specific process or use MPI_ANY_SOURCE when the sender does not need to be known in advance.
And a blocking MPI_Recv waits until a matching message arrives.
This explicit matching exists because in a parallel program there may be many messages in flight at once.
The receiver cannot simply say:
Give me “the message”.
Which message?
From whom?
For which phase of the algorithm?
Inside which group of processes?
Communication correctness increasingly depends not just on moving bytes, but on ensuring the right communication operations find each other.
MPI has made the hidden relationship between sender and receiver visible.
Then Point-to-Point Becomes Awkward
Suppose several processes are independently calculating pieces of one result.
At the end, Process 0 needs the total.
We could build that using individual sends.
Process 1 sends its partial result to Process 0.
Process 2 sends its partial result.
Process 3 sends its partial result.
Process 0 receives everything and adds it.
That works.
Then the process count grows.
We can make the algorithm more clever by combining partial results through a tree.
Now processes exchange intermediate sums.
Fewer communication steps.
Better use of parallelism.
And eventually an obvious question appears:
Why should every program rebuild the same communication pattern manually?
That is where collective communication becomes natural.
Instead of describing dozens of individual point-to-point messages, express the group operation itself.
MPI_Reduce says:
Combine values contributed by the processes and place the result at one destination.
MPI_Allreduce says:
Combine them and give the result to everybody.
MPI_Bcast says:
One process has data; distribute it to the group.
MPI_Scatter says:
Divide data from one process among the group.
MPI_Gather says:
Collect distributed pieces onto one process.
MPI defines these explicitly as collective operations involving groups of processes rather than as mere convenience wrappers around an arbitrary collection of sends.
The abstraction has moved upward.
We are no longer saying how each pair of processes should communicate.
We are describing what the group is trying to accomplish.
A Collective Is a Group Contract
This is where collective communication can punish a point-to-point mental model.
Suppose Process 0 calls MPI_Reduce.
Process 1 decides to be clever and calls MPI_Recv instead.
They both seem related to receiving data.
They are not interchangeable.
The program is erroneous.
Collective operations have collective semantics.
The processes participating in the communicator need to participate compatibly.
The MPI standard requires all processes in the communicator’s group to call the collective routine, and collective operations must be executed in the same order across that communication group.
This can feel annoyingly strict until the deeper model becomes visible.
A collective operation is not:
Process 0 happens to receive several messages.
It is:
This group of processes is participating in one coordinated communication operation.
The communicator defines who belongs to that contract.
The operation defines what role their data plays.
The ordering lets MPI match collective operations correctly and select efficient implementation strategies.
Once we accept that, the rules stop looking arbitrary.
The operation is bigger than any one sender.
“Send” Was Really a Question About Responsibility
Look back at how far we travelled from: send(message)
Transient communication asked whether the receiver needed to be present.
Persistent communication moved responsibility for storing the message into the communication system.
Asynchronous communication allowed the sender to continue after submission.
Synchronous communication made the sender wait for stronger evidence.
Receipt-based synchronization waited for the receiving host.
Delivery-based synchronization waited for the receiving application.
Response-based synchronization waited for processing to come back around.
Queues separated sender and receiver in time.
Publish/subscribe separated them in space and knowledge of one another.
Message brokers took responsibility for translating between different message formats.
MPI moved toward explicit, high-performance communication where buffering and synchronization became programmer-visible concerns.
Collective communication went further and made the whole group operation the unit of meaning.
None of these mechanisms exists because engineers could not settle on one good send() function.
They exist because distributed applications need different answers to the same questions:
Who owns the message now?
How long will it survive?
Who must be alive?
Who has to wait?
What counts as delivery?
What ordering is guaranteed?
Who participates?
What happens when the operation involves an entire group instead of one sender and one receiver?
Once those questions differ, the communication operation has to differ too.
Make the Contract Say What the System Means
The broader mistake is treating communication like physical movement.
Alice had bytes.
Now Bob has bytes.
Done.
But useful distributed communication is rarely just about where the bytes are.
It is about what the participants are allowed to assume after the communication operation returns.
A queue lets the producer assume the infrastructure accepted responsibility.
A response-based call lets the sender assume processing reached a response.
A synchronous MPI send gives a different synchronization relationship than a buffered one.
A collective says something about an entire group, not merely two endpoints.
Those are semantic guarantees.
And stronger guarantees are not free.
Persistence requires storage.
Synchronization introduces waiting.
Loose coupling requires intermediaries.
Ordering restricts implementation freedom.
Collective coordination requires compatible participation.
Performance improves when we stop demanding guarantees we do not need.
Correctness improves when we stop assuming guarantees we were never given.
So the next time The Origami Software Engineer’s API says send,
the useful question is not simply:
Where will this message go?
It is:
What exactly will be true when this operation says it is finished?
That is when messaging stops looking like a collection of oddly named primitives and starts looking like a set of deliberate contracts between independent processes.
“Send” was never one operation.
We just started with a mental model too small to notice.
That’s not failure.
That’s evolution.
The “I liked this” Starter Pack:
Don’t let your fingers get lazy now.
- Like : It tells me this was worth writing.
- A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
- Boost it: Especially with that one developer who definitely needs this.
Thanks for being here. It genuinely helps more than you know!
— Aaroophan Varatharajan
Find me elsewhere:
- Professional stuff: linkedin.com/in/Aaroophan
- Code stuff: github.com/Aaroophan
- UI stuff: aaroophan.dev/Aaroophan
- Life stuff: instagram.com/Aaroophan
