The API can hide the distance. It cannot erase what the distance changes.
"Explore the CAP theorem through practical examples of replication, network partitions, split brain, quorum, consistency, and availability and learn what distributed systems sacrifice when communication fails."
Suppose a client needs some work done by another component.
On one machine, this is boring in the best possible way.
Call a procedure.
Pass some arguments.
The procedure runs.
Get the result back.
✨Continue✨
Then we move that procedure onto another computer.
The first instinct is almost irresistible:
Why should the application care?
If the remote machine exposes the same operation, perhaps we can make calling it look exactly like calling a local procedure.
Instead of forcing every application developer to think in sockets, packets, addresses, message formats, and transport protocols, give them something familiar.
result = calculate(x, y)
The caller should not have to stare into the abyss and think:
- open socket
- encode x
- encode y
- construct message
- send bytes
- wait
- receive bytes
- decode result
That is the promise of Remote Procedure Call, or RPC.
Call a procedure on another machine just as you would call a local procedure.
Birrell and Nelson’s original RPC design was built around exactly this idea:
Provide communication between programs across a network through the familiar abstraction of a procedure call. (Microsoft)
It is an excellent abstraction.
Which is why it is worth understanding where the abstraction stops.
The Network Does Not Know What a Function Is
Before RPC can hide the network, something still has to use the network.
Distributed communication commonly happens over IP using transports such as TCP or UDP.
TCP gives us a connection-oriented byte stream.
UDP gives us connectionless datagrams.
Neither one understands: calculate(x, y)
The network understands messages and bytes.
So somewhere between the line of application code and the remote implementation, the procedure call has to be transformed into communication.
That is where RPC begins doing its little stage trick.
The application calls a client stub as though it were an ordinary procedure.
The stub builds a message and passes it to the communication middleware.
The message crosses the network.
On the server, middleware passes it to a server stub, or skeleton.
The server side reconstructs the arguments and calls the real local implementation.
The implementation returns a result.
Then the whole journey happens in reverse.
Server implementation → skeleton → middleware → network → client middleware → client stub → application.
The application sees: result = calculate(x, y)
The distributed system sees a round trip between two independent processes.
The ONC RPC specification describes essentially this model: the caller sends a call message containing the procedure parameters, waits for a reply message containing the results, and resumes once that reply arrives. (RFC Editor)
The procedure call did not cross the network.
A representation of the procedure call did.
That distinction is going to become expensive.
Even the Arguments Cannot Travel as Themselves
Suppose x is an integer.
Easy enough.
Put the integer into the message.
Except two machines may not even represent that integer identically.
One architecture may use one byte order.
Another may use another.
Strings may have different encoding or alignment rules.
Floating-point representations have to agree.
Arrays and structures have to be flattened into something that can travel and reconstructed at the other end.
Memory addresses are even more suspicious. A pointer that means something inside the client’s address space does not suddenly become meaningful inside the server’s address space.
So the stub has another responsibility.
Marshalling.
Data is encoded into a transmissible representation.
On the receiving side it is unmarshalled, or decoded, back into usable data structures.
Protocols therefore need agreed representations for parameters and results. ONC RPC, for example, uses External Data Representation to describe the messages exchanged between participants. (RFC Editor)
This is the first crack in the illusion.
A local call can hand another procedure values that already live in the same computational world.
A remote call has to translate one machine’s world into something another machine can reconstruct.
The API may look local.
The data most definitely knows it travelled.
Then We Have to Find the Procedure
There is another detail a local call gets almost for free.
When code calls a local procedure, the runtime knows where that procedure is.
A remote service may be somewhere else entirely.
Which machine?
Which process?
Which endpoint?
Which version?
So remote invocation needs binding.
A client has to discover and connect to the service that implements the desired interface.
That may involve some kind of directory or registry.
Once bound, the client can continue pretending that it simply possesses something callable.
This is location transparency doing useful work.
The client wants:
Give me the service that performs this operation.
It does not necessarily want:
Please make me personally manage the physical location and network configuration of the process currently implementing it.
Even formal RPC specifications separate the call protocol from the higher-level mechanism that binds a client to a particular service and transport endpoint. (RFC Editor)
Again, perfectly reasonable abstraction.
Again, the machinery underneath has not disappeared.
We have hidden where the call goes.
We still have to deal with what happens while it is going there.
Local Calls Have a Very Comfortable Timeline
Consider an ordinary synchronous procedure call.
You call.
You wait.
The procedure returns.
You continue.
RPC can reproduce that programming model.
But once the operation becomes communication, other interaction patterns suddenly become useful.
The client may send a message and wait for a response.
Or send without waiting at all: fire and forget.
Or send and wait only for an acknowledgement that the request was received.
A client may block.
It may poll.
It may provide a callback.
The call may be asynchronous or use deferred synchronization.
The ONC RPC model itself notes that implementations are not restricted to the simple blocking model; asynchronous execution is possible so the client can continue doing useful work while the remote operation proceeds. (RFC Editor)
Why does a supposedly local-looking call suddenly need all these options?
Because latency exists.
A local function call is usually close enough that waiting is the natural default.
A remote invocation can spend meaningful time crossing a network, waiting in another process, doing work, and crossing the network again.
Distribution changed the cost of waiting.
So even before anything fails, the network has already leaked into the programming model.
Then something fails.
The Timeout Is Where the Illusion Gets Cursed
The client sends a request.
Nothing comes back.
So it waits.
Eventually it times out.
What happened?
That sounds like a simple question. It is not.
Perhaps the request packet was lost before reaching the server.
Perhaps the request reached the server, but the acknowledgement was lost.
Perhaps the server executed the procedure and the reply was lost.
Perhaps the network became unavailable.
Perhaps the server process failed.
Perhaps the entire server machine failed.
From the client’s point of view, several very different realities can collapse into exactly the same observation:
No reply arrived.
This does not happen in the same way with an ordinary local procedure call.
The network has introduced ambiguity.
And ambiguity is where “remote is just like local” finally stops being an innocent simplification.
Fine. Retransmit It.
For ordinary communication failures, there are reasonable techniques.
Give requests unique identifiers or sequence numbers.
Use acknowledgements.
Detect duplicates.
If a message appears to have been lost, retransmit it.
This works beautifully for a lost packet.
Client sends request.
No acknowledgement.
Timeout.
Client sends request again.
Server receives it.
Reply arrives.
_Problem solved.
_Except there is another possible history.
Client sends request.
Server receives it.
Server performs the operation.
The acknowledgement or reply disappears.
Client times out.
Client sends the request again.
Now the server has seen the same operation twice.
For something harmless, perhaps that does not matter.
For an operation with side effects, it matters quite a lot.
Imagine the remote procedure is: TransferMoney(100)
The client did not ask: MaybeTransferMoneySomeNumberOfTimes(100)
Retries make the transport more reliable.
They can simultaneously make the operation less obviously correct.
This is why duplicate detection exists. A server can recognize an identifier it has already processed and avoid executing the same logical request again. ONC RPC similarly uses transaction IDs to match requests and replies and describes retaining those IDs as a way of obtaining a degree of execute-at-most-once behaviour. (RFC Editor)
The retry mechanism is not merely recovering lost packets.
It is trying to recover the meaning of the original invocation.
Did the Server Run It?
Now we reach the unpleasant question.
The client sends a request.
The server executes it.
Before the client receives the reply, communication disappears.
What should the client conclude?
The tempting answer is:
The call failed.
But what exactly does failed mean?
It certainly means the client did not receive a successful result.
It does not necessarily mean the server did not perform the operation.
That is the hidden difference.
The client knows what it observed.
It does not automatically know what happened on the other machine.
Even with a reliable transport such as TCP, receiving a reply allows the caller to know the remote operation executed, but receiving no reply does not prove that it never executed. The RPC specification states this distinction explicitly. (RFC Editor)
That gives a timeout a very different meaning from the one our local-call intuition wants to assign to it.
A timeout does not say:
The procedure did not execute.
It says:
I can no longer determine the outcome from the communication I received.
That sounds like a small wording difference.
It changes application design.
Retries.
Idempotency.
Error handling.
Transaction semantics.
Recovery.
All of them eventually run into the same question:
Did nothing happen, or did something happen and I simply failed to hear about it?
The network can make those realities indistinguishable from the caller’s side.
Failure Transparency Has a Limit
RPC originally aimed for a highly transparent programming model.
Hide location.
Hide communication.
Hide parameter conversion.
Make remote interaction feel ordinary.
But failures are where complete transparency becomes dangerous.
If a local-looking call can fail because another machine disappeared, the caller eventually needs some way to know that this failure category exists.
Modern RPC designs therefore acknowledge the things the abstraction cannot safely erase: network failures, remote process failures, communication latency, asynchronous behaviour, and the possibility that a request’s outcome is uncertain.
A remote call is still a convenient abstraction.
It is just no longer pretending that distribution has no semantic consequences.
This is a healthier contract.
The abstraction says:
You do not need to manually construct network messages for every invocation.
It does not say:
You may reason as though there is no network.
Those are very different promises.
Then We Tried It With Objects
Once procedures could be invoked remotely, the next progression was natural.
What about objects?
That leads from RPC toward Remote Method Invocation, or RMI.
Instead of invoking a standalone remote procedure, a client invokes a method on a remote object.
The client first binds to that distributed object.
A proxy representing the remote interface lives on the client side, playing a role similar to the RPC client stub.
The proxy serializes or marshals the invocation into messages.
On the server, a skeleton receives the invocation, unmarshals it, and calls the actual object’s implementation.
Results travel back in the opposite direction.
But an important boundary remains.
The remote object’s entire memory does not magically teleport into the client.
What becomes remotely accessible is its interface.
The interface says what operations can be invoked.
The implementation and state still live elsewhere.
Java RMI makes that separation explicit: a remote interface declares the methods that can be invoked from another virtual machine, and failures during remote invocation include communication problems as well as marshalling and protocol errors. (Oracle Docs)
The object may look near.
Its state is still remote.
Remote Exceptions Are the Abstraction Admitting the Truth
RMI produces one particularly honest design choice.
Remote exceptions.
Suppose the remote method itself throws an application exception.
That failure can be marshalled into a response, sent back to the client, reconstructed, and raised as an exception on the caller’s side.
From the application’s point of view, that can feel pleasantly local.
But remote invocation also has failure modes that exist because the method is remote.
The server may be unreachable.
The connection may fail.
Arguments may fail to marshal.
The response may fail to unmarshal.
The protocol may fail.
Java RMI therefore requires remote interfaces to account for RemoteException, which represents communication-related failures that can occur during a remote invocation. (Oracle Docs)
That requirement is almost philosophical.
The API still gives us the elegance of: RemoteObject.DoSomething()
But the type system quietly taps us on the shoulder:
Remember. This can fail in ways a local method cannot.
That is not the abstraction giving up.
That is the abstraction becoming honest.
Transparency Should Remove Work, Not Reality
RPC succeeded because the original idea was good.
The Origami Software Engineer should not have to rebuild networking primitives every time one program needs to ask another program to do something.
Stubs are useful.
Marshalling is useful.
Binding is useful.
Location transparency is useful.
Remote interfaces are useful.
Turning messages into something that resembles ordinary procedure or method invocation is one of the most productive abstractions in distributed computing. The original RPC work and later standardized RPC protocols both preserve that familiar call model while explicitly dealing with binding, message representation, transport, and failure semantics. (Microsoft)
The mistake is not using the abstraction.
The mistake is believing it literally.
A local procedure call and a remote procedure call may share syntax.
They do not share physics.
One crosses an address-space boundary.
The other crosses machines, networks, representations, clocks, failure domains, and potentially administrative boundaries.
One can usually tell you whether control returned.
The other may leave you asking whether the operation happened at all.
So good distributed abstractions do something subtler than making the network disappear.
They hide the mechanics that programmers should not have to repeat.
Then they expose the consequences that programmers cannot safely ignore.
The remote function call was never going to become truly local.
It did something more useful.
It became familiar without pretending that distance was free.
That’s not failure.
That’s evolution.
The “I liked this” Starter Pack:
Don’t let your fingers get lazy now.
- Like : It tells me this was worth writing.
- A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
- Boost it: Especially with that one developer who definitely needs this.
Thanks for being here. It genuinely helps more than you know!
— Aaroophan Varatharajan
Find me elsewhere:
- Professional stuff: linkedin.com/in/Aaroophan
- Code stuff: github.com/Aaroophan
- UI stuff: aaroophan.dev/Aaroophan
- Life stuff: instagram.com/Aaroophan
