Real-time updates on the web is an add-on. The classical web is built on a stateless request/response-oriented model where clients pull data from servers when needed.
Low-latency updates for some applications, like games and video-conferencing, require long-lived streaming transports like WebSocket or Server-Sent Events.
The question is how far you can push the stateless model using modern HTTP features before needing that upgrade.
Applications have different real-time requirements.
Some applications only need to receive updates every few seconds at the most and every few days at the least.
The choice of technology is a tradeoff between real-time requirement, the cost of network data transfer, scalability, and management of server state.
The question
What is the network cost of approximating real-time updates
using HTTP polling and conditional requests?
Approximate real-time in this context means update latencies on the order of seconds,
not milliseconds, with predictable and bounded network overhead.
The motivation for examining this question is to justify the use of naive HTTP
polling in developing the local.html link network.
To better evaluate the feasibility of this architectural choice, four separate experiments with connection-oriented protocols and HTTP polling are conducted to measure network cost.
Hypothesis
Modern HTTP reduces the cost of polling to the point where it can approximate real-time delivery for low-frequency updates.
This relies on conditional requests to avoid needlessly retransmitting payloads (304 responses) and reduction of per-request overhead through HTTP/2 header compression and connection reuse.
The experiments
The experiments are designed to compare relative network cost under a specific workload, not to provide a comprehensive evaluation of the protocols.
In each experiment a server is set up to deliver a payload of 6209 bytes to a client every 2 minutes over a 6 minute period.
Server apps are written in Go using
net/http and
github.com/coder/websocket
packages.
Client apps use
WebSocket,
EventSource, and
Fetch
browser APIs.
Chrome browser is used as network client in all experiments. A new incognito session is started for each run to clear caches and close TCP connections.
Client and server apps are served from different origins. Server apps bind only to IPv4 interfaces.
Each experiment is run 10 times on localhost and 10 times on a remote VM in North America. The client apps connect to the remote servers from a laptop connected to a mobile Wi-Fi hotspot in Northern Europe.
The localhost setup provides a low-latency reference for network cost measurements.
The remote setup tests byte overhead under a realistic WAN path, including retransmissions and connection churn.
Network cost is measured as total bytes transferred on the wire in both directions during the experiment.
This includes the size of the payload, application layer protocol overhead (HTTP request and response headers, WebSocket frames, SSE event stream formatting), security layer protocol overhead (TLS handshake and session management), and transport layer protocol overhead (TCP connection setup and retransmissions).
WebSocket
In the WebSocket experiment the server creates a listener that accepts connections and sends
a payload when client connects and subsequently every 2 minutes.
The server sends ping frames every 10 seconds.
In the server config read, write and idle timeouts are disabled.
The client runs a script that creates a WebSocket object, sends nothing and
logs all received messages to the console.
|
Avg. throughput |
Total bytes |
| localhost |
107.56 B/s |
37648 |
| remote VM |
105.35 B/s |
36922 |
The WebSocket connections are typically very long lived.
In this experiment the short benchmark duration means that the initial TLS handshake remains a noticeable fraction of total network cost.
Server-Sent Events (SSE)
In the SSE experiment the server responds to requests with a
text/event-stream response. A payload is sent immediately and
subsequently every 2 minutes.
The server sends sequence numbers with every event.
The server also sends a heartbeat event every 10 seconds.
In the server config read, write and idle timeouts are disabled.
The client runs a script that creates an EventSource object and
logs all messages to the console.
|
Avg. throughput |
Total bytes |
| localhost |
125.80 B/s |
44033 |
| remote VM |
126.97 B/s |
44501 |
Like WebSocket, SSE typically uses long-lived connections, so in practical deployments the cost of the initial TLS handshake is amortized over much longer periods than in this experiment.
HTTP polling
The polling experiment is run using both HTTP/1.1 and HTTP/2 protocols.
Both the HTTP/1.1 and HTTP/2 server apps conditionally serves the payload to the client using a Last-Modified validator.
The servers are configured to update the payload modification time every 2 minutes, which will trigger a full download on the client.
The servers set Cache-Control: no-cache header to force deterministic revalidation.
Server timeouts:
- Read timeout: 10 seconds
- Write timeout: 30 seconds
- Idle timeout: 120 seconds
The client uses the local.html app to subscribe to updates from the server.
The local.html app is configured in this experiment to poll the server with a fixed base interval of 5 seconds, applying full jitter to randomize each delay.
The jitter is the app's default behavior and not a parameter of the experiment.
HTTP/1.1
|
Avg. throughput |
Total bytes |
| localhost |
285.03 B/s |
101964 |
| remote VM |
249.50 B/s |
88942 |
The HTTP/1 polling experiment shows spikes in data transfer between update intervals comparable to the payload. The packet trace shows that these spikes coincide with client-initiated connection tear-down. The spikes are comparable in size to the TLS handshake (4-6 KB) and likely correspond to client-initiated connection replacement. Even with keep-alive and constant polling, the client will use an heuristic to independently determine the best time to close existing connections.
Total bytes transferred is larger on localhost than across the Atlantic. The same amount of application data is transferred, so the difference must be transport overhead. The loopback traces contain substantially more packet records, suggesting that the difference is caused by differences in packetization and transport-layer behavior between loopback and WAN paths.
HTTP/2
|
Avg. throughput |
Total bytes |
| localhost |
185.41 B/s |
66354 |
| remote VM |
150.15 B/s |
53221 |
The HTTP/2 polling experiment has the same irregularities in transport protocol network cost caused by client connection management heuristics and differences in packet fragmentation as the HTTP/1.1 experiment.
Total bytes transferred is 1.6× lower for HTTP/2, which is likely due to HPACK compression.
Discussion
In this setup the HTTP polling server performed at 1.4× the network overhead of
WebSocket protocol and at 1.2× the network overhead of SSE protocol while polling at approximately 5 second intervals.
While far from the capabilities of a WebSocket connection it is within the limits of a tolerably slow chat session, such as a web IRC bridge or support chat.
For systems with infrequent updates, this overhead remains bounded
and predictable, even when using a stateless request–response model. In the context of local.html, where updates occur on the order of minutes or hours,
this difference translates to a modest absolute increase in bandwidth usage.
Persistent connection models such as WebSocket and Server-Sent Event protocols are well suited for very low-latency, small updates.
HTTP/2 polling incurs higher overhead per update, but in this setup only by a factor of 1.2-1.4×.
For applications with relaxed real-time requirements, HTTP/2 polling can achieve similar behavior with a modest increase in bandwidth, while simplifying deployment and scaling.
The friend link network is characterized by a low bar of entry, decentralized topology, use of classical web technologies, stateless HTTP, no central coordination, user-defined relationships and
automatic discovery.
In this architecture polling is not just a transport workaround, but enables a distribution model where updates are discovered by clients, not distributed by a server.
The result favors this special use-case: an alternative architecture that, in the spirit
of the open web, assumes independent origins, no prior coordination, stateless HTTP, and pushes authority and responsibility to the edge of the network.
The default polling model for local.html is to poll 5 minutes after an update and then gradually back off to a maximum polling interval of 24 hours. Polling frequency is configurable per source, and can, as demonstrated in these experiments, practically be reduced to intervals as low as 5 seconds.
Source code
Benchmark client and server source code as a git bundle.
local.html app source code.