Skip to content

Networking architecture

How the R-Type server and client are built on the engine's net module: which thread does what, how messages cross between threads, what goes over TCP and over UDP, at which rates, and how each side notices that the other is gone. The bytes on the wire are specified in the protocol RFC; the engine's building blocks are listed in the architecture overview. The R-Type code lives in the project-rtype repository: server/, client/network/ and protocol/.

TCP and UDP

TCP UDP
Carries The session: LoginRequest, LoginAccepted, LoginRejected, Logout (RFC 3) The game: Connect / Accept, Input, Snapshot, Event, Heartbeat, Disconnect (RFC 2, 4, 5)
Why Reliable and ordered: a login must arrive, once No head-of-line blocking: a late snapshot is replaced by the next one rather than holding it up
Lifetime Opened first, kept open for the whole session; its closing ends the session Bound to the session by Connect once logged in
Engine type net::TcpListener, net::TcpConnection net::UdpSocket

The session token issued over TCP is in the header of every datagram; the server only accepts a datagram from the endpoint bound to the session of its token (RFC 3.4).

Threads on the server

The server has two threads, and each piece of state belongs to one of them:

flowchart LR
    subgraph main["Main thread: network"]
        ctx["NetworkContext<br/>run_for(50 ms), then update()"]
        net["ServerNetwork<br/>sockets, sessions, ReliableChannel per player"]
        ctx --- net
    end
    subgraph simthread["Simulation thread: game"]
        sim["Simulation<br/>MainLoop, 60 Hz fixed step"]
        game["GameInstance<br/>ServerGame, ECS world"]
        sim --- game
    end
    net -->|"Simulation::post(): join, leave, input"| sim
    sim -->|"NetworkContext::post(): events, snapshots"| net
  • Main thread: the network. r-type_server runs the NetworkContext on its main thread, in slices of 50 ms (network_slice), and calls ServerNetwork::update() between two slices for everything that is timed: login and UDP timeouts, events to resend, connections to close once their Logout has left, rate-limit windows and drop summaries. ServerNetwork (sockets, SessionTable, one net::ReliableChannel per bound player) is only touched by this thread.
  • Simulation thread: the game. Simulation runs the GameInstance on its own thread with the engine's core::MainLoop: a fixed step of 1/60 s, sleeping until each step is due, catching up at most 250 ms after a stall. The game (ServerGame, its ECS registry, the EventTracker) is only touched by this thread.
  • From the network to the game: Simulation::post(). A player whose endpoint is bound joins, a session that ends leaves, the most recent Input of a player sets its actions: each is a command, queued under a mutex and run by the simulation thread before its next tick, in order. A command is never dropped, so the queue grows instead of having a fixed capacity; there are a few per tick at most.
  • From the game to the network: NetworkContext::post(). After each tick, the simulation thread encodes the tick's events, and every third tick a snapshot, then posts a task with the bytes to the context; the main thread runs it and sends them to every bound player. The bytes are copied into the task, so the threads share nothing.
  • Shutdown. Ctrl+C or SIGTERM only sets an atomic flag; the main loop then sends Logout(Shutdown) to every player, runs the network for at most 1 second (shutdown_grace) for them to leave, and destroys GameServer, which stops the simulation thread before closing the sockets.

GameServer (server/include/rtype/game_server.hpp) wires the two: it owns the ServerNetwork and the Simulation, and turns the network's callbacks into posted commands and the simulation's ticks into posted sends.

The engine also offers net::NetworkThread, which runs a context on a thread of its own, and core::SpscQueue, a lock-free queue of fixed capacity (architecture overview). The R-Type server does not need them: its main thread has nothing else to do than the network, and the messages between its two threads are few and must never be dropped.

A tick on the server

sequenceDiagram
    participant N as Main thread (network)
    participant S as Simulation thread
    N->>S: post: Input of player 1 (most recent)
    Note over S: run the commands, then tick (16.7 ms)
    S->>S: EventTracker: what appeared, died, disappeared
    S->>N: post: the tick's events
    S->>N: post: a snapshot (every third tick)
    N->>N: ReliableChannel::send() per player, Event datagrams
    N->>N: Snapshot datagrams, one header per player
    Note over N: update(): resend the events not acknowledged within 100 ms
  • Events (Spawn, Destroy, Death, PlayerJoined, PlayerLeft) are found by EventTracker, which compares the replicated entities of the world from one tick to the next. Each player's ReliableChannel queues them and resends them until a datagram that carried them is acknowledged (RFC 5.3).
  • Snapshots are taken by take_snapshot() (server/include/rtype/world_snapshot.hpp) and split by write_snapshot() into datagrams of at most 51 entities. They are not resent: the next one replaces a lost one.
  • Acknowledgements ride in the header of every datagram, both ways, so they need no packet of their own (RFC 5.1).
  • A player who joins gets a PlayerJoined of every player already there, then a Spawn of every entity alive; the others get its PlayerJoined (RFC 4.6).

The client

The client has one network thread, the one that calls ClientSession::update(): for now r-type_client runs the NetworkContext in slices of 20 ms and calls update() between them; the game loop will do the same each frame. ClientSession (client/network/include/rtype/client_session.hpp) goes through the states of a session:

stateDiagram-v2
    [*] --> Connecting: start()
    Connecting --> LoggingIn: TCP connected, LoginRequest sent
    LoggingIn --> Binding: LoginAccepted
    Binding --> InGame: Accept
    Connecting --> Failed
    LoggingIn --> Failed: LoginRejected, no answer within 5 s
    Binding --> Failed: no Accept within 5 s
    InGame --> Lost: Logout, Disconnect, TCP closed, server silent for 5 s
    InGame --> Closed: leave()
  • Binding: Connect every 250 ms until Accept; without it within 5 seconds, the UDP port is likely blocked: Logout, and the player is told.
  • In game: an Input every tick (1/60 s), with the actions held; the client's ClientWorld applies the snapshots and events to its own ECS registry, with the order rules of RFC 4.5: an entity exists from its Spawn to its Destroy, snapshots only move it.
  • Failed is a session that never started, Lost one that ended while in game; error() says why, for the screens.

Rates

What Rate Constant
Server simulation 60 ticks per second rtype::tuning::tick_rate
Snapshots, server to each client 20 per second (every third tick) rtype::tuning::snapshot_rate, snapshot_due()
Inputs, client to server 60 per second ClientSession::input_interval
Event resend 100 ms after the last copy, until acknowledged net::ReliabilitySettings::resend_delay
Connect, while binding every 250 ms ClientSession::connect_interval
Server network slice 50 ms between two update() network_slice (server/main.cpp)
Measured tick rate in the log every 10 seconds Simulation::report_period

Each side sends at least one datagram every 50 ms (RFC 5.5): the server's snapshots and the client's inputs already do, so neither sends a separate Heartbeat in game.

Disconnect detection

Cause Server Client
TCP connection closed (process killed, crash) The session ends at once: PlayerLeft to the others, the ship is destroyed Lost: "the server closed the connection"
No valid datagram for 5 seconds Logout(TimedOut), then the session ends (udp_timeout) Lost, after a Logout over TCP (server_timeout)
Logout or Disconnect received The session ends Lost, with the server's reason
256 events not acknowledged The session is lost (max_events_in_flight): the peer receives but no longer acknowledges
No LoginRequest within 5 seconds The connection is closed (login_timeout)
A packet the server cannot handle Only its sender is kicked: Logout(ProtocolError)
Server stopped (Ctrl+C, SIGTERM) Logout(Shutdown) to every player Lost: "the server stopped"

The other players learn who left and why from PlayerLeft, then the Destroy of its ship.

Protection

  • Validation: every datagram is checked whole before anything changes, in the order of RFC 6.3 (size, header, direction, source, payload, sequence); a dropped datagram is never answered and never acknowledged.
  • Rate limit: one IP address may send 300 datagrams per second (--max-packet-rate) and hold 8 TCP connections (RateLimits); past that, its datagrams are dropped before being read. A flood of 180,000 datagrams per second leaves the simulation at 60 ticks per second, as the game runs on its own thread.
  • Logs: dropped datagrams are summed up every 10 seconds per address at most, never one line per datagram (RFC 6.7); the session token is never logged.

Where in the code

Piece Repository and path
Sockets, framing, headers, acknowledgements, reliable events aegis2d: public/aegis2d/net/
TCP messages, snapshots and events on the wire project-rtype: protocol/ (login.hpp, snapshot.hpp, events.hpp)
Sessions, sockets and datagram checks of the server project-rtype: server/include/rtype/server_network.hpp, sessions.hpp, rate_limit.hpp
Game thread and its link to the network project-rtype: server/include/rtype/simulation.hpp, game_server.hpp, game_instance.hpp
What changed in a tick, the state of the world project-rtype: server/include/rtype/event_tracker.hpp, world_snapshot.hpp
Client session and world project-rtype: client/network/include/rtype/client_session.hpp, client_world.hpp