3.7 KiB
Design
aprsc's basic design was drawn out in a pizza session in early 2008. The design goals were:
- High throughput and small enough latency
- Support for thousands of clients per server
- Support for heavy bursts of new clients (CWOP hits every 5 or 10 minutes)
- Scalability over multiple CPUs
- Low context switch overhead
- Low lock contention between threads
A modern hybrid threaded / event-driven approach was selected. All recently developed high-performance Internet servers work in this mode (some with multiple event-driven processes, some with event-driven threads). There is a small, fixed number of threads, close to the number of CPU cores on the server, so that multiple CPU cores can be utilized, but the relatively expensive context switches between a high number of threads will not cause serious overhead.
When the server is under heavy load, data transfers between threads happen in blocks of multiple data units, so that contention on mutexes and read-write locks will not block concurrent execution of the threads. Lock contention makes many multi-threaded servers effectively single-threaded and unable to utilize more than a single CPU core.
Main work is done by 1 to N worker threads. In real-world APRS-IS today, 1 worker thread is enough, but if a server was really heavily loaded with thousands of clients, 1 less than the number of CPU cores would be optimal.
A worker thread's workflow goes like this:
- Read data from connected clients
- Do initial APRS-IS packet parsing (SRCCALL>DSTCALL,PATH:DATA)
- Do Q-construct processing (,qAx in the PATH)
- Parse APRS formatted information in the DATA to extract enough details to support filtering in the outgoing / filtering phase
- Pass on received packets to the dupecheck thread for duplicate removal
- Get packets, sorted to unique and duplicate packets, from the dupecheck thread
- Send out packets to clients as instructed by the listening port's configuration and the client's filter settings
The Dupecheck thread maintains a cache of packets heard during the past 30 seconds. There is a dedicated thread for this cache, so that the worker threads do not need to compete for access to the shared resource. The thread gets packets from the worker threads, does dupe checking, and puts the unique and duplicate packets in two global ordered buffer queues. The workers then walk through those buffers and do filtering to decide which packets should be sent to which clients.
An Uplink threads initiates connections to upstream servers and reconnects them as needed. After a successful connection the socket will be passed on to a Worker thread which will proceed to exchange traffic with the remote server for the duration of the connection.
An Accept thread listens on the TCP ports for new incoming connections, does access list checks, and distributes allowed connections evenly across worker threads.
An HTTP thread runs an event-driven HTTP server (libevent2 based) to support the status page and HTTP position uploads. Since implementing nice web user interfaces in plain C is not very convenient or effective, the status page is produced using modern Web 2.0 methods. The HTTP server can only generate a dynamic JSON-encoded status file and serve static files. An empty index.html file loads a static JavaScript file, which then periodically loads the JSON status data and formats the contents of the status page within the client's browser. This approach allowed clean separation of server code (C) and web presentation (HTML5/JavaScript/jQuery/flot).
Both developers are experienced professional Unix C programmers, so the programming language was easy to select. We also had plenty of existing code that could be re-used in this project.