RPC

A key component of darlingserver is remote procedure calls, or RPC, between the server and its client processes.

darlingserver RPC is implemented with Unix datagram sockets. Each managed thread has its own socket to communicate with the server. This is done so that each thread has its own address and can communicate independently with the server.

An alternative implementation would be to use Unix seq-packet sockets, which would simplify some aspects of process and thread lifetime management. However, with seq-packet sockets, the server needs to open a separate descriptor for each connection, whereas it only needs one single socket with the datagram-based implementation. Thus, to reduce server-side descriptor usage, the datagram-based implementation was chosen.

Boilerplate/Wrapper Generation Script

Because Unix datagram IPC requires a lot of boilerplate code on both sides, a script (generate-rpc-wrappers.py) is used to automatically generate this boilerplate code for both the client and server. The client can then just invoke a wrapper for each call and let the wrapper worry about sending the call and receiving the reply. Likewise, the server can focus on implementing the call and just let the helper methods take care of creating the reply.

In addition to sending and receiving message content/data, the wrappers generated by this script automatically take care of passing file descriptors back and forth using SCM_RIGHTS ancillary data. Both the client and the server can easily send any number of FDs using the wrappers.

Client Side

On the client side, the script generates wrappers for each call that provide a simple call interface and take care of all the IPC plumbing behind the scenes. The basic structure of each wrapper (also explained in System Call Emulation) is as follows:

int dserver_rpc_CALL_NAME_HERE(int arg1, char arg2 /* and arg3, arg4, etc. */, int* return_val1, char* return_val2 /* and return_val3, return_val4, etc. */);

All return value pointers are optional (NULL can be passed if you don't care about the value). As for the actual return value of the wrapper (int), that is returned by all wrapper scripts and it is actually an error code. If it is negative, it indicates an internal RPC error occurred (e.g. communication with the server was interrupted, the server died, the socket was closed, invalid reply, etc.). If it is positive, it is an error code from the server for that specific call (e.g. ESRCH if a certain process or thread wasn't found, ENOENT if a particular file wasn't found, etc.). As with most error codes, 0 indicates success.

For example, the following call entry in the RPC wrapper generator script:

('mldr_path', [
  ('buffer', 'char*', 'uint64_t'),
  ('buffer_size', 'uint64_t'),
], [
  ('length', 'uint64_t'),
]),

produces the following wrapper prototype:

int dserver_rpc_mldr_path(char* buffer, uint64_t buffer_size, uint64_t* out_length);

This function accepts a pointer to a buffer for the server to write the path to mldr into and the size of this buffer, and then it returns the actual length of the path (the full length, even if it was longer than the given buffer). The char* buffer argument is internally converted to a uint64_t for RPC so that the server can receive the full pointer no matter what pointer size the client process is using (e.g. 32-bit pointers). The server receives the pointer value and is in charge of writing to the buffer using the pointer. With the length return value, however, the value is sent as a serialized value in the RPC reply; the wrappers take care of this and the server does not receive the pointer value directly.

RPC Hooks

The code generated by the script is only half the story, however. The other half lies within the RPC hooks that the generated code requires in order to actually send and receive messages. The reason this is done this way is so that client-side RPC can be used in different environments: currently, this means it can be used in both libsystem_kernel (which operates in a Darwin environment) and mldr (which operates in a Linux environment).

The client RPC hooks provide functions for actually sending and receiving messages to and from the client socket, printing debug information in the case of an error, and retrieving process and architecture information, among other things. Additionally, they also provide environment-specific definitions for the wrapper code to use for things like msghdr, iovec, and cmsg structures and sizes as well as constants like SCM_RIGHTS and SOL_SOCKET.

Client RPC hooks are also responsible for handling S2C calls from the server. See the S2C Calls section for more information.

Interrupts/Signals

Most RPC calls are uninterruptible: once the call is started, the thread cannot be interrupted by a signal until the reply is received. This simplifies the code needed to perform the call and allows it behave much like a regular function call. This is okay because most calls don't require the server to wait or sleep (locks notwithstanding). However, some calls include long periods of waiting where it would not be okay to wait uninterruptibly. For those calls, the ALLOW_INTERRUPTIONS flag in the RPC wrapper generator script indicates that, if any syscalls return EINTR (e.g. sendmsg and recvmsg), the call should be aborted and -EINTR should be returned.

Another interrupt-related thing that the RPC wrappers handle is out-of-order replies. However, the handling of this is only enabled for a specific pair of calls (interrupt_enter and interrupt_exit). See Interrupts for more information.

Server Side

The server side portion of darlingserver RPC is more complex than that of the client side. Unlike the wrapper code generated for the client side, the code generated for the server side can be very different depending on the flags set on the call entry in the script. There are three main types of calls the server can handle: Mach traps, BSD traps, and generic server calls.

Mach and BSD Traps

Calls marked as Mach traps with XNU_TRAP_CALL in the script produce server-side code that automatically calls the corresponding Mach trap from a duct-taped context. These calls do not return any values separately; all their return values are written directly into client pointers because that's how those calls behave in the XNU kernel. Calls marked as BSD traps behave similarly, except that they do return one value: a status code that's only valid when the call succeeds; this is because BSD traps return 2 status codes: one for success and one for failure.

Generic Server Calls

All other calls are treated as generic server calls that have handlers on the C++ side. The basic structure for each handler is:

void DarlingServer::Call::PascalCaseCallNameHere::processCall() {
  /* ... */
};

The generator script creates a _body member for each call that contains the message data structure from the client. For example, for the mldr_path call we saw earlier, the _body structure contains buffer (a uint64_t) and buffer_size (another uint64_t).

Each call class also has a _thread member that is a weak pointer to the thread that made the call. See Threads, processes, and microthreads for more details on the information available in the Thread class, including how to write to client memory directly (usually into pointers received from clients in call arguments).

Finally, each call class has a _sendReply method whose prototype depends on that call's specific interface. All _sendReply methods accept the call status code as their first argument, but additional arguments (if any) depend on the call's return values as specified in the generator script. This function must be called in order for the call's reply to be sent to the client; if it is never called, the client will be left waiting forever.

For example, for the mldr_path call, the _sendReply method accepts an additional length parameter (a uint64_t). This will be sent back to the client in the reply.

Conventions

Status Code Sign Indicates Origin

As noted earlier, negative status codes from RPC calls indicate internal RPC errors (usually fatal, but sometimes not, e.g. -EINTR) whereas positive status codes indicate error codes specific to each call. As always, 0 indicates success.

Strings/Buffer Arguments

String/buffer arguments should come in pairs of arguments, one for the string/buffer address and the other for the size/length. Additionally, for cases where the server is writing into the given string/buffer, you should add a return value that indicates how much data the server wanted to write—not how much data was actually written. This is done so that clients can retry the call with a bigger string/buffer if necessary.

S2C Calls

Most RPC calls originate from the client to the server. However, there is a small number of calls that the server makes to the client instead. These calls are used to access client resources that the server cannot access directly on Linux systems.

For example, on macOS, any process with the VM (virtual memory) port for another process is able to map, unmap, and protect memory (among other things) in that other process. However, on Linux, there is no way to perform these actions on another process' virtual memory. Therefore, whenever the server needs to perform these actions in a client process, it makes an S2C call to ask the client to perform those actions on itself on the server's behalf.

Request Methods

The server can request for a client to perform an S2C call by either sending it a special reply to an ongoing call or by sending it an S2C signal. Which method is chosen depends on the server-side state of the target thread: if it is currently processing a call, the special reply method is used; otherwise, the S2C signal method is used.

The special reply method is straightforward: the client is waiting for a reply from the server for a particular call that it made. When the client receives the special reply from the server, it knows to handle this differently and execute the S2C call using the information in the reply. It then sends back the result and continues waiting for a reply to original call.

The S2C signal method sends a special POSIX real-time signal to the target thread. When it receives this signal, it knows to message the server to receive the details of the S2C call it needs to execute. Once it completes the call and sends back the result, it exits the signal handler and continues with whatever it was doing when it was interrupted.

An alternative method was considered where the client doesn't even need to be aware that an S2C call is occurring. This method uses ptrace to temporarily take over the target thread and have it execute whatever code the server needs, resuming normal execution once complete. The main problem with this method, however, is that there is no asynchronous ptrace API available; any implementation using it would require blocking an actual server thread (not microthread). As such, it was rejected in favor of the asynchronous methods described earlier.