Introduction

tocat is a socat-inspired relay built on tokio. It connects two endpoints (sockets, files, subprocesses, stdio, etc.) and copies bytes between them in both directions. Unlike socat, connections can be described in a TOML config file with editor completion and validation, and the bytes in flight can be passed through a pipeline of plugins.

$ tocat tcp-listen:8080,fork tee,format=hex tcp:example.com:80

Everything tocat does is built out of four ideas:

• an endpoint is a scheme, a target, and a set of options, and a run has exactly two of them, a source and a sink
• a path is one direction of the relay, source to sink or sink to source, and each path is copied independently
• a stage is one instance of a plugin, on one path of one connection, with its own options and its own state
• a unit is what a stage emits between boundaries, and is what becomes one write, one datagram, or one call to the stage below

How this book is organised

• The user guide is the reference for running tocat: how a run is described, every endpoint scheme, every plugin that ships with it, and the cross-cutting subjects of buffers, progress reporting, configuration files and logging.
• The plugin API section describes the contract in the tocat-api crate, and is what you want if you are writing a plugin.
• The design section describes how tocat is put together and why, and is what you want if you are changing tocat itself.

For what is implemented and what is still missing, see the status section in the repository README.

Conventions

Command-line examples are shown as shell sessions:

$ tocat --from - --to tcp:localhost:9000

Options are given in tables. An option written key=VALUE takes a value, and one written on its own is a flag, which means true when present.

Identifiers are matched leniently everywhere: scheme names, endpoint option keys, plugin names, plugin option keys and enum values all ignore case and treat dashes and underscores as noise, so max-connections, max_connections and MaxConnections are one option. Values (paths, commands, labels, aliases) are never touched.

Sizes take binary suffixes (b, k/kb/kib, m/mb/mib, g/gb/gib), so 64k is 65536 and 1M is a mebibyte. A bare number is bytes. In a config file a size may be written as a number or as a string.

Durations take ns, us, ms, s, m, h, d, w, and may be compounded (1m30s). A bare number is seconds. The rate plugin’s interval option has its own smaller grammar, described on its page.

Installation

$ cargo install --locked tocat

From a checkout, where the crate lives in crates/tocat-cli and is named for the binary it produces:

$ cargo install --path crates/tocat-cli

The workspace pins a stable toolchain in rust-toolchain.toml, and .cargo/config.toml sets --cfg tokio_unstable for the whole build graph, which tokio-console requires.

A Nix flake is provided. nix develop gives the toolchain, the cargo tooling, mdbook and tombi, and a handful of things worth having next to a relay (socat, websocat, netcat, tcpdump, pv, hyperfine, tokio-console). nix build builds the binary.

Choosing plugins

Plugins are cargo features of the tocat crate. The default feature all-plugins turns on everything, and features are additive, so subtracting one means turning the defaults off and naming what you want back.

$ cargo install --locked tocat
# Only tee
$ cargo install --locked tocat --no-default-features --features tee
# tee and rate
$ cargo install --locked tocat --no-default-features --features tee,rate
# Everything except WASM, which is most of the build time
$ cargo install --locked tocat --no-default-features --features all-plugins-no-wasm
# No plugins at all
$ cargo install --locked tocat --no-default-features

Every plugin is a module of tocat-plugins, and a feature there enables both the module and whatever optional dependencies it needs, so a build without compress never compiles zstd and one without wasm never compiles wasmtime. The tocat crate forwards its own features to that one and never names a plugin.

A binary knows what it was built with, so tocat --list-plugins is the authoritative answer for any given install: it prints each plugin and its one line description. Naming a plugin the binary does not have is an error at startup, with a suggestion drawn from the ones it does have.

Building a guest

The wasm plugin loads modules rather than building them, so nothing above produces one. sdk/wasm/ is a CMake package that installs the guest ABI header and a tocat_add_wasm_guest() function, and builds the example guests with it. Installing it needs no compiler; building a guest needs a clang, any clang, since a guest imports nothing and has no WASI to link against.

$ cmake -S sdk/wasm -B build/wasm \
        -DCMAKE_TOOLCHAIN_FILE=$PWD/sdk/wasm/cmake/wasm32-toolchain.cmake
$ cmake --build build/wasm

Building this book

The documentation is an mdbook in docs/. mdbook is in the dev shell; otherwise cargo install mdbook.

$ mdbook serve docs --open
$ mdbook build docs

Invocation

$ tocat -h
socat-inspired relay

Usage: tocat [OPTIONS] [SPEC]...

Arguments:
  [SPEC]...  SOURCE [PLUGIN ...] SINK. The outer specs are endpoints; anything between them is a pipeline entry. Slots already filled by --from/--to are skipped.

Options:
  -c, --config <PATH>       Configuration file to use.
      --no-config           Disable configuration file merging.
      --dump-config         Render the final configuration as TOML.
  -b, --buffer-size <SIZE>  Bytes per copy, e.g. 65536 or 256KiB. One buffer per direction per connection.
  -f, --from <ADDR>         Source endpoint. Fills the first positional slot.
  -t, --to <ADDR>           Sink endpoint. Fills the last positional slot.
  -p, --plugin <SPEC>       Pipeline entry: NAME[:DIRECTION][,key=value...]. Repeatable, applied in order.
      --no-plugins          Ignore plugins declared in the configuration file.
      --list-plugins        List the plugins compiled into this binary and exit.
  -P, --progress[=<WHEN>]   Draw a progress line on stderr. Bare, or 'auto', draws only when stderr is a terminal; 'always' draws regardless. [possible values: never, auto, always]
  -v, --verbose...          Simple verbosity level.
      --log-level <LEVEL>   Explicit verbosity level. [possible values: off, error, warn, info, debug, trace]
  -h, --help                Print help (see more with '--help')
  -V, --version             Print version

tocat can be used similarly to socat, with the source and sink endpoints given as strings. You can also use the explicit --from and --to flags, or a configuration file.

$ tocat --from - --to tcp:localhost:9000
$ tocat - tcp:localhost:9000

Positional slots

There are exactly two endpoint slots, and roles are decided by position, never by looking at the text. --from and --to fill the first and last slots; the outer positional arguments fill whichever of those slots are still open; whatever remains in the middle is the pipeline.

$ tocat SRC SINK                  # no plugins
$ tocat SRC tee compress SINK     # two entries, in that order
$ tocat -f SRC tee SINK           # one entry; SINK fills the open slot
$ tocat -f SRC -t SINK tee        # one entry; both slots already filled

A lone positional with both slots open fills the source, which is what the older two-positional form did. When both endpoints come from flags or from a config file, -p is the way to add a stage: a bare positional would be read as an endpoint.

$ tocat -f tcp-listen:8080,fork -t tcp:example.com:80 -p 'tee,format=hex'

Quoting matters at the shell, not to tocat: an entry containing spaces or shell metacharacters ('process,command=gzip -c', "system:grep -v DEBUG | sort -u") has to reach tocat as one argument.

Flags that answer questions

These exit before any endpoint is opened.

-v (repeatable: once for debug, twice or more for trace) and --log-level conflict with each other; --config and --no-plugins conflict with --no-config. Running with -v logs the resolved chains for both directions, which is the quickest way to confirm a long command line came out in the order you meant.

Stopping

The first SIGINT or SIGTERM stops accepting new connections and drains what is in flight; a second exits immediately with status 130. A relay that finishes normally exits 0, and one that fails to start or fails mid-transfer logs the error and exits 1.

Draining includes the pipeline. The signal reaches a plugin as end of stream, so a stage holding buffered bytes still emits them and a stage that writes an epilogue still writes it: interrupting hash prints its digest, interrupting compress closes the frame, and a block stage flushes its partial tail. Wait for the exit rather than sending a second signal if you want that output.

One case cannot be interrupted promptly: the synchronous copy path (chosen when there are no plugins and both endpoints are file, pipe or stdio) checks for shutdown between chunks, so a read that never returns, such as a FIFO with no writer, is left behind after a short grace period rather than waited on forever.

Endpoints

The fundamental unit of tocat is the endpoint. An endpoint is a scheme, a target, and a set of options.

scheme:target,option,option=value

Bare options mean true, so fork and fork=true are equivalent. Schemes and options may have various aliases. If an option is specified multiple times, the last instance takes precedence. Options belong to the scheme that documents them: anything else is an error rather than being ignored, so tcp:80,append is rejected. Spelling is forgiving. Case is ignored and dashes and underscores are removed for schemes and option keys. Values remain untouched.

Schemes

Two properties in that table decide how the rest of tocat behaves around an endpoint.

Duplex or one way. A duplex endpoint can be both read and written, so a run with duplex endpoints on both sides relays in both directions. A one-way endpoint is read when it is the source and written when it is the sink, and the opposite path has nothing to carry.

That second half is what makes a one-way transfer end. tocat file:in.bin tcp:host:9000 exits at end of file because the reverse direction does not exist rather than because it finished: a direction with no half to read is skipped outright, so nothing is left waiting on the peer. A duplex endpoint has no such exit, which is why file: opens by role even where the path underneath it could be opened both ways.

Bytes or datagrams. A byte endpoint carries a stream, and a chunk is an arbitrary slice of it. A datagram endpoint carries messages, and the boundaries are part of the data. That distinction is what the datagram rules in the pipeline are about. It is about what the endpoint carries rather than how it connects: unix-seqpacket accepts connections and has an end of stream like tcp-listen, and is still a datagram endpoint, because a message it delivers is one the peer sent whole.

Layers, which stack over a transport

An endpoint is a transport and, optionally, handshakes stacked over it. A transport opens a connection; a layer takes one and returns another. tls is the first of them, and noise, proxy CONNECT, SOCKS and WebSocket are the same shape.

The command line spells a common stack as a single scheme, so tls:host:443 is a TCP transport with one TLS layer. A config file can write either that or the composed form, and --dump-config prints the composed one:

[sink]
type = "tcp"
addr = "backend:443"

[[sink.layers]]
type = "tls"
cafile = "/etc/ca.pem"

Options on a sugared scheme go to whichever of the two wants them, with the layer asked first. There is no ambiguity today, since the only key both could take is name=, which layers refuse: an endpoint has one name, not one per level.

A layer decides what the endpoint carries. TLS fuses message boundaries, so an endpoint with a TLS layer is a byte stream whatever the transport underneath it was. That is why only the stream transports accept layers at all, and why a layer over udp: is refused before anything is opened rather than discovered later.

Socket options, which the socket schemes share

tcp, tcp-listen, unix, unix-listen, udp and udp-listen take the options below, as far as their kind of socket has them. An option a scheme cannot honour is an error rather than a no-op, so unix:/tmp/s,nagle=false is refused: Nagle’s algorithm is a TCP thing.

socat’s spellings are accepted where they differ: nodelay (which is nagle inverted), rcvbuf, sndbuf, keepintvl, keepcnt, so-reuseaddr and so-linger.

$ tocat 'tcp-listen:9000,fork,backlog=64,nagle=false' 'tcp:backend:80,keepalive=30s'

recv-buffer is not buffer-size. The kernel buffer is how much the operating system will hold for this socket; buffer-size is how much tocat copies at a time and, on a datagram endpoint, the largest message it can carry. Sizing the wrong one is the usual mistake.

On a listener the options belong to each accepted connection, not to the listening socket, so under fork every client gets them. reuseaddr and the buffer sizes are the exceptions: they have to be set before the bind, so they apply to the listening socket itself.

Resilience, on the schemes that can be reopened

tcp, unix, unix-seqpacket, exec and system can be opened again, so they take the options below. A listening scheme does not: its answer to a peer that went away is fork, which keeps accepting, and a bind that fails is a configuration error rather than something to wait out. file: does not either, because reopening one raises a question about the offset that has no good answer; a FIFO whose writer comes and goes is pipe: with hold.

# Wait for a backend that has not started yet
$ tocat tcp-listen:9000 'tcp:backend:8080,retry=forever,interval=2s,connect-timeout=5s'

Only a failure counts. A peer that closes has said it is finished, so end of stream ends the run under every mode. Without that rule an ordinary relay would be impossible to end.

connect-timeout is worth setting whenever retry is. An address that blackholes packets leaves the connect waiting on the kernel for minutes, and retrying cannot help because the first attempt never finishes.

interval and reconnect-delay answer different questions. The first waits between connects that failed, the second between connections that worked and then did not. A peer that accepts and immediately resets is the case the second one exists for, and it is why the default is not zero.

The three reconnect modes

none is the default and ends the run, which is what a relay has always done.

restart reopens both endpoints and starts over with fresh plugin instances, as though the command had been rerun. The old path gets its end of stream first, so a stage holding bytes hands them over before it is replaced. Because both ends reopen, a listening source drops its current client and accepts the next one. It cannot be combined with fork: there every connection is already independent, and restarting the run would drop the ones still working.

keep reopens only the endpoint that failed, underneath the pipeline, so the stages carry on with the state they had and the other side of the relay never learns anything happened. It needs an endpoint that opens a two-way stream and is refused on anything else.

A keep endpoint does not end on its own. The direction reading it is reading a stream that reopens itself, so a peer that never closes cleanly means a relay that never finishes. That is correct for a reconnecting endpoint and it is also a way to leave something running forever by accident. Bound it with a timeout stage on both directions if the peer cannot be relied on to say when it is done.

keep preserves stage state, not the stream. Bytes the kernel had accepted but not delivered are gone, and a write interrupted halfway leaves an unknowable prefix on the wire. A reconnect is a hole. Pair it with a stage that needs message boundaries, unframe for instance, and that stage will resync on whatever follows the hole. restart, which gives every stage a clean start, is the safer default when the pipeline carries framing.

A relay that reconnects logs each attempt at warn level, so one waiting for a server to come up says so rather than looking hung.

What is not here yet

Under fork there is no per-connection restart. reconnect=restart is refused with fork because restarting reopens the whole relay, which would drop every other connection being served; reconnect=keep does work, since each connection reopens its own endpoint.

name, which every scheme takes

Every scheme accepts name=TEXT, which replaces the label the endpoint is known by. Labels appear in log records, in tee headers, and in the upstream -> downstream description a stage is given at build time.

$ tocat 'tcp-listen:9000,fork,name=frontend' 'tcp:10.0.0.5:8080,name=backend'

Four schemes keep their own label even when name is given, because the target is the identity: file: shows its path, and exec:, system: and pty-exec: show the command line.

Default labels are tcp://addr, udp://addr, unix://path, unix-seqpacket://path, unix-dgram://path, pipe://path, file://path, EXEC(argv), SYSTEM(command) and STDIO. Under fork the accepted peer is appended to the listening side’s label, so a dump or a log line says which connection it belongs to.

exec and system

Both run a child process with its stdin and stdout wired to the relay. They differ only in how the command line is interpreted.

exec - subprocesses

Runs a program with its stdin and stdout wired to the relay. The target is split on whitespace and passed directly to the program. This is not a shell, so no globbing, quoting, or metacharacters. The child’s stderr is inherited, so its diagnostics go to your terminal rather than into the relayed data.

$ tocat tcp-listen:9000,fork "exec:/usr/bin/env cat"

system - shell commands

Runs the given string through $SHELL -c (or sh -c when SHELL is unset), so pipes, redirection, globbing, and variable expansion all work. Anything the string contains runs with tocat’s privileges. Don’t use system with a command built from untrusted input, or in a config file others can write.

$ tocat tcp-listen:9000,fork "system:grep -v DEBUG | sort -u"

Both

Both also take the resilience options, since a child that fails to spawn can be spawned again. What they cannot do is give a respawned child the state the last one had: reconnect=keep preserves plugin state, not the process’s. A respawned cat is fine; a respawned stateful filter starts over mid-stream.

The child is killed when the relay drops the connection, and is reaped in the background: the relay ends when the pipes close rather than when the process does, and a non-zero exit is logged as a warning. That is the opposite of the process plugin, where a bad exit fails the direction, because there the child’s output is mid-pipeline and being wrong matters.

The pipes to and from the child are enlarged to match the copy buffer where the platform allows it, see Buffers.

To put a filter between the endpoints rather than at one end, use process, which has the same two ways of naming a child and adds control over its stderr.

For a child that needs a terminal rather than a pair of pipes, use pty-exec. Anything interactive wants it, and so does anything whose output never appears until it exits, which is libc block-buffering a pipe.

file

Alias: open. Files are unidirectional. They are read when they are the source and written to when they are the sink, so the writing options are simply not consulted on the source side, where the file is opened read-only.

Being unidirectional is what makes a transfer end. tocat file:in.bin tcp:host:9000 exits at end of file because the reverse direction does not exist, so nothing is left waiting on the peer. Opening the file both ways would keep the relay alive indefinitely, which is why a device that genuinely needs both directions gets its own scheme rather than a flag here: see tty.

$ tocat file:/tmp/payload tcp:localhost:9000
$ tocat tcp-listen:9000 file:/tmp/capture,truncate

Devices

A block device or a plain character device is a file that happens to live in /dev, so it belongs here rather than in a scheme of its own. Two options make that safe and useful.

device asserts the path already is one. Without it, create is on, so a wrong path or an unplugged adapter turns into a regular file of that name the moment it is used as a sink. As root that is a new file in /dev. With it, the path is checked before anything is opened, and the error tells you whether it was missing or was the wrong kind of thing.

$ tocat file:/dev/sda,device file:disk.img,truncate

device contradicts create, truncate and append, and asking for both is an error rather than a silent override: a device has no length to change, and nothing to create.

seek=SIZE starts somewhere other than the beginning, reading from that offset as the source and writing to it as the sink. It takes the usual size suffixes, so seek=1MiB is a mebibyte in.

$ tocat file:/dev/sda,device,seek=1MiB file:partition.img,truncate

An offset into a block device that is not a multiple of its block size is warned about rather than refused, since the kernel will not complain and the symptom is a shifted image rather than an error. seek contradicts append, where every write goes to the end whatever the offset says.

A terminal is the exception to all of this. It is duplex on one descriptor, which this scheme cannot produce, and its settings have to be restored afterwards, which this scheme has nowhere to keep: see tty.

FIFOs

file: pointed at a FIFO works, but the open blocks until a peer appears and the stream ends when the last writer leaves. That is a legitimate thing to want, so it warns rather than refusing. See pipe for the version that outlives its producers.

Two other things follow from being a file:

• A regular file as the source is the one case where tocat knows the size of the transfer up front, which is what lets --progress draw a bar, a percentage and an ETA rather than only counts.
• With no plugins declared and the other endpoint also being file:, pipe: or stdio:, the relay takes its synchronous copy path. See The data path.

noise

Noise is a layer, like tls: a TCP transport with a handshake stacked over it, and a byte stream at the top. Unlike TLS it has no certificates and no names. A Noise peer is a key, and everything here is about which keys the two sides have and when.

Reach for it when TLS is more machinery than the job needs: two ends you control, no public certificate authority in the picture, a shared secret or a pair of keys you can distribute yourself.

The two schemes

$ tocat - noise:gateway.internal:7000,pattern=nnpsk0,psk-file=/etc/tocat/psk
$ tocat 'noise-listen:7000,fork,pattern=nnpsk0,psk-file=/etc/tocat/psk' tcp:localhost:9000

Both take the socket options of the TCP transport underneath.

Pick a pattern

pattern= is required, and it is the only decision that really matters. There is no default, because the only pattern that works with no key material is the one that proves nothing.

A Noise pattern is two letters. The first is what the dialling side does with a static key, the second what the listening side does:

All twelve combinations work: nn, nk, nx, kn, kk, kx, xn, xk, xx, in, ik, ix.

Two rules follow from the letters, and they are the whole of the configuration:

• You need key when your own letter is K, X or I.
• You need peer when the other side’s letter is K, because a K key is never sent and so has to be there already.

Where the other side’s letter is X or I its key arrives during the handshake, so peer is optional there. It is a pin, and setting it is what turns the handshake from opportunistic into authenticated. Without one, any peer that answers is accepted.

Which to choose

nnpsk0 if you have one secret to share. Nothing else to distribute, both sides authenticated. The smallest thing that is actually secure.

xx if each machine should have its own key. Both sides send their static key during the handshake, and each names the other’s with peer=. Nothing has to be known in advance, which makes it the easiest of the key-based patterns to deploy.

ik when the dialling side already knows who it is calling. One round trip shorter than xx, and the dialling side’s own key stays hidden from anyone watching. The dialling side must set peer=.

nk for a server with a published key and anonymous clients. The client proves the server and stays unidentified itself.

nn proves nothing. A listener cannot read the traffic; anything in the path can read it, change it, and impersonate either end. tocat logs a warning naming any run whose configuration authenticates nobody, which includes nn and also xx or ix left without a peer=. It exists for testing and for talking to something that offers nothing better.

Mixing in a shared secret

Any pattern takes a pre-shared key modifier: append psk0, psk1, psk2 or psk3 to its name, as in nnpsk0, xxpsk3 or ikpsk1. The number is which handshake message the secret is folded into, and both sides must use the same one.

$ tocat - noise:gateway:7000,pattern=xxpsk2,key-file=id,peer=$THEIRS,psk-file=secret

Lower numbers protect more of the handshake; psk0 mixes the secret in before anything else is sent. Most patterns have two messages, so psk3 only works with xn, xk and xx, which have three. Using it elsewhere is refused before anything connects.

A psk is not a replacement for peer= in a pattern that transmits a static key, but it does authenticate both sides on its own, which is why nnpsk0 needs no keys at all.

Cipher and hash

Both ends must be set the same. Noise negotiates nothing, so a mismatch is a failed handshake, not a fallback. The defaults are what WireGuard uses and are a good choice unless something you are talking to needs otherwise.

The Diffie-Hellman function is always 25519.

Keys

Every key here is 32 bytes. Generate one with any tool that emits random bytes:

$ head -c 32 /dev/urandom | xxd -p -c 32 > /etc/tocat/psk
$ chmod 600 /etc/tocat/psk

The three sources for one key are alternatives, not a list, and giving two is an error rather than a silent preference.

Prefer psk-file and key-file to psk and key. A key on the command line is visible to anything that can read the process list. Whitespace is ignored when decoding, so a file with a trailing newline reads as what it looks like.

For any pattern with a K, X or I you need the public half to give the other side. tocat does not derive it for you today, so generate the pair with a tool that prints both.

Pinning

Where the other side’s letter is X or I, peer= is what turns encryption into authentication. Without it the handshake succeeds with whoever answers, exactly like tls with verify=none. With it, a peer presenting any other key is refused once the handshake completes and the connection dropped.

Where the other side’s letter is K, peer= is required rather than optional, because that key is never sent and the handshake cannot be built without it.

Rotating the key mid-connection

rekey=N replaces each direction’s cipher key after every N records, which bounds how much data sits under any one key. Both sides count the same records, so nothing is negotiated and nothing is sent; set the same number on both ends.

$ tocat - noise:gateway:7000,pattern=nnpsk0,psk-file=/etc/tocat/psk,rekey=4096

Leave it unset unless you have a reason. A session key already changes on every connection, and the nonce is a 64 bit counter that no transfer will exhaust.

Binding a session to something else

prologue= mixes text into the handshake. Both ends must write the same thing or the handshake fails, which is a cheap way to make a key usable only in the context you meant:

$ tocat - noise:gateway:7000,pattern=nnpsk0,psk-file=/etc/tocat/psk,prologue=metrics-v2

What this is not

It is not the encrypt plugin, and neither replaces the other. encrypt transforms bytes with a key you already hold and needs no peer at all, which is why it can encrypt a file into another file or into a pipe. noise needs a live peer running the same handshake, and in exchange derives a fresh key for every connection and can prove who the peer is. Use encrypt for data going somewhere to be read later, and noise for a connection between two processes.

It is TCP only. There is no noise over unix sockets or a subprocess today, the same as for tls.

There is no close notification. A peer that hangs up cleanly and a connection someone cut look the same. A record cut in half is caught and reported; a cut between records is not distinguishable from the peer finishing. Anything that needs to know a transfer was complete has to say so itself, above tocat.

It is a byte stream, not messages. A Noise record holds whatever happened to be in the buffer, so an endpoint with a noise layer is a stream endpoint like tls, not a message one like ws.

pipe

Alias: fifo. Unidirectional like file, but a rendezvous rather than storage: read as the source, written as the sink.

By default tocat holds the FIFO open read-write, making itself a writer. Two things follow: opening never blocks, and the stream never ends, so producers can come and go without the relay noticing. That is the difference from file: pointed at the same path, which relays one producer and exits. POSIX leaves read-write on a FIFO undefined, every platform tocat targets implements it, and there is no other way to hold one open.

$ tocat pipe:/tmp/events,create,mode=660 tcp:collector:9000
$ myapp >> /tmp/events          # restart this all day

Both halves come from the same descriptor but are exposed half-duplex on purpose: a held FIFO is readable and writable, so treating it as duplex would feed tocat’s own writes back into its reader.

Two tocats sharing a FIFO chains relays with different pipelines, and a FIFO on each side makes a test harness you can drive entirely from a shell:

$ tocat pipe:/tmp/in,create redact pipe:/tmp/out,create &
$ cat /tmp/out &
$ echo 'Authorization: Bearer abc123' > /tmp/in

Because a held FIFO never reaches end of stream, it behaves like a datagram source in one respect: end-of-stream work such as a rate summary or a final short block never runs. Use hold=false when you want the relay to finish with its producer.

proxy and socks5

Two ways to reach a peer through something else. Both are layers, both dial the proxy as their transport and carry the real destination themselves, and both are transparent once the tunnel is open: after the handshake the bytes are the target’s.

$ tocat - proxy:proxy.corp:8080:api.example.com:443
$ tocat - socks5:127.0.0.1:1080:api.example.com:443

The address is proxyhost:port:targethost:port. The first two components are the transport, which is what a scheme’s address has always meant here, and everything after is the target.

Naming more than one credential source is an error rather than a precedence rule. Both layers are client only: a listening endpoint has nobody to ask for a tunnel, and saying so is a parse error rather than a surprise at connect time.

Which one

proxy is HTTP CONNECT: a request, a status line, and then the connection is yours. It works through anything that fronts an HTTP proxy, which in a corporate network is usually what exists.

socks5 sends the target as a name unless you set resolve-locally, so the proxy does the lookup. That is the reason to prefer it where both are available: a host only the proxy’s network can resolve still works, and no lookup for it happens on your side. It also carries the reply codes, so a refusal arrives as “not allowed by ruleset” rather than as a dropped connection.

SOCKS4a is not supported. Everything that speaks it speaks SOCKS5.

Stacking TLS over a tunnel

This is the arrangement the layer model exists for, and the one that has to get the name right: above the tunnel you are talking to the target, so the certificate is checked against the target and not against the proxy.

There is no single scheme for it, because sugar spells one layer. Write the stack:

[sink]
type = "tcp"
addr = "127.0.0.1:1080"

[[sink.layers]]
type = "socks5"
target = "api.example.com:443"

[[sink.layers]]
type = "tls"

Layers are bottom first, so this reads as: dial the proxy, ask it for a route to api.example.com:443, then handshake TLS with api.example.com through it.

Two things that look like bugs

curl -x will not work against a tocat relay. -x tells curl the address is an HTTP proxy, so it sends CONNECT and waits for a 200. A relay answers nothing: it carries bytes to an endpoint that is already decided. If your sink is api.example.com:443, point curl at the relay directly and let it speak the protocol the far side expects.

A name based virtual host needs the name in the request. A relay does not rewrite anything, so a request arriving with Host: 127.0.0.1:12345 reaches the target with that header and gets whatever the target does with an unknown host, often a 404. Send the name you mean:

$ curl -H 'Host: api.example.com' http://127.0.0.1:12345

What is not here yet

proxy offers only basic authentication; digest and NTLM are not supported. socks5 does not do UDP association or BIND, so it tunnels outbound TCP and nothing else. Neither layer can accept a connection, so tocat cannot be a proxy, only use one.

pty and pty-exec

Both allocate a pseudo-terminal and relay its master side. They differ in what ends up on the other end: pty-exec spawns a child there, pty publishes it as a device path and waits for something else to open it.

pty-exec - a command on a terminal

Like exec, but the child gets a controlling terminal instead of a pair of pipes.

$ tocat tcp-listen:9000,fork pty-exec:bash

Three things change when a program runs on a terminal, and one of them is usually why you are here:

• libc line-buffers stdout instead of block-buffering it, so output arrives a line at a time rather than in 4KiB blocks. If you have piped a program through exec and watched nothing come out until it exited, this is the fix.
• The kernel supplies a line discipline, so line editing, ^C and job control work.
• Programs that check for a terminal and refuse, or fall back to a degraded mode, run properly.

The pty is the child’s stdin, stdout and stderr. There is no second descriptor to inherit, so unlike exec the child’s diagnostics are relayed along with everything else. On a terminal that is what a terminal shows.

With shell, the target is handed to $SHELL -c whole rather than split on whitespace, which is pty-exec playing the part system plays for exec:

$ tocat tcp-listen:9000 "pty-exec:tail -f /var/log/syslog | grep -i oom,shell"

pty - a device for something else to open

Allocates a pty, relays the master, and spawns nothing. The target is a symlink to create pointing at the slave device, which is how another program finds it:

$ tocat pty:/tmp/ttyfake tcp:10.0.0.5:4001

Anything that wants a serial port can now be pointed at /tmp/ttyfake, and its traffic goes over TCP. The symlink is removed when the relay exits, and a dangling one left by a previous run is replaced. A link whose target still exists is not, since that would steal a path something else is using.

The target is optional. Without it the allocated device is logged and nothing is linked, which is fine at a prompt and useless from a script.

Terminal settings, which both take

raw is on by default and you should usually leave it. A pty in cooked mode is not a transparent pipe: it translates carriage returns, turns ^C into a signal rather than a byte, and refuses a line longer than 4096 bytes. Relaying anything but text through that is corruption. Turn it off with raw=false when you want the line discipline, which mostly means when a human is typing at the other end and nothing else is providing line editing.

echo is off because a relay that echoes writes back at its own reader is a loop rather than a feature. Turn it on when the far end is a person who expects to see what they type and no local terminal is doing it for them.

size is unset because there is no correct value to guess. A fresh pty reports 0x0, which most full-screen programs read as “no idea” and a few read as a terminal one column wide. Set it if you are running something full-screen:

$ tocat tcp-listen:9000,fork "pty-exec:htop,size=40x120,term=xterm-256color"

The size is fixed for the life of the connection. Nothing relays a terminal resize, because there is no channel in a byte stream to carry one.

Which to reach for

To open a terminal that already exists rather than allocating one, use tty.

Use exec when you want a filter: it is cheaper, and a pty’s line discipline is a liability when nothing needs it. Use pty-exec when the program behaves differently on a terminal, which is most interactive ones. Use pty when the thing you are connecting expects to open a device rather than be spawned.

To put a program between the endpoints rather than at one end, use the process plugin. It has no pty option: a stage sits mid-pipeline where a line discipline rewriting the payload would be a bug.

stdio

Also spelled -. Note that tocat’s stdout carries relayed data when stdio is the sink. Logs and dumps go to stderr for this reason, and tocat refuses to open stdout as a dump target at all.

$ dd if=/dev/zero | tocat - tcp:localhost:9000

Both descriptors are resized when they turn out to be pipes. Nobody declares them as such, but tocat … | pv makes fd 1 one, and its 64 KiB default would otherwise cap every write. See Buffers.

On the synchronous path tocat reads and writes the descriptors directly rather than going through std::io::stdin/stdout: the standard handles bring a LineWriter that scans every byte for newlines and an 8 KiB BufReader, both of which are wrong for binary payload.

Three things contend for stderr: log records, a tee pointed there, and the --progress line. Logs and the progress line are coordinated under one lock; a stderr dump is not, and tocat warns about that combination.

tcp and tcp-listen

tcp - connect to a TCP socket

Aliases: tcp-connect, connect. The target is a host:port passed to the resolver, so a name or a bracketed IPv6 literal both work.

$ tocat - tcp:example.com:80
$ tocat - tcp:[::1]:9000

Plus the shared socket options and the resilience options, since a dialled socket is one that can be dialled again. Naming a local address or setting a pre-connect option means the socket has to exist before the connect, so one resolved address is dialled rather than each in turn.

tcp-listen - accept inbound TCP connections

Aliases: tcplisten, listen. The target is host:port, host, port or nothing, and what is missing defaults to 127.0.0.1:8000. Loopback rather than the wildcard is deliberate: exposing a relay to the network should be something you asked for.

$ tocat tcp-listen:9000 -
$ tocat tcp-listen:0.0.0.0:9000,fork tcp:localhost:8080

Plus the shared socket options, which apply to every accepted connection.

Either endpoint may be the listening one: tocat tcp:backend:80 tcp-listen:9000,fork forks on the sink and dials the source for each accepted client.

Under fork everything stateful is per connection: a buffer per direction, a fresh instance of every pipeline stage, and its own tick schedules. Only the side channels a stage writes to (a tee file, a rate sample file) are shared. That multiplies, so see the notes on cost in rate and Buffers before pairing a large buffer with a high ceiling.

The accepted peer’s address is appended to the listening endpoint’s label for the life of that connection, which is what keeps one shared hex dump readable.

tty

Opens a terminal that already exists: a /dev/pts/N somebody else made, or a serial device. pty allocates one; this one borrows.

$ tocat tty:/dev/ttyUSB0,b115200 tcp-listen:9000,fork

That is a serial console reachable over the network, with the line settings on the endpoint rather than arranged with stty beforehand and hoped to survive.

Why not file:

file: opens a device happily, and gets three things wrong for a terminal.

It is half-duplex by role, so file:/dev/pts/5 writes to the terminal and nothing comes back. A terminal is duplex on one descriptor, and file: has no spelling that opens both ways.

It opens without O_NOCTTY, so a relay that is a session leader with no controlling terminal acquires one, and signals generated by that terminal start arriving at tocat.

It applies no line settings, so the device keeps whatever it had. A serial port at the wrong speed reads as noise rather than as an error, and a terminal left in cooked mode translates carriage returns, turns ^C into a signal instead of a byte, and refuses a line over 4096 bytes.

What gets put back

A borrowed device outlives the relay. Leaving /dev/ttyUSB0 in raw mode changes what the next program to open it sees, and doing it to the terminal you launched from leaves your shell unusable, so the settings found at open are restored when the transfer ends. A pty needs none of that, because it is discarded rather than returned.

Restoration is best effort by necessity, since it runs during teardown. A failure is logged rather than raised, because there is nothing useful left to raise it to.

Terminal settings

The same three pty takes, with the same defaults and the same reasons.

Line settings

These are the ones a serial cable needs and a pts ignores.

clocal defaults on, and turning it off looks like a hang. Without it the open waits for carrier detect, which a three-wire cable never asserts. Turn it off when you actually have modem control and want to know when the far end goes away.

exclusive is worth setting on anything long-running. Without it a second program can open the same device and the two of you interleave reads, each getting an arbitrary share of the bytes, which looks like corruption rather than contention.

When the transfer ends

It does not, on its own. Both directions run, and a terminal has no half-close the way a socket does, so there is nothing for one side to shut down when it runs out. A tty: relay ends on a signal, or when the endpoint at the other end closes both ways.

That is the trade for duplex, and it is why file: stays half-duplex by role: tocat file:in.bin tcp:host:9000 exits at end of file precisely because the reverse direction does not exist to wait on.

serial as an alias

serial:/dev/ttyUSB0,b9600 is the same scheme under a name that says which of the two cases you are in. The behaviour is identical; the aliases exist because tty: reads oddly for a USB adapter and serial: reads oddly for a /dev/pts.

tun and tap

These open one of the kernel’s virtual network interfaces and hand the relay its packets, so you can put an interface on one side of a tocat pipeline and something else on the other.

$ tocat tun:tocat0 udp:relay.example:5555

The body is the interface name. Leave it off and the kernel picks one, which is then logged so you know what to configure.

These are message endpoints

One read is one packet and one write is one packet, the same as udp:. Three things follow, and they are the ones that catch people out:

• A packet longer than the copy buffer is truncated, not split. tocat reads the interface MTU when it opens and refuses to start if it is larger than the buffer, naming both numbers, so this fails loudly instead of quietly losing the end of every large packet. Raise --buffer-size when you raise the MTU.
• The other side has to keep boundaries too. Relaying to udp: or unix-dgram: works. Relaying to tcp: needs a plugin that frames, or the packets run together into a stream nobody can take apart again.
• Layers are refused. Neither tls nor noise keeps message boundaries, so a layer over an interface is rejected before anything opens.

Running without root

Creating an interface needs CAP_NET_ADMIN. Attaching to one that already exists does not, so the usual arrangement is that root makes the interface once and the relay runs as an ordinary user:

# ip tuntap add mode tap user alice name tap0
# ip addr add 10.99.0.1/24 dev tap0
# ip link set tap0 up

$ tocat tap:tap0 udp:peer.example:5555

Nothing in that second command needs privilege. tocat does not bring the interface up, set an address, or change the MTU unless you ask it to, which is what keeps the unprivileged case working.

Note the lifetimes differ. An interface tocat creates itself is not persistent and disappears when the relay ends. One made with ip tuntap add stays, together with its addresses and routes, which is usually what you want for anything that restarts.

Options

Everything but name is unset by default, because each of the others needs privilege the attach case does not have. packet-info is off to match ip tuntap add without its pi flag, and the two ends have to agree.

up has three states rather than two, and the spellings are up, up=true and up=false as everywhere else in tocat. Leaving it out means tocat does not touch the interface state at all, which is what lets an attach run unprivileged. That is not the same as up=false, which brings the interface down and needs the same privilege as bringing it up.

Addressing an interface tocat creates

ipv4= and ipv6= take a CIDR:

$ tocat tun:t0,ipv4=10.10.0.1/24 udp:peer.example:5555

This is mainly for an interface tocat creates rather than attaches to, and the reason is the lifetime difference above. A created interface does not exist until tocat opens it and is gone when tocat exits, so there is no moment at which ip addr add could reach it: before is too early, and after is a race against your own relay. These options are the only way to address one.

For an interface you made with ip tuntap add, address it there instead. It needs privilege either way, and doing it once outside the relay means the address survives a restart.

Setting an address implies up, since an address on a down interface does nothing. Add up=false if you want the address without it.

Pinning a tap’s hardware address

A tap is given a random hardware address when it is created, so an interface tocat creates has a different one every run. Anything keyed on it stops matching after a restart: a DHCP reservation, a firewall rule, a bridge forwarding entry, a peer’s ARP cache. mac= pins it.

$ tocat tap:tap0,mac=02:00:00:00:00:01,ipv4=10.10.0.1/24 udp:peer.example:5555

Either separator works, so 02-00-00-00-00-01 is the same address. The low bit of the first octet has to be clear, because that bit marks a multicast address and an interface cannot be given one; tocat says so rather than letting the kernel report it as “cannot assign requested address”, which reads like a conflict. Setting the second bit, as in the 02: above, marks the address as locally administered, which is what you want for one you invented rather than one from a vendor’s range.

tun: has no such option. A tun interface is point to point and has no hardware address, so the option is refused rather than accepted and ignored.

When it will not open

“An existing interface of the other layer is rejected by the kernel.” You pointed tun: at a TAP interface or the reverse. The layer is fixed when the interface is created and cannot be changed by opening it differently.

A permission error. Either the interface does not exist and creating it needs privilege, or you asked for up or mtu= and those need privilege even on an interface you can otherwise open. Create it in advance with ip tuntap add and drop those options.

Note that up is not a switch between “bring it up” and “bring it down”. Without it tocat does not touch the interface state at all, which is what lets an attach work unprivileged; an interface someone else already brought up stays up.

A packet arrives with four bytes of prefix, or is rejected as malformed. packet-info disagrees with how the interface was made.

A worked example

Two hosts, an Ethernet segment bridged over UDP:

# on each host
# ip tuntap add mode tap user alice name tap0
# ip link set tap0 up

# alice, on host A
$ tocat tap:tap0 udp:hostB:5555

# alice, on host B
$ tocat tap:tap0 'udp-listen:5555'

Frames that reach tap0 on either host come out of tap0 on the other. Add noise and it would be refused, because a layer cannot sit on a message endpoint; encrypt the UDP side instead, or put the encrypt plugin in the pipeline where boundaries are preserved.

tls

TLS is a layer rather than a plugin. A plugin stage is synchronous and pure and has no way to send bytes back upstream, and a handshake is a conversation with the peer, so it belongs to the endpoint. tls:example.com:443 is a TCP transport with one TLS layer over it, and --dump-config prints that composed form.

tls - dial a TLS server

Aliases: ssl, openssl. The target is the same host:port that tcp takes, and the transport options work as they do there.

$ tocat - tls:example.com:443
$ tocat tcp-listen:8080,fork 'tls:backend.internal:443,cafile=/etc/ca.pem'

Plus the socket options and the resilience options of the TCP transport underneath, except reconnect=keep, which is refused under a layer.

tls-listen - accept TLS connections

$ tocat 'tls-listen:8443,fork,cert=/etc/cert.pem,keyfile=/etc/key.pem' tcp:localhost:8080

Plus everything tcp-listen takes, including fork.

Mutual TLS

Both sides can present a certificate and both can check one. The same four keys do it, read from whichever side they are on:

• cert= and keyfile= are the identity this side presents. Required on tls-listen, optional on tls, where supplying them is the client half.
• cafile= is who this side trusts. On tls it replaces the platform store as who may vouch for the server. On tls-listen it names who may issue a client certificate.

$ tocat 'tls-listen:8443,fork,cert=server.pem,keyfile=server-key.pem,cafile=ca.pem,client-auth=required' \
    tcp:localhost:8080

$ tocat - 'tls:api.internal:8443,cafile=ca.pem,cert=client.pem,keyfile=client-key.pem'

client-auth has three settings. required refuses a client that presents no certificate. optional asks for one and checks it if it arrives, but accepts a client that presents none, so it authenticates the clients that have a certificate without restricting who may connect. none is the default and asks for nothing.

Two combinations are refused rather than accepted quietly:

• client-auth without cafile=. Falling back to the platform store would mean every public authority could vouch for anyone connecting, which is not a restriction anybody wants. Who may issue a client certificate has no default worth guessing.
• cafile= on a listener with client-auth=none. Nothing would consult it, and an option that sits there looking configured is worse than an error.

To try it with mkcert:

$ mkcert localhost
$ mkcert -client localhost
$ CA="$(mkcert -CAROOT)/rootCA.pem"

$ tocat "tls-listen:8443,fork,cert=./localhost.pem,keyfile=./localhost-key.pem,cafile=$CA,client-auth=required" \
    tcp:localhost:8080

Talking to something with a self-signed certificate

The usual reason to reach for verify=none is an appliance that presents a certificate no public authority signed. There are two better answers, and both are real checks rather than the absence of one.

If you have the issuing certificate, name it:

$ tocat - 'tls:appliance.local:443,cafile=/etc/appliance-ca.pem'

If you do not, pin the certificate itself. Its fingerprint is stable until it is replaced, so this authenticates the peer even though nothing signed it:

$ openssl s_client -connect appliance.local:443 </dev/null 2>/dev/null \
    | openssl x509 -noout -fingerprint -sha256
SHA256 Fingerprint=2F:8A:...

$ tocat - 'tls:appliance.local:443,pin=sha256:2F:8A:...'

The colons are optional and the case does not matter. pin= checks the leaf certificate and nothing else: no chain, no expiry, no name. That is the trade, and it is a reasonable one against a peer whose certificate you have seen.

pin= and verify=none together are refused, because they contradict each other.

verify=none

$ tocat - 'tls:appliance.local:443,verify=none'

The connection is encrypted and the peer is unauthenticated. Anything able to sit between you and the peer can present its own certificate, and TLS will accept it, so this protects against a passive listener and not against an active one.

It is spelled as a word rather than as a number so that a config file carrying it reads as an admission, and every run that uses it logs a warning naming what it gave up. That is deliberate: the failure mode of a verification bypass is that it is invisible afterwards, and the one thing that can be done about it is to make it noisy in the place it ended up.

The signature checks in the handshake still run, which proves the peer holds the key in the certificate it sent. That is worth nothing on its own, since the certificate is unverified, but it costs nothing and keeps the handshake honest.

Boundaries

TLS declares Fuse. A record is not a message: one write by the peer may arrive as several records or share one with the next write, so an endpoint with a TLS layer carries a byte stream whatever is underneath it. A stage that needs message boundaries is refused over TLS, for the same reason it is refused on tcp:.

The composed form

The command line spells the common stack as a single scheme. A config file can write either, and --dump-config prints the composed one:

[source]
type = "tcp"
addr = "backend.internal:443"

[[source.layers]]
type = "tls"
cafile = "/etc/ca.pem"
alpn = ["h2"]

The two are the same endpoint. The composed form is what makes a deeper stack expressible once there is more than one layer to stack.

What is not here yet

DTLS is out of scope, so a layer over udp: is refused rather than attempted. And reconnect=keep is refused under a layer, because reopening would have to redo the handshake underneath a pipeline that is mid-stream; use reconnect=restart.

udp and udp-listen

Messages rather than a byte stream. udp: fixes a peer to send to; udp-listen: binds a port and either peers with the first sender or, with fork, serves every sender separately. The target follows the same host:port rules as tcp-listen, defaulting to 127.0.0.1:8000.

$ tocat udp-listen:9000 tcp:backend:80
$ tocat - udp:127.0.0.1:5353
$ tocat udp-listen:5353 'tee,format=hex' udp:8.8.8.8:53
$ tocat udp-listen:5353,fork 'timeout:both,timeout=30s' udp:8.8.8.8:53

Plus reuseaddr, recv-buffer and send-buffer from the shared socket options. linger and the TCP options are refused: there is no connection to close and no Nagle’s algorithm to turn off.

A bind= address is now resolved before the socket is created, so a name with several addresses takes the first.

Multicast

Both forms take the options below, because one endpoint is often both a sender and a receiver: a relay answering on a group address sends from the socket it joined on.

# Receive a group and hand it to a local service
$ tocat 'udp-listen:0.0.0.0:5353,reuseaddr,multicast-group=239.1.1.1' tcp:localhost:8080

# Send to a group, off this host, without hearing the echo
$ tocat - 'udp:239.1.1.1:5353,multicast-ttl=32,multicast-loop=false'

A receiver has to bind the wildcard or the group address. Joining a group on a socket bound to a specific unicast address will not receive the group’s traffic, and it fails by receiving nothing rather than by erroring, which is the hardest kind of failure to diagnose. udp-listen:0.0.0.0:5353 or udp-listen:239.1.1.1:5353, not udp-listen:192.168.1.5:5353.

The group decides the address family, including for multicast-ttl. IPv4 and IPv6 have separate socket options for the hop limit and for the loopback flag, so the group is what says which pair to set. Without a group, an unqualified ttl= is read as IPv4. multicast-interface follows the same split: an address for IPv4, an interface index for IPv6.

reuseaddr is usually wanted on a receiver, since more than one process on the host may be listening to the same group and port.

What is not here yet

The join happens once, when the socket is bound. There is no way to leave a group, to join more than one, or to change the membership of a running relay, and a reconnect that reopens the socket rejoins from scratch. Source-specific multicast is not supported either.

udp: resolves the peer before binding, so that the local socket lands in the same address family. Without fork, udp-listen: peeks the first datagram to learn who the peer is and then connects to it, leaving that datagram queued for the relay rather than eating it.

Forking

With fork the socket is left unconnected and datagrams are routed by source address. Each new address becomes a session: its own dialled peer, its own buffers, its own plugin instances, its own span in the log. Replies go back out of the same socket to that address, so no extra port is opened per sender.

Nothing ends a session on its own, because UDP has no close to observe. Use the timeout plugin, on both directions:

$ tocat udp-listen:9000,fork 'timeout:both,timeout=30s' tcp:backend:80

:both matters. A forward-only halt ends the path from the sender but leaves the reverse pump reading a sink that may never close, and it is the session task finishing that releases the connection permit and the map entry. Without it, sessions accumulate until max-connections is reached and every new sender is dropped from then on; the log says so the first time it happens.

A halt is a real end of stream, so a hash digest, a compress epilogue or a rate summary do arrive when the session closes. A sender that goes quiet and comes back gets a new session with fresh plugin state, the way a reconnecting TCP client would.

Because one receive loop serves every sender, a session that cannot keep up has its datagrams dropped rather than being allowed to stall the others. The first drop is a warning and the rest are at debug, so -v is where to look if a peer is losing messages.

Three consequences of the datagram shape are worth keeping in mind.

No end of stream, unless a stage makes one. A datagram source runs until it is interrupted, so anything that only happens at end of stream never happens on that path: a rate summary, a compress ratio report, the final short block. A timeout stage halting the path is the one thing that produces one.

The copy buffer is the message ceiling. One receive is one datagram, and a message longer than the buffer is truncated by the kernel. The default is 256 KiB, which is above any datagram that will survive a real network, but -b set low enough will start cutting messages.

Boundaries are data. A stage that may not preserve them draws a warning when the destination on its path is a datagram endpoint. See Datagrams.

unix and unix-listen

$ tocat - unix:/run/app/app.sock
$ tocat unix-listen:/tmp/tocat.sock,fork,unlink,mode=660 tcp:localhost:8080

Plus linger, recv-buffer and send-buffer from the shared socket options, and, on unix rather than unix-listen, the resilience options. The TCP ones are refused here rather than ignored: a unix socket has no Nagle’s algorithm, no keepalive and no address to hold in TIME_WAIT.

unlink is about the stale path rather than the fresh one. Binding fails on an existing path whether or not anything is listening on it, so with unlink set tocat probes the path first: a refused connection means the owner is gone and the path can be removed, while a successful one means a live server and is an error rather than something to unlink out from under.

A bound socket is removed again when the relay finishes.

In a config file the scheme may also be written unix-connect. On the command line only unix: is accepted.

These two carry a byte stream. For local sockets that carry messages instead, see unix-seqpacket, which is connected and preserves boundaries, and unix-dgram, which is connectionless.

Addresses

Every unix scheme takes its address the same way, on the command line and in a config file alike.

A plain address is a path. An address beginning with @ names the Linux abstract namespace, which lives outside the filesystem:

$ tocat unix-listen:@tocat,fork tcp:localhost:8080
$ tocat - unix:@tocat

source = { type = "unix-listen", path = "@tocat", fork = true }

An abstract address has no directory entry, which changes three things.

• Nothing is created and nothing is cleaned up. The kernel releases the name when the last socket holding it closes, so there is no stale address for unlink to clear and nothing left behind if tocat is killed.
• mode is rejected rather than ignored. There is no file to change the permissions of, and no permission check at all: anything in the network namespace can connect. tocat fails the run instead of letting mode=600 look like it did something.
• Reaching one needs the same namespace. Containers, and anything else with a network namespace of its own, cannot see each other’s abstract addresses even when they share a filesystem. The reverse is also true, which is the usual reason to prefer one.

To use a file that really is called @name, write it as ./@name or give the absolute path.

unix-dgram and unix-dgram-listen

Local messages without a connection. unix-dgram: fixes a peer to send to; unix-dgram-listen: binds an address and receives. This is the shape /dev/log and most local logging protocols use.

$ tocat unix-dgram-listen:/tmp/log.sock,unlink file:/var/log/collected.log
$ tocat - unix-dgram:/dev/log
$ tocat unix-dgram-listen:/tmp/log.sock,unlink 'tee,format=hex' udp:collector:514
$ tocat unix-dgram-listen:@tocat,fork 'timeout:both,timeout=30s' tcp:backend:80

The scheme is also spelled unix-datagram and uds-dgram, with -listen on the end for the listening form.

Replies need an address

A unix datagram socket that never bound has no address, so nothing sent to it can be answered: the kernel has nowhere to deliver a reply to. This is the one place unix datagrams differ sharply from UDP, where an unbound sender is given a port automatically and can always be replied to.

unix-dgram: therefore binds a local address before it connects, so the reverse direction works by default. Without one, a relay whose sink is duplex would sit waiting for replies the kernel could not deliver, which looks like a hang rather than a misconfiguration.

The default is a path in the temporary directory, created with mode 600 and removed when the relay ends. bind= names one yourself, which is what you want when the peer authorises callers by path:

$ tocat - unix-dgram:/run/app/app.sock,bind=/run/app/client.sock,mode=660

On the listening side the same rule decides who can be answered. The first sender becomes the peer, and if it never bound, tocat says so and the path becomes receive-only:

WARN the first sender has no address of its own, so this path can receive but
not send; a peer that expects replies has to bind before it sends

That is the ordinary case for logging, where nothing is expected back and the sink has nothing to send anyway.

Listening

Without fork, unix-dgram-listen: receives one message to learn who the sender is, connects to it, and hands that first message to the pipeline as the first thing it carries. Nothing is lost, and once connected the kernel filters to that sender, so later messages from anyone else are dropped. A sender with no address of its own leaves the socket unconnected, and then messages from everyone are received.

Forking

With fork the socket stays unconnected and messages are routed by sending address. Each new address becomes a session: its own dialled peer, its own buffers, its own plugin instances, its own span in the log. Replies go back out of the same socket, so no extra socket is created per sender.

A session needs a sender it can both recognise and answer, and that is exactly the address that also carries replies. Senders that bound a path are served. Senders that did not are dropped, because two of them cannot be told apart and neither can be answered:

WARN dropping datagram; further drops are logged at debug
  peer=unnamed reason=the sender has no address to reply to

Senders in the abstract namespace are dropped for a narrower reason: replies here go out by path, and an abstract name is not one. Bind the senders to paths if they need to be forked to.

If your senders are anonymous and you want a session each, they need a connection rather than an address: unix-seqpacket-listen is the message transport that accepts.

Nothing ends a session on its own, because a datagram socket has no close to observe. Use the timeout plugin on both directions, exactly as on udp-listen; without it, sessions accumulate until max-connections is reached and new senders are dropped from then on.

Consequences of the datagram shape

No end of stream, unless a stage makes one. Anything that only happens at end of stream never happens on that path: a rate summary, a compress ratio report, the final short block. A timeout stage halting the path is the one thing that produces one. unix-seqpacket is the local message transport that does have an end of stream.

The copy buffer is the message ceiling. One receive is one message, and a longer one is truncated by the kernel, silently. The default is 256 KiB.

Boundaries are data. A stage that may not preserve them draws a warning when the destination on its path is a message endpoint. See Datagrams.

Addresses

Paths and abstract names both work, and the rules for unlink, mode and cleanup are the same as for unix.

unix-seqpacket and unix-seqpacket-listen

A local socket that keeps message boundaries and still has a connection and an end of stream. What the peer sent as three messages arrives as three messages, in order, and closing the connection is visible on the other side.

$ tocat - unix-seqpacket:/run/app/app.sock
$ tocat unix-seqpacket-listen:/tmp/tocat.sock,fork,unlink,mode=660 tcp:localhost:8080
$ tocat unix-seqpacket-listen:@tocat 'frame,format=lv32' tcp:collector:9000

Both forms take linger, recv-buffer and send-buffer from the shared socket options, and on the listening form they apply to every accepted connection. The TCP options are refused: a unix socket has no Nagle’s algorithm, no keepalive and no address to hold in TIME_WAIT.

unix-seqpacket also takes the resilience options. reconnect=keep is refused here: it needs an endpoint that opens a two-way byte stream, and this one carries messages.

The scheme is also spelled unix-seqpkt, uds-seqpacket and seqpacket, with -listen on the end for the listening form.

Choosing it over unix

Both are local sockets between processes on one machine, and the difference is what a read gives you back.

unix is a byte stream: what arrives is whatever the kernel had ready, so a protocol carried over it needs framing of its own, and tocat’s frame plugin exists for that. unix-seqpacket carries the boundaries for you. If the peer speaks a message protocol, this is the endpoint that stops you inventing a framing layer to recover what the sender already knew.

That also changes what a pipeline may do. A seqpacket endpoint is a message endpoint, so the same rules apply to it as to udp: a stage that does not preserve boundaries draws a warning when the destination on its path is one of these, and a stage that needs boundaries arriving is satisfied by one. See Datagrams.

The buffer is the message ceiling

One receive is one message, and a message longer than the copy buffer is truncated, with the rest lost. Unlike UDP the kernel says so, and tocat warns once per connection:

$ tocat -b 1KiB unix-seqpacket:/run/app/app.sock -
WARN a message did not fit the buffer and the rest of it is lost; raise -b past
the largest message the peer sends

Further truncations on the same connection are logged at debug, so a flood does not bury the rest of the log. The default buffer is 256 KiB.

Ending

A seqpacket connection has a real end of stream, which is what separates it from unix-dgram and from UDP. When tocat has nothing further to send it half closes the connection, so a peer waiting for that before it replies gets what it is waiting for, and one-way transfers finish on their own:

$ tocat file:messages.bin unix-seqpacket:/run/app/app.sock

The reverse arrives as an ordinary end of stream too, so hash digests, rate summaries and compress epilogues are all produced.

One consequence worth knowing: a zero-length message and a peer shutdown look the same at the receiving end, so sending an empty message ends the path.

Peers under fork

A connected unix socket has no address, so accepted connections are all labelled unnamed, the same as a unix-listen client that never bound. Each one still gets its own plugin instances and its own dialled peer.

Addresses

Paths and abstract names both work, and the rules for unlink, mode and cleanup are the same as for unix.

ws

WebSocket is a layer, like tls, and the first one that changes what an endpoint carries. TLS takes a byte stream and gives back a byte stream; WebSocket takes one and gives back messages, so a ws: endpoint is a datagram endpoint sitting on a TCP transport.

That is what makes it interesting beyond parity: it is the first message oriented network transport tocat has, so frame, unframe and the other boundary preserving stages become useful over a network rather than only over unix sockets.

The four schemes

$ tocat - ws:echo.example.com:80/socket
$ tocat 'wss-listen:8443,fork,cert=cert.pem,keyfile=key.pem' tcp:localhost:9000

The address is host:port as everywhere else in tocat, not a URL: write wss:echo.example.com:443, not wss://echo.example.com. The port is always required, since no scheme here defaults one.

wss and wss-listen also take every tls option, and all four take the socket options of the TCP transport underneath.

The path is part of the endpoint

The same value on both sides: asked for when dialling, required when accepting. Unset is / either way, so ws-listen:9000 serves / and answers a client asking for anything else with a 404 rather than upgrading it. The query string is the client’s business and is not compared.

A path is not authentication. It stops a misdirected client from being served, not a determined one.

Messages, not bytes

One message in is one message out. A ws: endpoint is under the same datagram rules as udp: and unix-seqpacket:: a stage that cannot carry boundaries is refused at build time rather than quietly joining your messages together.

A message larger than the copy buffer is an error. The other message endpoints truncate, because a datagram sender cannot know what the receiver will accept. A WebSocket peer believes it sent a whole message, so delivering part of one is worse than failing. Raise -b if you meet this.

Text and binary

Received text frames are passed through as their bytes; nothing is lost either way. What is sent is binary, unless you set text.

$ tocat - 'ws:api.example.com:80/socket,text'

A text frame promises the peer that its payload is valid UTF-8, and tocat cannot know whether the bytes crossing it are text. text is you making that promise. If a message turns out not to be valid UTF-8 the run fails rather than sending something the peer is entitled to reject.

What is not here yet

Subprotocol negotiation and Origin checking, both of which belong in the same place as the path check. Compression is not offered.

Plugins

Bytes can be passed through a pipeline of plugins on their way between the endpoints. A plugin can watch the stream, rewrite it, or drop parts of it.

Plugins are written like endpoints (a name, an optional direction, and a set of options) and go between the source and the sink on the command line.

name[:direction],option,option=value

$ tocat tcp-listen:8080,fork tee,format=hex tcp:example.com:80

The same entries can be given with -p, which is the only way to add one when both endpoints come from flags or a config file. Repeat it as needed; entries apply in the order written, after any in the config file.

$ tocat -f tcp-listen:8080,fork -t tcp:example.com:80 -p 'tee,format=hex'

Run tocat --list-plugins to see what your build has, and --no-plugins to ignore the ones in a config file.

What ships with tocat

“Datagram paths” is what each stage reports about itself, and it is what tocat checks when the destination on that path is a datagram endpoint. The default for a stage that says nothing, including any plugin from outside the binary, is that it is not safe.

A few stages go further and need message boundaries rather than merely preserving them. encrypt and decrypt in record mode are the two that ship, and an unmet requirement is an error rather than a warning: see When a stage needs boundaries below.

Writing an entry

On the command line an entry is comma-separated, as an endpoint is. A bare key means true (tee,append), a value that parses as an integer becomes one, true and false become booleans, and everything else stays a string. Sizes and durations are therefore written as they are elsewhere (bytes=10MiB, flush=250ms), and a value containing a comma has to go in a config file.

Option keys are matched leniently: case, dashes and underscores are noise, so at-limit, at_limit and atLimit are the same option. An option the plugin does not declare is an error rather than being ignored.

In a config file, an entry is a [[plugin]] table with the plugin’s own options alongside name, direction, as and detach:

[[plugin]]
name = "tee"
direction = "sink-to-source"
file = "/var/log/tocat-capture.bin"
format = "raw-binary"

Direction

An entry applies to one path or both. Omitting the direction means the forward path, from the source to the sink.

both builds two independent instances, one per path, so per-direction state (byte offsets, codec state) never leaks across paths. That also means two of whatever the stage counts: limit,bytes=1MiB,direction=both lets a megabyte past in each direction rather than a megabyte between them, and a ticking stage declared that way builds a timer per path per connection. Asking for both is therefore worth doing on purpose, which is why it is no longer what you get by saying nothing.

An asymmetric stage is not its own inverse, so both is wrong for it and the pair is written out:

$ tocat - compress decompress:reverse tcp:relay.internal:9000

The command line is a picture of the wire, read left to right, and the reverse path reads it right to left:

tocat SRC  a  b  c  SINK

source -> sink:   SRC -> a -> b -> c -> SINK
sink -> source:   SRC <- a <- b <- c <- SINK

So a stage written earlier sits nearer the source, and bytes coming back from the sink reach the later stages first. Write the stages in the order the forward path would see them, and the reverse path nests correctly, which is what you want for anything that wraps the payload.

The source/sink aliases are accepted but read badly: tee:sink means sink-to-source, not “tee at the sink”. Prefer forward and reverse.

Common options

Two options are handled by tocat rather than the plugin, and work on any entry.

The #n suffixes are only added where a name would otherwise appear twice on the same path, so a lone tee stays tee.

compress and decompress default to detached, and detach=false on one is honoured. process runs as a child process and always has its own task, so detach=false on one is rejected rather than ignored.

If nothing is declared on a path, that direction is copied straight through with no plugin machinery in the way.

Datagrams

On a byte stream a chunk is an arbitrary slice, and a stage may buffer, split or coalesce it freely. On a datagram path the chunk is the message: one call per datagram, and each unit the stage emits is sent as exactly one datagram. A stage that holds bytes across calls, or emits two messages’ worth from one, will produce well-formed datagrams containing nonsense.

Every stage declares whether it may sit on such a path. When one that may not is on a path whose destination is a datagram endpoint, tocat names it, warns, and relays anyway:

stage may not preserve message boundaries; datagrams sent to this endpoint may be split, merged, or malformed

That is a warning rather than a refusal because rewriting the message stream is sometimes the point: block,size=1400 on a UDP path emits one datagram per block, which is a reasonable thing to ask for and still not a preservation of what the peer sent.

The check is against the destination, not the source. Sending datagrams into a stream sink is unremarkable and draws no warning. So udp-listen:9000 compress:forward tcp:collector:9000 is fine, while udp-listen:9000 compress udp:peer:9000 will warn. A process stage always warns: its stdin and stdout are byte streams, so boundaries are gone the moment bytes cross the pipe.

When a stage needs boundaries

encrypt and decrypt do in record mode, a plugin from outside the binary can say that it does, and a WebAssembly guest declares it through TOCAT_NEEDS_UPSTREAM and TOCAT_NEEDS_DOWNSTREAM. A stage that reads whole messages needs one message per call; a stage that writes whole messages needs the ones it emits to arrive intact. Neither is a matter of taste, so an unmet requirement stops the relay before it starts rather than warning, and names both the stage and what took the boundaries away:

wasm on source-to-sink needs message boundaries arriving, and the source tcp://host:9000 is a byte
stream: put an unframe above it, or use a stage that does not need them

There are two ways to satisfy one. A datagram endpoint on the relevant side supplies boundaries by itself, so nothing more is needed on a UDP path. On a byte stream, frame and unframe are what put them there: unframe above a stage that reads messages, frame below one that writes them. Either in mode=null satisfies the requirement without writing any framing, which is what you want when the far end is a terminal rather than a peer, and is not what you want otherwise.

The scan stops at the first stage that settles the question, so a frame anywhere below covers everything under it: once the boundary is in the payload no later stage can lose it. A stage needing its units delivered is therefore happy with frame,mode=length below it even when a compress sits below that, and unhappy with the same two the other way round, since the units are gone by the time the framer sees them.

The check goes as far as confirming something downstream can put the boundaries on the wire. It cannot confirm the bytes reach the peer intact, or that the peer unframes them: see What framing survives.

One more thing a datagram sink does silently: an empty emission is not sent. At end of stream a pipeline is drained once more, and on a datagram sink that would otherwise put a spurious zero-length message on the wire, which many peers read as a close. The cost is that a genuine zero-length datagram is dropped rather than forwarded.

base64 and unbase64 - base64

Behind the base64 cargo feature. Carries arbitrary bytes across a hop that only tolerates text. The two are paired, one per path, as compress and decompress are:

$ tocat - base64 unbase64:reverse tcp:relay.internal:9000

direction=both would encode the replies too and hand the peer double-encoded text.

The far end of the link runs the mirror image, and the two relays form a base64-armored tunnel over a hop that would otherwise mangle binary.

Both are cheap enough per byte to run inline, and both preserve message boundaries, so either may sit on a datagram path.

One message per chunk

Each chunk is one complete message. Neither stage holds bytes back between calls, and base64 pads every chunk it encodes.

That is the datagram contract, and on a datagram path it comes for free: one call per message, one message out. On a byte stream a chunk is an arbitrary slice, and base64 packs three bytes into four characters, so a chunk cut anywhere other than a group boundary cannot be decoded on its own. Boundaries on a stream path come from frame and unframe, which belong immediately outside these stages.

A chunk whose length is not a whole number of groups is reported as a configuration error rather than relayed:

message is not a whole number of base64 groups: unbase64 decodes one complete message per call, so it needs an unframe stage ahead of it

The one mistake that cannot be caught is a chunk cut exactly on a group boundary, which decodes to a short payload. Framing is what rules it out.

Base64 output is text, so a newline delimiter is safe by construction:

$ tocat - base64 unbase64:reverse frame unframe:reverse tcp:relay.internal:9000

Padding

base64 always pads. unbase64 requires padding, because under one message per chunk a short final group is far more likely to be a framing bug than a peer that omits =. accept-unpadded is for a peer that really does omit it, and costs that diagnostic: a message truncated by one or two characters then decodes to a short payload instead of erroring. A length remainder of 1 is rejected either way, since no padding rule makes it valid.

Whitespace is not stripped. A trailing newline is a frame delimiter, and swallowing it here would paper over framing that is missing or misconfigured.

Size

Base64 is a 4-to-3 expansion, so an encoded message is a third larger than the payload, plus padding. On a datagram path that is the difference between fitting in a datagram and not: a 1400-byte payload leaves as 1868 bytes.

block - cut the stream into fixed-size records

Holds bytes back until it has a full block, then emits it as one unit.

$ tocat file:stream.bin block,size=512,pad file:/dev/st0
$ tocat -f tcp-listen:9000,fork -t udp:collector:9999 -p 'block,size=1400,flush=250ms'

This is dd’s obs, with the difference that a block is a unit rather than merely an amount: each one is delivered on its own rather than being concatenated with its neighbours. On a byte sink that only changes where the writes fall, which is what a tape drive, a raw device or anything opened O_DIRECT cares about. On a datagram sink each block becomes one message, and across a detach boundary each block becomes one parcel. See Units and boundaries for the mechanism.

A chunk off the wire is usually several blocks long, so one call can emit several, and a full block never waits: it goes out the moment it fills. flush bounds the short one, and the bound is real rather than approximate, because the stage restarts its own schedule when a block starts filling. The interval is therefore measured from the first byte held, not from wherever a shared cadence happened to be, so flush=250ms means a byte waits at most 250ms and not “until the next quarter-second boundary that comes round”.

flush=0 is not a timer at all: it emits whatever is in hand on every write, and builds no schedule. Without flush there is no timer either, and a short block waits for end of stream, which is what a device wants and an interactive stream does not.

pad only ever affects a short block, which in practice means the last one and any cut short by flush. A full block is already size bytes. An idle path stays quiet: a tick with nothing in hand emits nothing, which is what stops pad putting a block of zeroes on the wire once per interval forever.

Framing costs something, so reach for this only when the splits are the point. Every stage below a block is called once per unit rather than once per chunk, so size=512 against a 256 KiB buffer runs the rest of the pipeline 512 times per read. detach on a stage below one is worse still, since each block then costs its own task wakeup.

Not safe on a datagram path: this stage holds bytes across calls and the boundaries it emits are its own rather than the ones the peer sent. That is sometimes exactly what you want (block,size=1400 into a UDP sink is one datagram per block), so it warns rather than refusing.

The block buffer is reserved at startup and the reservation is fallible, so size=64GiB is a startup error rather than an allocation failure later.

compress and decompress - zstd

Behind the compress cargo feature. Compression is asymmetric, so the two are paired, one per path. An entry with no direction is on the forward path, which is what the first of these relies on:

$ tocat - compress decompress:reverse tcp:relay.internal:9000

direction=both would compress the replies too and hand the peer garbage.

The far end of the link runs the mirror image, and the two relays form a compressed tunnel over an otherwise plaintext hop.

A relay must not sit on bytes waiting for a better compression window, so flush is on by default: whatever arrived is on the wire before the call returns. Turn it off for bulk transfers where throughput matters and nobody is waiting on a prompt, and output then appears only as zstd fills its own buffer and at end of stream.

Both default to detach, since they are expensive enough per byte to be worth their own task. detach=false is accepted if you would rather have them inline.

compress writes a zstd epilogue at end of stream; without it a peer’s decoder rejects the stream as truncated. Two paths never reach end of stream, a datagram source and a held pipe, so neither the epilogue nor report happens there. Neither stage preserves message boundaries, so a datagram destination on the same path draws a warning.

encrypt and decrypt - symmetric encryption

Behind the encrypt cargo feature. Encryption is asymmetric, so the two are paired, one per path:

$ tocat - encrypt,cipher=aes-256-ctr,key-file=k.hex decrypt:reverse,cipher=aes-256-ctr,key-file=k.hex tcp:relay.internal:9000

direction=both would encrypt the replies too and hand the peer ciphertext it would try to decrypt.

The far end of the link runs the mirror image, and the two relays form an encrypted tunnel over a hop that would otherwise carry plaintext.

Exactly one of key, key-file, key-env and random-key is required.

Every option except on-fail, random-key and key-out has to match at both ends. Nothing is negotiated and nothing is described on the wire, so a mismatched cipher, mode, padding or rotate-after is a stream neither end can read.

Both default to detach, since they do enough work per byte to be worth their own task. detach=false is accepted if you would rather have them inline.

Two modes

record mode makes each chunk a self-contained unit:

nonce || ciphertext || tag

A fresh nonce is drawn for every record, so nothing is carried between them and a lost or reordered message costs exactly that message. This is the mode for a datagram path, and the only mode the authenticated ciphers have.

stream mode makes the whole path one session:

nonce || ciphertext ...

The nonce goes out once, ahead of the first byte, and the cipher’s state continues across every chunk after it. Nothing frames it, so no frame stage is needed, and it is the default for the ciphers that allow it.

The choice is between where the overhead goes and what a lost byte costs. A session pays for one nonce and then nothing; a record pays for a nonce and a tag every time, and on a datagram path that is what makes each datagram readable on its own. A session that loses a byte anywhere never recovers.

Records need boundaries

A record has to arrive exactly as it left, so encrypt in record mode requires that the units it emits survive downstream, and decrypt requires whole records arriving. On a datagram path that comes for free. On a byte stream it comes from frame and unframe, and without them tocat refuses to start:

encrypt on source-to-sink needs the units it emits to survive, and the destination tcp://host:9000 is a
byte stream: put a frame below it, or use a stage that does not need them

The pair belongs immediately outside the encryption, so the forward path frames what it encrypted and the reverse path unframes before decrypting:

$ tocat - encrypt,cipher=aes-256-gcm,key-file=k.hex decrypt:reverse,cipher=aes-256-gcm,key-file=k.hex frame unframe:reverse tcp:relay.internal:9000

frame,mode=length is the one to reach for here: the payload is ciphertext, so it can contain any byte, and a delimiter would have to be escaped.

Which cipher to pick

If nothing else decides it, use aes-256-gcm, or chacha20-poly1305 on hardware without AES instructions. Both authenticate, both are record mode, and both cost 28 bytes a record.

Everything else in the table below is there to talk to something that already speaks it. The list is long because a mode and a block cipher combine freely, not because a hundred and twenty three of them are worth choosing.

xchacha20-poly1305 earns its extra twelve bytes on a path carrying enormous numbers of records, since its 24-byte nonce is what makes a random one safe that far out. ascon-aead128 is the NIST lightweight selection, for talking to something small. aes-256-siv is nonce-misuse resistant, which matters where the same nonce may be drawn twice.

Ciphers

A name is primitive-keybits-mode, so aria-192-ctr is ARIA with a 192-bit key in counter mode. Names are matched the way every identifier in tocat is, so aes-256-gcm, AES256GCM and aes_256_gcm are the same cipher.

Every combination below exists, at each key size listed:

Plus the ciphers that are not a mode over a block cipher:

The two SIV key sizes are the sum of the two keys RFC 5297 splits them into, so aes-128-siv takes 32 bytes and aes-256-siv takes 64.

AES, ARIA and Camellia come in three key sizes, and leaving it out means the largest: aes-ctr is aes-256-ctr, and a bare aes is aes-256-gcm. The others have one size each and carry none in the name, so it is sm4-ctr and kuznyechik-gcm; bare sm4 is sm4-gcm and bare magma is magma-ecb.

The rest of the aliases are spellings: 3des- and tdes- for the des-ede3- modes, grasshopper- for the kuznyechik- ones, chacha, xchacha, salsa and xsalsa for the bare keystreams, chachapoly and xchachapoly for the two Poly1305 AEADs, and ascon for ascon-aead128.

What each mode costs on the wire

The AEAD modes all take a 12-byte nonce here, so a record costs 28 bytes.

The IV a keystream mode transmits is the primitive’s block, so it is 8 bytes for des, des-ede3 and magma and 16 elsewhere. GOST counter mode is the one exception: magma-ctr puts 4 bytes on the wire and kuznyechik-ctr puts 8, because GOST starts the counter at the transmitted half followed by zeroes.

rc4 has no IV at all. That makes mode=record unsafe with it, since every record would then be encrypted under the same keystream from the start, and it is also why rotate-after is refused for it.

The unauthenticated modes keep the payload secret and do nothing at all about tampering: anyone who can change bytes in transit can change the plaintext that comes out, and decrypt will hand it downstream without complaint. ECB goes further and encrypts identical blocks identically, so the shape of the plaintext survives into the ciphertext.

Among them CTR is the one that goes fast. CBC, CFB and OFB chain one block into the next and are several times slower to encrypt for that reason; see Cost.

des has a 56-bit effective key and is breakable by anyone who cares to. The 8-byte block ciphers, des, des-ede3 and magma, also reach the birthday bound after a few tens of gigabytes under one key, which on a relay is an afternoon rather than a theoretical limit.

Standards

Every cipher records the document that says what its bytes are, and a combination that no document specifies is refused:

invalid configuration for plugin `decrypt`: camellia-256-gcm-siv is not specified
by any document, so nothing else is likely to read it: pass nonstandard to use it
anyway

These are combinations the crates behind them will happily compute, and which round trip against tocat at both ends, but which no other implementation is likely to agree with byte for byte. They are the gcm-siv rows other than aes-128-gcm-siv and aes-256-gcm-siv, RFC 8452 having specified GCM-SIV for AES at those two sizes and nothing else.

nonstandard opts in, and both ends need it, since both refuse without it:

$ tocat - 'encrypt,cipher=sm4-gcm-siv,key-file=k.hex,nonstandard' ...

The flag is about whether a document defines the bytes, not about whether the choice is wise. A withdrawn standard still counts as one, which is why des-cbc needs no opt-in.

Authentication and what happens without it

An AEAD cipher tags every record, and decrypt checks the tag before it emits anything. A record that fails is never forwarded; on-fail decides what happens instead.

drop is what a datagram path usually wants, where one bad message among many is a thing to note rather than a reason to stop. error is the safer default everywhere else: a record that fails is either a misconfiguration or someone changing bytes in transit, and neither is a thing to carry on through quietly.

on-fail is a record mode option. Dropping part of a session desynchronises everything after it, so a session that cannot be decoded ends the path.

The unauthenticated ciphers have no tag to check. CBC and ECB still reject malformed padding, which catches a wrong key most of the time and nothing an attacker does deliberately. The keystream modes, CTR, OFB, CFB and the bare stream ciphers, cannot tell a good record from a bad one at all.

Keys

The key is bytes, and every source below gives the same bytes, so a key written as hex in one place is the same key as its base64 elsewhere. A key of the wrong length for the cipher is rejected at startup rather than truncated or padded.

$ tocat - encrypt,cipher=aes-256-gcm,key-file=/etc/tocat/relay.key ... 
$ tocat - encrypt,cipher=aes-256-gcm,key-env=RELAY_KEY ...

key-format=raw reads a file’s bytes exactly, which is what head -c 32 /dev/urandom > relay.key produces. hex and base64 ignore whitespace, so a file with a trailing newline reads as intended.

key= on the command line puts the key in ps output and in shell history. It is there for a throwaway and for tests; key-file or key-env is what to reach for otherwise.

random-key generates one and reports it on stderr, or to key-out:

$ tocat file:secrets.tar 'encrypt,cipher=aes-256-gcm,random-key,key-out=key.txt' frame file:secrets.enc

Nothing else can hold that key, so the far end of a link cannot be configured to match it. It is for encrypting something you will decrypt yourself later, with the key the stage wrote down. The line is written once, when the path first carries bytes, in the same shape hash uses:

0001…1e1f  key (aes-256-gcm) [file:secrets.tar -> file:secrets.enc | encrypt]

stdout is refused, since on a stdio endpoint it carries payload.

Rotation

rotate-after=N ends a session every N bytes and starts another, with a fresh nonce on the wire between them. It applies wherever there is a nonce to replace: the keystream modes, where it bounds the keystream drawn under one nonce, and cbc, where it bounds the chain. ecb and rc4 have no nonce and are refused. So is record mode, where every record already draws its own.

$ tocat - 'encrypt,cipher=aes-256-ctr,key-file=k.hex,rotate-after=1GiB' ...

N counts ciphertext, not payload, and it is what each closed session puts on the wire after its nonce. For the keystream modes those are the same number. For cbc they are not: a session ends with a padding block, so a 64-byte budget carries three blocks of payload and one of padding, and how much payload rides in that last block depends on where the sender’s chunks fell. Counting ciphertext is what lets the receiver find the next nonce, since it chunks differently and has nothing else to go on.

For a block mode the budget must therefore be a whole number of blocks, and with pkcs7 at least two of them, since one would be all padding and no payload. Both ends must be given the same rotate-after.

What rotation does not do is change the key. The amount of data one key can safely carry is set by the key and the cipher’s block size, and a fresh nonce does not extend it. That matters most for a cipher with a small block, where the bound arrives sooner than anyone expects. Rotation bounds what a single nonce covers; it is not a way to keep one key in service for longer.

Datagram paths

Use record mode. Each datagram is then one self-contained record: it is readable on its own, a lost datagram costs nothing but itself, and reordering does not matter.

stream mode on a datagram path is what the boundary warning is about:

stage may not preserve message boundaries; datagrams sent to this endpoint may be split, merged, or malformed

A session assumes every byte arrives once and in order, which is exactly what a datagram path does not promise. One lost datagram desynchronises everything after it, and the far end will carry on emitting plaintext that is not what was sent.

Cost

What a cipher costs depends on the machine and on how the binary was built, so the useful thing to know is not a number but which of four things you are paying for. Measure the rest yourself: put a rate stage next to the encryption and read it off.

$ tocat file:/dev/zero limit,bytes=10GiB 'encrypt,cipher=aes-128-gcm,random-key' rate,interval=1s frame,mode=length file:/dev/null

One pass or two. The unauthenticated modes make one pass over the payload. The authenticated ones make two, one to encrypt and one to authenticate, so they cost roughly the cipher again. That is the price of the tag, and it is what you are buying when you pick an AEAD cipher.

Whether the blocks can overlap. CTR and ECB encrypt every block independently, so a modern CPU keeps several in flight at once and its AES units stay busy. CBC, CFB and OFB cannot: each block’s encryption feeds the next, so the work is a chain of latencies with nothing to overlap, and they run several times slower than CTR under the same key. This is a property of the modes rather than of this implementation, and it is asymmetric for CBC, whose decryption has no such chain and runs at CTR-like speed. A CBC path is therefore much slower in one direction than the other.

How many rounds. AES-256 is fourteen rounds against AES-128’s ten, and pays that difference on the encryption pass only. The authentication pass does not care about key length, so the gap between 128 and 256 is wider for the unauthenticated modes than for the AEAD ones.

Whether the hardware has the instructions. AES-GCM wants AES and carry-less multiply instructions, which every x86-64 part worth relaying on and most ARM ones have. ChaCha20-Poly1305 wants vector instructions instead, and is the one to reach for where the AES instructions are missing. The other primitives here run in software whatever the machine has, so expect any of them to be slower than AES by a wide margin. ascon-aead128 is designed for small constrained hardware rather than for throughput, and is slower than either on a machine of this kind.

Beyond the cipher itself, encryption copies each chunk once, and decryption of a block mode copies twice, since it accumulates whole blocks and holds the last one back until it knows it is not the padded tail. Both stages detach by default, so they run concurrently with the endpoints, and the slowest stage sets the rate for the path.

Record overhead is per record, so it is set by how often the pipeline calls the stage rather than by the size of the transfer. On a byte stream with a 256 KiB buffer, AES-GCM’s 28 bytes are nothing; on a datagram path carrying 100-byte messages they are 28 percent, and frame,mode=length above them would add four more. The same is true of the work done once per record rather than once per byte: a fresh IV, and for the unauthenticated modes a fresh key schedule, both invisible against a large buffer and worth counting against a small message.

frame and unframe - message boundaries

Puts message boundaries on a byte stream and takes them off again. The two are paired, one per path:

$ tocat - frame unframe:reverse tcp:relay.internal:9000

direction=both would frame the replies too, which is only right if the peer speaks the same framing back.

What they are for

On a datagram path a chunk is a message and there is nothing to do. On a byte stream a chunk is an arbitrary slice, so a stage that needs whole messages (unbase64, anything with a per-message header) has no way to find them. unframe is where that knowledge lives: it accumulates until it has a whole message, then emits it as one unit, and the pipeline turns each unit into one call at every stage below it.

frame is the inverse, and it takes the boundaries it is given: one unit in, one framed unit out. That makes it meaningful on a datagram path, or after a stage that declared boundaries of its own such as block, and pointless anywhere else, where a chunk is whatever the last read happened to return.

An option a mode ignores is an error rather than a no-op, so mode=cobs,delimiter=\n is refused instead of quietly going on using a zero byte.

frame preserves the message stream it is handed, so it may sit on a datagram path. unframe replaces that stream with the sender’s framing, so it warns there, and there is nothing for it to do on a path that already has messages.

Modes

Five that frame, in two families, and null, which does not. The terminator family scans for a byte string and pays for a payload that could contain it. The counted family reads a header and pays nothing for the payload at all.

delimiter

Cheap, greppable, and only correct when the payload cannot contain the delimiter. That pairs naturally with base64, whose output is text by construction:

$ tocat - base64 unbase64:reverse frame unframe:reverse tcp:relay.internal:9000

Encode then frame going out, unframe then decode coming back: the reverse path walks the entry list backwards, so writing each pair in forward order makes the two nest correctly.

cobs and slip

Both escape the payload so the terminator cannot appear in it, which makes them exact for arbitrary binary.

COBS costs one byte per frame plus one per 254 bytes of payload, and a receiver that joins a stream mid-flight resynchronises at the next zero byte.

SLIP escapes 0xc0 and 0xdb, so its overhead depends on the payload and doubles in the worst case. It is here because serial hardware and embedded stacks already speak it: reach for it to talk to something that requires it, and for COBS otherwise. An empty SLIP frame is a flush rather than a message, since RFC 1055 has senders lead with a terminator to clear line noise, so unframe skips it. An escape followed by anything but 0xdc or 0xdd is undefined in the RFC and is an error here, rather than passing a peer’s bug through as payload.

length and netstring

A header says how long the message is, so the payload is written and read untouched and the overhead does not depend on it. These are also the only modes that know how big a message is before reading it, which is what lets max-message reject an oversized one from its header instead of after buffering it.

length writes a fixed-width big-endian prefix, which is what most binary protocols on a TCP hop already speak. Set endian=little for a peer that got that wrong. A message too large for the width is refused by frame rather than truncated into a header the far end could never resynchronise from.

netstring is the djb format, length:payload, with a decimal length. Same properties, with a self-describing header you can read in a hex dump, and the trailing comma turns a desynchronised stream into an error on the very next message instead of never. The length must be canonical: no sign, no spaces, and no leading zero to pad the header out with.

Neither counted mode can resynchronise. A receiver that joins mid-stream, or a sender that gets one length wrong, is lost until the connection is remade, where the terminator modes recover at the next terminator.

null

No framing at all. frame forwards the message as it stands and unframe forwards each chunk as it arrives, so the bytes come out exactly as they went in.

It is there for what it declares, not for what it does. A stage that needs message boundaries to survive below it, or to arrive from above, will not build without a frame or an unframe on that side: the requirement is an error rather than a warning, as described in When a stage needs boundaries. mode=null settles that requirement without putting a byte on the wire, which is what you want when the far end is your terminal:

$ tocat file:/dev/urandom block,size=16 limit,chunks=4 \
    encrypt,cipher=aes-128-gcm,mode=record,random-key,key-out=/dev/null \
    hexify frame,mode=null -

encrypt in record mode needs its messages to reach the far end intact, and here the far end is you, reading hex. Nothing downstream is going to parse it, so there is nothing for framing to protect, and mode=delimiter would only scatter newlines through the output. Inspecting a path rather than feeding a peer wants this shape whatever the stages in the middle are.

The cost is that the declaration is not backed by anything, and tocat will not tell you so, because skipping that check is the point. A real peer below frame,mode=null cannot tell one message from the next, and a stage below unframe,mode=null is handed whatever the last read returned rather than a message. Use it when the far side is a person; use a mode that frames when the far side is a program.

null takes no options except max-message. Pairing it with delimiter, length-bytes, endian, check or at-eof is refused, since each of those describes framing that is not happening. max-message is accepted and does nothing: it bounds the bytes unframe holds while it waits for the rest of a message, and this mode never holds any.

The delimiter has to be absent from the payload

Framing a message that contains the delimiter puts two messages on the wire, and nothing downstream can tell that apart from two messages the sender meant. frame scans for it rather than trusting the peer:

message would frame as two: it contains the delimiter, or ends with a prefix of one

The second half of that catches a subtler case, and only bites for a delimiter that overlaps itself. A message ending in a, terminated by aa, puts aaa on the wire, and the far end reads the boundary one byte early even though the message contains no delimiter at all.

check=false turns the scan off for a peer whose parser is known to tolerate the ambiguity. Any other mode makes the question go away.

Bounded memory

unframe holds a partial message, so a peer whose framing does not match is a peer asking the relay to buffer without limit. max-message caps the message and defaults to 1MiB. Exceeding it is reported as a protocol error rather than accommodated:

no complete message in 1048577 bytes (max-message is 1MiB): this stage's framing does not match the peer's

A counted mode does better, and refuses the message from its header before a byte of the payload has been buffered:

a header declares 1073741824 bytes, over the max-message of 1MiB

The cap is on one message, not on the stream: a stream of small messages never approaches it however long it runs. Framing bytes do not count against it, so a message of exactly max-message still passes. max-message=0 removes the cap, which is worth doing only for a peer you control.

null mode holds nothing back at all, so the cap has nothing to bound. It is accepted there and does nothing.

What framing survives, and what it does not

Framing puts the boundary into the payload, so it outlives a stage below that loses message boundaries: block, compress, process. That is what makes

tocat file:in.bin 'frame,mode=cobs' 'compress' tcp:collector:9000

meaningful where the same two stages without frame would leave the far end unable to tell one message from the next.

None of this section applies to mode=null. It reports that the boundary survives, which is what lets a pipeline build, and writes nothing into the payload to make that so.

What survives is the bytes, though, not the framing as such, and that puts two conditions on the stage below.

The first is that the bytes have to come out the other end. A stage that copies them (block) or transforms them reversibly (compress, or a process running gzip) is fine, because the framing bytes are still in there. A stage that drops or rewrites them is not: process with argv = ["grep", "ERROR"] throws away whole frames and half of a COBS escape sequence with them, and the far end sees a corrupt stream rather than fewer messages. tocat cannot tell the two apart, and does not try: what a subprocess does to its stdin is not something the relay can inspect.

The second is that the inverse has to run on the far side, above the unframe rather than below it. The framing bytes are inside the compressed stream, so they only reappear once it has been decompressed:

send:    ... -> frame -> compress -> tcp
receive: tcp -> decompress -> unframe -> ...

Reversing that nesting is also valid and means something else. compress above frame frames whatever chunks the compressor emits rather than the messages it was given, and the far end recovers those chunks, not your messages.

Two smaller things about a subprocess in the middle. Pick a binary-safe mode: anything that emits arbitrary bytes rules out delimiter unless something upstream guarantees the payload cannot contain it, which is what cobs is for. And check the child flushes. libc block-buffers stdout when it is a pipe, so a framed message can sit in the child waiting for 4KiB of company that never arrives, which on a relay is a stall rather than a slowdown. stdbuf -o0, or whatever the program’s own flag is.

The end of the stream

A stream can end part way through a message. at-eof decides what happens to those bytes: emit sends them as a final message, error fails the path, and drop discards them.

The default depends on the mode, because the same bytes mean different things. A text stream whose last line has no newline is routine, so delimiter mode emits. Everywhere else a partial frame was cut off in transit, so the default is error. The counted modes refuse at-eof=emit outright: what they hold is a header and part of a payload, and there is no message in it to emit. null mode refuses the option in every form, holding nothing back and so never having a partial message to decide about.

Empty messages

A zero-length message describes nothing a pipeline can carry: an empty unit is not emitted, and a datagram sink drops an empty message rather than putting a spurious zero-length datagram on the wire. Every mode that frames can express one on the wire, and none of them will deliver one. In delimiter mode a blank line is therefore swallowed. In cobs mode two terminators in a row cannot come from a conforming sender at all, so they are reported as a corrupt frame. In null mode an empty message writes no bytes, so it leaves no trace to express.

hash - digest the stream

Computes a digest of everything crossing this point on the path, and writes it where you point it.

$ tocat file:big.iso hash tcp:host:9000
$ tocat -f tcp-listen:9000,fork -t file:capture.bin -p 'hash,algo=blake3,file=digests.txt'

md5, sha1, sha224, sha256, sha384, sha512, sha3-224, sha3-256, sha3-384, sha3-512, blake2 and blake3 are available. Spelling is forgiving, so SHA-256 and sha_256 are the same thing, and sha2 and blake name the common member of their family. md5 and sha1 are there for checking against a legacy tool’s output rather than for deciding whether two things are the same.

Lines follow the sha256sum shape, with the algorithm and the hop appended:

ba78…15ad  stream (sha256) [tcp://example.com:80_10.0.0.4:52134 -> STDIO | hash]

The trailing bracket is what makes a shared file readable. Entries naming the same file share one writer, and under fork with the default direction=both there are two instances per connection, so without the hop and the stage name the lines could not be told apart. as= names an instance everywhere, including here.

Like tee and rate it never touches the payload, so it can go anywhere in a chain, including on a datagram path, and its position decides only what it sees: before a compress stage it digests the payload, after it digests the wire. Two instances give you both.

$ tocat file:big.iso 'hash,as=plain' compress 'hash,as=wire' tcp:relay:9000

Direction works the same way: an entry digests the forward path unless it says otherwise. direction=both gives each path its own hasher, so a duplex connection produces two digests rather than one over the concatenation, which would be a number with no meaning.

Chunks

chunks reports the digest of each chunk alone, not a running value, which is what makes it useful for locating where two captures diverge: the first line that differs is the first chunk that differed. It costs a finalisation and a write per chunk, so it is off by default.

A chunk is an arbitrary slice of a byte stream, decided by the copy buffer and by what the peer sent, so the same bytes relayed twice do not have to produce the same chunk lines. On a datagram path they do, since there a chunk is a message. block above a hash makes the splits deterministic, at the price of the framing it imposes.

There is not always an end of stream

summary needs one, and two paths never reach it: a datagram source, and a pipe held open across producers. On those, chunks is the only thing that will ever report, which is the same reason rate reports on a timer as well as at the end.

Turning both off is rejected at startup rather than accepted as a stage that hashes nothing.

hexify and unhexify - hex

Carries arbitrary bytes across a hop that only tolerates text, as base64 does, and for twice the bytes. The two are paired, one per path:

$ tocat - hexify unhexify:reverse tcp:relay.internal:9000

direction=both would encode the replies too and hand the peer double-encoded text.

The far end of the link runs the mirror image, and the two relays form a hex-armored tunnel over a hop that would otherwise mangle binary.

Both are cheap enough per byte to run inline, and both preserve message boundaries, so either may sit on a datagram path.

unhexify takes no options. It accepts either case regardless of how the far end is set, so an option there would have nothing to do.

One message per chunk

Each chunk is one complete message. Neither stage holds bytes back between calls.

That is the datagram contract, and on a datagram path it comes for free: one call per message, one message out. On a byte stream a chunk is an arbitrary slice, and hex packs one byte into two characters, so a chunk cut on an odd offset cannot be decoded on its own. Boundaries on a stream path have to come from frame and unframe, which have not landed yet; until they do, use these stages on datagram paths or on a peer that already sends one message per read.

A chunk with an odd number of digits is reported as a configuration error rather than relayed:

message has an odd number of hex digits: unhexify decodes one complete message per call, so it needs an unframe stage ahead of it

The mistake that cannot be caught is a chunk cut on an even offset, which decodes to a short payload. Hex has the shortest group of any codec here, so that is also the likeliest: half of all cuts land on one, against a quarter for base64. Framing is what rules it out.

Case

hexify writes lowercase by default. unhexify accepts both cases and mixtures of the two, whatever the encoding end is set to, so case never has to match across a hop the way base64’s alphabet does. Set it for the benefit of whatever is reading the wire.

Whitespace is not stripped. A trailing newline is a frame delimiter, and swallowing it here would paper over framing that is missing or misconfigured. Nor is a 0x prefix accepted, for the same reason: this is a wire codec, not a parser for hex written for people.

Size

Hex is a 2-to-1 expansion, exactly, with no padding: an encoded message is always twice the payload. On a datagram path that is the difference between fitting in a datagram and not, and a starker one than base64’s: a 1400-byte payload leaves as 2800 bytes, where base64 would send 1868.

limit - end the transfer after N bytes or N packets

Counts what passes its own position and, on reaching the limit, asks the host to stop reading.

$ tocat tcp:host:9000 limit,bytes=10MiB file:sample.bin
$ tocat udp-listen:9000 limit,packets=100 file:capture.bin
$ tocat -f tcp-listen:9000,fork -t tcp:backend:8080 -p 'limit,bytes=1M,direction=source'

Give one of bytes and packets. An entry with both is refused, and an entry with neither has nothing to count. To stop on whichever comes first, write two entries:

$ tocat udp-listen:9000 limit,packets=1000 limit,bytes=10MiB file:capture.bin

at-limit belongs to bytes and is refused alongside packets: a packet is counted only once it has passed whole, so no packet is ever cut.

Reaching the limit is upstream end of stream arriving early, not an error: bytes already emitted are written, the stages below get their end of stream, sinks are flushed and closed, and tocat exits successfully. The stop is logged at info:

limit: limit of 10MiB reached at 10MiB
limit: limit of 100 packets reached

A byte limit names both the limit and where the transfer actually ended, which differ under overshoot. A packet limit stops on the packet that reaches the count, so there is only one number to report, and it says packets whichever of the three spellings was written.

Anything still arriving after that (a chunk already in flight from a stage above) is dropped rather than announcing the limit again.

bytes=1MiB is a megabyte from the source. direction=both builds one instance per path, each with its own budget, so it is a megabyte in each direction rather than a megabyte between them, and packets=100 the same way. Position matters too: before a compress stage it caps the payload, after it caps the wire.

Drawing the limit

bytes and packets also take two forms that are settled when the pipeline is built rather than written down, for testing what a transfer does when it ends somewhere you did not pick.

$ tocat tcp:host:9000 limit,bytes=1KiB..1MiB file:sample.bin
$ tocat tcp:host:9000 limit,bytes=0.1% file:sample.bin
$ tocat udp-listen:9000 limit,packets=100..1000 file:capture.bin

A rate is per byte rather than per chunk, so it does not move with buffer-size or with how the peer happened to write: four writes of one byte and one write of four stop in the same place under the same seed. It has no fixed stopping point, so a short transfer often finishes untouched. That is the shape of the thing rather than a miss, and the way to use it is to run the same command many times.

Either form names the seed it came from in the halt line:

limit: limit of 651KiB reached at 651KiB (seed=1774028891573910000)

Give that number back as seed to draw the same limit again:

$ tocat tcp:host:9000 limit,bytes=1KiB..1MiB,seed=1774028891573910000 file:sample.bin

Without seed the number is taken from the clock and reported, so a run is reproducible after the fact as well as before it. A seed draws the same limit on any machine running the same tocat release.

The crossing chunk

Exactly one chunk straddles a byte limit, and there are three things to do with it.

exact is the default and is what a byte count usually means. drop is the hard ceiling: never put more than this many bytes into that file, that pipe, that quota. overshoot is the one to reach for on a datagram path, where a limit landing mid-message leaves a real choice: dropping throws away a message already received on a transfer that is ending anyway, while overshooting delivers it whole and then stops.

Splitting is also the only thing here that is unsafe on a datagram path, so the stage reports itself safe under drop and overshoot and unsafe under exact. Half a datagram is a corrupt message rather than a short read. A packet limit splits nothing and is safe on any path.

What counts as a packet

One chunk, meaning one delivery from whatever sits above the stage.

On a datagram path that is exactly one message, which is what the option is for: packets=100 on a UDP source takes a hundred datagrams and stops. Put the stage below an unframe to get the same thing on a byte stream, where the messages are the ones unframe found.

Everywhere else a chunk is one read, sized by buffer-size and by when the peer’s bytes happened to arrive. Two runs of the same transfer need not stop in the same place. That is still a well defined thing to ask for, so tocat counts it rather than refusing: reads are a usable proxy when what you want is “let a hundred deliveries through and stop”, and above a process stage they are the child’s writes. It is not a message count, and nothing in the config will pretend otherwise.

A count is written the way sizes are, suffixes and all, so packets=1k is 1024 chunks rather than 1000. It is reported as the plain number it is: a limit of 1k logs as limit of 1024 packets reached.

Nothing is copied under either limit. A chunk below a byte limit is passed through untouched and only the crossing one is ever materialised; a packet limit materialises none at all.

process - pipe through a subprocess

Hands this path’s bytes to a program’s stdin and takes its stdout back, so any filter that reads stdin and writes stdout becomes a stage.

$ tocat tcp-listen:9000,fork 'process:forward,command=gzip -c' tcp:backend:8080

[[plugin]]
name = "process"
direction = "source-to-sink"
argv = ["gzip", "-c"]
stderr = "log"

Give exactly one of argv or command; both, neither, or an empty one is a startup error. Only command works on the command line, since there is no way to write an array there (and a command containing a comma has to go in a config file, because commas separate options).

Filters that emit nothing until their input closes work correctly: tocat feeds the child and drains it concurrently, and closes its stdin at end of stream so it flushes. sort, tac and gzip all behave. A loop that wrote a chunk and waited for the reply would deadlock against exactly those filters, which is why the two halves run at once.

A non-zero exit fails that direction rather than being logged and ignored, since it means the bytes the child produced were incomplete or wrong.

Two exceptions to that shape are worth knowing. On a datagram source there is no end of stream, so the child’s stdin is never closed and it never flushes on its own: only a filter that streams its output is usable there. On a datagram sink the child’s output has no boundaries in it, so the messages sent are invented from wherever the reads land.

Cost

Two copies and two syscalls per chunk in each direction, plus a process per connection per direction. Under fork with direction = "both", sixty-four clients means one hundred and twenty-eight children. That is the correct price for “any tool becomes a stage”, but it should be paid knowingly: prefer a native stage for anything hot.

Because the child is driven concurrently rather than per chunk, process always has its own task and detach = false on one is rejected rather than ignored. It is a host plugin rather than an implementation of the Plugin trait for the same reason.

rate - measure throughput

Reports how fast bytes are moving past this point on the path. Like tee it never touches the payload, so it can go anywhere in a chain, including on a datagram path.

$ tocat -f tcp-listen:9000,fork -t tcp:backend:8080 -p 'rate,interval=30s'
$ tocat file:big.iso 'rate,as=plain' compress 'rate,as=wire' tcp:relay:9000

Each instance measures the traffic at its own position, so the two reports are the payload before compression and the bytes actually going on the wire.

interval takes a plain number of seconds (10, 0.5) or a suffixed string (500ms, 30s, 2m). This is the one place a fractional value is accepted, which is why the suffixes here are only ms, s and m rather than the full duration grammar.

Reports are logs, so they follow the log sinks and the log level. Every record a stage emits is logged under the plugin target with a stage field, so RUST_LOG=plugin=info selects plugin output and nothing else. The messages read:

10.2MiB/s (51.2MiB in 5.0s), 1.23GiB total
stalled, nothing in 5.0s, 1.23GiB total
transferred 1.23GiB in 0:02:04 (10.1MiB/s average, 11.8MiB/s peak)

The peak is the fastest window, so it appears only when there were windows to compare, which means not under interval=0.

With file= each interval writes one CSV row instead, with no header:

stage,elapsed,total,window,rate

elapsed is seconds since the first byte, total and window are byte counts, and rate is bytes per second for that window.

Reports are driven by the clock, not by chunks arriving, so they land on schedule and a stalled stream is reported rather than silent. The per-chunk cost is two adds and a branch; the first chunk stamps the start, so idle time waiting for a peer is not averaged into the transfer’s rate, and after that the data path never reads the clock at all.

A stall is announced once and not repeated (a connection that goes quiet for an hour should not say so seven hundred times) and the report after traffic resumes covers the whole gap. Samples written with file= are a time series rather than a narrative, so those are written every interval, zeroes included. A stage that has never seen a byte stays quiet either way.

A datagram source never reaches end of stream, so it never prints a summary, but its periodic reports work as usual.

Ticking costs one timer per direction per connection for any pipeline containing a ticking stage, which multiplies under fork. interval=0 is how you opt out of it and keep only the summary.

For throughput at the endpoints rather than at a point inside the pipeline, and for a live display rather than log lines, see --progress.

tee - mirror the stream

Writes a copy of everything on its path to a file or to stderr, without touching the payload. An entry with no direction watches the forward path only, so capturing both halves of a conversation means tee:both, or two entries when they should go to different files.

$ tocat tcp-listen:8080,fork tee,format=hex tcp:example.com:80
$ tocat tcp-listen:8080,fork 'tee:both,format=hex' tcp:example.com:80
$ tocat tcp-listen:8080,fork 'tee,file=req.bin' 'tee:reverse,file=resp.bin' tcp:example.com:80

Hex entries are headed with the hop and the stage, then the length and the offset within this path:

[tcp://example.com:80_10.0.0.4:52134 -> STDIO | audit] 4 bytes @ 0x0
00000000  70 69 6e 67                                      |ping|

Entries naming the same file share one buffered writer, so their output interleaves at chunk granularity rather than racing. Hex entries stay separable because each carries the connection and the stage name in its header; raw entries do not, so two raw tees on one file will produce something you cannot untangle.

tocat refuses to write a dump to stdout, which may be carrying relayed payload. A dump on stderr and --progress also do not mix, and tocat warns when both are asked for: the painter and the log writer coordinate over the line on screen, and a dump does not.

tee never materialises the payload: it passes the chunk through and copies into the host’s staging buffer, so inserting one costs a virtual call per chunk plus the dump itself. Its position decides only what it sees: before a compress stage it captures the payload, after it captures the wire.

throttle - constrict bandwidth

Holds the path to a ceiling by slowing the reader down.

$ tocat file:big.iso throttle,rate=256k tcp:host:9000
$ tocat -f tcp-listen:9000,fork -t tcp:backend:8080 -p 'throttle,rate=1MiB,burst=4MiB'

Nothing is buffered. Every chunk passes through untouched and the stage asks the relay to wait before reading again, so there is no queue to grow when the source outruns the limit. It also throttles the right end: a read that does not happen leaves the receive buffer full, which closes the TCP window and slows the sender at source. Buffering here would let the sender keep sending at full speed into memory that has to go somewhere, which is a queue, not a limit.

The allowance is a token bucket. It accrues at rate bytes per second up to burst, and each chunk spends its own length. Spending is not capped, so a chunk larger than the bucket is paid for with a proportionally longer wait rather than being split, which is what keeps the stage safe on a datagram path: a message goes out as the message it arrived as, just later. The bucket starts full, so a path that has just opened may move burst bytes before the ceiling first bites.

The wait lands between reads, so the stall comes in units of chunks. At the 256 KiB default buffer, rate=64k is four seconds of silence followed by 256 KiB at once. The average is right either way, but for smooth pacing give the relay a buffer at or below the per-second rate (-b 64k here).

The wait is also applied after the chunk in hand has been written, so nothing is held hostage by it. Where several stages on one path ask to wait, the longest request wins rather than the sum: two stages each asking for a second are satisfied by one second.

As with limit, an instance paces its own path, so rate=1MiB paces the source and direction=both is a megabyte per second each way rather than between them. To stop a stream rather than slow it, use limit; rate=0 is rejected.

timeout - end an idle path

Ends the transfer when nothing has crossed this point for a while.

$ tocat tcp-listen:9000,fork 'timeout,timeout=30s' tcp:backend:8080
$ tocat -f tcp-listen:9000,fork -t tcp:backend:8080 -p 'timeout:forward,wait=2m'

Reaching the timeout is upstream end of stream arriving early, not an error: bytes already emitted are written, the stages below get their end of stream, sinks are flushed and closed, and tocat exits successfully. The stop is logged at info by the host:

timeout: no data for 30s

Like tee and rate it never touches the payload, so it can go anywhere in a chain, including on a datagram path, and its position decides only what counts as activity: before a compress stage it watches the payload, after it watches the wire.

One clock per path

The window is per path and per connection, and by default that is the forward path. direction=both gives each path its own clock, which is usually not what an idle-connection reaper wants: a request/response protocol whose server thinks before answering has a quiet reverse path by construction, and ending it closes the write half back to the client. Pick the one path whose silence actually means the connection is dead.

The clock starts when the path opens rather than at the first byte, so a connection that is accepted and then says nothing is ended after one window. That is the point of the stage, but it does mean the timeout has to be longer than the peer’s think time, not just longer than its gaps.

Ending one path does not by itself end the run: the other direction is still being pumped. In practice the closed write half is what tells the peer to go away, and the other path then reaches its own end of stream, but a peer that ignores that keeps the connection open until it too times out. On a path that never reaches end of stream anyway (a datagram source, a held pipe) this stage is the only thing that will ever end it, which is one good reason to reach for it there.

A forked udp-listen is the case where direction=both is the right answer rather than the unusual one. A session there has no close to observe on either side, and a datagram sink has no write half to shut down, so a forward-only halt leaves the reverse path pumping forever. It is the session task finishing that hands back the connection permit, so without a clock on both paths the sessions accumulate until the ceiling is full.

$ tocat udp-listen:9000,fork 'timeout:both,timeout=30s' tcp:backend:8080

Accuracy and cost

The halt lands within a quarter of the timeout of where it was asked for, and never early. The stage asks the host for a tick four times per window and counts consecutive idle ticks, rather than asking for one tick per window: the host’s timer runs at a fixed cadence, so a single-tick implementation would fire anywhere between one and two windows after the last byte. Below 400ms it stops subdividing, so a very short timeout is coarser (up to twice what was asked) rather than waking every forked connection every few milliseconds.

That means four wakeups per window per direction per connection, which multiplies under fork, plus one clock read per chunk to restart the window. Both are cheap, but a one-second timeout across a thousand connections is four thousand wakeups a second, and the answer there is a longer timeout rather than a tighter one.

wasm - run a WebAssembly guest

Runs a .wasm module as a stage. The guest sits where it was written, takes a direction, and forwards or rewrites what it is handed like any other stage.

$ tocat tcp-listen:8080,fork 'wasm,module=/usr/lib/tocat/redact.wasm' tcp:example.com:80

[[plugin]]
name = "wasm"
module = "/usr/lib/tocat/redact.wasm"
direction = "source-to-sink"
config = { patterns = ["authorization", "cookie"] }

config is the guest’s own options, and the guest validates them and writes its own error messages. In a config file it is a table, which is the readable form. On the command line it is a JSON string, and since commas separate options there, anything with more than one key has to go in a config file.

What a guest cannot do

A guest imports nothing at all. No clock, no files, no sockets, no host functions of any kind, and no way to spawn anything. That is not a promise the documentation makes, it is a property the loader enforces: a module that imports so much as WASI is refused when it is loaded, naming the import.

It does not need any of them. A stage decides what to forward and queues everything else for the host to perform, so the things a guest would reach for are already messages rather than calls: it can forward bytes, frame them into units, log, ask to wait, ask to stop, and ask for its tick schedule to be restarted. The one capability the ABI does not carry yet is side channels, so a guest that wants to record something logs it. See The guest ABI for how to write one, crates/tocat-wasm-sdk for writing one in Rust, and sdk/wasm/ for a C and C++ SDK. Working guests for all three are in the repository.

Two consequences worth stating plainly. A guest cannot exfiltrate the payload, because it has nowhere to put it. And a guest cannot be a host plugin: spawning is a host capability by construction, so a subprocess stage will always be native.

Cost

A guest is expensive enough per byte that it defaults to detach, running on its own task. detach=false still works for one cheap enough not to want its own.

The module is compiled once per process, at startup, however many stages and connections name it. What is per stage is an instantiation and a linear memory, and a stage is per direction per connection: under fork with the default direction=both, a hundred clients is two hundred instances. The memory ceiling is the number that multiplies, so memory-max=64MiB against a thousand connections is a promise you may not want to make.

Fuel is per call, not per connection, so the budget is “how much work may one chunk cost”. A guest that runs out is trapped, which fails that direction: the bytes it was handed have gone nowhere, so there is nothing else honest to do. The default is generous enough for any straightforward per-byte transform of a full buffer and tight enough that a runaway loop is caught in milliseconds. fuel=0 turns metering off, and with it the guarantee that a guest cannot hang the relay: on_bytes runs on the copy task, and nothing else on that task moves while a guest is inside it.

When a guest does nothing

A guest that loads, runs, and silently drops the stream is almost always handing the host offsets into its own arena rather than addresses in its linear memory. Nothing traps, because both sides read memory that exists, and the outbox the host reads is whatever happens to be at the address the guest named, which is usually zeros, which decodes as “drop everything”. See Pointers are absolute.

Run the stage at -v and the resolved chain will confirm it is where you think it is. Beyond that, a guest is easiest to debug outside the relay: load the module, call the exports by hand, and decode the outbox after each call.

Datagrams

A guest declares for itself whether it preserves message boundaries, and one that says nothing is assumed not to, which is the same default every native stage has. A guest that emits one unit per call it was given one is safe; one that buffers across calls or reframes what it was handed is not, and gets the usual warning when the destination on its path is a datagram endpoint.

Buffers

tocat copies through one buffer per direction per connection, 256 KiB by default. -b on the command line or buffer-size in the config file changes it, as a byte count or with a binary suffix.

$ tocat -b 1MiB file:/big.iso tcp:host:9000

buffer-size = "64k"

Buffers are page-aligned and allocated whole pages at a time, though the extra is never exposed: asking for 1000 bytes gives a 1000-byte slice.

The buffer size is the largest chunk a stage can be handed, so it also sets the granularity of anything acting between reads. That is why throttle paces in units of chunks, and why a block size far below the buffer runs the rest of the pipeline many times per read. On a datagram path it is also the message ceiling: one receive is one datagram, and anything longer is truncated.

Under fork this multiplies, since every connection has its own buffer each way. tocat warns at startup when the buffer size and the connection ceiling together allow more than a gibibyte of copy buffers, which -b 1MiB against the default ceiling of 1024 already does.

Kernel pipe buffers

Kernel pipe buffers are a separate resource, and they cap throughput independently: a pipe holds 64 KiB by default, so a writer stalls once that much is unread however large tocat’s own buffer is. Where the pipe belongs to tocat it is enlarged to match the copy buffer automatically:

• stdin and stdout, whenever they turn out to be pipes, which is what tocat … | pv makes of fd 1
• the stdin and stdout of exec:, system: and process children, which always are

Named FIFOs are left alone unless you ask with size=, because their buffer is shared with whoever else has the path open.

Pipe resizing is Linux-only and best-effort, and it is done by asking rather than by checking first. The kernel rounds up to a power of two; a descriptor that is not a pipe, or one with more buffered than the new size would hold, is left alone and noted at debug. The one case reported louder is refusal to exceed /proc/sys/fs/pipe-max-size (1 MiB by default) without CAP_SYS_RESOURCE, which warns and carries on at the existing size.

Progress

--progress draws a pv-style line on stderr while the relay runs.

$ tocat --progress file:big.iso tcp:host:9000
  1.23GiB 0:00:12 [   102MiB/s] [=============>        ]  42% ETA 0:00:16

The line is counts, elapsed time and rate, then a bar, percentage and ETA where those are knowable. It counts bytes at the endpoints, before any stage sees them, and it is redrawn on a timer (ten times a second) rather than when bytes arrive, so a stalled transfer is visibly stalled rather than frozen. The displayed rate is smoothed over roughly a second.

--progress on its own means auto: draw only when stderr is a terminal. --progress=always draws regardless, which is what you want when stderr is redirected to a file. --progress=never is the default: a relay is as often a daemon as it is a command. The same values work in the config file.

progress = "auto"

The bar, the percentage and the ETA need a total, which tocat only has when the source is a regular file: and neither endpoint forks. Everything else gets the counts, the elapsed time and the rate. Traffic coming back from the sink is reported separately once there is any:

1.23GiB out  340MiB in 0:00:12 [   102MiB/s]

Under fork the line aggregates every connection, and shows the count once more than one is open. An ETA longer than 99:59:59 is not an estimate, so it reads --:--:--.

When the relay finishes, the line is erased and replaced with one summary line, unless nothing moved at all:

1.23GiB in 0:00:12 (102MiB/s)

Two things share stderr with the display. Logs are handled: an event erases the line, prints, and the next tick redraws it, all under the same stderr lock, so a record cannot land in the middle of a frame. A tee pointed at stderr is not, and tocat warns when the two are used together. Send the dump to a file, or drop the flag. No ANSI is involved anywhere: erasing is a carriage return, spaces and another carriage return, which is what makes --progress=always readable in a file.

Measuring is not free. A relay with no plugins and stream sockets on both sides is normally handed to copy_bidirectional_with_sizes, and counting bytes needs a per-chunk hook that call does not offer, so --progress puts that case on the general copy path instead. See The data path.

For throughput at a point inside the pipeline rather than at the endpoints, see the rate plugin. The two answer different questions: --progress is what the transfer is doing, rate is what a particular stage is seeing.

Configuration

tocat looks for a tocat.toml, then a .tocat.toml, in the working directory, and merges what it finds with the command line, which wins. --config PATH selects a file explicitly, --no-config skips the search entirely, and --dump-config prints the merged result as TOML and exits.

Below is a configuration that accepts up to 64 connections on port 9000, connects via TCP to another server, and keeps a raw log of the data coming back.

#:schema ./tocat.schema.json

log-level = "info"
buffer-size = "256KiB"

[source]
type = "tcp-listen"
host = "0.0.0.0"
port = 9000
fork = true
max-connections = 64

[sink]
type = "tcp"
addr = "backend.internal:8080"

[[plugin]]
name = "tee"
direction = "sink-to-source"
file = "/var/log/tocat-capture.bin"
format = "raw-binary"

Endpoints can also be written as the string shorthand from the command line. The two forms parse to the same thing, so an option means what it means in either.

#:schema ./tocat.schema.json

source = "tcp-listen:9000,fork"
sink = "tcp:localhost:8080"

Keys

Unknown top-level keys are an error rather than being ignored, as unknown options are everywhere else.

Merging

Command-line values replace file values field by field: an endpoint given as a flag or a positional replaces the file’s, -b replaces buffer-size, -P replaces progress.

Plugins are the exception to “replace”: they accumulate. The order is the file’s entries, then the inline positional pipeline, then every -p, so a standing capture can live in tocat.toml while an ad-hoc stage is added for one run.

[[plugin]]
name = "tee"
as = "wire"
file = "session.hex"
format = "hex"

[[plugin]]
name = "compress"
direction = "forward"
level = 9

--no-plugins drops the file’s list and keeps the -p entries, which is how you run a standing configuration without its capture for one invocation. It cannot be combined with --no-config, which has already dropped everything.

Note that tocat.toml and .tocat.toml are in the repository’s .gitignore, since they are how the project’s own ad-hoc runs are configured.

Schema

tocat.schema.json is a JSON schema for the configuration file format. If you use tombi (in the dev shell) you get completion, hover documentation, and validation in your editor. Either add the directive comment shown above, or register it in tombi.toml:

[[schemas]]
path = "tocat.schema.json"
include = ["tocat.toml", ".tocat.toml"]

If tocat was compiled with the schema feature (on by default), use the --dump-schema flag to print the schema to stdout.

$ tocat --dump-schema > tocat.schema.json

The schema describes every endpoint scheme and every plugin that ships with tocat, one complete variant each rather than a shared base plus conditional fragments, because editors only offer completions from the schema they resolve. It also accepts entries for plugins it does not know about, which is necessary since the plugin set depends on the features the binary was built with.

tocat itself is the real validator: it rejects unknown options, unknown plugins and contradictory ones at startup, before either endpoint is opened. The schema is a convenience on top of that, and needs updating alongside any option that is added or renamed.

Logging

Logs go to stderr by default, in a compact format. Verbosity can be set by repeating -v (once for debug, twice for trace), explicitly with --log-level, or with log-level in the config file. An explicit --log-level wins; otherwise the higher of -v and the config file’s level is used.

$ RUST_LOG=tocat=trace,tokio=warn tocat tcp-listen:9000 -

RUST_LOG is applied on top of the default stderr sink, and can select per target. Two targets are worth knowing: tocat for the relay itself and plugin for records emitted by stages, each of which carries a stage field naming the instance. So RUST_LOG=plugin=info gives you rate reports and process diagnostics without the rest.

$ RUST_LOG=plugin=info tocat -f file:big.iso -t tcp:host:9000 -p 'rate,interval=1s'

Additional sinks can be configured in the config file. Any number of [[log]] entries are composed into one subscriber, each with its own format and level.

[[log]]
type = "stderr"
format = "compact"

[[log]]
type = "file"
path = "/var/log/tocat.log"
format = "json"
level = "debug"
rotation = "daily"
max_files = 7

Declaring any [[log]] entry replaces the default stderr sink rather than adding to it, so include one of type = "stderr" if you want both. Note that RUST_LOG is not applied to sinks declared this way: each takes its own level (or the global one) and nothing else.

Under rotation the path is split into a directory and a filename prefix, and file writing is non-blocking, with the worker flushed at exit.

Two other things share the stderr stream. Log records erase the --progress line before printing and let the next tick redraw it, under the same lock. Traffic dumps are not logs at all: they are written by the tee plugin, go wherever that entry points, and are unaffected by the log level.

Overview

Plugins implement the Plugin trait from the tocat-api crate. A plugin is a synchronous byte transformer: it is handed a chunk that arrived from upstream and decides what to forward downstream. Anything that touches the outside world (writing a dump file, emitting a log line, waiting) is not performed by the plugin. It is queued as an effect and applied by the host after the call returns.

Three properties follow from that split, and they are the reason for it:

• Testable. A stage is a function from a chunk and its state to bytes out plus a list of requested effects. A test drives it with chunks and asserts on both, with no runtime, no sockets and no temporary files. See Testing a stage.
• Off the async runtime. Nothing in a stage awaits, so no stage can stall a reactor thread by accident, and all I/O stays on the host’s runtime where it can be batched and overlapped with the downstream write.
• Portable to a guest. It is the shape a WASM guest has to take, and the wasm stage is that: it implements Plugin by forwarding each call into a module and applying what comes back, and nothing in the relay knows the difference. Because effects are queued rather than performed, a guest needs no host imports at all, which is what makes “imports nothing” an enforceable rule rather than a wish. See The guest ABI.

The same split covers time. A stage cannot await and cannot read a clock, so one that needs time rather than traffic to drive it declares a period and is called back on it. See Ticks and timers.

A few stages cannot satisfy that contract at all. A subprocess decides nothing synchronously and may emit bytes belonging to chunks it was given long ago, so it is described to the host and run by it. See Host plugins.

Vocabulary

The crate

tocat-api depends only on serde and serde_json. It has no tokio, no I/O and no knowledge of the relay, which is what lets a plugin be built and tested on its own.

Where to go next

• The Plugin trait for the calls a stage receives and what it may do in them.
• Options and building for the factory, config deserialization and registration.
• Units and boundaries for controlling how what a stage emits is delivered.
• Ticks and timers for stages driven by time rather than by traffic.
• Effects and channels for reaching the outside world.
• Host plugins for stages the trait cannot express.
• Testing a stage for driving one without a relay.

The Plugin trait

pub trait Plugin: Send {
    fn name(&self) -> &str;

    fn on_bytes(&mut self, ctx: &mut Ctx<'_>, input: &[u8]) -> Result<()>;

    fn on_eof(&mut self, ctx: &mut Ctx<'_>) -> Result<()> { Ok(()) }

    fn tick_interval(&self) -> Option<Duration> { None }
    fn on_tick(&mut self, ctx: &mut Ctx<'_>) -> Result<()> { Ok(()) }

    fn boundaries(&self) -> Boundaries { Boundaries::Fuse }
    fn needs(&self) -> Needs { Needs::Nothing }
}

Instances are per direction and per connection. Under fork that means a fresh set for every accepted client, and a both entry means two instances, one per path, so per-direction state (byte offsets, codec state) never leaks across paths.

input is the same slice as ctx.input(); it is passed separately because it is the hot argument.

Deciding what happens to the chunk

Every call must say what becomes of the bytes. Ctx offers three answers and one qualifier:

Passthrough is the fast path and costs nothing: the pipeline hands the original read buffer down the chain and ultimately to the socket. Mixing the two is allowed and does the obvious thing, at the price of materialising the input: pass_through followed by forward copies the input first so the order is preserved.

Forwarding nothing is a legitimate steady state, not an error. block does it for every chunk that does not fill a block.

What a stage must not do is block. No socket, no file, no sleep, no lock held across a call. Everything that needs the outside world is requested through ctx and performed by the host: see Effects and channels.

An error returned from any call fails that path rather than being logged and ignored, on the grounds that a stage which cannot process the bytes it was given has produced an incomplete or wrong stream. Use PluginError::config for anything wrong with the entry and PluginError::runtime for anything that goes wrong mid-stream.

End of stream

on_eof is the last call a stage receives, and the last chance to emit: a codec writes its epilogue here, a buffering stage releases what it holds. Anything emitted continues downstream through the stages below, which then receive their own on_eof in turn, so a chain of buffering stages drains from the top down in one cascade. ctx.input() is empty, so there is nothing to pass through.

End of stream is a normal event rather than an error, and it can also arrive early: ctx.halt() is how a stage ends a transfer deliberately (this is all limit does), and the path then closes down exactly as it would have anyway.

Some paths never reach it. A datagram source has no end of stream, and neither does a held pipe. A stage whose only output happens at on_eof therefore produces nothing at all on such a path, which is why rate reports on a timer as well as in a summary.

Boundaries

boundaries says what the stage does to the messages passing through it, and needs says what it requires of the path it was placed on. Both are read once after build and never on the per-chunk path.

Fuse is the default because it claims nothing, which is the safe answer for a stage that has not thought about it, including any plugin loaded from outside the binary. A pure observer that only calls pass_through can say Preserve. Anything holding bytes across calls, emitting on a tick, or reframing what it was given should not, even when doing so is the whole point of the stage: the host warns and relays anyway, so the honest answer costs nothing.

The difference between the two methods is the difference between a warning and an error. boundaries is advisory, because rewriting the message stream is sometimes exactly what was asked for. needs is not: a stage saying Needs::Upstream cannot read what arrives unless every call carries one whole message, and one saying Needs::Downstream cannot have what it wrote read back unless the units it emitted survive. Neither is a matter of taste, so an unmet requirement fails the build.

The two sides are separate because the stages that want them want opposite ones. A stage that seals a message and appends a tag makes its own boundaries and needs them to survive downwards; the stage that verifies and strips that tag needs whole messages from above and does not care what happens below it.

The host answers a requirement by walking away from the stage until something settles it. A datagram endpoint on that side supplies boundaries, a frame below satisfies a downstream requirement however many stages fuse under it, an unframe above satisfies an upstream one, and the first stage that carries neither is named as the cause.

Options and building

A plugin is registered as a factory, and the factory is what the host talks to when it resolves an entry.

pub trait PluginFactory: Send + Sync + 'static {
    fn name(&self) -> &str;
    fn description(&self) -> &str { "" }
    fn execution(&self) -> Execution { Execution::Inline }
    fn build(&self, ctx: &mut BuildCtx<'_>) -> Result<Stage>;
}

name is the spelling users write and --list-plugins prints; the registry stores it normalized, so case, dashes and underscores do not matter at the call site. description is the one line beside it in that listing.

execution is the default placement: Inline (the default) runs the stage on the reading task, Detached gives it its own task behind a bounded channel, which buys concurrency with the reader for one copy and one wakeup per unit. An entry’s detach = true|false overrides it. Only reach for Detached when the stage does real work per byte, as compress does.

build runs once per direction per connection. It is where everything expensive or fallible belongs, so that the per-chunk path is a synchronous call that either forwards a slice or writes into a buffer.

BuildCtx

StageInfo::upstream and downstream are the stage’s actual neighbours on this path: the adjacent stages’ display names, or an endpoint label at either end. A tee wedged between two other stages therefore describes the hop it is really watching rather than the endpoints it is nowhere near. label() on either type formats the pair as upstream -> downstream, oriented for this path.

Position cannot change after construction, so anything derived from it should be computed here and cached. So should the answer to tick_interval, which the host reads exactly once, at the end of construction.

Config types

Declare a config struct and deserialize it. Nothing about tocat’s lenient matching appears in the plugin:

#[derive(Debug, Clone, Deserialize)]
#[serde(rename_all = "kebab-case", deny_unknown_fields)]
pub struct BlockConfig {
    #[serde(default = "default_size")]
    pub size: ByteSize,
    #[serde(default)]
    pub flush: Option<Interval>,
    #[serde(default)]
    pub pad: bool,
}

ctx.config() deserializes through Forgiving, which matches option keys and enum values the way every other identifier in tocat is matched: case insensitively, with dashes and underscores treated as noise. deny_unknown_fields is worth setting on every config type, since it is what turns a misspelled option into a startup error instead of a silent difference in behaviour. A #[serde(alias)] still works, because a candidate that matches no declared field is passed through untouched for serde to resolve.

Use ByteSize and Interval from tocat-api for sizes and durations rather than rolling your own. Both accept a number or a string, which matters: the command line coerces values to integers where they parse as one, so size=4096 arrives as a number and size=4k as a string.

Validate in build and fail there. block rejects a zero size, throttle a zero rate, compress a level outside 1 to 22, and process an entry giving both argv and command or neither. All of them are startup errors, before an endpoint is opened.

Registration

build returns a Stage, which is either a Filter wrapping a Plugin or an External describing a subprocess:

Ok(Stage::filter(Block { buf, size, flush, pad }))

Plugins compiled into the binary are modules of tocat-plugins, each behind a cargo feature, and registered in one place:

#[cfg(feature = "block")]
mod block;

#[cfg(feature = "block")]
registry.register(block::BlockFactory);

Adding one is a module, a feature, and a line in register_native, plus a matching feature in the tocat crate that forwards to it. A plugin with its own dependency tree needs no crate of its own for that: an optional dependency enabled by the same feature keeps it out of a build that did not ask for it, which is how compress gets zstd and wasm gets wasmtime.

The facade exports nothing but register_native, so no module can reach into another and the binary cannot reach into any of them. That is the property worth keeping, rather than any particular arrangement of crates.

Units and boundaries

Whatever a stage emits in one call is delivered as one unit, however many times it forwards: the pieces concatenate and travel together. A stage that needs them kept apart calls ctx.boundary() between them, and each unit then becomes

• one write at a byte sink,
• one message at a datagram sink,
• one parcel across a detach boundary,
• and one on_bytes call at every stage below.

That is the difference between a stage that merely accumulates bytes and one that records them. block forwards its block and then calls boundary; nothing else that ships with tocat calls it at all.

Unit counts multiply down a chain, which is why this is an explicit request rather than something inferred from a stage emitting more than once. A stage that forwards a header and then a body has not asked for two writes and should not silently get them.

Two conveniences worth knowing. The trailing unit needs no boundary: whatever is forwarded after the last one is closed automatically, so no bytes can be lost by forgetting. And a boundary with nothing forwarded since the last one is ignored, so a stage cannot emit an empty unit by accident.

What it costs

Passing through stays free underneath a framing stage. A stage handed framed bytes is called once per unit, and one that hands every unit back untouched copies nothing and still answers passthrough, exactly as it would on an unframed chunk. The copy starts at the first unit it rewrites, drops or reframes, and even then the pipeline ping-pongs between two buffers it owns rather than allocating.

What does cost is the call count. Every stage below a boundary runs once per unit rather than once per chunk, so block,size=512 against a 256 KiB buffer runs the rest of the pipeline 512 times per read, and a detach below that turns each unit into its own task wakeup. Ask for boundaries when the splits are the point, which is what a tape drive, a raw device, an O_DIRECT sink or a datagram peer needs, and not otherwise.

A byte sink is the one destination that does not pay: it has no framing of its own, so the whole emission goes out in one write. The peer cannot tell one call from several and one syscall is cheaper.

Where a unit shows up

The host sees an emission as Emitted, which is the bytes plus the offsets that frame them. An empty boundary list means one unit covering everything, which is the shape of every chunk off the wire, so the common path allocates nothing and every destination can ignore units entirely.

Boundaries and datagrams

A stage declares through boundaries what it does to the messages passing through it, and through needs what it requires of the path. boundary is what makes the useful case sayable rather than silent: a stage that emits several units emits several messages. That is still a rewrite of the peer’s message stream rather than a preservation of it, so a stage doing it reports Fuse and lets the host decide whether to warn.

Declaring the truth matters more than declaring safety. block fuses and says so, and it is still the right stage to reach for when one datagram per 1400 bytes is exactly what you want. See The datagram model.

Ticks and timers

A stage that needs time rather than traffic to drive it implements tick_interval and on_tick. The host asks once, at the end of construction, how often the stage wants calling, and then calls it on that schedule for as long as the pipeline lives, whether or not bytes are moving. Without it a stalled stream and a finished one are indistinguishable from inside a plugin.

ctx.input() is empty during a tick, so there is nothing to pass through: anything emitted is forwarded, and it continues downstream through the stages below this one, the same way on_eof output cascades. The stages above are upstream of a chunk that did not come from them. Emitting nothing is the common case and costs nothing. Whatever is emitted is one unit unless ctx.boundary() says otherwise, exactly as in on_bytes.

tick_interval is read exactly once, so it must not depend on anything that changes later. A stage whose interval is configurable reads its config in build and answers from that. A zero period reads as “no ticks”, which is the same answer as None and is what a stage configured with an interval of zero means by it.

Cadence, not delay

The host owns the clock, because a guest cannot reach one. What a stage gets is therefore a cadence rather than a delay: a tick that came due while bytes were flowing fires at the next opportunity, which can be immediately after the bytes it is about arrived.

A stage that means “an interval after I started waiting” says so with ctx.rearm(), which restarts its own schedule from now. It is cheap and harmless to call often: it sets a flag the host reads once at the end of the call, and it is ignored for a stage that asked for no ticks. A tick may rearm too, which is how a stage that has just given up waiting says so.

That is how block turns flush into a latency bound it can actually promise: it rearms when a block starts filling, so the interval is measured from the first byte held rather than from wherever a shared cadence happened to be.

What it costs

The pipeline owns the schedules, one deadline per ticking stage, and the host owns a single timer that asks whether any of them are due. The timer runs at the shortest period any stage in that segment asked for; a stage that wanted a longer one is simply not due on most wakeups, which is cheaper than a timer each. A segment with nothing ticking builds no timer at all and awaits its read directly.

So the cost is one timer per segment per direction per connection, which multiplies under fork: a stage asking for milliseconds is asking every forked connection to wake up that often. A stage should return None when its options do not require ticking, which is what rate,interval=0 does.

A stage that needs a deadline rather than a cadence needs both halves: rearm so the count starts from the last byte, and a period some fraction of the deadline so that the host’s cadence rounds the answer by that fraction rather than by the whole thing. timeout asks for a quarter of its window and halts on the fourth consecutive idle tick, which is what bounds its error at a quarter rather than at one whole window.

Two details of the host’s timing are worth relying on. Reads win the race against the timer, so payload never waits behind bookkeeping. And a segment that fell behind does not then fire a burst of catch-up ticks: a missed schedule resumes from now, and a stage that wants to know how long it was away measures that itself.

Effects and channels

A plugin never opens a file, never sleeps and never touches a socket. It asks, and the host does it after the call returns. Everything is staged per call and applied in one pass, which is what lets the flush overlap the downstream write.

Side channels

At build time a stage describes the sink it wants and receives an opaque ChannelId; at run time it queues writes against that id.

let channel = ctx.open_channel(ChannelTarget::file("session.hex"))?;

The host owns the descriptor and de-duplicates identical targets, so two entries naming the same path share one buffered writer and their output interleaves at chunk granularity rather than racing. Each sink has its own lock, so two directions contend only when they genuinely target the same file. A side write costs one extend_from_slice into a staging buffer that is reused across chunks, so there is no allocation and no lock inside the plugin call.

ChannelTarget is Stderr or File { path, append }. There is deliberately no stdout: on a stdio endpoint that stream carries relay payload, and the host refuses a file target that resolves to it. stderr is written unbuffered, since it is shared with the log writer and a dump stranded behind a partial line would be worse than the syscall.

Channels are opened once for the whole process, before any endpoint is touched, so an unwritable dump file is a startup error rather than a surprise on the first byte. That also means a stage cannot ask for a new target after startup.

Logging

ctx.log takes a LogLevel and a message. The host emits it under the plugin target with a stage field, so a user can select plugin output with RUST_LOG=plugin=info and can tell two instances apart by the name as gave them. Do not repeat the stage name in the message.

Pacing and halting

pace is the entire throttling mechanism: the host simply does not read for that long. Nothing is buffered, and on a socket the stalled read closes the receive window and slows the peer at source. It is applied after whatever was emitted has been written, so the bytes in hand are never held hostage by the wait. Where several stages ask on one chunk, the longest request wins rather than the sum.

halt ends the transfer deliberately. Everything already emitted is still written, on_eof still cascades, and the path closes down its normal way, so the relay exits successfully and the reason is logged at info. It is a decision, not a failure, and must not be reported as one. Where several stages ask, the first wins, which is the one nearest upstream.

Both default to doing nothing on the EffectSink trait, so a host with no reader to hold (a test harness, an offline driver) is not obliged to honour them.

Host plugins

Some stages cannot be expressed as a synchronous byte transformer. A subprocess decides nothing per chunk, may emit output belonging to chunks it was handed long ago, and has to be fed and drained at the same time or a filter like sort deadlocks against it.

Rather than bend Plugin into something a subprocess could satisfy, and lose the property that makes it portable, such a stage is described to the host and run by it. build returns the other variant of Stage:

Ok(Stage::External(ExternalStage {
    argv,
    shell,
    stderr: config.stderr,
    name: ctx.stage().name.to_string(),
}))

The plugin never sees a byte. It validates the entry, decides whether the command line is a program with arguments or a shell string, and names itself for logs and for attributing the child’s stderr. The host owns the process, the pipes, the tasks, the concurrent feed and drain, the stdin close at end of stream, and the exit status.

Why the split exists

• The trait stays honest. If it had to cover concurrent, latency-shifted stages, every plugin author would face a contract shaped by a case they do not have, and the guarantees the simple case relies on would be weaker.
• Capability is structural. Spawning is a host capability by construction, so a WASM guest can only ever be a filter. A guest cannot spawn a process because there is nothing in its interface that spawns a process, rather than because it is asked not to, and the loader refuses a module that imports anything at all.
• The host already owns the awkward parts. Pipe sizing, killing the child when the connection drops, closing stdin so the child flushes, and mapping a non-zero exit onto a failed direction are host responsibilities anyway.

What it still shares

From the user’s point of view an external stage is an ordinary pipeline entry: it takes a direction, it takes as=, it appears in the resolved chain and in logs the same way, and it sits where it was written. What differs is that it is always its own segment, so detach = false on one is rejected rather than ignored, and that it can never carry datagrams: its stdin and stdout are byte streams, so message boundaries are gone the moment bytes cross the pipe.

The one shipped example is process. Its cost model is worth reading before adding another: two copies and two syscalls per chunk each way, plus a process per connection per direction.

The guest ABI

A WebAssembly guest is a plugin like any other: the wasm stage implements Plugin by forwarding each call into the module and applying what comes back. This page is the guest half of that contract, ABI version 2.

The whole ABI is one call in and one struct out. A guest imports nothing. There are no host functions to link against, which is why a module built against WASI is refused at load time. That is not a sandbox bolted on afterwards: the trait already says that a stage decides what to forward and queues everything else for the host, so there is nothing left for a guest to import.

Exports

tocat_alloc is an arena rather than a heap: the host never frees, writes exactly len bytes, and calls it again on the next chunk. One static buffer, grown when a chunk does not fit, is a complete implementation. Returning 0 refuses the chunk, and the host fails that direction saying so.

Both tocat_tick_interval_ns and tocat_boundaries are read once, after tocat_init, because that is when the guest knows its options and because the host reads them once too. A guest that asks for a tick without exporting tocat_on_tick gets no timer.

Declaring boundaries

tocat_boundaries packs two answers into one word. Bits 0 and 1 say what the stage does to the message boundaries passing through it, and bits 2 and 3 say what it needs of the path it was placed on.

A guest that does not export the function answers zero, which claims nothing and asks for nothing. Any bit outside those two fields is a guest built against a later ABI than the host speaks, and the host refuses it at load rather than reading it as a claim of nothing.

The effect is a warning and the requirement is an error. The host warns when a fusing stage sits on a path whose destination is a datagram endpoint, since the operator may know the peer tolerates it; an unmet requirement fails the build, since the stage cannot work where it was put whatever the peer tolerates.

Pointers are absolute

Every pointer crossing this ABI, in both directions, is a byte offset into the guest’s linear memory as a whole. Not an offset into a buffer, and not an offset into whatever static the guest is using as its arena.

This is the one mistake the ABI invites, because a guest written around a static array reads naturally with offsets into that array:

static mut ARENA: [u8; 131072] = [0; 131072];

pub extern "C" fn tocat_outbox() -> i32 {
    0                                   // wrong: 0 is the start of memory,
}                                       // not the start of ARENA

The linker decides where ARENA lands, usually somewhere above 1 KiB of data segments, so a guest doing that writes its outbox into ARENA[0..48] while the host reads address 0, and reads its input from ARENA[64..] while the host wrote to address 64. Every pointer is wrong by the same constant, and nothing traps: both sides are reading memory that exists.

The symptom is an outbox that decodes as all zeros, which is emit = 0, which is a stage that drops everything. If a guest that should be rewriting the stream is silently dropping it, this is the first thing to check.

The fix is to add the arena’s own address to every offset, and to treat the pointer the host hands to tocat_on_bytes as already absolute:

fn base() -> usize {
    &raw const ARENA as usize
}

pub extern "C" fn tocat_outbox() -> i32 {
    (base() + OUTBOX) as i32
}

A module written in WAT tends not to hit this, because a (data (i32.const 0) ...) segment really is at address 0. A module compiled from a language with a data layout almost always does.

The outbox

After every call the host reads a fixed-layout struct at tocat_outbox() and applies it. It is little-endian and 48 bytes long, so decoding is a handful of loads and no allocation. Every pointer in it is an offset into the guest’s own memory.

Log levels are 0 trace, 1 debug, 2 info, 3 warn, 4 error.

Passthrough is the cheap answer and the host does not read the guest’s bytes at all: nothing is copied in either direction, exactly as for a native stage. Boundaries mean what they mean everywhere else, so read Units and boundaries before emitting more than one.

Two flags end the path and differ only in whether that is a success. halt is Ctx::halt: upstream end of stream arriving early, so stages below are drained, sinks are flushed, and tocat exits successfully. error fails the direction, and is also how tocat_init rejects an option, which is what makes a bad guest option a startup error carrying the guest’s own message.

A guest that forgets to reset the outbox between calls repeats itself rather than corrupting anything, which is the failure mode worth having. A guest that hands back a pointer outside its memory gets a runtime error naming the span: linear memory is the whole of what it can reach.

Where the ABI is defined

crates/tocat-wasm-abi is the one definition of the wire format, and everything that touches it reads that crate rather than its own copy: the host decodes an outbox with it, tocat-wasm-sdk writes one with it, and sdk/wasm/include/tocat/abi.h is generated from it.

$ ./scripts/regen-abi.sh
$ cargo run -p tocat-wasm-abi --example tocat-abi-header -- --check

The generator is an example rather than a binary so that cbindgen stays a dev-dependency and never appears in what a consumer of the crate resolves.

The header is committed so that a C guest builds without a Rust toolchain, and --check is what stops a committed generated file drifting: it writes nothing and exits non-zero when the two disagree. Everything a generator has no opinion about, which is the exports, the arena and the helpers, stays hand-written in tocat.h.

The crate asserts its own layout, so a change to the struct that moves a field fails to compile rather than becoming a stage that reads nonsense. Every wire value has two spellings on purpose: TOCAT_EMIT_BUFFERED is what C sees and Emit::Buffered is what Rust sees, and they cannot disagree because the enum’s discriminants are the constants.

Guests in Rust

crates/tocat-wasm-sdk is the Rust equivalent of tocat.hpp: a guest is a type implementing Guest, and export_guest! generates the arena, the outbox, the panic handler and all nine exports.

pub struct Upper {
    out: [u8; CAPACITY],
}

impl Guest for Upper {
    const INIT: Self = Self { out: [0; CAPACITY] };
    const BOUNDARIES: Boundaries = Boundaries::Preserve;

    fn on_bytes(&mut self, ctx: &mut Context, input: &[u8]) {
        for (out, byte) in self.out.iter_mut().zip(input) {
            *out = byte.to_ascii_uppercase();
        }

        ctx.emit(&self.out[..input.len()]);
    }
}

export_guest!(Upper, arena_bytes = CAPACITY);

init, on_eof, on_tick and tick_interval have defaults, so a guest writes the ones it needs. Two of the checks are sharper here than in C++. INIT is a const, which is how “global constructors do not run under --no-entry” becomes a compile error rather than a guest that silently reads zeroes. And every message handed to halt, fail and log is a &'static str, so the requirement that it outlive the call is a lifetime the compiler enforces rather than a comment.

examples/wasm/ is a separate cargo workspace, since guests are built for wasm32-unknown-unknown and pulling that into the main one would mean building the relay for it too.

$ cargo build --release --manifest-path examples/wasm/Cargo.toml

Guests in C and C++

sdk/wasm/ is the same contract in C and C++: tocat/abi.h is generated from the crate above, tocat/tocat.h adds the exports and the helpers, and a CMake package builds guests with both.

$ cmake -S sdk/wasm -B build/wasm \
        -DCMAKE_TOOLCHAIN_FILE=$PWD/sdk/wasm/cmake/wasm32-toolchain.cmake
$ cmake --build build/wasm

find_package(TocatWasm REQUIRED)

tocat_add_wasm_guest(redact SOURCES redact.c)

In C++ the exports are generated rather than written:

struct Upper {
    static constexpr uint32_t boundaries = TOCAT_BOUNDARIES_PRESERVE;

    void on_bytes(tocat::ctx &c, tocat::bytes input) { ... }
};

TOCAT_GUEST(Upper, 256 * 1024)

boundaries and needs are optional; a guest declaring neither claims nothing and asks for nothing. init, on_eof, on_tick and tick_interval are optional and detected too. ctx resets the outbox on construction, so the stale-flag mistake below cannot be made, and a halt reason is consteval from a string literal, so one that would not outlive the call does not compile. The header wraps the C one rather than restating it: two headers describing one ABI would drift, and the one that drifted would be the one nobody tested.

Three things differ from the Rust guest above and are worth knowing before writing one.

Pointers stop being a hazard: in C a pointer already is an address in linear memory, so TOCAT_ADDR is a cast rather than a calculation.

The layout can be checked at compile time. tocat.h declares the outbox as a struct and asserts its size and two of its offsets, which turns a mismatch into a build error rather than a stage that silently drops the stream. wasm32 puts the u64 at offset 32 on an eight-byte boundary, so there is no padding anywhere and the struct is exactly 48 bytes.

The compiler still expects a libc that is not there. A copy loop can be lowered into a call to memcpy whether or not you wrote one, so the header defines memcpy, memmove and memset. In C++ there is a sharper version of the same problem: with --no-entry nothing calls __wasm_call_ctors, so global constructors never run and anything needing one is silently uninitialised. Keep guest state trivially constructible, or export the initialiser and call it from tocat_init. The SDK’s README covers both.

Not here yet

Side channels. open_channel happens at build time and hands back an id, so it needs a round trip the ABI does not have: the guest would declare its targets during tocat_init, the host would open them, and the ids would have to be written back before the first chunk. Until then, a guest that wants to record something logs it, and the log sinks decide where that goes.

Testing a stage

Because a stage is synchronous and reaches nothing outside itself, a test is a plain unit test: build it, feed it chunks, assert on what came out and on what it asked for. No runtime, no sockets, no temporary files. Every plugin in the tree is tested this way.

Two small doubles are all it takes. A HostBuilder to hand out channel ids at build time:

struct CountingHost(u32);

impl HostBuilder for CountingHost {
    fn open_channel(&mut self, _target: ChannelTarget) -> Result<ChannelId> {
        let id = ChannelId(self.0);
        self.0 += 1;
        Ok(id)
    }
}

and an EffectSink that records whatever the stage asks for. Only write and log are required; pace and halt default to doing nothing, so implement them when the stage under test uses them.

#[derive(Default)]
struct Recorder(Vec<Vec<u8>>);

impl EffectSink for Recorder {
    fn write(&mut self, _channel: ChannelId, bytes: &[u8]) { self.0.push(bytes.to_vec()); }
    fn log(&mut self, _level: LogLevel, _stage: &str, _message: &str) {}
}

Building goes through the factory, so the config path is exercised too:

let map = json!({ "format": "hex" }).as_object().unwrap().clone();
let meta = PipelineMeta::new(Direction::SourceToSink, "tcp://a", "STDIO");
let stage = StageInfo { index: 0, total: 1, name: "audit", upstream: "tcp://a", downstream: "STDIO" };
let mut host = CountingHost(0);
let mut ctx = BuildCtx::new("tee", &map, &meta, stage, &mut host);
let mut plugin = match TeeFactory.build(&mut ctx)? { Stage::Filter(p) => p, _ => unreachable!() };

Driving it means constructing an Emission and a Ctx around each chunk, then reading the emission back:

let mut emission = Emission::new();
{
    let mut ctx = Ctx::new(&meta, "audit", input, &mut emission, &mut recorder);
    plugin.on_bytes(&mut ctx, input)?;
}

assert_eq!(emission.emit(), Emit::Passthrough);
assert_eq!(emission.bytes(), b"");
assert_eq!(emission.bounds(), &[] as &[usize]);

Emission exposes exactly what the host reads: bytes() is everything emitted concatenated, bounds() is the framing (empty means one unit), emit() is what the stage decided (Pending, Passthrough or Buffered), and rearm_requested() says whether it asked for its schedule to be restarted. reset() readies it for the next call while keeping the allocations, which is what the host does between chunks.

Three assertions are worth making by habit:

• A pure observer must never materialise the payload. emit() == Passthrough and bytes().is_empty() together are what prove tee and rate are free.
• Framing is what you meant. Assert on bounds(), not just on the concatenated bytes, or a stage that forgot its boundaries looks correct.
• The end matters. Call on_eof and assert on what it flushed, especially for anything holding bytes or writing an epilogue.

Ticks are just as direct: call on_tick with a Ctx whose input is empty, and check both what was emitted and rearm_requested(). Nothing has to wait for a real interval to pass.

Architecture

tocat is a small amount of machinery arranged so that the common case stays cheap. This section describes how the pieces fit and why, and is written for people changing tocat itself.

Crates

crates/
  tocat-api        the plugin contract: traits, pipeline, registry, shared grammars
  tocat-plugins    every native plugin, one module each, behind cargo features
  tocat-cli        the `tocat` binary: endpoints, config, relay, pumps, host services
  tocat-wasm-abi   the guest wire format, shared by the host, the SDK and the C header
  tocat-wasm-sdk   writing a guest in Rust
  tocat-wasm-shell a REPL for poking at a guest without a relay

The direction of the dependencies is the design. tocat-api depends on serde and nothing else: no tokio, no I/O, no relay. A plugin therefore depends on the contract rather than on the program, which is what lets it be built and tested alone, and is what a WASM guest needs, since a guest cannot link the relay at all. tocat-wasm-abi goes one further and depends on nothing, because it is also compiled into guests and translated into C.

The binary depends on tocat-plugins as a whole and never on one plugin, and the facade exposes nothing but [register_native], so a plugin can move, change its dependencies or disappear behind a feature without anything above noticing.

Modules in the binary

endpoint is deliberately one file per transport. Adding a scheme is a new file, one variant on EndpointSpec, and one line in the scheme table in endpoint::parse; each module owns its own fields, its parse, its label and its connect.

The shape of a run

1. Parse and merge. The config file (if any) and the command line become one Config, with the command line winning. --dump-config prints exactly this and exits. See Configuration resolution.
2. Resolve. Endpoint specs become EndpointSpecs and the merged plugin list stays as declarations. Anything wrong with an endpoint is an error here.
3. Validate. Relay::new builds both chains once, purely to discover which side channels they want, opens those channels, and freezes the plan. A misspelled plugin, an option no plugin declares, a detach = false on a subprocess stage or an unwritable dump file all fail here, before either endpoint is touched. The instances built for this pass are dropped. See Pipeline construction.
4. Warn. Three things are checked once, at this point, and warned about rather than refused: a stage that may not preserve message boundaries on a path whose destination is a datagram endpoint, a dump pointed at stderr while --progress is drawing there, and a buffer size and connection ceiling that together allow more than a gibibyte of copy buffers.
5. Connect. The endpoints are opened, or one of them listens and accepts, once or repeatedly under fork. Each connection builds its own chain pair; only the channel handles are shared.
6. Relay. The run takes the cheapest path that fits: see The data path.
7. Finish. End of stream on a path cascades through its stages, flushes and closes the sink. Channels are flushed on the way out. See Lifecycle and shutdown.

Design commitments

These hold across the codebase and are the reason for most of the local decisions.

• Errors are loud and early. Everything decidable from the configuration is decided before a byte moves. Unknown options are rejected rather than ignored, because a silently ignored option is a wrong relay that looks right.
• Roles come from position, never from text. The endpoint slots are filled positionally, and a pipeline entry is never promoted to an endpoint because it happens to look like one. Guessing would make the meaning of a command line depend on which plugins the binary was built with.
• One spelling rule. Schemes, endpoint options, plugin names, plugin option keys and enum values are all matched with case, dashes and underscores ignored, and a normalized string is never stored, forwarded or displayed. Free-form values never come near it.
• Paths are independent. Every stage instance belongs to one path of one connection. There is no shared mutable state between directions, which is what makes both safe for stateful stages and keeps fork off locks on the data path.
• The host owns the outside world. Stages do not touch files, sockets, clocks or tasks. Everything that does is requested and performed by the host, which keeps stages testable and portable and keeps I/O policy in one place.
• Pay for what you ask for. A run with no plugins builds no pipeline machinery, a segment with nothing ticking builds no timer, a stage that forwards untouched copies nothing, and an unframed emission allocates no boundary list. Anything that costs per byte or per connection is opt-in and documented as such.

The data path

Per connection, per direction, tocat owns one buffer and one loop: read into the buffer, hand the bytes to that path’s chain, write what comes out, repeat until end of stream. The buffer is 256 KiB by default and is the largest chunk any stage can see.

What actually runs depends on what was declared, and there are four arrangements, chosen so that nobody pays for machinery they did not ask for.

The whole-relay shortcuts

copy_bidirectional_with_sizes. Nothing declared on either path, no progress meter, and both endpoints handed back a Duplex stream, which in practice means TCP or Unix sockets on both sides. The relay hands the pair to tokio and stays out of the way. This is the case worth protecting: it is what most invocations are, and any per-chunk work tocat adds to it is pure overhead. Turning on --progress costs the shortcut, because counting needs a per-chunk hook that call does not offer, and that is documented rather than hidden.

The synchronous path. No plugins at all and both endpoints blocking-backed, which means file:, pipe: and stdio:. Tokio’s wrappers for those read into their own buffer and copy into yours, so a relay between two of them would pay two userspace copies of every byte. Instead each direction gets a spawn_blocking task running a plain read/write loop over one page-aligned buffer, which is structurally what socat does. Not std::io::copy: its kernel-offload specialisations only fire for concrete types, and through a dyn it falls back to an 8 KiB stack buffer.

A direction with no reader or no writer does not exist and is skipped, on this path and on the pumped ones below. It matters more than it sounds. A file: source paired with stdio would otherwise park on a stdin read whose bytes go to a null sink, holding the relay open for an EOF nobody will send; paired with udp: it would wait for one that cannot arrive at all, since a datagram socket has no end of stream. Either way the direction that mattered has already finished. Stages declared on a direction that does not exist are warned about rather than silently skipped, since they were built and are about to do nothing.

The pumps

Otherwise each direction is pumped independently, and a direction with nothing declared on it still gets the cheap treatment:

Datagrams cannot take the copy_buf shortcut even with an empty chain, because it is free to coalesce reads and would merge two messages into one send. An empty pipeline is run instead, which preserves the one-in-one-out mapping.

Segments and detach

A path is a chain of stages, and by default the whole chain runs inline on the reading task: one call per stage per chunk, no allocation, no wakeup. A stage declared Detached starts a new segment, and so does the first stage after a subprocess, since nothing can run inside one. Subsequent inline stages join the segment that is open rather than spawning more.

Segments are joined by bounded channels two parcels deep, with a return path so spent buffers are recycled rather than reallocated. That is what an OS pipe between stages would give you, minus the two syscalls and the kernel round trip. Crossing the boundary is the one place a copy is unavoidable: the downstream task outlives the stack frame that produced the bytes, so it needs owned ones.

Segments are spawned back to front, so every segment’s downstream exists before it starts, and the head stays on the calling task, since spawning it would buy nothing.

A downstream segment that has finished makes its upstream’s send fail, which is treated as end of stream rather than as an error: a limit that reached its cap or a sink that closed is a legitimate reason for the segment above to stop, and one that broke reports its own error from its own task.

Backpressure

tocat does not queue. Nothing on the data path grows without bound, and slowness propagates backwards rather than accumulating:

• A slow sink means the write does not complete, so the loop does not read again, so the receive buffer fills and the peer is slowed by the transport.
• throttle uses the same mechanism deliberately: it asks the host to wait before reading again rather than holding bytes back, which throttles the sender rather than tocat.
• A stage that buffers (block) is bounded by its own configuration rather than by demand.
• A detached link is two parcels deep, so a slow segment stalls the one above it almost immediately.

Kernel pipe buffers are the one fixed limit outside tocat that can cap throughput, which is why the pipes tocat owns are enlarged to match the copy buffer. See Buffers.

Ticks on the data path

A segment holding a stage that asked for ticks runs its read in a select! against one timer, so the stage hears from the clock even while the stream is idle. Reads are biased to win, because payload should not wait behind bookkeeping, and a segment with nothing ticking awaits the read directly.

That arrangement depends on the read being cancel-safe. Every arm of it bottoms out in poll_read, UdpSocket::recv or mpsc::Receiver::recv, none of which consume anything when dropped mid-poll. Anything added there has to keep that property or the select will quietly eat bytes.

When the timer fires, every stage that is due gets its turn, and what each emits is written before the next runs, all measured against one instant so that two stages due on the same wakeup do not disagree about the time.

Ending

End of stream on a path is delivered to the top stage, which may emit, and then cascades: each stage below sees that output and then its own end of stream. Once the chain has drained, the emission is written, staged effects are applied, and the sink is flushed and shut down. limit reaching its ceiling produces exactly this sequence early, which is why it is a successful exit rather than an error.

The run ends when every direction that exists has ended, which is not the same as when the interesting one has. Two duplex endpoints usually resolve this by themselves, since closing the write half is what tells the peer to hang up, and its close is the other direction’s end of stream. A datagram sink sends no such signal, so a genuinely bidirectional datagram relay ends when both paths do: that is what a timeout stage on both directions is for.

Pipeline construction

A run has two endpoint slots and an ordered list of pipeline entries. Turning that into two chains of stage instances is done by the registry, and it is where nearly all the validation lives.

Slots and positions

Roles are decided by position, never by looking at the text. --from and --to fill the first and last slots; the outer positional arguments fill whichever of those are still open, outermost first; whatever remains in the middle is the pipeline. A lone positional with both slots open fills the source.

tocat SRC SINK                    -> no plugins
tocat SRC tee compress SINK       -> two entries, in that order
tocat -f SRC tee SINK             -> one entry, SINK fills the open slot
tocat -f SRC -t SINK tee          -> one entry, both slots already filled

An endpoint and a plugin are written the same way, so guessing between them from the text would turn a typo into a different program, and would make the meaning of a command line depend on which plugins the binary was built with. The rule is worth keeping strictly.

From entries to chains

Each entry names a plugin, optionally a direction, and a set of options, of which as and detach are consumed by the host and the rest passed through untouched as JSON for the plugin’s own Deserialize. That is what lets the host stay ignorant of plugin schemas, including, eventually, those of plugins it has never seen.

Both chains are built from the same declaration list. Direction decides which one an entry joins:

both builds two independent instances rather than sharing one, so per-direction state never leaks across paths. Under fork the whole construction is repeated per connection, so an instance is scoped to one path of one connection and never needs a lock.

Order

The forward chain is the surviving entries in the order written; the reverse chain is the same list reversed. That is what makes the command line a picture of the wire:

tocat SRC  a  b  c  SINK

source -> sink:   SRC -> a -> b -> c -> SINK
sink -> source:   SRC <- a <- b <- c <- SINK

Mirroring rather than repeating is what makes wrapping stages nest correctly. Declaring compression nearer the source and framing nearer the sink gives, on the way back, unframing before decompression, with no second declaration and no way for the two directions to disagree.

Config-file entries come first, then the inline positional pipeline, then -p, so an ad-hoc stage lands nearest the sink on the forward path.

Naming

Each instance gets a display name: the as alias if given, otherwise the plugin name, with #1, #2 appended only where that name would appear more than once on the same path. It is what logs are tagged with, what tee puts in its header, and what a stage sees as StageInfo::name.

Each instance is also told its neighbours on this path, which are the adjacent stages’ display names or an endpoint label at either end. A stage wedged between two others therefore describes the hop it is really watching rather than the endpoints it is nowhere near. Under fork the accepted peer is folded into the listening endpoint’s label first, so those descriptions identify the connection too.

Segmentation

While building, stages accumulate into a draft segment. A stage whose placement resolves to Detached closes the draft and starts a new one, and an external stage closes the draft, becomes a segment of its own, and forces the next inline stage to start another. Placement is the entry’s detach if it gave one, otherwise the factory’s default.

What is decided here

• Which plugins exist in this binary, so a name the build does not have is a startup error, with a suggestion drawn from the ones it does have.
• Whether every option belongs to the plugin that documents it, and whether the values parse.
• Display names, including the #n suffixes.
• Placement, including rejecting detach = false on a stage that runs as a subprocess.
• Each stage’s tick interval, read once, so a segment with nothing ticking builds no timer.
• Which side channels exist, de-duplicated by target across both directions.
• Whether any stage on a path may break message boundaries, which produces the datagram warning.

Running with -v logs the resolved chains, their segment counts and the number of channels, which is the intended way to check that a long command line came out as intended.

The datagram model

A byte endpoint carries a stream; a datagram endpoint carries messages. tocat keeps that distinction rather than flattening everything to bytes, because the boundaries in a message stream are data: a peer that sent two datagrams did not send their concatenation.

The distinction is in the types. An open endpoint is a duplex stream, a split pair of halves, or a datagram socket, and the datagram variant is deliberately not AsyncRead/AsyncWrite: those traits describe a byte stream and have nowhere to put a boundary. Everything downstream of an endpoint therefore has to handle both shapes explicitly, which is what stops a datagram path quietly acquiring stream semantics in a refactor.

What sorts an endpoint into one or the other is what it carries, not how it connects. unix-seqpacket accepts connections, has an end of stream and is a datagram endpoint, because a message it delivers is one the peer sent whole. Putting it on the stream side would have been the easy choice, since the socket would satisfy those traits, and it would have thrown away the only property that distinguishes it from unix.

The rule

On a byte stream a chunk is an arbitrary slice, and both the host and a stage may buffer, split or coalesce freely. On a datagram path one receive is one message, one call to a stage is one message, and each unit a stage emits is sent as exactly one message. A stage that holds bytes across calls, or emits two messages’ worth from one, silently corrupts the protocol.

Units and boundaries are the mechanism that makes the good case expressible: what a stage emits between boundaries becomes one datagram.

Warn rather than refuse

Every stage declares whether it may carry datagrams, defaulting to false. When a stage that may not sits on a path whose destination is a datagram endpoint, the host names it, warns, and relays anyway.

Refusing would be wrong, because rewriting the message stream is sometimes exactly the point. block,size=1400 on a UDP path emits one datagram per block, which is a reasonable thing to ask for and still not a preservation of what the peer sent. The warning says what happened; the operator decides whether it is what they meant, and may well know the peer tolerates it.

The test is against the destination, not the source. Sending datagrams into a stream sink is unremarkable and draws no warning, since the sink has no boundaries to corrupt.

udp-listen:9000 compress:forward tcp:collector:9000   # fine
udp-listen:9000 compress         udp:peer:9000        # warns

A subprocess stage always warns: its stdin and stdout are byte streams, so boundaries are gone the moment bytes cross the pipe, whatever the child does.

Consequences elsewhere

Four properties of datagram paths ripple through the rest of the system, and are worth remembering when adding a feature.

There is no end of stream, unless a stage makes one, on the connectionless forms. A datagram source runs until interrupted, so anything that only happens at end of stream never happens: a rate summary, a compress epilogue and ratio report, the final short block. A feature that reports only at the end is a feature that does not exist on such a path, which is why rate also reports on a cadence. Held FIFOs share this property, so both cases need checking anywhere the code assumes end of stream will arrive.

A timeout stage halting the path is the one thing that produces one, and its halt is a real end of stream: on_eof runs and those reports do arrive. That also makes it the only way a forked session can end, which is why the endpoint grew no timer of its own.

unix-seqpacket is the exception on both counts, and the one datagram endpoint where the end of stream is the peer’s. A zero-length receive is its shutdown, so its recv can report the end without a stage halting the path, and it is the only one with a shutdown to send: the writing half half-closes at end of stream so a peer waiting for it hears one. Anything here that assumes a datagram endpoint runs forever has to check that variant.

Splitting is the unsafe operation, not truncating. This is why limit has three behaviours for the crossing chunk: drop and overshoot preserve the message and are safe, while exact splits it and is not. Half a datagram is a corrupt message rather than a short read.

The buffer is the message ceiling. One receive fills one buffer, and a longer message is truncated by the kernel. The copy buffer is therefore a protocol-visible setting on a datagram path in a way it is not on a stream.

An empty emission is not sent. The pipeline is drained once more at end of stream, and on a datagram sink that would put a spurious zero-length message on the wire, which many peers read as a close. The cost is that a genuine zero-length datagram is dropped rather than forwarded, which is the rarer case and the lesser harm.

Demultiplexing by sender

Without fork, a connectionless listener peers with the first sender and ignores everyone else. With fork it keeps the socket unconnected and routes by source address: one receive loop owns the socket, a map takes each datagram to that sender’s session, and a sender not in the map becomes a new session with its own dialled peer and its own chain instances. Two senders are two connections, because stage state is per path and per connection.

The loop is shared by udp-listen and unix-dgram-listen, over an enum of the two socket types and an enum of the two address types rather than a trait with generic sessions: two implementations do not earn a type parameter that every type around them would then have to carry, and a match against a syscall costs nothing.

The alternative, and what socat does, is a fresh SO_REUSEADDR socket per peer connected to that address, leaving the kernel to route by most specific match. That is one fd per peer, it leans on delivery rules that differ across platforms, SO_REUSEPORT on the same address load balances instead of specialising, and datagrams arriving between the receive and the connect land on the wrong socket anyway. One socket and a map has none of those problems and costs a task hop and a copy per datagram, which a pipeline was going to pay at its first detached boundary regardless.

A session needs a sender with an address. The map is keyed by the sender’s address and replies go back to it, so a sender with none can neither be told apart from another nor answered. UDP always satisfies this, because a socket that never bound is given an ephemeral port on its first send. AF_UNIX has no autobind, so an unbound sender arrives anonymous and its messages are dropped with the same accounting as any other drop. This is why unix-dgram binds a local address by default rather than leaving its own socket anonymous, and why unix-seqpacket-listen is the answer for peers that cannot bind: an accept gives an identity where an address does not.

Nothing ends a session but a stage. UDP has no close, so a session lasts as long as its task, and the task lasts until both directions finish. The endpoint deliberately has no idle option: timeout already ends a path that has gone quiet, already declares itself datagram safe, and already halts in the way that cascades on_eof, so a second timer here would have been the same feature under a different name. It has to be declared on both paths (timeout:both), since a forward-only halt leaves the reverse pump reading a sink that may never close, and it is the task ending that releases the permit and the map entry.

Two more properties follow from the receive loop serving every sender at once.

It never blocks on one session. A session’s queue is bounded, and a datagram for a queue that is full is dropped rather than made to wait, since waiting would stall every other peer. Drops are logged loudly once and at debug after, so a flood does not bury the rest of the log.

The ceiling is enforced there, not at the accept. max-connections is checked in the receive loop, because by the time a session reaches the accept loop it already exists. Finished sessions are reaped when the ceiling is reached, which is the only moment a dead entry is in anybody’s way.

Configuration resolution

There is one merged description of a run, and everything downstream reads only that. The command line and the config file are two ways of writing the same thing rather than two parallel mechanisms.

Order of operations

Startup is ordered around two constraints that are easy to break by accident.

A tracing subscriber can only be installed once, but configuration loading can fail and needs to report it. So a scoped bootstrap subscriber is installed first, at whatever level -v asked for, and is dropped before the real sinks are built from the merged config.

--list-plugins and --dump-config answer questions about the configuration, so they exit before anything opens a socket or a file. --list-plugins goes first of all, before the config file is even read, since the answer cannot depend on it.

Precedence

1. A config file is located: --config PATH, or tocat.toml then .tocat.toml in the working directory, unless --no-config was given.
2. Command-line values replace file values field by field. An endpoint given as a flag or a positional replaces the file’s, -b replaces buffer-size, -P replaces progress.
3. Plugin entries are the exception to replace: they accumulate, in the order file, then inline positionals, then -p. --no-plugins drops the file’s list and keeps the rest.
4. The log level is the explicit --log-level if given, otherwise the higher of -v and the file’s log-level. It is written back into the config, so the dump shows what will actually be used.
5. RUST_LOG is applied on top of the default stderr sink when the filter is built.

--dump-config renders the merged result as TOML, and is exactly what the rest of the program sees. It is worth keeping it that way rather than letting it drift into a pretty printer: it is the answer to “what is tocat actually about to do”.

Two spellings, one meaning

An endpoint can be written as a string (source = "tcp-listen:9000,fork") or as a table ([source] with type, host, port, fork). Both resolve to the same EndpointSpec, and the string form in the file goes through the same parser as the string form on the command line, so adding an option means adding it once, next to the field it sets, and both spellings and the error messages follow.

The serde representation is arranged to keep those two forms honest: the endpoint enum is internally tagged on type, and each variant is a newtype over the transport’s own struct, which flattens into the same table it would have produced inline.

Plugin entries work the same way. [[plugin]] carries name, direction, as and detach, with the plugin’s own options flattened alongside them, and the compact command-line grammar produces exactly that structure: a bare key is true, a value that parses as an integer becomes one, true and false become booleans, and anything else stays a string. The plugin’s config type decides the real shape from there, which is why ByteSize and Interval accept both a number and a string.

Keys are matched leniently in one direction only: case is ignored and dashes and underscores are removed for identifiers, while values stay untouched. A normalized string is never stored, forwarded or displayed. That keeps max-connections and max_connections the same option without ever second-guessing a path, a command or a label. Where a candidate matches no declared spelling it is passed through untouched rather than guessed at, which is what keeps a plugin’s own serde aliases working.

Validation

tocat is the validator, and everything it can check it checks before opening an endpoint: unknown top-level keys, unknown schemes, options belonging to another scheme, unknown plugins, options no plugin declares, contradictory ones, and channel targets that cannot be opened.

tocat.schema.json is a convenience on top of that, not a second source of truth. It gives completion, hover documentation and validation in an editor that speaks JSON Schema. It is written as one complete variant per known plugin rather than a shared base plus conditional fragments, because editors only offer completions from the schema they resolve and anything behind allOf or if/then is invisible to them. It deliberately accepts plugin entries it does not know about, since the plugin set depends on the cargo features the binary was built with.

When an option is added or renamed, the schema is updated alongside it. The schema being wrong is a documentation bug rather than a behaviour change.

Lifecycle and shutdown

Validation before I/O

Relay::new builds both chains once, before any endpoint is opened, purely to discover which side channels the declarations ask for. The instances from that pass are dropped: they are stateful and each connection needs its own. What survives is the frozen channel plan, which is then opened, and which every connection resolves its handles against without touching a lock.

So a misspelled plugin, an option no plugin declares, a detach = false on a subprocess stage or an unwritable dump file fails at startup rather than on the first byte of the first connection. The three warnings described in Architecture are raised in the same pass.

Forking

If either endpoint is a forking listener, that side binds and accepts in a loop while the other is dialled per connection. Concurrency is bounded by a semaphore of max-connections permits, acquired before the accept rather than after, so the listener stops taking connections it cannot serve instead of accepting and queueing them.

Each accepted connection gets its own chain pair, its own buffers and its own tick schedules, and runs on its own task inside a tracing span carrying the peer. Only the channel handles are shared, which is how several connections can dump into one file. Accept errors that are per-connection (aborted, interrupted, would-block) are logged and the loop continues; anything else ends the run.

udp-listen has no accept to run, so a forking one is served by a receive loop that demultiplexes the socket by source address and hands each new sender to the same accept loop as a session. Everything downstream of that is identical; what differs is that a session has no close to end it, so a timeout stage on both paths is what reclaims one, and that the connection ceiling is enforced in the receive loop, since a session exists by the time the accept loop sees it. See The datagram model.

Signals

The first SIGINT or SIGTERM flips a watch channel: the listener stops accepting, every connection still in flight is handed the same watch, and a task tracker waits for them to finish. A second signal exits immediately with status 130.

A connection observes the signal as end of stream, not as cancellation. The handle goes down to the reads that touch an endpoint, which stop returning chunks, and the ordinary end-of-stream path takes it from there: on_eof runs, a stage holding buffered bytes gets to emit them, side channels are applied and the writer is flushed and shut down. Racing the copy against the watch in a select! would skip all of that, which is why only the two paths with no stages do it: the plugin-free copy_bidirectional shortcut, and the synchronous copy.

Within a pipeline, only the head watches. A read from a segment boundary does not: the head stopping closes its outlet, which is end of stream for the segment below it, so on_eof cascades in order and parcels still in the channel are drained rather than dropped. A process stage stops feeding and closes the child’s stdin, which is the only end of stream a subprocess has, and then keeps draining its stdout so a filter holding a block still writes it out.

The synchronous copy path cannot do either, since a spawn_blocking task is not cancellable, so it polls between chunks instead. That leaves one case it cannot cover: a read that never returns, such as a FIFO with no writer. The runtime is therefore shut down with a short grace period rather than being dropped, so those tasks are left behind instead of making the process unkillable by signal.

Cleanup

Two kinds of state outlive the copy loop and have to be released in the right order.

Paths. A bound Unix socket, and a pipe: opened with unlink, hand back a guard that removes the path when it drops. The guard has to outlive the connection using it, or the socket is unlinked while still in use, or the FIFO is removed out from under a producer still writing to it.

Channels. Side-channel writers are buffered, so they are flushed after the relay finishes, whether it finished cleanly or not: on an error path that flush may be the last chance to get the dump to disk. All channels are flushed concurrently, so one failing sink does not cancel the others, and the first failure is reported.

The progress display is taken down before any error is reported, so its summary line lands above the diagnostics rather than wedged between them.

Children are handled differently at the two places they appear. An exec: or system: endpoint is reaped in the background, so the relay ends when the pipes close rather than when the process does, and a non-zero exit is a warning. A process stage is waited on, and a non-zero exit fails that direction, because there the child’s output is mid-pipeline and being wrong matters. Both are spawned with kill-on-drop, which is what ends a child blocked writing to a stdout nobody is draining.

Exit status

A stage stopping the transfer deliberately, which is what limit does, is a success: it is upstream end of stream arriving early, not a fault.