This version of the document targets the current stable branch of PyLink, and may be considerably outdated when compared to the **devel** branch where active development takes place. When developing new plugins or protocol modules, we highly recommend targeting the devel branch.
Starting with PyLink 2.x, a *protocol module* is any module containing a class derived from `PyLinkNetworkCore` (e.g. `InspIRCdProtocol`), along with a global `Class` attribute set equal to it (e.g. `Class = InspIRCdProtocol`). These modules do everything from managing connections to providing plugins with an API to send and receive data. New protocol modules may be implemented based off any of the classes in the following inheritance tree, with each containing a different amount of abstraction.
**Before you proceed, we highly recommend protocol module coders to get in touch with us** (e.g. via IRC at `#PyLink @ irc.overdrivenetworks.com`). Letting us know what you are working on can help coordinate coding efforts and better prepare for potential API breaks.
Note: The following notes in this section assume that you are working on some IRCd's server protocol, such that PyLink can spawn subservers and its own pseudoclients. If this is not the case, *virtual* clients and servers have to be spawned instead to emulate the correct state - the `clientbot` protocol module is a functional (though not very elegant) example of this.
`IRCNetwork` is the base IRC class which includes the state checking utilities from `PyLinkNetworkCore`, the generic IRC utilities from `PyLinkNetworkCoreWithUtils`, along with abstraction for establishing IRC connections and pinging the uplink at a set interval.
-`handle_events(self, data)`: given a line of text containing an IRC command, parse it and return a hook payload as specified in the [PyLink hooks reference](hooks-reference.md).
- In all of the official PyLink modules so far, handling for specific commands is delegated into submethods via [`getattr()`](https://github.com/jlu5/PyLink/blob/3922d44173593e4bcceae1218bbc6f267caa9fc1/protocols/ircs2s_common.py#L409-L412), and unknown commands are ignored.
-`post_connect(self)`: This method sends the server introduction commands to the uplink IRC server. This method replaces the `connect()` function defined by protocol modules prior to PyLink 2.x.
This class offers the most flexibility because the protocol module can choose how it wants to handle any command. However, because most IRC server protocols use the same RFC 1459-style message format, rewriting the entire event handler is often not worth doing. Instead, it may be better to use `IRCS2SProtocol`, as documented below, which includes a `handle_events` method which handles most cases (TS5/6, P10, and TS-less protocols such as ngIRCd).
- An exception to this general statement is `clientbot`, whose event handler also checks for unknown message senders and enumerates them when such a message is received.
`IRCCommonProtocol` (based off `IRCNetwork`) includes more IRC-specific methods such as parsers for ISUPPORT, as well as helper methods to parse arguments and recursively handle SQUIT. It also defines a default `_ping_uplink()` and incoming command handlers for commands that are the same across known protocols (AWAY, PONG, ERROR).
`IRCS2SProtocol` is the most complete base server class, including a generic `handle_events()` supporting most IRC S2S message styles (i.e. prefix-less messages, protocols with and without UIDs). It also defines some incoming and outgoing command functions that hardly vary between protocols: `invite()`, `kick()`, `message()`, `notice()`, `numeric()`, `part()`, `quit()`, `squit()`, and `topic()` as of PyLink 2.0. This list is subject to change in future releases.
Although no such transports have been implemented yet, PyLink leaves some level of abstraction for non-IRC protocols (e.g. Discord, Telegram, Slack, ...) by providing generic classes that only include state checking and utility functions.
Subclassing one of the `PyLinkNetworkCore*` classes means that a protocol module only needs to define one method of entry: `connect()`, and must set up its own message handling stack. Protocol configuration validation checks and autoconnect must also be reimplemented. IRC-style utility functions (i.e. `PyLinkNetworkCoreWithUtils` methods) should also be reimplemented / overridden when applicable.
For protocols that are closely related to existing ones, it may be wise to subclass off of an existing protocol class. For example, the `hybrid` and `ratbox` modules are based off of `ts6`. However, these protocol modules *do not guarantee API stability*, so we recommend letting us know of your intentions beforehand.
Unless otherwise noted, the camel-case variants of command functions (e.g. "`spawnClient`) are supported but deprecated. Protocol modules do *not* need to implement these aliases themselves; attempts to missing camel case functions are automatically coersed into their snake case variants via the [`structures.CamelCaseToSnakeCase`](https://github.com/jlu5/PyLink/blob/3922d44173593e4bcceae1218bbc6f267caa9fc1/structures.py#L172-L197) wrapper.
- **`spawn_client`**`(self, nick, ident='null', host='null', realhost=None, modes=set(), server=None, ip='0.0.0.0', realname=None, ts=None, opertype=None, manipulatable=False)` - Spawns a client on the PyLink server. No nick collision / valid nickname checks are done by protocol modules, as it is up to plugins to make sure they don't introduce anything invalid.
- The `manipulatable` option toggles whether the client spawned should be considered protected. Currently, all this does is prevent commands from plugins like `bots` from modifying these clients, but future client protections (anti-kill flood, etc.) may also depend on this.
- The `server` option optionally takes a SID of any PyLink server, and spawns the client on the one given. It should default to the root PyLink server if not specified.
- **`kick`**`(self, source, channel, target, reason=None)` - Sends a kick from a PyLink client/server. This should raise `NotImplementedError` if not supported by a protocol.
- **`kill`**`(self, source, target, reason)` - Sends a kill from a PyLink client/server. This should raise `NotImplementedError` if not supported by a protocol.
- **`knock`**`(self, source, target, text)` - Sends a KNOCK from a PyLink client. This should raise `NotImplementedError` if not supported by a protocol.
- **`mode`**`(self, source, target, modes, ts=None)` - Sends modes from a PyLink client/server. `modes` takes a set of `([+/-]mode char, mode arg)` tuples.
- **`numeric`**`(self, source, numeric, target, text)` - Sends a raw numeric `numeric` with `text` from the `source` server to `target`. This should raise `NotImplementedError` if not supported on a protocol.
- **`sjoin`**`(self, server, channel, users, ts=None, modes=set())` - Sends an SJOIN for a group of users to a channel. The sender should always be a Server ID (SID). TS is
- **`spawn_server`**`(self, name, sid=None, uplink=None, desc=None)` - Spawns a server off another PyLink server. `desc` (server description) defaults to the one in the config. `uplink` defaults to the main PyLink server, and `sid` (the server ID) is automatically generated if not given. Sanity checks for server name and SID validity ARE done by the protocol module here.
- **`update_client`**`(self, source, field, text)` - Updates the ident, host, or realname of a PyLink client. `field` should be either "IDENT", "HOST", "GECOS", or "REALNAME". If changing the field given on the IRCd isn't supported, `NotImplementedError` should be raised.
- Examples exist in the [UnrealIRCd](https://github.com/jlu5/PyLink/blob/1.0-beta1/protocols/unreal.py#L24-L27) and [InspIRCd](https://github.com/jlu5/PyLink/blob/1.0-beta1/protocols/inspircd.py#L25-L28) modules.
-`self.conf_keys`: a set of strings determining which server configuration options a protocol module needs to function; see the [Configuration key validation](#configuration-key-validation) section below.
-`self.cmodes` / `self.umodes`: These are mappings of named IRC modes (e.g. `inviteonly` or `moderated`) to a string list of mode letters, that should be either set during link negotiation or hardcoded into the protocol module. There are also special keys: `*A`, `*B`, `*C`, and `*D`, which **must** be set properly with a list of mode characters for that type of mode.
-`self.prefixmodes`: This defines a mapping of prefix modes (+o, +v, etc.) to their respective mode prefix. This will default to `{'o': '@', 'v': '+'}` (the standard op and voice) if not defined.
-`self.connected`: this is a `threading.Event` object that plugins use to determine if the network has finished bursting. Protocol modules should set this to True via `self.connected.set()` when ready.
PyLink 1.2 introduced the concept of protocol-defined capabilities, so that plugins wishing to use IRCd-specific features don't have to hard code protocol modules by name. Protocol capabilities are defined in `self.protocol_caps` (a set of strings) and may be changed freely before `self.connected` is set. Individual capabilities are then checked by plugins via `irc.has_cap(capability_name)`.
As of writing, the following protocol capabilities (case-sensitive) are implemented:
### Supported protocol capabilities
-`can-host-relay` - whether servers using this protocol can host a relay channel (for sanity reasons, this should be off for anything that's not IRC S2S)
-`can-spawn-clients` - determines whether any spawned clients are real or virtual (mainly for `services_support`).
-`can-track-servers` - determines whether servers are accurately tracked (for `servermaps` and other statistics)
-`has-statusmsg` - whether STATUSMSG messages (e.g. `@#channel`) are supported
-`has-ts` - determines whether channel and user timestamps are trackable (and not just spoofed)
-`slash-in-hosts` - determines whether `/` is allowed in hostnames
-`slash-in-nicks` - determines whether `/` is allowed in nicks
-`ssl-should-verify` - determines whether TLS certificates should be checked for validity by default - this should be enabled for any protocol modules needing to verify a remote server (e.g. Clientbot or a non-IRC API endpoint), and disabled for most IRC S2S links (where self-signed certs are widespread)
-`underscore-in-hosts` - determines whether `_` is allowed in client hostnames (yes, support for this actually varies by IRCd)
-`visible-state-only` - determines whether channels should be autocleared when the PyLink client leaves (for clientbot, etc.)
- Note: enabling this in a protocol module lets `coremods/handlers` automatically clean up old channels for you!
New protocol capabilities are generally added when needed - see https://github.com/jlu5/PyLink/issues/620
### Abstraction defaults
For reference, the `IRCS2SProtocol` class defines the following by default:
-`can-host-relay`
-`can-spawn-clients`
-`can-track-servers`
-`has-ts`
Whereas `PyLinkNetworkCore` defines no capabilities (i.e. the empty set) by default.
PyLink defines classes named `Server`, `User`, and `Channel` in the `classes` module, and stores dictionaries of these in the `servers`, `users`, and `channels` attributes of a protocol object respectively.
-`self.servers` is a dictionary mapping server IDs (SIDs) to `Server` objects. If a protocol module does not use SIDs, servers are stored by server name instead.
-`self.users` is a dictionary mapping user IDs (UIDs) to `User` objects. If a protocol module does not use UIDs, a pseudo UID (PUID) generator such as [`classes.PUIDGenerator`](https://github.com/jlu5/PyLink/blob/3922d44173593e4bcceae1218bbc6f267caa9fc1/classes.py#L1710-L1726) *must* be used instead.
- When sending text back to the protocol module, it may be helpful to use the [`_expandPUID()`](https://github.com/jlu5/PyLink/blob/4a363aee509c5a0488a38b9e60f93ec59a274c3c/classes.py#L1213-L1231) function in `PyLinkNetworkCoreWithUtils` to expand these pseudo-UIDs back to regular nicks.
-`self._channels` and `self.channels` are [IRC case-insensitive dictionaries](https://github.com/jlu5/PyLink/blob/4a363aee509c5a0488a38b9e60f93ec59a274c3c/structures.py#L114-L116) mapping channel names to Channel objects.
- The key difference between these two dictionaries is that `_channels` is powered by `classes.ChannelState` and creates new channels *automatically* when they are accessed by index. This makes writing protocol modules easier, as they can assume that the channels they wish to modify always exist (no chance of `KeyError`!).
-`Channel(self, name)` - First arg is the protocol object, second is the channel name.
-`User(self, nick, ts, uid, server, ident='null', host='null', realname='PyLink dummy client', realhost='null', ip='0.0.0.0', manipulatable=False, opertype='IRC Operator')` - These arguments are essentially the same as `spawn_client()`'s.
- When a user is introduced, their UID must be added to both `self.users` and to the `users` set in the `Server` object hosting the user (`self.servers[SID].users`). The latter list is used internally to track SQUITs.
- When a user joins a channel, the channel name is added to the User object's `channels` set (`self.users[UID].channels`), as well as the Channel object's user list (`self.channels[CHANNELNAME].users`)
- When a user disconnects, the `_remove_client` helper method can be called on their UID to automatically remove them from the relevant Server object, as well as all channels they were in.
- When a user leaves a channel, the `Channel.remove_user()` method can be used to easily remove them from the channel state, and vice versa.
Modes are stored not stored as strings, but lists of mode pairs in order to ease parsing. These lists of mode pairs are used both to represent mode changes in hooks and store modes internally.
`self.parse_modes(target, modestring)` is used to convert mode strings to mode lists. `target` is the channel name/UID the mode is being set on, while `modestring` takes either a string or string split by spaces (really a list).
`parse_modes()` will also automatically convert prefix mode targets from nicks to UIDs, and drop any duplicate (already set) or invalid (e.g. missing argument) modes.
**Note**: for protocols that accept or reject mode changes based on TS (i.e. practically every IRCd), you will want to use [`updateTS(...)`](https://github.com/jlu5/PyLink/blob/master/classes.py#L1484-L1487) instead to only apply the modes if the source TS is lower.
Internally, modes are stored in `Channel` and `User` objects as sets, **with the `+` prefixing each mode character omitted**. These sets are accessed via the `modes` attribute:
When receiving or sending topics, there is a `topicset` attribute in the `Channel` object that should be set to **True**. This boolean denotes that a topic has been set in the channel at least once; Relay uses it to know not to overwrite topics with empty ones during startup, when topics have not been received from all networks yet.
*Caveat:* Topic handlers on the current protocol modules do not follow TS rules (which vary by IRCd), and blindly accept data. See issue https://github.com/jlu5/PyLink/issues/277
Starting with PyLink 1.x, protocol modules can specify which config values within a server block they need in order to work. This is done by adjusting the `self.conf_keys` attribute, usually in the protocol module's `__init__()` method. The default set, defined in [`Classes.Protocol`](https://github.com/jlu5/PyLink/blob/1.0-beta1/classes.py#L1202-L1204), includes `{'ip', 'port', 'hostname', 'sid', 'sidrange', 'protocol', 'sendpass', 'recvpass'}`. Should any of these keys be missing from a server block, PyLink will bail with a configuration error.
As an example, one protocol module that tweaks this is [`Clientbot`](https://github.com/jlu5/PyLink/blob/1.0-beta1/protocols/clientbot.py#L17-L18), which removes all options except `ip`, `protocol`, and `port`.
In short, protocol modules have some very important jobs. If any of these aren't done correctly, you will be left with a very broken, desynced services server:
3) Make sure channel/user states are kept correctly. Joins, quits, parts, kicks, mode changes, nick changes, etc. should all be handled accurately where relevant.
5) Set the `threading.Event` instance `self.connected` to True (via `self.connected.set()`) when the connection with the uplink is fully established. This is important for Relay and the services API, which will refuse to initialize if the connection is not marked ready.
