diff --git a/.codeclimate.yml b/.codeclimate.yml deleted file mode 100644 index 7f6cb1d..0000000 --- a/.codeclimate.yml +++ /dev/null @@ -1,13 +0,0 @@ -# This file is used for the configuration of https://codeclimate.com/github/GLolol/PyLink/ -# You needn't change this if you're running your own copy of PyLink. - -engines: - duplication: - enabled: true - config: - languages: - - python - pep8: - enabled: true - fixme: - enabled: true diff --git a/.gitignore b/.gitignore index 8a951d3..ae38e2f 100644 --- a/.gitignore +++ b/.gitignore @@ -9,9 +9,11 @@ env/ build/ __pycache__/ +.idea/ *.py[cod] *.bak *~ +*# *.save* *.db *.pid @@ -19,3 +21,4 @@ __pycache__/ .eggs *.egg-info/ dist/ +log/ diff --git a/.mailmap b/.mailmap index 71e0734..bd36dfd 100644 --- a/.mailmap +++ b/.mailmap @@ -1,4 +1,7 @@ -James Lu -James Lu -Ken Spencer +James Lu +James Lu +James Lu +James Lu Ken Spencer +Ken Spencer +Ken Spencer diff --git a/README.md b/README.md index 04959d3..57870b0 100644 --- a/README.md +++ b/README.md @@ -24,24 +24,30 @@ PyLink and any bundled software are licensed under the Mozilla Public License, v **When upgrading between major versions, remember to read the [release notes](RELNOTES.md) for any breaking changes!** -Please report any bugs you find to the [issue tracker](https://github.com/GLolol/PyLink/issues). Pull requests are open if you'd like to contribute, though new stuff generally goes to the **devel** branch. +Please report any bugs you find to the [issue tracker](https://github.com/jlu5/PyLink/issues). Pull requests are open if you'd like to contribute, though new stuff generally goes to the **devel** branch. You can also find support via our IRC channel at `#PyLink @ irc.overdrivenetworks.com `([webchat][webchatlink]). Ask your questions and be patient for a response. ## Installation +### Pre-requisites +* CPython 3.4 or above (other intepreters are untested and unsupported) +* A Unix-like operating system: PyLink is actively developed on Linux only, so we cannot guarantee that things will work properly on other systems. + +If you are a developer and want to help make PyLink more portable, patches are highly appreciated. Talk to us! + ### Installing from source 1) First, make sure the following dependencies are met: - * Python 3.4+ * Setuptools (`pip3 install setuptools`) * PyYAML (`pip3 install pyyaml`) * ircmatch (`pip3 install ircmatch`) * *For password encryption*: Passlib (`pip3 install passlib`) + * *For enhanced cron support (by removing stale PID files): psutil (`pip3 install psutil`) * *For the servprotect plugin*: expiringdict (`pip3 install expiringdict`) -2) Clone the repository: `git clone https://github.com/GLolol/PyLink && cd PyLink` +2) Clone the repository: `git clone https://github.com/jlu5/PyLink && cd PyLink` 3) Pick your branch. * By default you'll be on the **master** branch, which contains the latest stable code. This branch is recommended for production networks that don't require new features or intensive bug fixes as they are developed. @@ -56,7 +62,7 @@ You can also find support via our IRC channel at `#PyLink @ irc.overdrivenetwork 2) Run `pip3 install pylinkirc` to download and install PyLink. pip will automatically resolve dependencies. -3) Download or copy https://github.com/GLolol/PyLink/blob/master/example-conf.yml for an example configuration. +3) Download or copy https://github.com/jlu5/PyLink/blob/master/example-conf.yml for an example configuration. ### Installing via Ubuntu PPA (14.04/Trusty and above) @@ -80,11 +86,12 @@ Upon installing the `pylink` package, example configuration and docs will be in These IRCds (in alphabetical order) are frequently tested and well supported. If any issues occur, please file a bug on the issue tracker. * [charybdis](https://github.com/charybdis-ircd/charybdis) (3.5+) - module `ts6` + - For KLINE support to work, a `shared{}` block should be added for PyLink on all servers. * [InspIRCd](http://www.inspircd.org/) 2.0.x - module `inspircd` - - For vHost setting to work, `m_chghost.so` must be loaded. + - For vHost setting to work, `m_chghost.so` must be loaded. For ident and realname changing support, `m_chgident.so` and `m_chgname.so` must be loaded respectively. - Supported channel, user, and prefix modes are negotiated on connect, but hotloading modules that change these is not supported. After changing module configuration, it is recommended to SQUIT PyLink to force a protocol renegotiation. * [Nefarious IRCu](https://github.com/evilnet/nefarious2) (2.0.0+) - module `p10` - - Note: Both account cloaks (user and oper) and hashed IP cloaks are optionally supported (HOST_HIDING_STYLE settings 0 to 3). Make sure you configure PyLink to match your IRCd settings. + - Note: Both account cloaks (user and oper) and hashed IP cloaks are optionally supported (`HOST_HIDING_STYLE` settings 0 to 3). Make sure you configure PyLink to match your IRCd settings. * [UnrealIRCd](https://www.unrealircd.org/) 4.x (4.0.12+) - module `unreal` - UnrealIRCd 4.x before version 4.0.12 suffers from [bug #4890](https://bugs.unrealircd.org/view.php?id=4890) which causes hostname desyncs on servers not directly linked to PyLink (e.g. `pylink<->serverA<->serverB` creates desynced hostnames on server B). This problem is fixed by upgrading your IRCds. - Linking to UnrealIRCd 3.2 servers is only possible when using an UnrealIRCd 4.x server as a hub, with topology such as `pylink<->unreal4<->unreal3.2`. We nevertheless encourage you to upgrade so all your IRCds are running the same version. @@ -93,16 +100,29 @@ These IRCds (in alphabetical order) are frequently tested and well supported. If Support for these IRCds exist, but are not tested as frequently and thoroughly. Bugs should be filed if there are any issues, though they may not always be fixed in a timely fashion. +* [beware-ircd](http://ircd.bircd.org/) (1.6.3) - module `p10` + - Because bircd disallows BURST after ENDBURST for regular servers, U-lines are required for all PyLink servers. Fortunately, wildcards are supported in U-lines, so you can add something along the lines of `U::` and `U:*.relay:` (adjust accordingly for your relay server suffix). + - Use `ircd: snircd` as the target IRCd. + - Halfops, `sethost` (`+h`), and account-based cloaking (`VHostStyle=1`) are supported. Crypted IPs and static hosts (`VHostStyle` 2 and 3) are NOT. +* [ChatIRCd](http://www.chatlounge.net/software) (1.2.x / git master) - module `ts6` + - For KLINE support to work, a `shared{}` block should be added for PyLink on all servers. * [Elemental-IRCd](https://github.com/Elemental-IRCd/elemental-ircd) (6.6.x / git master) - module `ts6` + - For KLINE support to work, a `shared{}` block should be added for PyLink on all servers. * [InspIRCd](http://www.inspircd.org/) 3.0.x (git master) - module `inspircd` + - The same notes for InspIRCd 2.x apply here as well. * [IRCd-Hybrid](http://www.ircd-hybrid.org/) (8.2.x / svn trunk) - module `hybrid` - - Note: for host changing support and optimal functionality, a `service{}` block / U-line should be added for PyLink on every IRCd across your network. -* [ircd-ratbox](http://www.ratbox.org/) (3.x) - module `ratbox` - - Host changing is not supported on ircd-ratbox. + - For host changing support and optimal functionality, a `service{}` block / U-line should be added for PyLink on every IRCd across your network. + - For KLINE support to work, a `shared{}` block should also be added for PyLink on all servers. +* [ircd-ratbox](http://www.ratbox.org/) (3.x) - module `ts6` + - Host changing is not supported. - On ircd-ratbox, all known IPs of users will be shown in `/whois`, even if the client is a cloaked relay client: if you're paranoid about this, turn off Relay IP forwarding by setting the `relay_no_ips` option in the ratbox network's `server:` block. + - For KLINE support to work, a `shared{}` block should be added for PyLink on all servers. * [IRCu](http://coder-com.undernet.org/) (u2.10.12.16+) - module `p10` - Host changing is not supported. -* [juno-ircd](https://github.com/cooper/yiria) (11.x / janet) - module `ts6` (see [configuration example](https://github.com/cooper/juno/blob/master/doc/ts6.md#pylink)) +* [juno-ircd](https://github.com/cooper/juno) (13.x / ava) - module `ts6` (see [configuration example](https://github.com/cooper/juno/blob/master/doc/ts6.md#pylink)) +* [ngIRCd](https://ngircd.barton.de/) (24+) - module `ngircd` + - For GLINEs to propagate, the `AllowRemoteOper` option must be enabled in ngIRCd. + - `+` (modeless) channels are not supported, and should be disabled for PyLink to function correctly. * [snircd](https://development.quakenet.org/) (1.3.x+) - module `p10` - Outbound host changing (i.e. for the `changehost` plugin) is not supported on P10 variants other than Nefarious. diff --git a/RELNOTES.md b/RELNOTES.md index 8be1a87..37fa7df 100644 --- a/RELNOTES.md +++ b/RELNOTES.md @@ -1,3 +1,258 @@ +# PyLink 2.0-rc1 (unreleased) + +#### Bug fixes +- relay: CHANDESC permissions are now given to opers if the `relay::allow_free_oper_links` option is true. +- Relay no longer forwards kills from servers, preventing extraneous kills for nick collisions and the like. +- bots: the `join` command now correctly uses `bots.join` as its permission name (keeping it consistent with the command name). + - The previous name `bots.joinclient` is still supported for compatibility reasons. + +#### Documentation updates +- Rewrote the Relay Quick Start Guide. [issue#619](https://github.com/jlu5/PyLink/issues/619) +- FAQ: expanded Relay section with common questions regarding Relay mechanics (i.e. kill, mode, and server bans handling) +- docs/technical, docs/permissions-reference: many updates to bring up to date with PyLink 2.0 +- various: updated the GitHub repository address + +# PyLink 2.0-beta1 (2018-06-27) + +This release contains all changes from 2.0-alpha3 as well as the following: + +#### New features +- **Added TLS certificate verification, which is now enabled by default on Clientbot networks**. [issue#592](https://github.com/jlu5/PyLink/issues/592) + - This adds the options `ssl_validate_hostname` and `ssl_accept_invalid_certs` options which have defaults as follows: + - | Server type | `ssl_validate_hostname` | `ssl_accept_invalid_certs` | + |--------------------------|-----------------------------------------------------|----------------------------| + | Full links (S2S) | false (implied by `ssl_accept_invalid_certs: true`) | true | + | Clientbot networks (C2S) | true | false | + - `ssl_validate_hostname` determines whether a network's TLS certificate will be checked for a matching hostname. + - `ssl_accept_invalid_certs` disables certificate checking entirely when enabled, and also turns off `ssl_validate_hostname`. + - The existing TLS certificate fingerprint options are unchanged by this and can be turned on and off regardless of these new options. +- New Relay features: + - **LINKACL now supports whitelisting networks in addition to the original blacklist implementation** (see `help LINKACL`). [issue#394](https://github.com/jlu5/PyLink/issues/394) + - relay: The defaults for CLAIM (on or off) and LINKACL (whitelist or blacklist mode) can now be pre-configured for new channels. [issue#581](https://github.com/jlu5/PyLink/issues/581) + - You can now set descriptions for channels in `LINKED` via the `CHANDESC` command. [issue#576](https://github.com/jlu5/PyLink/issues/576) + - `relay_clientbot` now supports setting clientbot styles by network. [issue#455](https://github.com/jlu5/PyLink/issues/455) + - New `relay::allow_free_oper_links` option allows disabling oper access to `CREATE/LINK/DELINK/DESTROY/CLAIM` by default +- New Antispam features (see the `antispam:` example block for configuration details): + - Antispam now supports text filtering with configured bad strings. [issue#359](https://github.com/jlu5/PyLink/issues/359) + - Added `block` as a punishment to only hide messages from other plugins like relay. [issue#616](https://github.com/jlu5/PyLink/issues/616) + - Antispam can now process PMs to PyLink clients for spam - this can be limited to service bots, enabled for all PyLink clients (including relay clones), or disabled entirely (the default). + - IRC formatting (bold, underline, colors, etc.) is now removed before processing text. [issue#615](https://github.com/jlu5/PyLink/issues/615) +- IPv4/IPv6 address selection is now automatic, detecting when an IPv6 address or bindhost is given. [issue#212](https://github.com/jlu5/PyLink/issues/212) +- Messages sent by most commands are now transparently word-wrapped to prevent cutoff. [issue#153](https://github.com/jlu5/PyLink/issues/153) +- The Global plugin now supports configuring exempt channels. [issue#453](https://github.com/jlu5/PyLink/issues/453) +- Automode now allows removing entries by entry numbers. [issue#506](https://github.com/jlu5/PyLink/issues/506) + +#### Feature changes +- Relay feature changes: + - Relay IP sharing now uses a pool-based configuration scheme (`relay::ip_share_pools`), deprecating the `relay::show_ips` and `relay_no_ips` options. + - IPs and real hosts are shared bidirectionally between all networks in an ipshare pool, and masked as `0.0.0.0` when sending to a network not in a pool and when receiving those networks' users. + - **KILL handling received a major rework** ([issue#520](https://github.com/jlu5/PyLink/issues/520): + - Instead of always bouncing, kills to a relay client can now be forwarded between networks in a killshare pool (`relay::kill_share_pools`). + - If the sender and target's networks are not in a killshare pool, the kill is forwarded as a kick to all shared channels that the sender + has CLAIM access on (e.g. when they are the home network, whitelisted in `CLAIM`, and/or an op). +- The PyLink service client no longer needs to be in channels to log to them. + +#### Bug fixes +- Fixed ping timeout handling (this was broken sometime during the port to select). +- relay: block networks not on the claim list from merging in modes when relinking (better support for modes set by e.g. services DEFCON). +- Reworked relay+clientbot op checks on to [be more consistent](https://github.com/jlu5/PyLink/compare/fee64ece045ad9dc49a07d1b438caa019c90a778~...d4bf407). +- inspircd: fix potential desyncs when sending a kill by removing the target immediately. [issue#607](https://github.com/jlu5/PyLink/issues/607) +- UserMapping: fixed a missing reference to the parent `irc` instance causing errors on nick collisions. +- clientbot: suppress warnings if `/mode #channel` doesn't show arguments to `+lk`, etc. [issue#537](https://github.com/jlu5/PyLink/issues/537) +- Relay now removes service persistent channels on unload, and when the home network for a link disconnects. +- relay: raise an error when trying to delink a leaf channel from another leaf network. + - Previously this would (confusingly) delink the channel from the network the command was called on instead of the intended target. +- opercmds: forbid killing the main PyLink client. + +#### Internal changes +- Login handling was rewritten and moved entirely from `coremods.corecommands` to `coremods.login`. [issue#590](https://github.com/jlu5/PyLink/issues/590) +- New stuff for the core network classes: + - `is_privileged_service(entityid)` returns whether the given UID or SID belongs to a privileged service (IRC U:line). + - `User`, `Channel` TS values are now consistently stored as `int`. [issue#594](https://github.com/jlu5/PyLink/issues/594) + - `match_host()` was split into `match_host()` and `match_text()`; the latter is now preferred as a simple text matcher for IRC-style globs. +- New stuff in utils: + - `remove_range()` removes a range string of (one-indexed) items from the list, where range strings are indices or ranges of them joined together with a "," (e.g. "5", "2", "2-10", "1,3,5-8") - [issue#506](https://github.com/jlu5/PyLink/issues/506) + - `get_hostname_type()` takes in an IP or hostname and returns an int representing the detected address type: 0 (none detected), 1 (IPv4), 2 (IPv6) - [issue#212](https://github.com/jlu5/PyLink/issues/212) + - `parse_duration()` takes in a duration string (in the form `1w2d3h4m5s`, etc.) and returns the equiv. amount of seconds - [issue#504](https://github.com/jlu5/PyLink/issues/504) +- The TLS/SSL setup bits in `IRCNetwork` were broken into multiple functions: `_make_ssl_context()`, `_setup_ssl`, and `_verify_ssl()` +- Removed deprecated attributes: `irc.botdata`, `irc.conf`, `utils.is*` methods, `PyLinkNetworkCoreWithUtils.check_authenticated()` - [issue#422](https://github.com/jlu5/PyLink/issues/422) +- PyLinkNCWUtils: The `allowAuthed`, `allowOper` options in `is_oper()` are now deprecated no-ops (they are set to False and True respectively) + +# PyLink 2.0-alpha3 (2018-05-10) + +This release contains all changes from 1.3.0, as well as the following: + +#### New features +- **Experimental daemonization support via `pylink -d`**. [issue#187](https://github.com/GLolol/PyLink/issues/187) +- New (alpha-quality) `antispam` plugin targeting mass-highlight spam: it supports any combination of kick, ban, quiet (mute), and kill as punishment. [issue#359](https://github.com/GLolol/PyLink/issues/359) +- Clientbot now supports expansions such as `$nick` in autoperform. +- Relay now translates STATUSMSG messages (e.g. `@#channel` messages) for target networks instead of passing them on as-is. [issue#570](https://github.com/GLolol/PyLink/issues/570) +- Relay endburst delay on InspIRCd networks is now configurable via the `servers::NETNAME::relay_endburst_delay` option. +- The servermaps plugin now shows the uplink server name for Clientbot links +- Added `--trace / -t` options to the launcher for integration with Python's `trace` module. + +#### Feature changes +- **Reverted the commit making SIGHUP shutdown the PyLink daemon**. Now, SIGUSR1 and SIGHUP both trigger a rehash, while SIGTERM triggers a shutdown. +- The `raw` command has been split into a new plugin (`plugins/raw.py`) with two permissions: `raw.raw` for Clientbot networks, and `raw.raw.unsupported_network` for other protocols. Using raw commands outside Clientbot is not supported. [issue#565](https://github.com/GLolol/PyLink/issues/565) +- The servermaps plugin now uses two permissions for `map` and `localmap`: `servermaps.map` and `servermaps.localmap` respectively +- `showuser` and `showchan` now consistently report times in UTC + +#### Bug fixes +- protocols/clientbot: fix errors when connecting to networks with mixed-case server names (e.g. AfterNET) +- relay: fix KeyError when a local client is kicked from a claimed channel. [issue#572](https://github.com/GLolol/PyLink/issues/572) +- Fix `irc.parse_modes()` incorrectly mangling modes changes like `+b-b *!*@test.host *!*@test.host` into `+b *!*@test.host`. [issue#573](https://github.com/GLolol/PyLink/issues/573) +- automode: fix handling of channels with multiple \#'s in them +- launcher: prevent protocol module loading errors (e.g. non-existent protocol module) from blocking the setup of other networks. + - This fixes a side-effect which can cause relay to stop functioning (`world.started` is never set) +- relay_clientbot: fix `STATUSMSG` (`@#channel`) notices from being relayed to channels that it shouldn't +- Fixed various 2.0-alpha2 regressions: + - Relay now relays service client messages as PRIVMSG and P10 WALL\* commands as NOTICE + - protocols/inspircd: fix supported modules list being corrupted when an indirectly linked server shuts down. [issue#567](https://github.com/GLolol/PyLink/issues/567) +- networks: `remote` now properly errors if the target service is not available on a network. [issue#554](https://github.com/GLolol/PyLink/issues/554) +- commands: fix `showchan` displaying status prefixes in reverse +- stats: route permission error replies to notice instead of PRIVMSG + - This prevents "Unknown command" flood loops with services which poll `/stats` on link. +- clientbot: fixed sending duplicate JOIN hooks and AWAY status updates. [issue#551](https://github.com/GLolol/PyLink/issues/551) + +#### Internal improvements +- **Reading from sockets now uses select instead of one thread per network.** + - This new code uses the Python selectors module, which automatically chooses the fastest polling backend available ([`epoll|kqueue|devpoll > poll > select`](https://github.com/python/cpython/blob/v3.6.5/Lib/selectors.py#L599-L601)). +- **API Break: significantly reworked channel handling for service bots**. [issue#265](https://github.com/GLolol/PyLink/issues/265) + - The `ServiceBot.extra_channels` attribute in previous versions is replaced with `ServiceBot.dynamic_channels`, which is accessed indirectly via new functions `ServiceBot.add_persistent_channel()`, `ServiceBot.remove_persistent_channel()`, `ServiceBot.get_persistent_channels()`. This API also replaces `ServiceBot.join()` for most plugins, which now joins channels *non-persistently*. + - This API change provides plugins with a way of registering dynamic persistent channels, which are consistently rejoined on kick or kill. + - Persistent channels are also "dynamic" in the sense that PyLink service bots will now part channels marked persistent when they become empty, and rejoin when it is recreated. + - This new implementation is also plugin specific, as plugins must provide a namespace (usually the plugin name) when managing persistent channels using `ServiceBot.(add|remove)_persistent_channel()`. + - New abstraction: `ServiceBot.get_persistent_channels()` which fetches the list of all persistent channels on a network (i.e. *both* the config defined channels and what's registered in `dynamic_channels`). + - New abstraction: `ServiceBot.part()` sends a part request to channels and only succeeds if it is not marked persistent by any plugin. This effectively works around the long-standing issue of relay-services conflicts. [issue#265](https://github.com/GLolol/PyLink/issues/265) +- Major optimizations to `irc.nick_to_uid`: `PyLinkNetworkCore.users` and `classes.User` now transparently maintain an index mapping nicks to UIDs instead of doing reverse lookup on every call. + - This is done via a new `UserMapping` class in `pylinkirc.classes`, which stores User objects by UID and provides a `bynick` attribute mapping case-normalized nicks to lists of UIDs. + - `classes.User.nick` is now a property, where the setter implicitly updates the `bynick` index with a pre-computed case-normalized version of the nick (also stored to `User.lower_nick`) +- Various relay optimizations: reuse target SID when bursting joins, and only look up nick once in `normalize_nick` +- Rewritten CTCP plugin, now extending to all service bots. [issue#468](https://github.com/GLolol/PyLink/issues/468), [issue#407](https://github.com/GLolol/PyLink/issues/407) +- Relay no longer spams configured U-lines with "message dropped because you aren't in a common channel" errors +- The `endburst_delay` option to `spawn_server()` was removed from the protocol spec, and replaced by a private API used by protocols/inspircd and relay. +- New API: hook handlers can now filter messages from lower-priority handlers by returning `False`. [issue#547](https://github.com/GLolol/PyLink/issues/547) +- New API: added `irc.get_server_option()` to fetch server-specific config variables and global settings as a fallback. [issue#574](https://github.com/GLolol/PyLink/issues/574) +- automode: replace assert checks with proper exceptions +- Renamed methods in log, utils, conf to snake case. [issue#523](https://github.com/GLolol/PyLink/issues/523) +- Remove `structures.DeprecatedAttributesObject`; it's vastly inefficient for what it accomplishes +- clientbot: removed unreliable pre-/WHO join bursting with `userhost-in-names` +- API change: `kick` and `kill` command funcitons now raise `NotImplementedError` when not supported by a protocol +- relay, utils: remove remaining references to deprecated `irc.proto` + +# PyLink 2.0-alpha2 (2018-01-16) +This release includes all changes from 1.2.2-dev, plus the following: + +#### New features +- relay_clientbot: add support for showing prefix modes in relay text, via a new `$mode_prefix` expansion. [issue#540](https://github.com/GLolol/PyLink/issues/540) +- Added new modedelta feature to Relay: + - Modedelta allows specifying a list of (named) modes to only apply on leaf channels, which can be helpful to fight spam if leaf networks don't have adequate spam protection. +- relay: added new option `server:::relay_forcetag_nicks`, a per-network list of nick globs to always tag when introducing users onto a network. [issue#564](https://github.com/GLolol/PyLink/issues/564) +- Added support for more channel modes in Relay: + * blockcaps: inspircd +B, elemental-ircd +G + * exemptchanops: inspircd +X + * filter: inspircd +g, unreal extban ~T:block ([issue#557](https://github.com/GLolol/PyLink/issues/557)) + * hidequits: nefarious +Q, snircd +u + * history: inspircd +H + * largebanlist: ts6 +L + * noamsg: snircd/nefarious +T + * blockhighlight: inspircd +V (extras module) + * kicknorejoin: elemental-ircd +J ([issue#559](https://github.com/GLolol/PyLink/issues/559)) + * kicknorejoin_insp: inspircd +J (with argument; [issue#559](https://github.com/GLolol/PyLink/issues/559)) + * repeat: elemental-ircd +E ([issue#559](https://github.com/GLolol/PyLink/issues/559)) + * repeat_insp: inspircd +K (with argument; [issue#559](https://github.com/GLolol/PyLink/issues/559)) +- Added support for UnrealIRCd extban `~T` in Relay. [issue#557](https://github.com/GLolol/PyLink/issues/557) +- p10: added proper support for STATUSMSG notices (i.e. messages to `@#channel` and the like) via WALLCHOPS/WALLHOPS/WALLVOICES +- p10: added outgoing /knock support by sending it as a notice +- ts6: added incoming /knock handling +- relay: added support for relaying /knock + +#### Backwards incompatible changes +- **The ratbox protocol module has been merged into ts6**, with a new `ircd: ratbox` option introduced to declare Ratbox as the target IRCd. [issue#543](https://github.com/GLolol/PyLink/issues/543) + +#### Bug fixes +- Fix default permissions not applying on startup (2.0-alpha1 regression). [issue#542](https://github.com/GLolol/PyLink/issues/542) +- Fix rejoin-on-kill for the main PyLink bot not working (2.0-alpha1/[94e05a6](https://github.com/GLolol/PyLink/commit/94e05a623314e9b0607de4eb01fab28be2e0c7e1) regression). +- Clientbot fixes: + - Fix desyncs caused by incomplete nick collision checking when a user on a Clientbot link changes their nick to match an existing virtual client. [issue#535](https://github.com/GLolol/PyLink/issues/535) + - Fix desync involving ghost users when a person leaves a channel, changes their nick, and rejoins. [issue#536](https://github.com/GLolol/PyLink/issues/536) + - Treat 0 as "no account" when parsing WHOX responses; this fixes incorrect "X is logged in as 0" output on WHOIS. +- protocols/p10: fix the `use_hashed_cloaks` server option not being effective. +- Fix long standing issues where relay would sometimes burst users multiple times on connect. [issue#529](https://github.com/GLolol/PyLink/issues/529) + - Also fix a regression from 2.0-alpha1 where users would not be joined if the hub link is down ([issue#548](https://github.com/GLolol/PyLink/issues/548)) +- Fix `$a:account` extbans being dropped by relay (they were being confused with `$a`). [issue#560](https://github.com/GLolol/PyLink/issues/560) +- Fix corrupt arguments when mixing the `remote` and `mode` commands. [issue#538](https://github.com/GLolol/PyLink/issues/538) +- Fix lingering queue threads when networks disconnect. [issue#558](https://github.com/GLolol/PyLink/issues/558) +- The relay and global plugins now better handle empty / poorly formed config blocks. +- bots: don't allow `spawnclient` on protocol modules with virtual clients (e.g. clientbot) +- bots: fix KeyError when trying to join previously nonexistent channels + +#### Internal improvements +- `Channel.sort_prefixes()` now consistently sorts modes from highest to lowest (i.e. from owner to voice). Also removed workaround code added to deal with the wonkiness of this function. +- ircs2s_common: add handling for `nick@servername` messages. +- `IRCNetwork` should no longer send multiple disconnect hooks for one disconnection. +- protocols/ts6 no longer requires `SAVE` support from the uplink. [issue#545](https://github.com/GLolol/PyLink/issues/545) +- ts6, hybrid: miscellaneous cleanup +- protocols/inspircd now tracks module (un)loads for `m_chghost.so` and friends. [issue#555](https://github.com/GLolol/PyLink/issues/555) +- Clientbot now logs failed attempts in joining channels. [issue#533](https://github.com/GLolol/PyLink/issues/533) + +# PyLink 2.0-alpha1 (2017-10-07) +The "Eclectic" release. This release includes all changes from 1.2.1, plus the following: + +#### New features +- **Login blocks can now be limited by network, user hostname, and IRCop status**: see the new "require_oper", "hosts", and "networks" options in the example config. +- **Added support for ngIRCd, ChatIRCd, and beware-ircd** (via protocol modules `ngircd`, `ts6`, `p10` respectively) +- **Add support for extbans in protocols and relay**: this is supported on UnrealIRCd, Charybdis (and derivatives), InspIRCd, and Nefarious P10. + - Yes, this means you can finally mute bothersome relay users. +- Clientbot is now more featureful: + - Added support for IRCv3 caps `account-notify`, `account-tag`, `away-notify`, `chghost`, `extended-join`, and `userhost-in-names` + - Configurable alternate / fallback nicks are now supported: look for the `pylink_altnicks` option in the example config. + - Added support for WHOX, to complement IRCv3 `account-notify` and `account-tag`. + - Relay_clientbot can now relay mode changes as text. + - Clientbot can now optionally sync ban lists when it joins a channel, allowing Relay modesync to unset bans properly. See the `fetch_ban_lists` option in the example config. +- PyLink received a new launcher, which now checks for stale PID files (when `psutil` is installed) and supports shutdown/restart/rehash via the command line. +- New commands for the opercmds plugin, including: + - `chghost`, `chgident`, and `chgname`, for IRCds that don't expose them as commands. + - `massban`, `masskill`, `massbanre`, and `masskillre` - these commands allow setting kickbans, kills, or glines on users matching a PyLink mask (`n!u@h` mask/exttarget) or regular expression. The hope is that these tools can help opers actively fight botnets as they are connected, similar to atheme's `clearchan` and Anope's `chankill` commands. + - `checkbanre` - a companion to `checkban`, using regex matching +- Better support for (pre-defined) U-lined services servers in relay: + - `CLAIM` restrictions are relaxed for service bots, which may now join with ops and set simple modes. This prevents mode floods when features such as `DEFCON` are enabled, and when a channel is accidentally registered on a network not on the CLAIM list. + - `DEFCON` modes set by services are ignored by Relay instead of bounced, and do not forward onto other networks unless the setting network is also in the channel's `CLAIM` list. + - To keep the spirit of `CLAIM` alive, opped services not in a channel's `CLAIM` list are still not allowed to kick remote users, set prefix modes (e.g. op) on others, or set list modes such as bans. +- Service bots' hostnames and real names are now fully configurable, globally and per network. +- Added per-network configuration of relay server suffixes. +- Added IRC `/STATS` support via the `stats` plugin (`/stats c`, `u`, and `o` are supported so far) +- PyLink's connection time is now displayed when WHOISing service bots. This info can be turned off using the `pylink:whois_show_startup_time` option. +- More specific permissions for the `remote` command, which now allows assigning permissions by target network, service bot, and command. +- New `$service` exttarget which matches service bots by name. + +#### Backwards incompatible changes +- Signal handling on Unix was updated to use `SIGUSR1` for rehash and `SIGHUP` for shutdown - this changes PyLink to be more in line with foreground programs, which generally close with the owning terminal. +- Some options were deprecated and renamed: + - The `p10_ircd` option for P10 servers is now named `ircd`, though the old option will still be read from. + - The `use_elemental_modes` setting on ts6 networks has been deprecated and replaced with an `ircd` option targeting charybdis, elemental-ircd, or chatircd. Supported values for `ircd` include `charybdis`, `elemental`, and `chatircd`. +- PID file checking is now enabled by default, along with checks for stale PID files *only* when [`psutil`](https://pythonhosted.org/psutil/) is installed. Users upgrading from PyLink < 1.1-dev without `psutil` installed will need remove PyLink's PID files before starting the service. +- The `fml` command in the `games` plugin was removed. + +#### Bug fixes +- Relay should stop bursting channels multiple times on startup now. (`initialize_channel` now skips execution if another thread is currently initializing the same channel) +- Fixed a long standing bug where fantasy responses would relay before a user's original command if the `fantasy` plugin was loaded before `relay`. (Bug #123) + +#### Internal changes +- **API Break**: The protocol module layer is completely rewritten, with the `Irc` and `Protocol`-derived classes combining into one. Porting **will** be needed for old protocol modules and plugins targetting 1.x; see the [new (WIP) protocol specification](https://github.com/GLolol/PyLink/blob/devel/docs/technical/pmodule-spec.md) for details. +- **API Break**: Channels are now stored in two linked dictionaries per IRC object: once in `irc._channels`, and again in `irc.channels`. The main difference is that `irc._channels` implicitly creates new channels when accessing them if they didn't previously exist (prefer this for protocol modules), while `irc.channels` does not raises and raises KeyError instead (prefer this for plugins). +- **API Break**: Most methods in `utils` and `classes` were renamed from camel case to snake case. `log`, `conf`, and others will be ported too before the final 2.0 release. +- **API Break**: IRC protocol modules' server introductions must now use **`post_connect()`** instead of **`connect()`** to prevent name collisions with the base connection handling code. +- Channels are now stored case insensitively internally, so protocol modules and new plugins no longer need to manually coerse names to lowercase. +- Plugins can now bind hooks as specific priorities via an optional `priority` option in `utils.add_hook`. Hooks with higher priorities will be called first; the default priority value us 500. +- Commands can now be properly marked as aliases, so that duplicates don't show in the `list` command. +- Added basic `GLINE/KLINE` support for most IRCds; work is ongoing to polish this off. +- PyLink accounts are now implicitly matched: i.e. `user1` is now equivalent to `$pylinkacc:user1` +- Added complete support for "Network Administrator" and "Network Service" as oper types on IRCds using user modes to denote them (e.g. UnrealIRCd, charybdis). +- User and server hop counts are now tracked properly instead of being hardcoded as 1. +- protocols/p10 now bursts IPv6 IPs to supported uplinks. +- Fixed compatibility with ircd-hybrid trunk after commit 981c61e (EX and IE are no longer sent in the capability list) + # PyLink 1.3.0 (2018-05-08) The 1.3 update focuses on backporting some commonly requested and useful features from the WIP 2.0 branch. This release includes all changes from 1.3-beta1, plus the following: @@ -32,7 +287,7 @@ The 1.3 update focuses on backporting some commonly requested and useful feature - Significantly revised example-conf for wording and consistency. - protocols/unreal: bumped protocol version to 4017 (no changes needed) -# PyLink 1.2.1 +# PyLink 1.2.1 (2017-09-19) The "Dancer" release. Changes from 1.2.0: #### Bug fixes @@ -56,7 +311,7 @@ The "Dancer" release. Changes from 1.2.0: - Minor logging cleanup for relay and `Irc.matchHost()`. - Fix cmode `+p` mapping on TS6 networks. -# PyLink 1.2.0 +# PyLink 1.2.0 (2017-08-14) The "Dragons" release. Changes since 1.2.0-rc1: #### Feature changes diff --git a/VERSION b/VERSION index f0bb29e..c6208b2 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.3.0 +2.0-rc0 diff --git a/classes.py b/classes.py index 719830f..634032e 100644 --- a/classes.py +++ b/classes.py @@ -12,80 +12,255 @@ import time import socket import ssl import hashlib -from copy import deepcopy -import inspect -import re -from collections import defaultdict import ipaddress import queue +import functools +import string +import re +import collections +import collections.abc +import textwrap try: import ircmatch except ImportError: raise ImportError("PyLink requires ircmatch to function; please install it and try again.") -from . import world, utils, structures, conf, __version__ +from . import world, utils, structures, conf, __version__, selectdriver from .log import * - -### Exceptions - -class ProtocolError(RuntimeError): - pass +from .utils import ProtocolError # Compatibility with PyLink 1.x ### Internal classes (users, servers, channels) -class Irc(utils.DeprecatedAttributesObject): - """Base IRC object for PyLink.""" - SOCKET_REPOLL_WAIT = 1.0 +class ChannelState(structures.IRCCaseInsensitiveDict): + """ + A dictionary storing channels case insensitively. Channel objects are initialized on access. + """ + def __getitem__(self, key): + key = self._keymangle(key) - def __init__(self, netname, proto, conf): + if key not in self._data: + log.debug('(%s) ChannelState: creating new channel %s in memory', self._irc.name, key) + self._data[key] = newchan = Channel(self._irc, key) + return newchan + + return self._data[key] + +class TSObject(): + """Base class for classes containing a type-normalized timestamp.""" + def __init__(self, *args, **kwargs): + self._ts = int(time.time()) + + @property + def ts(self): + return self._ts + + @ts.setter + def ts(self, value): + if (not isinstance(value, int)) and (not isinstance(value, float)): + log.warning('TSObject: Got bad type for TS, converting from %s to int', + type(value), stack_info=True) + value = int(value) + self._ts = value + +class User(TSObject): + """PyLink IRC user class.""" + def __init__(self, irc, nick, ts, uid, server, ident='null', host='null', + realname='PyLink dummy client', realhost='null', + ip='0.0.0.0', manipulatable=False, opertype='IRC Operator'): + super().__init__() + self._nick = nick + self.lower_nick = irc.to_lower(nick) + + self.ts = ts + self.uid = uid + self.ident = ident + self.host = host + self.realhost = realhost + self.ip = ip + self.realname = realname + self.modes = set() # Tracks user modes + self.server = server + self._irc = irc + + # Tracks PyLink identification status + self.account = '' + + # Tracks oper type (for display only) + self.opertype = opertype + + # Tracks external services identification status + self.services_account = '' + + # Tracks channels the user is in + self.channels = structures.IRCCaseInsensitiveSet(self._irc) + + # Tracks away message status + self.away = '' + + # This sets whether the client should be marked as manipulatable. + # Plugins like bots.py's commands should take caution against + # manipulating these "protected" clients, to prevent desyncs and such. + # For "serious" service clients, this should always be False. + self.manipulatable = manipulatable + + # Cloaked host for IRCds that use it + self.cloaked_host = None + + # Stores service bot name if applicable + self.service = None + + @property + def nick(self): + return self._nick + + @nick.setter + def nick(self, newnick): + oldnick = self.lower_nick + self._nick = newnick + self.lower_nick = self._irc.to_lower(newnick) + + # Update the irc.users bynick index: + if oldnick in self._irc.users.bynick: + # Remove existing value -> key mappings. + self._irc.users.bynick[oldnick].remove(self.uid) + + # Remove now-empty keys as well. + if not self._irc.users.bynick[oldnick]: + del self._irc.users.bynick[oldnick] + + # Update the new nick. + self._irc.users.bynick.setdefault(self.lower_nick, []).append(self.uid) + + def get_fields(self): """ - Initializes an IRC object. This takes 3 variables: the network name - (a string), the name of the protocol module to use for this connection, - and a configuration object. + Returns all template/substitution-friendly fields for the User object in a read-only dictionary. """ - self.deprecated_attributes = { - 'conf': 'Deprecated since 1.2; consider switching to conf.conf', - 'botdata': "Deprecated since 1.2; consider switching to conf.conf['bot']", - } + fields = self.__dict__.copy() + + # These don't really make sense in text substitutions + for field in ('manipulatable', '_irc'): + del fields[field] + + # Pre-format the channels list. FIXME: maybe this should be configurable somehow? + fields['channels'] = ','.join(sorted(self.channels)) + + # Swap SID and server name for convenience + fields['sid'] = self.server + try: + fields['server'] = self._irc.get_friendly_name(self.server) + except KeyError: + pass # Keep it as is (i.e. as the SID) if grabbing the server name fails + + # Network name + fields['netname'] = self._irc.name + + # Join umodes together + fields['modes'] = self._irc.join_modes(self.modes) + + # Add the nick attribute; this isn't in __dict__ because it's a property + fields['nick'] = self._nick + + return fields + + def __repr__(self): + return 'User(%s/%s)' % (self.uid, self.nick) +IrcUser = User + +# Bidirectional dict based off https://stackoverflow.com/a/21894086 +class UserMapping(collections.abc.MutableMapping, structures.CopyWrapper): + """ + A mapping storing User objects by UID, as well as UIDs by nick via + the 'bynick' attribute + """ + def __init__(self, irc, data=None): + if data is not None: + assert isinstance(data, dict) + self._data = data + else: + self._data = {} + self.bynick = collections.defaultdict(list) + self._irc = irc + + def __getitem__(self, key): + return self._data[key] + + def __setitem__(self, key, userobj): + assert hasattr(userobj, 'lower_nick'), "Cannot add object without lower_nick attribute to UserMapping" + if key in self._data: + log.warning('(%s) Attempting to replace User object for %r: %r -> %r', self._irc.name, + key, self._data.get(key), userobj) + + self._data[key] = userobj + self.bynick.setdefault(userobj.lower_nick, []).append(key) + + def __delitem__(self, key): + # Remove this entry from the bynick index + if self[key].lower_nick in self.bynick: + self.bynick[self[key].lower_nick].remove(key) + + if not self.bynick[self[key].lower_nick]: + del self.bynick[self[key].lower_nick] + + del self._data[key] + + # Generic container methods. XXX: consider abstracting this out in structures? + def __repr__(self): + return "%s(%s)" % (self.__class__.__name__, self._data) + + def __iter__(self): + return iter(self._data) + + def __len__(self): + return len(self._data) + + def __contains__(self, key): + return self._data.__contains__(key) + + def __copy__(self): + return self.__class__(self._irc, data=self._data.copy()) + +class PyLinkNetworkCore(structures.CamelCaseToSnakeCase): + """Base IRC object for PyLink.""" + + def __init__(self, netname): self.loghandlers = [] self.name = netname - self.conf = conf + self.conf = conf.conf self.sid = None - self.serverdata = conf['servers'][netname] - self.botdata = conf['bot'] - self.protoname = proto.__name__.split('.')[-1] # Remove leading pylinkirc.protocols. - self.proto = proto.Class(self) + self.serverdata = conf.conf['servers'][netname] + + self.protoname = self.__class__.__module__.split('.')[-1] # Remove leading pylinkirc.protocols. + self.proto = self.irc = self # Backwards compat + + # Protocol stuff + self.casemapping = 'rfc1459' + self.hook_map = {} + + # Lists required conf keys for the server block. + self.conf_keys = {'ip', 'port', 'hostname', 'sid', 'sidrange', 'protocol', 'sendpass', + 'recvpass'} + + # Defines a set of PyLink protocol capabilities + self.protocol_caps = set() # These options depend on self.serverdata from above to be set. self.encoding = None - self.pingfreq = self.serverdata.get('pingfreq') or 90 - self.pingtimeout = self.pingfreq * 2 - - self.queue = None self.connected = threading.Event() - self.aborted = threading.Event() - self.reply_lock = threading.RLock() - - self.pingTimer = None + self._aborted = threading.Event() + self._aborted_send = threading.Event() + self._reply_lock = threading.RLock() # Sets the multiplier for autoconnect delay (grows with time). self.autoconnect_active_multiplier = 1 - self.initVars() + self.was_successful = False - if world.testing: - # HACK: Don't thread if we're running tests. - self.connect() - else: - self.connection_thread = threading.Thread(target=self.connect, - name="Listener for %s" % - self.name) - self.connection_thread.start() + self._init_vars() - def logSetup(self): + def log_setup(self): """ Initializes any channel loggers defined for the current network. """ @@ -117,20 +292,15 @@ class Irc(utils.DeprecatedAttributesObject): self.loghandlers.append(handler) log.addHandler(handler) - def initVars(self): + def _init_vars(self): """ (Re)sets an IRC object to its default state. This should be called when an IRC object is first created, and on every reconnection to a network. """ self.encoding = self.serverdata.get('encoding') or 'utf-8' - self.pingfreq = self.serverdata.get('pingfreq') or 90 - self.pingtimeout = self.pingfreq * 3 + # Tracks the main PyLink client's UID. self.pseudoclient = None - self.lastping = time.time() - - self.maxsendq = self.serverdata.get('maxsendq', 4096) - self.queue = queue.Queue(self.maxsendq) # Internal variable to set the place and caller of the last command (in PM # or in a channel), used by fantasy command support. @@ -138,11 +308,16 @@ class Irc(utils.DeprecatedAttributesObject): self.called_in = None # Intialize the server, channel, and user indexes to be populated by - # our protocol module. For the server index, we can add ourselves right - # now. + # our protocol module. self.servers = {} - self.users = {} - self.channels = structures.KeyedDefaultdict(IrcChannel) + self.users = UserMapping(self) + + # Two versions of the channels index exist in PyLink 2.0, and they are joined together + # - irc._channels which implicitly creates channels on access (mostly used + # in protocol modules) + # - irc.channels which does not (recommended for use by plugins) + self._channels = ChannelState(self) + self.channels = structures.IRCCaseInsensitiveDict(self, data=self._channels._data) # This sets the list of supported channel and user modes: the default # RFC1459 modes are implied. Named modes are used here to make @@ -166,11 +341,16 @@ class Irc(utils.DeprecatedAttributesObject): '*A': 'b', '*B': 'k', '*C': 'l', - '*D': 'imnpstr'} + '*D': 'imnpst'} self.umodes = {'invisible': 'i', 'snomask': 's', 'wallops': 'w', 'oper': 'o', '*A': '', '*B': '', '*C': '', '*D': 'iosw'} + # Acting extbans such as +b m:n!u@h on InspIRCd + self.extbans_acting = {} + # Matching extbans such as R:account on InspIRCd and $a:account on TS6. + self.extbans_matching = {} + # This max nick length starts off as the config value, but may be # overwritten later by the protocol module if such information is # received. It defaults to 30. @@ -184,312 +364,30 @@ class Irc(utils.DeprecatedAttributesObject): self.start_ts = int(time.time()) # Set up channel logging for the network - self.logSetup() + self.log_setup() - def processQueue(self): - """Loop to process outgoing queue data.""" - while True: - throttle_time = self.serverdata.get('throttle_time', 0.005) - if not self.aborted.wait(throttle_time): - data = self.queue.get() - if data is None: - log.debug('(%s) Stopping queue thread due to getting None as item', self.name) - break - elif self not in world.networkobjects.values(): - log.debug('(%s) Stopping stale queue thread; no longer matches world.networkobjects', self.name) - break - elif data: - self._send(data) - else: - break + def __repr__(self): + return "<%s object for network %r>" % (self.__class__.__name__, self.name) + + ## Stubs + def validate_server_conf(self): + return def connect(self): - """ - Runs the connect loop for the IRC object. This is usually called by - __init__ in a separate thread to allow multiple concurrent connections. - """ - while True: - - self.aborted.clear() - self.initVars() - - try: - self.proto.validateServerConf() - except AssertionError as e: - log.exception("(%s) Configuration error: %s", self.name, e) - return - - ip = self.serverdata["ip"] - port = self.serverdata["port"] - checks_ok = True - try: - # Set the socket type (IPv6 or IPv4). - stype = socket.AF_INET6 if self.serverdata.get("ipv6") else socket.AF_INET - - # Creat the socket. - self.socket = socket.socket(stype) - self.socket.setblocking(0) - - # Set the socket bind if applicable. - if 'bindhost' in self.serverdata: - self.socket.bind((self.serverdata['bindhost'], 0)) - - # Set the connection timeouts. Initial connection timeout is a - # lot smaller than the timeout after we've connected; this is - # intentional. - self.socket.settimeout(self.pingfreq) - - # Resolve hostnames if it's not an IP address already. - old_ip = ip - ip = socket.getaddrinfo(ip, port, stype)[0][-1][0] - log.debug('(%s) Resolving address %s to %s', self.name, old_ip, ip) - - # Enable SSL if set to do so. - self.ssl = self.serverdata.get('ssl') - if self.ssl: - log.info('(%s) Attempting SSL for this connection...', self.name) - certfile = self.serverdata.get('ssl_certfile') - keyfile = self.serverdata.get('ssl_keyfile') - - context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - # Disable SSLv2 and SSLv3 - these are insecure - context.options |= ssl.OP_NO_SSLv2 - context.options |= ssl.OP_NO_SSLv3 - - # Cert and key files are optional, load them if specified. - if certfile and keyfile: - try: - context.load_cert_chain(certfile, keyfile) - except OSError: - log.exception('(%s) Caught OSError trying to ' - 'initialize the SSL connection; ' - 'are "ssl_certfile" and ' - '"ssl_keyfile" set correctly?', - self.name) - checks_ok = False - - self.socket = context.wrap_socket(self.socket) - - log.info("Connecting to network %r on %s:%s", self.name, ip, port) - self.socket.connect((ip, port)) - self.socket.settimeout(self.pingtimeout) - - # If SSL was enabled, optionally verify the certificate - # fingerprint for some added security. I don't bother to check - # the entire certificate for validity, since most IRC networks - # self-sign their certificates anyways. - if self.ssl and checks_ok: - peercert = self.socket.getpeercert(binary_form=True) - - # Hash type is configurable using the ssl_fingerprint_type - # value, and defaults to sha256. - hashtype = self.serverdata.get('ssl_fingerprint_type', 'sha256').lower() - - try: - hashfunc = getattr(hashlib, hashtype) - except AttributeError: - log.error('(%s) Unsupported SSL certificate fingerprint type %r given, disconnecting...', - self.name, hashtype) - checks_ok = False - else: - fp = hashfunc(peercert).hexdigest() - expected_fp = self.serverdata.get('ssl_fingerprint') - - if expected_fp and checks_ok: - if fp != expected_fp: - # SSL Fingerprint doesn't match; break. - log.error('(%s) Uplink\'s SSL certificate ' - 'fingerprint (%s) does not match the ' - 'one configured: expected %r, got %r; ' - 'disconnecting...', self.name, hashtype, - expected_fp, fp) - checks_ok = False - else: - log.info('(%s) Uplink SSL certificate fingerprint ' - '(%s) verified: %r', self.name, hashtype, - fp) - else: - log.info('(%s) Uplink\'s SSL certificate fingerprint (%s) ' - 'is %r. You can enhance the security of your ' - 'link by specifying this in a "ssl_fingerprint"' - ' option in your server block.', self.name, - hashtype, fp) - - if checks_ok: - - self.queue_thread = threading.Thread(name="Queue thread for %s" % self.name, - target=self.processQueue, daemon=True) - self.queue_thread.start() - - self.sid = self.serverdata.get("sid") - # All our checks passed, get the protocol module to connect and run the listen - # loop. This also updates any SID values should the protocol module do so. - self.proto.connect() - - log.info('(%s) Enumerating our own SID %s', self.name, self.sid) - host = self.hostname() - - self.servers[self.sid] = IrcServer(None, host, internal=True, - desc=self.serverdata.get('serverdesc') - or conf.conf['bot']['serverdesc']) - - log.info('(%s) Starting ping schedulers....', self.name) - self.schedulePing() - log.info('(%s) Server ready; listening for data.', self.name) - self.autoconnect_active_multiplier = 1 # Reset any extra autoconnect delays - self.run() - else: # Configuration error :( - log.error('(%s) A configuration error was encountered ' - 'trying to set up this connection. Please check' - ' your configuration file and try again.', - self.name) - # self.run() or the protocol module it called raised an exception, meaning we've disconnected! - # Note: socket.error, ConnectionError, IOError, etc. are included in OSError since Python 3.3, - # so we don't need to explicitly catch them here. - # We also catch SystemExit here as a way to abort out connection threads properly, and stop the - # IRC connection from freezing instead. - except (OSError, RuntimeError, SystemExit) as e: - log.exception('(%s) Disconnected from IRC:', self.name) - - self.disconnect() - - # If autoconnect is enabled, loop back to the start. Otherwise, - # return and stop. - autoconnect = self.serverdata.get('autoconnect') - - # Sets the autoconnect growth multiplier (e.g. a value of 2 multiplies the autoconnect - # time by 2 on every failure, etc.) - autoconnect_multiplier = self.serverdata.get('autoconnect_multiplier', 2) - autoconnect_max = self.serverdata.get('autoconnect_max', 1800) - # These values must at least be 1. - autoconnect_multiplier = max(autoconnect_multiplier, 1) - autoconnect_max = max(autoconnect_max, 1) - - log.debug('(%s) Autoconnect delay set to %s seconds.', self.name, autoconnect) - if autoconnect is not None and autoconnect >= 1: - log.debug('(%s) Multiplying autoconnect delay %s by %s.', self.name, autoconnect, self.autoconnect_active_multiplier) - autoconnect *= self.autoconnect_active_multiplier - # Add a cap on the max. autoconnect delay, so that we don't go on forever... - autoconnect = min(autoconnect, autoconnect_max) - - log.info('(%s) Going to auto-reconnect in %s seconds.', self.name, autoconnect) - # Continue when either self.aborted is set or the autoconnect time passes. - # Compared to time.sleep(), this allows us to stop connections quicker if we - # break while while for autoconnect. - self.aborted.clear() - self.aborted.wait(autoconnect) - - # Store in the local state what the autoconnect multiplier currently is. - self.autoconnect_active_multiplier *= autoconnect_multiplier - - if self not in world.networkobjects.values(): - log.debug('Stopping stale connect loop for old connection %r', self.name) - return - - else: - log.info('(%s) Stopping connect loop (autoconnect value %r is < 1).', self.name, autoconnect) - return + raise NotImplementedError def disconnect(self): - """Handle disconnects from the remote server.""" - was_successful = self.connected.is_set() - log.debug('(%s) disconnect: got %s for was_successful state', self.name, was_successful) + raise NotImplementedError - log.debug('(%s) disconnect: Clearing self.connected state.', self.name) - self.connected.clear() - - log.debug('(%s) disconnect: Setting self.aborted to True.', self.name) - self.aborted.set() - - log.debug('(%s) Removing channel logging handlers due to disconnect.', self.name) - while self.loghandlers: - log.removeHandler(self.loghandlers.pop()) - - try: - log.debug('(%s) disconnect: Shutting down socket.', self.name) - self.socket.shutdown(socket.SHUT_RDWR) - except Exception as e: # Socket timed out during creation; ignore - log.debug('(%s) error on socket shutdown: %s: %s', self.name, type(e).__name__, e) - - self.socket.close() - - # Stop the queue thread. - if self.queue: - # XXX: queue.Queue.queue isn't actually documented, so this is probably not reliable in the long run. - self.queue.queue.appendleft(None) - - # Stop the ping timer. - if self.pingTimer: - log.debug('(%s) Canceling pingTimer at %s due to disconnect() call', self.name, time.time()) - self.pingTimer.cancel() - - # Internal hook signifying that a network has disconnected. - self.callHooks([None, 'PYLINK_DISCONNECT', {'was_successful': was_successful}]) - - log.debug('(%s) disconnect: Clearing state via initVars().', self.name) - self.initVars() - - def run(self): - """Main IRC loop which listens for messages.""" - buf = b"" - data = b"" - while not self.aborted.is_set(): - try: - data = self.socket.recv(2048) - except (BlockingIOError, ssl.SSLWantReadError, ssl.SSLWantWriteError): - log.debug('(%s) No data to read, trying again later...', self.name) - if self.aborted.wait(self.SOCKET_REPOLL_WAIT): - return - continue - except OSError: - # Suppress socket read warnings from lingering recv() calls if - # we've been told to shutdown. - if self.aborted.is_set(): - return - raise - - buf += data - if not data: - log.error('(%s) No data received, disconnecting!', self.name) - return - elif (time.time() - self.lastping) > self.pingtimeout: - log.error('(%s) Connection timed out.', self.name) - return - - while b'\n' in buf: - line, buf = buf.split(b'\n', 1) - line = line.strip(b'\r') - line = line.decode(self.encoding, "replace") - self.runline(line) - - def runline(self, line): - """Sends a command to the protocol module.""" - log.debug("(%s) <- %s", self.name, line) - try: - hook_args = self.proto.handle_events(line) - except Exception: - log.exception('(%s) Caught error in handle_events, disconnecting!', self.name) - log.error('(%s) The offending line was: <- %s', self.name, line) - self.aborted.set() - return - # Only call our hooks if there's data to process. Handlers that support - # hooks will return a dict of parsed arguments, which can be passed on - # to plugins and the like. For example, the JOIN handler will return - # something like: {'channel': '#whatever', 'users': ['UID1', 'UID2', - # 'UID3']}, etc. - if hook_args is not None: - self.callHooks(hook_args) - - return hook_args - - def callHooks(self, hook_args): + ## General utility functions + def call_hooks(self, hook_args): """Calls a hook function with the given hook args.""" numeric, command, parsed_args = hook_args # Always make sure TS is sent. if 'ts' not in parsed_args: parsed_args['ts'] = int(time.time()) hook_cmd = command - hook_map = self.proto.hook_map + hook_map = self.hook_map # If the hook name is present in the protocol module's hook_map, then we # should set the hook name to the name that points to instead. @@ -508,11 +406,18 @@ class Irc(utils.DeprecatedAttributesObject): command, hook_cmd) # Iterate over registered hook functions, catching errors accordingly. - for hook_func in world.hooks[hook_cmd]: + for hook_pair in world.hooks[hook_cmd].copy(): + hook_func = hook_pair[1] try: log.debug('(%s) Calling hook function %s from plugin "%s"', self.name, hook_func, hook_func.__module__) - hook_func(self, numeric, command, parsed_args) + retcode = hook_func(self, numeric, command, parsed_args) + + if retcode is False: + log.debug('(%s) Stopping hook loop for %r (command=%r)', self.name, + hook_func, command) + break + except Exception: # We don't want plugins to crash our servers... log.exception('(%s) Unhandled exception caught in hook %r from plugin "%s"', @@ -521,55 +426,14 @@ class Irc(utils.DeprecatedAttributesObject): hook_args) continue - def _send(self, data): - """Sends raw text to the uplink server.""" - # Safeguard against newlines in input!! Otherwise, each line gets - # treated as a separate command, which is particularly nasty. - data = data.replace('\n', ' ') - encoded_data = data.encode(self.encoding, 'replace') + b"\n" - - log.debug("(%s) -> %s", self.name, data) - - try: - self.socket.send(encoded_data) - except (OSError, AttributeError): - log.exception("(%s) Failed to send message %r; did the network disconnect?", self.name, data) - - def send(self, data, queue=True): - """send() wrapper with optional queueing support.""" - if self.aborted.is_set(): - log.debug('(%s) refusing to queue data %r as self.aborted is set', self.name, data) - return - if queue: - # XXX: we don't really know how to handle blocking queues yet, so - # it's better to not expose that yet. - self.queue.put_nowait(data) - else: - self._send(data) - - def schedulePing(self): - """Schedules periodic pings in a loop.""" - self.proto.ping() - - self.pingTimer = threading.Timer(self.pingfreq, self.schedulePing) - self.pingTimer.daemon = True - self.pingTimer.name = 'Ping timer loop for %s' % self.name - self.pingTimer.start() - - log.debug('(%s) Ping scheduled at %s', self.name, time.time()) - - def __repr__(self): - return "" % self.name - - ### General utility functions - def callCommand(self, source, text): + def call_command(self, source, text): """ Calls a PyLink bot command. source is the caller's UID, and text is the full, unparsed text of the message. """ world.services['pylink'].call_cmd(self, source, text) - def msg(self, target, text, notice=None, source=None, loopback=True): + def msg(self, target, text, notice=None, source=None, loopback=True, wrap=True): """Handy function to send messages/notices to clients. Source is optional, and defaults to the main PyLink client if not specified.""" if not text: @@ -580,27 +444,35 @@ class Irc(utils.DeprecatedAttributesObject): return source = source or self.pseudoclient.uid - if notice: - self.proto.notice(source, target, text) - cmd = 'PYLINK_SELF_NOTICE' - else: - self.proto.message(source, target, text) - cmd = 'PYLINK_SELF_PRIVMSG' + def _msg(text): + if notice: + self.notice(source, target, text) + cmd = 'PYLINK_SELF_NOTICE' + else: + self.message(source, target, text) + cmd = 'PYLINK_SELF_PRIVMSG' - if loopback: - # Determines whether we should send a hook for this msg(), to relay things like services + # Determines whether we should send a hook for this msg(), to forward things like services # replies across relay. - self.callHooks([source, cmd, {'target': target, 'text': text}]) + if loopback: + self.call_hooks([source, cmd, {'target': target, 'text': text}]) + + # Optionally wrap the text output. + if wrap: + for line in self.wrap_message(source, target, text): + _msg(line) + else: + _msg(text) def _reply(self, text, notice=None, source=None, private=None, force_privmsg_in_private=False, - loopback=True): + loopback=True, wrap=True): """ Core of the reply() function - replies to the last caller in the right context (channel or PM). """ if private is None: # Allow using private replies as the default, if no explicit setting was given. - private = conf.conf['bot'].get("prefer_private_replies") + private = conf.conf['pylink'].get("prefer_private_replies") # Private reply is enabled, or the caller was originally a PM if private or (self.called_in in self.users): @@ -613,7 +485,7 @@ class Irc(utils.DeprecatedAttributesObject): else: target = self.called_in - self.msg(target, text, notice=notice, source=source, loopback=loopback) + self.msg(target, text, notice=notice, source=source, loopback=loopback, wrap=wrap) def reply(self, *args, **kwargs): """ @@ -622,7 +494,7 @@ class Irc(utils.DeprecatedAttributesObject): This function wraps around _reply() and can be monkey-patched in a thread-safe manner to temporarily redirect plugin output to another target. """ - with self.reply_lock: + with self._reply_lock: self._reply(*args, **kwargs) def error(self, text, **kwargs): @@ -630,52 +502,293 @@ class Irc(utils.DeprecatedAttributesObject): # This is a stub to alias error to reply self.reply("Error: %s" % text, **kwargs) - def toLower(self, text): - """Returns a lowercase representation of text based on the IRC object's - casemapping (rfc1459 or ascii).""" - if self.proto.casemapping == 'rfc1459': + ## Configuration-based lookup functions. + def version(self): + """ + Returns a detailed version string including the PyLink daemon version, + the protocol module in use, and the server hostname. + """ + fullversion = 'PyLink-%s. %s :[protocol:%s, encoding:%s]' % (__version__, self.hostname(), self.protoname, self.encoding) + return fullversion + + def hostname(self): + """ + Returns the server hostname used by PyLink on the given server. + """ + return self.serverdata.get('hostname', world.fallback_hostname) + + def get_full_network_name(self): + """ + Returns the full network name (as defined by the "netname" option), or the + short network name if that isn't defined. + """ + return self.serverdata.get('netname', self.name) + + def get_service_option(self, servicename, option, default=None, global_option=None): + """ + Returns the value of the requested service bot option on the current network, or the + global value if it is not set for this network. This function queries and returns: + + 1) If present, the value of the config option servers::::_