d47139a6d6
After a connection loss on an IRC session with a ngIRCd, the alertmanager-irc-relay was unable to reconnect. After some debugging, the error's origin was the state tracking within the used goirc library. When using an unidentified session, ngIRCd prefixes the user's ident with a `~`. The state tracking registers this and keeps `~${NICK}` as the current and the new ident for future reconnects. However, `~` is not a valid char for the `<user>` part in the `USER` command, at least not for ngIRCd. To clarify this behaviour, take a look at the following log. First, the initial connection is begin established correctly. Keep an eye on the `USER` command being sent to the server. > http.go:132: INFO Starting HTTP server > irc.go:308: INFO Connected to IRC server, waiting to establish session > connection.go:543: DEBUG -> NICK alertbot > connection.go:543: DEBUG -> USER alertbot 12 * :Alertmanager IRC Relay > connection.go:474: DEBUG <- :__SERVER__ 001 alertbot :Welcome to the Internet Relay Network alertbot!~alertbot@__IP__ Now, there was a network incident and the session needs to be recreated. > connection.go:466: ERROR irc.recv(): read tcp __REDACTED__: read: connection timed out > connection.go:577: INFO irc.Close(): Disconnected from server. > irc.go:150: INFO Disconnected from IRC > reconciler.go:129: INFO Channel #alerts monitor: context canceled while monitoring > irc.go:300: INFO Connecting to IRC __SERVER__ > backoff.go:111: INFO Backoff for 0s starts > backoff.go:114: INFO Backoff for 0s ends > connection.go:390: INFO irc.Connect(): Connecting to __SERVER__. > irc.go:308: INFO Connected to IRC server, waiting to establish session > connection.go:543: DEBUG -> NICK alertbot > connection.go:543: DEBUG -> USER ~alertbot 12 * :Alertmanager IRC Relay > connection.go:474: DEBUG <- ERROR :Invalid user name > connection.go:577: INFO irc.Close(): Disconnected from server. > irc.go:150: INFO Disconnected from IRC > irc.go:319: WARN Receiving a session down before the session is up, this is odd This time, the used `user` part of the `USER` command has the prefixed `~` and fails. However, without using `-debug` and taking a very close look, this error can be missed very easy. As the new ident is invalid, the alertmanager-irc-relay is now stuck in an endless reconnection loop. This fix is kind of straight forward and just checks if the ident has changed before trying to reconnect. It might not be the prettiest solution, but recreating the whole *irc.Config resulted in other bugs as it was still referenced - even after being `Close`d. |
||
---|---|---|
.github/workflows | ||
logging | ||
.gitignore | ||
backoff_test.go | ||
backoff.go | ||
config_test.go | ||
config.go | ||
context.go | ||
CONTRIBUTING.md | ||
data.go | ||
Dockerfile | ||
fake_delayer.go | ||
fake_timeteller.go | ||
format_test.go | ||
format.go | ||
go.mod | ||
go.sum | ||
http_test.go | ||
http.go | ||
irc_server_for_test.go | ||
irc_test.go | ||
irc.go | ||
LICENSE | ||
main.go | ||
README.md | ||
reconciler_test.go | ||
reconciler.go | ||
testdata.go | ||
time.go |
Alertmanager IRC Relay
Alertmanager IRC Relay is a bot relaying Prometheus alerts to IRC. Alerts are received from Prometheus using Webhooks and are relayed to an IRC channel.
Configuring and running the bot
To configure and run the bot you need to create a YAML configuration file and pass it to the service. Running the service without a configuration will use the default test values and connect to a default IRC channel, which you probably do not want to do.
Example configuration:
# Start the HTTP server receiving alerts from Prometheus Webhook binding to
# this host/port.
#
http_host: localhost
http_port: 8000
# Connect to this IRC host/port.
#
# Note: SSL is enabled by default, use "irc_use_ssl: no" to disable.
# Set "irc_verify_ssl: no" to accept invalid SSL certificates (not recommended)
irc_host: irc.example.com
irc_port: 7000
# Optionally set the server password
irc_host_password: myserver_password
# Use this IRC nickname.
irc_nickname: myalertbot
# Password used to identify with NickServ
irc_nickname_password: mynickserv_key
# Use this IRC real name
irc_realname: myrealname
# Optionally pre-join certain channels.
#
# Note: If an alert is sent to a non # pre-joined channel the bot will join
# that channel anyway before sending the message. Of course this cannot work
# with password-protected channels.
irc_channels:
- name: "#mychannel"
- name: "#myprivatechannel"
password: myprivatechannel_key
# Define how IRC messages should be sent.
#
# Send only one message when webhook data is received.
# Note: By default a message is sent for each alert in the webhook data.
msg_once_per_alert_group: no
#
# Use PRIVMSG instead of NOTICE (default) to send messages.
# Note: Sending PRIVMSG from bots is bad practice, do not enable this unless
# necessary (e.g. unless NOTICEs would weaken your channel moderation policies)
use_privmsg: yes
# Define how IRC messages should be formatted.
#
# The formatting is based on golang's text/template .
msg_template: "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}"
# Note: When sending only one message per alert group the default
# msg_template is set to
# "Alert {{ .GroupLabels.alertname }} for {{ .GroupLabels.job }} is {{ .Status }}"
# Set the internal buffer size for alerts received but not yet sent to IRC.
alert_buffer_size: 2048
# Patterns used to guess whether NickServ is asking us to IDENTIFY
# Note: If you need to change this because the bot is not catching a request
# from a rather common NickServ, please consider sending a PR to update the
# default config instead.
nickserv_identify_patterns:
- "identify via /msg NickServ identify <password>"
- "type /msg NickServ IDENTIFY password"
- "authenticate yourself to services with the IDENTIFY command"
# Rarely NickServ or ChanServ is reached at a specific hostname. Specify an
# override here
nickserv_name: NickServ
chanserv_name: ChanServ
Running the bot (assuming GOPATH* and *PATH are properly setup for go):
$ go install github.com/google/alertmanager-irc-relay
$ alertmanager-irc-relay --config /path/to/your/config/file
The configuration file can reference environment variables. It is then possible to specify certain parameters directly when running the bot:
$ cat /path/to/your/config/file
...
http_port: $MY_SERVICE_PORT
...
irc_nickname_password: $NICKSERV_PASSWORD
...
$ export MY_SERVICE_PORT=8000 NICKSERV_PASSWORD=mynickserv_key
$ alertmanager-irc-relay --config /path/to/your/config/file
Prometheus configuration
Prometheus can be configured following the official Webhooks
documentation. The url
must specify the IRC channel name
that alerts should be sent to:
send_resolved: false
url: http://localhost:8000/mychannel