Send Prometheus Alerts to IRC using Webhooks
Go to file
Alvar Penning 5aed9591e1 Detect connection loss through IRC PONG
In the current state, the alertmanager-irc-relay already sends minutely
IRC PINGs. This allows to check the IRC connection's health in protocol
without having to deal with specific TCP settings. However, even when
we are sending those PINGs, we don't process the server's PONGs or their
absence.

On one of my alertmanager-irc-relay instances, the time between the last
received PONG and the TCP read to fail was round about fifteen minutes.
All this time, the connection was already dead, but there was no attempt
to reestablish it.

The introduces changes keep book on the last received PONG and fails if
there was no new PONG within twice the pingFrequencySecs time. When
establishing a new connection during the SetupPhase, the current time
will be set as the last PONG's time to reset the time comparison.
2023-06-07 15:39:54 +02:00
.github/workflows Update go version list used in github workflows 2022-10-19 12:19:00 +02:00
logging logging: clarify call depth value 2021-04-13 10:44:23 +02:00
.gitignore Add Go modules vendoring. 2020-01-30 17:43:17 +01:00
backoff_test.go Add tests for channel join retry logic 2021-03-29 16:06:36 +02:00
backoff.go add own logging and define debug flag 2021-04-07 03:20:52 +02:00
config_test.go Make alert channel size configurable (and bigger) 2020-11-25 16:06:12 +01:00
config.go Detect connection loss through IRC PONG 2023-06-07 15:39:54 +02:00
context.go add own logging and define debug flag 2021-04-07 03:20:52 +02:00
CONTRIBUTING.md Initial code check-in 2018-05-21 15:49:47 +01:00
data.go s/notice/msg/ 2020-01-25 16:42:59 +00:00
Dockerfile Add a basic Docker file. 2021-05-11 23:04:17 +01:00
fake_delayer.go add own logging and define debug flag 2021-04-07 03:20:52 +02:00
fake_timeteller.go Add tests for channel join retry logic 2021-03-29 16:06:36 +02:00
format_test.go Handle alert messages with newlines 2021-11-26 12:16:45 +01:00
format.go Handle alert messages with newlines 2021-11-26 12:16:45 +01:00
go.mod Bump github.com/prometheus/client_golang from 1.9.0 to 1.11.1 2023-02-15 04:08:30 +00:00
go.sum Bump github.com/prometheus/client_golang from 1.9.0 to 1.11.1 2023-02-15 04:08:30 +00:00
http_test.go Use Context and WaitGroup for routines coordination 2021-02-24 15:49:33 +01:00
http.go add own logging and define debug flag 2021-04-07 03:20:52 +02:00
irc_server_for_test.go add own logging and define debug flag 2021-04-07 03:20:52 +02:00
irc_test.go Detect connection loss through IRC PONG 2023-06-07 15:39:54 +02:00
irc.go Detect connection loss through IRC PONG 2023-06-07 15:39:54 +02:00
LICENSE Initial code check-in 2018-05-21 15:49:47 +01:00
main.go add own logging and define debug flag 2021-04-07 03:20:52 +02:00
README.md Allow customized NickServ and ChanServ 2022-10-08 11:02:16 -05:00
reconciler_test.go Add tests for channel join retry logic 2021-03-29 16:06:36 +02:00
reconciler.go Allow customized NickServ and ChanServ 2022-10-08 11:02:16 -05:00
testdata.go Update tests to handle new fingerprint field 2020-01-25 16:16:10 +00:00
time.go Add tests for channel join retry logic 2021-03-29 16:06:36 +02:00

Alertmanager IRC Relay

Alertmanager IRC Relay is a bot relaying Prometheus alerts to IRC. Alerts are received from Prometheus using Webhooks and are relayed to an IRC channel.

Configuring and running the bot

To configure and run the bot you need to create a YAML configuration file and pass it to the service. Running the service without a configuration will use the default test values and connect to a default IRC channel, which you probably do not want to do.

Example configuration:

# Start the HTTP server receiving alerts from Prometheus Webhook binding to
# this host/port.
#
http_host: localhost
http_port: 8000

# Connect to this IRC host/port.
#
# Note: SSL is enabled by default, use "irc_use_ssl: no" to disable.
# Set "irc_verify_ssl: no" to accept invalid SSL certificates (not recommended)
irc_host: irc.example.com
irc_port: 7000
# Optionally set the server password
irc_host_password: myserver_password

# Use this IRC nickname.
irc_nickname: myalertbot
# Password used to identify with NickServ
irc_nickname_password: mynickserv_key
# Use this IRC real name
irc_realname: myrealname

# Optionally pre-join certain channels.
#
# Note: If an alert is sent to a non # pre-joined channel the bot will join
# that channel anyway before sending the message. Of course this cannot work
# with password-protected channels.
irc_channels:
  - name: "#mychannel"
  - name: "#myprivatechannel"
    password: myprivatechannel_key

# Define how IRC messages should be sent.
#
# Send only one message when webhook data is received.
# Note: By default a message is sent for each alert in the webhook data.
msg_once_per_alert_group: no
#
# Use PRIVMSG instead of NOTICE (default) to send messages.
# Note: Sending PRIVMSG from bots is bad practice, do not enable this unless
# necessary (e.g. unless NOTICEs would weaken your channel moderation policies)
use_privmsg: yes

# Define how IRC messages should be formatted.
#
# The formatting is based on golang's text/template .
msg_template: "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}"
# Note: When sending only one message per alert group the default
# msg_template is set to
# "Alert {{ .GroupLabels.alertname }} for {{ .GroupLabels.job }} is {{ .Status }}"

# Set the internal buffer size for alerts received but not yet sent to IRC.
alert_buffer_size: 2048

# Patterns used to guess whether NickServ is asking us to IDENTIFY
# Note: If you need to change this because the bot is not catching a request
# from a rather common NickServ, please consider sending a PR to update the
# default config instead.
nickserv_identify_patterns:
  - "identify via /msg NickServ identify <password>"
  - "type /msg NickServ IDENTIFY password"
  - "authenticate yourself to services with the IDENTIFY command"

# Rarely NickServ or ChanServ is reached at a specific hostname.  Specify an
# override here
nickserv_name: NickServ
chanserv_name: ChanServ

Running the bot (assuming GOPATH* and *PATH are properly setup for go):

$ go install github.com/google/alertmanager-irc-relay
$ alertmanager-irc-relay --config /path/to/your/config/file

The configuration file can reference environment variables. It is then possible to specify certain parameters directly when running the bot:

$ cat /path/to/your/config/file
...
http_port: $MY_SERVICE_PORT
...
irc_nickname_password: $NICKSERV_PASSWORD
...
$ export MY_SERVICE_PORT=8000 NICKSERV_PASSWORD=mynickserv_key
$ alertmanager-irc-relay --config /path/to/your/config/file

Prometheus configuration

Prometheus can be configured following the official Webhooks documentation. The url must specify the IRC channel name that alerts should be sent to:

send_resolved: false
url: http://localhost:8000/mychannel