pratyush/alertmanager-irc-relay
1
0
Fork 0
mirror of https://github.com/google/alertmanager-irc-relay.git synced 2024-11-26 21:19:30 +01:00
Code Issues Projects Releases Wiki Activity

Initial code check-in

Browse Source 
Signed-off-by: Luca Bigliardi <shammash@google.com>
This commit is contained in:
Luca Bigliardi 2018-05-21 15:36:45 +01:00
parent 522822e703
commit 60632b16e6
14 changed files with 2250 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,28 @@
# How to Contribute
We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.
## Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution,
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to <https://cla.developers.google.com/> to see
your current agreements on file or to sign a new one.
You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.
## Code reviews
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
## Community Guidelines
This project follows [Google's Open Source Community
Guidelines](https://opensource.google.com/conduct/).

202
LICENSE Normal file
View File

@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

78
README.md Normal file
View File

@ -0,0 +1,78 @@
# Alertmanager IRC Relay
Alertmanager IRC Relay is a bot relaying [Prometheus](https://prometheus.io/) alerts to IRC.
Alerts are received from Prometheus using
[Webhooks](https://prometheus.io/docs/alerting/configuration/#webhook-receiver-<webhook_config>)
and are relayed to an IRC channel.
### Configuring and running the bot
To configure and run the bot you need to create a YAML configuration file and
pass it to the service. Running the service without a configuration will use
the default test values and connect to a default IRC channel, which you
probably do not want to do.
Example configuration:
```
# Start the HTTP server receiving alerts from Prometheus Webhook binding to
# this host/port.
#
http_host: localhost
http_port: 8000
# Connect to this IRC host/port.
#
# Note: SSL is enabled by default, use "irc_use_ssl: no" to disable.
irc_host: irc.example.com
irc_port: 7000
# Use this IRC nickname.
irc_nickname: myalertbot
# Password used to identify with NickServ
irc_nickname_password: mynickserv_key
# Use this IRC real name
irc_realname: myrealname
# Optionally pre-join certain channels.
#
# Note: If an alert is sent to a non # pre-joined channel the bot will join
# that channel anyway before sending the notice. Of course this cannot work
# with password-protected channels.
irc_channels:
- name: "#mychannel"
- name: "#myprivatechannel"
password: myprivatechannel_key
# Define how IRC messages should be sent.
#
# Send only one notice when webhook data is received.
# Note: By default a notice is sent for each alert in the webhook data.
notice_once_per_alert_group: no
# Define how IRC messages should be formatted.
#
# The formatting is based on golang's text/template .
notice_template: "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}"
# Note: When sending only one notice per alert group the default
# notice_template is set to
# "Alert {{ .GroupLabels.alertname }} for {{ .GroupLabels.job }} is {{ .Status }}"
```
Running the bot (assuming *$GOPATH* and *$PATH* are properly setup for go):
```
$ go install github.com/google/alertmanager-irc-relay
$ alertmanager-irc-relay --config /path/to/your/config/file
```
### Prometheus configuration
Prometheus can be configured following the official
[Webhooks](https://prometheus.io/docs/alerting/configuration/#webhook-receiver-<webhook_config>)
documentation. The `url` must specify the IRC channel name that alerts should
be sent to:
```
send_resolved: false
url: http://localhost:8000/mychannel
```

102
backoff.go Normal file
View File

@ -0,0 +1,102 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"log"
"math"
"math/rand"
"time"
)
type JitterFunc func(int) int
type TimeFunc func() time.Time
type Delayer interface {
Delay()
}
type Backoff struct {
step float64
maxBackoff float64
resetDelta float64
lastAttempt time.Time
durationUnit time.Duration
jitterer JitterFunc
timeGetter TimeFunc
}
func jitterFunc(input int) int {
if input == 0 {
return 0
}
return rand.Intn(input)
}
func NewBackoff(maxBackoff float64, resetDelta float64,
durationUnit time.Duration) *Backoff {
return NewBackoffForTesting(
maxBackoff, resetDelta, durationUnit, jitterFunc, time.Now)
}
func NewBackoffForTesting(maxBackoff float64, resetDelta float64,
durationUnit time.Duration, jitterer JitterFunc, timeGetter TimeFunc) *Backoff {
return &Backoff{
step: 0,
maxBackoff: maxBackoff,
resetDelta: resetDelta,
lastAttempt: timeGetter(),
durationUnit: durationUnit,
jitterer: jitterer,
timeGetter: timeGetter,
}
}
func (b *Backoff) maybeReset() {
now := b.timeGetter()
lastAttemptDelta := float64(now.Sub(b.lastAttempt) / b.durationUnit)
b.lastAttempt = now
if lastAttemptDelta >= b.resetDelta {
b.step = 0
}
}
func (b *Backoff) GetDelay() time.Duration {
b.maybeReset()
var synchronizedDuration float64
// Do not add any delay the first time.
if b.step == 0 {
synchronizedDuration = 0
} else {
synchronizedDuration = math.Pow(2, b.step)
}
if synchronizedDuration < b.maxBackoff {
b.step++
} else {
synchronizedDuration = b.maxBackoff
}
duration := time.Duration(b.jitterer(int(synchronizedDuration)))
return duration * b.durationUnit
}
func (b *Backoff) Delay() {
delay := b.GetDelay()
log.Printf("Backoff for %s", delay)
time.Sleep(delay)
}

80
backoff_test.go Normal file
View File

@ -0,0 +1,80 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"testing"
"time"
)
type FakeTime struct {
timeseries []int
lastIndex int
durationUnit time.Duration
}
func (f *FakeTime) GetTime() time.Time {
timeDelta := time.Duration(f.timeseries[f.lastIndex]) * f.durationUnit
fakeTime := time.Unix(0, 0).Add(timeDelta)
f.lastIndex++
return fakeTime
}
func FakeJitter(input int) int {
return input
}
func RunBackoffTest(t *testing.T,
maxBackoff float64, resetDelta float64,
elapsedTime []int, expectedDelays []int) {
fakeTime := &FakeTime{
timeseries: elapsedTime,
lastIndex: 0,
durationUnit: time.Millisecond,
}
backoff := NewBackoffForTesting(maxBackoff, resetDelta, time.Millisecond,
FakeJitter, fakeTime.GetTime)
for i, value := range expectedDelays {
expected_delay := time.Duration(value) * time.Millisecond
delay := backoff.GetDelay()
if expected_delay != delay {
t.Errorf("Call #%d of GetDelay returned %s (expected %s)",
i, delay, expected_delay)
}
}
}
func TestBackoffIncreasesAndReachesMax(t *testing.T) {
RunBackoffTest(t,
8,
32,
// Simple sequential time
[]int{0, 0, 1, 2, 3, 4, 5, 6, 7},
// Exponential ramp-up to max, then keep max.
[]int{0, 2, 4, 8, 8, 8, 8, 8},
)
}
func TestBackoffReset(t *testing.T) {
RunBackoffTest(t,
8,
32,
// Simulate two intervals bigger than resetDelta
[]int{0, 0, 1, 2, 50, 51, 100, 101, 102},
// Delays get reset each time
[]int{0, 2, 4, 0, 2, 0, 2, 4},
)
}

80
config.go Normal file
View File

@ -0,0 +1,80 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"gopkg.in/yaml.v2"
"io/ioutil"
)
const (
defaultNoticeOnceTemplate = "Alert {{ .GroupLabels.alertname }} for {{ .GroupLabels.job }} is {{ .Status }}"
defaultNoticeTemplate = "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}"
)
type IRCChannel struct {
Name string `yaml:"name"`
Password string `yaml:"password"`
}
type Config struct {
HTTPHost string `yaml:"http_host"`
HTTPPort int `yaml:"http_port"`
IRCNick string `yaml:"irc_nickname"`
IRCNickPass string `yaml:"irc_nickname_password"`
IRCRealName string `yaml:"irc_realname"`
IRCHost string `yaml:"irc_host"`
IRCPort int `yaml:"irc_port"`
IRCUseSSL bool `yaml:"irc_use_ssl"`
IRCChannels []IRCChannel `yaml:"irc_channels"`
NoticeTemplate string `yaml:"notice_template"`
NoticeOnce bool `yaml:"notice_once_per_alert_group"`
}
func LoadConfig(configFile string) (*Config, error) {
config := &Config{
HTTPHost: "localhost",
HTTPPort: 8000,
IRCNick: "alertmanager-irc-relay",
IRCNickPass: "",
IRCRealName: "Alertmanager IRC Relay",
IRCHost: "irc.freenode.net",
IRCPort: 7000,
IRCUseSSL: true,
IRCChannels: []IRCChannel{IRCChannel{Name: "#airtest"}},
NoticeOnce: false,
}
if configFile != "" {
data, err := ioutil.ReadFile(configFile)
if err != nil {
return nil, err
}
if err := yaml.Unmarshal(data, config); err != nil {
return nil, err
}
}
// Set default template if config does not have one.
if config.NoticeTemplate == "" {
if config.NoticeOnce {
config.NoticeTemplate = defaultNoticeOnceTemplate
} else {
config.NoticeTemplate = defaultNoticeTemplate
}
}
return config, nil
}

180
config_test.go Normal file
View File

@ -0,0 +1,180 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"fmt"
"gopkg.in/yaml.v2"
"io/ioutil"
"os"
"testing"
)
func TestNoConfig(t *testing.T) {
noConfigFile := ""
config, err := LoadConfig(noConfigFile)
if config == nil {
t.Errorf("Expected a default config, got: %s", err)
}
}
func TestLoadGoodConfig(t *testing.T) {
expectedConfig := &Config{
HTTPHost: "test.web",
HTTPPort: 8888,
IRCNick: "foo",
IRCHost: "irc.example.com",
IRCPort: 1234,
IRCUseSSL: true,
IRCChannels: []IRCChannel{IRCChannel{Name: "#foobar"}},
NoticeTemplate: defaultNoticeTemplate,
NoticeOnce: false,
}
expectedData, err := yaml.Marshal(expectedConfig)
if err != nil {
t.Errorf("Could not serialize test data: %s", err)
}
tmpfile, err := ioutil.TempFile("", "airtestconfig")
if err != nil {
t.Errorf("Could not create tmpfile for testing: %s", err)
}
defer os.Remove(tmpfile.Name())
if _, err := tmpfile.Write(expectedData); err != nil {
t.Errorf("Could not write test data in tmpfile: %s", err)
}
if err := tmpfile.Close(); err != nil {
t.Errorf("Could not close tmpfile: %s", err)
}
config, err := LoadConfig(tmpfile.Name())
if config == nil {
t.Errorf("Expected a config, got: %s", err)
}
configData, err := yaml.Marshal(config)
if err != nil {
t.Errorf("Could not serialize loaded config")
}
if string(expectedData) != string(configData) {
t.Errorf("Loaded config does not match expected config: %s", configData)
}
}
func TestLoadBadFile(t *testing.T) {
tmpfile, err := ioutil.TempFile("", "airtestbadfile")
if err != nil {
t.Errorf("Could not create tmpfile for testing: %s", err)
}
tmpfile.Close()
os.Remove(tmpfile.Name())
config, err := LoadConfig(tmpfile.Name())
if config != nil {
t.Errorf("Expected no config upon non-existent file.")
}
}
func TestLoadBadConfig(t *testing.T) {
tmpfile, err := ioutil.TempFile("", "airtestbadconfig")
if err != nil {
t.Errorf("Could not create tmpfile for testing: %s", err)
}
defer os.Remove(tmpfile.Name())
badConfigData := []byte("footest\nbarbaz\n")
if _, err := tmpfile.Write(badConfigData); err != nil {
t.Errorf("Could not write test data in tmpfile: %s", err)
}
tmpfile.Close()
config, err := LoadConfig(tmpfile.Name())
if config != nil {
t.Errorf("Expected no config upon bad config.")
}
}
func TestNoticeOnceDefaultTemplate(t *testing.T) {
tmpfile, err := ioutil.TempFile("", "airtesttemmplateonceconfig")
if err != nil {
t.Errorf("Could not create tmpfile for testing: %s", err)
}
defer os.Remove(tmpfile.Name())
noticeOnceConfigData := []byte("notice_once_per_alert_group: yes")
if _, err := tmpfile.Write(noticeOnceConfigData); err != nil {
t.Errorf("Could not write test data in tmpfile: %s", err)
}
tmpfile.Close()
config, err := LoadConfig(tmpfile.Name())
if config == nil {
t.Errorf("Expected a config, got: %s", err)
}
if config.NoticeTemplate != defaultNoticeOnceTemplate {
t.Errorf("Expecting defaultNoticeOnceTemplate when NoticeOnce is true")
}
}
func TestNoticeDefaultTemplate(t *testing.T) {
tmpfile, err := ioutil.TempFile("", "airtesttemmplateconfig")
if err != nil {
t.Errorf("Could not create tmpfile for testing: %s", err)
}
defer os.Remove(tmpfile.Name())
if _, err := tmpfile.Write([]byte("")); err != nil {
t.Errorf("Could not write test data in tmpfile: %s", err)
}
tmpfile.Close()
config, err := LoadConfig(tmpfile.Name())
if config == nil {
t.Errorf("Expected a config, got: %s", err)
}
if config.NoticeTemplate != defaultNoticeTemplate {
t.Errorf("Expecting defaultNoticeTemplate when NoticeOnce is false")
}
}
func TestGivenTemplateNotOverwritten(t *testing.T) {
tmpfile, err := ioutil.TempFile("", "airtestexpectedtemmplate")
if err != nil {
t.Errorf("Could not create tmpfile for testing: %s", err)
}
defer os.Remove(tmpfile.Name())
expectedTemplate := "Alert {{ .Status }}: {{ .Annotations.SUMMARY }}"
configData := []byte(fmt.Sprintf("notice_template: \"%s\"", expectedTemplate))
if _, err := tmpfile.Write(configData); err != nil {
t.Errorf("Could not write test data in tmpfile: %s", err)
}
tmpfile.Close()
config, err := LoadConfig(tmpfile.Name())
if config == nil {
t.Errorf("Expected a config, got: %s", err)
}
if config.NoticeTemplate != expectedTemplate {
t.Errorf("Template does not match configuration")
}
}

19
data.go Normal file
View File

@ -0,0 +1,19 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
type AlertNotice struct {
Channel, Alert string
}

148
http.go Normal file
View File

@ -0,0 +1,148 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"bytes"
"encoding/json"
"io"
"io/ioutil"
"log"
"net/http"
"github.com/gorilla/mux"
promtmpl "github.com/prometheus/alertmanager/template"
"strconv"
"strings"
"text/template"
)
type HTTPListener func(string, http.Handler) error
type HTTPServer struct {
StoppedRunning chan bool
Addr string
Port int
NoticeTemplate *template.Template
NoticeOnce bool
AlertNotices chan AlertNotice
httpListener HTTPListener
}
func NewHTTPServer(config *Config, alertNotices chan AlertNotice) (
*HTTPServer, error) {
return NewHTTPServerForTesting(config, alertNotices, http.ListenAndServe)
}
func NewHTTPServerForTesting(config *Config, alertNotices chan AlertNotice,
httpListener HTTPListener) (*HTTPServer, error) {
tmpl, err := template.New("notice").Parse(config.NoticeTemplate)
if err != nil {
return nil, err
}
server := &HTTPServer{
StoppedRunning: make(chan bool),
Addr: config.HTTPHost,
Port: config.HTTPPort,
NoticeTemplate: tmpl,
NoticeOnce: config.NoticeOnce,
AlertNotices: alertNotices,
httpListener: httpListener,
}
return server, nil
}
func (server *HTTPServer) FormatNotice(data interface{}) string {
output := bytes.Buffer{}
var msg string
if err := server.NoticeTemplate.Execute(&output, data); err != nil {
msg_bytes, _ := json.Marshal(data)
msg = string(msg_bytes)
log.Printf("Could not apply notice template on alert (%s): %s",
err, msg)
log.Printf("Sending raw alert")
} else {
msg = output.String()
}
return msg
}
func (server *HTTPServer) GetNoticesFromAlertMessage(ircChannel string,
data *promtmpl.Data) []AlertNotice {
notices := []AlertNotice{}
if server.NoticeOnce {
msg := server.FormatNotice(data)
notices = append(notices,
AlertNotice{Channel: ircChannel, Alert: msg})
} else {
for _, alert := range data.Alerts {
msg := server.FormatNotice(alert)
notices = append(notices,
AlertNotice{Channel: ircChannel, Alert: msg})
}
}
return notices
}
func (server *HTTPServer) RelayAlert(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
ircChannel := "#" + vars["IRCChannel"]
body, err := ioutil.ReadAll(io.LimitReader(r.Body, 1024*1024*1024))
if err != nil {
log.Printf("Could not get body: %s", err)
return
}
var alertMessage = promtmpl.Data{}
if err := json.Unmarshal(body, &alertMessage); err != nil {
log.Printf("Could not decode request body (%s): %s", err, body)
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
w.WriteHeader(422) // Unprocessable entity
if err := json.NewEncoder(w).Encode(err); err != nil {
log.Printf("Could not write decoding error: %s", err)
return
}
return
}
for _, alertNotice := range server.GetNoticesFromAlertMessage(
ircChannel, &alertMessage) {
select {
case server.AlertNotices <- alertNotice:
default:
log.Printf("Could not send this alert to the IRC routine: %s",
alertNotice)
}
}
}
func (server *HTTPServer) Run() {
router := mux.NewRouter().StrictSlash(true)
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
server.RelayAlert(w, r)
})
router.Path("/{IRCChannel}").Handler(handler).Methods("POST")
listenAddr := strings.Join(
[]string{server.Addr, strconv.Itoa(server.Port)}, ":")
log.Printf("Starting HTTP server")
if err := server.httpListener(listenAddr, router); err != nil {
log.Printf("Could not start http server: %s", err)
}
server.StoppedRunning <- true
}

218
http_test.go Normal file
View File

@ -0,0 +1,218 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"reflect"
"strings"
"testing"
)
type FakeHTTPListener struct {
StartedServing chan bool
StopServing chan bool
AlertNotices chan AlertNotice // kinda ugly putting it here, but convenient
router http.Handler
}
func (listener *FakeHTTPListener) Serve(_ string, router http.Handler) error {
listener.router = router
listener.StartedServing <- true
<-listener.StopServing
return nil
}
func NewFakeHTTPListener() *FakeHTTPListener {
return &FakeHTTPListener{
StartedServing: make(chan bool),
StopServing: make(chan bool),
AlertNotices: make(chan AlertNotice, 10),
}
}
func MakeHTTPTestingConfig() *Config {
return &Config{
HTTPHost: "test.web",
HTTPPort: 8888,
NoticeTemplate: "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}",
}
}
func RunHTTPTest(t *testing.T,
alertData string, url string,
testingConfig *Config, listener *FakeHTTPListener) *http.Response {
httpServer, err := NewHTTPServerForTesting(testingConfig,
listener.AlertNotices, listener.Serve)
if err != nil {
t.Fatal(fmt.Sprintf("Could not create HTTP server: %s", err))
}
go httpServer.Run()
<-listener.StartedServing
alertDataReader := strings.NewReader(alertData)
request, err := http.NewRequest("POST", url, alertDataReader)
if err != nil {
t.Fatal(fmt.Sprintf("Could not create HTTP request: %s", err))
}
responseRecorder := httptest.NewRecorder()
listener.router.ServeHTTP(responseRecorder, request)
listener.StopServing <- true
<-httpServer.StoppedRunning
return responseRecorder.Result()
}
func TestAlertsDispatched(t *testing.T) {
listener := NewFakeHTTPListener()
testingConfig := MakeHTTPTestingConfig()
expectedAlertNotices := []AlertNotice{
AlertNotice{
Channel: "#somechannel",
Alert: "Alert airDown on instance1:3456 is resolved",
},
AlertNotice{
Channel: "#somechannel",
Alert: "Alert airDown on instance2:7890 is resolved",
},
}
expectedStatusCode := 200
response := RunHTTPTest(
t, testdataSimpleAlertJson, "/somechannel",
testingConfig, listener)
if expectedStatusCode != response.StatusCode {
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
expectedStatusCode, response.StatusCode))
}
for _, expectedAlertNotice := range expectedAlertNotices {
alertNotice := <-listener.AlertNotices
if !reflect.DeepEqual(expectedAlertNotice, alertNotice) {
t.Error(fmt.Sprintf(
"Unexpected alert notice.\nExpected: %s\nActual: %s",
expectedAlertNotice, alertNotice))
}
}
}
func TestAlertsDispatchedOnce(t *testing.T) {
listener := NewFakeHTTPListener()
testingConfig := MakeHTTPTestingConfig()
testingConfig.NoticeOnce = true
testingConfig.NoticeTemplate = "Alert {{ .GroupLabels.alertname }} is {{ .Status }}"
expectedAlertNotices := []AlertNotice{
AlertNotice{
Channel: "#somechannel",
Alert: "Alert airDown is resolved",
},
}
expectedStatusCode := 200
response := RunHTTPTest(
t, testdataSimpleAlertJson, "/somechannel",
testingConfig, listener)
if expectedStatusCode != response.StatusCode {
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
expectedStatusCode, response.StatusCode))
}
for _, expectedAlertNotice := range expectedAlertNotices {
alertNotice := <-listener.AlertNotices
if !reflect.DeepEqual(expectedAlertNotice, alertNotice) {
t.Error(fmt.Sprintf(
"Unexpected alert notice.\nExpected: %s\nActual: %s",
expectedAlertNotice, alertNotice))
}
}
}
func TestRootReturnsError(t *testing.T) {
listener := NewFakeHTTPListener()
testingConfig := MakeHTTPTestingConfig()
expectedStatusCode := 404
response := RunHTTPTest(
t, testdataSimpleAlertJson, "/",
testingConfig, listener)
if expectedStatusCode != response.StatusCode {
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
expectedStatusCode, response.StatusCode))
}
}
func TestInvalidDataReturnsError(t *testing.T) {
listener := NewFakeHTTPListener()
testingConfig := MakeHTTPTestingConfig()
expectedStatusCode := 422
response := RunHTTPTest(
t, testdataBogusAlertJson, "/somechannel",
testingConfig, listener)
if expectedStatusCode != response.StatusCode {
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
expectedStatusCode, response.StatusCode))
}
}
func TestTemplateErrorsCreateRawAlertNotice(t *testing.T) {
listener := NewFakeHTTPListener()
testingConfig := MakeHTTPTestingConfig()
testingConfig.NoticeTemplate = "Bogus template {{ nil }}"
expectedAlertNotices := []AlertNotice{
AlertNotice{
Channel: "#somechannel",
Alert: `{"status":"resolved","labels":{"alertname":"airDown","instance":"instance1:3456","job":"air","service":"prometheus","severity":"ticket","zone":"global"},"annotations":{"DESCRIPTION":"service /prometheus has irc gateway down on instance1","SUMMARY":"service /prometheus air down on instance1"},"startsAt":"2017-05-15T13:49:37.834Z","endsAt":"2017-05-15T13:50:37.835Z","generatorURL":"https://prometheus.example.com/prometheus/..."}`,
},
AlertNotice{
Channel: "#somechannel",
Alert: `{"status":"resolved","labels":{"alertname":"airDown","instance":"instance2:7890","job":"air","service":"prometheus","severity":"ticket","zone":"global"},"annotations":{"DESCRIPTION":"service /prometheus has irc gateway down on instance2","SUMMARY":"service /prometheus air down on instance2"},"startsAt":"2017-05-15T11:47:37.834Z","endsAt":"2017-05-15T11:48:37.834Z","generatorURL":"https://prometheus.example.com/prometheus/..."}`,
},
}
expectedStatusCode := 200
response := RunHTTPTest(
t, testdataSimpleAlertJson, "/somechannel",
testingConfig, listener)
if expectedStatusCode != response.StatusCode {
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
expectedStatusCode, response.StatusCode))
}
for _, expectedAlertNotice := range expectedAlertNotices {
alertNotice := <-listener.AlertNotices
if !reflect.DeepEqual(expectedAlertNotice, alertNotice) {
t.Error(fmt.Sprintf(
"Unexpected alert notice.\nExpected: %s\nActual: %s",
expectedAlertNotice, alertNotice))
}
}
}

251
irc.go Normal file
View File

@ -0,0 +1,251 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"crypto/tls"
irc "github.com/fluffle/goirc/client"
"log"
"strconv"
"strings"
"time"
)
const (
pingFrequencySecs = 60
connectionTimeoutSecs = 30
nickservWaitSecs = 10
ircConnectMaxBackoffSecs = 300
ircConnectBackoffResetSecs = 1800
)
func loggerHandler(_ *irc.Conn, line *irc.Line) {
log.Printf("Received: '%s'", line.Raw)
}
type ChannelState struct {
Channel IRCChannel
BackoffCounter Delayer
}
type IRCNotifier struct {
// Nick stores the nickname specified in the config, because irc.Client
// might change its copy.
Nick string
NickPassword string
Client *irc.Conn
StopRunning chan bool
StoppedRunning chan bool
AlertNotices chan AlertNotice
// irc.Conn has a Connected() method that can tell us wether the TCP
// connection is up, and thus if we should trigger connect/disconnect.
// We need to track the session establishment also at a higher level to
// understand when the server has accepted us and thus when we can join
// channels, send notices, etc.
sessionUp bool
sessionUpSignal chan bool
sessionDownSignal chan bool
PreJoinChannels []IRCChannel
JoinedChannels map[string]ChannelState
NickservDelayWait time.Duration
BackoffCounter Delayer
}
func NewIRCNotifier(config *Config, alertNotices chan AlertNotice) (*IRCNotifier, error) {
ircConfig := irc.NewConfig(config.IRCNick)
ircConfig.Me.Ident = config.IRCNick
ircConfig.Me.Name = config.IRCRealName
ircConfig.Server = strings.Join(
[]string{config.IRCHost, strconv.Itoa(config.IRCPort)}, ":")
ircConfig.SSL = config.IRCUseSSL
ircConfig.SSLConfig = &tls.Config{ServerName: config.IRCHost}
ircConfig.PingFreq = pingFrequencySecs * time.Second
ircConfig.Timeout = connectionTimeoutSecs * time.Second
ircConfig.NewNick = func(n string) string { return n + "^" }
backoffCounter := NewBackoff(
ircConnectMaxBackoffSecs, ircConnectBackoffResetSecs,
time.Second)
notifier := &IRCNotifier{
Nick: config.IRCNick,
NickPassword: config.IRCNickPass,
Client: irc.Client(ircConfig),
StopRunning: make(chan bool),
StoppedRunning: make(chan bool),
AlertNotices: alertNotices,
sessionUpSignal: make(chan bool),
sessionDownSignal: make(chan bool),
PreJoinChannels: config.IRCChannels,
JoinedChannels: make(map[string]ChannelState),
NickservDelayWait: nickservWaitSecs * time.Second,
BackoffCounter: backoffCounter,
}
notifier.Client.HandleFunc(irc.CONNECTED,
func(*irc.Conn, *irc.Line) {
log.Printf("Session established")
notifier.sessionUpSignal <- true
})
notifier.Client.HandleFunc(irc.DISCONNECTED,
func(*irc.Conn, *irc.Line) {
log.Printf("Disconnected from IRC")
notifier.sessionDownSignal <- false
})
notifier.Client.HandleFunc(irc.KICK,
func(_ *irc.Conn, line *irc.Line) {
notifier.HandleKick(line.Args[1], line.Args[0])
})
for _, event := range []string{irc.NOTICE, "433"} {
notifier.Client.HandleFunc(event, loggerHandler)
}
return notifier, nil
}
func (notifier *IRCNotifier) HandleKick(nick string, channel string) {
if nick != notifier.Client.Me().Nick {
// received kick info for somebody else
return
}
state, ok := notifier.JoinedChannels[channel]
if ok == false {
log.Printf("Being kicked out of non-joined channel (%s), ignoring", channel)
return
}
log.Printf("Being kicked out of %s, re-joining", channel)
go func() {
state.BackoffCounter.Delay()
notifier.Client.Join(state.Channel.Name, state.Channel.Password)
}()
}
func (notifier *IRCNotifier) CleanupChannels() {
log.Printf("Deregistering all channels.")
notifier.JoinedChannels = make(map[string]ChannelState)
}
func (notifier *IRCNotifier) JoinChannel(channel *IRCChannel) {
if _, joined := notifier.JoinedChannels[channel.Name]; joined == true {
return
}
log.Printf("Joining %s", channel.Name)
notifier.Client.Join(channel.Name, channel.Password)
state := ChannelState{
Channel: *channel,
BackoffCounter: NewBackoff(
ircConnectMaxBackoffSecs, ircConnectBackoffResetSecs,
time.Second),
}
notifier.JoinedChannels[channel.Name] = state
}
func (notifier *IRCNotifier) JoinChannels() {
for _, channel := range notifier.PreJoinChannels {
notifier.JoinChannel(&channel)
}
}
func (notifier *IRCNotifier) MaybeIdentifyNick() {
if notifier.NickPassword == "" {
return
}
// Very lazy/optimistic, but this is good enough for my irssi config,
// so it should work here as well.
currentNick := notifier.Client.Me().Nick
if currentNick != notifier.Nick {
log.Printf("My nick is '%s', sending GHOST to NickServ to get '%s'",
currentNick, notifier.Nick)
notifier.Client.Privmsgf("NickServ", "GHOST %s %s", notifier.Nick,
notifier.NickPassword)
time.Sleep(notifier.NickservDelayWait)
log.Printf("Changing nick to '%s'", notifier.Nick)
notifier.Client.Nick(notifier.Nick)
}
log.Printf("Sending IDENTIFY to NickServ")
notifier.Client.Privmsgf("NickServ", "IDENTIFY %s", notifier.NickPassword)
time.Sleep(notifier.NickservDelayWait)
}
func (notifier *IRCNotifier) MaybeSendAlertNotice(alertNotice *AlertNotice) {
if !notifier.sessionUp {
log.Printf("Cannot send alert to %s : IRC not connected",
alertNotice.Channel)
return
}
notifier.JoinChannel(&IRCChannel{Name: alertNotice.Channel})
notifier.Client.Notice(alertNotice.Channel, alertNotice.Alert)
}
func (notifier *IRCNotifier) Run() {
keepGoing := true
for keepGoing {
if !notifier.Client.Connected() {
log.Printf("Connecting to IRC")
notifier.BackoffCounter.Delay()
if err := notifier.Client.Connect(); err != nil {
log.Printf("Could not connect to IRC: %s", err)
select {
case <-notifier.StopRunning:
log.Printf("IRC routine not connected but asked to terminate")
keepGoing = false
default:
}
continue
}
log.Printf("Connected to IRC server, waiting to establish session")
}
select {
case alertNotice := <-notifier.AlertNotices:
notifier.MaybeSendAlertNotice(&alertNotice)
case <-notifier.sessionUpSignal:
notifier.sessionUp = true
notifier.MaybeIdentifyNick()
notifier.JoinChannels()
case <-notifier.sessionDownSignal:
notifier.sessionUp = false
notifier.CleanupChannels()
notifier.Client.Quit("see ya")
case <-notifier.StopRunning:
log.Printf("IRC routine asked to terminate")
keepGoing = false
}
}
if notifier.Client.Connected() {
log.Printf("IRC client connected, quitting")
notifier.Client.Quit("see ya")
if notifier.sessionUp {
log.Printf("Session is up, wait for IRC disconnect to complete")
select {
case <-notifier.sessionDownSignal:
case <-time.After(notifier.Client.Config().Timeout):
log.Printf("Timeout while waiting for IRC disconnect to complete, stopping anyway")
}
}
}
notifier.StoppedRunning <- true
}

720
irc_test.go Normal file
View File

@ -0,0 +1,720 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"bufio"
"fmt"
irc "github.com/fluffle/goirc/client"
"io"
"log"
"net"
"reflect"
"strings"
"sync"
"testing"
"time"
)
type LineHandlerFunc func(*bufio.ReadWriter, *irc.Line) error
func h_USER(conn *bufio.ReadWriter, line *irc.Line) error {
r := fmt.Sprintf(":example.com 001 %s :Welcome\n", line.Args[0])
conn.WriteString(r)
return nil
}
func h_QUIT(conn *bufio.ReadWriter, line *irc.Line) error {
return fmt.Errorf("client asked to terminate")
}
type closeEarlyHandler func()
type testServer struct {
net.Listener
Client net.Conn
ServingWaitGroup sync.WaitGroup
ConnectionsWaitGroup sync.WaitGroup
lineHandlersMu sync.Mutex
lineHandlers map[string]LineHandlerFunc
Log []string
closeEarlyMu sync.Mutex
closeEarlyHandler
}
func (s *testServer) setDefaultHandlers() {
if s.lineHandlers == nil {
s.lineHandlers = make(map[string]LineHandlerFunc)
}
s.lineHandlers["USER"] = h_USER
s.lineHandlers["QUIT"] = h_QUIT
}
func (s *testServer) getHandler(cmd string) LineHandlerFunc {
s.lineHandlersMu.Lock()
defer s.lineHandlersMu.Unlock()
return s.lineHandlers[cmd]
}
func (s *testServer) SetHandler(cmd string, h LineHandlerFunc) {
s.lineHandlersMu.Lock()
defer s.lineHandlersMu.Unlock()
if h == nil {
delete(s.lineHandlers, cmd)
} else {
s.lineHandlers[cmd] = h
}
}
func (s *testServer) handleLine(conn *bufio.ReadWriter, line *irc.Line) error {
s.Log = append(s.Log, strings.Trim(line.Raw, " \r\n"))
handler := s.getHandler(line.Cmd)
if handler == nil {
log.Printf("=Server= No handler for command '%s', skipping", line.Cmd)
return nil
}
return handler(conn, line)
}
func (s *testServer) handleConnection(conn net.Conn) {
defer func() {
s.Client = nil
conn.Close()
s.ConnectionsWaitGroup.Done()
}()
bufConn := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))
for {
msg, err := bufConn.ReadBytes('\n')
if err != nil {
if err == io.EOF {
log.Printf("=Server= Client %s disconnected", conn.RemoteAddr().String())
} else {
log.Printf("=Server= Could not read from %s: %s", conn.RemoteAddr().String(), err)
}
return
}
log.Printf("=Server= Received %s", msg)
line := irc.ParseLine(string(msg))
if line == nil {
log.Printf("=Server= Could not parse received line")
continue
}
err = s.handleLine(bufConn, line)
if err != nil {
log.Printf("=Server= Closing connection: %s", err)
return
}
bufConn.Flush()
}
}
func (s *testServer) SetCloseEarly(h closeEarlyHandler) {
s.closeEarlyMu.Lock()
defer s.closeEarlyMu.Unlock()
s.closeEarlyHandler = h
}
func (s *testServer) handleCloseEarly(conn net.Conn) bool {
s.closeEarlyMu.Lock()
defer s.closeEarlyMu.Unlock()
if s.closeEarlyHandler == nil {
return false
}
log.Printf("=Server= Closing connection early")
conn.Close()
s.closeEarlyHandler()
return true
}
func (s *testServer) Serve() {
defer s.ServingWaitGroup.Done()
for {
conn, err := s.Listener.Accept()
if err != nil {
log.Printf("=Server= Stopped accepting new connections")
return
}
log.Printf("=Server= New client connected from %s", conn.RemoteAddr().String())
if s.handleCloseEarly(conn) {
continue
}
s.Client = conn
s.ConnectionsWaitGroup.Add(1)
s.handleConnection(conn)
}
}
func (s *testServer) Stop() {
s.Listener.Close()
s.ServingWaitGroup.Wait()
s.ConnectionsWaitGroup.Wait()
}
func makeTestServer(t *testing.T) (*testServer, int) {
server := new(testServer)
server.Log = make([]string, 0)
server.setDefaultHandlers()
addr, err := net.ResolveTCPAddr("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("=Server= Could not resolve tcp addr: %s", err)
}
listener, err := net.ListenTCP("tcp", addr)
if err != nil {
t.Fatalf("=Server= Could not create listener: %s", err)
}
addr = listener.Addr().(*net.TCPAddr)
log.Printf("=Server= Test server listening on %s", addr.String())
server.Listener = listener
server.ServingWaitGroup.Add(1)
go func() {
server.Serve()
}()
addr = listener.Addr().(*net.TCPAddr)
return server, addr.Port
}
type FakeDelayer struct {
}
func (f *FakeDelayer) Delay() {
log.Printf("Faking Backoff")
}
func makeTestIRCConfig(IRCPort int) *Config {
return &Config{
IRCNick: "foo",
IRCNickPass: "",
IRCHost: "127.0.0.1",
IRCPort: IRCPort,
IRCUseSSL: false,
IRCChannels: []IRCChannel{
IRCChannel{Name: "#foo"},
IRCChannel{Name: "#bar"},
IRCChannel{Name: "#baz"},
},
}
}
func makeTestNotifier(t *testing.T, config *Config) (*IRCNotifier, chan AlertNotice) {
alertNotices := make(chan AlertNotice)
notifier, err := NewIRCNotifier(config, alertNotices)
if err != nil {
t.Fatal(fmt.Sprintf("Could not create IRC notifier: %s", err))
}
notifier.Client.Config().Flood = true
notifier.BackoffCounter = &FakeDelayer{}
return notifier, alertNotices
}
func TestPreJoinChannels(t *testing.T) {
server, port := makeTestServer(t)
config := makeTestIRCConfig(port)
notifier, _ := makeTestNotifier(t, config)
var testStep sync.WaitGroup
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
// #baz is configured as the last channel to pre-join
if line.Args[0] == "#baz" {
testStep.Done()
}
return nil
}
server.SetHandler("JOIN", joinHandler)
testStep.Add(1)
go notifier.Run()
testStep.Wait()
notifier.StopRunning <- true
server.Stop()
expectedCommands := []string{
"NICK foo",
"USER foo 12 * :",
"JOIN #foo",
"JOIN #bar",
"JOIN #baz",
"QUIT :see ya",
}
if !reflect.DeepEqual(expectedCommands, server.Log) {
t.Error("Did not pre-join channels")
}
}
func TestSendAlertOnPreJoinedChannel(t *testing.T) {
server, port := makeTestServer(t)
config := makeTestIRCConfig(port)
notifier, alertNotices := makeTestNotifier(t, config)
var testStep sync.WaitGroup
testChannel := "#foo"
testMessage := "test message"
// Send the alert after configured channels have joined, to ensure we
// check for no re-join attempt.
joinedHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
if line.Args[0] == testChannel {
testStep.Done()
}
return nil
}
server.SetHandler("JOIN", joinedHandler)
testStep.Add(1)
go notifier.Run()
testStep.Wait()
server.SetHandler("JOIN", nil)
noticeHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
testStep.Done()
return nil
}
server.SetHandler("NOTICE", noticeHandler)
testStep.Add(1)
alertNotices <- AlertNotice{Channel: testChannel, Alert: testMessage}
testStep.Wait()
notifier.StopRunning <- true
server.Stop()
expectedCommands := []string{
"NICK foo",
"USER foo 12 * :",
"JOIN #foo",
"JOIN #bar",
"JOIN #baz",
"NOTICE #foo :test message",
"QUIT :see ya",
}
if !reflect.DeepEqual(expectedCommands, server.Log) {
t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n"))
}
}
func TestSendAlertAndJoinChannel(t *testing.T) {
server, port := makeTestServer(t)
config := makeTestIRCConfig(port)
notifier, alertNotices := makeTestNotifier(t, config)
var testStep sync.WaitGroup
testChannel := "#foobar"
testMessage := "test message"
// Send the alert after configured channels have joined, to ensure log
// ordering.
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
// #baz is configured as the last channel to pre-join
if line.Args[0] == "#baz" {
testStep.Done()
}
return nil
}
server.SetHandler("JOIN", joinHandler)
testStep.Add(1)
go notifier.Run()
testStep.Wait()
server.SetHandler("JOIN", nil)
noticeHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
testStep.Done()
return nil
}
server.SetHandler("NOTICE", noticeHandler)
testStep.Add(1)
alertNotices <- AlertNotice{Channel: testChannel, Alert: testMessage}
testStep.Wait()
notifier.StopRunning <- true
server.Stop()
expectedCommands := []string{
"NICK foo",
"USER foo 12 * :",
"JOIN #foo",
"JOIN #bar",
"JOIN #baz",
// #foobar joined before sending message
"JOIN #foobar",
"NOTICE #foobar :test message",
"QUIT :see ya",
}
if !reflect.DeepEqual(expectedCommands, server.Log) {
t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n"))
}
}
func TestSendAlertDisconnected(t *testing.T) {
server, port := makeTestServer(t)
config := makeTestIRCConfig(port)
notifier, alertNotices := makeTestNotifier(t, config)
var testStep, holdUserStep sync.WaitGroup
testChannel := "#foo"
disconnectedTestMessage := "disconnected test message"
connectedTestMessage := "connected test message"
// First send an alert while the session is not established.
testStep.Add(1)
holdUserStep.Add(1)
holdUser := func(conn *bufio.ReadWriter, line *irc.Line) error {
log.Printf("=Server= Wait before completing session")
testStep.Wait()
log.Printf("=Server= Completing session")
holdUserStep.Done()
return h_USER(conn, line)
}
server.SetHandler("USER", holdUser)
go notifier.Run()
alertNotices <- AlertNotice{Channel: testChannel, Alert: disconnectedTestMessage}
testStep.Done()
holdUserStep.Wait()
// Make sure session is established by checking that pre-joined
// channels are there.
testStep.Add(1)
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
// #baz is configured as the last channel to pre-join
if line.Args[0] == "#baz" {
testStep.Done()
}
return nil
}
server.SetHandler("JOIN", joinHandler)
testStep.Wait()
// Now send and wait until a notice has been received.
testStep.Add(1)
noticeHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
testStep.Done()
return nil
}
server.SetHandler("NOTICE", noticeHandler)
alertNotices <- AlertNotice{Channel: testChannel, Alert: connectedTestMessage}
testStep.Wait()
notifier.StopRunning <- true
server.Stop()
expectedCommands := []string{
"NICK foo",
"USER foo 12 * :",
"JOIN #foo",
"JOIN #bar",
"JOIN #baz",
// Only message sent while being connected is received.
"NOTICE #foo :connected test message",
"QUIT :see ya",
}
if !reflect.DeepEqual(expectedCommands, server.Log) {
t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n"))
}
}
func TestReconnect(t *testing.T) {
server, port := makeTestServer(t)
config := makeTestIRCConfig(port)
notifier, _ := makeTestNotifier(t, config)
var testStep sync.WaitGroup
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
// #baz is configured as the last channel to pre-join
if line.Args[0] == "#baz" {
testStep.Done()
}
return nil
}
server.SetHandler("JOIN", joinHandler)
testStep.Add(1)
go notifier.Run()
// Wait until the last pre-joined channel is seen.
testStep.Wait()
// Simulate disconnection.
testStep.Add(1)
server.Client.Close()
// Wait again until the last pre-joined channel is seen.
testStep.Wait()
notifier.StopRunning <- true
server.Stop()
expectedCommands := []string{
// Commands from first connection
"NICK foo",
"USER foo 12 * :",
"JOIN #foo",
"JOIN #bar",
"JOIN #baz",
// Commands from reconnection
"NICK foo",
"USER foo 12 * :",
"JOIN #foo",
"JOIN #bar",
"JOIN #baz",
"QUIT :see ya",
}
if !reflect.DeepEqual(expectedCommands, server.Log) {
t.Error("Reconnection did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n"))
}
}
func TestConnectErrorRetry(t *testing.T) {
server, port := makeTestServer(t)
config := makeTestIRCConfig(port)
// Attempt SSL handshake. The server does not support it, resulting in
// a connection error.
config.IRCUseSSL = true
notifier, _ := makeTestNotifier(t, config)
var testStep, joinStep sync.WaitGroup
testStep.Add(1)
earlyHandler := func() {
testStep.Done()
}
server.SetCloseEarly(earlyHandler)
go notifier.Run()
testStep.Wait()
// We have caused a connection failure, now check for a reconnection
notifier.Client.Config().SSL = false
joinStep.Add(1)
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
// #baz is configured as the last channel to pre-join
if line.Args[0] == "#baz" {
joinStep.Done()
}
return nil
}
server.SetHandler("JOIN", joinHandler)
server.SetCloseEarly(nil)
joinStep.Wait()
notifier.StopRunning <- true
server.Stop()
expectedCommands := []string{
"NICK foo",
"USER foo 12 * :",
"JOIN #foo",
"JOIN #bar",
"JOIN #baz",
"QUIT :see ya",
}
if !reflect.DeepEqual(expectedCommands, server.Log) {
t.Error("Reconnection did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n"))
}
}
func TestIdentify(t *testing.T) {
server, port := makeTestServer(t)
config := makeTestIRCConfig(port)
config.IRCNickPass = "nickpassword"
notifier, _ := makeTestNotifier(t, config)
notifier.NickservDelayWait = 0 * time.Second
var testStep sync.WaitGroup
// Wait until the last pre-joined channel is seen (joining happens
// after identification).
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
// #baz is configured as the last channel to pre-join
if line.Args[0] == "#baz" {
testStep.Done()
}
return nil
}
server.SetHandler("JOIN", joinHandler)
testStep.Add(1)
go notifier.Run()
testStep.Wait()
notifier.StopRunning <- true
server.Stop()
expectedCommands := []string{
"NICK foo",
"USER foo 12 * :",
"PRIVMSG NickServ :IDENTIFY nickpassword",
"JOIN #foo",
"JOIN #bar",
"JOIN #baz",
"QUIT :see ya",
}
if !reflect.DeepEqual(expectedCommands, server.Log) {
t.Error("Identification did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n"))
}
}
func TestGhostAndIdentify(t *testing.T) {
server, port := makeTestServer(t)
config := makeTestIRCConfig(port)
config.IRCNickPass = "nickpassword"
notifier, _ := makeTestNotifier(t, config)
notifier.NickservDelayWait = 0 * time.Second
var testStep, usedNick, unregisteredNickHandler sync.WaitGroup
// Trigger 433 for first nick
usedNick.Add(1)
unregisteredNickHandler.Add(1)
nickHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
if line.Args[0] == "foo" {
conn.WriteString(":example.com 433 * foo :nick in use\n")
}
usedNick.Done()
unregisteredNickHandler.Wait()
return nil
}
server.SetHandler("NICK", nickHandler)
// Wait until the last pre-joined channel is seen (joining happens
// after identification).
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
// #baz is configured as the last channel to pre-join
if line.Args[0] == "#baz" {
testStep.Done()
}
return nil
}
server.SetHandler("JOIN", joinHandler)
testStep.Add(1)
go notifier.Run()
usedNick.Wait()
server.SetHandler("NICK", nil)
unregisteredNickHandler.Done()
testStep.Wait()
notifier.StopRunning <- true
server.Stop()
expectedCommands := []string{
"NICK foo",
"USER foo 12 * :",
"NICK foo^",
"PRIVMSG NickServ :GHOST foo nickpassword",
"NICK foo",
"PRIVMSG NickServ :IDENTIFY nickpassword",
"JOIN #foo",
"JOIN #bar",
"JOIN #baz",
"QUIT :see ya",
}
if !reflect.DeepEqual(expectedCommands, server.Log) {
t.Error("Ghosting did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n"))
}
}
func TestStopRunningWhenHalfConnected(t *testing.T) {
server, port := makeTestServer(t)
config := makeTestIRCConfig(port)
notifier, _ := makeTestNotifier(t, config)
var testStep, holdQuitWait sync.WaitGroup
// Send a StopRunning request while the client is connected but the
// session is not up
testStep.Add(1)
holdUser := func(conn *bufio.ReadWriter, line *irc.Line) error {
log.Printf("=Server= NOT completing session")
testStep.Done()
return nil
}
server.SetHandler("USER", holdUser)
// Ignore quit, but wait for it to have deterministic test commands
holdQuitWait.Add(1)
holdQuit := func(conn *bufio.ReadWriter, line *irc.Line) error {
log.Printf("=Server= Ignoring quit")
holdQuitWait.Done()
return nil
}
server.SetHandler("QUIT", holdQuit)
go notifier.Run()
testStep.Wait()
notifier.StopRunning <- true
<-notifier.StoppedRunning
holdQuitWait.Wait()
// Client has left, cleanup the server side before stopping
server.Client.Close()
server.Stop()
expectedCommands := []string{
"NICK foo",
"USER foo 12 * :",
"QUIT :see ya",
}
if !reflect.DeepEqual(expectedCommands, server.Log) {
t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n"))
}
}

67
main.go Normal file
View File

@ -0,0 +1,67 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"flag"
"log"
"os"
"os/signal"
"syscall"
)
func main() {
configFile := flag.String("config", "", "Config file path.")
flag.Parse()
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM)
config, err := LoadConfig(*configFile)
if err != nil {
log.Printf("Could not load config: %s", err)
return
}
alertNotices := make(chan AlertNotice, 10)
ircNotifier, err := NewIRCNotifier(config, alertNotices)
if err != nil {
log.Printf("Could not create IRC notifier: %s", err)
return
}
go ircNotifier.Run()
httpServer, err := NewHTTPServer(config, alertNotices)
if err != nil {
log.Printf("Could not create HTTP server: %s", err)
return
}
go httpServer.Run()
select {
case <-httpServer.StoppedRunning:
log.Printf("Http server terminated, exiting")
case <-ircNotifier.StoppedRunning:
log.Printf("IRC notifier stopped running, exiting")
case s := <-signals:
log.Printf("Received %s, exiting", s)
ircNotifier.StopRunning <- true
log.Printf("Waiting for IRC to quit")
<-ircNotifier.StoppedRunning
}
}

77
testdata.go Normal file
View File

@ -0,0 +1,77 @@
// Copyright 2018 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
const (
testdataSimpleAlertJson = `
{
"status": "resolved",
"receiver": "example_receiver",
"groupLabels": {
"alertname": "airDown",
"service": "prometheus"
},
"commonLabels": {
"alertname": "airDown",
"job": "air",
"service": "prometheus",
"severity": "ticket",
"zone": "global"
},
"commonAnnotations": {},
"externalURL": "https://prometheus.example.com/alertmanager",
"alerts": [
{
"annotations": {
"SUMMARY": "service /prometheus air down on instance1",
"DESCRIPTION": "service /prometheus has irc gateway down on instance1"
},
"endsAt": "2017-05-15T13:50:37.835Z",
"generatorURL": "https://prometheus.example.com/prometheus/...",
"labels": {
"alertname": "airDown",
"instance": "instance1:3456",
"job": "air",
"service": "prometheus",
"severity": "ticket",
"zone": "global"
},
"startsAt": "2017-05-15T13:49:37.834Z",
"status": "resolved"
},
{
"annotations": {
"SUMMARY": "service /prometheus air down on instance2",
"DESCRIPTION": "service /prometheus has irc gateway down on instance2"
},
"endsAt": "2017-05-15T11:48:37.834Z",
"generatorURL": "https://prometheus.example.com/prometheus/...",
"labels": {
"alertname": "airDown",
"instance": "instance2:7890",
"job": "air",
"service": "prometheus",
"severity": "ticket",
"zone": "global"
},
"startsAt": "2017-05-15T11:47:37.834Z",
"status": "resolved"
}
]
}
`
testdataBogusAlertJson = `{"this is not": "a valid alert",}`
)