Initial code check-in
Signed-off-by: Luca Bigliardi <shammash@google.com>
This commit is contained in:
Luca Bigliardi
parent
522822e703
commit
60632b16e6
|
|
@ -0,0 +1,28 @@
|
|||
# How to Contribute
|
||||
|
||||
We'd love to accept your patches and contributions to this project. There are
|
||||
just a few small guidelines you need to follow.
|
||||
|
||||
## Contributor License Agreement
|
||||
|
||||
Contributions to this project must be accompanied by a Contributor License
|
||||
Agreement. You (or your employer) retain the copyright to your contribution,
|
||||
this simply gives us permission to use and redistribute your contributions as
|
||||
part of the project. Head over to <https://cla.developers.google.com/> to see
|
||||
your current agreements on file or to sign a new one.
|
||||
|
||||
You generally only need to submit a CLA once, so if you've already submitted one
|
||||
(even if it was for a different project), you probably don't need to do it
|
||||
again.
|
||||
|
||||
## Code reviews
|
||||
|
||||
All submissions, including submissions by project members, require review. We
|
||||
use GitHub pull requests for this purpose. Consult
|
||||
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
|
||||
information on using pull requests.
|
||||
|
||||
## Community Guidelines
|
||||
|
||||
This project follows [Google's Open Source Community
|
||||
Guidelines](https://opensource.google.com/conduct/).
|
||||
|
|
@ -0,0 +1,202 @@
|
|||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
# Alertmanager IRC Relay
|
||||
|
||||
Alertmanager IRC Relay is a bot relaying [Prometheus](https://prometheus.io/) alerts to IRC.
|
||||
Alerts are received from Prometheus using
|
||||
[Webhooks](https://prometheus.io/docs/alerting/configuration/#webhook-receiver-<webhook_config>)
|
||||
and are relayed to an IRC channel.
|
||||
|
||||
### Configuring and running the bot
|
||||
|
||||
To configure and run the bot you need to create a YAML configuration file and
|
||||
pass it to the service. Running the service without a configuration will use
|
||||
the default test values and connect to a default IRC channel, which you
|
||||
probably do not want to do.
|
||||
|
||||
Example configuration:
|
||||
```
|
||||
# Start the HTTP server receiving alerts from Prometheus Webhook binding to
|
||||
# this host/port.
|
||||
#
|
||||
http_host: localhost
|
||||
http_port: 8000
|
||||
|
||||
# Connect to this IRC host/port.
|
||||
#
|
||||
# Note: SSL is enabled by default, use "irc_use_ssl: no" to disable.
|
||||
irc_host: irc.example.com
|
||||
irc_port: 7000
|
||||
|
||||
# Use this IRC nickname.
|
||||
irc_nickname: myalertbot
|
||||
# Password used to identify with NickServ
|
||||
irc_nickname_password: mynickserv_key
|
||||
# Use this IRC real name
|
||||
irc_realname: myrealname
|
||||
|
||||
# Optionally pre-join certain channels.
|
||||
#
|
||||
# Note: If an alert is sent to a non # pre-joined channel the bot will join
|
||||
# that channel anyway before sending the notice. Of course this cannot work
|
||||
# with password-protected channels.
|
||||
irc_channels:
|
||||
- name: "#mychannel"
|
||||
- name: "#myprivatechannel"
|
||||
password: myprivatechannel_key
|
||||
|
||||
# Define how IRC messages should be sent.
|
||||
#
|
||||
# Send only one notice when webhook data is received.
|
||||
# Note: By default a notice is sent for each alert in the webhook data.
|
||||
notice_once_per_alert_group: no
|
||||
|
||||
# Define how IRC messages should be formatted.
|
||||
#
|
||||
# The formatting is based on golang's text/template .
|
||||
notice_template: "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}"
|
||||
# Note: When sending only one notice per alert group the default
|
||||
# notice_template is set to
|
||||
# "Alert {{ .GroupLabels.alertname }} for {{ .GroupLabels.job }} is {{ .Status }}"
|
||||
```
|
||||
|
||||
Running the bot (assuming *$GOPATH* and *$PATH* are properly setup for go):
|
||||
```
|
||||
$ go install github.com/google/alertmanager-irc-relay
|
||||
$ alertmanager-irc-relay --config /path/to/your/config/file
|
||||
```
|
||||
|
||||
### Prometheus configuration
|
||||
|
||||
Prometheus can be configured following the official
|
||||
[Webhooks](https://prometheus.io/docs/alerting/configuration/#webhook-receiver-<webhook_config>)
|
||||
documentation. The `url` must specify the IRC channel name that alerts should
|
||||
be sent to:
|
||||
```
|
||||
send_resolved: false
|
||||
url: http://localhost:8000/mychannel
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"log"
|
||||
"math"
|
||||
"math/rand"
|
||||
"time"
|
||||
)
|
||||
|
||||
type JitterFunc func(int) int
|
||||
|
||||
type TimeFunc func() time.Time
|
||||
|
||||
type Delayer interface {
|
||||
Delay()
|
||||
}
|
||||
|
||||
type Backoff struct {
|
||||
step float64
|
||||
maxBackoff float64
|
||||
resetDelta float64
|
||||
lastAttempt time.Time
|
||||
durationUnit time.Duration
|
||||
jitterer JitterFunc
|
||||
timeGetter TimeFunc
|
||||
}
|
||||
|
||||
func jitterFunc(input int) int {
|
||||
if input == 0 {
|
||||
return 0
|
||||
}
|
||||
return rand.Intn(input)
|
||||
}
|
||||
|
||||
func NewBackoff(maxBackoff float64, resetDelta float64,
|
||||
durationUnit time.Duration) *Backoff {
|
||||
return NewBackoffForTesting(
|
||||
maxBackoff, resetDelta, durationUnit, jitterFunc, time.Now)
|
||||
}
|
||||
|
||||
func NewBackoffForTesting(maxBackoff float64, resetDelta float64,
|
||||
durationUnit time.Duration, jitterer JitterFunc, timeGetter TimeFunc) *Backoff {
|
||||
return &Backoff{
|
||||
step: 0,
|
||||
maxBackoff: maxBackoff,
|
||||
resetDelta: resetDelta,
|
||||
lastAttempt: timeGetter(),
|
||||
durationUnit: durationUnit,
|
||||
jitterer: jitterer,
|
||||
timeGetter: timeGetter,
|
||||
}
|
||||
}
|
||||
|
||||
func (b *Backoff) maybeReset() {
|
||||
now := b.timeGetter()
|
||||
lastAttemptDelta := float64(now.Sub(b.lastAttempt) / b.durationUnit)
|
||||
b.lastAttempt = now
|
||||
|
||||
if lastAttemptDelta >= b.resetDelta {
|
||||
b.step = 0
|
||||
}
|
||||
}
|
||||
|
||||
func (b *Backoff) GetDelay() time.Duration {
|
||||
b.maybeReset()
|
||||
|
||||
var synchronizedDuration float64
|
||||
// Do not add any delay the first time.
|
||||
if b.step == 0 {
|
||||
synchronizedDuration = 0
|
||||
} else {
|
||||
synchronizedDuration = math.Pow(2, b.step)
|
||||
}
|
||||
|
||||
if synchronizedDuration < b.maxBackoff {
|
||||
b.step++
|
||||
} else {
|
||||
synchronizedDuration = b.maxBackoff
|
||||
}
|
||||
duration := time.Duration(b.jitterer(int(synchronizedDuration)))
|
||||
return duration * b.durationUnit
|
||||
}
|
||||
|
||||
func (b *Backoff) Delay() {
|
||||
delay := b.GetDelay()
|
||||
log.Printf("Backoff for %s", delay)
|
||||
time.Sleep(delay)
|
||||
}
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type FakeTime struct {
|
||||
timeseries []int
|
||||
lastIndex int
|
||||
durationUnit time.Duration
|
||||
}
|
||||
|
||||
func (f *FakeTime) GetTime() time.Time {
|
||||
timeDelta := time.Duration(f.timeseries[f.lastIndex]) * f.durationUnit
|
||||
fakeTime := time.Unix(0, 0).Add(timeDelta)
|
||||
f.lastIndex++
|
||||
return fakeTime
|
||||
}
|
||||
|
||||
func FakeJitter(input int) int {
|
||||
return input
|
||||
}
|
||||
|
||||
func RunBackoffTest(t *testing.T,
|
||||
maxBackoff float64, resetDelta float64,
|
||||
elapsedTime []int, expectedDelays []int) {
|
||||
fakeTime := &FakeTime{
|
||||
timeseries: elapsedTime,
|
||||
lastIndex: 0,
|
||||
durationUnit: time.Millisecond,
|
||||
}
|
||||
backoff := NewBackoffForTesting(maxBackoff, resetDelta, time.Millisecond,
|
||||
FakeJitter, fakeTime.GetTime)
|
||||
|
||||
for i, value := range expectedDelays {
|
||||
expected_delay := time.Duration(value) * time.Millisecond
|
||||
delay := backoff.GetDelay()
|
||||
if expected_delay != delay {
|
||||
t.Errorf("Call #%d of GetDelay returned %s (expected %s)",
|
||||
i, delay, expected_delay)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackoffIncreasesAndReachesMax(t *testing.T) {
|
||||
RunBackoffTest(t,
|
||||
8,
|
||||
32,
|
||||
// Simple sequential time
|
||||
[]int{0, 0, 1, 2, 3, 4, 5, 6, 7},
|
||||
// Exponential ramp-up to max, then keep max.
|
||||
[]int{0, 2, 4, 8, 8, 8, 8, 8},
|
||||
)
|
||||
}
|
||||
|
||||
func TestBackoffReset(t *testing.T) {
|
||||
RunBackoffTest(t,
|
||||
8,
|
||||
32,
|
||||
// Simulate two intervals bigger than resetDelta
|
||||
[]int{0, 0, 1, 2, 50, 51, 100, 101, 102},
|
||||
// Delays get reset each time
|
||||
[]int{0, 2, 4, 0, 2, 0, 2, 4},
|
||||
)
|
||||
}
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"gopkg.in/yaml.v2"
|
||||
"io/ioutil"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultNoticeOnceTemplate = "Alert {{ .GroupLabels.alertname }} for {{ .GroupLabels.job }} is {{ .Status }}"
|
||||
defaultNoticeTemplate = "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}"
|
||||
)
|
||||
|
||||
type IRCChannel struct {
|
||||
Name string `yaml:"name"`
|
||||
Password string `yaml:"password"`
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
HTTPHost string `yaml:"http_host"`
|
||||
HTTPPort int `yaml:"http_port"`
|
||||
IRCNick string `yaml:"irc_nickname"`
|
||||
IRCNickPass string `yaml:"irc_nickname_password"`
|
||||
IRCRealName string `yaml:"irc_realname"`
|
||||
IRCHost string `yaml:"irc_host"`
|
||||
IRCPort int `yaml:"irc_port"`
|
||||
IRCUseSSL bool `yaml:"irc_use_ssl"`
|
||||
IRCChannels []IRCChannel `yaml:"irc_channels"`
|
||||
NoticeTemplate string `yaml:"notice_template"`
|
||||
NoticeOnce bool `yaml:"notice_once_per_alert_group"`
|
||||
}
|
||||
|
||||
func LoadConfig(configFile string) (*Config, error) {
|
||||
config := &Config{
|
||||
HTTPHost: "localhost",
|
||||
HTTPPort: 8000,
|
||||
IRCNick: "alertmanager-irc-relay",
|
||||
IRCNickPass: "",
|
||||
IRCRealName: "Alertmanager IRC Relay",
|
||||
IRCHost: "irc.freenode.net",
|
||||
IRCPort: 7000,
|
||||
IRCUseSSL: true,
|
||||
IRCChannels: []IRCChannel{IRCChannel{Name: "#airtest"}},
|
||||
NoticeOnce: false,
|
||||
}
|
||||
|
||||
if configFile != "" {
|
||||
data, err := ioutil.ReadFile(configFile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := yaml.Unmarshal(data, config); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// Set default template if config does not have one.
|
||||
if config.NoticeTemplate == "" {
|
||||
if config.NoticeOnce {
|
||||
config.NoticeTemplate = defaultNoticeOnceTemplate
|
||||
} else {
|
||||
config.NoticeTemplate = defaultNoticeTemplate
|
||||
}
|
||||
}
|
||||
|
||||
return config, nil
|
||||
}
|
||||
|
|
@ -0,0 +1,180 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"gopkg.in/yaml.v2"
|
||||
"io/ioutil"
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestNoConfig(t *testing.T) {
|
||||
noConfigFile := ""
|
||||
|
||||
config, err := LoadConfig(noConfigFile)
|
||||
if config == nil {
|
||||
t.Errorf("Expected a default config, got: %s", err)
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
func TestLoadGoodConfig(t *testing.T) {
|
||||
expectedConfig := &Config{
|
||||
HTTPHost: "test.web",
|
||||
HTTPPort: 8888,
|
||||
IRCNick: "foo",
|
||||
IRCHost: "irc.example.com",
|
||||
IRCPort: 1234,
|
||||
IRCUseSSL: true,
|
||||
IRCChannels: []IRCChannel{IRCChannel{Name: "#foobar"}},
|
||||
NoticeTemplate: defaultNoticeTemplate,
|
||||
NoticeOnce: false,
|
||||
}
|
||||
expectedData, err := yaml.Marshal(expectedConfig)
|
||||
if err != nil {
|
||||
t.Errorf("Could not serialize test data: %s", err)
|
||||
}
|
||||
|
||||
tmpfile, err := ioutil.TempFile("", "airtestconfig")
|
||||
if err != nil {
|
||||
t.Errorf("Could not create tmpfile for testing: %s", err)
|
||||
}
|
||||
defer os.Remove(tmpfile.Name())
|
||||
|
||||
if _, err := tmpfile.Write(expectedData); err != nil {
|
||||
t.Errorf("Could not write test data in tmpfile: %s", err)
|
||||
}
|
||||
if err := tmpfile.Close(); err != nil {
|
||||
t.Errorf("Could not close tmpfile: %s", err)
|
||||
}
|
||||
|
||||
config, err := LoadConfig(tmpfile.Name())
|
||||
if config == nil {
|
||||
t.Errorf("Expected a config, got: %s", err)
|
||||
}
|
||||
|
||||
configData, err := yaml.Marshal(config)
|
||||
if err != nil {
|
||||
t.Errorf("Could not serialize loaded config")
|
||||
}
|
||||
|
||||
if string(expectedData) != string(configData) {
|
||||
t.Errorf("Loaded config does not match expected config: %s", configData)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadBadFile(t *testing.T) {
|
||||
tmpfile, err := ioutil.TempFile("", "airtestbadfile")
|
||||
if err != nil {
|
||||
t.Errorf("Could not create tmpfile for testing: %s", err)
|
||||
}
|
||||
tmpfile.Close()
|
||||
os.Remove(tmpfile.Name())
|
||||
|
||||
config, err := LoadConfig(tmpfile.Name())
|
||||
if config != nil {
|
||||
t.Errorf("Expected no config upon non-existent file.")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadBadConfig(t *testing.T) {
|
||||
tmpfile, err := ioutil.TempFile("", "airtestbadconfig")
|
||||
if err != nil {
|
||||
t.Errorf("Could not create tmpfile for testing: %s", err)
|
||||
}
|
||||
defer os.Remove(tmpfile.Name())
|
||||
|
||||
badConfigData := []byte("footest\nbarbaz\n")
|
||||
if _, err := tmpfile.Write(badConfigData); err != nil {
|
||||
t.Errorf("Could not write test data in tmpfile: %s", err)
|
||||
}
|
||||
tmpfile.Close()
|
||||
|
||||
config, err := LoadConfig(tmpfile.Name())
|
||||
if config != nil {
|
||||
t.Errorf("Expected no config upon bad config.")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoticeOnceDefaultTemplate(t *testing.T) {
|
||||
tmpfile, err := ioutil.TempFile("", "airtesttemmplateonceconfig")
|
||||
if err != nil {
|
||||
t.Errorf("Could not create tmpfile for testing: %s", err)
|
||||
}
|
||||
defer os.Remove(tmpfile.Name())
|
||||
|
||||
noticeOnceConfigData := []byte("notice_once_per_alert_group: yes")
|
||||
if _, err := tmpfile.Write(noticeOnceConfigData); err != nil {
|
||||
t.Errorf("Could not write test data in tmpfile: %s", err)
|
||||
}
|
||||
tmpfile.Close()
|
||||
|
||||
config, err := LoadConfig(tmpfile.Name())
|
||||
if config == nil {
|
||||
t.Errorf("Expected a config, got: %s", err)
|
||||
}
|
||||
|
||||
if config.NoticeTemplate != defaultNoticeOnceTemplate {
|
||||
t.Errorf("Expecting defaultNoticeOnceTemplate when NoticeOnce is true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoticeDefaultTemplate(t *testing.T) {
|
||||
tmpfile, err := ioutil.TempFile("", "airtesttemmplateconfig")
|
||||
if err != nil {
|
||||
t.Errorf("Could not create tmpfile for testing: %s", err)
|
||||
}
|
||||
defer os.Remove(tmpfile.Name())
|
||||
|
||||
if _, err := tmpfile.Write([]byte("")); err != nil {
|
||||
t.Errorf("Could not write test data in tmpfile: %s", err)
|
||||
}
|
||||
tmpfile.Close()
|
||||
|
||||
config, err := LoadConfig(tmpfile.Name())
|
||||
if config == nil {
|
||||
t.Errorf("Expected a config, got: %s", err)
|
||||
}
|
||||
|
||||
if config.NoticeTemplate != defaultNoticeTemplate {
|
||||
t.Errorf("Expecting defaultNoticeTemplate when NoticeOnce is false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGivenTemplateNotOverwritten(t *testing.T) {
|
||||
tmpfile, err := ioutil.TempFile("", "airtestexpectedtemmplate")
|
||||
if err != nil {
|
||||
t.Errorf("Could not create tmpfile for testing: %s", err)
|
||||
}
|
||||
defer os.Remove(tmpfile.Name())
|
||||
|
||||
expectedTemplate := "Alert {{ .Status }}: {{ .Annotations.SUMMARY }}"
|
||||
configData := []byte(fmt.Sprintf("notice_template: \"%s\"", expectedTemplate))
|
||||
if _, err := tmpfile.Write(configData); err != nil {
|
||||
t.Errorf("Could not write test data in tmpfile: %s", err)
|
||||
}
|
||||
tmpfile.Close()
|
||||
|
||||
config, err := LoadConfig(tmpfile.Name())
|
||||
if config == nil {
|
||||
t.Errorf("Expected a config, got: %s", err)
|
||||
}
|
||||
|
||||
if config.NoticeTemplate != expectedTemplate {
|
||||
t.Errorf("Template does not match configuration")
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
type AlertNotice struct {
|
||||
Channel, Alert string
|
||||
}
|
||||
|
|
@ -0,0 +1,148 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"log"
|
||||
"net/http"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
promtmpl "github.com/prometheus/alertmanager/template"
|
||||
"strconv"
|
||||
"strings"
|
||||
"text/template"
|
||||
)
|
||||
|
||||
type HTTPListener func(string, http.Handler) error
|
||||
|
||||
type HTTPServer struct {
|
||||
StoppedRunning chan bool
|
||||
Addr string
|
||||
Port int
|
||||
NoticeTemplate *template.Template
|
||||
NoticeOnce bool
|
||||
AlertNotices chan AlertNotice
|
||||
httpListener HTTPListener
|
||||
}
|
||||
|
||||
func NewHTTPServer(config *Config, alertNotices chan AlertNotice) (
|
||||
*HTTPServer, error) {
|
||||
return NewHTTPServerForTesting(config, alertNotices, http.ListenAndServe)
|
||||
}
|
||||
|
||||
func NewHTTPServerForTesting(config *Config, alertNotices chan AlertNotice,
|
||||
httpListener HTTPListener) (*HTTPServer, error) {
|
||||
tmpl, err := template.New("notice").Parse(config.NoticeTemplate)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
server := &HTTPServer{
|
||||
StoppedRunning: make(chan bool),
|
||||
Addr: config.HTTPHost,
|
||||
Port: config.HTTPPort,
|
||||
NoticeTemplate: tmpl,
|
||||
NoticeOnce: config.NoticeOnce,
|
||||
AlertNotices: alertNotices,
|
||||
httpListener: httpListener,
|
||||
}
|
||||
|
||||
return server, nil
|
||||
}
|
||||
|
||||
func (server *HTTPServer) FormatNotice(data interface{}) string {
|
||||
output := bytes.Buffer{}
|
||||
var msg string
|
||||
if err := server.NoticeTemplate.Execute(&output, data); err != nil {
|
||||
msg_bytes, _ := json.Marshal(data)
|
||||
msg = string(msg_bytes)
|
||||
log.Printf("Could not apply notice template on alert (%s): %s",
|
||||
err, msg)
|
||||
log.Printf("Sending raw alert")
|
||||
} else {
|
||||
msg = output.String()
|
||||
}
|
||||
return msg
|
||||
}
|
||||
|
||||
func (server *HTTPServer) GetNoticesFromAlertMessage(ircChannel string,
|
||||
data *promtmpl.Data) []AlertNotice {
|
||||
notices := []AlertNotice{}
|
||||
if server.NoticeOnce {
|
||||
msg := server.FormatNotice(data)
|
||||
notices = append(notices,
|
||||
AlertNotice{Channel: ircChannel, Alert: msg})
|
||||
} else {
|
||||
for _, alert := range data.Alerts {
|
||||
msg := server.FormatNotice(alert)
|
||||
notices = append(notices,
|
||||
AlertNotice{Channel: ircChannel, Alert: msg})
|
||||
}
|
||||
}
|
||||
return notices
|
||||
}
|
||||
|
||||
func (server *HTTPServer) RelayAlert(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
ircChannel := "#" + vars["IRCChannel"]
|
||||
|
||||
body, err := ioutil.ReadAll(io.LimitReader(r.Body, 1024*1024*1024))
|
||||
if err != nil {
|
||||
log.Printf("Could not get body: %s", err)
|
||||
return
|
||||
}
|
||||
|
||||
var alertMessage = promtmpl.Data{}
|
||||
if err := json.Unmarshal(body, &alertMessage); err != nil {
|
||||
log.Printf("Could not decode request body (%s): %s", err, body)
|
||||
|
||||
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
|
||||
w.WriteHeader(422) // Unprocessable entity
|
||||
if err := json.NewEncoder(w).Encode(err); err != nil {
|
||||
log.Printf("Could not write decoding error: %s", err)
|
||||
return
|
||||
}
|
||||
return
|
||||
}
|
||||
for _, alertNotice := range server.GetNoticesFromAlertMessage(
|
||||
ircChannel, &alertMessage) {
|
||||
select {
|
||||
case server.AlertNotices <- alertNotice:
|
||||
default:
|
||||
log.Printf("Could not send this alert to the IRC routine: %s",
|
||||
alertNotice)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (server *HTTPServer) Run() {
|
||||
router := mux.NewRouter().StrictSlash(true)
|
||||
|
||||
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
server.RelayAlert(w, r)
|
||||
})
|
||||
router.Path("/{IRCChannel}").Handler(handler).Methods("POST")
|
||||
|
||||
listenAddr := strings.Join(
|
||||
[]string{server.Addr, strconv.Itoa(server.Port)}, ":")
|
||||
log.Printf("Starting HTTP server")
|
||||
if err := server.httpListener(listenAddr, router); err != nil {
|
||||
log.Printf("Could not start http server: %s", err)
|
||||
}
|
||||
server.StoppedRunning <- true
|
||||
}
|
||||
|
|
@ -0,0 +1,218 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type FakeHTTPListener struct {
|
||||
StartedServing chan bool
|
||||
StopServing chan bool
|
||||
AlertNotices chan AlertNotice // kinda ugly putting it here, but convenient
|
||||
router http.Handler
|
||||
}
|
||||
|
||||
func (listener *FakeHTTPListener) Serve(_ string, router http.Handler) error {
|
||||
listener.router = router
|
||||
|
||||
listener.StartedServing <- true
|
||||
<-listener.StopServing
|
||||
return nil
|
||||
}
|
||||
|
||||
func NewFakeHTTPListener() *FakeHTTPListener {
|
||||
return &FakeHTTPListener{
|
||||
StartedServing: make(chan bool),
|
||||
StopServing: make(chan bool),
|
||||
AlertNotices: make(chan AlertNotice, 10),
|
||||
}
|
||||
}
|
||||
|
||||
func MakeHTTPTestingConfig() *Config {
|
||||
return &Config{
|
||||
HTTPHost: "test.web",
|
||||
HTTPPort: 8888,
|
||||
NoticeTemplate: "Alert {{ .Labels.alertname }} on {{ .Labels.instance }} is {{ .Status }}",
|
||||
}
|
||||
}
|
||||
|
||||
func RunHTTPTest(t *testing.T,
|
||||
alertData string, url string,
|
||||
testingConfig *Config, listener *FakeHTTPListener) *http.Response {
|
||||
httpServer, err := NewHTTPServerForTesting(testingConfig,
|
||||
listener.AlertNotices, listener.Serve)
|
||||
if err != nil {
|
||||
t.Fatal(fmt.Sprintf("Could not create HTTP server: %s", err))
|
||||
}
|
||||
|
||||
go httpServer.Run()
|
||||
|
||||
<-listener.StartedServing
|
||||
|
||||
alertDataReader := strings.NewReader(alertData)
|
||||
request, err := http.NewRequest("POST", url, alertDataReader)
|
||||
if err != nil {
|
||||
t.Fatal(fmt.Sprintf("Could not create HTTP request: %s", err))
|
||||
}
|
||||
responseRecorder := httptest.NewRecorder()
|
||||
|
||||
listener.router.ServeHTTP(responseRecorder, request)
|
||||
|
||||
listener.StopServing <- true
|
||||
<-httpServer.StoppedRunning
|
||||
return responseRecorder.Result()
|
||||
}
|
||||
|
||||
func TestAlertsDispatched(t *testing.T) {
|
||||
listener := NewFakeHTTPListener()
|
||||
testingConfig := MakeHTTPTestingConfig()
|
||||
|
||||
expectedAlertNotices := []AlertNotice{
|
||||
AlertNotice{
|
||||
Channel: "#somechannel",
|
||||
Alert: "Alert airDown on instance1:3456 is resolved",
|
||||
},
|
||||
AlertNotice{
|
||||
Channel: "#somechannel",
|
||||
Alert: "Alert airDown on instance2:7890 is resolved",
|
||||
},
|
||||
}
|
||||
expectedStatusCode := 200
|
||||
|
||||
response := RunHTTPTest(
|
||||
t, testdataSimpleAlertJson, "/somechannel",
|
||||
testingConfig, listener)
|
||||
|
||||
if expectedStatusCode != response.StatusCode {
|
||||
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
|
||||
expectedStatusCode, response.StatusCode))
|
||||
}
|
||||
|
||||
for _, expectedAlertNotice := range expectedAlertNotices {
|
||||
alertNotice := <-listener.AlertNotices
|
||||
if !reflect.DeepEqual(expectedAlertNotice, alertNotice) {
|
||||
t.Error(fmt.Sprintf(
|
||||
"Unexpected alert notice.\nExpected: %s\nActual: %s",
|
||||
expectedAlertNotice, alertNotice))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAlertsDispatchedOnce(t *testing.T) {
|
||||
listener := NewFakeHTTPListener()
|
||||
testingConfig := MakeHTTPTestingConfig()
|
||||
testingConfig.NoticeOnce = true
|
||||
testingConfig.NoticeTemplate = "Alert {{ .GroupLabels.alertname }} is {{ .Status }}"
|
||||
|
||||
expectedAlertNotices := []AlertNotice{
|
||||
AlertNotice{
|
||||
Channel: "#somechannel",
|
||||
Alert: "Alert airDown is resolved",
|
||||
},
|
||||
}
|
||||
expectedStatusCode := 200
|
||||
|
||||
response := RunHTTPTest(
|
||||
t, testdataSimpleAlertJson, "/somechannel",
|
||||
testingConfig, listener)
|
||||
|
||||
if expectedStatusCode != response.StatusCode {
|
||||
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
|
||||
expectedStatusCode, response.StatusCode))
|
||||
}
|
||||
|
||||
for _, expectedAlertNotice := range expectedAlertNotices {
|
||||
alertNotice := <-listener.AlertNotices
|
||||
if !reflect.DeepEqual(expectedAlertNotice, alertNotice) {
|
||||
t.Error(fmt.Sprintf(
|
||||
"Unexpected alert notice.\nExpected: %s\nActual: %s",
|
||||
expectedAlertNotice, alertNotice))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRootReturnsError(t *testing.T) {
|
||||
listener := NewFakeHTTPListener()
|
||||
testingConfig := MakeHTTPTestingConfig()
|
||||
|
||||
expectedStatusCode := 404
|
||||
|
||||
response := RunHTTPTest(
|
||||
t, testdataSimpleAlertJson, "/",
|
||||
testingConfig, listener)
|
||||
|
||||
if expectedStatusCode != response.StatusCode {
|
||||
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
|
||||
expectedStatusCode, response.StatusCode))
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidDataReturnsError(t *testing.T) {
|
||||
listener := NewFakeHTTPListener()
|
||||
testingConfig := MakeHTTPTestingConfig()
|
||||
|
||||
expectedStatusCode := 422
|
||||
|
||||
response := RunHTTPTest(
|
||||
t, testdataBogusAlertJson, "/somechannel",
|
||||
testingConfig, listener)
|
||||
|
||||
if expectedStatusCode != response.StatusCode {
|
||||
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
|
||||
expectedStatusCode, response.StatusCode))
|
||||
}
|
||||
}
|
||||
|
||||
func TestTemplateErrorsCreateRawAlertNotice(t *testing.T) {
|
||||
listener := NewFakeHTTPListener()
|
||||
testingConfig := MakeHTTPTestingConfig()
|
||||
testingConfig.NoticeTemplate = "Bogus template {{ nil }}"
|
||||
|
||||
expectedAlertNotices := []AlertNotice{
|
||||
AlertNotice{
|
||||
Channel: "#somechannel",
|
||||
Alert: `{"status":"resolved","labels":{"alertname":"airDown","instance":"instance1:3456","job":"air","service":"prometheus","severity":"ticket","zone":"global"},"annotations":{"DESCRIPTION":"service /prometheus has irc gateway down on instance1","SUMMARY":"service /prometheus air down on instance1"},"startsAt":"2017-05-15T13:49:37.834Z","endsAt":"2017-05-15T13:50:37.835Z","generatorURL":"https://prometheus.example.com/prometheus/..."}`,
|
||||
},
|
||||
AlertNotice{
|
||||
Channel: "#somechannel",
|
||||
Alert: `{"status":"resolved","labels":{"alertname":"airDown","instance":"instance2:7890","job":"air","service":"prometheus","severity":"ticket","zone":"global"},"annotations":{"DESCRIPTION":"service /prometheus has irc gateway down on instance2","SUMMARY":"service /prometheus air down on instance2"},"startsAt":"2017-05-15T11:47:37.834Z","endsAt":"2017-05-15T11:48:37.834Z","generatorURL":"https://prometheus.example.com/prometheus/..."}`,
|
||||
},
|
||||
}
|
||||
expectedStatusCode := 200
|
||||
|
||||
response := RunHTTPTest(
|
||||
t, testdataSimpleAlertJson, "/somechannel",
|
||||
testingConfig, listener)
|
||||
|
||||
if expectedStatusCode != response.StatusCode {
|
||||
t.Error(fmt.Sprintf("Expected %d status in response, got %d",
|
||||
expectedStatusCode, response.StatusCode))
|
||||
}
|
||||
|
||||
for _, expectedAlertNotice := range expectedAlertNotices {
|
||||
alertNotice := <-listener.AlertNotices
|
||||
if !reflect.DeepEqual(expectedAlertNotice, alertNotice) {
|
||||
t.Error(fmt.Sprintf(
|
||||
"Unexpected alert notice.\nExpected: %s\nActual: %s",
|
||||
expectedAlertNotice, alertNotice))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,251 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
irc "github.com/fluffle/goirc/client"
|
||||
"log"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
pingFrequencySecs = 60
|
||||
connectionTimeoutSecs = 30
|
||||
nickservWaitSecs = 10
|
||||
ircConnectMaxBackoffSecs = 300
|
||||
ircConnectBackoffResetSecs = 1800
|
||||
)
|
||||
|
||||
func loggerHandler(_ *irc.Conn, line *irc.Line) {
|
||||
log.Printf("Received: '%s'", line.Raw)
|
||||
}
|
||||
|
||||
type ChannelState struct {
|
||||
Channel IRCChannel
|
||||
BackoffCounter Delayer
|
||||
}
|
||||
|
||||
type IRCNotifier struct {
|
||||
// Nick stores the nickname specified in the config, because irc.Client
|
||||
// might change its copy.
|
||||
Nick string
|
||||
NickPassword string
|
||||
Client *irc.Conn
|
||||
StopRunning chan bool
|
||||
StoppedRunning chan bool
|
||||
AlertNotices chan AlertNotice
|
||||
|
||||
// irc.Conn has a Connected() method that can tell us wether the TCP
|
||||
// connection is up, and thus if we should trigger connect/disconnect.
|
||||
// We need to track the session establishment also at a higher level to
|
||||
// understand when the server has accepted us and thus when we can join
|
||||
// channels, send notices, etc.
|
||||
sessionUp bool
|
||||
sessionUpSignal chan bool
|
||||
sessionDownSignal chan bool
|
||||
|
||||
PreJoinChannels []IRCChannel
|
||||
JoinedChannels map[string]ChannelState
|
||||
|
||||
NickservDelayWait time.Duration
|
||||
BackoffCounter Delayer
|
||||
}
|
||||
|
||||
func NewIRCNotifier(config *Config, alertNotices chan AlertNotice) (*IRCNotifier, error) {
|
||||
|
||||
ircConfig := irc.NewConfig(config.IRCNick)
|
||||
ircConfig.Me.Ident = config.IRCNick
|
||||
ircConfig.Me.Name = config.IRCRealName
|
||||
ircConfig.Server = strings.Join(
|
||||
[]string{config.IRCHost, strconv.Itoa(config.IRCPort)}, ":")
|
||||
ircConfig.SSL = config.IRCUseSSL
|
||||
ircConfig.SSLConfig = &tls.Config{ServerName: config.IRCHost}
|
||||
ircConfig.PingFreq = pingFrequencySecs * time.Second
|
||||
ircConfig.Timeout = connectionTimeoutSecs * time.Second
|
||||
ircConfig.NewNick = func(n string) string { return n + "^" }
|
||||
|
||||
backoffCounter := NewBackoff(
|
||||
ircConnectMaxBackoffSecs, ircConnectBackoffResetSecs,
|
||||
time.Second)
|
||||
|
||||
notifier := &IRCNotifier{
|
||||
Nick: config.IRCNick,
|
||||
NickPassword: config.IRCNickPass,
|
||||
Client: irc.Client(ircConfig),
|
||||
StopRunning: make(chan bool),
|
||||
StoppedRunning: make(chan bool),
|
||||
AlertNotices: alertNotices,
|
||||
sessionUpSignal: make(chan bool),
|
||||
sessionDownSignal: make(chan bool),
|
||||
PreJoinChannels: config.IRCChannels,
|
||||
JoinedChannels: make(map[string]ChannelState),
|
||||
NickservDelayWait: nickservWaitSecs * time.Second,
|
||||
BackoffCounter: backoffCounter,
|
||||
}
|
||||
|
||||
notifier.Client.HandleFunc(irc.CONNECTED,
|
||||
func(*irc.Conn, *irc.Line) {
|
||||
log.Printf("Session established")
|
||||
notifier.sessionUpSignal <- true
|
||||
})
|
||||
|
||||
notifier.Client.HandleFunc(irc.DISCONNECTED,
|
||||
func(*irc.Conn, *irc.Line) {
|
||||
log.Printf("Disconnected from IRC")
|
||||
notifier.sessionDownSignal <- false
|
||||
})
|
||||
|
||||
notifier.Client.HandleFunc(irc.KICK,
|
||||
func(_ *irc.Conn, line *irc.Line) {
|
||||
notifier.HandleKick(line.Args[1], line.Args[0])
|
||||
})
|
||||
|
||||
for _, event := range []string{irc.NOTICE, "433"} {
|
||||
notifier.Client.HandleFunc(event, loggerHandler)
|
||||
}
|
||||
|
||||
return notifier, nil
|
||||
}
|
||||
|
||||
func (notifier *IRCNotifier) HandleKick(nick string, channel string) {
|
||||
if nick != notifier.Client.Me().Nick {
|
||||
// received kick info for somebody else
|
||||
return
|
||||
}
|
||||
state, ok := notifier.JoinedChannels[channel]
|
||||
if ok == false {
|
||||
log.Printf("Being kicked out of non-joined channel (%s), ignoring", channel)
|
||||
return
|
||||
}
|
||||
log.Printf("Being kicked out of %s, re-joining", channel)
|
||||
go func() {
|
||||
state.BackoffCounter.Delay()
|
||||
notifier.Client.Join(state.Channel.Name, state.Channel.Password)
|
||||
}()
|
||||
|
||||
}
|
||||
|
||||
func (notifier *IRCNotifier) CleanupChannels() {
|
||||
log.Printf("Deregistering all channels.")
|
||||
notifier.JoinedChannels = make(map[string]ChannelState)
|
||||
}
|
||||
|
||||
func (notifier *IRCNotifier) JoinChannel(channel *IRCChannel) {
|
||||
if _, joined := notifier.JoinedChannels[channel.Name]; joined == true {
|
||||
return
|
||||
}
|
||||
log.Printf("Joining %s", channel.Name)
|
||||
notifier.Client.Join(channel.Name, channel.Password)
|
||||
state := ChannelState{
|
||||
Channel: *channel,
|
||||
BackoffCounter: NewBackoff(
|
||||
ircConnectMaxBackoffSecs, ircConnectBackoffResetSecs,
|
||||
time.Second),
|
||||
}
|
||||
notifier.JoinedChannels[channel.Name] = state
|
||||
}
|
||||
|
||||
func (notifier *IRCNotifier) JoinChannels() {
|
||||
for _, channel := range notifier.PreJoinChannels {
|
||||
notifier.JoinChannel(&channel)
|
||||
}
|
||||
}
|
||||
|
||||
func (notifier *IRCNotifier) MaybeIdentifyNick() {
|
||||
if notifier.NickPassword == "" {
|
||||
return
|
||||
}
|
||||
|
||||
// Very lazy/optimistic, but this is good enough for my irssi config,
|
||||
// so it should work here as well.
|
||||
currentNick := notifier.Client.Me().Nick
|
||||
if currentNick != notifier.Nick {
|
||||
log.Printf("My nick is '%s', sending GHOST to NickServ to get '%s'",
|
||||
currentNick, notifier.Nick)
|
||||
notifier.Client.Privmsgf("NickServ", "GHOST %s %s", notifier.Nick,
|
||||
notifier.NickPassword)
|
||||
time.Sleep(notifier.NickservDelayWait)
|
||||
|
||||
log.Printf("Changing nick to '%s'", notifier.Nick)
|
||||
notifier.Client.Nick(notifier.Nick)
|
||||
}
|
||||
log.Printf("Sending IDENTIFY to NickServ")
|
||||
notifier.Client.Privmsgf("NickServ", "IDENTIFY %s", notifier.NickPassword)
|
||||
time.Sleep(notifier.NickservDelayWait)
|
||||
}
|
||||
|
||||
func (notifier *IRCNotifier) MaybeSendAlertNotice(alertNotice *AlertNotice) {
|
||||
if !notifier.sessionUp {
|
||||
log.Printf("Cannot send alert to %s : IRC not connected",
|
||||
alertNotice.Channel)
|
||||
return
|
||||
}
|
||||
notifier.JoinChannel(&IRCChannel{Name: alertNotice.Channel})
|
||||
notifier.Client.Notice(alertNotice.Channel, alertNotice.Alert)
|
||||
}
|
||||
|
||||
func (notifier *IRCNotifier) Run() {
|
||||
keepGoing := true
|
||||
for keepGoing {
|
||||
if !notifier.Client.Connected() {
|
||||
log.Printf("Connecting to IRC")
|
||||
notifier.BackoffCounter.Delay()
|
||||
if err := notifier.Client.Connect(); err != nil {
|
||||
log.Printf("Could not connect to IRC: %s", err)
|
||||
select {
|
||||
case <-notifier.StopRunning:
|
||||
log.Printf("IRC routine not connected but asked to terminate")
|
||||
keepGoing = false
|
||||
default:
|
||||
}
|
||||
continue
|
||||
}
|
||||
log.Printf("Connected to IRC server, waiting to establish session")
|
||||
}
|
||||
|
||||
select {
|
||||
case alertNotice := <-notifier.AlertNotices:
|
||||
notifier.MaybeSendAlertNotice(&alertNotice)
|
||||
case <-notifier.sessionUpSignal:
|
||||
notifier.sessionUp = true
|
||||
notifier.MaybeIdentifyNick()
|
||||
notifier.JoinChannels()
|
||||
case <-notifier.sessionDownSignal:
|
||||
notifier.sessionUp = false
|
||||
notifier.CleanupChannels()
|
||||
notifier.Client.Quit("see ya")
|
||||
case <-notifier.StopRunning:
|
||||
log.Printf("IRC routine asked to terminate")
|
||||
keepGoing = false
|
||||
}
|
||||
}
|
||||
if notifier.Client.Connected() {
|
||||
log.Printf("IRC client connected, quitting")
|
||||
notifier.Client.Quit("see ya")
|
||||
|
||||
if notifier.sessionUp {
|
||||
log.Printf("Session is up, wait for IRC disconnect to complete")
|
||||
select {
|
||||
case <-notifier.sessionDownSignal:
|
||||
case <-time.After(notifier.Client.Config().Timeout):
|
||||
log.Printf("Timeout while waiting for IRC disconnect to complete, stopping anyway")
|
||||
}
|
||||
}
|
||||
}
|
||||
notifier.StoppedRunning <- true
|
||||
}
|
||||
|
|
@ -0,0 +1,720 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
irc "github.com/fluffle/goirc/client"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type LineHandlerFunc func(*bufio.ReadWriter, *irc.Line) error
|
||||
|
||||
func h_USER(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
r := fmt.Sprintf(":example.com 001 %s :Welcome\n", line.Args[0])
|
||||
conn.WriteString(r)
|
||||
return nil
|
||||
}
|
||||
|
||||
func h_QUIT(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
return fmt.Errorf("client asked to terminate")
|
||||
}
|
||||
|
||||
type closeEarlyHandler func()
|
||||
|
||||
type testServer struct {
|
||||
net.Listener
|
||||
Client net.Conn
|
||||
|
||||
ServingWaitGroup sync.WaitGroup
|
||||
ConnectionsWaitGroup sync.WaitGroup
|
||||
|
||||
lineHandlersMu sync.Mutex
|
||||
lineHandlers map[string]LineHandlerFunc
|
||||
|
||||
Log []string
|
||||
|
||||
closeEarlyMu sync.Mutex
|
||||
closeEarlyHandler
|
||||
}
|
||||
|
||||
func (s *testServer) setDefaultHandlers() {
|
||||
if s.lineHandlers == nil {
|
||||
s.lineHandlers = make(map[string]LineHandlerFunc)
|
||||
}
|
||||
s.lineHandlers["USER"] = h_USER
|
||||
s.lineHandlers["QUIT"] = h_QUIT
|
||||
}
|
||||
|
||||
func (s *testServer) getHandler(cmd string) LineHandlerFunc {
|
||||
s.lineHandlersMu.Lock()
|
||||
defer s.lineHandlersMu.Unlock()
|
||||
return s.lineHandlers[cmd]
|
||||
}
|
||||
|
||||
func (s *testServer) SetHandler(cmd string, h LineHandlerFunc) {
|
||||
s.lineHandlersMu.Lock()
|
||||
defer s.lineHandlersMu.Unlock()
|
||||
if h == nil {
|
||||
delete(s.lineHandlers, cmd)
|
||||
} else {
|
||||
s.lineHandlers[cmd] = h
|
||||
}
|
||||
}
|
||||
|
||||
func (s *testServer) handleLine(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
s.Log = append(s.Log, strings.Trim(line.Raw, " \r\n"))
|
||||
handler := s.getHandler(line.Cmd)
|
||||
if handler == nil {
|
||||
log.Printf("=Server= No handler for command '%s', skipping", line.Cmd)
|
||||
return nil
|
||||
}
|
||||
return handler(conn, line)
|
||||
}
|
||||
|
||||
func (s *testServer) handleConnection(conn net.Conn) {
|
||||
defer func() {
|
||||
s.Client = nil
|
||||
conn.Close()
|
||||
s.ConnectionsWaitGroup.Done()
|
||||
}()
|
||||
bufConn := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))
|
||||
for {
|
||||
msg, err := bufConn.ReadBytes('\n')
|
||||
if err != nil {
|
||||
if err == io.EOF {
|
||||
log.Printf("=Server= Client %s disconnected", conn.RemoteAddr().String())
|
||||
} else {
|
||||
log.Printf("=Server= Could not read from %s: %s", conn.RemoteAddr().String(), err)
|
||||
}
|
||||
return
|
||||
}
|
||||
log.Printf("=Server= Received %s", msg)
|
||||
line := irc.ParseLine(string(msg))
|
||||
if line == nil {
|
||||
log.Printf("=Server= Could not parse received line")
|
||||
continue
|
||||
}
|
||||
err = s.handleLine(bufConn, line)
|
||||
if err != nil {
|
||||
log.Printf("=Server= Closing connection: %s", err)
|
||||
return
|
||||
}
|
||||
bufConn.Flush()
|
||||
}
|
||||
}
|
||||
|
||||
func (s *testServer) SetCloseEarly(h closeEarlyHandler) {
|
||||
s.closeEarlyMu.Lock()
|
||||
defer s.closeEarlyMu.Unlock()
|
||||
s.closeEarlyHandler = h
|
||||
}
|
||||
|
||||
func (s *testServer) handleCloseEarly(conn net.Conn) bool {
|
||||
s.closeEarlyMu.Lock()
|
||||
defer s.closeEarlyMu.Unlock()
|
||||
if s.closeEarlyHandler == nil {
|
||||
return false
|
||||
}
|
||||
log.Printf("=Server= Closing connection early")
|
||||
conn.Close()
|
||||
s.closeEarlyHandler()
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *testServer) Serve() {
|
||||
defer s.ServingWaitGroup.Done()
|
||||
for {
|
||||
conn, err := s.Listener.Accept()
|
||||
if err != nil {
|
||||
log.Printf("=Server= Stopped accepting new connections")
|
||||
return
|
||||
}
|
||||
log.Printf("=Server= New client connected from %s", conn.RemoteAddr().String())
|
||||
if s.handleCloseEarly(conn) {
|
||||
continue
|
||||
}
|
||||
s.Client = conn
|
||||
s.ConnectionsWaitGroup.Add(1)
|
||||
s.handleConnection(conn)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *testServer) Stop() {
|
||||
s.Listener.Close()
|
||||
s.ServingWaitGroup.Wait()
|
||||
s.ConnectionsWaitGroup.Wait()
|
||||
}
|
||||
|
||||
func makeTestServer(t *testing.T) (*testServer, int) {
|
||||
server := new(testServer)
|
||||
server.Log = make([]string, 0)
|
||||
server.setDefaultHandlers()
|
||||
|
||||
addr, err := net.ResolveTCPAddr("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatalf("=Server= Could not resolve tcp addr: %s", err)
|
||||
}
|
||||
listener, err := net.ListenTCP("tcp", addr)
|
||||
if err != nil {
|
||||
t.Fatalf("=Server= Could not create listener: %s", err)
|
||||
}
|
||||
addr = listener.Addr().(*net.TCPAddr)
|
||||
log.Printf("=Server= Test server listening on %s", addr.String())
|
||||
|
||||
server.Listener = listener
|
||||
|
||||
server.ServingWaitGroup.Add(1)
|
||||
go func() {
|
||||
server.Serve()
|
||||
}()
|
||||
|
||||
addr = listener.Addr().(*net.TCPAddr)
|
||||
return server, addr.Port
|
||||
}
|
||||
|
||||
type FakeDelayer struct {
|
||||
}
|
||||
|
||||
func (f *FakeDelayer) Delay() {
|
||||
log.Printf("Faking Backoff")
|
||||
}
|
||||
|
||||
func makeTestIRCConfig(IRCPort int) *Config {
|
||||
return &Config{
|
||||
IRCNick: "foo",
|
||||
IRCNickPass: "",
|
||||
IRCHost: "127.0.0.1",
|
||||
IRCPort: IRCPort,
|
||||
IRCUseSSL: false,
|
||||
IRCChannels: []IRCChannel{
|
||||
IRCChannel{Name: "#foo"},
|
||||
IRCChannel{Name: "#bar"},
|
||||
IRCChannel{Name: "#baz"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func makeTestNotifier(t *testing.T, config *Config) (*IRCNotifier, chan AlertNotice) {
|
||||
alertNotices := make(chan AlertNotice)
|
||||
notifier, err := NewIRCNotifier(config, alertNotices)
|
||||
if err != nil {
|
||||
t.Fatal(fmt.Sprintf("Could not create IRC notifier: %s", err))
|
||||
}
|
||||
notifier.Client.Config().Flood = true
|
||||
notifier.BackoffCounter = &FakeDelayer{}
|
||||
|
||||
return notifier, alertNotices
|
||||
}
|
||||
|
||||
func TestPreJoinChannels(t *testing.T) {
|
||||
server, port := makeTestServer(t)
|
||||
config := makeTestIRCConfig(port)
|
||||
notifier, _ := makeTestNotifier(t, config)
|
||||
|
||||
var testStep sync.WaitGroup
|
||||
|
||||
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
// #baz is configured as the last channel to pre-join
|
||||
if line.Args[0] == "#baz" {
|
||||
testStep.Done()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("JOIN", joinHandler)
|
||||
|
||||
testStep.Add(1)
|
||||
go notifier.Run()
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
notifier.StopRunning <- true
|
||||
server.Stop()
|
||||
|
||||
expectedCommands := []string{
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"JOIN #foo",
|
||||
"JOIN #bar",
|
||||
"JOIN #baz",
|
||||
"QUIT :see ya",
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(expectedCommands, server.Log) {
|
||||
t.Error("Did not pre-join channels")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSendAlertOnPreJoinedChannel(t *testing.T) {
|
||||
server, port := makeTestServer(t)
|
||||
config := makeTestIRCConfig(port)
|
||||
notifier, alertNotices := makeTestNotifier(t, config)
|
||||
|
||||
var testStep sync.WaitGroup
|
||||
|
||||
testChannel := "#foo"
|
||||
testMessage := "test message"
|
||||
|
||||
// Send the alert after configured channels have joined, to ensure we
|
||||
// check for no re-join attempt.
|
||||
joinedHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
if line.Args[0] == testChannel {
|
||||
testStep.Done()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("JOIN", joinedHandler)
|
||||
|
||||
testStep.Add(1)
|
||||
go notifier.Run()
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
server.SetHandler("JOIN", nil)
|
||||
|
||||
noticeHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
testStep.Done()
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("NOTICE", noticeHandler)
|
||||
|
||||
testStep.Add(1)
|
||||
alertNotices <- AlertNotice{Channel: testChannel, Alert: testMessage}
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
notifier.StopRunning <- true
|
||||
server.Stop()
|
||||
|
||||
expectedCommands := []string{
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"JOIN #foo",
|
||||
"JOIN #bar",
|
||||
"JOIN #baz",
|
||||
"NOTICE #foo :test message",
|
||||
"QUIT :see ya",
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(expectedCommands, server.Log) {
|
||||
t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSendAlertAndJoinChannel(t *testing.T) {
|
||||
server, port := makeTestServer(t)
|
||||
config := makeTestIRCConfig(port)
|
||||
notifier, alertNotices := makeTestNotifier(t, config)
|
||||
|
||||
var testStep sync.WaitGroup
|
||||
|
||||
testChannel := "#foobar"
|
||||
testMessage := "test message"
|
||||
|
||||
// Send the alert after configured channels have joined, to ensure log
|
||||
// ordering.
|
||||
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
// #baz is configured as the last channel to pre-join
|
||||
if line.Args[0] == "#baz" {
|
||||
testStep.Done()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("JOIN", joinHandler)
|
||||
|
||||
testStep.Add(1)
|
||||
go notifier.Run()
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
server.SetHandler("JOIN", nil)
|
||||
|
||||
noticeHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
testStep.Done()
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("NOTICE", noticeHandler)
|
||||
|
||||
testStep.Add(1)
|
||||
alertNotices <- AlertNotice{Channel: testChannel, Alert: testMessage}
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
notifier.StopRunning <- true
|
||||
server.Stop()
|
||||
|
||||
expectedCommands := []string{
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"JOIN #foo",
|
||||
"JOIN #bar",
|
||||
"JOIN #baz",
|
||||
// #foobar joined before sending message
|
||||
"JOIN #foobar",
|
||||
"NOTICE #foobar :test message",
|
||||
"QUIT :see ya",
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(expectedCommands, server.Log) {
|
||||
t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSendAlertDisconnected(t *testing.T) {
|
||||
server, port := makeTestServer(t)
|
||||
config := makeTestIRCConfig(port)
|
||||
notifier, alertNotices := makeTestNotifier(t, config)
|
||||
|
||||
var testStep, holdUserStep sync.WaitGroup
|
||||
|
||||
testChannel := "#foo"
|
||||
disconnectedTestMessage := "disconnected test message"
|
||||
connectedTestMessage := "connected test message"
|
||||
|
||||
// First send an alert while the session is not established.
|
||||
testStep.Add(1)
|
||||
holdUserStep.Add(1)
|
||||
holdUser := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
log.Printf("=Server= Wait before completing session")
|
||||
testStep.Wait()
|
||||
log.Printf("=Server= Completing session")
|
||||
holdUserStep.Done()
|
||||
return h_USER(conn, line)
|
||||
}
|
||||
server.SetHandler("USER", holdUser)
|
||||
|
||||
go notifier.Run()
|
||||
|
||||
alertNotices <- AlertNotice{Channel: testChannel, Alert: disconnectedTestMessage}
|
||||
|
||||
testStep.Done()
|
||||
holdUserStep.Wait()
|
||||
|
||||
// Make sure session is established by checking that pre-joined
|
||||
// channels are there.
|
||||
testStep.Add(1)
|
||||
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
// #baz is configured as the last channel to pre-join
|
||||
if line.Args[0] == "#baz" {
|
||||
testStep.Done()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("JOIN", joinHandler)
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
// Now send and wait until a notice has been received.
|
||||
testStep.Add(1)
|
||||
noticeHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
testStep.Done()
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("NOTICE", noticeHandler)
|
||||
|
||||
alertNotices <- AlertNotice{Channel: testChannel, Alert: connectedTestMessage}
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
notifier.StopRunning <- true
|
||||
server.Stop()
|
||||
|
||||
expectedCommands := []string{
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"JOIN #foo",
|
||||
"JOIN #bar",
|
||||
"JOIN #baz",
|
||||
// Only message sent while being connected is received.
|
||||
"NOTICE #foo :connected test message",
|
||||
"QUIT :see ya",
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(expectedCommands, server.Log) {
|
||||
t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestReconnect(t *testing.T) {
|
||||
server, port := makeTestServer(t)
|
||||
config := makeTestIRCConfig(port)
|
||||
notifier, _ := makeTestNotifier(t, config)
|
||||
|
||||
var testStep sync.WaitGroup
|
||||
|
||||
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
// #baz is configured as the last channel to pre-join
|
||||
if line.Args[0] == "#baz" {
|
||||
testStep.Done()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("JOIN", joinHandler)
|
||||
|
||||
testStep.Add(1)
|
||||
go notifier.Run()
|
||||
|
||||
// Wait until the last pre-joined channel is seen.
|
||||
testStep.Wait()
|
||||
|
||||
// Simulate disconnection.
|
||||
testStep.Add(1)
|
||||
server.Client.Close()
|
||||
|
||||
// Wait again until the last pre-joined channel is seen.
|
||||
testStep.Wait()
|
||||
|
||||
notifier.StopRunning <- true
|
||||
server.Stop()
|
||||
|
||||
expectedCommands := []string{
|
||||
// Commands from first connection
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"JOIN #foo",
|
||||
"JOIN #bar",
|
||||
"JOIN #baz",
|
||||
// Commands from reconnection
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"JOIN #foo",
|
||||
"JOIN #bar",
|
||||
"JOIN #baz",
|
||||
"QUIT :see ya",
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(expectedCommands, server.Log) {
|
||||
t.Error("Reconnection did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectErrorRetry(t *testing.T) {
|
||||
server, port := makeTestServer(t)
|
||||
config := makeTestIRCConfig(port)
|
||||
// Attempt SSL handshake. The server does not support it, resulting in
|
||||
// a connection error.
|
||||
config.IRCUseSSL = true
|
||||
notifier, _ := makeTestNotifier(t, config)
|
||||
|
||||
var testStep, joinStep sync.WaitGroup
|
||||
|
||||
testStep.Add(1)
|
||||
earlyHandler := func() {
|
||||
testStep.Done()
|
||||
}
|
||||
|
||||
server.SetCloseEarly(earlyHandler)
|
||||
|
||||
go notifier.Run()
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
// We have caused a connection failure, now check for a reconnection
|
||||
notifier.Client.Config().SSL = false
|
||||
joinStep.Add(1)
|
||||
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
// #baz is configured as the last channel to pre-join
|
||||
if line.Args[0] == "#baz" {
|
||||
joinStep.Done()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("JOIN", joinHandler)
|
||||
server.SetCloseEarly(nil)
|
||||
|
||||
joinStep.Wait()
|
||||
|
||||
notifier.StopRunning <- true
|
||||
server.Stop()
|
||||
|
||||
expectedCommands := []string{
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"JOIN #foo",
|
||||
"JOIN #bar",
|
||||
"JOIN #baz",
|
||||
"QUIT :see ya",
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(expectedCommands, server.Log) {
|
||||
t.Error("Reconnection did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentify(t *testing.T) {
|
||||
server, port := makeTestServer(t)
|
||||
config := makeTestIRCConfig(port)
|
||||
config.IRCNickPass = "nickpassword"
|
||||
notifier, _ := makeTestNotifier(t, config)
|
||||
notifier.NickservDelayWait = 0 * time.Second
|
||||
|
||||
var testStep sync.WaitGroup
|
||||
|
||||
// Wait until the last pre-joined channel is seen (joining happens
|
||||
// after identification).
|
||||
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
// #baz is configured as the last channel to pre-join
|
||||
if line.Args[0] == "#baz" {
|
||||
testStep.Done()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("JOIN", joinHandler)
|
||||
|
||||
testStep.Add(1)
|
||||
go notifier.Run()
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
notifier.StopRunning <- true
|
||||
server.Stop()
|
||||
|
||||
expectedCommands := []string{
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"PRIVMSG NickServ :IDENTIFY nickpassword",
|
||||
"JOIN #foo",
|
||||
"JOIN #bar",
|
||||
"JOIN #baz",
|
||||
"QUIT :see ya",
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(expectedCommands, server.Log) {
|
||||
t.Error("Identification did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGhostAndIdentify(t *testing.T) {
|
||||
server, port := makeTestServer(t)
|
||||
config := makeTestIRCConfig(port)
|
||||
config.IRCNickPass = "nickpassword"
|
||||
notifier, _ := makeTestNotifier(t, config)
|
||||
notifier.NickservDelayWait = 0 * time.Second
|
||||
|
||||
var testStep, usedNick, unregisteredNickHandler sync.WaitGroup
|
||||
|
||||
// Trigger 433 for first nick
|
||||
usedNick.Add(1)
|
||||
unregisteredNickHandler.Add(1)
|
||||
nickHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
if line.Args[0] == "foo" {
|
||||
conn.WriteString(":example.com 433 * foo :nick in use\n")
|
||||
}
|
||||
usedNick.Done()
|
||||
unregisteredNickHandler.Wait()
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("NICK", nickHandler)
|
||||
|
||||
// Wait until the last pre-joined channel is seen (joining happens
|
||||
// after identification).
|
||||
joinHandler := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
// #baz is configured as the last channel to pre-join
|
||||
if line.Args[0] == "#baz" {
|
||||
testStep.Done()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("JOIN", joinHandler)
|
||||
|
||||
testStep.Add(1)
|
||||
go notifier.Run()
|
||||
|
||||
usedNick.Wait()
|
||||
server.SetHandler("NICK", nil)
|
||||
unregisteredNickHandler.Done()
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
notifier.StopRunning <- true
|
||||
server.Stop()
|
||||
|
||||
expectedCommands := []string{
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"NICK foo^",
|
||||
"PRIVMSG NickServ :GHOST foo nickpassword",
|
||||
"NICK foo",
|
||||
"PRIVMSG NickServ :IDENTIFY nickpassword",
|
||||
"JOIN #foo",
|
||||
"JOIN #bar",
|
||||
"JOIN #baz",
|
||||
"QUIT :see ya",
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(expectedCommands, server.Log) {
|
||||
t.Error("Ghosting did not happen correctly. Received commands:\n", strings.Join(server.Log, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestStopRunningWhenHalfConnected(t *testing.T) {
|
||||
server, port := makeTestServer(t)
|
||||
config := makeTestIRCConfig(port)
|
||||
notifier, _ := makeTestNotifier(t, config)
|
||||
|
||||
var testStep, holdQuitWait sync.WaitGroup
|
||||
|
||||
// Send a StopRunning request while the client is connected but the
|
||||
// session is not up
|
||||
testStep.Add(1)
|
||||
holdUser := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
log.Printf("=Server= NOT completing session")
|
||||
testStep.Done()
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("USER", holdUser)
|
||||
|
||||
// Ignore quit, but wait for it to have deterministic test commands
|
||||
holdQuitWait.Add(1)
|
||||
holdQuit := func(conn *bufio.ReadWriter, line *irc.Line) error {
|
||||
log.Printf("=Server= Ignoring quit")
|
||||
holdQuitWait.Done()
|
||||
return nil
|
||||
}
|
||||
server.SetHandler("QUIT", holdQuit)
|
||||
|
||||
go notifier.Run()
|
||||
|
||||
testStep.Wait()
|
||||
|
||||
notifier.StopRunning <- true
|
||||
|
||||
<-notifier.StoppedRunning
|
||||
|
||||
holdQuitWait.Wait()
|
||||
|
||||
// Client has left, cleanup the server side before stopping
|
||||
server.Client.Close()
|
||||
|
||||
server.Stop()
|
||||
|
||||
expectedCommands := []string{
|
||||
"NICK foo",
|
||||
"USER foo 12 * :",
|
||||
"QUIT :see ya",
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(expectedCommands, server.Log) {
|
||||
t.Error("Alert not sent correctly. Received commands:\n", strings.Join(server.Log, "\n"))
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"log"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
func main() {
|
||||
|
||||
configFile := flag.String("config", "", "Config file path.")
|
||||
|
||||
flag.Parse()
|
||||
|
||||
signals := make(chan os.Signal, 1)
|
||||
signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM)
|
||||
|
||||
config, err := LoadConfig(*configFile)
|
||||
if err != nil {
|
||||
log.Printf("Could not load config: %s", err)
|
||||
return
|
||||
}
|
||||
|
||||
alertNotices := make(chan AlertNotice, 10)
|
||||
|
||||
ircNotifier, err := NewIRCNotifier(config, alertNotices)
|
||||
if err != nil {
|
||||
log.Printf("Could not create IRC notifier: %s", err)
|
||||
return
|
||||
}
|
||||
go ircNotifier.Run()
|
||||
|
||||
httpServer, err := NewHTTPServer(config, alertNotices)
|
||||
if err != nil {
|
||||
log.Printf("Could not create HTTP server: %s", err)
|
||||
return
|
||||
}
|
||||
go httpServer.Run()
|
||||
|
||||
select {
|
||||
case <-httpServer.StoppedRunning:
|
||||
log.Printf("Http server terminated, exiting")
|
||||
case <-ircNotifier.StoppedRunning:
|
||||
log.Printf("IRC notifier stopped running, exiting")
|
||||
case s := <-signals:
|
||||
log.Printf("Received %s, exiting", s)
|
||||
ircNotifier.StopRunning <- true
|
||||
log.Printf("Waiting for IRC to quit")
|
||||
<-ircNotifier.StoppedRunning
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
// Copyright 2018 Google LLC
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
const (
|
||||
testdataSimpleAlertJson = `
|
||||
{
|
||||
"status": "resolved",
|
||||
"receiver": "example_receiver",
|
||||
"groupLabels": {
|
||||
"alertname": "airDown",
|
||||
"service": "prometheus"
|
||||
},
|
||||
"commonLabels": {
|
||||
"alertname": "airDown",
|
||||
"job": "air",
|
||||
"service": "prometheus",
|
||||
"severity": "ticket",
|
||||
"zone": "global"
|
||||
},
|
||||
"commonAnnotations": {},
|
||||
"externalURL": "https://prometheus.example.com/alertmanager",
|
||||
"alerts": [
|
||||
{
|
||||
"annotations": {
|
||||
"SUMMARY": "service /prometheus air down on instance1",
|
||||
"DESCRIPTION": "service /prometheus has irc gateway down on instance1"
|
||||
},
|
||||
"endsAt": "2017-05-15T13:50:37.835Z",
|
||||
"generatorURL": "https://prometheus.example.com/prometheus/...",
|
||||
"labels": {
|
||||
"alertname": "airDown",
|
||||
"instance": "instance1:3456",
|
||||
"job": "air",
|
||||
"service": "prometheus",
|
||||
"severity": "ticket",
|
||||
"zone": "global"
|
||||
},
|
||||
"startsAt": "2017-05-15T13:49:37.834Z",
|
||||
"status": "resolved"
|
||||
},
|
||||
{
|
||||
"annotations": {
|
||||
"SUMMARY": "service /prometheus air down on instance2",
|
||||
"DESCRIPTION": "service /prometheus has irc gateway down on instance2"
|
||||
},
|
||||
"endsAt": "2017-05-15T11:48:37.834Z",
|
||||
"generatorURL": "https://prometheus.example.com/prometheus/...",
|
||||
"labels": {
|
||||
"alertname": "airDown",
|
||||
"instance": "instance2:7890",
|
||||
"job": "air",
|
||||
"service": "prometheus",
|
||||
"severity": "ticket",
|
||||
"zone": "global"
|
||||
},
|
||||
"startsAt": "2017-05-15T11:47:37.834Z",
|
||||
"status": "resolved"
|
||||
}
|
||||
]
|
||||
}
|
||||
`
|
||||
|
||||
testdataBogusAlertJson = `{"this is not": "a valid alert",}`
|
||||
)
|
||||
Loading…
Reference in New Issue