Mykyta Bulatnikov
← All writing

I audited a week of our Slack alerts and most of them were noise

An alert bot everyone had learned to scroll past. What a week of its own messages showed, and the four changes that cut the volume by about two thirds.

We had a Slack bot that nobody trusted anymore.

Not because it was wrong. It was right about most things. But it posted five times a day, every day, and after two weeks the team had learned to scroll past it. When I asked for honest feedback, one colleague said: “Important signals drown in the noise.” Another said: “Generally fine, but it’s hard to tell what actually needs action.”

So before changing anything, I spent a week looking at what the bot had actually been sending.

What the bot does

The company runs a property-management SaaS. Residents file requests (a broken lift, a roof leak), and staff work through them. The bot is a monitoring system I built to surface what matters: five Cloud Run functions, each watching a different slice of the CRM data in BigQuery and posting to Slack on a schedule.

ModuleWhat it doesSchedule
Emergency pulseTop-priority requests still unassigned after 60+ minutesEvery 15 min, 8–22h
Daily heartbeatDay summary: closed, overdue, backlog trendDaily, 8:00
Backlog (14+ days)Overdue requests, top 20 oldestDaily, 8:00
Backlog (30+ days)Critical requests stuck for a monthDaily, 8:00
Anomaly detectorResidents filing abusive or spam volumes3× daily

Five messages, each useful on its own. Together they were a wall of text that taught the team to ignore all of it.

What a week of data showed

I exported every message the bot sent over one week and went through them one by one. Some things are invisible in a single day.

The anomaly detector repeated the same residents three times a day

It runs at 8:30, 12:30 and 16:30 over a 24-hour window. So a resident who filed four requests on Monday appeared in all three Monday alerts, with the exact same request IDs. That is 21 alerts a week where about 7 carried news. The detector had no memory between runs.

The 30+ day backlog was the same list every day

Comparing the daily reports side by side:

May 1:  Request A: 121 days open
May 2:  Request A: 122 days open
May 3:  Request A: 123 days open

Seven @channel pings about the same requests, each one a day older. The team was being trained to ignore the @channel itself, so when something genuinely new appeared in that report, they missed it.

The heartbeat reported a big number with no context

✅ Closed yesterday: 10
⚠️ Current overdue requests: ~1,900

On its own that reads like a disaster. In fact the trend that week was improving, and “10 closed” was a Sunday. Without a delta, every number in the report relied on a baseline the reader had to remember.

The “top 20 oldest” list was one nobody could act on

The first entry was a request 76 days old. What could anyone do about it today? Nothing. It needed an escalation, not a daily nudge, and it was crowding out the three requests that did need attention right now.

What I changed

Deduplicate with a state table

An alert_state table in BigQuery stores a hash of the last payload per (alert_type, entity_key). Before posting, the detector reads it and skips anything whose composition has not changed since the last alert.

def filter_new_spam(client, spam_data):
    last = _state_get_last_hashes(client, 'anomaly_spam')
    fresh = []
    for row in spam_data:
        h = _hash_payload(sorted(row['request_ids']))
        if last.get(row['resident_id']) != h:
            fresh.append(row)
    return fresh

The detector now fires only when there is something new: from 21 alerts a week to 5–7.

”Actionable today” instead of “oldest”

I rebuilt the daily report around one question: can someone do something about this request today? Anything older than 30 days is systemic debt and needs an escalation, not a daily mention.

def is_actionable(o):
    if o['days_open'] > 30:
        return False                    # systemic debt, not today's job
    if o['priority_level'] in (1, 2) and o['status'] in ('new', 'consideration'):
        return True                     # emergency, not yet picked up
    if 14 <= o['days_open'] <= 21 and o['status'] in ('new', 'consideration'):
        return True                     # just crossed the 14-day line
    if (o['status'] == 'in_progress' and 14 <= o['days_open'] <= 30
            and o['days_since_last_update'] >= 7):
        return True                     # in progress but no movement for a week
    return False

The chronic backlog became a single number at the bottom of the report, with a warning if any critical request sits in that bucket.

Context on every number

Current overdue requests: ~1,900 📉 −12 since yesterday
Closed yesterday: 10 (weekend)

The 30+ report moved to Mondays

Weekly instead of daily, with no @channel, and a new section for requests that newly crossed 30 days this week, because those are the ones where action is still possible.

Results

MetricBefore (per week)After
Anomaly alerts~215–7
30+ day @channel pings71 (Monday, no @channel)
Daily actionable items20, of which 18 irrelevant~10, all relevant
Trend visible in the heartbeatNoYes (±N per day)

Getting the code under version control

Up to this point the functions lived in a folder on my laptop: no history, no changelog, no way to answer “what changed and why”. Moving to GitHub was supposed to be routine, and it had two surprises.

GitHub blocked the first push. Secret scanning found a hardcoded Slack webhook URL in five files. That was good news in disguise, because a webhook in git history is there permanently even if you delete it later. The fix was to read it from an environment variable.

The next run crashed. I had set the variable on the old revision and forgotten it on the new one, so the function read an empty string and failed on requests.post('') with a stack trace that said nothing useful. Every function now checks for it up front and fails with a message that says what is missing.

What I took away

An alert without novelty is noise. The cheapest fix for repeats is a state table and a payload hash: one BigQuery table and a few lines of Python.

Daily is not the same as comprehensive. A daily report should answer “what do I do today”. Anything older than 30 days is a different conversation, a weekly audit or an escalation. Mixing the two makes both worse.

Context beats raw numbers. “1,900 overdue” triggers panic. “1,900, down 12 since yesterday” informs.

Feedback gives direction, data gives scale. The team said “too much noise”, and they were right. A week of messages showed how much: 21 alerts where there should have been 7, and seven @channel pings about the same requests.

Version control is not for git enthusiasts. Without a repository and a changelog I could not have written this post. That is the real cost of skipping it: you lose the ability to explain what you built and why.

Stack: BigQuery, Cloud Run functions (Python 3.11), Cloud Scheduler, Slack webhooks.