The Case of the Disappearing Slack Notifications

We happened to notice that slack messages that we were used to getting were no longer showing up. This is a walk through of how they were setup, what we figured out, and what our next steps are.

Getting an app to send slack notifications

To start with you need to configure an Incoming Webhook, the essence of which involves:

• creating a Slack app
• getting the app approved
• enabling Incoming Webhooks
• then add a webhook by specifying a channel messages are sent to

Next you need to have your app generate a JSON message and post it to the url provided by the webhook. Then, violá, you have a notification in slack from your application.

So why did we stop getting some of our notifications?

We noticed that we no longer were getting notifications when some of or processes failed. We'd get success messages, so we knew some notifications were working fine, just not some of the failures.

What happens when a message is sent to a non-existant webhook?

It seems that slack has several error response. This doesn't help much if the apps aren't expecting this. So this is one thing to improve, but why would a webhook that was working start returning "no_service"?

We have had changes on the team and one notification was put together by an intern working on our team who's account was recently deactivated...

So what happens when your account is deactivated?

Slack provides some details on this, with a small teasing statement:

Apps the members set up may be disabled. You can manage and re-enable these apps if you'd like to reconnect them.

Following the link in that statement, but isn't as precise as we'd like:

...some apps they've installed (whether third-party or custom) will automatically disable. Some features remain active (bot users, slash commands, incoming webhooks, etc.), but apps that require member-specific permissions will disable entriely.

Finally, after a bit more digging some details were found in the Installing apps section of Managing Slack apps. Under the "A pleasant vacation from unexpected uninstallation" it notes how apps will not be automatically uninstalled if they limit their scope, but then it mentions a caveat:

One caveat: this exception doesn't apply to the folks who created an app or were added as an App Collaborator. When they leave, the app is still uninstalled.

So that explains why our notifications went away. Though I'm not sure how to avoid or alert on that.

Maybe collaborators?

My first thought was adding other team members to the collaborators so that more people could do the work, but the previous section implies that it might also delete it. Regardless, that's a stop gap at best.

Maybe notification on deletion?

I'm not sure who manages removing slack users, but I'm curious if it is possible to notify their former team of the apps that will be deactivated?

At least, Admins and Owners of our slack Workspace are able to reenable apps that have been deacivated.  So if we remember who set it up or to ask if anything recently got deactivated after someone has left..

Next steps

1. Improve the error handling when posting a slack message.
2. Add alerting if our apps cannot post a slack message (not just a log message).
3. Follow up with Slack Workspace Admins / Management on:
   
   1. If they can find apps owned by deactivated users
      
      • Unfortunately, they can only see the list of deactivated apps in chronological order.
   2. If we can get notified of apps that were deactivated when an account is inactivated.
      
      • The Slack Admins weren't aware of this process so more digging needs to happen to figure this out.
   3. Also worth asking these questions of Slack if I can find out where to ask them.