How-to · 23 Oct
How-to · 3 min read
How to add a moderation check to your own chat app
You are building an app with comments, chat or user profiles, and you need to stop abuse, scams and explicit images before other users see them. Writing a banned word list takes an afternoon and catches almost nothing that matters. Training your own classifier is a project in itself. You want one API call you can put in front of every new message, image or username, with a clear yes or no and a reason. Ideally it should also catch the messages that only look bad in the context of the conversation around them.
01
Why it's hard
Word filters only match words. People get around them with numbers in place of letters, odd Unicode characters or sarcasm that contains no bad word at all. Harassment is often spread across several messages, each harmless on its own, so checking one message in isolation misses it. Images and short video clips need a separate system, and running every frame of every clip through a model gets slow and costly fast. Small teams usually end up with patchy rules that annoy honest users and still let the worst content through. And every hour spent tuning those rules is an hour not spent on the product people came for. Meanwhile your users decide within a few bad experiences whether your app feels safe enough to keep using.
This part describes the usual way. It makes no claims about Supervisor, so nothing is quoted here.
02
How Supervisor does it
Supervisor gives you a REST API that reads intent instead of matching keywords. You send text or a base64 image to one endpoint and get back whether it was flagged and which labels apply, from harassment and hate to spam, scams and explicit content. A separate endpoint checks usernames and returns a confidence score, and short videos are handled by sampling only the frames where the picture changes. There are official SDKs for Python, JavaScript, Go, Rust and Java if you would rather not write raw HTTP calls. Each video frame it checks comes back with a timestamp, so you can find the exact flagged moment in a clip instead of only knowing that something in it was wrong.
Moderate text content using AI models. Optionally include context analysis to determine if message history is needed.Quoted from supervisor.gg
Check whether a username is allowed. Returns a flag and a confidence score.Quoted from supervisor.gg
frames carries a timestamp_ms per analysed frame, so a flagged moment can be located in the clip rather than only reported for the video as a whole.Quoted from supervisor.gg
03
Step by step
- 1Sign up and create an API key in the dashboard.
- 2Send each new message to POST /moderate with your key in the Authorization header.
- 3Pick a model: observer for high volume, sentinel as the default, arbiter for the most careful checks.
- 4Turn on only the labels your community rules care about with enabled_labels.
- 5Hide or hold anything that comes back flagged, and show the label to your moderators.
- 6Check new usernames with POST /username before the account is created.
04
Tips
Use the batch endpoint for backlogs. If you are cleaning up old comments or importing content, send many texts in one request instead of looping single calls.
Moderate multiple texts in a single request. Useful for processing large volumes of content efficiently.Quoted from supervisor.gg
Send text and images as separate requests. The image field takes base64 data and replaces the text field for that call.
Send text and images in separate requests.Quoted from supervisor.gg
Respect the video limits before you upload. Clips over 10 MB or 60 seconds are refused, and the SDKs check the size on your side so the call fails early.
The SDKs wrap this, and check the size limit locally so an oversized clip fails before it is uploadedQuoted from supervisor.gg
Turn on context analysis when harassment is your main worry. With include_context set, the response tells you whether the message needs its history to judge, which catches the quiet campaigns a single message check misses. It is included with Premium.
"include_context": boolean (optional, default: false, included with Premium)Quoted from supervisor.gg
05
Questions
Which models can I choose from?
Three. Observer is the fast one for high volume, sentinel is balanced and arbiter is the most accurate, each priced per KB.
observer (v2.1): Fast, lightweight model for high-volume applications (~£0.0014/KB)Quoted from supervisor.gg
Does it work with languages other than English?
Yes. It moderates content in more than 100 languages and supports Unicode.
any of 100+ languages, Supervisor catches harmful content in any language. Unicode is also supported.Quoted from supervisor.gg
Can I moderate on behalf of my own users?
Yes. The Platform API adds OAuth2 client credentials, per-user provisioning and delegated moderation.
The Platform API adds OAuth2 client credentials, per-user provisioning, and delegated moderation.Quoted from supervisor.gg