Scout: My AI Tool That Watches Your App and Explains the Errors

What is Scout?#

Scout is a backend tool I am building. It is also called ObservaAI in the code.

Scout has one simple job:

1. It reads the metrics of your app from Prometheus.
2. It checks if something is wrong (for example, too many errors, or slow responses).
3. If something is wrong, it finds the part of the code that causes the problem.
4. Then it uses an AI model (like Claude) to explain the problem and suggest a fix.

In short: Scout watches your app, finds the broken part, and tells you how to fix it. All of this happens automatically, without a human.

Why did I build this?#

When an app has a problem in production, a developer must:

• Look at the metrics.
• Find the service with the error.
• Open the code.
• Understand what is wrong.
• Think of a solution.

This takes time. Scout does the first steps for you. It gives the developer a clear report, with the cause of the problem and an idea to solve it.

How does Scout work? (Step by step)#

Every 20 minutes, Scout does this:

1. Get the metrics. Scout connects to Prometheus (or reads the app's /metrics page directly) and gets the numbers: error rate, latency, etc.
2. Detect anomalies. Scout compares the numbers to normal limits. If a number is too high, it is an "anomaly".
3. Find the endpoint. Scout needs to know which part of the code has the problem. It tries different methods, from easy to hard:
   • First, look in the database (fast and simple).
   • Second, read the metric label (fast and simple).
   • Third, search in the project files (still simple, no AI).
   • Fourth, only if the other methods fail, ask an AI model to find the file. This step is slower and costs money, so Scout uses it only as a last option.
4. Create an alert. Scout saves the problem as an alert. If the same problem happens again, Scout just adds a number to the same alert (it does not create a new one every time).
5. Analyze the problem with AI. Scout collects the related code (the controller, the use case, the domain logic, the repository) and sends it to an AI model. The AI reads the code and the metrics, and answers with a diagnosis and a suggestion — for example, "this query is slow because there is no index."
6. Save the result. Scout saves the AI's answer as a report, connected to the alert.

Smart decisions inside Scout#

I want to explain some ideas that make Scout better and safer.

1. Simple methods first, AI only when needed. Most of the time, Scout can find the answer with simple, fast, and free methods (database lookup, text search). The AI model is only used when nothing else works. This saves time and money.

2. No duplicate alerts. If the same problem happens again in 30 minutes, Scout does not create a new report. It knows this is "the same problem" and just updates the counter. This also saves AI calls.

3. Memory of past problems. Scout remembers old problems. It saves a short summary of every analysis in a special database (called a vector database). Next time a similar problem happens, Scout can say: "I saw something like this before, and this was the fix."

4. Protecting sensitive data. Before sending any code to the AI, Scout removes passwords, API keys, tokens, and other secret information. It also blocks attempts to read files outside the project folder. This keeps the code safe.

5. The AI must answer in a fixed format. Scout asks the AI to answer only with a JSON object (a structured format), so the app can read the answer easily and show it in a clean report.

The technology behind Scout#

Here is a short list, in simple words:

• FastAPI (Python) — for the web server. It is fast and works well with many requests at the same time.
• PostgreSQL + pgvector — the database. pgvector also stores the "memory" of past problems, so Scout can search for similar cases.
• APScheduler — a small internal clock that runs the check every 20 minutes.
• Claude / OpenAI / DeepSeek — the AI models. Scout can use different ones, in case one has a problem or is too expensive.
• Hexagonal architecture — a way to organize the code so the "core logic" does not depend on external tools (database, AI, etc.). This makes the project easier to change and to test.

What I learned#

Building Scout is helping me learn:

• How to connect a real application to an AI model in a safe way.
• How to use a vector database for "memory" (this is called RAG — Retrieval-Augmented Generation).
• How to design a system where AI is a last resort, not the first answer — a cheaper and more reliable approach.
• How to protect private data before sending anything to an external AI service.

What is next?#

Scout is still a work in progress. Some ideas for the future:

• Better protection against prompt injection (when someone hides bad instructions inside the code).
• Support for more types of applications, not only PHP.
• A simple dashboard to see all the alerts and reports in one place.

Thanks for reading! This project helps me practice both AI backend engineering and English, two skills I am improving at the same time.