Live building of agents s02e01 "Does my agent do the right thing?"

Topics: evaluations, benchmarks, eval-driven-development, overevaluation vs underevaluation, human-in-the-loop and monkey-in-the-loop, overseeing strategies, information compression
---
Hey everyone,

after the summer break we will open the second season of our meetup series with looking into problems of controlling and overseeing agents.

What is it about
The agents got very powerful, and in some areas they possess superhuman abilities. At the same time sometimes they do weird and unpredictable things.

• How do we ensure, that the our agent actually does the right thing?
• How do we oversee the work of the agent without babysitting it?
• How do we keep experimenting with the agent architecture while ensuring it keeps running robustly?

We will look into tools and techniques which help you to navigate approaches to overseeing and controlling agents.

For whom it's relevant
The event is most relevant for people who plan using agents on repeated tasks, designing automations or building customer-facing agents for given domain and less relevant for people using agents on arbitrary one-shot tasks.

Hands-on approach
This meetup is hands-on - participants will be building, running and tweaking agents.

• Coding skills are not mandatory - you will be provided with no-code tools for building agents.
• Previous agent building experience will help, but is not mandatory
• Laptops are recommended (Wi-Fi will be provided), but there will also be a way to build the agents with a smartphone.
• If you're software developer or vibecoder - you are welcome to use your own tools, environments and techniques for building agents. All the challenges and environments for the agents with be accessible via API. So you will have the full freedom of building your agents however like, while being able to plug them into the system along with the agents built with the standard tools.
• Be ready to spend approximately €10 on OpenRouter credits (you might not need it, if you have LLM subscriptions already, but better assume you will spend it)

Agent competition (optional)
As usual, at the end we will have a small friendly blitz-competition - we will let the agent built by participants compete in a shared envrionment and then we will compare which agent architectures and configurations were the most effective.

Community

• Linkedin
• Discord