Cards / needs / LLM observability and evaluation

LLM observability and evaluation

Self-hosted alternatives to LangSmith, Helicone, Braintrust and Weights & Biases Weave.

What people say

• see what my AI app is doing
• trace prompts and responses
• track LLM costs per user
• evaluate prompts and models
• keep prompt logs private

Why people self-host it

People leave LangSmith, Helicone, Braintrust, Weights & Biases Weave for it. Why, strongest first: privacy, cost, compliance.

Does it suit a machine of your own?

Good fit prompt logs hold users' data, which is the strongest reason to keep them on a machine you control

Questions that decide it

• roughly how many requests a day?
• do prompts contain users' personal data?

How to publish it

Usually

the dashboard on org; the ingestion API on api-key

When the data is private or sensitive

the same, with a retention limit on stored prompts

When the data is regulated

prompt logs inherit the data's rules: if users send health or client data, the logs are that data

The machine

a database that grows with traffic; disk for logs

Cards for it

No card yet. GitHub repositories tagged llmops or llm-observability, and tensorchord/Awesome-LLMOps (for example Langfuse, Arize Phoenix, Opik) An agent with the edgible-cards skill can shortlist apps and build one, or you can ask for one.