๐Ÿ“– Navigation Menu
Chapter 1 • Getting Started

What is Feenion?

Feenion is an open-source, 100% self-hosted observability and debugging platform built specifically for LLM applications, RAG pipelines, and autonomous multi-agent tool loops.

Traditional APMs only capture raw HTTP status codes or generic server errors. When an agent loops infinitely, retrieves irrelevant vector chunks, or hallucinates due to prompt bloat, traditional tools remain blind. Feenion captures the full causal execution graph: exact prompts, completions, tokens, calculated costs, tool arguments, and retrieval scores.

Docker Quickstart

Spin up the complete Feenion server and React dashboard with embedded SQLite WAL storage in one command:

git clone https://github.com/DarshanAguru/feenion.git
cd feenion
docker compose up -d

Open http://localhost:8000 in your browser to view the live dashboard.

Python SDK Setup

Install the official lightweight Python SDK in your application environment:

pip install feenion

Your First 5-Minute Trace

Configure the client and annotate your functions with @trace and span():

from feenion import trace, span, configure

# Point SDK to your local or self-hosted server
configure(server_url="http://localhost:8000")

@trace(name="customer_support_agent", span_type="agent")
def handle_query(user_query: str):
    with span("vector_kb_search", span_type="retrieval", input={"q": user_query}):
        docs = ["Reset password at /settings", "API keys are under Security"]

    with span("llm_synthesis", span_type="llm"):
        return f"Answer for {user_query}: {docs[0]}"

handle_query("How do I reset my password?")
Chapter 2 • Core Architecture

Traces, Spans & Events Data Model

Feenion models execution as a hierarchical Directed Acyclic Graph (DAG):

  • Trace: The top-level root operation representing the end-to-end request (e.g. an API endpoint or agent task).
  • Span: A timed unit of work within a trace (e.g., LLM generation, vector search, tool execution, function call).
  • Event: An instantaneous timestamped milestone attached to a span (e.g. streaming first token received).

Context Propagation

Feenion uses Python's native contextvars.ContextVar to maintain the current active trace and parent span IDs across nested sync and async calls without requiring manual ID passing.

Pluggable Telemetry Exporters

Feenion provides 5 built-in, decoupled exporters for any production or testing setup:

  • AsyncExporter(inner): Non-blocking daemon worker with an in-memory queue flushing batches every 500ms without slowing down request loops.
  • HTTPExporter(endpoint, api_key, project_id): Direct HTTP ingestion transport to the Feenion server with tenant API key authentication.
  • JSONLExporter(filepath): Structured append-only local file logging for air-gapped environments, CI pipelines, and offline trace replay.
  • ConsoleExporter(verbose): Real-time formatted terminal output for local developer interactive debugging.
  • CompositeExporter([exporters]): Broadcasts telemetry to multiple targets simultaneously (e.g. streaming to remote HTTP while saving local JSONL audit files).
from feenion import configure
from feenion.exporters import AsyncExporter, HTTPExporter, CompositeExporter, JSONLExporter

# Composite: Async HTTP upload + local JSONL audit trail
configure(
    exporter=CompositeExporter([
        AsyncExporter(HTTPExporter("http://localhost:8000", api_key="feenion_key", project_id="finance-prod")),
        JSONLExporter("traces_audit.jsonl"),
    ])
)

Multi-Tenant Workspace Routing & Authentication

Telemetry in Feenion is routed strictly using each workspace's unique Workspace ID (obtained from the Settings page):

๐Ÿ” Authentication & Local Hosting

API key is optional for local development on the same PC. When deploying Feenion in production with authentication enabled, provide your workspace's API key.

import feenion
from feenion import trace

# Option 1: Global Workspace Configuration by Workspace ID
feenion.configure(
    server_url="http://localhost:8000",
    workspace_id="60d03b94-82a1-4328-874e-7b5fbfbc4402",
    api_key="fn_live_...",  # Optional for local instance, required for auth
)

# Option 2: Dynamic Per-Agent / Per-Trace Workspace Routing (with Auth)
@trace(
    name="compliance_scanner",
    workspace_id="60d03b94-82a1-4328-874e-7b5fbfbc4402",
    api_key="fn_live_...",
)
def run_compliance():
    # Or attach dynamically inside any function:
    feenion.set_workspace_id("60d03b94-82a1-4328-874e-7b5fbfbc4402")
    feenion.set_api_key("fn_live_...")
    return "completed"
Chapter 3 • Python SDK Deep-Dive

@trace & @async_trace Decorators

Decorate any synchronous or asynchronous Python function to automatically track execution:

from feenion import async_trace

@async_trace(name="async_pipeline", span_type="agent")
async def process_stream(payload: dict):
    await asyncio.sleep(0.05)
    return {"status": "success"}

span() Context Manager

Use the span() context manager for sub-operations:

with span("db_lookup", span_type="tool", input={"user_id": 42}) as s:
    result = db.query(...)
    s.set_output(result)
    s.set_metric("rows_returned", len(result))

Token & Cost Tracking • Multi-Currency Support

Feenion contains built-in pricing tables for standard models (e.g. GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash, DeepSeek). You can also set custom token usage and model overrides in code or via the Web Dashboard:

s.set_tokens(prompt_tokens=512, completion_tokens=128)
s.set_attribute("model", "gpt-4o")

# Override pricing directly in code (per 1M tokens):
from feenion import set_model_pricing
set_model_pricing("my-fine-tuned-model", prompt_per_1m=1.20, completion_per_1m=3.80)
๐Ÿ’ฑ Dashboard Currency & Pricing Editor

In the Web Dashboard under Settings → Currency & FX Display Preferences, you can switch between USD ($), INR (โ‚น), EUR (โ‚ฌ), GBP (ยฃ), CNY (ยฅ), and JPY (ยฅ) with customizable exchange rates, and edit token rates for any model with a live simulation calculator.

Client-Side Sensitive PII & Secret Redaction

Mask sensitive fields in payload dicts before they leave application memory:

from feenion.redaction import Redactor

redactor = Redactor(sensitive_keys={"api_key", "password", "auth_token", "ssn"})
safe_payload = redactor.redact(raw_payload)
Chapter 4 • Framework Auto-Instrumentation

Azure OpenAI & Azure AI Foundry

Wrap Azure OpenAI (openai.AzureOpenAI or LangChain langchain_openai.AzureChatOpenAI) in one call:

from langchain_openai import AzureChatOpenAI
from feenion.integrations import wrap_azure_openai

# Auto-instruments .invoke(), .ainvoke(), tokens, and pricing
llm = wrap_azure_openai(AzureChatOpenAI(
    azure_endpoint="https://my-resource.openai.azure.com/",
    api_key="AZURE_API_KEY",
    azure_deployment="gpt-4o",
    api_version="2024-02-01",
))

response = llm.invoke([("user", "Analyze compliance report #1024")])

OpenAI Auto-Instrumentation

Wrap the official OpenAI client or LangChain ChatOpenAI:

from openai import OpenAI
from feenion.integrations.openai import wrap_openai

client = wrap_openai(OpenAI())

Google Gemini Auto-Instrumentation

Wrap Google Gemini clients (google-genai or google-generativeai) in one line:

from google import genai
from feenion.integrations.gemini import instrument_gemini

client = genai.Client()
instrument_gemini(client)

response = client.models.generate_content(
    model="gemini-2.0-flash",
    contents="Explain vector embeddings in machine learning."
)

Anthropic Claude Auto-Instrumentation

from anthropic import Anthropic
from feenion.integrations.anthropic import instrument_anthropic

client = Anthropic()
instrument_anthropic(client)

LangChain & LangGraph Callbacks

from feenion.integrations.langchain import FeenionCallbackHandler

handler = FeenionCallbackHandler(trace_name="rag_chain")
chain.invoke({"topic": "AI Debugging"}, config={"callbacks": [handler]})

RAG & Vector Retrieval Observability

Instrument retrieval spans to capture similarity scores, chunk IDs, and query text:

with span("chroma_kb_query", span_type="retrieval", input={"query": q, "top_k": 5}) as s:
    chunks = vector_store.similarity_search_with_score(q, k=5)
    s.set_output([{"text": doc.page_content, "score": score} for doc, score in chunks])

Autonomous Agents & Tool Calling

Track multi-turn reasoning loops, tool call arguments, execution outputs, and error recovery:

with span("sql_executor", span_type="tool", input={"sql": query_str}) as s:
    s.set_attribute("tool_name", "database_reader")
    res = db.execute(query_str)
    s.set_output(res)
Chapter 5 • Debugging Workflows

Waterfall Timeline & Critical Path

The Waterfall Timeline view provides an interactive flamegraph of all child spans. Feenion automatically highlights the critical path that contributed most to the overall request duration, allowing you to instantly spot latency bottlenecks.

D3 Execution DAG Mind Map

The Mind Map view renders the full parent-child causality tree of your AI execution. Link distances scale proportionally to latency, and color-coded nodes represent LLMs (purple), retrievals (amber), tools (cyan), and exceptions (rose).

Semantic Error Intelligence

Feenion clusters similar exceptions using semantic error fingerprinting. Click any fingerprint to see the full stack trace, occurrence frequency, affected models, and deep links to sample traces.

Comparative Trace Regression Diff

Open two traces side-by-side to immediately compare durations, token inflations, cost variances, and prompt differences to determine why a regression occurred.

Chapter 6 • Self-Hosting & Operations

Production Docker Deployment

Deploy Feenion as a single all-in-one container using SQLite Write-Ahead Logging:

version: '3.8'
services:
  feenion:
    image: feenion/feenion:latest
    ports:
      - "8000:8000"
    environment:
      - FEENION_DATABASE_URL=sqlite:////app/data/feenion.db
      - FEENION_RETENTION_DAYS=30
    volumes:
      - feenion_data:/app/data
    restart: unless-stopped

Environment Variables Reference

Variable Default Description
FEENION_DATABASE_URL sqlite:////app/data/feenion.db Database connection string (SQLite WAL or PostgreSQL)
FEENION_REDIS_URL "" Optional Redis queue URL for distributed workers
FEENION_LOG_LEVEL INFO Logging level (DEBUG, INFO, WARNING, ERROR)
FEENION_RETENTION_DAYS 30 Automatic retention period for traces and spans
Chapter 6 • Multi-Tenancy & Workspaces

Workspaces & Isolation Architecture

Feenion provides built-in multi-tenant workspace isolation, allowing teams to separate telemetry data across environments (e.g. Production, Staging, Dev) or distinct AI microservices under a single self-hosted server.

The Workspace Lifecycle Flow

1 Create Workspace

Create a workspace (e.g. prod-agent-service) via the Web Dashboard Settings or POST /api/v1/projects.

2 Generate API Key

The server generates a unique cryptographic ingestion API key (fn_...) stored as a SHA-256 hash.

3 Instrument App

Pass the API key to configure(api_key="fn_...") in your Python backend or HTTP exporter.

4 Isolated Queries

Switch workspaces instantly in the dashboard top navigation bar to view isolated traces, analytics, and costs.

Configuring Python SDK with Workspace Key

from feenion import configure, trace, span

# Bind all telemetry from this process to your dedicated workspace:
configure(
    server_url="http://localhost:8000",
    api_key="fn_abc123yourSecretKeyHere...",
)

@trace(name="customer_support_pipeline", span_type="agent")
def run_task(query: str):
    with span("kb_lookup", span_type="retrieval"):
        pass

๐Ÿ”’ Zero Data Bleed

Traces, span error stacks, token costs, and LLM prompts are partitioned by workspace ID at the database layer. One workspace cannot access telemetry from another.

๐Ÿ—‘๏ธ Protected Cascade Deletion

Deleting a workspace requires entering the exact confirmation text delete workspace and cascades through all associated spans, events, and API keys. The system prevents deleting the last remaining workspace.

Server API • Complete Reference

REST & WebSocket API Reference

The Feenion server exposes high-performance asynchronous endpoints for ingestion, query filtering, deep analytics, project isolation, and live WebSocket telemetry push.

๐Ÿ“ฅ 1. Ingestion & Live Streaming

POST /api/v1/traces
Batch Ingestion

Ingests an array of execution traces, child spans, token metrics, and execution events with asynchronous persistence. Supports Gzip compression via Content-Encoding: gzip.

Headers: Content-Type: application/jsonContent-Encoding: gzip (optional) • X-Feenion-Api-Key: <key> (optional)
Response (200): {"accepted": 10, "schema_version": "1.0"}
WS /api/v1/ws/telemetry
Real-time Stream

Bi-directional WebSocket connection for live telemetry streaming. The server broadcasts events (trace_ingested, trace_deleted, data_cleared) directly to connected dashboard sessions.

๐Ÿ” 2. Traces & Spans Exploration

GET /api/v1/traces
Filter & Paginate

List and search traces with multi-dimensional filtering, free-text prompt search, and sorting.

Query Parameters:
  • time_window: 15m, 1h, 6h, 24h, 7d, 30d, all
  • status: all, ok, error
  • environment: all, production, staging, development
  • span_type: all, llm, retrieval, tool, agent
  • model: Filter by LLM model name (e.g. gemini-2.0-flash, gpt-4o)
  • search: Full-text substring search across trace names, prompt inputs, model names, and IDs
  • min_duration_ms / max_duration_ms: Filter by execution latency
  • sort_by: newest, oldest, slowest, fastest, most_tokens, most_cost, most_spans, error
  • limit (1-500, default 100) • offset (default 0)
GET /api/v1/traces/{trace_id}
Full Trace Details

Retrieves the complete causality tree for a single trace, including child spans, prompt inputs, completions, error stack traces, token breakdowns, and calculated dollar costs.

GET /api/v1/traces/{trace_id}/spans
Span Waterfall

Returns the ordered flat list of child spans with start timestamps, latencies, and parent IDs for timeline and flamegraph visualization.

GET /api/v1/count/traces
Counter

Returns fast count of total ingested traces: {"count": 1420}.

๐Ÿ“Š 3. Observability & Deep Analytics

GET /api/v1/analytics/overview
Health & Regression

Returns system health score (0-100), latency percentiles (p50, p75, p90, p95, p99), KPI deltas vs previous window, time breakdown (LLM vs Retrieval vs Tools vs Other), and automated "What Changed?" regression analysis.

GET /api/v1/analytics/models
Model Breakdown

Aggregates performance across LLM providers and models: requests, prompt & completion tokens, total spend, p50/p95 latency, error rates, and average cost per query.

GET /api/v1/analytics/tools
MCP & Tool Calling

Monitors tool invocations, execution failures, p50/p95 latency per tool, and latest invocation timestamps.

GET /api/v1/analytics/retrieval
Vector RAG Metrics

Analyzes vector search and retrieval pipelines: average document chunks retrieved, average cosine/relevance scores, slow retrievals (>1s), and empty result alerts.

GET /api/v1/analytics/agents
Multi-Step Agent Loops

Evaluates autonomous agent execution runs: average step count, loop candidate detection (repeated tool calling >3 times), total tokens, and goal completion rates.

GET /api/v1/errors
Error Fingerprinting

Groups exceptions into unique semantic fingerprints with stack traces, first seen / latest occurrence, occurrence counts, and list of affected model names and trace IDs.

๐Ÿข 4. Multi-Tenant Workspaces & API Keys

GET /api/v1/projects
List Projects

Lists all active workspaces and their creation dates.

POST /api/v1/projects
Create Project & Key

Creates a new workspace and generates a raw API key (feenion_live_...) for SDK telemetry ingestion.

Payload: {"name": "production-llm-service"}
DELETE /api/v1/projects/{project_id}
Cascade Delete

Deletes a workspace and cascade-deletes all its API keys, traces, spans, and telemetry logs. (Safety rule: cannot delete the only remaining workspace).

๐Ÿ›ก๏ธ 5. Admin & Data Maintenance

DELETE / POST /api/v1/admin/traces • /api/v1/admin/traces/purge
Full Purge

Permanently purges all database and in-memory telemetry data. Requires explicit confirmation payload {"confirmation": "delete everything"}.

POST /api/v1/admin/traces/batch-delete
Batch Delete

Deletes multiple selected traces by IDs with {"trace_ids": ["..."], "confirmation": "delete selected"}.

DELETE /api/v1/admin/traces/{trace_id}
Single Delete

Deletes a specific trace and its child spans and events, broadcasting a trace_deleted event over WebSocket.

โš™๏ธ 6. System Probes & Health

GET /health

Liveness probe for load balancers and container orchestrators: {"status": "ok"}.

GET /ready

Readiness check verifying database connection and background worker thread status.

Troubleshooting & FAQ

Does Feenion slow down my application?

No. The Python SDK pushes spans to an in-memory queue, and a background thread flushes them asynchronously. Spans execute with zero network delay on user requests.

Can I use Feenion with Ollama or local LLMs?

Yes. Simply use the span() context manager or OpenAI client wrapper pointed to your local Ollama/vLLM base URL.