All projects

BACKEND ENGINEERING / AI EVALUATION

EvalForge AI Reliability Platform

EvalForge is a backend-focused platform for testing model behavior, detecting regressions, and measuring run-to-run consistency. It accepts evaluation requests through an Express API, processes jobs in a Redis and BullMQ worker, runs cases against model providers, grades the responses, and stores results for inspection and comparison.

Node.jsExpressJavaScriptPostgreSQLRedisBullMQDockerGemini / OpenAI

01 / PROBLEM

Repeatable checks, not one good demo.

AI responses can vary between runs, providers, and model versions. A single successful response does not show whether behavior is consistent or whether a change caused a regression. EvalForge was built to make evaluation runs inspectable and repeatable by keeping datasets, graders, outputs, and run results together.

02 / SOLUTION

Saved runs and per-case results.

The platform accepts versioned JSON datasets, queues evaluation work, calls a selected provider adapter, grades each response, and records the result and execution details. The dashboard provides recent-run pagination, dataset management, a run inspector, run comparison, and reliability reporting.

Dataset versionsAsync evaluation jobsRule-based gradersRun comparisonReliability campaigns
ComponentResponsibilityImplementation
APIAccepts dataset and evaluation requests, returns run identifiers, and serves reports.Node.js / Express
Data storePersists datasets, versions, runs, per-case outcomes, and traces.PostgreSQL
QueueMoves evaluation work outside the request-response cycle.Redis / BullMQ
WorkerExecutes queued evaluation cases and records completion or error states.Node.js worker
Provider adaptersConnects to deterministic mock responses and live model providers.Mock, Gemini, OpenAI
DashboardDisplays recent runs, datasets, graded cases, model details, and reliability summaries.Web interface

03 / ARCHITECTURE

Runs in the background, API stays fast.

The API creates and tracks work. Redis and BullMQ hold queued jobs, and a separate worker processes the evaluation. PostgreSQL stores the data needed to inspect what was tested and how each case finished.

EvalForge architecture diagram showing the dashboard, Express API, PostgreSQL, Redis and BullMQ worker, model providers, graders, and saved results
Architecture overview. A simplified view of the components documented in the project README and Docker Compose setup.
01DashboardCreates runs and inspects results
02Express APIValidates requests and queues work
03Redis + BullMQHolds background evaluation jobs
04WorkerCalls providers and runs graders
05PostgreSQLStores cases, traces, and reports

04 / EVALUATION FLOW

From dataset to inspectable report.

  1. Prepare a dataset: create or import JSON test cases, then save edits as a new immutable dataset version.
  2. Queue a run: submit an evaluation request. The API returns a run identifier while background processing continues.
  3. Call a provider: the worker executes cases using the Mock provider, Google Gemini, or OpenAI adapter.
  4. Grade the response: configured graders check output behavior and capture tool/function call proposals without executing them.
  5. Persist each case: store pass/fail/error status, response details, traces, actual model information, latency, and token usage when available.
  6. Inspect and compare: review the run, compare it with another run, or start a repeated-run reliability campaign.

05 / GRADING

Explicit graders, visible criteria.

The project includes rule-based graders for non-empty output, required and forbidden keywords, exact-match and expected-output phrases, valid JSON, context-aware abstention, and required or forbidden tool calls. An optional LLM-as-judge can apply a rubric, but its output is treated as a model-generated assessment that needs calibration, not as ground truth.

Deterministic checks

Keyword, exact-match, JSON, and tool-call checks produce inspectable pass or fail results.

Context-aware checks

Abstention checks can verify whether the assistant avoids answering when the supplied context is insufficient.

Optional judge

An LLM judge can score against a rubric, with the limitation that judge results are not ground truth.

Captured traces

Tool/function call proposals are recorded for assessment. EvalForge does not execute those calls.

06 / OBSERVABILITY

Record what actually happened during a run.

EvalForge separates the requested model from the model that actually responded. This matters when a provider fallback is used. Run details include fallback reasons, provider errors, latency, token counts, and cost estimates based on configured rates. Cost figures are estimates from recorded usage and project configuration, not billing statements.

Model identityRequested model and actual responding model are tracked separately.
Fallback visibilityPer-case fallback reasons are recorded so a run is not misread as a single-model benchmark.
Latency and tokensRun and case details include timing and token usage when returned by the provider.
Cost estimateConfigured per-model rates are used, with missing or incomplete rates reported instead of guessed.

07 / VALIDATION

Tested with suites and a live dataset.

The repository README records the following validation snapshot from project development. The figures describe the listed test suite and the specific Policy Grounding dataset, not a general guarantee that a model will always be correct.

69 tests passedAutomated tests across 9 files.
12 / 12 passedLive Policy Grounding evaluation using gemini-3.1-flash-lite in a single-model run.
3 of 3 repeatsReliability campaign recorded 100% on the same 12-case dataset.
0 run-to-run std. dev.Observed across those three repeated runs on this dataset.

How to read these results: Policy Grounding uses a synthetic demonstration policy. The dataset is small, and three repetitions are an encouraging smoke test, not a large statistical sample. Cases and graders should be reviewed before adapting the setup to an organization's real policies.

08 / SCOPE & LIMITATIONS

A portfolio MVP on solid foundations.

EvalForge demonstrates backend patterns such as asynchronous job processing, persistence, dataset versioning, provider adapters, trace capture, and automated regression checks. It is not presented as a hardened public multi-tenant service.

01

Access control is not implemented

Authentication, per-user permissions, run ownership, and multi-tenant isolation are not implemented. Do not expose an unprotected instance with real user data or secrets.

02

Live model output can vary

Graders need representative cases and careful calibration. Fallback may also produce mixed-model runs that should not be treated as a strict single-model benchmark.

03

Judge results need review

The optional LLM judge can be wrong and should be checked against labeled examples before its scores are relied on.

04

Evaluation is not deployment proof

Small datasets and few repetitions provide limited evidence. Broader datasets and operational hardening remain future work.

09 / EVIDENCE

Inspect the dashboard and run-level views.

These screenshots are embedded from the public EvalForge repository. Open the repository for the full project source, setup instructions, automated test workflow, and validation notes. The image files are stored in the repository under docs/screenshots.

Open the screenshot files in GitHub

10 / REPOSITORY

Explore the implementation.

The public repository includes the Express API, worker, database migrations, Docker Compose setup, dashboard, automated tests, screenshots, and project documentation.

Next projectDesignHub Studio