Codefuse DevOps Eval

Evaluates foundation models against 7,486 multiple-choice questions spanning 8 DevOps lifecycle categories (Plan, Code, Build, Test, Release, Deploy, Operate, Monitor) using a three-component architecture: Core Evaluation System orchestrating the pipeline, curated Datasets with dev/test splits for zero-shot and few-shot scenarios, and Evaluation Scripts that load models, build contexts, and calculate metrics. The framework uses configuration files to determine which models and datasets participate in evaluation runs, enabling systematic comparison of model performance across DevOps knowledge domains.

Evaluates models on 2,840 AIOps-specific samples covering log parsing, time series anomaly detection, root cause analysis, time series classification, and time series forecasting. The framework structures these tasks to test model capabilities in operational intelligence scenarios, with separate evaluation metrics tailored to each AIOps task type. Samples are organized into dev and test splits, allowing models to be evaluated in few-shot contexts where exemplars from the dev split inform predictions on test samples.

Provides data preprocessing utilities to transform raw evaluation data into the framework's expected format (dev/test splits with category metadata, proper field naming, token count calculation). The preprocessing API includes functions for parsing raw datasets, normalizing field names, splitting into dev/test, assigning category labels, and calculating statistics. Users can apply preprocessing to custom datasets before integrating them into the framework, ensuring consistency with existing datasets.

Evaluates models on 1,509 Tool Learning samples spanning 59 fields and 239 tool categories, assessing the model's ability to invoke functions and tools correctly. The framework implements specialized evaluation metrics for tool learning that measure whether models select the correct tool, format arguments properly, and chain multiple tool calls in sequence. Tool learning evaluation is integrated into the pipeline with dedicated data format specifications and metric calculations that differ from standard accuracy metrics.

Implements a ModelAndTokenizerLoader base class that provides default implementations for loading models and tokenizers from Hugging Face, with extensibility hooks for custom model architectures. Models are registered in model_conf.json, which maps model identifiers to their loader implementations and configuration parameters. The system allows users to override default loading behavior for specialized models (e.g., quantized models, custom fine-tuned variants) by implementing custom loader subclasses that inherit from ModelAndTokenizerLoader.

Implements a ContextBuilder abstraction that formats evaluation inputs (questions, few-shot exemplars, tool schemas) into model-specific prompt formats. Different models require different context structures (e.g., chat templates, instruction formats, tool-calling schemas), so the framework allows registering custom ContextBuilder implementations per model. The context builder receives raw evaluation data and produces formatted strings or structured inputs that the model's inference engine expects, enabling consistent evaluation across models with heterogeneous input requirements.

Supports both zero-shot evaluation (no exemplars) and few-shot evaluation (with exemplars from dev split) through configuration-driven pipeline control. The framework organizes datasets into dev and test splits, where dev split contains exemplars that can be included in prompts for few-shot scenarios. The evaluation pipeline reads configuration files that specify the evaluation mode, number of exemplars, and exemplar selection strategy, then constructs contexts accordingly and measures model performance under each condition.

Organizes evaluation data into three main dataset categories (DevOps General with 7,486 samples, AIOps with 2,840 samples, Tool Learning with 1,509 samples), each with dev/test splits and category-level metadata. Datasets are structured to enable filtering by DevOps lifecycle stage (Plan, Code, Build, Test, Release, Deploy, Operate, Monitor), AIOps task type (log parsing, anomaly detection, RCA, classification, forecasting), or tool category. The framework provides APIs to load datasets, filter by category, and access both raw samples and preprocessed versions.

Calculates evaluation metrics tailored to each dataset type: accuracy and per-category breakdown for DevOps General, task-specific metrics (precision/recall for anomaly detection, RMSE for forecasting, etc.) for AIOps, and tool-specific metrics (tool selection accuracy, argument binding accuracy, chaining accuracy) for Tool Learning. The framework aggregates metrics at multiple levels (overall, per-category, per-task) and produces detailed reports showing model performance across different dimensions. Metrics are calculated using standard ML evaluation libraries (scikit-learn) with custom aggregation logic for domain-specific metrics.

Provides extension points for integrating custom models and datasets into the evaluation pipeline through well-defined interfaces: custom models are added by implementing ModelAndTokenizerLoader and ContextBuilder subclasses and registering them in model_conf.json; custom datasets are added by implementing data preprocessing logic and organizing samples into the framework's expected format (dev/test splits with category metadata). The framework is designed to be modular, allowing users to extend it without modifying core evaluation code.

Orchestrates the entire evaluation process through configuration files (JSON) that specify which models to evaluate, which datasets to use, evaluation mode (zero-shot/few-shot), and output format. The pipeline reads configuration, loads specified models and datasets, builds contexts, runs inference, calculates metrics, and generates reports—all without requiring code changes. Configuration files decouple evaluation logic from execution parameters, enabling reproducible evaluations and easy parameter sweeps (e.g., evaluating multiple models on multiple datasets).