Qwen3.5-9B可观测性：OpenTelemetry接入+请求链路追踪实践

Qwen3.5-9B可观测性OpenTelemetry接入请求链路追踪实践1. 项目概述Qwen3.5-9B是一款拥有90亿参数的开源大语言模型具备以下核心能力强逻辑推理能够处理复杂的逻辑问题和推理任务代码生成支持多种编程语言的代码生成和补全多轮对话保持上下文连贯的长对话能力多模态理解支持图文输入通过Qwen3.5-9B-VL变体长上下文支持最高可处理128K tokens的上下文2. 环境准备2.1 基础环境配置项目运行环境配置如下搭建日期: 2026-03-25 模型: Qwen3.5-9B (多模态) 环境: torch28 (Conda) 端口: 7860 进程管理: Supervisor2.2 项目结构项目目录结构如下/root/qwen3.5-9b/ ├── app.py # 主程序 (Gradio WebUI) ├── start.sh # 启动脚本 ├── service.log # 运行日志 └── history.json # 对话历史记录3. OpenTelemetry接入实践3.1 OpenTelemetry简介OpenTelemetry是一套开源的观测性框架用于收集、处理和导出遥测数据指标、日志和追踪。它为分布式系统提供了端到端的可观测性能力。3.2 接入步骤3.2.1 安装依赖首先安装必要的Python包conda activate torch28 pip install opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation-flask opentelemetry-exporter-otlp3.2.2 修改app.py在应用主文件中添加OpenTelemetry初始化代码from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.instrumentation.flask import FlaskInstrumentor # 初始化OpenTelemetry trace.set_tracer_provider(TracerProvider()) otlp_exporter OTLPSpanExporter(endpointhttp://localhost:4317) trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter)) # 自动检测Flask应用 FlaskInstrumentor().instrument_app(app)3.2.3 配置Jaeger接收数据使用Docker快速启动Jaeger服务docker run -d --name jaeger \ -e COLLECTOR_OTLP_ENABLEDtrue \ -p 16686:16686 \ -p 4317:4317 \ jaegertracing/all-in-one:latest4. 请求链路追踪实现4.1 自定义追踪在关键业务逻辑中添加自定义追踪from opentelemetry import trace tracer trace.get_tracer(__name__) app.route(/generate, methods[POST]) def generate_text(): with tracer.start_as_current_span(generate_text) as span: # 添加请求参数作为span属性 span.set_attribute(model, Qwen3.5-9B) span.set_attribute(max_tokens, request.json.get(max_tokens, 512)) # 业务逻辑处理 result model.generate(request.json[prompt]) # 记录响应信息 span.set_attribute(response_length, len(result)) return jsonify({result: result})4.2 多模态请求追踪对于图片处理请求可以添加更多上下文信息app.route(/analyze_image, methods[POST]) def analyze_image(): with tracer.start_as_current_span(analyze_image) as span: image request.files[image] span.set_attribute(image_format, image.content_type) span.set_attribute(image_size, len(image.read())) # 重置文件指针以便处理 image.seek(0) # 调用模型处理图片 result model.analyze_image(image) return jsonify({result: result})5. 监控指标收集5.1 性能指标收集模型推理的关键性能指标from opentelemetry import metrics from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter # 初始化指标收集 metric_reader PeriodicExportingMetricReader( OTLPMetricExporter(endpointhttp://localhost:4317) ) meter_provider MeterProvider(metric_readers[metric_reader]) metrics.set_meter_provider(meter_provider) meter metrics.get_meter(__name__) request_counter meter.create_counter( qwen.request.count, descriptionTotal number of requests, unit1 ) latency_histogram meter.create_histogram( qwen.request.latency, descriptionRequest latency in milliseconds, unitms )5.2 在请求处理中使用指标import time app.route(/generate, methods[POST]) def generate_text(): start_time time.time() # 记录请求计数 request_counter.add(1, {endpoint: /generate}) try: result model.generate(request.json[prompt]) # 记录延迟 latency (time.time() - start_time) * 1000 latency_histogram.record(latency, {endpoint: /generate}) return jsonify({result: result}) except Exception as e: # 记录错误计数 request_counter.add(1, {endpoint: /generate, status: error}) raise e6. 日志集成6.1 结构化日志配置配置OpenTelemetry日志收集from opentelemetry import _logs from opentelemetry.sdk._logs import LoggingHandler, LoggerProvider from opentelemetry.sdk._logs.export import BatchLogRecordProcessor from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter # 设置日志处理器 logger_provider LoggerProvider() _logs.set_logger_provider(logger_provider) otlp_log_exporter OTLPLogExporter(endpointhttp://localhost:4317) logger_provider.add_log_record_processor( BatchLogRecordProcessor(otlp_log_exporter) ) # 将OpenTelemetry日志处理器添加到Python日志系统 handler LoggingHandler() logging.getLogger().addHandler(handler)6.2 日志与追踪关联在日志中自动添加追踪上下文import logging logger logging.getLogger(__name__) app.route(/generate, methods[POST]) def generate_text(): try: logger.info(Processing generate request, extra{ prompt: request.json[prompt], max_tokens: request.json.get(max_tokens, 512) }) result model.generate(request.json[prompt]) logger.info(Request completed successfully, extra{ response_length: len(result) }) return jsonify({result: result}) except Exception as e: logger.error(Request failed, exc_infoTrue) raise e7. 可视化与分析7.1 Jaeger界面启动Jaeger后可以通过http://localhost:16686访问Jaeger UI查看请求链路追踪数据服务列表选择qwen3.5-9b服务追踪查询按时间范围或特定条件筛选追踪记录详情查看点击单个追踪查看完整的请求链路和耗时分布7.2 Prometheus与Grafana配置Prometheus收集指标数据# prometheus.yml scrape_configs: - job_name: otel-collector static_configs: - targets: [otel-collector:8889]在Grafana中导入OpenTelemetry仪表板模板监控关键指标请求速率响应延迟分布错误率资源使用情况8. 高级配置8.1 采样策略根据业务需求配置采样策略from opentelemetry.sdk.trace.sampling import TraceIdRatioBased # 配置采样率为50% sampler TraceIdRatioBased(0.5) trace.set_tracer_provider(TracerProvider(samplersampler))8.2 资源属性添加服务资源属性便于识别和分类from opentelemetry.sdk.resources import Resource resource Resource.create({ service.name: qwen3.5-9b, service.version: 1.0.0, deployment.environment: production }) trace.set_tracer_provider(TracerProvider(resourceresource)) meter_provider MeterProvider(resourceresource, metric_readers[metric_reader]) logger_provider LoggerProvider(resourceresource)8.3 自定义导出器根据需要配置不同的导出器# 控制台导出器用于调试 from opentelemetry.sdk.trace.export import ConsoleSpanExporter console_exporter ConsoleSpanExporter() trace.get_tracer_provider().add_span_processor( BatchSpanProcessor(console_exporter) ) # Jaeger导出器 from opentelemetry.exporter.jaeger.thrift import JaegerExporter jaeger_exporter JaegerExporter( agent_host_namelocalhost, agent_port6831, ) trace.get_tracer_provider().add_span_processor( BatchSpanProcessor(jaeger_exporter) )9. 总结通过OpenTelemetry接入Qwen3.5-9B模型服务我们实现了端到端追踪完整记录请求处理链路便于排查问题性能监控实时监控模型推理延迟和吞吐量结构化日志日志与追踪关联提供完整上下文可视化分析通过Jaeger、Grafana等工具直观展示系统状态这种可观测性方案不仅适用于Qwen3.5-9B也可以扩展到其他AI模型服务为生产环境部署提供有力保障。获取更多AI镜像想探索更多AI镜像和应用场景访问 CSDN星图镜像广场提供丰富的预置镜像覆盖大模型推理、图像生成、视频生成、模型微调等多个领域支持一键部署。