
Vector 的 vector 源与汇 TLS 加密实例间安全数据中继的完整配置指南【免费下载链接】vectorA high-performance observability data pipeline.项目地址: https://gitcode.com/GitHub_Trending/vect/vectorVector 的vector源source与vector汇sink是实例间互传观测数据的原生通道二者均支持通过tls.*选项启用 TLS 协议从而在 Vector 实例之间安全地转发日志、指标与追踪数据。本文围绕这一功能展开先讲清vector源/汇的 TLS 配置项全集与默认值再结合当前仓库的源码剖析 TLS 设置如何被解析、校验并注入到 gRPC 服务端与客户端连接器中最后给出一套可直接复用的源-汇对接配置示例。功能定位为什么需要 vector 源/汇的 TLS 支持vector源/汇解决的问题是多实例级联cascading架构下游的 Vector 聚合器aggregator通过vector源监听端口接收上游数据上游的 Vector 节点agent通过vector汇把数据批量推送到下游。这一通道在 0.9.0 版本中加入了 TLS 协议支持对应仓库中的历史特性文档 website/content/en/highlights/2020-03-11-vector-sink-tls.md核心提示是查看tls.*系列选项。在 TLS 之前实例间流量是明文的启用tls.*后两端之间的 gRPCHTTP/2连接被 TLS 加密并可做证书校验与主机名校验避免数据在跨主机传输过程中被窃听或篡改。vector 源的 TLS 配置vector源的完整配置结构定义在 src/sources/vector/mod.rspub struct VectorConfig { version: OptionVectorConfigVersion, /// The socket address to listen for connections on. It _must_ include a port. pub address: SocketAddr, #[serde(default)] tls: OptionTlsEnableableConfig, #[serde(default, deserialize_with bool_or_struct)] acknowledgements: SourceAcknowledgementsConfig, #[serde(default)] keepalive: GrpcKeepaliveConfig, ... }关键配置项选项说明address监听地址必须包含端口默认0.0.0.0:6000tls.enabled是否启用 TLS启用后必须提供身份证书tls.crt_file与tls.key_filetls.ca_file用于校验客户端证书的 CA 文件DER/PEM或内联 PEM 字符串tls.crt_file/tls.key_file服务端身份证书与私钥DER、PEM 或 PKCS#12tls.verify_certificate/tls.verify_hostname证书与主机名校验开关tls.alpn_protocols支持的 ALPN 协议列表tls.server_name出站 SNI 服务端名源场景下一般用不到acknowledgements是否启用端到端确认布尔或结构化配置keepalivegRPC keepalive 参数如max_connection_age_secs一个启用 TLS 的vector源示例sources: vector_in: type: vector address: 0.0.0.0:6000 acknowledgements: true tls: enabled: true crt_file: /path/to/host_certificate.crt key_file: /path/to/host_certificate.key # 如需校验客户端证书双向 TLS提供 CA # ca_file: /path/to/certificate_authority.crt源码剖析TLS 设置如何生效在VectorConfig::buildsrc/sources/vector/mod.rs 约 L183-L230中配置先被转换为运行时设置let tls_settings MaybeTlsSettings::from_config(self.tls.as_ref(), true)?; ... let source run_grpc_server_with_routes( self.address, tls_settings, None, builder.routes(), self.keepalive.clone(), cx.shutdown, )注意第二个实参true——它表示按服务端语义解析 TLS 配置。这与源端的另一处细节相呼应服务端的 gRPC 解码上限被显式绑定到全局的解压缩大小上限// Tonic added a default of 4MB in 0.9. Bound this by the global decompressed-size // cap rather than usize::MAX so a single oversized message cannot drive unbounded // allocation on this unauthenticated listener. .max_decoding_message_size(max_decompressed_size_bytes());MaybeTlsSettings::from_config的底子是 lib/vector-core/src/tls/settings.rs 中的TlsConfig::from_options_base(options, for_server)客户端for_server falseverify_certificate与verify_hostname默认为true若显式设为false会打印安全告警this may lead to security vulnerabilities。服务端for_server true两项默认均为false即源端默认不强制校验客户端证书如需 mTLS应显式启用并配置ca_file。证书解析ca_file加载为受信 CA 栈authoritiescrt_filekey_file可选key_pass解锁加密私钥组成identityALPN 协议按声明顺序优先协商server_name用于出站 SNI。证书格式支持CA/证书可为 DER 或 PEM也可内联 PEM 字符串crt_file还支持 PKCS#12此时可不再单独提供key_filekey_file需为 DER 或 PEMPKCS#8。另外源端同时暴露了自定义的health_checkRPC 与标准 gRPC 健康服务tonic_health供下游vector汇在启动时做健康检查——这与 TLS 无直接关系但同属该 gRPC 服务面配置vector汇的 healthcheck 时会用到。vector 汇的 TLS 配置vector汇的配置结构在 src/sinks/vector/config.rs#[configurable_component(sink(vector, Relay observability data to a Vector instance.))] pub struct VectorConfig { /// The downstream Vector address to which to connect. /// Both IP address and hostname are accepted formats. The address _must_ include a port. #[configurable(deprecated This option has been deprecated, use routing.endpoints instead.)] address: OptionString, /// Routing options for sending requests to one or more downstream Vector endpoints. routing: OptionRoutingConfig, /// Compression algorithm for requests. Supports none, gzip, or zstd. compression: VectorCompression, batch: BatchConfigRealtimeEventBasedDefaultBatchSettings, request: TowerRequestConfig, #[serde(default)] tls: OptionTlsEnableableConfig, /// HTTP/2 keepalive settings for the sinks gRPC connections. keepalive: OptionVectorKeepaliveConfig, acknowledgements: AcknowledgementsConfig, }与源端共用同一套tls.*选项但语义不同汇是出站连接因此crt_file/key_file在汇端只在需要向下游证明身份mTLS时才用ca_file用于验证下游 Vector 实例的证书。典型配置sinks: vector_out: type: vector inputs: [my_transform] address: https://aggregator.example.com:6000 # 已弃用新配置请用 routing.endpoints # 或 routing: endpoints: - https://agg-1.example.com:6000 - https://agg-2.example.com:6000 strategy: load_balance # load_balance | failover | failover_primary compression: gzip # none | gzip | zstd batch: max_bytes: 1_000_000 request: retry_backoff_secs: 1 tls: enabled: true ca_file: /path/to/certificate_authority.crt # 双向认证时再补充 # crt_file: /path/to/client_certificate.crt # key_file: /path/to/client_certificate.key关键机制一scheme 省略时的默认协议随 TLS 切换address/routing.endpoints中的端点字符串必须包含端口但 scheme 可以省略。源码中这条规则很明确src/sinks/vector/config.rs 的ValidatedEndpointA scheme-less endpoint defaults tohttp, orhttpswhen TLS is enabled.具体逻辑在with_schemefn with_scheme(self, tls: bool) - crate::ResultUri { let mut uri self.endpoint.as_uri().clone(); if tls self.scheme_defaulted { let mut parts uri.into_parts(); parts.scheme Some(https.parse().unwrap()); uri Uri::from_parts(parts)?; } Ok(uri) }也就是说当你写了tls.enabled true而端点没带 scheme如10.0.0.2:6000汇会自动按https处理反之无 scheme 且未启用 TLS 时默认http并且源码会对这种默认http的行为打印DEPRECATED警告提示未来版本会改为默认https——因此最佳实践是始终显式写明http://或https://。解析上还有一个容易踩的坑被源码注释点明scheme-less 的host:port[/path]若直接用http::Uri解析host 会被误读为 scheme所以汇使用HttpEndpoint::parse_default_http先补默认 scheme 再解析并在validate阶段完成解析、把结果传递给build避免二次解析的不一致。关键机制二TLS 连接器与 HTTP/2 客户端的组装ValidatedVector::build中汇端 TLS 的注入路径是let tls MaybeTlsSettings::from_config(self.tls.as_ref(), false)?; // false 客户端语义 ... let client new_client(tls, cx.proxy(), self.keepalive)?;new_clientsrc/sinks/vector/config.rs 约 L874-L896构建了一个hyper::ClientProxyConnectorHttpsConnectorHttpConnector, BoxBodyfn new_client(tls_settings: MaybeTlsSettings, proxy_config: ProxyConfig, keepalive: OptionVectorKeepaliveConfig) - crate::Resulthyper::ClientProxyConnectorHttpsConnectorHttpConnector, BoxBody { let proxy build_proxy_connector(tls_settings.clone(), proxy_config)?; let mut builder hyper::Client::builder(); builder.http2_only(true); // Keepalive is opt-in: PINGs on idle connections detect dead pooled // connections before reuse. if let Some(keepalive) keepalive { builder.http2_keep_alive_interval(Duration::from_secs(keepalive.interval_secs.get())) .http2_keep_alive_timeout(Duration::from_secs(keepalive.timeout_secs.get())) .http2_keep_alive_while_idle(true); } Ok(builder.build(proxy)) }几个值得注意的实现事实客户端严格http2_only(true)——vector汇只走 gRPC/HTTP/2这与源端的 gRPC 服务面对齐ALPN 协商tls.alpn_protocols如h2在这一链路上保证双方都选到 HTTP/2。TLS 设置被传给build_proxy_connector即使同时配置了proxyTLS 也会正确作用于连接层。keepalive是可选增强启用后对空闲连接发送 HTTP/2 PING默认间隔 60s、超时 20s注释指出 60s 是 gRPC 社区建议的下限避免触发中间代理的too_many_pings策略用于在复用前剔除已死连接保证重试总是落到活连接上。关键机制三健康检查与端点策略汇端的健康检查会调用源端暴露的health_checkRPCsrc/sinks/vector/config.rs 的healthcheck函数并依据routing.strategy决定检查哪些端点load_balance默认启动时检查所有端点运行期用 Tower distributed service 按routing.health配置做健康跟踪与退避探活failover/failover_primary默认只检查第一个端点除非显式设置healthcheck.uri。运行期失败时按策略轮转端点且请求被强制串行化in_flight信号量恒为 1以保持同一时间只有一个活动端点。gRPC 错误码的重试策略同样值得了解VectorGrpcRetryLogic对NotFound、InvalidArgument、AlreadyExists、PermissionDenied、OutOfRange、Unimplemented、Unauthenticated、DataLoss这类不可重试状态码直接失败其余如Unavailable、超时进入重试——这与 TLS 握手失败、连接重置等瞬时网络问题的重试行为直接相关。端到端示例一个 TLS 化的两级 Vector 部署把上面两侧的选项组合起来一个典型的两级部署配置agent 端与 aggregator 端各一份配置文件如下。Aggregator 端接收方vector源启用 TLSsources: upstream: type: vector address: 0.0.0.0:6000 acknowledgements: true tls: enabled: true crt_file: /etc/vector/server.crt # DER/PEM/PKCS#12或内联 PEM key_file: /etc/vector/server.key # DER/PEM(PKCS#8)可用 key_pass 解锁 # 后续处理与外发…… transforms: process: type: remap inputs: [upstream] source: . parse_json!(.message)Agent 端发送方vector汇启用 TLSsources: logs: type: file include: [/var/log/*.log] sinks: upstream: type: vector inputs: [logs] routing: endpoints: - https://aggregator-1:6000 - https://aggregator-2:6000 strategy: load_balance compression: gzip batch: max_bytes: 1_000_000 tls: enabled: true ca_file: /etc/vector/ca.crt # 验证聚合器证书 verify_certificate: true # 客户端默认即 true verify_hostname: true alpn_protocols: [h2]验证手段运行vector validate config可先做配置级校验端点语法、必填项互斥检查如address与routing只能二选一均在 src/sinks/vector/config.rs 的validate_endpoint_options中实现。启动时vector汇默认对端点执行 healthcheckTLS 配置错误证书过期、CA 不匹配、主机名不符会在此阶段以健康检查失败的形式暴露而不是等到发数据时才报错。若把verify_certificate/verify_hostname显式设为false客户端路径会打印安全告警日志——生产环境不应关闭这两项。小结vector源与vector汇共用TlsEnableableConfig定义于 lib/vector-core/src/tls/settings.rs核心选项为enabled、ca_file、crt_file、key_file、key_pass、verify_certificate、verify_hostname、alpn_protocols、server_name。源端服务端语义默认不校验客户端证书启用 TLS 必须提供身份证书汇端客户端语义默认开启证书与主机名校验。汇端省略 scheme 的端点在tls.enabled true时自动按https解释但显式写 scheme 是当前源码明确建议的做法。汇的 gRPC 客户端强制 HTTP/2、可选 keepalive、按策略做端点健康检查与可重试错误退避与源端的 gRPC 服务面含 health RPC 与解码上限保护共同构成一条可审计的加密实例间中继链路。【免费下载链接】vectorA high-performance observability data pipeline.项目地址: https://gitcode.com/GitHub_Trending/vect/vector创作声明:本文部分内容由AI辅助生成(AIGC),仅供参考