Telegraf Amazon Timestream Output 插件完全指南配置、数据映射与源码级剖析【免费下载链接】telegrafAgent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.项目地址: https://gitcode.com/GitHub_Trending/te/telegraf本篇技术指南以 Telegraf 仓库中 plugins/outputs/timestream/README.md 为骨架结合 timestream.go 源码、sample.conf 配置样例与 timestream_test.go 单元测试系统讲解如何将 Telegraf 采集的指标写入 AWS Amazon Timestream 时序数据库。读完本文你将掌握该插件的完整配置项含义、四种 Influx 到 Timestream 的数据映射模式、凭据加载链、批量写入与错误重试机制并能据此搭建可上线的监控数据落库方案。插件概览outputs.timestream是 Telegraf 官方提供的输出插件负责把 Telegraf 指标metrics写入 AWS 的 Amazon Timestream 托管时序数据库服务。该插件自Telegraf v1.16.0起可用属于cloud、datastore类目支持所有平台all。从源码结构看插件在 timestream.go 中通过outputs.Add(timestream, ...)注册并在 plugins/outputs/all/timestream.go 中完成默认构建时的导入注册。其底层基于 AWS SDK for Go v2 的timestreamwrite服务客户端核心接口WriteClient仅封装了三个操作CreateTable、WriteRecords与DescribeDatabase见 timestream.go并通过WriteFactory变量注入客户端以便单元测试替换 mocktimestream.go。认证机制Authentication插件使用AWS 凭据链credential chain向 Timestream API 端点进行认证按以下顺序尝试若同时指定role_arn与web_identity_token_file则通过 STS 使用Web identity provider凭据若指定role_arn则通过 STS 使用AssumeRole 假定凭据源凭据按后续规则依次评估。注意endpoint_url只作用于 Timestream 服务本身拉取凭据时一律使用 STS 全局端点通过access_key、secret_key、token显式指定凭据通过profile指定共享配置文件中的 profile环境变量AWS_ACCESS_KEY_ID等共享凭据文件Shared Credentials FileEC2 实例角色EC2 Instance Profile。源码实现印证该凭据链在 plugins/common/aws/credentials.go 中实现CredentialConfig.Credentials()首先判断RoleARN是否为空——非空则走configWithAssumeCredentials()内部基于 STS 客户端创建AssumeRoleProvider或WebIdentityRoleProvider否则走configWithRootCredentials()依次叠加 profile、共享凭据文件、静态凭据最后回落到 SDK 默认的环境变量/实例角色链。Timestream 插件通过内嵌common_aws.CredentialConfig复用这套机制timestream.go。配置详解以下为完整配置样例与 sample.conf 一致随后逐项说明# Configuration for sending metrics to Amazon Timestream. [[outputs.timestream]] ## Amazon Region region us-east-1 ## Amazon Credentials ## Credentials are loaded in the following order: ## 1) Web identity provider credentials via STS if role_arn and ## web_identity_token_file are specified ## 2) Assumed credentials via STS if role_arn is specified ## 3) explicit credentials from access_key and secret_key ## 4) shared profile from profile ## 5) environment variables ## 6) shared credentials file ## 7) EC2 Instance Profile #access_key #secret_key #token #role_arn #web_identity_token_file #role_session_name #profile #shared_credential_file ## Endpoint to make request against, the correct endpoint is automatically ## determined and this option should only be set if you wish to override the ## default. ## ex: endpoint_url http://localhost:8000 # endpoint_url ## Timestream database where the metrics will be inserted. ## The database must exist prior to starting Telegraf. database_name yourDatabaseNameHere ## Specifies if the plugin should describe the Timestream database upon ## starting to validate if it has access, necessary permissions, connection, ## etc., as a safety check. If the describe operation fails, the plugin will ## not start and therefore the Telegraf agent will not start. describe_database_on_start false ## Specifies how the data is organized in Timestream. ## Valid values are: single-table, multi-table. ## When mapping_mode is set to single-table, all of the data is stored in a ## single table. When mapping_mode is set to multi-table, the data is ## organized and stored in multiple tables. The default is multi-table. mapping_mode multi-table ## Specifies if the plugin should create the table, if it doesnt exist. create_table_if_not_exists true ## Specifies the Timestream table magnetic store retention period in days. ## Check Timestream documentation for more details. ## NOTE: This property is valid when create_table_if_not_exists true. create_table_magnetic_store_retention_period_in_days 365 ## Specifies the Timestream table memory store retention period in hours. ## Check Timestream documentation for more details. ## NOTE: This property is valid when create_table_if_not_exists true. create_table_memory_store_retention_period_in_hours 24 ## Specifies how the data is written into Timestream. ## Valid values are: true, false ## When use_multi_measure_records is set to true, all of the tags and fields ## are stored as a single row in a Timestream table. ## When use_multi_measure_record is set to false, Timestream stores each field ## in a separate table row, thereby storing the tags multiple times (once for ## each field). The recommended setting is true. The default is false. use_multi_measure_records false ## Specifies the measure_name to use when sending multi-measure records. ## NOTE: This property is valid when use_multi_measure_recordstrue and ## mapping_modemulti-table measure_name_for_multi_measure_records telegraf_measure ## Specifies the name of the table to write data into ## NOTE: This property is valid when mapping_modesingle-table. # single_table_name ## Specifies the name of dimension when all of the data is being stored in a ## single table and the measurement name is transformed into the dimension ## value (see Mapping data from Influx to Timestream for details) ## NOTE: This property is valid when mapping_modesingle-table. # single_table_dimension_name_for_telegraf_measurement_name namespace ## Only valid and optional if create_table_if_not_exists true ## Specifies the Timestream table tags. ## Check Timestream documentation for more details # create_table_tags { foo bar, environment dev} ## Specify the maximum number of parallel go routines to ingest/write data ## If not specified, defaulted to 1 go routines max_write_go_routines 25参数逐项说明配置项类型/取值默认值说明region字符串必填AWS 区域例如us-east-1。access_key/secret_key/token字符串空显式静态凭据优先级低于 STS 假定凭据。role_arn字符串空通过 STS AssumeRole 获取临时凭据若同时配置web_identity_token_file则使用 Web Identity 方式。web_identity_token_file字符串空Web identity token 文件路径与role_arn搭配使用如 EKS IRSA 场景。role_session_name字符串空AssumeRole 会话名便于在 CloudTrail 中溯源。profile字符串空指定共享凭据文件~/.aws/credentials中的 profile 名。shared_credential_file字符串空自定义共享凭据文件路径。endpoint_url字符串空覆盖 Timestream 服务端点仅在你需要指向自定义/本地模拟端点如http://localhost:8000时设置。从 credentials.go 可确认其仅作用于 Timestream 客户端不影响 STS。database_name字符串必填目标数据库名该数据库必须在启动 Telegraf 之前已存在。describe_database_on_startboolfalse启动时调用DescribeDatabase校验权限、连通性等若该操作失败插件不会启动Telegraf agent 也随之启动失败。mapping_modesingle-table/multi-tablemulti-table数据组织方式详见下文映射模式。create_table_if_not_existsbooltrue表不存在时是否自动建表。create_table_magnetic_store_retention_period_in_daysint64365磁存储Magnetic Store保留天数仅当create_table_if_not_existstrue时生效。create_table_memory_store_retention_period_in_hoursint6424内存存储Memory Store保留小时数仅当create_table_if_not_existstrue时生效。use_multi_measure_recordsboolfalsetrue时一条 Telegraf 指标的所有 tag 与 field 以单行 Multi-Measure 记录写入false时每个 field 单独占一行tag 随之重复存储。官方推荐设为true。measure_name_for_multi_measure_records字符串telegraf_measure仅当use_multi_measure_recordstrue且mapping_modemulti-table时有效指定 Multi-Measure 记录的 measure_name。single_table_name字符串空仅mapping_modesingle-table时有效指定写入的单表表名。single_table_dimension_name_for_telegraf_measurement_name字符串namespace仅mapping_modesingle-table时有效指定存放 measurement 名的维度名measurement 名会转换为该维度的值。create_table_tagsmap[string]string空仅当create_table_if_not_existstrue时可选指定建表时的表标签如{ foo bar, environment dev}。max_write_go_routinesint1并发写数据的 Go 协程数上限。源码常量maxWriteRoutinesDefault 1timestream.go表明未配置时默认 1。配置校验规则源码视角在Connect()中timestream.go插件会执行严格的参数校验测试用例TestConnectValidatesConfigParameterstimestream_test.go覆盖了这些约束database_name与mapping_mode为必填项缺失直接报错mapping_mode仅允许single-table与multi-table两个取值single-table 模式下必须配置single_table_name当use_multi_measure_recordsfalse时还必须配置single_table_dimension_name_for_telegraf_measurement_namemeasurement 名需要有个维度去承载此时若又配置了measure_name_for_multi_measure_records则报错因为单表 多测点场景下measurement 名直接充当 multi-measure 名multi-table 模式下禁止配置single_table_name与single_table_dimension_name_for_telegraf_measurement_name当use_multi_measure_recordstrue时必须配置measure_name_for_multi_measure_records若create_table_if_not_existstrue两个保留期参数都必须大于 0max_write_go_routines 0时回落到默认值 1。字段类型与 Unsigned Integers 处理Timestream不支持无符号 64 位整型unsigned int64。convertValue()timestream.go的类型转换规则如下Telegraf 字段类型Timestream MeasureValueType说明int,int8,int16,int32,int64BIGINT直接转十进制字符串。uint,uint8,uint16,uint32BIGINT直接转十进制字符串。uint64BIGINT若值 math.MaxInt64原样输出超过最大有符号 int64 的值会被截断capped为9223372036854775807。float32,float64DOUBLE使用FormatFloat(..., f, -1, ...)输出。boolBOOLEAN输出true/false。stringVARCHAR直接输出。其他类型—被跳过okfalse详见错误处理一节。测试TestBuildMultiMeasuresInSingleAndMultiTableModetimestream_test.go专门验证了uint64(math.MaxUint64)被转换为9223372036854775807的行为。批量写入Batching与并发MultithreadingCommonAttributes 批量优化插件利用 TimestreamWriteRecordsInput.CommonAttributes高效批量写入。核心逻辑在TransformMetrics()timestream.go每条 Telegraf 指标按表名tableName分组聚合——single-table 模式统一落到single_table_namemulti-table 模式以 measurement 名作为表名由于 Telegraf 指标已被 metric 的序列化逻辑按 Name、Tag Keys、Time 归一化相同表下的记录可以共享CommonAttributes空 Record 占位维度与时间放在单条 Record 上从而压缩请求体针对Timestream WriteRecords API 单次调用 100 条记录的上限源码常量maxRecordsPerCall 100timestream.gopartitionRecords()会将超出上限的请求自动拆分为多个不超过 100 条的WriteRecordsInput。测试TestTransformMetricsRequestsAboveLimitAreSplittimestream_test.go验证了 101 条记录会被拆成 100 1 两个请求TestWriteMultiMeasuresSingleTableMode验证了 101 条多测点记录产生 2 次WriteRecords调用。多线程写入README 中说明默认单线程写入但从源码看Write()timestream.go会根据max_write_go_routines启动至多 N 个 worker 协程并发消费写任务 channel任务数少于协程数时取小者。MaxWriteGoRoutinesCount 0时回落默认 1即经典的单协程行为。每个 worker 调用writeToTimestream()执行实际的WriteRecords。错误处理与重试语义ErrorswriteToTimestream()timestream.go按 AWS 异常类型区分处理策略这是理解数据会不会丢的关键字段类型不受支持该字段被丢弃错误写入日志。例如TestTransformMetricsUnsupportedFieldsAreSkipped中time.Time类型的 field 被跳过timestream_test.goThrottlingException 或 InternalServerException5xx错误返回给 TelegrafTelegraf 会把这些指标保留在输出缓冲区中在下一个 flush 周期自动重试写入。测试TestThrottlingErrorIsReturnedToTelegraf验证了这一行为timestream_test.goResourceNotFoundException表不存在若create_table_if_not_existstrue插件先尝试建表createTableAndRetry()timestream.go建表成功则重写一次记录建表失败则记录错误并跳过若create_table_if_not_existsfalse记录被丢弃并输出错误日志RejectedRecordsException逐条输出被拒绝记录的原因与索引reject reason/record index整批记录不重试重试无意义返回nil避免 Telegraf 反复重试。测试TestRejectedRecordsErrorResultsInMetricsBeingSkipped验证该行为timestream_test.go其他 AWS 错误OperationError记录被丢弃并记录日志因为重试这类请求基本不会成功非 AWS 网络错误作为可重试错误返回给 Telegraf。建表本身createTable()timestream.go会携带两个保留期配置并打上create_table_tags若遇到ConflictException并发下别人已建表则视为成功。日志与调试Logging开启 Telegraf 的debug 标志可输出详细日志包括正在写入 Timestream 的记录内容。源码中的日志点包括Constructing Timestream client for multi-table mode客户端构建Describing database ... in region ...与 Describe 结果仅describe_database_on_starttrue时WriteToTimestream - Metrics size: N request size: M time(ms): T每次 flush 的指标数、请求数与耗时统计各类错误日志统一带数据库名与表名便于定位。单元测试Testing插件附带完整单元测试在插件目录下执行go test -v ./plugins/outputs/timestream/...测试文件 timestream_test.go 通过覆盖全局WriteFactory注入 mock 客户端mockTimestreamClient/mockTimestreamErrorClient无需真实 AWS 环境即可验证配置校验TestConnectValidatesConfigParameters多测点记录在单表/多表模式下的生成与写入次数TestWriteMultiMeasuresSingleTableMode、TestWriteMultiMeasuresMultiTableMode超过 100 条上限的请求拆分TestTransformMetricsRequestsAboveLimitAreSplit不同维度、不同时间戳、不同 measurement 的分组写入策略TestTransformMetricsSameDimensionsSameTimestampsAreWrittenTogether、TestTransformMetricsSameDimensionsDifferentTimestampsAreWrittenSeparate、TestTransformMetricsDifferentMetricsAreWrittenToDifferentTablesInMultiTableMapping等错误类型对应的重试/丢弃语义TestThrottlingErrorIsReturnedToTelegraf、TestRejectedRecordsErrorResultsInMetricsBeingSkipped自定义端点TestCustomEndpoint。从 Influx 到 Timestream 的数据映射Mapping默认情况下Telegraf 的 Influx 行协议数据映射到 Timestream 遵循四条规则时间戳→ Timestream 的time字段Tags→ Timestream 的dimensions维度Fields→ Timestream 的measures度量Measurement 名→ Timestream 的表名。时间戳换算由getTimestreamTime()timestream.go实现它会自动选择能无损表达该时间戳的最小粒度TimeUnit——整秒用SECONDS、整毫秒用MILLISECONDS、整微秒用MICROSECONDS否则用NANOSECONDS从而兼顾精度与存储开销。下面以两行 Influx 行协议数据为例完整演示四种映射模式weather,locationus-midwest,seasonsummer temperature82,humidity71 1465839830100400200 airquality,locationus-west no25,pm2516 1465839830100400200其中weather、airquality是 measurement 名location、season是 tagtemperature、humidity、no2、pm25是 field。模式一multi-table use_multi_measure_recordstrue每 measurement 一张表每行多字段插件创建weather、airquality两张表mapping_modemulti-table每行包含该指标的多个字段use_multi_measure_recordstrue表weather内容如下measure_name_for_multi_measure_records即该配置项的实际值timelocationseasonmeasure_nametemperaturehumidity2016-06-13 17:43:50us-midwestsummermeasure_name_for_multi_measure_records8271表airquality内容如下timelocationmeasure_nameno2pm252016-06-13 17:43:50us-westmeasure_name_for_multi_measure_records516模式二multi-table use_multi_measure_recordsfalse每 measurement 一张表每字段一行插件创建weather、airquality两张表mapping_modemulti-table每个字段单独占一行use_multi_measure_recordsfalse表weather内容如下timelocationseasonmeasure_namemeasure_value::bigint2016-06-13 17:43:50us-midwestsummertemperature822016-06-13 17:43:50us-midwestsummerhumidity71表airquality内容如下timelocationmeasure_namemeasure_value::bigint2016-06-13 17:43:50us-westno252016-06-13 17:43:50us-westpm2516模式三single-table use_multi_measure_recordstrue单表每行多字段插件创建名为single_table_name的一张表mapping_modesingle-table每行包含多个字段use_multi_measure_recordstruemeasurement 名作为single_table_dimension_name_for_telegraf_measurement_name维度的值写入timelocationseasonsingle_table_dimension_name_for_telegraf_measurement_namemeasure_nametemperaturehumidityno2pm252016-06-13 17:43:50us-midwestsummerweathermeasure_name_for_multi_measure_records8271nullnull2016-06-13 17:43:50us-westnullairqualitymeasure_name_for_multi_measure_recordsnullnull516注意此模式下源码会忽略配置的measure_name_for_multi_measure_records而直接使用 measurement 名作为 multi-measure 名见 timestream.go 与TestBuildMultiMeasuresInSingleAndMultiTableMode中 single-table 分支的断言。模式四single-table use_multi_measure_recordsfalse单表每字段一行插件创建名为single_table_name的一张表mapping_modesingle-table每个字段单独占一行use_multi_measure_recordsfalse表内容如下以namespace作为承载 measurement 名的维度为例timelocationseasonnamespacemeasure_namemeasure_value::bigint2016-06-13 17:43:50us-midwestsummerweathertemperature822016-06-13 17:43:50us-midwestsummerweatherhumidity712016-06-13 17:43:50us-westNULLairqualityno252016-06-13 17:43:50us-westNULLairqualitypm2516上表中single_table_name、single_table_dimension_name_for_telegraf_measurement_name、measure_name_for_multi_measure_records均代表对应配置项的实际取值。如何选择映射模式多表模式天然贴合 Influx 的 measurement → 表 语义查询隔离性好适合 measurement 集合固定、且希望按业务分表管理的场景单表模式将所有数据聚合到一张表配合single_table_dimension_name_for_telegraf_measurement_name维度区分来源适合表数量敏感、希望统一管理保留策略与权限的场景**Multi-Measure 记录推荐**能显著减少行数、压缩维度重复存储开销适合字段较多的高频指标。通用配置与运行约束该插件同样支持 Telegraf 输出插件的通用配置能力metric 过滤、tag/field 改名、别名、插件顺序等详见 CONFIGURATION.md 与 plugins 通用文档。几点上线前必须确认的前提数据库需预先创建database_name指向的数据库必须在启动前存在表可由插件自动创建create_table_if_not_existstrue数据库则不能IAM 权限插件运行需要timestream:DescribeDatabase、timestream:WriteRecords、timestream:CreateTable后者仅自动建表时需要等权限具体以 AWS IAM 策略为准保留期参数仅当create_table_if_not_existstrue且需要自动建表时才必须大于 0且均需符合 Timestream 服务的取值约束写入失败重试只有 Throttling / 5xx 类可重试错误会触发 Telegraf 缓冲重试其余错误如字段类型非法、RejectedRecords会被丢弃并记日志因此建议开启 debug 日志观察写入健康度。【免费下载链接】telegrafAgent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.项目地址: https://gitcode.com/GitHub_Trending/te/telegraf创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Telegraf Amazon Timestream Output 插件完全指南：配置、数据映射与源码级剖析
Telegraf Amazon Timestream Output 插件完全指南配置、数据映射与源码级剖析【免费下载链接】telegrafAgent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.项目地址: https://gitcode.com/GitHub_Trending/te/telegraf本篇技术指南以 Telegraf 仓库中 plugins/outputs/timestream/README.md 为骨架结合 timestream.go 源码、sample.conf 配置样例与 timestream_test.go 单元测试系统讲解如何将 Telegraf 采集的指标写入 AWS Amazon Timestream 时序数据库。读完本文你将掌握该插件的完整配置项含义、四种 Influx 到 Timestream 的数据映射模式、凭据加载链、批量写入与错误重试机制并能据此搭建可上线的监控数据落库方案。插件概览outputs.timestream是 Telegraf 官方提供的输出插件负责把 Telegraf 指标metrics写入 AWS 的 Amazon Timestream 托管时序数据库服务。该插件自Telegraf v1.16.0起可用属于cloud、datastore类目支持所有平台all。从源码结构看插件在 timestream.go 中通过outputs.Add(timestream, ...)注册并在 plugins/outputs/all/timestream.go 中完成默认构建时的导入注册。其底层基于 AWS SDK for Go v2 的timestreamwrite服务客户端核心接口WriteClient仅封装了三个操作CreateTable、WriteRecords与DescribeDatabase见 timestream.go并通过WriteFactory变量注入客户端以便单元测试替换 mocktimestream.go。认证机制Authentication插件使用AWS 凭据链credential chain向 Timestream API 端点进行认证按以下顺序尝试若同时指定role_arn与web_identity_token_file则通过 STS 使用Web identity provider凭据若指定role_arn则通过 STS 使用AssumeRole 假定凭据源凭据按后续规则依次评估。注意endpoint_url只作用于 Timestream 服务本身拉取凭据时一律使用 STS 全局端点通过access_key、secret_key、token显式指定凭据通过profile指定共享配置文件中的 profile环境变量AWS_ACCESS_KEY_ID等共享凭据文件Shared Credentials FileEC2 实例角色EC2 Instance Profile。源码实现印证该凭据链在 plugins/common/aws/credentials.go 中实现CredentialConfig.Credentials()首先判断RoleARN是否为空——非空则走configWithAssumeCredentials()内部基于 STS 客户端创建AssumeRoleProvider或WebIdentityRoleProvider否则走configWithRootCredentials()依次叠加 profile、共享凭据文件、静态凭据最后回落到 SDK 默认的环境变量/实例角色链。Timestream 插件通过内嵌common_aws.CredentialConfig复用这套机制timestream.go。配置详解以下为完整配置样例与 sample.conf 一致随后逐项说明# Configuration for sending metrics to Amazon Timestream. [[outputs.timestream]] ## Amazon Region region us-east-1 ## Amazon Credentials ## Credentials are loaded in the following order: ## 1) Web identity provider credentials via STS if role_arn and ## web_identity_token_file are specified ## 2) Assumed credentials via STS if role_arn is specified ## 3) explicit credentials from access_key and secret_key ## 4) shared profile from profile ## 5) environment variables ## 6) shared credentials file ## 7) EC2 Instance Profile #access_key #secret_key #token #role_arn #web_identity_token_file #role_session_name #profile #shared_credential_file ## Endpoint to make request against, the correct endpoint is automatically ## determined and this option should only be set if you wish to override the ## default. ## ex: endpoint_url http://localhost:8000 # endpoint_url ## Timestream database where the metrics will be inserted. ## The database must exist prior to starting Telegraf. database_name yourDatabaseNameHere ## Specifies if the plugin should describe the Timestream database upon ## starting to validate if it has access, necessary permissions, connection, ## etc., as a safety check. If the describe operation fails, the plugin will ## not start and therefore the Telegraf agent will not start. describe_database_on_start false ## Specifies how the data is organized in Timestream. ## Valid values are: single-table, multi-table. ## When mapping_mode is set to single-table, all of the data is stored in a ## single table. When mapping_mode is set to multi-table, the data is ## organized and stored in multiple tables. The default is multi-table. mapping_mode multi-table ## Specifies if the plugin should create the table, if it doesnt exist. create_table_if_not_exists true ## Specifies the Timestream table magnetic store retention period in days. ## Check Timestream documentation for more details. ## NOTE: This property is valid when create_table_if_not_exists true. create_table_magnetic_store_retention_period_in_days 365 ## Specifies the Timestream table memory store retention period in hours. ## Check Timestream documentation for more details. ## NOTE: This property is valid when create_table_if_not_exists true. create_table_memory_store_retention_period_in_hours 24 ## Specifies how the data is written into Timestream. ## Valid values are: true, false ## When use_multi_measure_records is set to true, all of the tags and fields ## are stored as a single row in a Timestream table. ## When use_multi_measure_record is set to false, Timestream stores each field ## in a separate table row, thereby storing the tags multiple times (once for ## each field). The recommended setting is true. The default is false. use_multi_measure_records false ## Specifies the measure_name to use when sending multi-measure records. ## NOTE: This property is valid when use_multi_measure_recordstrue and ## mapping_modemulti-table measure_name_for_multi_measure_records telegraf_measure ## Specifies the name of the table to write data into ## NOTE: This property is valid when mapping_modesingle-table. # single_table_name ## Specifies the name of dimension when all of the data is being stored in a ## single table and the measurement name is transformed into the dimension ## value (see Mapping data from Influx to Timestream for details) ## NOTE: This property is valid when mapping_modesingle-table. # single_table_dimension_name_for_telegraf_measurement_name namespace ## Only valid and optional if create_table_if_not_exists true ## Specifies the Timestream table tags. ## Check Timestream documentation for more details # create_table_tags { foo bar, environment dev} ## Specify the maximum number of parallel go routines to ingest/write data ## If not specified, defaulted to 1 go routines max_write_go_routines 25参数逐项说明配置项类型/取值默认值说明region字符串必填AWS 区域例如us-east-1。access_key/secret_key/token字符串空显式静态凭据优先级低于 STS 假定凭据。role_arn字符串空通过 STS AssumeRole 获取临时凭据若同时配置web_identity_token_file则使用 Web Identity 方式。web_identity_token_file字符串空Web identity token 文件路径与role_arn搭配使用如 EKS IRSA 场景。role_session_name字符串空AssumeRole 会话名便于在 CloudTrail 中溯源。profile字符串空指定共享凭据文件~/.aws/credentials中的 profile 名。shared_credential_file字符串空自定义共享凭据文件路径。endpoint_url字符串空覆盖 Timestream 服务端点仅在你需要指向自定义/本地模拟端点如http://localhost:8000时设置。从 credentials.go 可确认其仅作用于 Timestream 客户端不影响 STS。database_name字符串必填目标数据库名该数据库必须在启动 Telegraf 之前已存在。describe_database_on_startboolfalse启动时调用DescribeDatabase校验权限、连通性等若该操作失败插件不会启动Telegraf agent 也随之启动失败。mapping_modesingle-table/multi-tablemulti-table数据组织方式详见下文映射模式。create_table_if_not_existsbooltrue表不存在时是否自动建表。create_table_magnetic_store_retention_period_in_daysint64365磁存储Magnetic Store保留天数仅当create_table_if_not_existstrue时生效。create_table_memory_store_retention_period_in_hoursint6424内存存储Memory Store保留小时数仅当create_table_if_not_existstrue时生效。use_multi_measure_recordsboolfalsetrue时一条 Telegraf 指标的所有 tag 与 field 以单行 Multi-Measure 记录写入false时每个 field 单独占一行tag 随之重复存储。官方推荐设为true。measure_name_for_multi_measure_records字符串telegraf_measure仅当use_multi_measure_recordstrue且mapping_modemulti-table时有效指定 Multi-Measure 记录的 measure_name。single_table_name字符串空仅mapping_modesingle-table时有效指定写入的单表表名。single_table_dimension_name_for_telegraf_measurement_name字符串namespace仅mapping_modesingle-table时有效指定存放 measurement 名的维度名measurement 名会转换为该维度的值。create_table_tagsmap[string]string空仅当create_table_if_not_existstrue时可选指定建表时的表标签如{ foo bar, environment dev}。max_write_go_routinesint1并发写数据的 Go 协程数上限。源码常量maxWriteRoutinesDefault 1timestream.go表明未配置时默认 1。配置校验规则源码视角在Connect()中timestream.go插件会执行严格的参数校验测试用例TestConnectValidatesConfigParameterstimestream_test.go覆盖了这些约束database_name与mapping_mode为必填项缺失直接报错mapping_mode仅允许single-table与multi-table两个取值single-table 模式下必须配置single_table_name当use_multi_measure_recordsfalse时还必须配置single_table_dimension_name_for_telegraf_measurement_namemeasurement 名需要有个维度去承载此时若又配置了measure_name_for_multi_measure_records则报错因为单表 多测点场景下measurement 名直接充当 multi-measure 名multi-table 模式下禁止配置single_table_name与single_table_dimension_name_for_telegraf_measurement_name当use_multi_measure_recordstrue时必须配置measure_name_for_multi_measure_records若create_table_if_not_existstrue两个保留期参数都必须大于 0max_write_go_routines 0时回落到默认值 1。字段类型与 Unsigned Integers 处理Timestream不支持无符号 64 位整型unsigned int64。convertValue()timestream.go的类型转换规则如下Telegraf 字段类型Timestream MeasureValueType说明int,int8,int16,int32,int64BIGINT直接转十进制字符串。uint,uint8,uint16,uint32BIGINT直接转十进制字符串。uint64BIGINT若值 math.MaxInt64原样输出超过最大有符号 int64 的值会被截断capped为9223372036854775807。float32,float64DOUBLE使用FormatFloat(..., f, -1, ...)输出。boolBOOLEAN输出true/false。stringVARCHAR直接输出。其他类型—被跳过okfalse详见错误处理一节。测试TestBuildMultiMeasuresInSingleAndMultiTableModetimestream_test.go专门验证了uint64(math.MaxUint64)被转换为9223372036854775807的行为。批量写入Batching与并发MultithreadingCommonAttributes 批量优化插件利用 TimestreamWriteRecordsInput.CommonAttributes高效批量写入。核心逻辑在TransformMetrics()timestream.go每条 Telegraf 指标按表名tableName分组聚合——single-table 模式统一落到single_table_namemulti-table 模式以 measurement 名作为表名由于 Telegraf 指标已被 metric 的序列化逻辑按 Name、Tag Keys、Time 归一化相同表下的记录可以共享CommonAttributes空 Record 占位维度与时间放在单条 Record 上从而压缩请求体针对Timestream WriteRecords API 单次调用 100 条记录的上限源码常量maxRecordsPerCall 100timestream.gopartitionRecords()会将超出上限的请求自动拆分为多个不超过 100 条的WriteRecordsInput。测试TestTransformMetricsRequestsAboveLimitAreSplittimestream_test.go验证了 101 条记录会被拆成 100 1 两个请求TestWriteMultiMeasuresSingleTableMode验证了 101 条多测点记录产生 2 次WriteRecords调用。多线程写入README 中说明默认单线程写入但从源码看Write()timestream.go会根据max_write_go_routines启动至多 N 个 worker 协程并发消费写任务 channel任务数少于协程数时取小者。MaxWriteGoRoutinesCount 0时回落默认 1即经典的单协程行为。每个 worker 调用writeToTimestream()执行实际的WriteRecords。错误处理与重试语义ErrorswriteToTimestream()timestream.go按 AWS 异常类型区分处理策略这是理解数据会不会丢的关键字段类型不受支持该字段被丢弃错误写入日志。例如TestTransformMetricsUnsupportedFieldsAreSkipped中time.Time类型的 field 被跳过timestream_test.goThrottlingException 或 InternalServerException5xx错误返回给 TelegrafTelegraf 会把这些指标保留在输出缓冲区中在下一个 flush 周期自动重试写入。测试TestThrottlingErrorIsReturnedToTelegraf验证了这一行为timestream_test.goResourceNotFoundException表不存在若create_table_if_not_existstrue插件先尝试建表createTableAndRetry()timestream.go建表成功则重写一次记录建表失败则记录错误并跳过若create_table_if_not_existsfalse记录被丢弃并输出错误日志RejectedRecordsException逐条输出被拒绝记录的原因与索引reject reason/record index整批记录不重试重试无意义返回nil避免 Telegraf 反复重试。测试TestRejectedRecordsErrorResultsInMetricsBeingSkipped验证该行为timestream_test.go其他 AWS 错误OperationError记录被丢弃并记录日志因为重试这类请求基本不会成功非 AWS 网络错误作为可重试错误返回给 Telegraf。建表本身createTable()timestream.go会携带两个保留期配置并打上create_table_tags若遇到ConflictException并发下别人已建表则视为成功。日志与调试Logging开启 Telegraf 的debug 标志可输出详细日志包括正在写入 Timestream 的记录内容。源码中的日志点包括Constructing Timestream client for multi-table mode客户端构建Describing database ... in region ...与 Describe 结果仅describe_database_on_starttrue时WriteToTimestream - Metrics size: N request size: M time(ms): T每次 flush 的指标数、请求数与耗时统计各类错误日志统一带数据库名与表名便于定位。单元测试Testing插件附带完整单元测试在插件目录下执行go test -v ./plugins/outputs/timestream/...测试文件 timestream_test.go 通过覆盖全局WriteFactory注入 mock 客户端mockTimestreamClient/mockTimestreamErrorClient无需真实 AWS 环境即可验证配置校验TestConnectValidatesConfigParameters多测点记录在单表/多表模式下的生成与写入次数TestWriteMultiMeasuresSingleTableMode、TestWriteMultiMeasuresMultiTableMode超过 100 条上限的请求拆分TestTransformMetricsRequestsAboveLimitAreSplit不同维度、不同时间戳、不同 measurement 的分组写入策略TestTransformMetricsSameDimensionsSameTimestampsAreWrittenTogether、TestTransformMetricsSameDimensionsDifferentTimestampsAreWrittenSeparate、TestTransformMetricsDifferentMetricsAreWrittenToDifferentTablesInMultiTableMapping等错误类型对应的重试/丢弃语义TestThrottlingErrorIsReturnedToTelegraf、TestRejectedRecordsErrorResultsInMetricsBeingSkipped自定义端点TestCustomEndpoint。从 Influx 到 Timestream 的数据映射Mapping默认情况下Telegraf 的 Influx 行协议数据映射到 Timestream 遵循四条规则时间戳→ Timestream 的time字段Tags→ Timestream 的dimensions维度Fields→ Timestream 的measures度量Measurement 名→ Timestream 的表名。时间戳换算由getTimestreamTime()timestream.go实现它会自动选择能无损表达该时间戳的最小粒度TimeUnit——整秒用SECONDS、整毫秒用MILLISECONDS、整微秒用MICROSECONDS否则用NANOSECONDS从而兼顾精度与存储开销。下面以两行 Influx 行协议数据为例完整演示四种映射模式weather,locationus-midwest,seasonsummer temperature82,humidity71 1465839830100400200 airquality,locationus-west no25,pm2516 1465839830100400200其中weather、airquality是 measurement 名location、season是 tagtemperature、humidity、no2、pm25是 field。模式一multi-table use_multi_measure_recordstrue每 measurement 一张表每行多字段插件创建weather、airquality两张表mapping_modemulti-table每行包含该指标的多个字段use_multi_measure_recordstrue表weather内容如下measure_name_for_multi_measure_records即该配置项的实际值timelocationseasonmeasure_nametemperaturehumidity2016-06-13 17:43:50us-midwestsummermeasure_name_for_multi_measure_records8271表airquality内容如下timelocationmeasure_nameno2pm252016-06-13 17:43:50us-westmeasure_name_for_multi_measure_records516模式二multi-table use_multi_measure_recordsfalse每 measurement 一张表每字段一行插件创建weather、airquality两张表mapping_modemulti-table每个字段单独占一行use_multi_measure_recordsfalse表weather内容如下timelocationseasonmeasure_namemeasure_value::bigint2016-06-13 17:43:50us-midwestsummertemperature822016-06-13 17:43:50us-midwestsummerhumidity71表airquality内容如下timelocationmeasure_namemeasure_value::bigint2016-06-13 17:43:50us-westno252016-06-13 17:43:50us-westpm2516模式三single-table use_multi_measure_recordstrue单表每行多字段插件创建名为single_table_name的一张表mapping_modesingle-table每行包含多个字段use_multi_measure_recordstruemeasurement 名作为single_table_dimension_name_for_telegraf_measurement_name维度的值写入timelocationseasonsingle_table_dimension_name_for_telegraf_measurement_namemeasure_nametemperaturehumidityno2pm252016-06-13 17:43:50us-midwestsummerweathermeasure_name_for_multi_measure_records8271nullnull2016-06-13 17:43:50us-westnullairqualitymeasure_name_for_multi_measure_recordsnullnull516注意此模式下源码会忽略配置的measure_name_for_multi_measure_records而直接使用 measurement 名作为 multi-measure 名见 timestream.go 与TestBuildMultiMeasuresInSingleAndMultiTableMode中 single-table 分支的断言。模式四single-table use_multi_measure_recordsfalse单表每字段一行插件创建名为single_table_name的一张表mapping_modesingle-table每个字段单独占一行use_multi_measure_recordsfalse表内容如下以namespace作为承载 measurement 名的维度为例timelocationseasonnamespacemeasure_namemeasure_value::bigint2016-06-13 17:43:50us-midwestsummerweathertemperature822016-06-13 17:43:50us-midwestsummerweatherhumidity712016-06-13 17:43:50us-westNULLairqualityno252016-06-13 17:43:50us-westNULLairqualitypm2516上表中single_table_name、single_table_dimension_name_for_telegraf_measurement_name、measure_name_for_multi_measure_records均代表对应配置项的实际取值。如何选择映射模式多表模式天然贴合 Influx 的 measurement → 表 语义查询隔离性好适合 measurement 集合固定、且希望按业务分表管理的场景单表模式将所有数据聚合到一张表配合single_table_dimension_name_for_telegraf_measurement_name维度区分来源适合表数量敏感、希望统一管理保留策略与权限的场景**Multi-Measure 记录推荐**能显著减少行数、压缩维度重复存储开销适合字段较多的高频指标。通用配置与运行约束该插件同样支持 Telegraf 输出插件的通用配置能力metric 过滤、tag/field 改名、别名、插件顺序等详见 CONFIGURATION.md 与 plugins 通用文档。几点上线前必须确认的前提数据库需预先创建database_name指向的数据库必须在启动前存在表可由插件自动创建create_table_if_not_existstrue数据库则不能IAM 权限插件运行需要timestream:DescribeDatabase、timestream:WriteRecords、timestream:CreateTable后者仅自动建表时需要等权限具体以 AWS IAM 策略为准保留期参数仅当create_table_if_not_existstrue且需要自动建表时才必须大于 0且均需符合 Timestream 服务的取值约束写入失败重试只有 Throttling / 5xx 类可重试错误会触发 Telegraf 缓冲重试其余错误如字段类型非法、RejectedRecords会被丢弃并记日志因此建议开启 debug 日志观察写入健康度。【免费下载链接】telegrafAgent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.项目地址: https://gitcode.com/GitHub_Trending/te/telegraf创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考