Sklearn 管道与工作流sklearn.pipeline提供了将多个处理步骤串联/并联的工具确保数据泄露防范和代码简洁。 PipelinePipeline— 顺序管道 ⭐fromsklearn.pipelineimportPipeline pipelinePipeline(steps[(scaler,StandardScaler()),(pca,PCA(n_components10)),(classifier,LogisticRegression())],memoryNone,# 缓存中间变换结果路径字符串或 joblib.MemoryverboseFalse)pipeline.fit(X_train,y_train)# 预测y_predpipeline.predict(X_test)y_probpipeline.predict_proba(X_test)y_scorepipeline.decision_function(X_test)# 评分scorepipeline.score(X_test,y_test)# 变换只执行到最后一个 transformerX_transformedpipeline.transform(X)# 逆变换如果所有步骤都支持X_reconstructedpipeline.inverse_transform(X_transformed)# 访问步骤print(pipeline.named_steps[scaler])print(pipeline.named_steps[pca])print(pipeline[0])# 按索引print(pipeline[-1])# 最后一步make_pipeline()— 快捷管道 ⭐无需命名步骤名称自动小写生成。fromsklearn.pipelineimportmake_pipeline pipelinemake_pipeline(StandardScaler(),PCA(n_components10),LogisticRegression())# 步骤名: standardscaler, pca, logisticregressionpipeline.fit(X_train,y_train)️ 超参数访问管道中每个步骤的参数用stepname__paramname格式访问fromsklearn.pipelineimportPipelinefromsklearn.preprocessingimportStandardScalerfromsklearn.svmimportSVC pipelinePipeline([(scaler,StandardScaler()),(svc,SVC())])# 设置参数pipeline.set_params(svc__C10,svc__kernelrbf,svc__gammascale)# 查看参数print(pipeline.get_params())# {memory: None, steps: [...],# scaler: StandardScaler(),# svc: SVC(),# scaler__copy: True,# scaler__with_mean: True,# scaler__with_std: True,# svc__C: 10,# svc__kernel: rbf, ...}# 配合 GridSearchCVfromsklearn.model_selectionimportGridSearchCV param_grid{svc__C:[0.1,1,10,100],svc__gamma:[scale,auto,0.01,0.1],svc__kernel:[rbf,linear]}gridGridSearchCV(pipeline,param_grid,cv5,n_jobs-1)grid.fit(X_train,y_train)print(grid.best_params_)# {svc__C: 10, svc__gamma: 0.1, svc__kernel: rbf} FeatureUnionFeatureUnion— 特征并联 ⭐并行应用多个变换器然后拼接结果。fromsklearn.pipelineimportFeatureUnion unionFeatureUnion(transformer_list[(numeric,StandardScaler()),(text,TfidfVectorizer()),(poly,PolynomialFeatures(degree2))],n_jobs-1,transformer_weightsNone,# 每个变换器的权重verboseFalse)X_combinedunion.fit_transform(X)make_union()— 快捷并联fromsklearn.pipelineimportmake_union unionmake_union(StandardScaler(),PCA(n_components5)) 完整工作流示例1. 数值类别混合数据处理 ⭐fromsklearn.pipelineimportPipelinefromsklearn.composeimportColumnTransformerfromsklearn.preprocessingimportStandardScaler,OneHotEncoderfromsklearn.imputeimportSimpleImputerfromsklearn.ensembleimportRandomForestClassifier# 数值特征管道numeric_pipelinePipeline([(imputer,SimpleImputer(strategymedian)),(scaler,StandardScaler())])# 类别特征管道categorical_pipelinePipeline([(imputer,SimpleImputer(strategymost_frequent)),(encoder,OneHotEncoder(handle_unknownignore))])# 列变换器preprocessorColumnTransformer([(numeric,numeric_pipeline,[age,income,score]),(categorical,categorical_pipeline,[gender,city,education])],remainderdrop)# 完整管道full_pipelinePipeline([(preprocessor,preprocessor),(classifier,RandomForestClassifier(n_estimators100,random_state42))])# 一次性训练评估full_pipeline.fit(X_train,y_train)accuracyfull_pipeline.score(X_test,y_test)2. 文本分类管道fromsklearn.pipelineimportPipelinefromsklearn.feature_extraction.textimportCountVectorizer,TfidfTransformerfromsklearn.naive_bayesimportMultinomialNB text_pipelinePipeline([(vectorizer,CountVectorizer(max_features10000,ngram_range(1,2),stop_wordsenglish)),(tfidf,TfidfTransformer()),(classifier,MultinomialNB(alpha0.1))])text_pipeline.fit(texts_train,labels_train)predictedtext_pipeline.predict(texts_test)3. 复杂异构数据管道fromsklearn.composeimportColumnTransformerfromsklearn.pipelineimportPipeline,FeatureUnionfromsklearn.feature_extraction.textimportTfidfVectorizerfromsklearn.decompositionimportTruncatedSVDfromsklearn.preprocessingimportStandardScaler,OneHotEncoder# 文本特征降维后text_featuresPipeline([(tfidf,TfidfVectorizer(max_features5000)),(svd,TruncatedSVD(n_components50))])# 数值特征多个处理numeric_featuresPipeline([(imputer,SimpleImputer(strategymedian)),(scaler,StandardScaler())])# 类别特征categorical_featuresPipeline([(encoder,OneHotEncoder(handle_unknownignore,sparse_outputFalse))])# 用 ColumnTransformer 组合不同类型的列preprocessorColumnTransformer([(text,text_features,review_text),(numeric,numeric_features,[price,rating]),(categorical,categorical_features,[category])])# 最终管道final_pipelinePipeline([(preprocessor,preprocessor),(classifier,GradientBoostingClassifier(random_state42))])final_pipeline.fit(df_train,df_train[label])4. 目标变换管道fromsklearn.composeimportTransformedTargetRegressorfromsklearn.preprocessingimportPowerTransformerfromsklearn.linear_modelimportRidge# 对目标变量取对数后拟合再逆变换预测modelTransformedTargetRegressor(regressorRidge(alpha1.0),transformerPowerTransformer(methodyeo-johnson),check_inverseTrue)model.fit(X_train,y_train)y_predmodel.predict(X_test)# 自动做逆变换 管道缓存对耗时的中间步骤启用缓存fromjoblibimportMemoryfromsklearn.pipelineimportPipeline# 缓存到磁盘memoryMemory(location./pipeline_cache,verbose0)pipelinePipeline([(slow_step,SomeExpensiveTransformer()),(classifier,LogisticRegression())],memorymemory)# 首次拟合会计算并缓存 slow_step 的结果pipeline.fit(X_train,y_train)# 修改后续步骤参数时不重新计算前面的步骤pipeline.set_params(classifier__C10)pipeline.fit(X_train,y_train)# slow_step 的结果从缓存加载 常用辅助函数check_array()/check_X_y()— 输入验证fromsklearn.utilsimportcheck_array,check_X_yimportnumpyasnp# 验证并转换输入Xcheck_array(X,accept_sparseTrue,dtypenumeric)# 同步验证 X 和 yX,ycheck_X_y(X,y,accept_sparseTrue,dtypenumeric,multi_outputFalse,y_numericTrue)clone()— 复制估计器fromsklearn.baseimportclone estimatorLogisticRegression(C1.0)estimator_cloneclone(estimator)# 深拷贝不复制拟合结果 调试管道# 查看管道结构print(pipeline)# 查看某一步的输出fromsklearn.pipelineimportPipeline# 提取中间结果X_intermediatepipeline.named_steps[preprocessor].transform(X)# 使用 set_output 控制输出类型1.2fromsklearnimportset_config set_config(transform_outputpandas)# 全部 transformer 输出 DataFrame# 或针对单个pipeline.named_steps[scaler].set_output(transformpandas)[[sklearn-总览|← 返回总览]] Python-sklearn-管道
Sklearn 管道与工作流sklearn.pipeline提供了将多个处理步骤串联/并联的工具确保数据泄露防范和代码简洁。 PipelinePipeline— 顺序管道 ⭐fromsklearn.pipelineimportPipeline pipelinePipeline(steps[(scaler,StandardScaler()),(pca,PCA(n_components10)),(classifier,LogisticRegression())],memoryNone,# 缓存中间变换结果路径字符串或 joblib.MemoryverboseFalse)pipeline.fit(X_train,y_train)# 预测y_predpipeline.predict(X_test)y_probpipeline.predict_proba(X_test)y_scorepipeline.decision_function(X_test)# 评分scorepipeline.score(X_test,y_test)# 变换只执行到最后一个 transformerX_transformedpipeline.transform(X)# 逆变换如果所有步骤都支持X_reconstructedpipeline.inverse_transform(X_transformed)# 访问步骤print(pipeline.named_steps[scaler])print(pipeline.named_steps[pca])print(pipeline[0])# 按索引print(pipeline[-1])# 最后一步make_pipeline()— 快捷管道 ⭐无需命名步骤名称自动小写生成。fromsklearn.pipelineimportmake_pipeline pipelinemake_pipeline(StandardScaler(),PCA(n_components10),LogisticRegression())# 步骤名: standardscaler, pca, logisticregressionpipeline.fit(X_train,y_train)️ 超参数访问管道中每个步骤的参数用stepname__paramname格式访问fromsklearn.pipelineimportPipelinefromsklearn.preprocessingimportStandardScalerfromsklearn.svmimportSVC pipelinePipeline([(scaler,StandardScaler()),(svc,SVC())])# 设置参数pipeline.set_params(svc__C10,svc__kernelrbf,svc__gammascale)# 查看参数print(pipeline.get_params())# {memory: None, steps: [...],# scaler: StandardScaler(),# svc: SVC(),# scaler__copy: True,# scaler__with_mean: True,# scaler__with_std: True,# svc__C: 10,# svc__kernel: rbf, ...}# 配合 GridSearchCVfromsklearn.model_selectionimportGridSearchCV param_grid{svc__C:[0.1,1,10,100],svc__gamma:[scale,auto,0.01,0.1],svc__kernel:[rbf,linear]}gridGridSearchCV(pipeline,param_grid,cv5,n_jobs-1)grid.fit(X_train,y_train)print(grid.best_params_)# {svc__C: 10, svc__gamma: 0.1, svc__kernel: rbf} FeatureUnionFeatureUnion— 特征并联 ⭐并行应用多个变换器然后拼接结果。fromsklearn.pipelineimportFeatureUnion unionFeatureUnion(transformer_list[(numeric,StandardScaler()),(text,TfidfVectorizer()),(poly,PolynomialFeatures(degree2))],n_jobs-1,transformer_weightsNone,# 每个变换器的权重verboseFalse)X_combinedunion.fit_transform(X)make_union()— 快捷并联fromsklearn.pipelineimportmake_union unionmake_union(StandardScaler(),PCA(n_components5)) 完整工作流示例1. 数值类别混合数据处理 ⭐fromsklearn.pipelineimportPipelinefromsklearn.composeimportColumnTransformerfromsklearn.preprocessingimportStandardScaler,OneHotEncoderfromsklearn.imputeimportSimpleImputerfromsklearn.ensembleimportRandomForestClassifier# 数值特征管道numeric_pipelinePipeline([(imputer,SimpleImputer(strategymedian)),(scaler,StandardScaler())])# 类别特征管道categorical_pipelinePipeline([(imputer,SimpleImputer(strategymost_frequent)),(encoder,OneHotEncoder(handle_unknownignore))])# 列变换器preprocessorColumnTransformer([(numeric,numeric_pipeline,[age,income,score]),(categorical,categorical_pipeline,[gender,city,education])],remainderdrop)# 完整管道full_pipelinePipeline([(preprocessor,preprocessor),(classifier,RandomForestClassifier(n_estimators100,random_state42))])# 一次性训练评估full_pipeline.fit(X_train,y_train)accuracyfull_pipeline.score(X_test,y_test)2. 文本分类管道fromsklearn.pipelineimportPipelinefromsklearn.feature_extraction.textimportCountVectorizer,TfidfTransformerfromsklearn.naive_bayesimportMultinomialNB text_pipelinePipeline([(vectorizer,CountVectorizer(max_features10000,ngram_range(1,2),stop_wordsenglish)),(tfidf,TfidfTransformer()),(classifier,MultinomialNB(alpha0.1))])text_pipeline.fit(texts_train,labels_train)predictedtext_pipeline.predict(texts_test)3. 复杂异构数据管道fromsklearn.composeimportColumnTransformerfromsklearn.pipelineimportPipeline,FeatureUnionfromsklearn.feature_extraction.textimportTfidfVectorizerfromsklearn.decompositionimportTruncatedSVDfromsklearn.preprocessingimportStandardScaler,OneHotEncoder# 文本特征降维后text_featuresPipeline([(tfidf,TfidfVectorizer(max_features5000)),(svd,TruncatedSVD(n_components50))])# 数值特征多个处理numeric_featuresPipeline([(imputer,SimpleImputer(strategymedian)),(scaler,StandardScaler())])# 类别特征categorical_featuresPipeline([(encoder,OneHotEncoder(handle_unknownignore,sparse_outputFalse))])# 用 ColumnTransformer 组合不同类型的列preprocessorColumnTransformer([(text,text_features,review_text),(numeric,numeric_features,[price,rating]),(categorical,categorical_features,[category])])# 最终管道final_pipelinePipeline([(preprocessor,preprocessor),(classifier,GradientBoostingClassifier(random_state42))])final_pipeline.fit(df_train,df_train[label])4. 目标变换管道fromsklearn.composeimportTransformedTargetRegressorfromsklearn.preprocessingimportPowerTransformerfromsklearn.linear_modelimportRidge# 对目标变量取对数后拟合再逆变换预测modelTransformedTargetRegressor(regressorRidge(alpha1.0),transformerPowerTransformer(methodyeo-johnson),check_inverseTrue)model.fit(X_train,y_train)y_predmodel.predict(X_test)# 自动做逆变换 管道缓存对耗时的中间步骤启用缓存fromjoblibimportMemoryfromsklearn.pipelineimportPipeline# 缓存到磁盘memoryMemory(location./pipeline_cache,verbose0)pipelinePipeline([(slow_step,SomeExpensiveTransformer()),(classifier,LogisticRegression())],memorymemory)# 首次拟合会计算并缓存 slow_step 的结果pipeline.fit(X_train,y_train)# 修改后续步骤参数时不重新计算前面的步骤pipeline.set_params(classifier__C10)pipeline.fit(X_train,y_train)# slow_step 的结果从缓存加载 常用辅助函数check_array()/check_X_y()— 输入验证fromsklearn.utilsimportcheck_array,check_X_yimportnumpyasnp# 验证并转换输入Xcheck_array(X,accept_sparseTrue,dtypenumeric)# 同步验证 X 和 yX,ycheck_X_y(X,y,accept_sparseTrue,dtypenumeric,multi_outputFalse,y_numericTrue)clone()— 复制估计器fromsklearn.baseimportclone estimatorLogisticRegression(C1.0)estimator_cloneclone(estimator)# 深拷贝不复制拟合结果 调试管道# 查看管道结构print(pipeline)# 查看某一步的输出fromsklearn.pipelineimportPipeline# 提取中间结果X_intermediatepipeline.named_steps[preprocessor].transform(X)# 使用 set_output 控制输出类型1.2fromsklearnimportset_config set_config(transform_outputpandas)# 全部 transformer 输出 DataFrame# 或针对单个pipeline.named_steps[scaler].set_output(transformpandas)[[sklearn-总览|← 返回总览]]