Sample Usage Guide【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/geFunction DescriptionThis sample uses the BatchMatMulV2 flatten fusion pass as an example to introduce the implementation of a fusion pass that flattens BatchMatMul to MatMul, providing ATC tool offline model compilation verification. The pass is implemented using eager style API and fusion interfaces.Fusion Principle: Flattens the BatchMatMulV2 operation of A[b,m,k] B[k,n] to a MatMulV2 operation of [b*m,k][k,n] through Reshape, then restores to [b,m,n] output shape through Reshape.This sample only covers the educational scenario where A is 3-dimensional, B is 2-dimensional, both inputs have the same dtype of float/float16/bfloat16, no bias/offset_w, offset_x0, and transpose attributes are all false. BatchMatMul/BatchMatMulV2 with batch dimension broadcasting, bias, offset_w, or non-zero offset_x are not within the scope of this sample optimization.Directory Structure├── README.md // C sample documentation ├── src │ ├──batch_matmul_flatten_pass.cpp // Pass implementation file ├── CMakeLists.txt // Build script ├── data │ ├──gen_onnx.py // ONNX export script for ATC offline verification (supports --batch/--m/--k/--n parameters) │ ├──quick_verify.sh // One-click verification script (supports shape and execution count parameters) │ ├──benchmark_model.cpp // Performance test program source code ├── gen_es_api │ ├──CMakeLists.txt // Build script for generating eager style APIEnvironment RequirementsCompiler: GCC 7.3.xPython and its dependency library versions: python3.9, onnxEnvironment Preparation completed.Implementation StepsDefine classBatchMatmulFlattenPassinheriting fromPatternFusionPass.Override 3 functions from base classPatternFusionPass:Patternsdefines matching templates to capture topologies matching the template in the entire graph.pattern-CaptureTensor()captures BatchMatMul nodes for reading attributes and input shapes.MeetRequirementsfilters the matched topologies.Checks that only x1/x2 valid inputs exist, input A is 3-dimensional, input B is 2-dimensional, both inputs have same floating-point dtype, offset_x0, transpose attributes are false.Replacementdefines the replacement part.Constructs ReshapeMatMulV2Reshape replacement graph, choosing Const or dynamically computed reshape target shape based on whether shape is dynamic.UsesInferShapeAndCheckSupportto verify replacement graph correctness.RegisterBatchMatmulFlattenPassas a custom fusion pass with execution phase AfterInferShape.Program CompilationAssume the CANN software package installation directory is INSTALL_PATH, for example/home/HwHiAiUser/Ascend/.Configure environment variables.Run the environment variable script in the software package with the following command:source ${ASCEND_PATH}/set_env.sh${ASCEND_PATH}is the cann path under the CANN software package installation directory. Replace with the actual installation path of the relevant software package, for example${INSTALL_PATH}/cann.Modify the following information in theCMakeLists.txtfile in the current directory according to actual conditions.ASCEND_PATH: You can set the default software package path. If$ASCEND_HOME_PATHis set viaset_env.sh, no modification needed.PASS_SO_DIR: You can set the custom fusion pass dynamic library installation directory name, default ispass_so_dir.target_include_directories: Header files to include. For this sample, no modification needed. For user-developed code, add header files directly below the example when needed. Note: do not delete existing entries. If the network has custom operators, add custom operator prototype definition header files.target_link_libraries: Libraries to link. For this sample, no modification needed. For user-developed code, add link libraries directly below the example when needed. Note: do not delete existing entries.Do not link other so files in the software package, otherwise compatibility issues may occur during future upgrades.Execute sequentially:mkdir build cd build cmake ..After execution, the es_all_build/generated_code directory generated in thebuilddirectory contains header files and source code for ES graph building API.Executemakecommand to compile the custom pass so. After successful compilation, install the dynamic library filelibbatch_matmul_flatten_pass.soto the custom fusion pass directory viamake install. You can add optional parameter-j$(nproc)aftermakefor parallel build tasks.$(nproc)dynamically gets the CPU core count.make -j$(nproc) batch_matmul_flatten_pass make installAfter sample verification is complete, execute the following command to clean the custom pass so installed in the CANN package to avoid affecting subsequent UT/ST:make clean_custom_passProgram ExecutionConfigure environment variables (skip if already executed).Run the environment variable script in the software package with the following command:source ${ASCEND_PATH}/set_env.shReplace${ASCEND_PATH}with the actual installation path of the relevant software package.Use ATC for offline inference.Set environment variables to dump the model graph during compilation:export DUMP_GE_GRAPH1Navigate to thedatadirectory in the current directory and execute the.pyfile to export onnx (the file uses the onnx library, ensure its installed before running):python gen_onnx.pyYou can also specify shape parameters to export models with different shapes:python gen_onnx.py --batch 32 --m 64 --k 512 --n 256After execution, a.onnxformat model file namedmodel.onnxis generated in thedatadirectory.Execute the ATC tool command (for detailed ATC tool instructions, visit Ascend Documentation and search for ATC Offline Model Compilation Tool), replacesoc_versionaccording to your actual environment:atc --model./model.onnx --framework5 --soc_versionxxx --output./model_fusedThe following output appears in the logs:Define pattern for BatchMatmulFlattenPass Define MeetRequirements for BatchMatmulFlattenPass Define replacement for BatchMatmulFlattenPass Created node: Reshape Created node: MatMulV2 Created node: Reshape InferShapeAndCheckSupport successOne-click Verification (Optional)Use thequick_verify.shscript to complete compilation, ATC, dump graph inspection, and performance testing in one click. Thesoc_versionin the script defaults toAscend910B3, modify the--soc_versionparameter in the atc command in the script according to your actual environment (refer to Using ATC for Offline Inference):cd data ./quick_verify.sh [batch] [m] [k] [n] [test_rounds]Default parameters: batch32, m64, k512, n256, test_rounds3The script automatically:Checks and compiles Pass (if not compiled)Checks and compiles benchmark_modelGenerates ONNX modelATC compiles fusion modelChecks dump graph to verify fusion effectRuns multiple rounds of performance testingCleans custom pass so installed in CANN packageIf the pre-fusion dump graph has no BatchMatMul node, or the post-fusion graph doesnt have the ReshapeMatMulV2Reshape replacement structure, the script will exit with failure.View ResultsAfter the ATC tool command completes, a series of.pbtxtand.txtfiles are generated in the directory. Compare the following dump graphs:ge_proto_xxxxx_graph_x_PreRunBegin.txtpre-execution dump graph, should contain BatchMatMulV2 nodege_proto_xxxxx_graph_x_RunCustomPass_AfterInferShape.txtcustom pass dump graph after InferShape, should contain ReshapeMatMulV2Reshape nodes, no longer contains BatchMatMulV2 nodeYou can see the model has been optimized as expected, i.e., BatchMatMulV2 is replaced by ReshapeMatMulV2Reshape.If expected results are not obtained, you can set the following environment variables (if using atc command, also add parameter--logdebug) to print logs to screen for troubleshooting:export ASCEND_SLOG_PRINT_TO_STDOUT1 export ASCEND_GLOBAL_LOG_LEVEL0【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考