GE Memory Constraints Document【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/geStatic Memory ReuseCode Location:compiler/graph/build/memory/Constraint 1: Graph compilation module memory reuse phase forbids graph modificationPrecise boundary:Scope of graph modification prohibition:BlockMemAssigner::AssignMemoryWithReuseimplementation and all functions it triggersMulti-threaded entry:HybridMemAssigner::Assignstarts multiple threads, concurrently calling AssignMemoryWithReuseExplicitly prohibited:Adding, modifying, deleting attributes on ComputeGraphs NodeSafe operations:Reading attributes/OpDesc is safe (system reads OpDesc extensively during traversal to judge memory allocation strategy)Constraint 2: Dynamic multi-batch scenario impact analysisCode entity:DynamicBatchMemAssigner(dynamic_batch_mem_assigner.h)Meaning:System identifies different batches throughbatch_label_(set by user or GE upper framework), supports different memory reuse strategies between different batchesConflict with static memory reuse:Continuous input memory in different batches will be merged into one large block for alignmentNo reuse within/between batches, but alignment strategy between batches exists, leading to possibly lower memory usage efficiencyMaximum split size limit:kMaxSplitSizeForDynamicBatch 400MB(dynamic_batch_mem_assigner.h)Constraint 3: Scenarios to consider for static graph memory new featuresScenarioCode marker/basisImpact on reuseContinuous memorycontinuous_block_(block_mem_assigner.h),ContinuousMemMng(continuous_mem.cc)Supports memory merging for continuous input nodes; continuous memory in different batches can merge and reuseAtomic centralized zero-clearatomic_addr_clean_id_(block_mem_assigner.h)Memory blocks needing atomic zero-clear cannot be reused by other nodes; if node has no related attribute, skip zero-clearZero copyis_zero_copy_(block_mem_assigner.h),IsNodeAndPeerNodeTaskSupportZeroCopy(block_mem_assigner.cc)Zero copy blocks can be reused across nodes (IsRealSizeReuseBlock); zero copy memory cannot merge (multiple user input addresses may be discontinuous)Immutable address outputis_fixed_addr_prior_(block_mem_assigner.h)Output addresses of constant/const/variable/fileconstant/constplaceholder type operators are fixed at compile time, fixed address priority memory blocks can be reused but addresses are immutableOperators not supporting address refreshHCOM/rtsStreamSwitchByIndex etcInput/output addresses of these operators must be stable, cannot use zero copyP2P memory typeRT_MEMORY_P2P_DDR(block_mem_assigner.cc)P2P memory cannot merge zero-clear with other memory types (graph_mem_assigner.cc)Constraint 4: Particularity of HCOM operatorsMeaning of continuous:Logically continuous, not physically continuous. Outputs of multiple HCOM operators form continuous memory region logically, managed throughContinuousMemMngmanager for allocation and reuse. —continuous_mem.ccfeatureBaseRefreshable configuration:Get method:ge::GetContext().GetOption(ge::OPTION_FEATURE_BASE_REFRESHABLE, refreshable)—block_mem_assigner.ccMember variable:is_feature_map_refreshable_(block_mem_assigner.h)Default value:false, set totruewhen configuration value is 1Effect: Controls whether feature map is refreshable, affectsIsNoNeedAssignMemoryjudgmentConstraint 5: Other constraintsPreAssign/SetOpMemOffset not thread-safe:Can only be called by single thread, other concurrent operations need attention. —block_mem_assigner.hAlignment strategy difference:Zero copy memory uses 32-byte alignment, others use 512-byte alignment. —graph_mem_assigner.hSubgraph NETOUTPUT special handling:NETOUTPUT nodes in subgraphs cannot perform zero copy. —block_mem_assigner.ccMulti-batch shape data node constraint:Multi-batch shape data nodes do not support zero copy. —block_mem_assigner.ccSuspended memory block management:Suspended memory blocks are released during next node allocation, lifecycle managed throughlife_time_begin_andlife_time_end_, cannot be modified once set. —block_mem_assigner.hReuse strategy configurability:Supports dynamic configuration through parameters likeuse_range_,ascending_sort_,reuse_first_release_,memory_priority_mode_. —block_mem_assigner.hDynamic Memory ReuseCode Location:v2 layer:runtime/v2/kernel/memory/allocator/(ScalableAllocator, MemoryPool)v1 layer:runtime/v1/graph/manager/active_memory_allocator.h(ActiveMemoryAllocator, ExpandableActiveMemoryAllocator, PhysicalMemoryAllocator)Bridge layer:runtime/v2/kernel/memory/device/device_allocator.h(DeviceAllocator)Constraint 1: ScalableAllocator does not support multi-threaded concurrencyCode location:runtime/v2/kernel/memory/allocator/scalable_allocator.hLock-free design basis:Class internally has nostd::mutexorstd::recursive_mutex, onlystatic std::atomic_size_t global_allocator_id_for generating unique ID (scalable_allocator.h)Safety guarantee method:Guaranteed byaclmdlExecutecall constraint for single-threaded calling (seedocs/graph_engine_api/aclmdlExecute.mdfor details), underlying allocator guarantees thread safety through recursive_mutexUnderlying has lock protection:v1 layers PhysicalMemoryAllocator usesstd::recursive_mutex(active_memory_allocator.h), ExpandableActiveMemoryAllocatorImp also usesstd::recursive_mutex(active_memory_allocator.h)Constraint 2: ActiveMemoryAllocator/ExpandableActiveMemoryAllocator/PhysicalMemoryAllocator support multi-threadingThread safety mechanism:Usesstd::recursive_mutexto protect shared resourcesNew code requirements:Must lock when accessing shared resources, follow existing lock usage patternsMemory ManagementCode Location:runtime/v2/kernel/memory/(excluding allocator subdirectory)Constraint 1: Device id correctnessCode location:memory_kernel.ccusesaclrtGetDeviceto get device_idRequirement:When calling rts interfaces, device id must be explicitly passed correct value, avoid using default parameters (default is 0), need to verify multi-device scenario test casesConstraint 2: Memory release timingOrder:First stream synchronization → then release memory → finally destroy deviceCode association:caching_mem_allocator.cc,AllocateWithTryRecyclemethod ensures synchronization before releaseConstraint 3: Virtual memory compatibility designrtReserveMemAddress purpose:Virtual address reservation, used for dynamic shape pre-allocated address spaceActual call location:runtime/v1/graph/manager/active_memory_allocator.ccFallback path:WhenrtReserveMemAddressfails, mark as not supporting virtual address reservation, fallback to physical address allocation mode. —runtime/v1/graph/manager/active_memory_allocator.cc(Maybe not support rtReserveMemAddress.)Requirement:Need to ensure business flow is normal, no ERROR logs【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 CANN/ge内存约束设计文档
GE Memory Constraints Document【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/geStatic Memory ReuseCode Location:compiler/graph/build/memory/Constraint 1: Graph compilation module memory reuse phase forbids graph modificationPrecise boundary:Scope of graph modification prohibition:BlockMemAssigner::AssignMemoryWithReuseimplementation and all functions it triggersMulti-threaded entry:HybridMemAssigner::Assignstarts multiple threads, concurrently calling AssignMemoryWithReuseExplicitly prohibited:Adding, modifying, deleting attributes on ComputeGraphs NodeSafe operations:Reading attributes/OpDesc is safe (system reads OpDesc extensively during traversal to judge memory allocation strategy)Constraint 2: Dynamic multi-batch scenario impact analysisCode entity:DynamicBatchMemAssigner(dynamic_batch_mem_assigner.h)Meaning:System identifies different batches throughbatch_label_(set by user or GE upper framework), supports different memory reuse strategies between different batchesConflict with static memory reuse:Continuous input memory in different batches will be merged into one large block for alignmentNo reuse within/between batches, but alignment strategy between batches exists, leading to possibly lower memory usage efficiencyMaximum split size limit:kMaxSplitSizeForDynamicBatch 400MB(dynamic_batch_mem_assigner.h)Constraint 3: Scenarios to consider for static graph memory new featuresScenarioCode marker/basisImpact on reuseContinuous memorycontinuous_block_(block_mem_assigner.h),ContinuousMemMng(continuous_mem.cc)Supports memory merging for continuous input nodes; continuous memory in different batches can merge and reuseAtomic centralized zero-clearatomic_addr_clean_id_(block_mem_assigner.h)Memory blocks needing atomic zero-clear cannot be reused by other nodes; if node has no related attribute, skip zero-clearZero copyis_zero_copy_(block_mem_assigner.h),IsNodeAndPeerNodeTaskSupportZeroCopy(block_mem_assigner.cc)Zero copy blocks can be reused across nodes (IsRealSizeReuseBlock); zero copy memory cannot merge (multiple user input addresses may be discontinuous)Immutable address outputis_fixed_addr_prior_(block_mem_assigner.h)Output addresses of constant/const/variable/fileconstant/constplaceholder type operators are fixed at compile time, fixed address priority memory blocks can be reused but addresses are immutableOperators not supporting address refreshHCOM/rtsStreamSwitchByIndex etcInput/output addresses of these operators must be stable, cannot use zero copyP2P memory typeRT_MEMORY_P2P_DDR(block_mem_assigner.cc)P2P memory cannot merge zero-clear with other memory types (graph_mem_assigner.cc)Constraint 4: Particularity of HCOM operatorsMeaning of continuous:Logically continuous, not physically continuous. Outputs of multiple HCOM operators form continuous memory region logically, managed throughContinuousMemMngmanager for allocation and reuse. —continuous_mem.ccfeatureBaseRefreshable configuration:Get method:ge::GetContext().GetOption(ge::OPTION_FEATURE_BASE_REFRESHABLE, refreshable)—block_mem_assigner.ccMember variable:is_feature_map_refreshable_(block_mem_assigner.h)Default value:false, set totruewhen configuration value is 1Effect: Controls whether feature map is refreshable, affectsIsNoNeedAssignMemoryjudgmentConstraint 5: Other constraintsPreAssign/SetOpMemOffset not thread-safe:Can only be called by single thread, other concurrent operations need attention. —block_mem_assigner.hAlignment strategy difference:Zero copy memory uses 32-byte alignment, others use 512-byte alignment. —graph_mem_assigner.hSubgraph NETOUTPUT special handling:NETOUTPUT nodes in subgraphs cannot perform zero copy. —block_mem_assigner.ccMulti-batch shape data node constraint:Multi-batch shape data nodes do not support zero copy. —block_mem_assigner.ccSuspended memory block management:Suspended memory blocks are released during next node allocation, lifecycle managed throughlife_time_begin_andlife_time_end_, cannot be modified once set. —block_mem_assigner.hReuse strategy configurability:Supports dynamic configuration through parameters likeuse_range_,ascending_sort_,reuse_first_release_,memory_priority_mode_. —block_mem_assigner.hDynamic Memory ReuseCode Location:v2 layer:runtime/v2/kernel/memory/allocator/(ScalableAllocator, MemoryPool)v1 layer:runtime/v1/graph/manager/active_memory_allocator.h(ActiveMemoryAllocator, ExpandableActiveMemoryAllocator, PhysicalMemoryAllocator)Bridge layer:runtime/v2/kernel/memory/device/device_allocator.h(DeviceAllocator)Constraint 1: ScalableAllocator does not support multi-threaded concurrencyCode location:runtime/v2/kernel/memory/allocator/scalable_allocator.hLock-free design basis:Class internally has nostd::mutexorstd::recursive_mutex, onlystatic std::atomic_size_t global_allocator_id_for generating unique ID (scalable_allocator.h)Safety guarantee method:Guaranteed byaclmdlExecutecall constraint for single-threaded calling (seedocs/graph_engine_api/aclmdlExecute.mdfor details), underlying allocator guarantees thread safety through recursive_mutexUnderlying has lock protection:v1 layers PhysicalMemoryAllocator usesstd::recursive_mutex(active_memory_allocator.h), ExpandableActiveMemoryAllocatorImp also usesstd::recursive_mutex(active_memory_allocator.h)Constraint 2: ActiveMemoryAllocator/ExpandableActiveMemoryAllocator/PhysicalMemoryAllocator support multi-threadingThread safety mechanism:Usesstd::recursive_mutexto protect shared resourcesNew code requirements:Must lock when accessing shared resources, follow existing lock usage patternsMemory ManagementCode Location:runtime/v2/kernel/memory/(excluding allocator subdirectory)Constraint 1: Device id correctnessCode location:memory_kernel.ccusesaclrtGetDeviceto get device_idRequirement:When calling rts interfaces, device id must be explicitly passed correct value, avoid using default parameters (default is 0), need to verify multi-device scenario test casesConstraint 2: Memory release timingOrder:First stream synchronization → then release memory → finally destroy deviceCode association:caching_mem_allocator.cc,AllocateWithTryRecyclemethod ensures synchronization before releaseConstraint 3: Virtual memory compatibility designrtReserveMemAddress purpose:Virtual address reservation, used for dynamic shape pre-allocated address spaceActual call location:runtime/v1/graph/manager/active_memory_allocator.ccFallback path:WhenrtReserveMemAddressfails, mark as not supporting virtual address reservation, fallback to physical address allocation mode. —runtime/v1/graph/manager/active_memory_allocator.cc(Maybe not support rtReserveMemAddress.)Requirement:Need to ensure business flow is normal, no ERROR logs【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考