1. 倚天710的DDR5子系统
倚天710支持支持最先进的DDR5 DRAM,为云计算和HPC提供巨大的内存带宽。倚天710有8 DDR5通道(channel),每个Die上有4个。每个通道相互独立地服务系统的内存请求,分别支持用于1DPC(DIMM Per Channel)的DDR5-4400和2DPC的DDR5-4000。
1.2 DDR5 Architecture
DDR5的一个主要变化是新的DIMM通道结构(Fig 2中Channel Architecture)。DDR4 DIMM的总线位宽为72比特,由64比特数据位和8比特ECC位组成。DDR5的每个DIMM有两个独立的子通道。两个通道中的总线位宽都为40比特:32比特的数据位和8比特的ECC位。尽管DDR4和DDR5的数据位宽相同(总共64比特),但两个独立通道可以提高内存访问效率并减少延迟。单通道单次任务只能读或写,双通道的DDR5则读写可以同时进行。
1.2 DDR5 理论带宽
倚天2DPC的DDR5-4000的理论带宽为:
- 4000MHz *32bit / 8 *8 *2 = 128 *10^9 *2 bytes = 128GB/s *2= 256 GB/s
- 内存等效频率(4000MHz)_ 子通道位宽(32 bit)/ 8 _ 子通道数(8)* Die (2)
注意GB和GiB的不同:
- 1 GB = 1000000000 bytes (= 1000^3 B = 10^9 B)
- 1 GiB = 1073741824 bytes (= 1024^3 B = 2^30 B).
2. 倚天710 DDRSS PMU
倚天710的DDRSS为每个子通道都实现了独立的PMU,用于性能和功能调试,每个子通道的PMU包含16个通用计数器。
带宽计算公式为:
- DRAM Read Bandwidth = perf_hif_rd *DDRC_WIDTH *DDRC_Freq / DDRC_Cycle
- DRAM Write Bandwidth = (perf_hif_wr + perf_hif_rmw) *DDRC_WIDTH *DDRC_Freq / DDRC_Cycle
- DDRC_WIDTH: Units of 64 bytes
3. Cloud-kernel对DDRSS PMU的支持
#lscpu
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 1
Core(s) per socket: 128
Socket(s): 1
NUMA node(s): 2
...
测试环境为1个Socket,2个Die,包含两个NUMA node。
#numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 0 size: 257416 MB
node 0 free: 187991 MB
node 1 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
node 1 size: 257014 MB
node 1 free: 194504 MB
node distances:
node 0 1
0: 10 15
1: 15 10
每个NUMA node有 256 GB内存。
#dmidecode|grep -P -A5 "Memorys+Device"|grep Size|grep -v Range
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: 32 GB
Size: No Module Installed
...
#dmidecode -t memory | grep Speed:
Speed: 4000 MHz
Configured Clock Speed: 4000 MHz
2DPC,共插了16根DIMM,每个Die8根DIMM,有效频率为 4000MHz。
#ls /sys/bus/event_source/devices/ | grep drw
ali_drw_21000
ali_drw_21080
ali_drw_23000
ali_drw_23080
ali_drw_25000
ali_drw_25080
ali_drw_27000
ali_drw_27080
ali_drw_40021000
ali_drw_40021080
ali_drw_40023000
ali_drw_40023080
ali_drw_40025000
ali_drw_40025080
ali_drw_40027000
ali_drw_40027080
2DPC满插时一共16个PMU设备,其中ali_drw_21000
与ali_drw_21080
为Die 0上同一个DIMM的两个子通道,ali_drw_2X000
为Die 0的PMU设备,ali_drw_4002X000
为Die 1的PMU设备。
4. DDR 带宽准确性验证
4.1 TL;DR
带宽单位:MB/s
可以看到,DDR PMU的带宽统计误差不超过 1%。测试原理,请阅读《倚天710性能监控 —— CMN Flit Traffic Trace with Watchpoint Event》。
4.2 C0M0 rd
# First, run bw_mem as backgroud workload
# numactl --cpubind=0 --membind=0 ./bw_mem 40960M rd
# Then run perf command in another console
perf stat -e ali_drw_21000/perf_hif_wr/ -e ali_drw_21000/perf_hif_rd/ -e ali_drw_21000/perf_hif_rmw/ -e ali_drw_21000/perf_cycle/ -e ali_drw_21080/perf_hif_wr/ -e ali_drw_21080/perf_hif_rd/ -e ali_drw_21080/perf_hif_rmw/ -e ali_drw_21080/perf_cycle/ -e ali_drw_23000/perf_hif_wr/ -e ali_drw_23000/perf_hif_rd/ -e ali_drw_23000/perf_hif_rmw/ -e ali_drw_23000/perf_cycle/ -e ali_drw_23080/perf_hif_wr/ -e ali_drw_23080/perf_hif_rd/ -e ali_drw_23080/perf_hif_rmw/ -e ali_drw_23080/perf_cycle/ -e ali_drw_25000/perf_hif_wr/ -e ali_drw_25000/perf_hif_rd/ -e ali_drw_25000/perf_hif_rmw/ -e ali_drw_25000/perf_cycle/ -e ali_drw_25080/perf_hif_wr/ -e ali_drw_25080/perf_hif_rd/ -e ali_drw_25080/perf_hif_rmw/ -e ali_drw_25080/perf_cycle/ -e ali_drw_27000/perf_hif_wr/ -e ali_drw_27000/perf_hif_rd/ -e ali_drw_27000/perf_hif_rmw/ -e ali_drw_27000/perf_cycle/ -e ali_drw_27080/perf_hif_wr/ -e ali_drw_27080/perf_hif_rd/ -e ali_drw_27080/perf_hif_rmw/ -e ali_drw_27080/perf_cycle/ -a -- sleep 1
Performance counter stats for 'system wide':
12398 ali_drw_21000/perf_hif_wr/
40160751 ali_drw_21000/perf_hif_rd/
743 ali_drw_21000/perf_hif_rmw/
500620725 ali_drw_21000/perf_cycle/
12252 ali_drw_21080/perf_hif_wr/
40161013 ali_drw_21080/perf_hif_rd/
767 ali_drw_21080/perf_hif_rmw/
500619340 ali_drw_21080/perf_cycle/
11960 ali_drw_23000/perf_hif_wr/
40159522 ali_drw_23000/perf_hif_rd/
737 ali_drw_23000/perf_hif_rmw/
500613505 ali_drw_23000/perf_cycle/
12044 ali_drw_23080/perf_hif_wr/
40159066 ali_drw_23080/perf_hif_rd/
773 ali_drw_23080/perf_hif_rmw/
500607620 ali_drw_23080/perf_cycle/
12698 ali_drw_25000/perf_hif_wr/
40160138 ali_drw_25000/perf_hif_rd/
709 ali_drw_25000/perf_hif_rmw/
500601240 ali_drw_25000/perf_cycle/
12521 ali_drw_25080/perf_hif_wr/
40160169 ali_drw_25080/perf_hif_rd/
727 ali_drw_25080/perf_hif_rmw/
500594755 ali_drw_25080/perf_cycle/
12171 ali_drw_27000/perf_hif_wr/
40159404 ali_drw_27000/perf_hif_rd/
706 ali_drw_27000/perf_hif_rmw/
500589945 ali_drw_27000/perf_cycle/
12290 ali_drw_27080/perf_hif_wr/
40157620 ali_drw_27080/perf_hif_rd/
710 ali_drw_27080/perf_hif_rmw/
500583305 ali_drw_27080/perf_cycle/
1.000923276 seconds time elapsed
>>> 40159522*8*64/1000/1000.0
20561.675
# set CPU and memory to the same NUMA node
numactl --cpubind=0 --membind=0 ./bw_mem 40960M rd
40960.00 20507.82
4.3 C1M1 rd
# First, run bw_mem as backgroud workload
# numactl --cpubind=1 --membind=1 ./bw_mem 40960M rd
# Then run perf command in another console
perf stat -e ali_drw_40021000/perf_hif_wr/ -e ali_drw_40021000/perf_hif_rd/ -e ali_drw_40021000/perf_hif_rmw/ -e ali_drw_40021000/perf_cycle/ -e ali_drw_40021080/perf_hif_wr/ -e ali_drw_40021080/perf_hif_rd/ -e ali_drw_40021080/perf_hif_rmw/ -e ali_drw_40021080/perf_cycle/ -e ali_drw_40023000/perf_hif_wr/ -e ali_drw_40023000/perf_hif_rd/ -e ali_drw_40023000/perf_hif_rmw/ -e ali_drw_40023000/perf_cycle/ -e ali_drw_40023080/perf_hif_wr/ -e ali_drw_40023080/perf_hif_rd/ -e ali_drw_40023080/perf_hif_rmw/ -e ali_drw_40023080/perf_cycle/ -e ali_drw_40025000/perf_hif_wr/ -e ali_drw_40025000/perf_hif_rd/ -e ali_drw_40025000/perf_hif_rmw/ -e ali_drw_40025000/perf_cycle/ -e ali_drw_40025080/perf_hif_wr/ -e ali_drw_40025080/perf_hif_rd/ -e ali_drw_40025080/perf_hif_rmw/ -e ali_drw_40025080/perf_cycle/ -e ali_drw_40027000/perf_hif_wr/ -e ali_drw_40027000/perf_hif_rd/ -e ali_drw_40027000/perf_hif_rmw/ -e ali_drw_40027000/perf_cycle/ -e ali_drw_40027080/perf_hif_wr/ -e ali_drw_40027080/perf_hif_rd/ -e ali_drw_40027080/perf_hif_rmw/ -e ali_drw_40027080/perf_cycle/ -a -- sleep 1
Performance counter stats for 'system wide':
2329 ali_drw_40021000/perf_hif_wr/
40071983 ali_drw_40021000/perf_hif_rd/
58 ali_drw_40021000/perf_hif_rmw/
500572165 ali_drw_40021000/perf_cycle/
2374 ali_drw_40021080/perf_hif_wr/
40071737 ali_drw_40021080/perf_hif_rd/
39 ali_drw_40021080/perf_hif_rmw/
500569615 ali_drw_40021080/perf_cycle/
2330 ali_drw_40023000/perf_hif_wr/
40071063 ali_drw_40023000/perf_hif_rd/
74 ali_drw_40023000/perf_hif_rmw/
500565635 ali_drw_40023000/perf_cycle/
2372 ali_drw_40023080/perf_hif_wr/
40070344 ali_drw_40023080/perf_hif_rd/
54 ali_drw_40023080/perf_hif_rmw/
500561355 ali_drw_40023080/perf_cycle/
2362 ali_drw_40025000/perf_hif_wr/
40070906 ali_drw_40025000/perf_hif_rd/
45 ali_drw_40025000/perf_hif_rmw/
500557480 ali_drw_40025000/perf_cycle/
2385 ali_drw_40025080/perf_hif_wr/
40070168 ali_drw_40025080/perf_hif_rd/
46 ali_drw_40025080/perf_hif_rmw/
500552550 ali_drw_40025080/perf_cycle/
2333 ali_drw_40027000/perf_hif_wr/
40069233 ali_drw_40027000/perf_hif_rd/
28 ali_drw_40027000/perf_hif_rmw/
500548745 ali_drw_40027000/perf_cycle/
2211 ali_drw_40027080/perf_hif_wr/
40068227 ali_drw_40027080/perf_hif_rd/
30 ali_drw_40027080/perf_hif_rmw/
500544450 ali_drw_40027080/perf_cycle/
1.000863258 seconds time elapsed
>>> 40070906*8*64/1000/1000.0
20516.303
numactl --cpubind=1 --membind=1 ./bw_mem 40960M rd
40960.00 20492.53
4.4 C0M0 fwr
# First, run bw_mem as backgroud workload
# numactl --cpubind=0 --membind=0 ./bw_mem 40960M fwr
# Then run perf command in another console
perf stat -e ali_drw_21000/perf_hif_wr/ -e ali_drw_21000/perf_hif_rd/ -e ali_drw_21000/perf_hif_rmw/ -e ali_drw_21000/perf_cycle/ -e ali_drw_21080/perf_hif_wr/ -e ali_drw_21080/perf_hif_rd/ -e ali_drw_21080/perf_hif_rmw/ -e ali_drw_21080/perf_cycle/ -e ali_drw_23000/perf_hif_wr/ -e ali_drw_23000/perf_hif_rd/ -e ali_drw_23000/perf_hif_rmw/ -e ali_drw_23000/perf_cycle/ -e ali_drw_23080/perf_hif_wr/ -e ali_drw_23080/perf_hif_rd/ -e ali_drw_23080/perf_hif_rmw/ -e ali_drw_23080/perf_cycle/ -e ali_drw_25000/perf_hif_wr/ -e ali_drw_25000/perf_hif_rd/ -e ali_drw_25000/perf_hif_rmw/ -e ali_drw_25000/perf_cycle/ -e ali_drw_25080/perf_hif_wr/ -e ali_drw_25080/perf_hif_rd/ -e ali_drw_25080/perf_hif_rmw/ -e ali_drw_25080/perf_cycle/ -e ali_drw_27000/perf_hif_wr/ -e ali_drw_27000/perf_hif_rd/ -e ali_drw_27000/perf_hif_rmw/ -e ali_drw_27000/perf_cycle/ -e ali_drw_27080/perf_hif_wr/ -e ali_drw_27080/perf_hif_rd/ -e ali_drw_27080/perf_hif_rmw/ -e ali_drw_27080/perf_cycle/ -a -- sleep 1
Performance counter stats for 'system wide':
42910737 ali_drw_21000/perf_hif_wr/
108397 ali_drw_21000/perf_hif_rd/
495 ali_drw_21000/perf_hif_rmw/
500708510 ali_drw_21000/perf_cycle/
42911223 ali_drw_21080/perf_hif_wr/
117280 ali_drw_21080/perf_hif_rd/
515 ali_drw_21080/perf_hif_rmw/
500706780 ali_drw_21080/perf_cycle/
42910038 ali_drw_23000/perf_hif_wr/
109179 ali_drw_23000/perf_hif_rd/
516 ali_drw_23000/perf_hif_rmw/
500702100 ali_drw_23000/perf_cycle/
42911620 ali_drw_23080/perf_hif_wr/
111038 ali_drw_23080/perf_hif_rd/
523 ali_drw_23080/perf_hif_rmw/
500697340 ali_drw_23080/perf_cycle/
42910435 ali_drw_25000/perf_hif_wr/
111748 ali_drw_25000/perf_hif_rd/
469 ali_drw_25000/perf_hif_rmw/
500692500 ali_drw_25000/perf_cycle/
42908786 ali_drw_25080/perf_hif_wr/
110177 ali_drw_25080/perf_hif_rd/
456 ali_drw_25080/perf_hif_rmw/
500686595 ali_drw_25080/perf_cycle/
42908903 ali_drw_27000/perf_hif_wr/
114093 ali_drw_27000/perf_hif_rd/
490 ali_drw_27000/perf_hif_rmw/
500681405 ali_drw_27000/perf_cycle/
42908156 ali_drw_27080/perf_hif_wr/
109668 ali_drw_27080/perf_hif_rd/
489 ali_drw_27080/perf_hif_rmw/
500676420 ali_drw_27080/perf_cycle/
1.001100811 seconds time elapsed
>>> (42908156+489)*8*64/1000/1000.0
21969.226
numactl --cpubind=0 --membind=0 ./bw_mem 40960M fwr
40960.00 21936.50
4.5 C1M1 fwr
# First, run bw_mem as backgroud workload
# numactl --cpubind=1 --membind=1 ./bw_mem 40960M fwr
# Then run perf command in another console
perf stat -e ali_drw_40021000/perf_hif_wr/ -e ali_drw_40021000/perf_hif_rd/ -e ali_drw_40021000/perf_hif_rmw/ -e ali_drw_40021000/perf_cycle/ -e ali_drw_40021080/perf_hif_wr/ -e ali_drw_40021080/perf_hif_rd/ -e ali_drw_40021080/perf_hif_rmw/ -e ali_drw_40021080/perf_cycle/ -e ali_drw_40023000/perf_hif_wr/ -e ali_drw_40023000/perf_hif_rd/ -e ali_drw_40023000/perf_hif_rmw/ -e ali_drw_40023000/perf_cycle/ -e ali_drw_40023080/perf_hif_wr/ -e ali_drw_40023080/perf_hif_rd/ -e ali_drw_40023080/perf_hif_rmw/ -e ali_drw_40023080/perf_cycle/ -e ali_drw_40025000/perf_hif_wr/ -e ali_drw_40025000/perf_hif_rd/ -e ali_drw_40025000/perf_hif_rmw/ -e ali_drw_40025000/perf_cycle/ -e ali_drw_40025080/perf_hif_wr/ -e ali_drw_40025080/perf_hif_rd/ -e ali_drw_40025080/perf_hif_rmw/ -e ali_drw_40025080/perf_cycle/ -e ali_drw_40027000/perf_hif_wr/ -e ali_drw_40027000/perf_hif_rd/ -e ali_drw_40027000/perf_hif_rmw/ -e ali_drw_40027000/perf_cycle/ -e ali_drw_40027080/perf_hif_wr/ -e ali_drw_40027080/perf_hif_rd/ -e ali_drw_40027080/perf_hif_rmw/ -e ali_drw_40027080/perf_cycle/ -a -- sleep 1
Performance counter stats for 'system wide':
42906048 ali_drw_40021000/perf_hif_wr/
33939 ali_drw_40021000/perf_hif_rd/
76 ali_drw_40021000/perf_hif_rmw/
500629355 ali_drw_40021000/perf_cycle/
42905967 ali_drw_40021080/perf_hif_wr/
34018 ali_drw_40021080/perf_hif_rd/
63 ali_drw_40021080/perf_hif_rmw/
500631900 ali_drw_40021080/perf_cycle/
42905422 ali_drw_40023000/perf_hif_wr/
33843 ali_drw_40023000/perf_hif_rd/
75 ali_drw_40023000/perf_hif_rmw/
500628540 ali_drw_40023000/perf_cycle/
42905547 ali_drw_40023080/perf_hif_wr/
33858 ali_drw_40023080/perf_hif_rd/
68 ali_drw_40023080/perf_hif_rmw/
500623970 ali_drw_40023080/perf_cycle/
42905230 ali_drw_40025000/perf_hif_wr/
34028 ali_drw_40025000/perf_hif_rd/
56 ali_drw_40025000/perf_hif_rmw/
500620630 ali_drw_40025000/perf_cycle/
42904734 ali_drw_40025080/perf_hif_wr/
34141 ali_drw_40025080/perf_hif_rd/
61 ali_drw_40025080/perf_hif_rmw/
500615840 ali_drw_40025080/perf_cycle/
42903390 ali_drw_40027000/perf_hif_wr/
33712 ali_drw_40027000/perf_hif_rd/
84 ali_drw_40027000/perf_hif_rmw/
500610635 ali_drw_40027000/perf_cycle/
42903975 ali_drw_40027080/perf_hif_wr/
33916 ali_drw_40027080/perf_hif_rd/
106 ali_drw_40027080/perf_hif_rmw/
500606645 ali_drw_40027080/perf_cycle/
1.000953335 seconds time elapsed
>>> (42903975+106)*8*64/1000/1000.0
21966.889
#numactl --cpubind=1 --membind=1 ./bw_mem 40960M fwr
40960.00 21934.51
-
计数器
+关注
关注
32文章
2256浏览量
94475 -
DRAM芯片
+关注
关注
1文章
84浏览量
18011 -
HPC
+关注
关注
0文章
315浏览量
23751 -
PMU
+关注
关注
1文章
108浏览量
21591 -
DDR5
+关注
关注
1文章
422浏览量
24139
发布评论请先 登录
相关推荐
评论