{ "type": "excalidraw", "version": 2, "source": "https://excalidraw.com", "elements": [ { "id": "bg-top", "type": "rectangle", "x": 30, "y": 20, "width": 1740, "height": 86, "angle": 0, "strokeColor": "#16a34a", "backgroundColor": "#f8fafc", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8047, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "title", "type": "text", "x": 60, "y": 39, "width": 940.5000000000008, "height": 37.5, "angle": 0, "strokeColor": "#16a34a", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8048, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "FlashAttention v2 better parallelism + work partitioning", "fontSize": 30, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "FlashAttention v2 better parallelism + work partitioning", "autoResize": true, "lineHeight": 1.25 }, { "id": "subtitle", "type": "text", "x": 60, "y": 75, "width": 632.4, "height": 21.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8049, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "目标：数学和 v1 基本相同，把 GPU 利用率从 IO-aware 进一步推向接近 GEMM。", "fontSize": 17, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "目标：数学和 v1 基本相同，把 GPU 利用率从 IO-aware 进一步推向接近 GEMM。", "autoResize": true, "lineHeight": 1.25 }, { "id": "core-panel", "type": "rectangle", "x": 60, "y": 135, "width": 520, "height": 330, "angle": 0, "strokeColor": "#16a34a", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8050, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "core-title", "type": "text", "x": 85, "y": 160, "width": 216.7, "height": 27.5, "angle": 0, "strokeColor": "#15803d", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8051, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "v2 相对 v1 改了什么", "fontSize": 22, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "v2 相对 v1 改了什么", "autoResize": true, "lineHeight": 1.25 }, { "id": "core-body", "type": "text", "x": 85, "y": 205, "width": 447.10000000000014, "height": 191.25, "angle": 0, "strokeColor": "#14532d", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8052, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "v1 已经少搬 HBM，但还不够快：\n1. 非 matmul 操作占比高\n2. 只按 batch/head 并行，长序列小 batch 时 SM 不满\n3. sliced-K 需要 warp 间写 SMEM、同步、归并\n\nv2 的三板斧：\n减少 rescale/mask/bound check 等非 matmul\n沿 sequence 维度增加 CTA 并行\n从 sliced-K 改为 sliced-Q，减少 warp 间通信", "fontSize": 17, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "v1 已经少搬 HBM，但还不够快：\n1. 非 matmul 操作占比高\n2. 只按 batch/head 并行，长序列小 batch 时 SM 不满\n3. sliced-K 需要 warp 间写 SMEM、同步、归并\n\nv2 的三板斧：\n减少 rescale/mask/bound check 等非 matmul\n沿 sequence 维度增加 CTA 并行\n从 sliced-K 改为 sliced-Q，减少 warp 间通信", "autoResize": true, "lineHeight": 1.25 }, { "id": "pseudo-panel", "type": "rectangle", "x": 620, "y": 135, "width": 560, "height": 420, "angle": 0, "strokeColor": "#2563eb", "backgroundColor": "#dbeafe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8053, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "pseudo-title", "type": "text", "x": 645, "y": 160, "width": 399.3000000000002, "height": 27.5, "angle": 0, "strokeColor": "#1e40af", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8054, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "概念伪代码：更细的 query block 并行", "fontSize": 22, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "概念伪代码：更细的 query block 并行", "autoResize": true, "lineHeight": 1.25 }, { "id": "pseudo-code", "type": "text", "x": 645, "y": 205, "width": 334.4000000000002, "height": 380, "angle": 0, "strokeColor": "#1e3a8a", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8055, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "parallel for batch, head, q_block i:\n split Qi rows across warps\n keep K/V tile visible to all warps\n m,l,acc are per Q-row slice\n\n for k_block j:\n load Kj,Vj once to SMEM\n each warp handles its Q slice:\n S_w = Q_w @ Kj.T\n apply mask only when needed\n online_softmax_update\n acc_w += P_w @ Vj\n\n no cross-warp reduction for O slice\n store each warp output slice\n\nextra parallelism:\n more CTAs cover sequence blocks\n improves occupancy when B*H is small", "fontSize": 16, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "parallel for batch, head, q_block i:\n split Qi rows across warps\n keep K/V tile visible to all warps\n m,l,acc are per Q-row slice\n\n for k_block j:\n load Kj,Vj once to SMEM\n each warp handles its Q slice:\n S_w = Q_w @ Kj.T\n apply mask only when needed\n online_softmax_update\n acc_w += P_w @ Vj\n\n no cross-warp reduction for O slice\n store each warp output slice\n\nextra parallelism:\n more CTAs cover sequence blocks\n improves occupancy when B*H is small", "autoResize": true, "lineHeight": 1.25 }, { "id": "partition-panel", "type": "rectangle", "x": 60, "y": 510, "width": 1120, "height": 280, "angle": 0, "strokeColor": "#64748b", "backgroundColor": "#f8fafc", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8056, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "partition-title", "type": "text", "x": 85, "y": 535, "width": 314.60000000000014, "height": 27.5, "angle": 0, "strokeColor": "#334155", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8057, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "v1 sliced-K vs v2 sliced-Q", "fontSize": 22, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "v1 sliced-K vs v2 sliced-Q", "autoResize": true, "lineHeight": 1.25 }, { "id": "sliced-k", "type": "rectangle", "x": 95, "y": 590, "width": 470, "height": 145, "angle": 0, "strokeColor": "#ef4444", "backgroundColor": "#fee2e2", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8058, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "sliced-k-text", "type": "text", "x": 120, "y": 615, "width": 266.90000000000003, "height": 106.25, "angle": 0, "strokeColor": "#7f1d1d", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8059, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "v1 sliced-K\nwarp0..3 分别拿 K/V 的不同切片\n每个 warp 得到局部 O\n写 SMEM -> sync -> reduce\n痛点：共享内存读写和同步开销", "fontSize": 17, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "v1 sliced-K\nwarp0..3 分别拿 K/V 的不同切片\n每个 warp 得到局部 O\n写 SMEM -> sync -> reduce\n痛点：共享内存读写和同步开销", "autoResize": true, "lineHeight": 1.25 }, { "id": "sliced-q", "type": "rectangle", "x": 655, "y": 590, "width": 470, "height": 145, "angle": 0, "strokeColor": "#16a34a", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8060, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "sliced-q-text", "type": "text", "x": 680, "y": 615, "width": 294.9500000000001, "height": 106.25, "angle": 0, "strokeColor": "#14532d", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8061, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "v2 sliced-Q\nwarp0..3 分别拿 Q 的不同 row 切片\nK/V 对所有 warp 共享\n每个 warp 直接得到自己的 O 行\n收益：基本不需要 warp 间归并", "fontSize": 17, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "v2 sliced-Q\nwarp0..3 分别拿 Q 的不同 row 切片\nK/V 对所有 warp 共享\n每个 warp 直接得到自己的 O 行\n收益：基本不需要 warp 间归并", "autoResize": true, "lineHeight": 1.25 }, { "id": "partition-arrow", "type": "arrow", "x": 565, "y": 660, "width": 90, "height": 0, "angle": 0, "strokeColor": "#3b82f6", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 3, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8062, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 90, 0 ] ], "startBinding": null, "endBinding": null, "startArrowhead": null, "endArrowhead": "arrow" }, { "id": "move-panel", "type": "rectangle", "x": 60, "y": 830, "width": 1120, "height": 220, "angle": 0, "strokeColor": "#7c3aed", "backgroundColor": "#f3e8ff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8063, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "move-title", "type": "text", "x": 85, "y": 855, "width": 154, "height": 27.5, "angle": 0, "strokeColor": "#6d28d9", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8064, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "线程与搬运节奏", "fontSize": 22, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "线程与搬运节奏", "autoResize": true, "lineHeight": 1.25 }, { "id": "warp-0", "type": "rectangle", "x": 95, "y": 920, "width": 225, "height": 58, "angle": 0, "strokeColor": "#7c3aed", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8065, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "warp-0-t", "type": "text", "x": 110, "y": 938, "width": 156.75000000000003, "height": 18.75, "angle": 0, "strokeColor": "#6d28d9", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8066, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "warp0: Q rows 0..31", "fontSize": 15, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "warp0: Q rows 0..31", "autoResize": true, "lineHeight": 1.25 }, { "id": "warp-1", "type": "rectangle", "x": 355, "y": 920, "width": 225, "height": 58, "angle": 0, "strokeColor": "#7c3aed", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8067, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "warp-1-t", "type": "text", "x": 370, "y": 938, "width": 165.00000000000003, "height": 18.75, "angle": 0, "strokeColor": "#6d28d9", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8068, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "warp1: Q rows 32..63", "fontSize": 15, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "warp1: Q rows 32..63", "autoResize": true, "lineHeight": 1.25 }, { "id": "warp-2", "type": "rectangle", "x": 615, "y": 920, "width": 225, "height": 58, "angle": 0, "strokeColor": "#7c3aed", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8069, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "warp-2-t", "type": "text", "x": 630, "y": 938, "width": 165.00000000000003, "height": 18.75, "angle": 0, "strokeColor": "#6d28d9", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8070, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "warp2: Q rows 64..95", "fontSize": 15, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "warp2: Q rows 64..95", "autoResize": true, "lineHeight": 1.25 }, { "id": "warp-3", "type": "rectangle", "x": 875, "y": 920, "width": 225, "height": 58, "angle": 0, "strokeColor": "#7c3aed", "backgroundColor": "#ffffff", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8071, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "warp-3-t", "type": "text", "x": 890, "y": 938, "width": 173.25000000000003, "height": 18.75, "angle": 0, "strokeColor": "#6d28d9", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8072, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "warp3: Q rows 96..127", "fontSize": 15, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "warp3: Q rows 96..127", "autoResize": true, "lineHeight": 1.25 }, { "id": "move-body", "type": "text", "x": 95, "y": 995, "width": 883.9999999999995, "height": 21.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8073, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "K/V tile 只加载一份到 SMEM，所有 warp 读取；每个 warp 维护自己 Q 行的 m/l/acc，最后直接写对应 O 行。", "fontSize": 17, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "K/V tile 只加载一份到 SMEM，所有 warp 读取；每个 warp 维护自己 Q 行的 m/l/acc，最后直接写对应 O 行。", "autoResize": true, "lineHeight": 1.25 }, { "id": "right-panel", "type": "rectangle", "x": 1220, "y": 135, "width": 490, "height": 915, "angle": 0, "strokeColor": "#f59e0b", "backgroundColor": "#fffbeb", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8074, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "right-title", "type": "text", "x": 1245, "y": 160, "width": 300.29999999999995, "height": 27.5, "angle": 0, "strokeColor": "#b45309", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8075, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "面试讲法：v2 是工程调度升级", "fontSize": 22, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "面试讲法：v2 是工程调度升级", "autoResize": true, "lineHeight": 1.25 }, { "id": "right-body", "type": "text", "x": 1245, "y": 210, "width": 577.1500000000003, "height": 318.75, "angle": 0, "strokeColor": "#78350f", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8076, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "一句话：\nv2 没有改变 exact attention 的数学，主要是让计算更像高效 GEMM。\n\n三个关键词：\n1. fewer non-matmul FLOPs\n Tensor Core 做 matmul 很快，FP32 exp/max/mask 相对昂贵。\n\n2. sequence parallelism\n 长序列、小 batch、少 head 时，不能只靠 B*H 个 CTA。\n\n3. sliced-Q work partitioning\n Q 行天然独立，按 Q 切能减少 shared memory 通信。\n\n过渡到 v3：\nA100 上 v2 已经不错，但 H100 有 TMA/WGMMA/FP8，新硬件能力没吃满。", "fontSize": 17, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "一句话：\nv2 没有改变 exact attention 的数学，主要是让计算更像高效 GEMM。\n\n三个关键词：\n1. fewer non-matmul FLOPs\n Tensor Core 做 matmul 很快，FP32 exp/max/mask 相对昂贵。\n\n2. sequence parallelism\n 长序列、小 batch、少 head 时，不能只靠 B*H 个 CTA。\n\n3. sliced-Q work partitioning\n Q 行天然独立，按 Q 切能减少 shared memory 通信。\n\n过渡到 v3：\nA100 上 v2 已经不错，但 H100 有 TMA/WGMMA/FP8，新硬件能力没吃满。", "autoResize": true, "lineHeight": 1.25 }, { "id": "legend", "type": "rectangle", "x": 60, "y": 1110, "width": 1030, "height": 70, "angle": 0, "strokeColor": "#64748b", "backgroundColor": "#f8fafc", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8077, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "legend-text", "type": "text", "x": 80, "y": 1128, "width": 1109.999999999999, "height": 18.75, "angle": 0, "strokeColor": "#334155", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8078, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "读图方法：蓝色表示 HBM/全局内存，绿色表示 SMEM/SRAM，紫色表示寄存器/累加器，橙色表示 Tensor Core 或 CUDA Core 计算。箭头表示数据搬运或流水依赖。", "fontSize": 15, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "读图方法：蓝色表示 HBM/全局内存，绿色表示 SMEM/SRAM，紫色表示寄存器/累加器，橙色表示 Tensor Core 或 CUDA Core 计算。箭头表示数据搬运或流水依赖。", "autoResize": true, "lineHeight": 1.25 }, { "id": "source", "type": "rectangle", "x": 1220, "y": 1110, "width": 520, "height": 70, "angle": 0, "strokeColor": "#94a3b8", "backgroundColor": "#f8fafc", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "roundness": { "type": 3 }, "seed": 8079, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false }, { "id": "source-text", "type": "text", "x": 1240, "y": 1128, "width": 536.9000000000001, "height": 17.5, "angle": 0, "strokeColor": "#475569", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 8080, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "来源：FlashAttention-2 arXiv:2307.08691；Princeton NLP Group FA2 博客。", "fontSize": 14, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "来源：FlashAttention-2 arXiv:2307.08691；Princeton NLP Group FA2 博客。", "autoResize": true, "lineHeight": 1.25 }, { "id": "practice-interview-panel-flashattention-v2-work-partitioning-原理详解-bg", "type": "rectangle", "x": 80, "y": 1260.0, "width": 1580, "height": 300, "angle": 0, "strokeColor": "#64748b", "backgroundColor": "#e2e8f0", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 70, "groupIds": [], "seed": 93001, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "roundness": { "type": 3 } }, { "id": "practice-interview-panel-flashattention-v2-work-partitioning-原理详解-title", "type": "text", "x": 108, "y": 1282.0, "width": 1524, "height": 38, "angle": 0, "strokeColor": "#1e40af", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 1, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 91001, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "练习题与面试题补充：FlashAttention IO-aware attention", "fontSize": 24, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "练习题与面试题补充：FlashAttention IO-aware attention", "autoResize": false, "lineHeight": 1.25 }, { "id": "practice-interview-panel-flashattention-v2-work-partitioning-原理详解-practice-box", "type": "rectangle", "x": 108, "y": 1338.0, "width": 748.0, "height": 190, "angle": 0, "strokeColor": "#2563eb", "backgroundColor": "#bfdbfe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 93002, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "roundness": { "type": 3 } }, { "id": "practice-interview-panel-flashattention-v2-work-partitioning-原理详解-practice-text", "type": "text", "x": 130, "y": 1358.0, "width": 725.0, "height": 150, "angle": 0, "strokeColor": "#243042", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 1, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 91001, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "练习题\n1. 手推一个 Q block 扫两个 K/V tile 的 online softmax 更新\n2. 画出 S tile、P tile、O tile 的生命周期\n3. 比较 v1/v2/v3/v4 在 work partition 和 pipeline 上的变化", "fontSize": 17, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "练习题\n1. 手推一个 Q block 扫两个 K/V tile 的 online softmax 更新\n2. 画出 S tile、P tile、O tile 的生命周期\n3. 比较 v1/v2/v3/v4 在 work partition 和 pipeline 上的变化", "autoResize": false, "lineHeight": 1.25 }, { "id": "practice-interview-panel-flashattention-v2-work-partitioning-原理详解-interview-box", "type": "rectangle", "x": 884.0, "y": 1338.0, "width": 748.0, "height": 190, "angle": 0, "strokeColor": "#7c3aed", "backgroundColor": "#ddd6fe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 93003, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "roundness": { "type": 3 } }, { "id": "practice-interview-panel-flashattention-v2-work-partitioning-原理详解-interview-text", "type": "text", "x": 906.0, "y": 1358.0, "width": 725.0, "height": 150, "angle": 0, "strokeColor": "#243042", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 1, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "seed": 91001, "version": 1, "isDeleted": false, "boundElements": null, "updated": 1, "link": null, "locked": false, "text": "面试题\n1. FlashAttention 为什么降低 HBM traffic\n2. online softmax 的 m/l/O 三个状态如何保证数值正确\n3. v2 的 work partitioning 相比 v1 改善了什么", "fontSize": 17, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "面试题\n1. FlashAttention 为什么降低 HBM traffic\n2. online softmax 的 m/l/O 三个状态如何保证数值正确\n3. v2 的 work partitioning 相比 v1 改善了什么", "autoResize": false, "lineHeight": 1.25 } ], "appState": { "gridSize": null, "viewBackgroundColor": "#ffffff" }, "files": {} }