{
  "type": "excalidraw",
  "version": 2,
  "source": "https://excalidraw.com",
  "elements": [
    {
      "id": "bg-top",
      "type": "rectangle",
      "x": 30,
      "y": 20,
      "width": 1740,
      "height": 86,
      "angle": 0,
      "strokeColor": "#ea580c",
      "backgroundColor": "#f8fafc",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8155,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "title",
      "type": "text",
      "x": 60,
      "y": 39,
      "width": 1468.4999999999982,
      "height": 37.5,
      "angle": 0,
      "strokeColor": "#ea580c",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8156,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "FlashAttention v4  Blackwell pipeline co-design: fully async MMA + TMEM + softmax rethink",
      "fontSize": 30,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "FlashAttention v4  Blackwell pipeline co-design: fully async MMA + TMEM + softmax rethink",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "subtitle",
      "type": "text",
      "x": 60,
      "y": 75,
      "width": 736.95,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8157,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "目标：B200/GB200 上 Tensor Core 增长更快，瓶颈转向 softmax、SMEM 带宽和流水线调度。",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "目标：B200/GB200 上 Tensor Core 增长更快，瓶颈转向 softmax、SMEM 带宽和流水线调度。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "core-panel",
      "type": "rectangle",
      "x": 60,
      "y": 135,
      "width": 520,
      "height": 370,
      "angle": 0,
      "strokeColor": "#ea580c",
      "backgroundColor": "#ffedd5",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8158,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "core-title",
      "type": "text",
      "x": 85,
      "y": 160,
      "width": 202.39999999999998,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#c2410c",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8159,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "为什么 v4 还要重写",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "为什么 v4 还要重写",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "core-body",
      "type": "text",
      "x": 85,
      "y": 205,
      "width": 517.6500000000002,
      "height": 276.25,
      "angle": 0,
      "strokeColor": "#7c2d12",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8160,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "Blackwell 的非对称扩展：\nTensor Core BF16 吞吐大幅提升\n但 shared memory 带宽、exp 单元等没有同幅提升\n\n结果：\n单纯把 v3 搬过来，matmul 更快了，softmax 和数据路径更显眼。\n\nv4 的方向：\n更深的 fully async MMA pipeline\n更大的 tile，提高 Tensor Core 连续工作量\nTMEM 存中间矩阵，减少 SMEM 往返\n软件 exp + 条件 rescale，压低非 matmul 开销\n反向用 2-CTA MMA 降低 dQ 原子累加和 SMEM 流量",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "Blackwell 的非对称扩展：\nTensor Core BF16 吞吐大幅提升\n但 shared memory 带宽、exp 单元等没有同幅提升\n\n结果：\n单纯把 v3 搬过来，matmul 更快了，softmax 和数据路径更显眼。\n\nv4 的方向：\n更深的 fully async MMA pipeline\n更大的 tile，提高 Tensor Core 连续工作量\nTMEM 存中间矩阵，减少 SMEM 往返\n软件 exp + 条件 rescale，压低非 matmul 开销\n反向用 2-CTA MMA 降低 dQ 原子累加和 SMEM 流量",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "pseudo-panel",
      "type": "rectangle",
      "x": 620,
      "y": 135,
      "width": 570,
      "height": 470,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "#dbeafe",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8161,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "pseudo-title",
      "type": "text",
      "x": 645,
      "y": 160,
      "width": 558.8000000000004,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#1e40af",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8162,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "概念伪代码：显式调度 main / softmax / correction",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "概念伪代码：显式调度 main / softmax / correction",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "pseudo-code",
      "type": "text",
      "x": 645,
      "y": 205,
      "width": 363.0000000000002,
      "height": 468.75,
      "angle": 0,
      "strokeColor": "#1e3a8a",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8163,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "for q_tile i:\n  init m,l,acc in registers/TMEM\n  prefetch Q,K,V stages\n\n  for k_tile j in pipelined order:\n    launch async MMA: S_j = Q @ K_j.T\n    while MMA runs:\n      schedule softmax for previous S\n      maybe skip rescale if max change small\n\n    softmax stage:\n      m_new = max(m, rowmax(S_j))\n      if need_rescale(m, m_new):\n        acc *= exp_approx(m - m_new)\n        l *= exp_approx(m - m_new)\n      P = exp_approx(S_j - m_new)\n\n    launch async MMA: acc += P @ V_j\n    keep intermediates in TMEM when possible\n\n  final normalize O = acc / l\n\nbackward idea:\n  2-CTA MMA cooperates on larger tile\n  reduce SMEM traffic and dQ atomics",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "for q_tile i:\n  init m,l,acc in registers/TMEM\n  prefetch Q,K,V stages\n\n  for k_tile j in pipelined order:\n    launch async MMA: S_j = Q @ K_j.T\n    while MMA runs:\n      schedule softmax for previous S\n      maybe skip rescale if max change small\n\n    softmax stage:\n      m_new = max(m, rowmax(S_j))\n      if need_rescale(m, m_new):\n        acc *= exp_approx(m - m_new)\n        l *= exp_approx(m - m_new)\n      P = exp_approx(S_j - m_new)\n\n    launch async MMA: acc += P @ V_j\n    keep intermediates in TMEM when possible\n\n  final normalize O = acc / l\n\nbackward idea:\n  2-CTA MMA cooperates on larger tile\n  reduce SMEM traffic and dQ atomics",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "pipeline-panel",
      "type": "rectangle",
      "x": 60,
      "y": 545,
      "width": 1130,
      "height": 330,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "#f8fafc",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8164,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "pipeline-title",
      "type": "text",
      "x": 85,
      "y": 570,
      "width": 636.9000000000003,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#334155",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8165,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "Blackwell 数据路径：减少 SMEM 压力，把 Tensor Core 喂满",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "Blackwell 数据路径：减少 SMEM 压力，把 Tensor Core 喂满",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v4-lane-lane-0",
      "type": "rectangle",
      "x": 95,
      "y": 630,
      "width": 245,
      "height": 72,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "#dbeafe",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8166,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "v4-lane-lane-text-0",
      "type": "text",
      "x": 113,
      "y": 647,
      "width": 90,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8167,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "HBM\nQ/K/V/O",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "HBM\nQ/K/V/O",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v4-lane-lane-1",
      "type": "rectangle",
      "x": 372,
      "y": 630,
      "width": 245,
      "height": 72,
      "angle": 0,
      "strokeColor": "#16a34a",
      "backgroundColor": "#dcfce7",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8168,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "v4-lane-lane-text-1",
      "type": "text",
      "x": 390,
      "y": 647,
      "width": 90,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#16a34a",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8169,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "SMEM\nstaging",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "SMEM\nstaging",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v4-lane-lane-2",
      "type": "rectangle",
      "x": 649,
      "y": 630,
      "width": 245,
      "height": 72,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#f3e8ff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8170,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "v4-lane-lane-text-2",
      "type": "text",
      "x": 667,
      "y": 647,
      "width": 90,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8171,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "TMEM\nMMA accum",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "TMEM\nMMA accum",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v4-lane-lane-3",
      "type": "rectangle",
      "x": 926,
      "y": 630,
      "width": 245,
      "height": 72,
      "angle": 0,
      "strokeColor": "#ea580c",
      "backgroundColor": "#ffedd5",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8172,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "v4-lane-lane-text-3",
      "type": "text",
      "x": 944,
      "y": 647,
      "width": 140.24999999999997,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#ea580c",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8173,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "Tensor Core\nfully async MMA",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "Tensor Core\nfully async MMA",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "v4-lane-arrow-0",
      "type": "arrow",
      "x": 340,
      "y": 666,
      "width": 32,
      "height": 0,
      "angle": 0,
      "strokeColor": "#3b82f6",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8174,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          32,
          0
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "v4-lane-arrow-1",
      "type": "arrow",
      "x": 617,
      "y": 666,
      "width": 32,
      "height": 0,
      "angle": 0,
      "strokeColor": "#3b82f6",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8175,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          32,
          0
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "v4-lane-arrow-2",
      "type": "arrow",
      "x": 894,
      "y": 666,
      "width": 32,
      "height": 0,
      "angle": 0,
      "strokeColor": "#3b82f6",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8176,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          32,
          0
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "pipeline-note",
      "type": "text",
      "x": 95,
      "y": 740,
      "width": 1139.8499999999992,
      "height": 42.5,
      "angle": 0,
      "strokeColor": "#334155",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8177,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "讲解重点：v4 不只是 tile 更大。Blackwell 上 MMA、load 更异步，TMEM 靠近 Tensor Core 存中间结果；这样避免很多中间量反复进出 SMEM。\n软 softmax 的意义：exp 和 rescale 变成瓶颈后，用近似/软件化分担特殊函数单元压力；条件 rescale 则跳过不必要的旧 acc/l 重缩放。",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "讲解重点：v4 不只是 tile 更大。Blackwell 上 MMA、load 更异步，TMEM 靠近 Tensor Core 存中间结果；这样避免很多中间量反复进出 SMEM。\n软 softmax 的意义：exp 和 rescale 变成瓶颈后，用近似/软件化分担特殊函数单元压力；条件 rescale 则跳过不必要的旧 acc/l 重缩放。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "thread-panel",
      "type": "rectangle",
      "x": 1220,
      "y": 135,
      "width": 500,
      "height": 740,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#f3e8ff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8178,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "thread-title",
      "type": "text",
      "x": 1245,
      "y": 160,
      "width": 148.49999999999997,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#6d28d9",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8179,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "线程/CTA 视角",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "线程/CTA 视角",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-0",
      "type": "rectangle",
      "x": 1245,
      "y": 215,
      "width": 430,
      "height": 82,
      "angle": 0,
      "strokeColor": "#16a34a",
      "backgroundColor": "#dcfce7",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8180,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "item-h-0",
      "type": "text",
      "x": 1265,
      "y": 229,
      "width": 158.95,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#16a34a",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8181,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "Producer / loader",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "Producer / loader",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-b-0",
      "type": "text",
      "x": 1265,
      "y": 257,
      "width": 319.5000000000001,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8182,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "异步搬 Q/K/V tile，维护 staging buffer。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "异步搬 Q/K/V tile，维护 staging buffer。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-a-0",
      "type": "arrow",
      "x": 1460,
      "y": 297,
      "width": 0,
      "height": 32,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8183,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          0,
          32
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "item-1",
      "type": "rectangle",
      "x": 1245,
      "y": 330,
      "width": 430,
      "height": 82,
      "angle": 0,
      "strokeColor": "#ea580c",
      "backgroundColor": "#ffedd5",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8184,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "item-h-1",
      "type": "text",
      "x": 1265,
      "y": 344,
      "width": 121.54999999999998,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#ea580c",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8185,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "MMA warpgroup",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "MMA warpgroup",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-b-1",
      "type": "text",
      "x": 1265,
      "y": 372,
      "width": 396.0000000000001,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8186,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "启动 fully async QK 和 PV MMA，结果优先留在 TMEM。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "启动 fully async QK 和 PV MMA，结果优先留在 TMEM。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-a-1",
      "type": "arrow",
      "x": 1460,
      "y": 412,
      "width": 0,
      "height": 32,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8187,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          0,
          32
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "item-2",
      "type": "rectangle",
      "x": 1245,
      "y": 445,
      "width": 430,
      "height": 82,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#ffffff",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8188,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "item-h-2",
      "type": "text",
      "x": 1265,
      "y": 459,
      "width": 158.95,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8189,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "Softmax warpgroup",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "Softmax warpgroup",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-b-2",
      "type": "text",
      "x": 1265,
      "y": 487,
      "width": 293.25000000000006,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8190,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "处理 rowmax、exp、sum、条件 rescale。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "处理 rowmax、exp、sum、条件 rescale。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-a-2",
      "type": "arrow",
      "x": 1460,
      "y": 527,
      "width": 0,
      "height": 32,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8191,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          0,
          32
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "item-3",
      "type": "rectangle",
      "x": 1245,
      "y": 560,
      "width": 430,
      "height": 82,
      "angle": 0,
      "strokeColor": "#0ea5e9",
      "backgroundColor": "#e0f2fe",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8192,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "item-h-3",
      "type": "text",
      "x": 1265,
      "y": 574,
      "width": 149.6,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#0ea5e9",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8193,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "Correction stage",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "Correction stage",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-b-3",
      "type": "text",
      "x": 1265,
      "y": 602,
      "width": 300.75000000000006,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8194,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "当 max 变化需要校正时，再缩放旧 acc/l。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "当 max 变化需要校正时，再缩放旧 acc/l。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-a-3",
      "type": "arrow",
      "x": 1460,
      "y": 642,
      "width": 0,
      "height": 32,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8195,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "points": [
        [
          0,
          0
        ],
        [
          0,
          32
        ]
      ],
      "startBinding": null,
      "endBinding": null,
      "startArrowhead": null,
      "endArrowhead": "arrow"
    },
    {
      "id": "item-4",
      "type": "rectangle",
      "x": 1245,
      "y": 675,
      "width": 430,
      "height": 82,
      "angle": 0,
      "strokeColor": "#f59e0b",
      "backgroundColor": "#fffbeb",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8196,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "item-h-4",
      "type": "text",
      "x": 1265,
      "y": 689,
      "width": 130.89999999999998,
      "height": 21.25,
      "angle": 0,
      "strokeColor": "#f59e0b",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8197,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "Backward 2-CTA",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "Backward 2-CTA",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "item-b-4",
      "type": "text",
      "x": 1265,
      "y": 717,
      "width": 381.0000000000002,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#374151",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8198,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "两个 CTA 协同更大 MMA tile，减少 dQ atomic add。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "两个 CTA 协同更大 MMA tile，减少 dQ atomic add。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "compare-panel",
      "type": "rectangle",
      "x": 60,
      "y": 920,
      "width": 1650,
      "height": 135,
      "angle": 0,
      "strokeColor": "#f59e0b",
      "backgroundColor": "#fffbeb",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8199,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "compare-title",
      "type": "text",
      "x": 85,
      "y": 945,
      "width": 272.79999999999995,
      "height": 27.5,
      "angle": 0,
      "strokeColor": "#b45309",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8200,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "从 v3 到 v4 的面试对比句",
      "fontSize": 22,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "从 v3 到 v4 的面试对比句",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "compare-body",
      "type": "text",
      "x": 85,
      "y": 990,
      "width": 1522.7999999999984,
      "height": 45,
      "angle": 0,
      "strokeColor": "#78350f",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8201,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "v3 的关键词是 Hopper 异步化：TMA/WGMMA + warp specialization，解决搬运和计算重叠。\nv4 的关键词是 Blackwell 非对称瓶颈：Tensor Core 太快后，softmax、SMEM、原子累加和调度成为主要问题，所以要 TMEM、更深 pipeline、软件 exp、条件 rescale、2-CTA MMA。",
      "fontSize": 18,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "v3 的关键词是 Hopper 异步化：TMA/WGMMA + warp specialization，解决搬运和计算重叠。\nv4 的关键词是 Blackwell 非对称瓶颈：Tensor Core 太快后，softmax、SMEM、原子累加和调度成为主要问题，所以要 TMEM、更深 pipeline、软件 exp、条件 rescale、2-CTA MMA。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "legend",
      "type": "rectangle",
      "x": 60,
      "y": 1110,
      "width": 1030,
      "height": 70,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "#f8fafc",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "dashed",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8202,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "legend-text",
      "type": "text",
      "x": 80,
      "y": 1128,
      "width": 1109.999999999999,
      "height": 18.75,
      "angle": 0,
      "strokeColor": "#334155",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8203,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "读图方法：蓝色表示 HBM/全局内存，绿色表示 SMEM/SRAM，紫色表示寄存器/累加器，橙色表示 Tensor Core 或 CUDA Core 计算。箭头表示数据搬运或流水依赖。",
      "fontSize": 15,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "读图方法：蓝色表示 HBM/全局内存，绿色表示 SMEM/SRAM，紫色表示寄存器/累加器，橙色表示 Tensor Core 或 CUDA Core 计算。箭头表示数据搬运或流水依赖。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "source",
      "type": "rectangle",
      "x": 1220,
      "y": 1110,
      "width": 520,
      "height": 70,
      "angle": 0,
      "strokeColor": "#94a3b8",
      "backgroundColor": "#f8fafc",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "dashed",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "roundness": {
        "type": 3
      },
      "seed": 8204,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false
    },
    {
      "id": "source-text",
      "type": "text",
      "x": 1240,
      "y": 1128,
      "width": 552.3000000000001,
      "height": 17.5,
      "angle": 0,
      "strokeColor": "#475569",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 8205,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "来源：FlashAttention-4 arXiv:2603.05451；PyTorch FlexAttention/FA4 博客。",
      "fontSize": 14,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "来源：FlashAttention-4 arXiv:2603.05451；PyTorch FlexAttention/FA4 博客。",
      "autoResize": true,
      "lineHeight": 1.25
    },
    {
      "id": "practice-interview-panel-flashattention-v4-blackwell-pipeline-原理详解-bg",
      "type": "rectangle",
      "x": 80,
      "y": 1260.0,
      "width": 1580,
      "height": 300,
      "angle": 0,
      "strokeColor": "#64748b",
      "backgroundColor": "#e2e8f0",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 70,
      "groupIds": [],
      "seed": 93001,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "roundness": {
        "type": 3
      }
    },
    {
      "id": "practice-interview-panel-flashattention-v4-blackwell-pipeline-原理详解-title",
      "type": "text",
      "x": 108,
      "y": 1282.0,
      "width": 1524,
      "height": 38,
      "angle": 0,
      "strokeColor": "#1e40af",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 1,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 91001,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "练习题与面试题补充：FlashAttention IO-aware attention",
      "fontSize": 24,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "练习题与面试题补充：FlashAttention IO-aware attention",
      "autoResize": false,
      "lineHeight": 1.25
    },
    {
      "id": "practice-interview-panel-flashattention-v4-blackwell-pipeline-原理详解-practice-box",
      "type": "rectangle",
      "x": 108,
      "y": 1338.0,
      "width": 748.0,
      "height": 190,
      "angle": 0,
      "strokeColor": "#2563eb",
      "backgroundColor": "#bfdbfe",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 93002,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "roundness": {
        "type": 3
      }
    },
    {
      "id": "practice-interview-panel-flashattention-v4-blackwell-pipeline-原理详解-practice-text",
      "type": "text",
      "x": 130,
      "y": 1358.0,
      "width": 725.0,
      "height": 150,
      "angle": 0,
      "strokeColor": "#243042",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 1,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 91001,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "练习题\n1. 手推一个 Q block 扫两个 K/V tile 的 online softmax 更新\n2. 画出 S tile、P tile、O tile 的生命周期\n3. 比较 v1/v2/v3/v4 在 work partition 和 pipeline 上的变化",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "练习题\n1. 手推一个 Q block 扫两个 K/V tile 的 online softmax 更新\n2. 画出 S tile、P tile、O tile 的生命周期\n3. 比较 v1/v2/v3/v4 在 work partition 和 pipeline 上的变化",
      "autoResize": false,
      "lineHeight": 1.25
    },
    {
      "id": "practice-interview-panel-flashattention-v4-blackwell-pipeline-原理详解-interview-box",
      "type": "rectangle",
      "x": 884.0,
      "y": 1338.0,
      "width": 748.0,
      "height": 190,
      "angle": 0,
      "strokeColor": "#7c3aed",
      "backgroundColor": "#ddd6fe",
      "fillStyle": "solid",
      "strokeWidth": 2,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 93003,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "roundness": {
        "type": 3
      }
    },
    {
      "id": "practice-interview-panel-flashattention-v4-blackwell-pipeline-原理详解-interview-text",
      "type": "text",
      "x": 906.0,
      "y": 1358.0,
      "width": 725.0,
      "height": 150,
      "angle": 0,
      "strokeColor": "#243042",
      "backgroundColor": "transparent",
      "fillStyle": "solid",
      "strokeWidth": 1,
      "strokeStyle": "solid",
      "roughness": 1,
      "opacity": 100,
      "groupIds": [],
      "seed": 91001,
      "version": 1,
      "isDeleted": false,
      "boundElements": null,
      "updated": 1,
      "link": null,
      "locked": false,
      "text": "面试题\n1. FlashAttention 为什么降低 HBM traffic\n2. online softmax 的 m/l/O 三个状态如何保证数值正确\n3. v2 的 work partitioning 相比 v1 改善了什么",
      "fontSize": 17,
      "fontFamily": 5,
      "textAlign": "left",
      "verticalAlign": "top",
      "containerId": null,
      "originalText": "面试题\n1. FlashAttention 为什么降低 HBM traffic\n2. online softmax 的 m/l/O 三个状态如何保证数值正确\n3. v2 的 work partitioning 相比 v1 改善了什么",
      "autoResize": false,
      "lineHeight": 1.25
    }
  ],
  "appState": {
    "gridSize": null,
    "viewBackgroundColor": "#ffffff"
  },
  "files": {}
}