Fast Dispatch / CQ Firmware on Blackhole

0. System overview

The pipeline has three cooperating actors:

1. Host — runs the Python in cq.py. Writes commands into a pinned hugepage ("issue region") and pushes 2-byte size entries onto a 16-bit prefetch_q in the prefetch core's L1.
2. Prefetch core (BRISC on (PREFETCH_NOC_X, PREFETCH_NOC_Y), NOC 0) — reads from host hugepage over PCIe-NOC, lands data in its cmddat_q, relays commands to the dispatch core's CB. Built from cq_prefetch.cpp.
3. Dispatch core (BRISC on (DISPATCH_NOC_X, DISPATCH_NOC_Y), NOC 1, with a co-resident NCRISC "subordinate") — drains the relayed command stream, executes individual dispatch commands. Built from cq_dispatch.cpp (BRISC) and cq_dispatch_subordinate.cpp (NCRISC).

The board layout is wired in device.py:

P100: tensix_x = 1..7, 10..14   prefetch=(14,2)  dispatch=(14,3)
P150: tensix_x = 1..7, 10..16   prefetch=(16,2)  dispatch=(16,3)
PCIE NOC: x=19  y=24   (absolute; not subject to NOC mirroring)

build.sh compiles the CQ kernels with -DPREFETCH_NOC_X/Y, -DDISPATCH_NOC_X/Y, -DPCIE_NOC_X=19 -DPCIE_NOC_Y=24 and -DNOC_INDEX={0,1}. Prefetch BRISC is NOC 0; dispatch BRISC + the subordinate NCRISC are both NOC 1.

1. Memory map and pointers

cq_fixed_config.hpp is the source of truth for L1 layout. Three include-guarded blocks select per processor.

1.1 Prefetch core L1 (NOC 0 BRISC)

Semaphores (16 B aligned, on the FD core type):

• MY_DOWNSTREAM_CB_SEM_ID = 0 — page credits granted by dispatch_d.
• MY_DISPATCH_S_CB_SEM_ID = 1 — page credits granted by dispatch_s.
• DOWNSTREAM_SYNC_SEM_ID = 2 — counter dispatch_d increments to unblock prefetch's STALL.
• MY_UPSTREAM_CB_SEM_ID = 3 — only used in _d variants (split path).

1.2 Dispatch core L1 (NOC 1 BRISC, "dispatch_d")

1.3 Dispatch subordinate L1 (NCRISC, NOC 1)

Co-resident with dispatch_d on the same tile. CB at 0x9A000, 32 KiB, log_page_size=8. Worker-completion stream registers live in the NOC stream register file from FIRST_STREAM_USED = 48. DISPATCH_S_SYNC_SEM_BASE_ADDR = 0x19710 — array of 16-byte-strided semaphores bumped by dispatch_d so dispatch_s knows when it's safe to send a go signal.

1.4 Host sysmem layout

+0                       (header / reserved)
+2 * PCIE_ALIGN          _HOST_CQ_WR_OFF      (4 B: completion-fifo wr-ptr | toggle)
+3 * PCIE_ALIGN          _HOST_CQ_RD_OFF      (4 B: host's rd-ptr | toggle echo)
+4 * PCIE_ALIGN          _HOST_ISSUE_BASE     issue region, 64 MiB
+   ...                  _HOST_COMPLETION_BASE  completion region, 32 MiB
+   ...                  _HOST_TIMESTAMP_BASE   4096 × 16 B   (CQTimestamp slots)
+   ...                  _HOST_CORE_TIMING_BASE 4096 × 16 B   (per-core BRISC start/end)

PCIE_ALIGN = 64,  L1_ALIGN = 16,  PAGE_SIZE = 4096

2. Host-side command construction (cq.py)

2.1 Commands

Every dispatch command is a 16-byte fixed-size header followed by an L1-aligned payload. The byte layout is defined by cq_commands.hpp and produced on the host by _cq_hdr(fmt, ...) which struct-packs and zero-pads to 16 bytes.

2.2 RELAY_INLINE wrapping

Each top-level dispatch command is wrapped on the host into a RELAY_INLINE prefetch command:

def _relay_inline(payload):
  stride = align_up(CQ_CMD_SIZE + len(payload), PCIE_ALIGN)
  hdr = _cq_hdr("<BBHII", _RELAY_INLINE, 0, 0, len(payload), stride)
  return hdr + payload.ljust(stride - CQ_CMD_SIZE, b"\0")

2.3 prefetch_q ring

2.4 Issue region writes

1. Bump _issue_wr to next PCIE_ALIGN boundary.
2. If record won't fit, wrap to 0.
3. Copy record into sysmem at _HOST_ISSUE_BASE + _issue_wr.
4. _issue_wr += len(record).
5. Spin waiting until prefetch_q[idx] == 0.
6. Atomically write 2-byte size (record_len >> 4) into prefetch_q[idx].
7. idx = (idx + 1) % CQ_PREFETCH_Q_ENTRIES.

2.5 Lowering programs (dispatch.py)

build_ir(program, ...) produces a list of IRCommands; _lower_ir then converts:

Write(cores, addr, bytes)         → CQWritePackedLarge + CQBarrier  (per 1 KiB chunk)
Write(cores, addr, list[bytes])   → CQWritePacked                   (per L1_ALIGN slice)
Launch(cores)                     → CQSetGoSignalNocData(cores)
                                    CQWaitStream(DONE_STREAM=48, 0)
                                    CQSendGoSignal(go_word, 48, 0, len(cores))
                                    CQWaitStream(DONE_STREAM, len(cores))

lower_fast(programs, ...) bookends each per-program IR with CQTimestamp writes. device._run_fast_dispatch() appends a single CQHostEvent — that is the event the host waits on.

3. Prefetcher startup and main loops

The prefetch firmware lives in cq_prefetch.cpp. It compiles to three variants (_h, _d, _hd) but for blackhole-py-old only the unified _hd is used because IS_H_VARIANT = IS_D_VARIANT = 1.

void kernel_main_hd() {
    uint32_t cmd_ptr = cmddat_q_base;
    uint32_t fence   = cmddat_q_base;
    bool done = false;
    uint32_t l1_cache[l1_cache_elements_rounded];
    PrefetchExecBufState exec_buf_state;

    cq_noc_async_write_init_state<...>(0, dst_dispatch, 0);    // BRISC_WR_CMD_BUF
    cq_noc_async_write_init_state<...>(0, dst_dispatch_s, 0);  // BRISC_WR_REG_CMD_BUF

    while (!done) {
        fetch_q_get_cmds<0>(fence, cmd_ptr, pcie_read_ptr);
        cmd = (volatile CQPrefetchCmd*)cmd_ptr;
        uint32_t stride;
        done = process_cmd<false, false>(cmd_ptr, downstream_data_ptr, stride, l1_cache, exec_buf_state);
        cmd_ptr += stride;
    }
}

3.1 process_cmd (prefetcher)

3.2 process_relay_inline_common — the hot path

uint32_t length = cmd->relay_inline.length;
uint32_t stride = cmd->relay_inline.stride;
uint32_t npages = (length + page_size - 1) >> log_page_size;

RelayInlineState::cb_writer.acquire_pages(npages);     // wait on credits from dispatch
CmdPayloadStream stream{cmd_ptr, exec_buf_state};
stream.init(sizeof(CQPrefetchCmd));                    // skip 16 B prefetch header
while (remaining > 0) {
    ByteSpan span = stream.next_span(remaining, stride);
    write_downstream<...>(span.ptr, local_downstream_data_ptr, span.len, ..., );
    remaining -= span.len;
}
local_downstream_data_ptr = round_up_pow2(local_downstream_data_ptr, page_size);
noc_async_writes_flushed(RelayInlineState::downstream_noc_index);
RelayInlineState::cb_writer.release_pages(npages, local_downstream_data_ptr);

4. Dispatch core (master, BRISC NOC 1)

void kernel_main() {
    set_l1_data_cache<true>();
    if constexpr (NOC_INDEX != UPSTREAM_NOC_INDEX) noc_local_state_init(UPSTREAM_NOC_INDEX);
    reset_worker_completion_stream_counts<FIRST_STREAM_USED=48, MAX_NUM_WORKER_SEMS=8>();
    dispatch_cb_reader.init();
    cmd_ptr = DISPATCH_CB_BASE;
    write_offset[0..2] = 0;
    completion_queue_wr_ptr_and_toggle = *get_cq_completion_write_ptr();
    while (!done) {
        dispatch_cb_reader.wait_for_available_data_and_release_old_pages(cmd_ptr);
        done = process_cmd<IS_D_VARIANT=true>(cmd_ptr, l1_cache);
        cmd_ptr = round_up_pow2(cmd_ptr, dispatch_cb_page_size);
    }
}

4.1 The dispatch CB — 4-block ring with deferred release

CBReaderWithReleasePolicy (defined in cq_common.hpp) maintains:

• local_count_ — pages logically consumed.
• cb_fence_ — byte address one past last guaranteed-present byte.
• block_next_start_addr_[4] — block boundary fences.
• rd_block_idx_ — current block (0..3).
• block_noc_writes_to_clear_ — snapshot of NOC nonposted-write counter.

4.2 process_cmd (dispatcher)

4.3 The hot path: process_write_packed

4.4 process_write_host_h (host event / read-back)

data_ptr = cmd_ptr;     // include cmd header so host can parse it
wlength  = cmd->write_linear_host.length;
init NOC state to pcie_noc_xy.
while (wlength != 0) {
    chunk_len = min(wlength, ~(dispatch_cb_page_size - 1));
    while (chunk_len != 0) {
        wait_for_data_in_cb(data_ptr);
        xfer = min(chunk_len, available_in_cb);
        npages = div_up(xfer, completion_page_size);
        completion_queue_reserve_back(npages);          // blocks on rd_ptr
        wr_addr = cq_write_interface.completion_fifo_wr_ptr << 4;
        if (wr_addr + xfer > completion_queue_end) { split + wrap; }
        write to (pcie_noc_xy, wr_addr) of size xfer
        completion_queue_push_back(npages);             // updates ptr in sysmem
        data_ptr += xfer; chunk_len -= xfer; wlength -= xfer;
    }
}

4.5 Sync commands (WAIT flags)

• BARRIER (0x01) → noc_async_write_barrier()
• WAIT_MEMORY (0x04) → spin until *addr >= count (using wrap_ge)
• WAIT_STREAM (0x08) → spin on stream's SPACE_AVAILABLE_REG until >= count
• CLEAR_STREAM (0x10) → atomically zero the stream by writing -(*sem) to the update reg
• NOTIFY_PREFETCH (0x02) → noc_semaphore_inc(prefetch's UPSTREAM_SYNC_SEM, 1)

5. Dispatch subordinate (NCRISC, NOC 1)

cq_dispatch_subordinate.cpp lives on the same tensix as dispatch_d but runs on the NCRISC. It uses NOC 1 with a customized command-buffer allocation so BRISC and NCRISC never collide:

In blackhole-py-old's fixed config DISTRIBUTED_DISPATCHER = 0, so the host never sends go-signal commands as DISPATCH_SUBORDINATE. The dispatch_s mostly idles after start-up — kept fully functional only so future split configurations can drop in.

6. Launch flow — how a program actually starts

_lower_ir lowers a Launch(cores) to four CQ commands in order:

CQSetGoSignalNocData(cores)                 # 17: SET_GO_SIGNAL_NOC_DATA
CQWaitStream(DONE_STREAM=48, 0)             # 7 + WAIT_STREAM + CLEAR_STREAM
CQSendGoSignal(go_word, 48, 0, len(cores))  # 14: SEND_GO_SIGNAL  (num_unicast = N)
CQWaitStream(DONE_STREAM, len(cores))       # 7 + WAIT_STREAM + CLEAR_STREAM

The go_word is built in device._go_word():

go = GoMsg()
go.bits.signal = DevMsgs.RUN_MSG_GO          # = 0x80
go.bits.master_x, go.bits.master_y = DISPATCH_NOC_X, DISPATCH_NOC_Y
go.bits.dispatch_message_offset = 0
return go.all                                # uint32: [signal | mx | my | offset]

6.1 SET_GO_SIGNAL_NOC_DATA

Copies a list of NOC xy encodings into go_signal_noc_data[256]. Subsequent SEND_GO_SIGNAL walks the array starting at noc_data_start_index.

6.2 The WAIT(stream=48, 0) + CLEAR

Trivially passes the WAIT check (≥0 always true), then atomically zeroes the stream counter by writing -(*sem) << REMOTE_DEST_BUF_WORDS_FREE_INC to the update reg. Streams are accumulator-style registers, not normal memory.

6.3 SEND_GO_SIGNAL

for (i = 0; i < num_unicasts; ++i) {
    dst = get_noc_addr_helper(go_signal_noc_data[idx++], UNICAST_GO_SIGNAL_ADDR);
    noc_async_write_one_packet(aligned_go_signal_storage, dst, sizeof(uint32_t));
}

The go word lands at 0x370 on each worker tensix. The worker BRISC spins on that address, sees signal == RUN_MSG_GO (0x80), starts the program; when done it writes back RUN_MSG_DONE = 0 and bumps stream 48 of the dispatch core by 1.

6.4 WAIT(stream=48, N) + CLEAR

Dispatcher spins until stream 48 reaches N, then clears it. This is the "bottom of cycle" fence — after it returns, the next launch can proceed.

7. Host-side completion path

while True:
  wr_raw = self._sysmem_read32(_HOST_CQ_WR_OFF)
  wr_16b, wr_toggle = wr_raw & 0x7FFF_FFFF, (wr_raw >> 31) & 1
  if (wr_16b, wr_toggle) != (self._completion_rd_16b, self._completion_rd_toggle):
    off = (self._completion_rd_16b << 4) - self.noc_local
    got = self._sysmem_read32(off + 16)            # skip echoed header
    self._completion_rd_16b += self._completion_page_16b
    if self._completion_rd_16b >= self._completion_end_16b:
      self._completion_rd_16b = self._completion_base_16b
      self._completion_rd_toggle ^= 1
    raw = (self._completion_rd_16b & 0x7FFF_FFFF) | (self._completion_rd_toggle << 31)
    self._dispatch_win.write32(CQ_COMPLETION_RD_PTR, raw)
    self._sysmem_write32(_HOST_CQ_RD_OFF, raw)
    if got != (event_id & 0xFFFFFFFF):
      raise RuntimeError("event mismatch")
    return
  time.sleep(0.0002)

8. Putting it all end-to-end

A concrete trace for device.run() on one program with N cores:

1. _compile_ir(program, ...) builds IR: Write(go_msg reset) + Write(go_msg_index) + Write(uniform_rta) + Write(launch, role) + Write(shared, role) + Launch(cores).
2. lower_fast prepends a CQTimestamp for slot 0, lowers IR to CQ commands, appends CQTimestamp for slot 1, then CQHostEvent(event_id).
3. Each CQ command is to_bytes()-serialized, wrapped in _relay_inline, appended to cq.stream.
4. CQSysmem.flush(cq) copies each record into sysmem and pushes its size onto prefetch_q.
5. Prefetch core's fetch_q_get_cmds sees the new slot, NOC-reads the record from hugepage into cmddat_q, marks the slot 0.
6. Prefetch decodes outer RELAY_INLINE, acquires N pages of dispatch CB credit, NOC-writes the inner command into the dispatch CB, releases credit via NOC atomic-inc.
7. Dispatch BRISC wakes; runs process_cmd — for WRITE_PACKED issues N NOC writes to workers; for WAIT_STREAM polls stream 48; for SEND_GO_SIGNAL unicasts the go word.
8. Workers receive go word, run, write completion to stream 48 on dispatch core.
9. Dispatch processes WRITE_LINEAR_H_HOST: writes 32 B (cmd hdr + event page) to completion region; updates wr_ptr in sysmem mirror.
10. Host's wait_completion polls _HOST_CQ_WR_OFF, sees move, reads event_id, advances rd_ptr, writes back. Done.

9. Pieces worth re-emphasizing

• Three independent rings: prefetch_q (16-bit sizes, 1534 entries), cmddat_q (256 KiB byte ring), dispatch CB (128×4 KiB page ring). Each has its own producer/consumer ptrs.
• Three independent semaphores per producer-consumer hop: input-CB credit, downstream-CB credit, sync (for STALL / NOTIFY_PREFETCH). All counted, all over-NOC atomic-incremented.
• Stateful NOC programming: each kernel pre-arms cmd buffers once at init and re-uses them every transaction. dispatch_s uses a different cmd-buf allocation than dispatch_d so they share NOC 1 without clobbering each other.
• PCIe NOC: prefetcher reads from host hugepage go over (19,24) on Blackhole P100 — a special endpoint bridging to the PCIe controller. The hugepage IOVA is pinned and bit 60 marks it as a PCIe-NOC address.
• Stream registers (48..55) carry worker-completion signals. CLEAR is implemented via the update reg with a shifted negative delta because stream regs are accumulators.
• Timestamps: every program is bracketed with two CQTimestamp commands that have the dispatcher sample its wall clock and write 8 B to a sysmem slot. With TT_CORE_TIMING=1, the host also arms each worker BRISC to log its own start/end into another pair of slots.
• Unused-in-this-build commands: RELAY_PAGED*, RELAY_LINEAR, RELAY_RINGBUFFER, EXEC_BUF*, WRITE_PAGED, WRITE_LINEAR(_H), NOTIFY_SUBORDINATE_GO_SIGNAL, mcast SEND_GO_SIGNAL. The host only emits WRITE_PACKED, WRITE_PACKED_LARGE, WAIT, SEND_GO_SIGNAL, SET_GO_SIGNAL_NOC_DATA, WRITE_LINEAR_H_HOST, TIMESTAMP.