On May 15, 2026, Pope Leo XIV released an encyclical letter expressing his thoughts on Artificial Intelligence for all of us to reflect on. This couldn’t have come at a better time, because today more than ever, the fast development of AI technology has been challenging me to reflect more deeply and with more urgency on what are the most important things to focus my life on, as a parent, a citizen, an engineer. I would thus like to relay his letter and offer some of my thoughts, especially on the first half of it, and in the humble spirit that claims no full knowledge of the truth.

The letter is long but worth a read. In it, Pope Leo XIV challenges us to think about whether the promise of AI technology will be pursued under the delusion of human infallibility and self-sufficiency or in the spirit of an authentic and responsible stewardship that honors and preserves. Rather than humans sacrificing their dignity and being subservient to technology, it is technology that must be directed to serve human dignity. The pope thus spends a good portion of his letter explaining the main principles developed by the Catholic Church to honor and preserve the dignity of every human being, the principle of common good, of universal destination of goods, of subsidiarity, of solidarity, and of social justice. “The fundamental dignity of each person, therefore, is neither acquired nor earned, nor does it need to be justified [..] Every human person possesses an infinite dignity, inalienably grounded in his or her very being, which prevails in and beyond every circumstance, state, or situation the person may ever encounter” [53]. The same spirit pervades the sacred American creed written 250 years ago, that all “men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness”.

Today, the challenges facing humanity are immense because society is dominated by the modern technological way of being. In order to confront this reality, we must be clear-eyed about what technology truly is, not the sum of all created technical things but a reflection of our own modern way of revealing the world, our “tendency to let the logic of efficiency, control, and profit alone shape personal, social and economic decisions”[92](emphasis mine). A new historical reality has set in, one in which the dominant actors driving these technological breakthroughs are private corporations large enough to mediate almost all societal interactions and with enough financial resources to bend the democratic system in their favor. As Pope Leo XIV says, “In many cases within the digital context, control over platforms, infrastructure, data and computing power does not rest with States, but with major economic and technological actors. These entities effectively set the conditions for access, determine the rules of visibility and shape the very possibilities for participation. When such power is concentrated in the hands of a few, it tends to become opaque and evade public oversight, increasing the risk of distorted forms of development that give rise to new dependencies, exclusions, manipulations and inequalities.”[95]

The impending arrival of what is now called Artificial General Intelligence (AGI) technology has been dazzling the world, feeding its craving for unlimited superhuman capabilities on demand, but at the same time auguring an era that deeply questions the role of humans in society. Pope Leo XIV warns us not to confuse artificial intelligence with human intelligence and not to succumb to the hubris of the Tower of Babel. AI systems are very useful but they are not human. They “do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences”[99]. All that AI is trained to do is to imitate language and simulate empathy and understanding. It feels nothing, it is simply a cold and empty “form of statistical adaptation based on data and feedback, which can be very effective, but does not imply inner growth”. The distinction between human and artificial intelligence is so fundamental that it ought not even be questioned, and yet here we are so lost to technological thinking that this fact is no longer “obvious” and needs to be argued. In the letter, the pope has a message of vigilance for those developing this technology because he understands technology is not a morally neutral instrument. All human creations embody in them the values of their creator. “Every technical tool embodies choices and priorities through what it measures, ignores and optimizes, and how it classifies people and situations.”[104] And for this reason, with AI even more than any other technology, it is important to understand and “examine how that system is designed and what vision of the human person and society is embedded in the data and models that guide it” [104]

Pope Leo XIV uses the provocative concept of Disarming AI to wake us up to the reality of a race to the bottom in an AI-weaponized world with catastrophic consequences. This expression brings our mind back to the time when the atomic bomb was detonated to end WWII and destroyed countless lives in the most horrific way possible. Shortly after humanity realized that it had created a technology that could bring it to its end. The evident potential for mutual destruction and annihilation were so overwhelming that they awakened global consciousness and sustained the movement to disarm nuclear weapons during the Cold War. We have not yet seen today’s AI Hiroshima moment. Still in the future lies the threat of uncontrollable autonomous AI weapons or society-wide AI automation leading to massive unemployment and breakdown of social relations. These dangers are much discussed today, but there is one more that the pope emphasized which I really appreciate, one that is much less talked about and less conceptualized. This is the subtle but slower and more profound way AI will impoverish our human existence as it presents itself as the ultimate possibility of liberation from human suffering. Trans-humanism and post-humanism [116] are today’s ideologies pushing furthest in this direction. In reality if left to seep through society without any sense of the impending danger, AI technology will turn out to be the biggest challenge to the human creative spirit and pursuit of happiness.

All my readings and experience indicate to me that for millennia now, from the early greek philosophical writings, to the first century apostolic gospels and epistles expressing the radical Christian message, to the 20th century modern existential philosophies responding to the calamities of two World Wars, human being at its core has been understood to mean overcoming a fallen, average, and dispersed absorption in worldly matters. But overcoming how? Certainly not by removing the body altogether and replacing it with a machine (or by escaping to Mars), but overcoming by a process of resolutely taking up the responsibility for our ownmost possibilities, freed to face the anxiety of our finitude, embedded in our communities, rooted in a place of belonging, and from the Christian point of view grounded in the unity of being that is God. The most optimistic supporters of AI will have us all believe our salvation lies in a human created technology. Pope Leo XIV, however, understands this not to be true, that there is no final solution to the problem of human existence, that the danger lies in thinking that there is, that instead it falls as a task to every generation to renew human existential possibilities and make them available to the largest number of people: “I am convinced that the concrete way of living out social relationships in the light of the Gospel is not established once and for all, but remains a task entrusted, from generation to generation, to the Christian community.”

There is more in this letter that is worth pondering and I hope to touch on in future posts after more reflection and reading. I would just like to close with the special message that Pope Leo XIV addressed to developers and engineers working on the development of AI technology, and I would say in technology in general and not just as engineers, whether in Silicon Valley or around the world, that is worth asking oneself about (emphasis mine):

“I wish to address a special appeal to those who develop artificial intelligence. In one sense, technological innovation can represent human participation in the divine act of creation. Developers, therefore, bear a particular ethical and spiritual responsibility, for every design choice reflects a vision of humanity. Just as the creator of an artistic or literary work must consider the values it conveys, so developers are called to embed values in their projects with due seriousness: with transparency, responsibility toward affected communities and careful attention to ensuring that what is being cultivated is a genuine good” [111].

Encyclical Letter, Magnifica Humanitas

Short speech by the pope summarizing his thoughts on Disarming AI

Traditional OLAP database execution engines were designed for the CPU: 1) optimized for a handful of powerful cores, 2) deep cache hierarchies, and 3) sequential or lightly vectorized processing. In the past decade, however, GPU performance and functionality have greatly advanced, driven largely by the generative AI revolution, to the point of becoming a viable platform for running relational workloads. GPU-accelerated data systems that can run queries orders of magnitude faster than their CPU equivalents will enable the next big revolution in analytics, fuelled by AI agents. Their architecture and programming models are, however, different enough from the CPU, that specialized algorithms must be developed to achieve high performance on analytical workloads. This post aims to illuminate the kind of algorithms NVIDIA’s RAPIDS project has built to close that gap. It is the first in a series exploring libcudf, NVIDIA’s core DataFrame library for single-node GPU data processing.

This report was produced with the help of AI agents.

1. Introduction: Relational Algebra on GPUs

Mapping relational algebra onto GPUs introduces a massive semantic gap compared to CPU. Operators like joins, aggregations, and sorts must be entirely reimagined for the GPU SIMT (Single Instruction, Multiple Thread) architecture. Conventional algorithms natively optimized for CPUs often hit brutal bottlenecks on GPUs due to thread divergence, uncoalesced memory access, and severe penalties for global synchronisation.

To bridge this runtime gap, NVIDIA developed libcudf: a C++ library implementing foundational DataFrame operations and relational primitives natively on the GPU. It has emerged as the de facto execution framework for a massive portion of the accelerated data ecosystem, underpinning projects like Spark RAPIDS, Dask-cuDF, Velox CuDF and numerous independent database research efforts.

The central questions driving this exploration are:

• How does libcudf translate fundamental relational operators into massively parallel GPU kernels?
• What are its structural strengths, and where does the GPU memory/compute model impose hard limits?
• What does the developer tooling look like, and how does one reason about its hardware utilization?

To answer these questions, I begin by identifying at a high level which algorithms underlie key primitives. I take as an example the simple groupby aggregation and break down into components showing how the library implements them. I identify the main data structures and provide illustration that show the movement of data as the algorithm runs. While some runtime info is provided for the 100M dataset in this post, the next one will focus on actual run and reviews learning from the debugging tools.

The Aggregation Problem

GROUP BY is one of the foundational operators in relational database systems. Its job is to partition an input relation into disjoint subsets then reduces each group to a single output row by applying one or more aggregate functions. For example, a simple question a retail merchant might ask is what the breakdown of the total price of orders by order status, identifying the amount of missed dollar opportunity for orders not completed and investigating improvements. We will use this example in the rest of the blog post.

In a query execution engine the GROUP BY physical operator must solve two logical subproblems:

1. Key partitioning: determine, for every input row, which output group it belongs to. This is effectively a dictionary-encoding problem: map an arbitrarily-typed key (integer, string, composite) to a dense integer group-id in [0, K) where K is the number of distinct keys.

2. Aggregation: reduce all rows assigned to the same group-id to a single scalar per aggregate column (e.g., sum all values from column C for group-id 3), using an aggregate function such as SUM, COUNT, MIN, MAX, AVG, etc

These two subproblems are algorithm-agnostic: the same logical goals can be achieved via two fundamentally different physical strategies.

• The sort-aggregate approach sorts all rows by key first, after which identical keys are contiguous and can be reduced in a single scan; comparison sort costs O(n log n), while radix sort can be linear for fixed-width keys.
• The hash-aggregate approach builds a hash table mapping each distinct key to its running accumulator, updating it in expected O(n) time, no sort required, but concurrent writes to shared buckets introduce contention.

The algorithm must also be adapted to the number of keys and columns being operated on as well as the kind of data types used for partition and aggregation. Simple primitive types like float and int come with hardware and basic language support, while more advanced ones like string and datetime require specialized handling.

Recently, a big use case has been supporting user-provided functions (UDFs) for aggregation, which come with their own challenges, mainly the requirement to compile them first into low-level code before running them efficiently.

CPU vs GPU Challenges

Various kinds of CPU-focused solutions have been developed over decades and evolved with CPU cache hierarchies, branch prediction, and thread-level parallelism. GPUs introduce a different execution model: aggregation algorithms must explicitly manage the memory hierarchy, limit global atomic contention, minimize warp divergence, and keep key-comparison logic executable entirely on-device.

I’ll focus on the high-level algorithm libcudf uses to compute a simple Groupby + Sum aggregation on the GPU, taking you through its flow from the initialization of data structures to identifying unique groupings, and aggregating the data into final output buffers. Throughout, I’ll highlight the libcudf CUDA kernels used, how they take advantage of the GPU’s limited but very fast shared memory to update intermediate results in a massively parallel way, and use block synchronization to ensure threads remain in lockstep. I’ll also describe some of the other libraries libcudf relies on, such as cuCollections for the static sets and hashmaps, and the Thrust library for lower-level data-parallel algorithms like scatter and for_each. Future posts will provide a more in-depth look at an actual run of the algorithm and its performance, also introducing a new visualization tool I have developed to understand the flow of interaction between CPU and GPU.

2. Setup: Software and Hardware used

Library Versions

The investigation was performed on RAPIDS v26.02.00, released on February 4, 2026. The cuCollections dependency used is pinned to commit d3701ae.

The code examples in this post link to my own annotated forks that include additional comments to aid understanding:

• My cuDF fork: github.com/jazracherif/cudf, v26.02.00_analysis
• My cuCollections fork: github.com/jazracherif/cuCollections, v26.02.00_analysis

GB10 Device

This analysis was performed on the DGX Spark running the GB10 NVIDIA GPU (Blackwell). Here are some key hardware specs relevant to this analysis to keep in mind:

The unified memory architecture means there is no PCIe transfer step for the input table, the Arrow Parquet file is read directly into the shared pool and is immediately accessible by both CPU and GPU. The memory bandwidth figure (~273–301 GB/s) will be the primary bottleneck for this workload, as the hash-set initialization and Interlude scans are purely bandwidth-bound.

Code Invoked

The input dataset is the order table from TPCH with 100 million rows (1.8 GB Parquet file). Future posts will explore much larger datasets. The below libcudf C++ code is invoked on an ingested table stored in the Apache Arrow format:

// Assume Table already loaded into GPU memory
cudf::table_view tv = cudf_table->view();

// Create GroupBy operator by specifying the `key` column to group on
cudf::groupby::groupby gb(cudf::table_view{{tv.column(src.key_col)}});

// create aggregation for each column, here only 1 SUM agg
cudf::groupby::aggregation_request req;
req.values = tv.column(src.value_col);
req.aggregations.push_back(cudf::make_sum_aggregation<cudf::groupby_aggregation>());

// Aggregate on default stream
auto [result_keys, agg_results] = gb.aggregate({req});

The equivalent SQL command is:

SELECT   o_orderstatus,
         SUM(o_totalprice) AS total_price
FROM     orders
GROUP BY o_orderstatus;

The goals are:

• Understand how libcudf selects and executes the groupby sum path using a string key and float64 column.
• Break down the algorithm into understandable pieces and show where in the code these are implemented
• Identify the main GPU kernel launch to its source location in cuDF and associated libraries.
• Explain the two-level shared-memory aggregation strategy that libcudf uses to reduce global atomic contention.

In a followup post, I will cover the following:

• Capture a real run of the algorithm with real Nsight Systems on GB10: confirms kernel names, ordering, and timing on a 100M-row workload.
• Review each kernel performance with Nsight Compute
• Look at the flow of messages using a custom viewer I have developed.

With 100M input rows that need to be reduced into K distinct o_orderstatus values, a naïve GPU approach, one global atomic-add per row directly into the output column, will suffer from severe memory contention, particularly when cardinality is low. cuDF avoids this by staging the reduction through shared memory.

3. Architecture at a Glance: The Four-Phase Data Flow

The diagram below shows the overall execution of the aggregation from the GPU’s perspective (from left to right), covering kernels and main data structures used.

Use it as a map that links together all the details. The following sections below will zoom into each region: the kernel implementations (§5), and a step-by-step trace (§6), and more details about the global_set structure (§8)

Four algorithmic steps are highlighted:

1. Initialization
2. Block Level Membership and Index Mapping
3. Interlude: Dense output index remapping
4. Final step: Shared-memory accumulation + flush

The diagram highlights the different data structures stored in global memory and those in the shared memory, with reference to the 100M row dataset.

Global memory (visible to all blocks):

• Input: o_orderstatus, o_totalprice (N=100M rows)
• Intermediary: global_set (800 MB, 200M slots each 4 bytes), local_mapping_indices (one int32 per row), global_mapping_indices (128 int32 entries per block)
• Output: total_price (K values)

Shared memory (private to each SM, discarded after the kernel):

• shared_set / __shared__ slots[128]: phase 2 only; the cuco::static_set_ref hash map backing store, used by find_local_mapping to probe for key existence and assign block-local ranks via CAS
• shared_set_indices[128]: phase 2 only; parallel flat array mapping each block-local rank to the first input row that claimed it (rank → representative row-index)
• shmem_agg_storage: phase 4 only; dynamic partial-sum accumulator storage indexed by block-local rank. For one double SUM over 128 ranks, the logical accumulator is 1 KB before alignment and any additional per-aggregation layout overhead.

4. The GroupBy.Sum() Algorithm

1. Initialization:
   • Before any row is processed, a global hash set (global_set) is initialized to size 2x the input (200M slots) with a SENTINEL value (spawning a cub::detail::for_each on the gpu side).

     global_set[0..200M) <- SENTINEL
     

2. Block Level Membership and Index Mapping:
   • This phase reads the key column and determines which o_orderstatus group every input row belongs to. At this point the grouping is only local to a block. Each CUDA block uses a private hash table in shared-memory to map its rows to at most 128 distinct keys, assigning each a block-local rank (see below what is a block-local rank?).
   • For each new key, using CAS (the compare-and-swap atomic instruction), it atomically elects a single representative row where that key was first seen and inserts it into global_set to be used across all blocks.
   • Two index arrays are maintained for use in later phases: 1) the local_mapping_indices stores the block-local rank value allocated to each row later used to generate the per block aggregation, and 2) the global_mapping_indices stores the winning row for each rank slot of that block, later turned into a global ranking.

     local_mapping_indices[row]        -> the block-local rank assigned to row  
     global_mapping_indices[blk*128+r] -> The representative row index for each rank slot in each block
     global_set insert/find(rep_row)   -> The winning representative row at the key hash slot
     

3. Interlude: Dense Output Index Remapping:
   • Between the two main kernels, a set of device operations scans global_set (via retrieve_all / cub::DeviceSelect::If) to collect the K representative row-indices, then builds a dense output ordering (0..K-1) via thrust::scatter, and rewrites global_mapping_indices in-place via thrust::for_each_n so every block agrees on the same output slot for each group. These algorithms launch over the full input or hash-set range even though only K ≪ N slots are populated, trading excess thread count for uniform, divergence-free execution that saturates memory bandwidth.

     global_mapping_indices[blk*128+r] -> dense output index in total_price[0..K-1]
     

4. Shared-Memory Accumulation + Global Reduction:
   • Now that membership and output ordering are known, each block accumulates its assigned o_totalprice values entirely within shared memory (no cross-block, no global atomics yet).
   • Each block then flushes only up to 128 partial o_totalprice sums to the correct output slot using the remapped global_mapping_indices, one atomic-add per distinct o_orderstatus value per block rather than one per row.
   • For this dataset the number of global atomics is reduced by a factor of roughly 100M / (num_blocks × avg_labels_per_block) compared to the naïve approach.
   • The logic is repeated based on the number of aggregation outputs to produce and available shared memory within a block

     r = local_mapping_indices[row]
     shmem_price_accum[r] += o_totalprice[row]
     global_label_idx = global_mapping_indices[blk*128+r]
     total_price[global_label_idx] += shmem_price_accum[r]
     

Note: Phase 2 and Phase 4 communicate through the index arrays produced by Phase 2 and rewritten by the Interlude; no inter-block GPU synchronisation is needed between Phase 2 and Phase 4.

What is a block-local rank?

• The key idea behind the fast path is that each CUDA block first locally deduplicates the keys among the rows it processes, then connects those per-block results to the final global output groups. This is done via block-local ranks.
• Each CUDA block assigns a small integer, starting from 0, to each distinct o_orderstatus value the first time it is encountered among that block’s assigned rows. That integer is the block-local rank: a dense index into the block’s private shared-memory accumulator array. In the fast path, valid ranks are 0..127. This numbering is private to this block; another block may assign rank 0 to “O” or any other o_orderstatus.
• The Interlude phase converts the representative row-indices stored in global_mapping_indices after Phase 2 into final dense global output indices (0..K-1), where K is the total number of unique keys across all rows. This ensures that all blocks agree on the same output slot for each group before phase 4 runs.

5. Deep Dive into each phase: Kernels and Data Structures

Now that the data flow and hash-set mechanics are established, this section revisits the same phases at the level of the actual kernels and helper functions. All the kernels below are invoked from a single host function, compute_single_pass_aggs().

Phase 1: Hash set initialization

Before any row is processed, a cub::detail::for_each kernel sweeps all 200M slots of global_set and writes the SENTINEL value (typically INT32_MAX) to each one. This establishes the “empty” state that insert_and_find’s CAS loop uses to distinguish occupied from free slots. At 4 bytes × 200M slots = 800 MB of writes, this kernel is purely memory-bandwidth-bound (~4.1 ms on this dataset).

Phase 2: Key insertion and index mapping

Every input row is processed by the mapping_indices_kernel kernel. For each row, the thread performs three steps:

1. Block-local deduplication: find_local_mapping() inserts the row’s key into shared_set, a block-private mini hash table cuco::static_set_ref backed by __shared__ slots[] (capacity = GROUPBY_CARDINALITY_THRESHOLD = 128 unique keys). shared_set is used only for existence checks (new key vs. duplicate); a separate flat __shared__ array shared_set_indices[rank] = row_idx maps each block-local rank to the first input row that claimed it. local_mapping_indices[row] is written with the block-local group rank (0..127): for a new key it is assigned by atomically incrementing cardinality; for a duplicate it is copied from local_mapping_indices[matched_row] after a block.sync(). local_mapping_indices provides a local per block grouping of the rows that will be re-used in phase 4 of the later accumulation step.

2. Global key registration: find_global_mapping() iterates over shared_set_indices[0..cardinality-1] and inserts each representative row-index into the global cuco::static_set. The CAS inside global_set.insert_and_find() atomically elects a single representative row for that key across all blocks. The winning row-index is stored in global_mapping_indices[block × 128 + rank]. Only one global insertion is made per distinct o_orderstatus value per block, not per row.

3. Overflow detection: if cardinality > 128, the needs_global_memory_fallback flag is set and all threads in the block break out of the input loop. After the kernel, the host checks this flag and if set, falls back a slower naïve global-memory aggregation path instead run_aggs_by_global_mem_kernel.

Phase 3, Interlude: Dense output index remapping

When there is no overflow, extract_populated_keys() is invoked to extract unique key row-indices from global_set into a contiguous buffer via cuco::static_set::retrieve_all(), which fires two CUB kernels (DeviceCompactInitKernel + DeviceSelectSweepKernel).

The key transition in this phase is the meaning of global_mapping_indices:

Before Interlude:

global_mapping_indices = representative input row-index, in range [0..N-1]

After Interlude:

global_mapping_indices = dense output index into total_price[], in range [0..K-1]

This is done in 2 steps:

1. A compute_key_transform_map() step builds the dense renumbering (key_transform_map) that maps any representative input row-index to a compact output slot [0, K):

   key_transform_map[representative_input_row_idx] 
        = output_group_index   (0..K-1)
   

2. A second thrust::for_each_n kernel then rewrites global_mapping_indices in place using this map so that every entry holds a finalized output group index.

Phase 4: Shared-memory accumulation + flush

This phase is implemented by a single kernel, single_pass_shmem_aggs_kernel

Each block declares extern __shared__ cuda::std::byte shmem_agg_storage[]: a dynamically-sized shared memory buffer laid out by calculate_columns_to_aggregate() as num_agg_columns × cardinality × sizeof(element_type) bytes (plus alignment padding), where cardinality ≤ GROUPBY_CARDINALITY_THRESHOLD = 128.

The kernel breaks the computation into a loop covering a number of aggregation output columns based on available shared memory, with the inner loop running the following two sub-phases:

┌─ Sub-phase 1: per-row accumulation into shared memory ──────────────────────┐
│  For each `row` assigned to a block, use previously generated               |
| `local_mapping_indices` to aggregated rows with same key in each block:     │
│    shmem_agg_storage[local_mapping_indices[row]] += source_value[row]       │
│    (via cudf::detail::atomic_add into shared memory)                        │
└─────────────────────────────────────────────────────────────────────────────┘
                              |
                          block.sync()
                              |
                              V
┌─ Sub-phase 2: flush partial results to global output columns ───────────────┐
│  For each `unique key` resident in this block:                              │
│    target_global_col[global_mapping_indices[blk×128+rank]]                  │
│        += shmem_agg_storage[rank]                                           │
│    (via cudf::detail::atomic_add into global memory)                        │
└─────────────────────────────────────────────────────────────────────────────┘

target_global_col will contain the final aggregation value for each column.

The global atomic_add in sub-phase 2 is reached via an inlined two-level compile-time template dispatch (type_dispatcher × aggregation_dispatcher) that resolves the runtime column type and aggregation kind to a single pre-compiled specialization with no GPU branching. For SUM on double input (o_totalprice), this lands at update_target_element_gmem<double, SUM>, which calls cudf::detail::atomic_add directly.

Wrapup: Output Key Gather

After aggregation, the unique key row-indices retrieved from the hash set are used to gather the corresponding rows from the original input keys table into a dense output keys table:

output_keys[i] = input_keys[unique_key_indices[i]]   for i in [0, K)

For string key columns this gather requires a multi-step CUB prefix scan over character offsets followed by a parallel character copy kernel (gather_chars_fn_char_parallel).

6. Step-By-Step Example of the algorithm: from input rows to final output indices

The example below traces the whole algorithm with two small blocks. The values are artificial, but the roles of local_mapping_indices, global_mapping_indices, unique_key_indices, and key_transform_map match the real execution.

Setup:

• 2 blocks (B0, B1),
• GROUPBY_CARDINALITY_THRESHOLD = 128
• K=3 unique aggregation key values, "F", "O", and "P"
• MurmurHash3 slot assignments in the 200M-slot global_set:
  • hash("F")%200M = 47_000_000
  • hash("O")%200M = 103_000_000
  • hash("P")%200M = 182_000_000.

Step 1: Input partitioning

Each block is assigned a contiguous slice of the 100M input rows:

Block0 (rows 1000..1004):

Block1 (rows 5000..5004):

Step 2: Phase 2 block-local rank assignment + global set insertion (compute_mapping_indices)

• Each block builds a private shmem hash set, assigning a rank to each new key on first encounter.
• For every key that is new to that block, it calls insert_and_find(row_idx) on the shared global_set (200M slots, cuda::thread_scope_device) to claim a globally unique slot via CAS.
• insert_and_find returns {iterator_to_slot, bool_inserted}.
• Dereferencing the iterator (*it) yields the row index stored in that slot; always the winning thread’s row_idx, regardless of which thread won the CAS race.
• That row index is what gets written to global_mapping_indices.

Assume Block0 wins the global CAS races, and each block assigns local ranks in first-seen row order. local_mapping_indices maps each input row to its block-local rank:

• Block0 first sees “F”, then “O”, then “P” → F=rank0, O=rank1, P=rank2
• Block1 first sees “O”, then “P”, then “F” → O=rank0, P=rank1, F=rank2

local_mapping_indices: block-local rank per row:

global_set after Phase 2 (200M slots, only 3 occupied), stores the winning representative row for this key, all from Block0 rows.

global_mapping_indices after Phase 2 contains representative input row indices, not yet mapped to the dense output grouping. Since B0 won the CAS races, B0’s winning rows are what gets stored:

Note: B1 also attempted to insert “O”, “P”, and “F” but the CAS returned DUPLICATE. The iterator still points to the existing slot, so *it gives the same row index B0 stored. Both blocks therefore agree on the same representative input row index per key.

Step 3: extract_populated_keys(): compact global_set → unique_key_indices

retrieve_all() scans global_set linearly from slot 0 to slot 199M via cub::DeviceSelect::If, collecting the row-index values stored in each non-SENTINEL slot:

scan order: slot hash("F")%200M comes first, then hash("O")%200M, then hash("P")%200M
            (i.e. in ascending slot-position order, regardless of insertion order)

unique_key_indices = [1000, 1001, 1003]   ← representative input row index per slot, in slot-scan order
                       i=0    i=1    i=2

These are the same row indices already in global_mapping_indices, just deduplicated by scanning the hash table. Their position in unique_key_indices (0, 1, 2) defines the dense output row each key will occupy.

Step 4: compute_key_transform_map(): invert unique_key_indices via thrust::scatter

Scatters counting values 0, 1, 2 to positions unique_key_indices[0,1,2]. The result is an array of size N (number of input rows), where each populated index is the representative input row index mapped to its final dense output row.

Step 5: thrust::for_each_n: rewrites global_mapping_indices in-place with dense output rows

Each non-SENTINEL entry (a representative input row index in 0..N-1) is replaced with key_transform_map[old_idx] (the corresponding dense output row in 0..K-1). The representative rows 1000, 1001, and 1003 are not usable as output indices directly; there are only K=3 output rows, so they must be remapped to 0, 1, and 2:

global_mapping_indices after remapping (dense output indices, replacing representative row indices): Notice that the ranks have the same value across all blocks; it is a global mapping.

Step 6: Kernel 2, accumulate + flush (compute_shared_memory_aggs)

Now we have a mapping from every row to its block-local accumulator, and from every block-local accumulator to its global output row. Each block reads its rows, accumulates o_totalprice into shmem using local_mapping_indices[row] as the shmem slot, then flushes at most 128 partial sums to global memory using global_mapping_indices[block*128 + local_rank] as the total_price output index.

7. Algorithm Complexity Summary

Assuming the following:

• N = total number of input rows (100M in this dataset).
• K = number of distinct groupby keys.
• capacity = hash-table size (2N slots = 200M).

Total: O(N) average with low constant factors when cardinality ≤ 128 groups per block. The asymptotic result is simple; the practical win comes from changing global atomic frequency from per-row to per-block-per-group.

8. Appendix: A Deeper Look into hash set global_set

The hash groupby is built around a device-side open-addressing hash set (cuco::static_set), referred to as global_set in the code, that stores one representative input row-index per unique key. It does not store the aggregation key values directly; instead, each stored row-index points back into the original key column, and the row hasher/comparator use that row to hash and compare the key value.

Since many rows can have the same aggregation key, insert_and_find() uses CAS (compare-and-swap) to claim empty global slots and elect one representative row for each key across all blocks. Each block first maintains its own block-private shared_set in shared memory to deduplicate rows locally, then only the block-local representative rows are inserted/looked up in global_set.

Multiple blocks may attempt to register the same key, but only the first successful CAS writes that key’s global representative row-index into the set. In order to minimize collision cost without knowing the distinct key count in advance, the set’s capacity is sized for the worst case in which every input row has a distinct key: twice the number of rows in the dataset.

global_set slot layout:

• N = 100M rows
• load factor = 50%
• capacity = 2 × num_input_rows = 200M slots

Example content with 2 unique key values:

Set design

The hash set is constructed in compute_groupby() with the following specifications:

• Key type: int32_t (cuDF size_type). Row hashing and equality comparison are performed by cuDF’s row comparator against the o_orderstatus (utf8) column. MurmurHash3 over character bytes, byte-wise equality.
• Capacity: 2 × N slots where N = num_input_rows (used as a worst-case upper bound for distinct key count; CUCO_DESIRED_LOAD_FACTOR = 0.5). For N = 100M rows: 200M slots × 4 bytes = 800 MB. Construction fires cub::detail::for_each::static_kernel<initialize_functor<long,int>> to fill all slots with the sentinel in parallel. For 100M rows, initialization costs 4.105 ms, ~23.5% of total groupby kernel time.
• Probing scheme: cuco::linear_probing<1, row_hasher_with_cache_t>. Linear probing with CGSize=1 (each probe step is handled by a single thread, advancing one slot at a time), with an optional row-hash cache (pre-computed hashes stored in a device_uvector).
• Thread scope: cuda::thread_scope_device. All GPU threads can access the same set.
• Sentinel: CUDF_SIZE_TYPE_SENTINEL = INT32_MAX. Marks empty slots.
• Memory: rmm::mr::polymorphic_allocator. Backed by the caller-supplied RMM pool.
• Storage layout: cuco::storage<BucketSize=1>. Two-level slot hierarchy: array of buckets, each holding BucketSize contiguous slots. BucketSize > 1 lets a thread probe multiple slots per step (beneficial for memory-bandwidth-bound workloads). For cuDF GroupBy, hardcoded to GROUPBY_BUCKET_SIZE = 1 (flat per-slot probing); appropriate here since key cardinality is low and contention is minimal.

Finding/Inserting a key in the set

The set stores row indices (int32_t), not actual key values. When the set needs to hash or compare a candidate slot, it calls back into the original input column data on the GPU (via d_row_hash). This indirection is set up before any kernels run, in dispatch_groupby():

1. preprocessed_table::create(keys, stream): copies the column_device_view metadata structs (data pointers, null masks, type IDs) into a GPU buffer so kernels can dereference them. The actual column bytes were already in GPU memory via RMM. Cost: ~143 bytes (one string column’s metadata, as seen in the RMM trace).
2. self_comparator: host factory that wraps the preprocessed_table and produces device_row_comparator, a GPU callable implementing operator()(i, j) → byte-wise string equality via type_dispatcher.
3. row_hasher: same pattern; produces device_row_hasher, a GPU callable implementing operator()(i) → MurmurHash3 over all columns of row i. Both share the same preprocessed_table via shared_ptr to avoid a redundant GPU upload.

These two callables are then embedded directly into the cuco::static_set constructor as the probing scheme and equality comparator, so every insert and lookup the set performs reaches back into the original key column memory.

insert_and_find(i) logic for row index i:

1. slot = d_row_hash(i) % 200M_slots          ← initial probe position from o_orderstatus string bytes

2. occupant = *slot
  pre-CAS check: d_row_equal(i, occupant)    ← does the row stored in this slot match row i's key?
      EQUAL     → return {slot, false}         ← key seen before; occupant is the representative (no CAS needed)
      AVAILABLE → go to step 3                 ← slot is empty (SENTINEL); attempt insert
      UNEQUAL   → slot += 1, repeat step 2    ← occupied by a different key; linear probe

3. CAS(slot, SENTINEL, i)                     ← atomically try to claim this empty slot
      SUCCESS   → return {slot, true}          ← we won; row i is now the representative
      DUPLICATE → return {slot, false}         ← another thread won the same key; slot holds the representative
      CONTINUE  → repeat step 2 at same slot  ← a different key raced us here; re-probe from this slot

Phase 2 kernel mapping_indices_kernel uses this operation in two scopes. First, each row probes the block-private shared_set to get a block-local rank. Then only the rows that represent keys new to that block probe the global global_set, where the CAS in step 3 performs the cross-block election: whichever thread wins the compare-and-swap for a given o_orderstatus value becomes the globally agreed representative row for that key. The CONTINUE result (a raced-but-different-key loss) sends the thread back to re-evaluate the slot it just lost, not to advance, since the winner may have written a key equal to i.

This post tackles the following sessions:

1. Quais Taraki (CTO, EDB) shows how standard Postgres breaks under agentic query loads and walks through PGAA — a GPU-accelerated HTAP solution that swaps the Postgres compute back-end for Iceberg + Spark RAPIDS, achieving 100× TPC-DS speedup and enabling a complete LangFlow-based agentic stack on top.

2. Liang Chen and Prudhvi Vatala from Snap detail how Spark RAPIDS cut A/B pipeline costs by 90% — not through any Spark tuning magic, but by rerouting 11,000 idle inference L4s into a three-tier fallback Spark fleet at near-zero incremental cost.

3. Harishankar G and Jalakandeshwaran A from Zoho give a deep dive into Velociraptor, their in-house GPU OLAP engine built as a Postgres extension, which runs all 22 TPC-H queries at 1 TB on a single H200 in under two minutes — then explain why PCIe is still the bottleneck even after every I/O optimization.

4. Hirakendu Das, Navin Kumar, and Rishi Chandra lead a hands-on Spark RAPIDS workshop covering the cuDF plugin, Project Aether’s automated qualify → tune → validate loop, and Ether Assistant’s LLM-based UDF rewriter.

5. Allison Ding walks through a full GPU data science pipeline — from zero-copy feature engineering with cuDF and GPU Polars, through cuML model training (k-means 40×, XGBoost 7×), to Triton Inference Server deployment with dynamic batching.

Industry Use Cases

🔗 [EDB] Supercharging Postgres for Agentic Analytics with Rapids Accelerator and Apache Iceberg

Quais Taraki · CTO, EDB

EDB (EnterpriseDB) develops solutions on top of PostgreSQL, such as the proprietary extension PostgreSQL Analytics Accelerator (PGAA) discussed in this talk. EDB recognizes the problem that Analytics Agents are now bottlenecked by CPU-based data systems for their OLAP needs. In this talk, Quais discusses PGAA, their hybrid OLTP/OLAP solution that relies on Spark Rapids GPU acceleration for large-scale analytics and making it available to Agents via technologies like langflow and kserve. More on PGAA in their blog post.

Takeaways

1. Agent queries time out on standard Postgres — the database becomes the bottleneck	@ 03:54
Agents generate far more complex and random query patterns than human-written queries. "We see customers hitting timeouts with large data sets, thereby starving the agent." Constraining the agents or doing an application rewrite are the typical mitigations — both undesirable. The fix must come from the database layer.
2. PGAA swaps the Postgres compute back-end: Iceberg + DataFusion + Spark Connect on RAPIDS	@ 05:11
EDB's Postgres Analytics Accelerator (PGAA) replaces the Postgres compute back-end with three components: (1) replicate data to object storage in Apache Iceberg format; (2) DataFusion — "a columnar vectorized open source query engine" — as a plug-and-play compute layer; (3) Spark Connect to offload massive distributed joins to a Spark cluster, further accelerated via Spark RAPIDS on GPU. The Postgres front-end and SQL interface remain unchanged.
3. TPC-DS: Postgres times out at 10 TB; Spark RAPIDS on L40S is 100× faster; Blackwell adds 14× more	@ 05:57
Three-tier TPC-DS comparison across data sizes: standard Postgres (orange) grows unbounded and times out at 10 TB; vanilla Spark+PGAA (blue) is a substantial improvement; Spark RAPIDS on L40S (green) lands "on the order of 100x over standard Postgres." Re-ran on RTX 6000 Pro (Blackwell): "a further 14x speedup" on top of that.
4. Full agentic stack: AIDB vectorization + MCP endpoints + NVIDIA NIMs + LangFlow on top of PGAA	@ 08:02
Above the analytics layer: the AIDB extension generates embedding vectors from Postgres data for semantic search; MCP endpoints expose all data stores to LLM agents; containerized NVIDIA NIMs run local model inference via KServe; LangFlow provides a low/no-code agent authoring environment. The entire stack runs on NVIDIA GPUs. Speaker's admission: "there are a lot of moving parts… a lot of security to consider" — EDB packages all of it so customers don't have to.
5. EDB ships the full stack as a sovereign, batteries-included Postgres platform	@ 09:41
Stack choices are opinionated: Lakekeeper as the Iceberg catalog ("a much more modern version than, say, something like Hive Metastore"), LangFlow for agent authoring, NVIDIA NIMs on KServe for inference. The platform is "complete, batteries included, modular, composable, sovereign, open source" — deployable on IBM mainframes, on-premise, all hyperscalers, or custom Supermicro+NVIDIA hardware.

🔗 [Snap] How Snap Saves Millions with Accelerated Apache Spark

Liang Chen · Staff Software Engineer, Snap, Inc.
Prudhvi Vatala · Sr. Engineering Manager, Snap, Inc.

This session takes us through Snap’s experience adopting Spark on RAPIDS for its A/B pipelines. All these experiments were done on GCP instances featuring L4 and T4 GPUs like the g2-standard-48, with an on-demand price of $1.76. Interestingly, using Spark on RAPIDS was mostly a smooth experience with little engineering effort; the main bottleneck was rather the scarcity of GPUs available for these data processing jobs, at a time when almost everything is going toward AI inference and training. The team’s main engineering effort was thus moving their system over to the AI K8s GKE cluster to take advantage of the idle GPUs often seen late at night. Nevertheless, the team highlights their collaborative effort with Google and NVIDIA in getting this to work. Look at this companion article from Snap Engineering for a good read.

Takeaways

1. Non I/O-bound jobs saw significant speedups with Spark Rapids, particularly in join and repartition stages	@ 04:21
For the merge sort join operator on 3 TB of input, total Spark execution time dropped 20×, leading to a wall-time speedup of 2.5×. For the union operator on 9 TB of input, total Spark execution time dropped ~4×, for a wall-time speedup of 1.8×. Aggregation's wall-time speedup was 1.6×. The authors note that in the first two cases, the speedup was due to no longer seeing terabytes of data spilling onto disk when using RAPIDS.
2. At Peak, the actual bottleneck is GPU Availability!	@ 08:54
A/B cluster uses 60k machines to finish within its time window. GPUs would reduce peak machine count by two-thirds — from 62,000 to ~20,000 GPUs simultaneously, still a huge amount! On-demand GPU procurement at that scale was not feasible. The breakthrough was finding idle capacity already inside Snap: the ML inference fleet for ad ranking and content recommendations drops to low utilization between 2–5 AM Pacific as users sleep, leaving thousands of GPUs sitting idle every night.
3. Reusing the inference fleet's idle window — the unlock for scale	@ 11:12
Snap built a Spark-on-GKE platform that runs batch jobs on the same infrastructure as the serving stack. Jobs use GPUs only when available and fall back to CPU GKE, then Dataproc — "GPU acceleration is just opportunistic, never mandatory." Pipelines were shifted into the 1–5 AM idle window by moving from client timestamps to server timestamps and rescheduling upstream Airflow dependencies to start at 2–3 AM.
4. Three-tier fallback: GPU GKE → CPU GKE → Dataproc; no job ever fails to complete	@ 15:46
At submission: no GPU quota → fall back to CPU Kubernetes. At runtime: GPU preempted by serving traffic → retry on GPU GKE, then CPU GKE, then Dataproc. "Every single job has a path to completion." Production shows 99% hourly job success rate and 96% daily — "and these numbers aren't even cherry picked. These numbers include all the infra failures even beyond GPUs."
5. 90% net cost reduction; 11,000 L4s, 81% less memory, zero new spend	@ 22:15
Because Snap reuses idle capacity, "there is no net new compute cost added." Net experimentation platform footprint dropped 90%. Apples-to-apples (counting GPU cost as incremental): 76% savings. The switch to g2-standard-48 instances leads to a dramatic drop in resource usage and costs: the pipeline now runs on ~11,000 L4s during the six-hour overnight window, "number of cores went down 62.5%," "memory went from about three petabytes to half a petabyte, 81% reduction."

🔗 [Zoho] Build a GPU-Accelerated Database Engine With CUDA

Harishankar G · Leadership Staff, Zoho Corp.
Jalakandeshwaran A · Leadership Staff, Zoho Corp.

Takeaways

1. Velociraptor processes all 22 TPC-H queries at SF1k (1TB) in under 2 minutes on a single H200 GPU	@ 03:37
Zoho's in-house GPU-accelerated OLAP engine, shipped as a Postgres extension, runs the full TPC-H benchmark at scale factor 1000 on a single GPU with 90 GB of memory. "The longest query executes in under eight seconds. The shortest one runs in under a second. And the median query execution time is five seconds." It currently powers Zoho Analytics' largest customers.
2. Plan conversion layer decouples GPU-optimal rewrites from the Postgres front-end and keeps the original plan for OOM fallback	@ 04:48
Postgres selects hash vs. sort group-by based on work_mem and expected cardinality — optimal for CPU, not for GPU. A plan conversion step lets Velociraptor substitute GPU-optimal choices while keeping the original Postgres plan intact for OOM fallback: "allows us to keep the original plan untouched in case we need to fall back due to situations like out of memory." It also let the team switch from Apache Calcite to Postgres as the front-end with minimal changes to code generation and execution layers.
3. Four-layer I/O stack: columnar storage + block filtering + compression + late materialization — all to minimize bytes sent to the GPU	@ 06:27
Techniques compound: (1) column-store layout skips unneeded columns; (2) per-block min/max metadata prunes blocks that can't pass a filter without reading them; (3) columnar layout boosts compression ratios since same-type data compresses better; (4) late materialization fetches only columns needed for the current operator. The result: sometimes only a very small amount of data per batch actually crosses the PCIe bus.
4. GPU decompression blows past the PCIe limit for high-compression data; LZ4 for strings is the weak link	@ 10:23
A three-stage pipeline — disk read → PCIe transfer → GPU decompress — runs concurrently using nvCOMP. On PCIe Gen 4 x8 (13 GB/s cap), GPU-accelerated decompression of cascaded RLE/delta columns exceeds the interconnect ceiling for highly compressed data. On H200 (PCIe Gen 5, 58 Gbps; 8.5× the memory bandwidth of the previous system), numbers are similarly strong. Weak link: LZ4 used for strings and doubles has lower throughput at low compression ratios — Blackwell's on-chip LZ4 decompressor is on the team's roadmap.
5. Even after all optimizations, GPU execution is only 25% of end-to-end query time — PCIe is still the bottleneck	@ 12:15
"Despite all of these optimizations… the actual time spent on executing the query on the GPU in a benchmark like TPC-H is only 25% of the end-to-end time. So we are still bottlenecked by the interconnect and decompression." The team is explicitly waiting for NVLink as a CPU-to-GPU interconnect on x86 — NVLink already reaches 900 GB/s for hosted devices vs. PCIe Gen 6 at 128 GB/s. They welcomed the Intel/NVIDIA NVLink fusion partnership as a step in that direction.

DLI Training Labs

🔗 Accelerate Apache Spark With GPU and AI: A Hands-On Workshop

Hirakendu Das · Principal Software Engineer, NVIDIA
Navin Kumar · Sr. System Software Engineer, NVIDIA
Rishi Chandra · Systems Software Engineer, NVIDIA

Apache Spark usage is predominant in enterprise. Four different use cases are highlighted at the beginning of the training session to remind us how central the system is and how enabling RAPIDS on Spark has wide impact across enterprise operations:

1. RAW ETL into the data lake
2. Analytics on this ingested data
3. Loading data from the lake into traditional data warehouses
4. Pre-processing this data for machine learning training and other data science use cases.

Takeaways

1. cuDF is the CUDA library at the center of RAPIDS Spark	@ 07:41
cuDF implements relational algebra on the GPU. Each node will convert the Spark physical plan into a plan that can be run over CuDF. Data is stored in columnar format and remains so until an unsupported operator forces a CPU fallback. The cost is the conversion round-trip not the operation itself. The RAPIDS qualification tools help determine if the overall query will overcome these challenges or not.
2. Spark on GPU wins on compute intensive tasks like joins, aggregates, sorts over high-cardinality data — not I/O-bound ops	@ 12:18
"If you have systems with very large amounts of high cardinality data.. and joins and aggregates and sorting, those tend to be the most ideal workloads for GPU." I/O-bound jobs, where most time is spent reading from or writing to a data store, see little benefit. Small datasets also underperform due to the overhead of staging data into GPU memory. Know which regime your job is in before expecting a speedup.
3. Project Aether automates the full qualify → submit → profile → tune → validate loop	@ 13:56
The old manual migration process — qualification, staging, POC, config iteration, production argument — requires enough engineering resources that "this process generally stops somewhere before getting the workloads migrated." Aether wraps all four steps (`qualify`, `submit`, `profile`, `report`) into a single aether run command. Results and configs are stored in a SQLite history database. Supports on-prem, Amazon EMR, and Google Dataproc.
4. Aether TuneML: XGBoost model predicts optimal Spark config changes with 90% AUC ranking accuracy	@ 43:10
Replacing rule-based formulas (QualX tunable with `aether profile`) with an XGBoost model trained on 100 NDS queries (90 train / 10 holdout), TuneML uses profiling metrics (input bytes, shuffle read/write bytes, spill rates) to predict speedup for candidate config changes. Ranking AUC ~90%: "90% of the time, the configs should lead to some speedup." Roadmap: replace XGBoost with a fine-tunable DNN and add reinforcement learning for efficient config space exploration.
5. The two critical GPU Spark configs are `sql.files.maxPartitionBytes` and `sql.shuffle.partitions`	@ 53:09
`maxPartitionBytes` controls data per read partition; `shuffle.partitions` controls the number of shuffle tasks for joins and group-bys. "In order to take advantage of the massive parallelism, you want to have bigger and bigger batch sizes or tasks" — but too large and the job spills. Memory spill metrics in event logs are the leading indicator that shuffle tasks are oversized. These two configs drive 80% of the tuning value.
6. UDFs force a full GPU→CPU PCIe round trip — columnar→row conversion included	@ 01:03:25
"You have to ship all the data back over PCIe to the CPU. You have to convert from those columnar batches back to row-by-row formats. You have to run the UDF, and then you have to reverse that process." The RAPIDS Accelerator cannot optimize inside a UDF — it's opaque to the query planner. Any UDF in a GPU Spark job is a performance cliff, especially for compute-heavy functions where the GPU speedup would have been highest.
7. Ether Assistant: LLM rewrites CPU UDFs to GPU columnar — test generation → conversion → benchmark	@ 01:05:18
Three-phase LLM pipeline: (1) generate unit tests for the existing CPU UDF, (2) rewrite the UDF to SQL or cuDF columnar using those tests for verification, (3) generate a synthetic dataset and benchmark both versions for speedup. Each phase is an iterative feedback loop before proceeding. The target is the RAPIDS UDF interface — a `evaluateColumnar()` override that hands the UDF GPU columnar batches directly, eliminating the PCIe round trip entirely.

🔗 From Ingestion to Inference: Mastering the High-Performance GPU Data Science Pipeline

Allison Ding · Senior Developer Advocate, Data Science, NVIDIA

This session covers the tools developed by NVIDIA for the full end-to-end machine learning workflow, from feature wrangling and exploration with cuDF, to accelerating various machine learning models for classification, regression, and clustering tasks with cuML, a drop-in replacement for scikit-learn, and finally how to profile and deploy the model to inference servers like Triton.

Takeaways

1. Apache Arrow is the zero-copy glue between cuDF, cuML, and GPU Polars	@ 10:33
All CUDA-X libraries — cuDF, cuML, cuGraph, cuVS — share data through Apache Arrow: "Arrow provides zero copy data transfers from pandas to CUDA-X, which means the data doesn't need to be copied or converted, just a pointer is passed." This is what allows the entire pipeline to stay on GPU without marshaling overhead between steps.
2. cuDF group-by: 200× faster; merges: 130× faster on GPU	@ 12:35
Key cuDF operation speedups: CSV reads ~20×, merges ~130×, group-by ~200×, select+filter ~3×. The full data processing + EDA + feature engineering pipeline on the IEEE CIS dataset runs in 43 seconds on GPU vs. 87 seconds on CPU — about 2× end-to-end, with the gains concentrated in the merge and aggregation steps.
3. GPU Polars uses cuDF as its engine — same performance, Polars syntax	@ 18:07
"GPU Polars uses CUDA-X as its execution engine" — it is not an alternative to cuDF, it is cuDF with a Polars API surface. The only change required: set `engine="gpu"` in the `.collect()` call. An email domain aggregation query runs in 78ms on CPU vs 11ms on GPU — 7×. Choose cuDF for pandas users, GPU Polars for Polars users; same hardware, same performance.
4. cuML accelerates highly parallelizable algorithms like UMAP: 40× on 2D, 20× on 3D — GPU makes it interactive	@ 37:45
2D UMAP projection: 48 seconds on CPU, 1.3 seconds on GPU (~37×). 3D UMAP: 56 seconds → 2.8 seconds (~20×). KNN graph construction, mutual reachability distance, and gradient descent layout steps are all parallelized. At these speeds, exploratory cluster visualization becomes interactive during model development rather than a batch job.
5. Both supervised and unsupervised algorithms can be accelerated: k-means 40× faster; XGBoost training 7× faster; grid search cross-validation 4×	@ 42:15
k-means: 4.9s CPU → 1.3s GPU, "over 40 times speedup." XGBoost single training run: 26.8s → 4.6s (7×). 5-fold cross-validation: 124s → 26.7s (~5×). 3-fold cross-validated grid search (9 XGBoost runs): 159s → 44s (4×). The acceleration applies to gradient/hessian computation, tree building, and loss evaluation — all four XGBoost phases are GPU-parallelized.
6. cuDF and cuML profilers give per-operation and line-by-line CPU vs GPU breakdown	@ 48:08
Both cuDF and cuML ship with two built-in profilers: an operation-level breakdown (what ran on CPU vs GPU) and a line-by-line profiler that pinpoints bottlenecks. Common performance killers to watch for: small batch sizes that cause repeated CPU↔GPU transfers, silent CPU fallbacks for unsupported operations, and complex string operations which don't accelerate well on GPU.
7. Triton Inference Server: dynamic batching from 128 to 1024, 100% success rate in testing	@ 01:16:26
Triton sits between client applications and GPUs, supporting Python, TensorRT, ONNX, and PyTorch backends. For the XGBoost fraud model (461 input features, 1 output probability), dynamic batching is configured from 128 to 1024. In the workshop demo, Triton reports "100% success rate, zero failure rate." Metrics surface latency, throughput, queue efficiency, and per-GPU utilization.

Cross-session themes

The five sessions share a common vocabulary captured in the takeaway tags. Here is what each theme amounted to across all sessions.

Connect With Experts

One of the advantages of being at the conference is the opportunity to meet with NVIDIA engineers working directly on these systems, and there were several opportunities to do so with the folks involved in the accelerated data stack, which I list below for reference.

🔗 Next-Gen Data Systems: GPU Acceleration for SQL and Vector Databases

Tanmay Gujar · Developer Technology Engineer, NVIDIA
Corey Nolet · Distinguished Engineer, Unstructured Data Processing & Database Acceleration, NVIDIA
Felipe Aramburu · Distinguished Solutions Architect, NVIDIA
Manas Singh · TPM Vector Search, NVIDIA
Benjamin Karsin · Senior Developer Technology Engineer, NVIDIA
Greg Kimball · Software Engineering Manager, NVIDIA

🔗 Boost Data Science Pipelines With Accelerated Libraries

Greg Kimball · Software Engineering Manager, NVIDIA
Alexandria Barghi · Senior Software Engineer, NVIDIA
Divye Gala · Senior Software Engineer, NVIDIA
Vyas Ramasubramani · Sr. Systems Software Engineer, NVIDIA
Bobby Evans · Distinguished Software Engineer, NVIDIA

← Part 1: Technical Deep Dives

Then there were many sessions covering these developments. In this post, I cover the four main technical ones.

1. Joshua Patterson and Todd Mostak open with a state-of-the-union on GPU-accelerated data processing where CPU analytics performance has stalled, how the NVIDIA ecosystem closes the gap, and what a next-generation analytics cluster looks like.

2. Greg Kimball and Zoltán Arnold Nagy then zoom into Presto specifically, walking through the concrete engineering required to turn GPU acceleration from theory into production reality at lakehouse scale.

3. Bobbi Yogatama and Xiangyao Yu bring that story down to a single node, showing how their Sirius extension turns DuckDB into a record-breaking analytics engine without changing a single query.

4. Finally, Felipe and Rodrigo Aramburu go one layer deeper with cuCascade and a custom telemetry tool, the composable building blocks behind Sirius’s ability to handle datasets far larger than GPU memory.

This is Part 1 of my series on Accelerated Analytics at GTC 2026. Read Part 2: Industry Use Cases and Training Labs.

Technical Deep Dives

🔗 The Era of GPU Data Processing: From SQL to Search and Back Again — S81769

Joshua Patterson · VP, Solutions Architecture, NVIDIA
Todd Mostak · Sr. Director of Engineering, NVIDIA

In this session, Joshua and Todd argue that CPU performance improvements on TPC-H have flattened over the past few years. They explore how the NVIDIA ecosystem can bring faster developments, either by adopting the new Vera CPU, migrating to cuDF/cuVS-accelerated databases, or redesigning data center clusters with analytics in mind to maximize the overlap of compute-intensive aggregations and joins vs IO-intensive tasks like shuffle and storage IO.

Takeaways

1. CPU TPC-H price/performance has flatlined	@ 04:27
Speakers argue that over the past few years CPU performance for analytics has not improved by orders of magnitude. For example, SQL Server and peers show only 15–20% gains every two years, probably just due to CPU refresh cycles. They also argue NVIDIA can help push the field forward.
2. Vera CPU accelerates analytics for free	@ 05:17
Vera CPU shows impressive performance improvements for analytics workloads. Some of the attributes that make it a good fit are: 1) massive amount of memory BW (1.2 TB/s) 2) "tons of cross-section BW that avoids the NUMA locality problem seen in multi-socket machines" 3) lots of BW/per core (14 GB/s per core, 3× that of x86/ARM) ➜ Vera gives analytics workloads a 2.5–3× lift with zero recompilation. Starburst, Kinetica, Redpanda all validate this. More GPU headroom per watt is the secondary win. Speakers show the performance on the TPC-DS Benchmark with Vera CPU compared to Intel and AMD for the Starburst Lakehouse, which is built on Trino.
3. Enterprise unstructured data: cuVS helps close the indexing gap	@ 08:54
90% of enterprise data is unstructured but only 10% is properly indexed. GPU-accelerated CAGRA (a graph nearest neighbor algorithm that outcompetes the CPU based HNSW) hits 10–13× faster indexing vs CPU HNSW at equivalent accuracy, on a cheaper instance. CuVS Plugin integration with Milvus, Elasticsearch, and OpenSearch means no rewrite required.
4. 2% of queries consume 92% of cluster resources.	@ 11:15
An interesting statistic from Trino usage at Airbnb shows that 2% of queries consume 92% of cluster resources. cuDF was created to help with this, first as a simple pandas drop-in replacement, but now more targeted at accelerating large-scale systems like databases and lakehouses such as Spark, Trino, Presto, and DuckDB. Companion libraries support efficient memory management, filesystem transfer, distributed communication, data prefetching, etc.
5. Sirius on DuckDB on a single GB300: 21 seconds for TPC-H 1 TB	@ 19:01
The speakers reference a later session on SiriusDB showcasing how fast the acceleration is. The Sirius cuDF-DuckDB integration delivers 7× TCO on ClickBench. Transwarp's TPC-DS 150 GB run clocked 26× faster than CPU DuckDB on the same GPU. Another accelerated database mentioned (@19:45) is Heavy AI / OmniSci, which will be open-sourced in late Q2 2026. It features LLVM compilation engine, Vulkan in-process rendering, geospatial/time series support, fast OLAP. Todd Mostak is now at NVIDIA leading this.
6. Theseus's async mini-executor architecture + GPU Direct Storage help break the memory wall	@ 20:41
The Theseus engine from Voltron was able to run TPCH-100TB on 2 DGX A100 servers (each with 8× A100 80GB/GPU, 640GB total GPU HBM2e memory, 2TB/s per-GPU memory bandwidth, connected via NVLink 3.0 at 600GB/s) + 200 Gbps InfiniBand + GPU Direct Storage. GDS enables all GPUs to talk directly to storage network instead of waiting for data to be served via a single CPU attached to the system. The practical consequence: petabytes of NVMe storage becomes queryable working memory. Replacing the monolithic executor with specialized actors — compute, memory-tier management, prefetch/decode, networking — enabled true overlap of I/O and compute. The speakers described it as "an agent swarm for query processing." See the Theseus paper for more on the architecture.
7. SPACE MICE: a reference design to push data analytics clusters to the next level	@ 24:53
This design consists of 1) 1 GB200 NVL72 2) 9 DGX B300 3) 10 RTX PRO 6000 nodes 4) 20 RTX 4500. NVLink is mainly for all GPU-to-GPU shuffle (east-west, ~1.8 TB/s) while CX8 NICs are dedicated to storage I/O (north-south, 3–4 TB/s). The two networks run simultaneously and non-overlapping. 18 GPUs × 100 TB reachable per TB of GPU memory = ~1.8 PB per rack. Vera Rubin (144 GPUs) pushes this to ~5 PB.

The next session zooms in on one of the distributed engines in that ecosystem — Presto — and walks through the concrete engineering work required to make GPU acceleration practical at scale.

🔗 Unlock Fast, Cost-Effective Interactive Analytics on Massive Data Lakehouses — S81563

Greg Kimball · Software Engineering Manager, NVIDIA
Zoltán Arnold Nagy · Sr. Software Engineer, IBM Research

This session focuses on the distributed SQL engine Presto and recent performance improvement for its GPU accelerated mode. It is presented by Greg from NVIDIA’s cuDF team, and Zoltan from IBM Research and working on Presto. Presto C++ workers use the open source project Velox as single node query engine and Velox provides experimental support for GPUs via the RAPIDS AI libraries including cuDF. See this article from last year on Velox over cuDF. The picture below shows the evolution of the Presto project from native Java to Native C++. Together with Spark, Presto is one of the most stable and widely adopted open source systems for distributed data processing.

Takeaways

1. Presto GPU node is 30× faster than an 8-node Grace CPU cluster on TPC-H SF1K	@ 05:29
The comparison baseline is "each node has a two socket Grace CPU" running the 22-query TPC-H-derived suite. Four B200 GPUs drop that to "30 times faster speed" — supports caching ingested Parquet data. Powered by Velox and cuDF under the hood.
2. Table scan & Parquet I/O dominates TPC-H runtime — not compute	@ 07:22
The operator breakdown at SF-100 and SF-1K shows the "big blue part, parquet data source" dwarfs hash join, filter, and partitioning combined and accounts for 60-70% of the runtime. Tuning Presto GPU is almost entirely an I/O problem.
3. Picking a good file format encoding + using NUMA pinning improves performance by ~30%	@ 08:29
"The Delta binary packed encoding came out with Parquet… it makes a huge difference on GPU execution." For integer physical types, switching to DBP encoding is described as a way to make your data lake "scream fast on GPU." Also, NUMA pinning on DGX boxes gives a significant GPU speedup. On a DGX, one CPU is connected close to four of the GPUs. Keeping that CPU in charge of all CUDA launching and copy activity for its four GPUs improves throughput.
4. 10x lower TCO for Presto GPU when running all TPC-H queries	@ 09:33
Greg's cost-per-run chart shows that for SF1K (1TB), a single GPU delivers better price/performance than four. At SF3K (3TB), three GPUs beat eight. The headline: "around a 10x benefit using Presto GPU versus Presto CPU" — but only if you optimize for cost, not raw speed. Perhaps counterintuitively, the $ cost per performance is more pronounced for smaller cluster with less capable hardware, "that's where the TCO story is" says Greg.
5. UCX exchange operator enables 900 GB/s NVLink5 for data shuffle	@ 13:22
Zoltan's comparison: normal Linux kernel TCP "just doesn't have the bandwidth" once you go multi-hundred gigabit. NVLink 5 on Blackwell delivers "1800 gigabytes a second bi-directional bandwidth" — "900 gigabytes a second to move between GPU to GPU." Presto uses UCXExchange to select NVLink when available and falls back gracefully to RoCE or TCP otherwise. It's a drop-in replacement. Zoltan shows a TPC-H SF1K benchmark comparison between 16x Grace CPU (8 nodes), 8xA100 with Http exchange and 8xA100 with CuDFExchange and shows the dramatic drop from 690s ➜ 453s ➜ 60s!
6. S3 on cloud has a *per-VM* BW ceiling AWS doesn't officially document	@ 19:18
AWS will throttle traffic from a single VM. The workaround is to spin up CPU instances that pull from S3 in parallel and write directly to GPU memory bypassing host memory entirely. On B300, "AWS gives you 800 gigabit per GPU (100GB/s), and it has eight GPUs, 6.4 terabits a second of bandwidth (800GB/s) to fill up", performance you would never be able to achieve on a single S3 connection.
7. Velox "Async data Cache" improvements drop the runtime further for hot queries	@ 24:29
By reading Parquet metadata first to predict what will be fetched, nearby small reads are combined into larger requests: "just coalescing the reads drops your request number from 700 to around 200." Parallelism is preserved but metadata overhead drops, pushing the hot run to ~20 seconds — described as "basically saturating the PCI Express bus on the RTX 6000."
8. UDF needs more support in libcuDF + overlapping shuffle with Parquet	@ 34:30
JIT compilation for supporting User defined functions in libcudf will be a big part of the story needed to bridge the gap to wider adoption in the industry. Also supporting parallel IO traffic from different sources such as for shuffle and table scan was a big challenge. A recent article digs into the latest JIT improvements in cuDF for string transform UDFs Efficient Transforms in cuDF Using JIT Compilation

From distributed clusters to single-node: the next session covers Sirius, the GPU extension that turns DuckDB into a record-breaking analytics engine.

🔗 Achieving 8x Lower Cost Analytics with GPU-Accelerated DuckDB — S81870

Bobbi Yogatama · Sr. Systems Software Engineer, NVIDIA
Xiangyao Yu · Assistant Professor, University of Wisconsin-Madison

GPU is becoming a general purpose computing system. OLAP data systems stand to benefit because there are lots of parallelizable algorithms in analytics. Recent SW/HW trends are helping overcome traditional challenges to GPU accelerated databases like GPU memory, CPU-GPU PCIe bottleneck, and engineering complexity. Sirius uses DuckDB’s modularity to bring GPU acceleration without requiring any changes to end-user queries. More in the recent article NVIDIA CUDA-X Powers the New Sirius GPU Engine for DuckDB, Setting ClickBench Records

Takeaways

1. ClickBench world record: Sirius holds #1 and #2 on hot run at $2/hr	@ 06:37
Sirius took both first and second place on the ClickBench hot-run leaderboard — GH200 on LambdaLabs and H100 on AWS — and holds first on the combined run. The GH200 instance costs $2/hr, far below the CPU-based systems it beats. When you normalize for cost, the gap widens even further.
2. GPU-only execution: fall back to DuckDB entirely, never hybrid	@ 07:53
Sirius does not mix CPU and GPU execution within a single query. When it encounters an unsupported operator, it falls back cleanly to stock DuckDB — "the worst performance that you can get is a DuckDB performance, which is pretty damn good." This avoids the complexity and performance cliffs of hybrid scheduling while preserving correctness. It's interesting that the direction of newer systems seems to be GPU or CPU only rather than hybrid CPU-GPU execution which has been a topic of research in the past few years from the speaker's own work with Mordred as well as other projects like CoGaDB, Ocelot, and HetExchange. See Yogatama's phD thesis for more details, including Sirius.
3. Live demo: TPC-H Q9 on 1TB Parquet — 2.5s vs 16s, data exceeds GPU memory	@ 10:30
On a GB300 DGX station, Sirius completes TPC-H Query 9 against a 1 TB Parquet dataset in 2.5 seconds; DuckDB on CPU takes 16 seconds. The data intentionally exceeds GPU memory — cuCascade spills transparently to host. Results between the two engines are identical. The CPU instance was c8id.metal-96xl which contains a 384 cores Intel Xeon CPU with 768GiB memory.
4. DuckDB creator announces Sirius as the official GPU extension	@ 12:03
Hannes Mühleisen, creator of DuckDB, appeared on stage to announce that Sirius will become a core DuckDB extension — "the blessed way of running queries on GPUs with DuckDB." This signals GPU-accelerated execution moving from an external experiment to an officially endorsed path in the DuckDB ecosystem.
5. Sirius implements separate executors for compute and spilling	@ 24:43
In Sirius a collection of Data Batches are managed by a Data Repository Manager that relies on cuCascade and a downgrader executor to manage spilling either back to the CPU host memory or to the GPU. Data spilling can happen simultaneously with the data processing on other data in the GPU. New operators are added in the plan to support spilling from GPUs.
6. Sirius achieves 5x speed for DuckDB on TPC-H queries using 1 DGX GB300	@ 24:43
Running the full TPC-H SF1K (1 TB) suite on a DGX GB300 node, Sirius completes all 22 queries in 21 seconds total — a 5× speedup over DuckDB. The presenter notes: "the TPC-H official record is actually slower than 21 seconds," implying this result, while informal, would be the fastest ever recorded at this scale factor. Compare with Presto performance shown in previous session which got 24s on a 1/2 DGX B200 node (using only 4x B200)
7. 9× TCO at the same $2/hr cost: GH200 Lambda Lab vs AWS CPU instance	@ 25:53
On the SF300 hot-run benchmark, Sirius on a GH200 (2022 Grace-Hopper on LambdaLabs, $2/hr) delivers 9× better TCO than DuckDB running on a comparably priced AWS Intel CPU instance (r8i.8xlarge). Same dollar spend, same wallclock budget — 9× the throughput. The implication: GPU instances are no longer a premium option; at equivalent cost they dominate.
8. Sirius manages its own pinned memory cache — 2× transfer speedup on GB300	@ 32:43
Rather than relying on the OS page cache, Sirius maintains its own pinned memory buffer. This allows it to transfer data from host in compressed form and decompress on the GPU, bypassing the pageable memory path entirely. On GB300, this yields up to 2× higher sustained transfer throughput — a gain that disappears if you let the OS manage the cache.

Sirius’s spilling story depends on a library that wasn’t widely known until this conference. The final session pulls back the curtain on cuCascade and the telemetry tooling built alongside it.

🔗 Shatter the Memory Wall: Composable Building Blocks for Massive Scale Analytics — S81873

Felipe Aramburu · Distinguished Solutions Architect, NVIDIA
Rodrigo Aramburu · Developer Relations for Data Processing, NVIDIA

This talk is all about piercing through the GPU memory wall and the experience Felipe and Rodrigo had working on this problem from BlazingSQL to Theseus engine and now at Nvidia. They believe that GPU memory capacity is no longer as big of a problem, rather it is data movement that is now the real cost and the primary challenge. They also emphasize memory frugality as the idea of using more compute if it leads to fewer memory accesses.

Takeaways

1. GPU main challenges: *memory movement* + *memory frugality*	@ 01:44
The session opens with introducing the memory wall challenge and conceptually inversion of the CPU paradigm. On CPU, you minimize compute to save memory bandwidth. On GPU, the argument is the opposite: "pay a higher computational cost because there is leftover compute inside the GPU to shrink the amount of bytes for the amount of time that you necessarily have them in memory." Memory is the scarce resource; compute is not.
2. Theseus engine ran 100 TB TPC-H on 1.28 TB GPU working memory — 20× ratio via spill	@ 03:09
Theseus — the research system behind cuCascade — processed the full 100 TB TPC-H/TPC-DS suite using only 1.28 TB of GPU working memory. With host memory (9.28 TB total), the effective data-to-GPU-memory ratio is roughly 20×. This is the existence proof for cuCascade's design: a single node can handle workloads orders of magnitude larger than its HBM capacity.
3. *cuCascade* solves topology, memory management, and data movement challenges using MICE design principles	@ 05:00
MICE: Modular, Interoperable, Composable, and Extensible enabling a multitude of use cases for cuCascade users: Engine developers can adopt it today to add multi-tier spill, memory reservation, and topology-aware scheduling without building these components from scratch, only choosing what they need or customizing for their use cases. Memory Reservation helps avoid oversubscribing memory and inevitable extra spilling and OOM. Allocation Policies are available to handle cases where Allocators reach their memory maximum. Data Format encoding, decoding, and conversion between different formats via Data Batch Representations (GPU cuDF Table, Host CPU fixed size pinned memory pages, custom). Automatic Topology discovery eliminates the human errors common with manual configuration file setup.
4. Sirius TPC-H-1k - Q9 required cuCascade's downgrade policy to pass	@ 19:01
Sirius was one of the first projects to integrate cuCascade for spilling (another one is RAPIDS MPF, specifically used for multi-GPU data pipelines like shuffle). During the SF1K run, Query 9 was the query that most aggressively exceeded GPU memory — "that's the one that actually ended up blowing up on us quite aggressively." Without cuCascade's downgrade policy (which detects memory pressure and transparently degrades to host spill), the query would have OOM'd and terminated. The downgrade policy is what separates a working system from a fragile one. Note the focus on Query 9 from the Sirius team in their talk as well. cuCascade was built from a blank slate starting January 1st, 2026. Three months later it was live on stage. cuCascade made possible Sirius' performance on TPCH-1k of 21 seconds across all 22 queries on a GB300 DGX station, preventing OOM failures in the hardest queries.
5. Built a custom telemetry tool at under 0.1% overhead - can be running in the background	@ 22:58
The custom Rust-based telemetry layer captures cluster-wide data movement, per-operator throughput, and memory tier transitions with less than 0.1% runtime overhead — "you're not gonna have to pay this penalty." Existing tools (OpenTelemetry, Grafana, Prometheus, NSYS) were evaluated and rejected as too heavyweight for always-on use at this scale. The system was built from scratch specifically for distributed GPU analytics. Query plan operators can be mapped to the metrics collected during its run. Data movement can also be tracked as it moves across memory space. The tool has been used with SiriusDB.
6. cuCascade makes cuDF multi-GPU — a capability cuDF doesn't have natively	@ 36:06
Speakers highlighted recent progress improving libcudf, particularly the use of the recent nvcomp compression engine and support for new file formats like vortex. The evolution of cuDF could be more integration with cuCascade, providing the topology-aware routing and data movement layer that lets data operators cross the GPU boundaries. "cuDF itself is not intended to be a multi-GPU library. But leveraging it with cuCascade... you're able to do that multi-GPU computation with cuDF."

Cross-session themes

The four sessions share a common vocabulary captured in the takeaway tags. Here is what each theme amounted to across all sessions.

Part 2: Industry Use Cases and Training Labs →

Previously in Parts 1 & 2: After setting the scene at GTC, Jensen spent the first half of the keynote celebrating CUDA’s 20-year flywheel, making the case for NVIDIA’s role in accelerating enterprise analytics (with partnerships from IBM, Dell, and Google Cloud), reviewing the CUDA-X library ecosystem, and laying out the economics of the AI inference boom, framing the $1T infrastructure wave ahead and how GB300 NVL72 became the inference king on tokens-per-watt.

Summary of Part 3 sections

The second half of the keynote covered the following sections:

Full Vera Rubin hardware stack — GPU, NVLink, Rubin Ultra, and Spectrum-X Groq LPX + DSX platform for AI factory optimization (38min)

A Decade of AI Infrastructure Innovation: From DGX-1 to Vera Rubin · 3:30min

Jensen narrates NVIDIA's decade of data center infrastructure innovation: 2016 DGX-1 —packages 8 Pascal GPUs, first supercomputer built for deep learning, one delivered to openAI that year 2017 Volta — introduces NVLink 2 switch, GPU-to-GPU interconnect inside nodes 2019 Mellanox acquisition — allows the data center to become a single unit of computing 2020 Ampere / DGX A100 SuperPOD — brings scale-up via NVLink 3, scale-out via ConnectX-6 InfiniBand 2022 Hopper — supports FP8 Transformer Engine for Gen AI, NVLink 4, ConnectX-7 2024 Blackwell / NVL72 — achieves 130 TB/s bandwidth and a deeper rack-level co-design for top performance 2026 Vera Rubin — built for agentic AI · 35× throughput/MW · 40M× cumulative compute over the decade

NVIDIA Vera Rubin · 2:27min

Jensen introduces the Vera Rubin hardware on stage

NVIDIA Vera Rubin, NVLink and Groq · 1:36min

He makes some interesting observations: with the recent tray designs, installation time falls down from 2 days to 2 hours. Also cooling is done with hot water at 45 degrees.

Spectrum-X Switch, Co-Packaged Optics, Vera and BlueField-4 · 2:09min

discusses the 8 grok 3rd gen tray which is in production and shows the Spectrum Co-packaged optics switch. Vera brings 2x performance per watt. ConnectX9 and storage platform are powered by Vera CPU.

Rubin Ultra · 2:03min

Jensen also shows VR Ultra and the new Kyber rack that can connect 144 gpus that now slide vertically into the rack. He also shows the new NVLink tray design that sits behind, also vertically.

Inference Performance and Efficiency Drive Company Results · 9:35min

Jensen's main message to CEOs is how they will need to evaluate their company's usage of tokens, and study the tradeoff between throughput (as Token per Sec per MW) vs Interactivity (as token per second per user). Input and output Context length are growing and usage depends on use case. Jensen shows a graph partitionned by kind of model at different prices and how nvidia's chips performs on this tradeoff. The value of Ultra lays enabling bigger more interactive models with better energy efficiencies. GB NVL72 has increased the medium tier by 35x and Vera rubin will increase high tier by 3x and increased premium tier by 10x. Rubin + Groq LPX increase most valuable tier by 35x. Ultra enables even better interactivity.

Uniting Processors of Extreme Performances · 3:36min

Jensen delves into the performance of Groq, which has high SRAM capacity (500MB) at very high throughput (150TB). This complements Rubin's 288GB of HBM4 memory at 22TB/s by providing statically compiled compute primitives specially used for the decode Feed Forward phase of AI inference, and helps achieve very low latency for token generation.

NVIDIA Groq 3 LPX · 0:38min

Jensen shows Groq LPX manufactured by samsung and say he expects to ship by Q3 this year.

Announcing NVIDIA Launch Partners · 1:56min

shows all the AI labs, cloud, and OEM/ODM that will launch Vera Rubin. Expects production in the 1000s per week. also shows launch partners for Vera CPU and BlueField storage systems

NVIDIA Vera Rubin: 7 Chips – 5 Rack Systems · 1:02min

Jensen shows how much progress was made by comparing x86 hopper generation to Vera Rubin GiGaWatt factory. VR can generate 350x more tokens per seconds than Hopper thanks to 35x more scale up BW per Rack (at 288TB/s) and with half as many GPUs.

NVIDIA Extreme Co-Design Delivering X-Factors Every Year · 3:37min

shows the roadmap to 2028 with Feynman. Oberon will enable scale up in both copper and optical to support NVL576 racks (Kyber) and then NVL1152 for Feynman with Kyber.

NVIDIA DSX AI Factory Platform · 2:10min

Jensen describes the importance of the NVIDIA Omniverse solution to help design GW factory digital twins and reach max performance at lowest possible energy usage. He talks about tools for simulation such as DSX Sim, DSX exchange, DSX flex power management and DSX Max Q for dynamic power adjustment in the data center.

How AI Factories Maximize Tokens, Power, and Profit With NVIDIA DSX · 3:25min

The video summarizes all the components of the DSX AI factory platform

Space-1 Vera Rubin Module · 0:43min

Jensen briefly mentions NVIDIA's foray in space with Space-1 Vera Rubin module and mentions the challenge of cooling in space.

OpenClaw, NemoClaw, Open Model Coalition (19min)

NemoClaw for OpenClaw · 1:24min

Jensen is very excited about OpenClaw, the most popular open source in history, with the fastest project to get the most stars in github

OpenClaw: The ChatGPT Moment for Long-Running, Autonomous Agents · 9:14min

He shows how openclaw grew as a project to 340k stars on GitHub since the end of january 2026. It is the operating system of agents and every enterprise will soon need an OpenClaw strategy.

NVIDIA Nemotron and Open Models · 0:28min

Jensen announces new models in Nvidia's open foundation model families: bioNemo for biomedical AI, earth-2 for Ai physics, Nemotron for Agentic AI, Cosmos for Physical AI, GROOT for Robotics, and Alpamayo for Autonomous Vehicles.

How NVIDIA Open Models Power Every Industry's AI · 4:17min

The video shows models from each of the Nvidia families. They are world class, doing well on benchmarks. Shows nemotron-3-super-120b as #4 on best open model for openClaw. Nemotron 3 ultra.

Announcing Global AI Leaders Join NVIDIA Nemotron Coalition · 2:57min

Jensen announces the NVIDIA Nemotron Coalition1 aimed at accelerating the co-development of open AI frontier models with partners Black Forest Labs, Cursor, LangChain, Mistral AI, Perplexity, Reflection AI, Sarvam and Thinking Machines Lab

Announcing NVIDIA NemoClaw Reference OpenClaw · 0:39min

Jensen says the openClaw event cannot be understated and is as big as linux and html. In response, Nvidia is releasing NemoClaw, a reference enterprise-ready solution to secure openClaw deployments inside enterprises.

Robotics, Physical AI, & recap (14min)

Physical AI and Robotics · 3:11min

Jensen talks robots, mentions there are 110 robots at GTC, announces 4 new auto partners: BYD, Hyundai, Nissan, and Geely are joining Mercedes, Toyota, and GM to build robotaxi technologies. Jensen also announces a partnership with Uber to launch a large fleet of autonomous vehicles for 2027 on the NVIDIA DRIVE AV stack2

The Age of Physical AI and Robotics · 4:27min

This video shows how autonomous cars have been improving thanks to NVIDIA's and partner ecosystem.

Olaf Takes the Stage With Jensen Huang · 1:55min

Jensen welcomes the only guest at the keynote. Last year it was a Star Wars inspired robot "blue", this year it is Olaf from Frozen

Official Keynote Closing Video · 4:02min

The Keynote ends with a generated video recapping the keynote with a jensen emoticon playing harmonica in the forest, surrounded by a band of robots playing instruments, a bit silly for my tast but again showcasing the power of the tools

Full keynote is available here and the slides here.

← Part 2: Intro, Analytics, CUDA-X & Inference

References

1. NVIDIA Launches Nemotron Coalition of Leading Global AI Labs to Advance Open Frontier Models — nvidianews.nvidia.com
2. NVIDIA DRIVE Hyperion Achieves Level 4 Autonomy with Uber Partnership — nvidianews.nvidia.com

I was back this year for the 2026 edition of NVIDIA’s GTC conference held at the San Jose Convention Center and surroundings from March 16-19.

Like last year, there was plenty of energy at the conference with attendee numbers said to have reached more than 30k. The conference was packed with interesting technical sessions on new developments in the NVIDIA ecosystem including technical sessions on CUDA-X libraries and industry and state partners presenting how they have integrated the NVIDIA stack into their products.

The conference expanded to the nearby hotels for additional space, the security check-ins were moved out of the convention center and onto the street and an additional lunch section was added in the parking lot in front of the Hylton Hotel on S. Almaden Road.

Finally the keynote was held like previous years at the SAP Center, a 15min walk away, with a larger pavilion setup just outside of it for free coffee and pastries and for hosting the “pre-game” show featuring executives and technical leaders of companies working with NVIDIA. Other than that, the conference looks about the same as last year!

In this post, I will only cover the keynote and will delve into the sessions I attended and the exhibit hall in followup posts.

The Keynote

The keynote was the main event held on the first day of conference and it was moved ahead to 11AM, making it easier to get there early and avoid long lines. Here are some pictures from the packed SAP Center stadium where it was held

As he does every year, Jensen showed hardware on stage, including the new Vera Rubin tray, the new Groq LPX tray, and the new Co-Packaged Optical switch tray for scaling up. He also showed Vera Ultra and its Kyber rack design where trays are inserted vertically instead of horizontally. The exhibit hall had all these nicely on display.

One interesting aspect I wasn’t expecting was Jensen spending 18 minutes almost at the outset of the keynote talking about how NVIDIA’s libraries are sitting at the foundation of accelerated analytics in Enterprise structured and unstructured data. He announced several partnerships with the cloud providers and highlighted how many of NVIDIA’s solutions accelerate CSP’s offerings. I will cover the analytics aspects of the conference in a separate post.

Jensen reveled in being crowned “inference king” by Semianalysis for GB NVL72 system! Also check their review¹ of the GTC conference.

CUDA is 20 years old

CUDA is now 20 years old, and Jensen celebrated that by spending a few extra minutes talking about its core importance to NVIDIA as a company. He emphasized the crucial flywheel role that CUDA-X plays for NVIDIA as an ecosystem of hundreds of libraries for accelerating all kinds of workloads. As the install base for CUDA has grown, reaching hundreds of millions of GPUs deployed around the world, so has the reach to developers, leading to new breakthroughs in many domains, each creating new markets and new customers who then want to buy more GPUs, further growing the user base.

The Vera Rubin POD is expanding: Seven Chips, Five Rack-scale Systems

One of the major reveals at this year’s conference and worth re-emphasizing is the addition of the Groq LPU to speed up AI inference and the addition of co-packaged optics for the scale network. The NVIDIA AI factory is built around five rack types, and a full Vera Rubin POD “features 40 racks, 1.2 quadrillion transistors, nearly 20,000 NVIDIA dies, 1,152 NVIDIA Rubin GPUs, 60 exaflops, and 10 PB/s total scale-up bandwidth”²

1. The VR NVL72 GPU node
2. The newly announced companion Groq LPU rack offloading part of the AI inference pass (decode)
3. BlueField-4 to store KV cache offloaded from the GPU memory
4. Vera CPU Rack for more general Agentic workloads and RL, and
5. the Spectrum-6 networking rack to connect the whole POD.

Summary of the Keynote by section

Here’s a short breakdown of the main section Jensen covered in the Keynote.

Find the breakdown below, linking directly into each section on the YouTube video, along with summary notes and section durations.

Intro, Cuda flywheel, Graphics improvements (16min)

Tokens, the Building Blocks of AI · 3:15min

Keynotes start with an inspiring video describing how AI tokens are the main "commodity" produced by AI factories and their power to unlock new knowledge and possibilities

Welcome to GTC 2026 · 2:47min

Jensen enters the stage and gives introductory remarks thanking the pre-game show hosts, and also how the conference will be covering the AI 5 layer cake, a reference to his blog post that divides the stack along: Energy, Chips, Infrastructure, Models, and Applications

20 Years of CUDA · 4:21min

Jensen reviews the flywheel that Cuda software has been enabling for the past 20 years.

GeForce · 3:27min

CUDA made GPUs programmable first on the consumer product GeForce in 2006, which then enabled the deep learning community to test the viability of training neural networks and launched the new AI revolution.

DLSS 5 · 2:29min

Jensen shows a video featuring the new DLSS5 capability, a Neural rendering technology that fuses 3d Graphics with AI to give more beautiful and detailed textures to videos. Video details triggered a backlash from game developers.

Accelerated Analytics (22min)

Structured Data is the Ground Truth of AI · 3:26min

Jensen says Analytics are ripe for acceleration with the arrival of AI agent and emphasizes CuDF and CuVS as foundation libraries powering the whole ecosystem.

IBM Reinvents Data Processing With NVIDIA · 18:10min

He announced partnerships with IBM for Watson-X, a major contributed to open source Presto C++ and user of Spark over Rapids, NVIDIA's own accelerated dataframe libraries. Also announced were partnerships with Dell for an AI platform over RTX6000 servers, and for Google Cloud's AI Hypercomputer. Jensen highlights NVIDIA's stack that accelerate many of the CSP's offerings for AI and he spent some time reviewing them for different cloud providers.

Cuda-X review and AI native companies (7min)

AI Inference Inflection + Overview of datacenter efficiency (Tokens/Watt) vs interactivity (Tokens/s per user) across different tiers (22min)

Inference Inflection Arrives · 4:42min

Jensen highlights 3 key moments for AI inference in the past 2 years: 2023) ChatGPT is released 2024) reasoning AI model with o1 and o3 takeoff and in 2025) Claude code agentic system revolutionizes software engineering.

"The inflection point for inference has arrived." · 1:40min

Agent thinking capabilities led to an explosion in the amount of inference by 10,000x since ChatGPT was released. Coupled with 100x increase in end-user demand, Jensen says we have 1M x more inference demand since 2023. We are now at an inflection point for inference

Inference Inflection Drives Strong Growth · 8:30min

Last year Jensen saw $500B demand for blackwell. This year through 2027, he see $1Tr in infrastructure investments on NVIDIA mainly for inference. 60% of the business is for hyperscalers (some of it for internal use), and 40% is all the rest, such as regional or sovereign cloud, enterprise, supercomputers and all the rest. GB + NVL72 + inference over fp4 for training , dynamo, tensorRT. DGX Cloud.

NVIDIA Extreme Co-Design Revolutionized Token Cost · 3:57min

Datacenters are constrained by a fixed amount of power (Watts) available. Emphasize Tokens Per Watt as the metric to maximize, and interactivity (token/s per User) as a use case differentiator.

InferenceMAX King · 1:23min

Shows how GB300NVL72 has improved on both efficiency and cost for inference and has been recognized by semianalysis as inference King!

NVIDIA is the Global Standard for AI Inference at Scale · 0:33min

Inference service providers should be seen as token factories. The output token rate from companies like eigen AI, together.ai, nebius, etc. has increased very fast, now reaching 400+ token/s for kimiK2.5 reasoning agent. Also see artificial analysis for a breakdown between providers.

AI Factories are the Industrial Infrastructure of the AI Era · 1:10min

Inference drives revenues and Token effectiveness is the most important metric.

Full Vera Rubin hardware stack — GPU, NVLink, Rubin Ultra, and Spectrum-X Groq LPX + DSX platform for AI factory optimization (38min)

A Decade of AI Infrastructure Innovation: From DGX-1 to Vera Rubin · 3:30min

Jensen narrates NVIDIA's decade of data center infrastructure innovation: 2016 DGX-1 — packages 8 Pascal GPUs, first supercomputer built for deep learning, one delivered to OpenAI that year 2017 Volta — introduces NVLink 2 switch, GPU-to-GPU interconnect inside nodes 2019 Mellanox acquisition — allows the data center to become a single unit of computing 2020 Ampere / DGX A100 SuperPOD — brings scale-up via NVLink 3, scale-out via ConnectX-6 InfiniBand 2022 Hopper — supports FP8 Transformer Engine for Gen AI, NVLink 4, ConnectX-7 2024 Blackwell / NVL72 — achieves 130 TB/s bandwidth and a deeper rack-level co-design for top performance 2026 Vera Rubin — built for agentic AI · 35× throughput/MW · 40M× cumulative compute over the decade

NVIDIA Vera Rubin · 2:27min

Jensen introduces the Vera Rubin hardware on stage

NVIDIA Vera Rubin, NVLink and Groq · 1:36min

He makes some interesting observations: with the recent tray designs, installation time falls down from 2 days to 2 hours. Also cooling is done with hot water at 45 degrees.

Spectrum-X Switch, Co-Packaged Optics, Vera and BlueField-4 · 2:09min

discusses the 8 grok 3rd gen tray which is in production and shows the Spectrum Co-packaged optics switch. Vera brings 2x performance per watt. ConnectX9 and storage platform are powered by Vera CPU.

Rubin Ultra · 2:03min

Jensen also shows VR Ultra and the new Kyber rack that can connect 144 gpus that now slide vertically into the rack. He also shows the new NVLink tray design that sits behind, also vertically.

Inference Performance and Efficiency Drive Company Results · 9:35min

Jensen's main message to CEOs is how they will need to evaluate their company's usage of tokens, and study the tradeoff between throughput (as Token per Sec per MW) vs Interactivity (as token per second per user). Input and output Context length are growing and usage depends on use case. Jensen shows a graph partitionned by kind of model at different prices and how nvidia's chips performs on this tradeoff. The value of Ultra lays enabling bigger more interactive models with better energy efficiencies. GB NVL72 has increased the medium tier by 35x and Vera rubin will increase high tier by 3x and increased premium tier by 10x. Rubin + Groq LPX increase most valuable tier by 35x. Ultra enables even better interactivity.

Uniting Processors of Extreme Performances · 3:36min

Jensen delves into the performance of Groq, which has high SRAM capacity (500MB) at very high throughput (150TB). This complements Rubin's 288GB of HBM4 memory at 22TB/s by providing statically compiled compute primitives specially used for the decode Feed Forward phase of AI inference, and helps achieve very low latency for token generation.

NVIDIA Groq 3 LPX · 0:38min

Jensen shows Groq LPX manufactured by samsung and say he expects to ship by Q3 this year.

Announcing NVIDIA Launch Partners · 1:56min

shows all the AI labs, cloud, and OEM/ODM that will launch Vera Rubin. Expects production in the 1000s per week. also shows launch partners for Vera CPU and BlueField storage systems

NVIDIA Vera Rubin: 7 Chips – 5 Rack Systems · 1:02min

Jensen shows how much progress was made by comparing x86 hopper generation to Vera Rubin GiGaWatt factory. VR can generate 350x more tokens per seconds than Hopper thanks to 35x more scale up BW per Rack (at 288TB/s) and with half as many GPUs.

NVIDIA Extreme Co-Design Delivering X-Factors Every Year · 3:37min

shows the roadmap to 2028 with Feynman. Oberon will enable scale up in both copper and optical to support NVL576 racks (Kyber) and then NVL1152 for Feynman with Kyber.

NVIDIA DSX AI Factory Platform · 2:10min

Jensen describes the importance of the NVIDIA Omniverse solution to help design GW factory digital twins and reach max performance at lowest possible energy usage. He talks about tools for simulation such as DSX Sim, DSX exchange, DSX flex power management and DSX Max Q for dynamic power adjustment in the data center.

How AI Factories Maximize Tokens, Power, and Profit With NVIDIA DSX · 3:25min

The video summarizes all the components of the DSX AI factory platform

Space-1 Vera Rubin Module · 0:43min

Jensen briefly mentions NVIDIA's foray in space with Space-1 Vera Rubin module and mentions the challenge of cooling in space.

OpenClaw, NemoClaw, Open Model Coalition (19min)

NemoClaw for OpenClaw · 1:24min

Jensen is very excited about OpenClaw, the most popular open source in history, with the fastest project to get the most stars in github

OpenClaw: The ChatGPT Moment for Long-Running, Autonomous Agents · 9:14min

He shows how openclaw grew as a project to 340k stars on GitHub since the end of january 2026. It is the operating system of agents and every enterprise will soon need an OpenClaw strategy.

NVIDIA Nemotron and Open Models · 0:28min

Jensen announces new models in Nvidia's open foundation model families: bioNemo for biomedical AI, earth-2 for Ai physics, Nemotron for Agentic AI, Cosmos for Physical AI, GROOT for Robotics, and Alpamayo for Autonomous Vehicles.

How NVIDIA Open Models Power Every Industry's AI · 4:17min

The video shows models from each of the Nvidia families. They are world class, doing well on benchmarks. Shows nemotron-3-super-120b as #4 on best open model for openClaw. Nemotron 3 ultra.

Announcing Global AI Leaders Join NVIDIA Nemotron Coalition · 2:57min

Jensen announces the NVIDIA Nemotron Coalition3 aimed at accelerating the co-development of open AI frontier models with partners Black Forest Labs, Cursor, LangChain, Mistral AI, Perplexity, Reflection AI, Sarvam and Thinking Machines Lab

Announcing NVIDIA NemoClaw Reference OpenClaw · 0:39min

Jensen says the openClaw event cannot be understated and is as big as linux and html. In response, Nvidia is releasing NemoClaw, a reference enterprise-ready solution to secure openClaw deployments inside enterprises.

Robotics, Physical AI, & recap (14min)

Physical AI and Robotics · 3:11min

Jensen talks robots, mentions there are 110 robots at GTC, announces 4 new auto partners: BYD, Hyundai, Nissan, and Geely are joining Mercedes, Toyota, and GM to build robotaxi technologies. Jensen also announces a partnership with Uber to launch a large fleet of autonomous vehicles for 2027 on the NVIDIA DRIVE AV stack4

The Age of Physical AI and Robotics · 4:27min

This video shows how autonomous cars have been improving thanks to NVIDIA's and partner ecosystem.

Olaf Takes the Stage With Jensen Huang · 1:55min

Jensen welcomes the only guest at the keynote. Last year it was a Star Wars inspired robot "blue", this year it is Olaf from Frozen

Official Keynote Closing Video · 4:02min

The Keynote ends with a generated video recapping the keynote with a jensen emoticon playing harmonica in the forest, surrounded by a band of robots playing instruments, a bit silly for my taste but again showcasing the power of the tools

Full keynote is available here and the slides here.

References

1. Semianalysis — Nvidia: The Inference Kingdom Expands — newsletter.semianalysis.com
2. NVIDIA — Vera Rubin POD: Seven Chips, Five Rack-Scale Systems, One AI Supercomputer — developer.nvidia.com
3. NVIDIA Launches Nemotron Coalition of Leading Global AI Labs to Advance Open Frontier Models — nvidianews.nvidia.com
4. NVIDIA DRIVE Hyperion Achieves Level 4 Autonomy with Uber Partnership — nvidianews.nvidia.com

Previously in Part 1: I covered the conference’s atmosphere, shared a bit about the keynote’s energy, NVIDIA’s celebration of CUDA’s 20th anniversary and the flywheel it has created, and how the introduction of the new Groq rack expended NIVDIA’s AI Factory Pod, now a five-rack system combining the Groq LPX, BlueField-4, Vera CPU, and Spectrum-6 networking racks alongside the Vera Rubin GPU node.

Summary of Part 2 sections

Here’s a short breakdown of the first hour of the Keynote. For each of the section i give how much time Jensen spent on it along with my impressions and summary notes. I also link directly into each section on the YouTube video.

Intro, Cuda flywheel, Graphics improvements (16min)

Tokens, the Building Blocks of AI · 3:15min

The keynote starts with an inspiring video describing how AI tokens are the main "commodity" produced by AI factories and their power to unlock new knowledge and possibilities

Welcome to GTC 2026 · 2:47min

Jensen enters the stage and gives introductory remarks thanking the pre-game show hosts, and also how the conference will be covering the AI 5 layer cake, a reference to his blog post that divides the stack along: Energy, Chips, Infrastructure, Models, and Applications

20 Years of CUDA · 4:21min

Jensen reviews the flywheel that Cuda software has been enabling for the past 20 years.

GeForce · 3:27min

CUDA made GPUs programmable first on the consumer product GeForce in 2006, which then enabled the deep learning community to test the viability of training neural networks and launched the new AI revolution.

DLSS 5 · 2:29min

Jensen shows a video featuring the new DLSS5 capability, a Neural rendering technology that fuses 3d Graphics with AI to give more beautiful and detailed textures to videos. Video details triggered a backlash from game developers.

Accelerated Analytics (22min)

Structured Data is the Ground Truth of AI · 3:26min

Jensen says Analytics are ripe for acceleration with the arrival of AI agent and emphasizes CuDF and CuVS as foundation libraries powering the whole ecosystem.

IBM Reinvents Data Processing With NVIDIA · 18:10min

He announced partnerships with IBM for Watson-X, a major contributed to open source Presto C++ and user of Spark over Rapids, NVIDIA's own accelerated dataframe libraries. Also announced were partnerships with Dell for an AI platform over RTX6000 servers, and for Google Cloud's AI Hypercomputer. Jensen highlights NVIDIA's stack that accelerate many of the CSP's offerings for AI and he spent some time reviewing them for different cloud providers.

Cuda-X review and AI native companies (7min)

AI Inference Inflection + Overview of datacenter efficiency (Tokens/Watt) vs interactivity (Tokens/s per user) across different tiers (22min)

Inference Inflection Arrives · 4:42min

Jensen highlights 3 key moments for AI inference in the past 2 years: 2023) ChatGPT is released 2024) reasoning AI model with o1 and o3 takeoff and in 2025) Claude code agentic system revolutionizes software engineering.

"The inflection point for inference has arrived." · 1:40min

Agent thinking capabilities led to an explosion in the amount of inference by 10,000x since ChatGPT was released. Coupled with 100x increase in end-user demand, Jensen says we have 1M x more inference demand since 2023. We are now at an inflection point for inference

Inference Inflection Drives Strong Growth · 8:30min

Last year Jensen saw $500B demand for blackwell. This year through 2027, he see $1Tr in infrastructure investments on NVIDIA mainly for inference. 60% of the business is for hyperscalers (some of it for internal use), and 40% is all the rest, such as regional or sovereign cloud, enterprise, supercomputers and all the rest. GB + NVL72 + inference over fp4 for training, dynamo, tensorRT. DGX Cloud.

NVIDIA Extreme Co-Design Revolutionized Token Cost · 3:57min

Datacenters are constrained by a fixed amount of power (Watts) available. Emphasize Tokens Per Watt as the metric to maximize, and interactivity (token/s per User) as a use case differentiator.

InferenceMAX King · 1:23min

Shows how GB300NVL72 has improved on both efficiency and cost for inference and has been recognized by semianalysis as inference King!

NVIDIA is the Global Standard for AI Inference at Scale · 0:33min

Inference service providers should be seen as token factories. The output token rate from companies like eigen AI, together.ai, nebius, etc. has increased very fast, now reaching 400+ token/s for kimiK2.5 reasoning agent. Also see artificial analysis for a breakdown between providers.

AI Factories are the Industrial Infrastructure of the AI Era · 1:10min

Inference drives revenues and Token effectiveness is the most important metric.

The first hour of the keynote established the foundations: CUDA’s flywheel, NVIDIA’s growing role in enterprise analytics, and the massive scale of the inference inflection. Part 3 shifts to the hardware itself where Jensen walks through the full Vera Rubin stack with Groq, then turns to what he called one of the most important open source moments in history.

← Part 1: Overview & Context · Part 3: Vera Rubin Hardware, OpenClaw & Robotics →

I was back this year for the 2026 edition of NVIDIA’s GTC conference held at the San Jose Convention Center and surroundings from March 16-19.

Like last year, there was plenty of energy at the conference with attendee numbers said to have reached more than 30k. The conference was packed with interesting technical sessions on new developments in the NVIDIA ecosystem including technical sessions on CUDA-X libraries and industry and state partners presenting how they have integrated the NVIDIA stack into their products.

The conference expanded to the nearby hotels for additional space, the security check-ins were moved out of the convention center and onto the street and an additional lunch section was added in the parking lot in front of the Hylton Hotel on S. Almaden Road.

Finally the keynote was held like previous years at the SAP Center, a 15min walk away, with a larger pavilion setup just outside of it for free coffee and pastries and for hosting the “pre-game” show featuring executives and technical leaders of companies working with NVIDIA. Other than that, the conference looks about the same as last year!

In this post, I will only cover the keynote and will delve into the sessions I attended and the exhibit hall in followup posts.

The Keynote

The keynote was the main event held on the first day of conference and it was moved ahead to 11AM, making it easier to get there early and avoid long lines. Here are some pictures from the packed SAP Center stadium where it was held

As he does every year, Jensen showed hardware on stage, including the new Vera Rubin tray, the new Groq LPX tray, and the new Co-Packaged Optical switch tray for scaling up. He also showed Vera Ultra and its Kyber rack design where trays are inserted vertically instead of horizontally. The exhibit hall had all these nicely on display.

One interesting aspect I wasn’t expecting was Jensen spending 18 minutes almost at the outset of the keynote talking about how NVIDIA’s libraries are sitting at the foundation of accelerated analytics in Enterprise structured and unstructured data. He announced several partnerships with the cloud providers and highlighted how many of NVIDIA’s solutions accelerate CSP’s offerings. I will cover the analytics aspects of the conference in a separate post.

Jensen reveled in being crowned “inference king” by Semianalysis for GB NVL72 system! Also check their review¹ of the GTC conference.

CUDA is 20 years old

CUDA is now 20 years old, and Jensen celebrated that by spending a few extra minutes talking about its core importance to NVIDIA as a company. He emphasized the crucial flywheel role that CUDA-X plays for NVIDIA as an ecosystem of hundreds of libraries for accelerating all kinds of workloads. As the install base for CUDA has grown, reaching hundreds of millions of GPUs deployed around the world, so has the reach to developers, leading to new breakthroughs in many domains, each creating new markets and new customers who then want to buy more GPUs, further growing the user base.

The Vera Rubin POD is expanding: Seven Chips, Five Rack-scale Systems

One of the major reveals at this year’s conference and worth re-emphasizing is the addition of the Groq LPU to speed up AI inference and the addition of co-packaged optics for the scale network. The NVIDIA AI factory is built around five rack types, and a full Vera Rubin POD “features 40 racks, 1.2 quadrillion transistors, nearly 20,000 NVIDIA dies, 1,152 NVIDIA Rubin GPUs, 60 exaflops, and 10 PB/s total scale-up bandwidth”²

1. The VR NVL72 GPU node
2. The newly announced companion Groq LPU rack offloading part of the AI inference pass (decode)
3. BlueField-4 to store KV cache offloaded from the GPU memory
4. Vera CPU Rack for more general Agentic workloads and RL, and
5. the Spectrum-6 networking rack to connect the whole POD.

With the stage now set (the packed keynote venue, Jensen’s excitement about the expanding CUDA ecosystem, and the broad strokes of the new Vera Rubin POD), Part 2 dives into the actual keynote sections, starting with the CUDA anniversary, accelerated analytics, and Jensen’s case for the AI inference inflection point.

Part 2: Intro, Analytics, CUDA-X & Inference →

References

Scope: This analysis applies to in-memory analytics — workloads whose active dataset fits within the system’s fast memory tier (HBM for GPUs, DRAM for CPUs). Once a workload spills to storage or a slower memory tier, the bandwidth and capacity comparisons change fundamentally: GPU HBM bandwidth advantages disappear when the bottleneck shifts to PCIe, NVMe, or network I/O, and CPU DRAM’s larger capacity becomes a decisive structural advantage. The conclusions here do not generalize to disk-spilling or out-of-core workloads.

The findings are consistent across all three views. GPU memory bandwidth is the most durable advantage — it crossed above parity between the GH200 and GB200 generations and holds steady at 3.5–5.4× at equal spend, whether bare-metal or cloud. The compute and cost advantages that originally drove GPU adoption are compressing: GPU prices are rising faster than per-chip compute gains, and the Perf/W lead is narrowing in parallel. The capacity gap between cheap DDR DRAM and expensive HBM collapses dramatically at equal budget — from 51–91× at compute parity to 8–11× at equal spend.

Table of Contents

1. The CPU Baseline
2. GPU Superchip Specifications
3. Study #1: NVIDIA GPU vs AMD CPU at Compute Parity
4. Study #2: Isocost Analysis - Bare Metal: What Does $1M of GPU Buy vs $1M of CPU?
5. Study #3: Isocost Analysis - Cloud Instance: GPU vs CPU at Equal Hourly Spend on AWS
6. Conclusion
7. Reference Tables

Methodology Caveat: This analysis is intentionally simplified and scoped to in-memory workloads — datasets that fit within the fast memory tier of each system. Real platform evaluation spans many additional dimensions – workload mix, software maturity, interconnect topology, memory tiering behavior, cluster-level networking, availability, and total cost of ownership over time. The comparisons here use a compute-parity model plus explicit assumptions (especially for cost and power) to make directional trends easier to inspect, not to claim a universally optimal chipset choice.

This is a companion post to The Case for GPU-Accelerated Data Analytics.

The CPU Baseline

Server CPUs from Intel and AMD have seen real but incremental progress over the same period. AMD EPYC has been the more aggressive of the two — Turin (2024) tripled memory bandwidth relative to Milan (2021) by upgrading from 8-channel DDR4-3200 (~205 GB/s) to 12-channel DDR5-6000 (~576 GB/s), while also tripling the core count to 192 (up from 64 in Milan).

In the tables below, FP32 numbers are theoretical peak using the widest SIMD available per generation, as a rough proxy for analytics workload compute. Actual sustained throughput varies with workload, instruction mix, and all-core clock. Peak FP32 is calculated without FMA-doubling to reflect typical analytics SQL, which rarely relies on fused multiply-add operations:

Peak FP32 = (Total SIMD units × (SIMD width ÷ 32) × Clock GHz) ÷ 1000

E.g., for Turin: 384 units × (256 / 32) × 2.4 GHz ÷ 1000 ≈ 7.4 TFLOPS. Most analytics operations are comparisons, aggregations, and reductions—not multiply-accumulate patterns. (FMA-doubling would apply to dense linear algebra or ML kernels, not analytics, so it would be misleading here.)

AMD EPYC (per socket)

Intel’s Xeon gains tell a two-part story. Within the Xeon Scalable lineage, progress was incremental: Emerald Rapids (2024) lifted bandwidth only ~1.2× over Sapphire Rapids — from ~307 GB/s to ~358 GB/s (both 8-ch DDR5) — while core count barely moved from 60 to 64 (+7%). The more significant step was Xeon 6 with Granite Rapids (also 2024), a new platform that doubled max cores to 128, pushed bandwidth to ~409 GB/s (8-ch DDR5-6400), and nearly tripled FP32 compute to ~10 TFLOPS. Still well below AMD’s Turin on bandwidth, but a meaningful inflection within Intel’s own trajectory.

Intel Xeon Platinum (per socket)

GPU Superchip Specifications

Looking at NVIDIA’s flagship data-center CPU-GPU superchips across three recent generations, each roughly one to two years apart:

Summary:

• Memory bandwidth has grown roughly 9× across three superchip generations: from 4.9 TB/s on the GH200 to 44 TB/s on the VR200.
• HBM capacity has grown 4× over the same span, from 144 GB to 576 GB.
• The NVLink-C2C architecture further extends this by exposing unified memory that spans both HBM and LPDDR5X — the VR200 makes up to 2.1 TB (576 GB HBM4 + 1.5 TB LPDDR5X) accessible to the GPU.
• That said, HBM remains roughly an order of magnitude more expensive per gigabyte than DDR5, and for workloads that spill beyond the fast HBM tier, performance falls back on the lower LPDDR5X bandwidth.

Study #1: NVIDIA GPU vs AMD CPU at Compute Parity

Raw compute capability is stark, and growing with each GPU generation, but compute alone does not determine analytics outcomes. The table below shows, for each superchip, the best contemporary AMD CPU, how many sockets are needed to match the GPU’s FP32 throughput, and how that compute-equivalent cluster compares on bandwidth, capacity, cost, and power efficiency.

Inter-node (shuffle) bandwidth is not shown here because NVL32/NVL72 are rack-scale products, individual superchips are not sold as standalone IB nodes. Within the rack, all superchip-to-superchip traffic flows over NVLink/NVSwitch at very high bandwidth; InfiniBand only exits at the rack boundary. Inter-node comparisons are covered in Study #2 and Study #3, where rack-level deployment makes the unit of comparison clearer.

Cost assumptions use a platform-normalized method (cost per platform ÷ superchips per platform) with current market prices for full-rack CAPEX:

• GH200 (Hopper) Platform: $1.1M – $1.4M (NVL32 Rack). At 32 superchips, this yields ~$34k – $44k per equivalent.
• GB200 (Blackwell) Platform: $2.9M – $3.4M (NVL72 Rack). At 36 superchips, this yields ~$80k – $95k per equivalent.
• VR200 (Rubin) Platform: $5.5M – $8.0M (NVL72 Rack). At 36 superchips, this yields ~$153k – $222k per equivalent.

Two narratives emerge from the data, one where GPUs are clearly gaining ground, and one where their traditional advantages are quietly eroding.

1) GPUs are closing the gap on Memory Metrics

• On Memory Bandwidth, the inflection point where GPU begins to outrun a compute-equivalent CPU cluster falls between the GH200 and GB200 generations. In the GH200 era, the Genoa cluster is actually ahead (~1.8× CPU advantage). By GB200, the GPU moves ahead (~1.3×). By VR200, the GPU lead widens further (~2.1×).

• On Memory Capacity, the direction is also positive for GPU: DDR-based clusters remain ~51–91× ahead on raw memory footprint, but the ratio is declining each generation. DDR is orders of magnitude cheaper per gigabyte than HBM, so this gap won’t close quickly — but it is shrinking. For workloads that spill beyond the HBM tier, the GPU must fall back to LPDDR5X unified memory or GPUDirect Storage; spill support is therefore a required capability for any GPU database aiming to compete at scale.

2) GPU’s traditional advantages are narrowing (negative for GPU) on cost and energy efficiency

• On Cost at FP32 Parity, the CPU-to-GPU cost ratio trends down from ~15.9× (GH200) to ~6.8× (VR200). A falling cost line means each successive GPU generation requires a larger single-chip capital outlay to deliver the same parity compute, eroding the hardware cost advantage that originally made GPU deployments attractive. These cost figures are the most assumption-sensitive inputs in the post.

• On Power Efficiency, the GPU’s Perf/W lead is also shrinking — from nearly 6.8× over a Genoa cluster to ~3.6× over a Turin cluster. NVIDIA is pushing the thermal limits of silicon — the VR200 superchip draws ~5 kW total, versus ~1 kW for the GH200 — to extract raw performance, while server CPUs have maintained a more conservative power envelope. The absolute efficiency advantage remains meaningful, but the trend is moving in the wrong direction for GPU advocates.

Study #2: Isocost Analysis - Bare Metal: What Does $1M of GPU Buy vs $1M of CPU?

The compute-parity table above asks how many CPUs does it take to match one GPU in raw FLOPS? An isocost analysis flips the question: for a fixed $1M capital budget, how many GPU superchips vs CPU sockets can you buy — and what do you get?

$1M is a meaningful procurement anchor: it buys nearly a full Hopper NVL32 rack worth of GH200s, a partial Blackwell rack of GB200s, or about five Vera Rubin superchips. On the CPU side, $1M buys a meaningful compute cluster — 125 Milan sockets (~62 nodes) or 71 Turin sockets (~36 nodes). This budget is large enough that multi-chip GPU NVLink effects start to matter, and realistic enough to represent a real infrastructure decision.

CPU socket prices are estimated market rates for the highest-core-count SKU at each generation — EPYC 7763 (Milan) at approximately $8k/socket and EPYC 9965 (Turin) at approximately $14k/socket. These are chip-level prices and do not include platform, memory, or networking, consistent with comparing silicon to silicon. GPU costs use the same rack-normalized per-superchip midpoints from the parity section above.

CPU bandwidth: sockets × per-socket bandwidth. CPU DRAM: 64 GB DIMMs, 2 per channel — ~1 TB/socket for Milan (8-ch DDR4), ~1.5 TB/socket for Turin (12-ch DDR5). CPU power adds ~30% platform overhead to socket TDP (Milan 280W, Turin 500W). GPU fleet sizes are partial racks: 25 GH200s ≈ 78% of an NVL32; 11 GB200s ≈ 30% of an NVL72; 5 VR200s ≈ 14% of an NVL72.

Five patterns emerge from the $1M isocost view:

GPU compute advantage is meaningful but compresses as GPU prices rise. A $1M GH200 fleet delivers 5.4× more FP32 than a $1M Milan cluster. By VR200, that lead falls to 2.5× — not because GPU compute scaled down, but because $1M buys far fewer Vera Rubin superchips (5) than GH200s (25). This is a direct effect of GPU price inflation per generation outpacing the compute-per-chip gains.

GPU bandwidth advantage is the most stable metric: 4.3–5.4× across all three generations. Unlike the compute ratio, bandwidth per dollar holds remarkably steady. Even when buying fewer chips, each VR200 contributes 44 TB/s, which keeps the fleet aggregate well ahead of the CPU cluster. This is the GPU’s most durable advantage at equal budget: memory bandwidth per dollar has not eroded the way compute per dollar has.

The CPU capacity advantage is real but much smaller than the parity view suggests — and worsens for GPU in the GB200 generation. At parity, CPU clusters hold 51–91× more DRAM. At $1M, that collapses to 8–11×. However, the ratio worsens for GPU going from GH200 to GB200: $1M buys many more GH200s (15.6 TB total HBM) than GB200s (9.5 TB), because GB200s are ~2.2× more expensive per chip with only a proportional HBM-per-dollar increase. The VR200 partially recovers (10.5 TB) thanks to its larger per-chip HBM. As rack-scale NVLink pooling becomes the default deployment model, the effective addressable GPU memory pool expands beyond what these single-fleet numbers reflect.

GPU Perf/W advantage is large and consistent. The ~9.7× lead at the GH200 generation narrows to ~4.6× by VR200 — consistent with the parity trend — and reflects that GPU silicon extracts substantially more analytics-relevant FP32 output per watt than CPU silicon at this budget. Notably, both the GPU and CPU fleets draw comparable absolute power at $1M (~25–30 kW GPU vs ~45–46 kW CPU — a factor of ~1.5–1.8×), so the Perf/W ratio is primarily a statement about performance density, not a dramatic difference in total energy draw.

Inter-node (shuffle) bandwidth: per-node advantage recovers at VR200, but the CPU fleet still wins total aggregate egress at $1M. Bare-metal GPU and CPU clusters connect over InfiniBand — HDR 200 Gbps (25 GB/s) for Milan era, NDR 400 Gbps (50 GB/s) for Genoa, Turin, and GB200. The GH200 generation carries a 2× per-node advantage over a Milan cluster (NDR vs HDR). By GB200, both GPU and CPU nodes sit on NDR 400 Gbps — 1× per-node parity. The VR200, estimated to ship with ConnectX-9 (XDR), breaks this parity at 1,600 Gbps (200 GB/s) — a ~4× per-node advantage over Turin’s 400 Gbps (50 GB/s) NDR. At equal $1M spend, the CPU fleet’s larger node count still dominates total aggregate shuffle egress: 36 Turin nodes × 400 Gbps = 14.4 Tbps vs 11 GB200 nodes × 400 Gbps = 4.4 Tbps — a ~3.3× CPU aggregate advantage in the GB200 era. For VR200, the per-node XDR lead narrows the aggregate gap significantly: 5 VR200 nodes × 1,600 Gbps = 8 Tbps vs 36 Turin nodes × 400 Gbps = 14.4 Tbps — ~1.8× CPU aggregate advantage. For workloads dominated by cross-node data movement (large hash joins, high-cardinality group-by across partitions), the CPU fleet at bare-metal $1M scale retains the aggregate shuffle throughput edge, though VR200 narrows the gap significantly.

Study #3: Isocost Analysis - Cloud Instance: GPU vs CPU at Equal Hourly Spend on AWS

The capital budget analysis above captures bare-metal procurement economics. Cloud deployments shift this to an operational model — pay by the hour, no upfront commitment, scale up or down. This section uses AWS on-demand Linux pricing from Vantage (April 2026) to ask the same isocost question with hourly rates.

Cloud pricing caveat: On-demand AWS rates are the most widely published and comparable benchmark, but they are not the cheapest option. GPU-specialized clouds — CoreWeave, Lambda Labs, Crusoe, and others — typically offer H100 capacity at $2.49–2.89/hr per GPU (~$20–23/hr for an 8-GPU node), roughly 60% below the AWS p5.48xlarge rate of $55.04/hr. GCP and Azure on-demand rates for equivalent instances are broadly similar to AWS. All three advantage ratios in this section (FP32, bandwidth, capacity) are sensitive to pricing: a cheaper GPU cloud means more GPU instances per $1k/hr, which shifts all ratios in the GPU’s favor. The AWS numbers here should be read as a specific pricing scenario, not a hardware-fundamental result.

A key property of cloud GPU instances: the instance price already includes the host CPU. The p4d.24xlarge bundles 8× A100 GPUs with an Intel Xeon Platinum host; the p5.48xlarge bundles 8× H100 GPUs with an AMD EPYC host; the p6-b200.48xlarge bundles 8× B200 GPUs with an Intel Xeon Emerald Rapids host. This is equivalent to the superchip pricing model — you pay for the full compute node, GPU and CPU together. AWS also offers p6e-gb200.36xlarge — 36 native Grace-Blackwell superchips — but on-demand pricing is not yet published for that instance.

At $1,000/hour on-demand:

FP32 estimates follow the same method as the parity section: SIMD units × (SIMD width ÷ 32) × all-core GHz. GPU FP32 uses ~19.5 TFLOPS per A100 SXM4 (NVIDIA published non-sparse FP32 peak), ~67 TFLOPS per H100, and ~75 TFLOPS per B200 (the GB200 superchip = 1 Grace CPU + 2× B200 GPUs = 150 TFLOPS total, so 75 TFLOPS per B200). CPU FP32 uses ~2.3 TFLOPS per 48-core Milan socket (EPYC 7R13: 96 SIMD units × 8 × 2.95 GHz / 1000), ~3.7 TFLOPS per 96-core Genoa socket, and ~3.7–4.0 TFLOPS per 96-core Turin socket at sustained all-core clock. CPU memory bandwidth uses per-socket figures (205 GB/s Milan, 461 GB/s Genoa, 576 GB/s Turin) multiplied by 2 sockets per instance (hpc6a/hpc7a/hpc8a are all 2-socket nodes), giving 410 GB/s, 922 GB/s, and 1,152 GB/s per instance respectively. GPU bandwidth specs are from NVIDIA datasheets; CPU bandwidth specs are from AMD datasheets. Prices are from Vantage.

Four observations from the cloud view:

The compute advantage is large but generation-sensitive. At equal hourly spend, 18 H100 instances outcompute 138 Genoa nodes by ~9.5×. By the Blackwell generation, that lead narrows to ~5.2×: the B200 is more powerful per GPU, but $1,000/hr buys only 8 p6-b200 instances versus 126 hpc8a nodes — because the B200 instance price (2.1× the H100 instance price) has risen faster than the per-GPU compute improvement (~1.1×).

GPU bandwidth advantage is stable across all three cloud generations (~3.5–3.9×). From A100/Milan (3.9×) to H100/Genoa (3.8×) to B200/Turin (3.5×), total memory bandwidth per dollar holds remarkably steady even as the compute ratio swings from 4.5× to 9.5× and back to 5.2×. Bandwidth is the GPU’s most durable and predictable cloud advantage.

The cloud capacity gap (8–9× on AWS) is comparable to the bare-metal view (8–11×). This convergence reflects a similar GPU-to-CPU price ratio across both procurement models — on AWS, GPU instances cost roughly 7–14× more per node than CPU HPC instances, broadly in the same range as the bare-metal silicon cost ratios. The specific numbers will shift on cheaper clouds: at CoreWeave H100 pricing (~$20/hr per node), the same $1k/hr buys ~50 GPU nodes instead of 18, compressing the CPU capacity advantage to roughly 3×.

On AWS, GPU instances carry a decisive per-node inter-node bandwidth advantage that extends the GPU lead from intra-node memory to inter-node shuffle. The H100 (p5.48xlarge) and B200 (p6-b200.48xlarge) instances both carry 3,200 Gbps (400 GB/s) EFA — roughly 11× more per-node inter-node bandwidth than the 300 Gbps (37.5 GB/s) EFA on hpc7a (Genoa) and hpc8a (Turin). For analytics workloads with significant data movement between nodes (hash joins, group-by on high-cardinality keys), the GPU fleet’s per-node network bandwidth means it is also faster at the shuffle phase, unlike the bare-metal case where both GPU and CPU nodes share NDR InfiniBand. The A100 era is more modest: p4d.24xlarge has 400 Gbps (50 GB/s) EFA vs hpc6a’s 100 Gbps (12.5 GB/s) — a 4× per-node advantage. At $1k/hr, the H100 fleet’s total shuffle capacity (18 × 3,200 Gbps = 57.6 Tbps) also exceeds the Genoa fleet (138 × 300 Gbps = 41.4 Tbps); for the A100 and B200 eras the CPU fleet regains a total aggregate lead through node count, but the GPU maintains a 4–11× per-node inter-node advantage throughout all three cloud generations.

Conclusion

Three lenses — compute parity, $1M bare-metal capital, and equal hourly AWS spend — tell a consistent story with two competing narratives.

GPUs are gaining decisively on bandwidth. Bandwidth per chip crossed above the Genoa cluster between the GH200 and GB200 generations, and that lead has widened steadily. Bandwidth per dollar is the most stable metric across all isocost views: 4.3–5.4× on bare metal across three GPU generations, and 3.5–3.9× on AWS across three cloud generations (A100 through B200). Of every metric tracked in this post, memory bandwidth per dollar has eroded the least — and for analytics workloads, that matters most.

The compute and cost advantages that once justified GPU deployments are compressing. The FP32 advantage at equal spend falls from ~5.4× (GH200 vs Milan, $1M bare metal) to ~2.5× (VR200 vs Turin) — not because GPU compute stagnated, but because GPU prices are rising faster than per-chip compute gains. The cloud view shows a non-monotonic pattern: the A100/Milan era started at ~4.5×, the H100/Genoa generation peaked at ~9.5× (H100 offered exceptional value at launch — ~3.4× more compute than A100 without proportional pricing), and the B200/Turin shows compression to ~5.2× as instance prices have overtaken compute gains. The Perf/W lead is narrowing for the same reasons.

The capacity gap is structural but not fixed by price. At compute parity, CPU DRAM clusters hold 51–91× more memory than GPU HBM. At equal budget — whether $1M capital or $1k/hr cloud — that collapses to 8–11×. The gap is real, but much of it is a pricing artifact: DDR capacity is cheap per gigabyte; HBM is not. For workloads that genuinely need TBs of fast-path memory, CPUs hold a durable structural advantage. For most analytics workloads that fit in HBM, the gap is largely academic.

We will revisit this comparison when AMD EPYC Venice (Zen 6) specs are finalized, to measure how much the CPU side shifts the balance versus NVIDIA Vera Rubin.

Reference Tables

Cloud Instance Pricing — AWS on-demand Linux rates from Vantage, April 2026. GPU instances include the host CPU. Prices may vary by region.

Why these instances?

• GPU side — maximum GPU density per generation:
  • p4d.24xlarge (8× A100), p5.48xlarge (8× H100), p6-b200.48xlarge (8× B200) are all the flagship 8-GPU nodes AWS offers per generation.
  • Using maximum GPU density per node minimizes fixed CPU/networking overhead per GPU and is the standard choice for GPU-heavy workloads.
• CPU side — HPC-optimized hpc* instances specifically:
  • The hpc6a/7a/8a family is chosen over general compute instances (like c6a, c7a) because they match the AMD EPYC generations used in the bare-metal section (Milan → Genoa → Turin), and they include high-bandwidth EFA networking — making the networking comparison fair vs the GPU instances.
• Generation alignment: Each pair is matched within the same GPU-era: A100 (2020) paired with Milan (2021), H100 (2022) with Genoa (2022), B200 (2024) with Turin (2024). This avoids comparing late-generation CPUs to early-generation GPUs or vice v

Memory Capacity and Bandwitdh - Inside a single chip

Interconnect Technologies — Intra-node bandwidth connects CPUs and GPUs within a single node (NVLink, PCIe, NUMA fabric). Inter-node bandwidth is the network fabric used for data exchange between nodes — the shuffle phase in distributed analytics.

  ①  Inter-chip BW          ②  Intra-node BW              ③  Inter-node BW

  GPU superchip (e.g. GB200 NVL72):
    ┌─── SoC ──────────┐
    │[CPU]──①──[GPU]   │──────────②──────────[GPU]  ···
    └──────────────────┘    NVLink / NVSwitch      │
         NVLink C2C                                └────③────[Node B]  ···
                                                        IB / EFA

  Cloud GPU node (e.g. p5.48xlarge):
    [Host CPU]──①──[GPU 0]──────②──────[GPU 1]  ···  [GPU 7]
                PCIe           NVLink              │
                                                   └────③────[Node B]  ···
                                                           EFA

  CPU cluster (e.g. hpc7a):
    (① n/a)   [Socket 0]──────②──────[Socket 1]
                          Inf. Fabric          │
                                               └────③────[Node B]  ···
                                                     IB / EFA

AWS EFA speeds from Vantage, April 2026. InfiniBand speeds reflect standard configurations: HDR = 200 Gbps / 25 GB/s (2021 era), NDR = 400 Gbps / 50 GB/s (2022+). The jump from p4d (400 Gbps / 50 GB/s EFA) to p5 (3,200 Gbps / 400 GB/s EFA) reflects AWS’s EFA3 fabric generation deployed for the Hopper generation. NVIDIA NVLink and GPU intra-node specs from NVIDIA datasheets.

IB (InfiniBand) is an open industry-standard network fabric — completely independent of CPU vendor. NVIDIA owns Mellanox (acquired 2020), the dominant InfiniBand hardware maker, but the HCAs (host channel adapters) plug into any server via PCIe regardless of whether it runs AMD or Intel CPUs. Standard generations used in this post: HDR = 200 Gbps / 25 GB/s (circa 2021), NDR = 400 Gbps / 50 GB/s (2022+), XDR = 1,600 Gbps (200 GB/s)/port ConnectX-9 (2024+, est.).

EFA (Elastic Fabric Adapter) is AWS’s proprietary high-performance network interface for HPC and ML workloads. Unlike standard cloud networking, EFA uses OS-bypass RDMA (via AWS’s SRD — Scalable Reliable Datagram — protocol), skipping the kernel network stack to deliver lower latency and higher throughput for tightly-coupled distributed workloads. EFA is not InfiniBand, but achieves similar application-level semantics. Generations: EFA2 (~400 Gbps / ~50 GB/s, p4d) → EFA3 (~3,200 Gbps / ~400 GB/s, p5/p6-b200).

References

The Analytics Workload Is Changing Fast — Enter AI Agents

In 2025, LLM-powered AI agents started proving their value, and their adoption has been rapidly spreading across enterprises, particularly for data analytics and insights extraction. The 2025 State of AI in Enterprise report shows that companies are now moving from piloting the technology to actually deploying it in production, noting that “many companies focused on experimenting last year [2025] have crossed the threshold into operational AI systems.”

Databricks is at the forefront of adopting LLMs and agent technology, and it is worthwhile to follow how they have been preparing for the coming explosion in their adoption. Through their latest Lakebase architecture, Databricks shows they are positioning for both OLTP and OLAP workloads required by agents for the full automation of the data exploration and productionization pipelines.

This architecture eliminates much of the cost, complexity, and lock-in that have defined databases for decades, and it is especially powerful for modern AI and agent-driven workloads, where developers want to launch many instances, experiment freely, and pay only for what they use.

Their latest Genie product is their version of the AI agents that will carry out this work, driven solely by high-level natural language commands tied to business needs.

Genie Code can autonomously carry out complex tasks such as building pipelines, debugging failures, shipping dashboards, and maintaining production systems.

Together, these advances will help bring AI agents to the market, simplifying much of the data science workflow. But Databricks believes a much bigger wave is ahead, one where agents are unleashed to search for insights by trying many different paths. They call this Agentic speculation, “a high-throughput process of exploration and solution formulation for the given task,” which Databricks engineers envision will require redesigning data systems to be agent-first¹.

Overall, as agentic workloads become more and more prevalent, the sheer scale and inefficiencies of agentic speculation will become the bottleneck, and our data systems will need to evolve in response

The impact on the analytics workload will be profound. Future systems will be designed almost exclusively with AI agents as first-class users, performing exploration, identifying insights, and productionizing their solutions. All of this will be done from raw structured and unstructured data.

Where will the engineering bottlenecks be? Agentic speculation will dramatically increase the velocity of both code generation and analytical queries, vastly increasing the effective memory bandwidth and working set memory requirements of the underlying systems. AI agents will also become more deeply integrated into the data infrastructure to provide intelligent exploration. Do we have the right software and hardware to support this movement? Today, we see massive investment in serving inference from GPUs, but not enough analytics workloads have been accelerated, and this is likely to become a major bottleneck.

The question I am posing is whether current CPU-centered data processing systems will be capable of handling the scale needed to support these new agentic workloads.

CPU-Centered Analytics: Decades of Dominance

In the past few decades, analytic query engines have been very successfully built around the CPU architecture, featuring a growing number of high-performance server cores (in the hundreds), deep cache hierarchies (in tens of MBs), and vectorized operations taking advantage of wider SIMD instruction sets (up to 512 bits wide).

To meet ever-larger volumes of data stored in object stores, these engines moved towards disaggregated architectures that enable elastic scaling of compute and storage. Coupled with open columnar data formats like Parquet and Arrow, this shift has fostered a wide ecosystem of query engines built on a composable data philosophy. It has been a remarkable run.

The milestones speak for themselves. Snowflake’s 2016 architecture pioneered separating compute from storage entirely, proving that cloud-native disaggregation could deliver elastic, multi-tenant analytics at scale. On the single-node analytical engine front, DuckDB brought embeddable, vectorized OLAP to the edge; ClickHouse pushed columnar execution to extreme throughput on commodity hardware; and Umbra/CedarDB pushed the boundary on single-node performance with JIT query compilation via LLVM and a hybrid row/columnar storage engine capable of handling both transactional and analytical workloads on a single system.

The composable data systems movement² has further decoupled execution from storage, built on two key standards: Apache Arrow as the universal in-memory columnar format enabling zero-copy data exchange between engines, and Substrait as a portable, cross-language query plan representation that lets a plan produced by one system be executed by another. On top of these, Velox (Meta) and Apache DataFusion provide reusable, modular physical execution engines that plug into larger systems rather than reinventing the wheel. This composability is now flowing upstream into the dominant distributed compute platforms — Gluten brings Velox-backed native execution into Apache Spark, Apache DataFusion Comet does the same using DataFusion as the native Rust backend, and Presto has adopted Velox as its native C++ evaluation engine — extending the CPU performance frontier by replacing JVM-based execution with optimized native kernels.

CPU vs GPU Hardware Trajectories: The In-Memory Gap Is Shifting in GPU’s Favor

Yet even as software pushes the CPU performance frontier further, the underlying hardware is hitting diminishing returns. AMD’s EPYC Turin, today’s server CPU bandwidth leader, peaks at ~576 GB/s per socket (+25% vs Genoa’s ~461 GB/s) and ~15 TFLOPS FP32 (+~40% vs Genoa’s ~11 TFLOPS), with max DRAM capacity flat at 6 TB across both generations. Intel’s Xeon 6 (Granite Rapids) reaches ~409 GB/s (+33% vs Sapphire Rapids’ ~307 GB/s) and ~10 TFLOPS FP32 (~2× vs Sapphire Rapids’ ~4.8 TFLOPS), with capacity likewise flat at 4 TB. Meaningful gains but incremental, and capacity has effectively plateaued.

GPUs tell a different story. Driven by the insatiable demand for AI training and inference, NVIDIA’s flagship data-center superchips have advanced at a fundamentally different pace across just three generationsm the GH200 (Grace Hopper, 2023), GB200 (Grace Blackwell, 2024), and VR200 (Vera Rubin, 2025): memory bandwidth grew 9x from 4.9 TB/s to 44 TB/s, FP32 compute grew from 67 to 260 TFLOPS, and total unified memory capacity grew 3.4x generation-over-generation: 624 GB -> 864 GB (+39%) -> 2.1 TB (+143%). That puts VR200 at about 76x the bandwidth of a single EPYC Turin socket.

But for in-memory analytics, workloads whose active dataset fits within the fast memory tier (HBM for GPUs, DRAM for CPUs), raw hardware scaling alone does not determine the winner. A three-generation study across three lenses (compute parity, $1M bare-metal budget, equal AWS hourly spend) reveals two distinct trends pulling in opposite directions:

• GPU memory bandwidth is the most durable advantage and is not eroding. It crossed above parity against a compute-equivalent CPU cluster between the GH200 and GB200 generations, and holds steady at 3.5–5.4× at equal spend across all three generations. The inflection is generational and structural.
• GPU compute, cost, and Perf/W advantages are real but compressing. At equal spend, the FP32 advantage peaked at the H100 generation (~9.5× over Genoa on AWS) and has since fallen to ~5.2× for B200 — not because GPU compute plateaued, but because GPU prices are rising faster than per-chip compute gains. The Perf/W lead is narrowing for the same reason.
• The capacity gap is large but driven by price, not physics. CPU DRAM holds 51–91× more memory at compute parity, but that collapses to 8–11× at equal spend. The difference is that DDR costs a fraction of HBM per gigabyte — once you normalize by budget, you are buying far more DDR capacity than the raw chip-count comparison suggests. If HBM prices fall relative to DDR over time, a trend already underway, this ratio will compress further in the GPU’s favor.

Assumptions: The directional conclusions above rest on specific cost and pricing inputs — rack-normalized GPU superchip prices ($39k–$188k per chip), AMD EPYC socket prices (~$8k–$14k), and AWS on-demand rates from April 2026. Cost figures are the most assumption-sensitive part of the analysis: GPU list prices vary by channel and contract, and cloud rates change frequently. The bandwidth and compute trends are hardware-spec-driven and more stable; the capacity and cost conclusions are pricing-driven and should be read as directional, not precise.

For a more detailed generation-by-generation comparison, see: GPU vs CPU for In-Memory Analytics: Bandwidth Holds as Compute and Cost Advantages Narrow Across Three Generations.

Coming next: The analysis above is scoped strictly to in-memory workloads. A follow-up post will delve into the big data case where datasets exceed GPU HBM capacity, and where the HBM bandwidth advantage disappears at the PCIe or NVMe bottleneck, and CPU DRAM’s structural capacity advantage becomes decisive for analytics at scale.

GPU-Accelerated Databases Are Rising in Research and Industry

Unsurprisingly, the database research community has been paying close attention since 2020, with top conferences like SIGMOD and VLDB regularly accepting papers evaluating and building GPU-accelerated databases — both hybrid CPU-GPU and fully GPU-native. Recent highlights include:

• Rethinking Analytical Processing in the GPU Era³ (CIDR 2026) — Sirius, a GPU plugin for DuckDB that rethinks analytical processing natively on the GPU.
• Scaling GPU-Accelerated Databases beyond GPU Memory Size⁴ (VLDB 2025) — tackles the fundamental GPU memory capacity bottleneck with a hybrid CPU-GPU filtering strategy, achieving a 3.5× speedup over SQL Server at 1 TB scale on a single A100.
• GPU Database Systems Characterization and Optimization⁵ (VLDB 2024) — systematically characterizes GPU database performance bottlenecks and proposes optimizations for modern workloads.
• A Study of the Fundamental Performance Characteristics of GPUs and CPUs for Database Analytics⁶ (SIGMOD 2020) — proposes Crystal, a GPU query library, and shows that full query GPU speedup can exceed the memory bandwidth ratio (up to 25×) due to CPU vectorization limitations.

On the industry side, 2025 saw GPU acceleration reach mainstream data tools:

• GPU execution landed in CPU dataframe engines like Velox⁷ and Polars⁸.
• The RAPIDS Accelerator for Apache Spark⁹ enabled faster migration to GPU-accelerated distributed data engineering and analytics workloads.
• Voltron published the design paper for Theseus¹⁰, their petabyte-scale GPU accelerated query engine.

Despite genuine progress, building correct and performant GPU implementations of the full relational algebra remains enormously difficult. Managing GPU memory limits, PCIe transfer bottlenecks, operator fusion, and full SQL coverage is a hard engineering problem with no easy shortcut.

NVIDIA’s Moat: RAPIDS and libcudf

NVIDIA has seen this challenge coming for a while and has been systematically building a solution through its RAPIDS AI¹¹ ecosystem, first launched in 2018¹², well before the generative AI and LLM revolution had taken hold. At its core is a little-known C++ library, libcudf (and its sister libraries), a highly optimized, native GPU foundation that underpins virtually all GPU-accelerated analytic systems being built today.

It is the de facto single-node physical operator infrastructure in this space, and understanding it is the key to understanding how GPU databases actually work. And yet, despite its central role, in-depth technical coverage of libcudf’s internals is surprisingly scarce. Most available material stays at the user-facing API level, leaving critical questions about kernel design, memory management, and performance characteristics largely undocumented outside of the source code itself.

In future posts, I’ll thus be diving deeper into the technical internals of libcudf and answering questions such as:

• ❓ How does libcudf translate relational operators into parallel GPU kernels?
• ❓ What is the tooling like to evaluate the library’s performance?
• ❓ How is the libcudf used as a building block for larger distributed systems?

We are at an inflection point. The hardware gap between CPUs and GPUs is no longer a niche concern for ML engineers — it is becoming structurally relevant for anyone building or operating data systems at scale. For in-memory analytics, the shift is already underway: GPU bandwidth has crossed into a durable lead and the remaining gaps in cost and capacity are narrowing, not widening. The harder question — how this plays out when datasets exceed HBM capacity and the bottleneck shifts to PCIe or storage — is the subject of a future post. The research momentum, the industry adoption, and NVIDIA’s deliberate infrastructure investment all point in the same direction: GPU-accelerated analytics is moving from experimental to essential. The open question is not whether it will happen, but how fast the ecosystem matures and how much of the existing CPU-centric stack it displaces versus complements.

Excited about the momentum of GPU-accelerated analytics? Have questions about the software or hardware stack? Let me know below! 👇

References