Why an AI chip sits idle, and what changes when it stops waiting on memory
A team of four students built a simulator for MobileNetV2, a network used for real-time camera perception in cars. In their baseline the chip spent 87% of its cycles waiting on memory. Keeping layer outputs on-chip made inference 2.1x faster at 2 MB of buffer, in simulation.
Cars run camera networks on embedded chips under strict time and energy limits, and on those chips data movement, not arithmetic, sets the pace. This team built a trace-driven simulator that holds the hardware fixed and changes only the on-chip buffer policy, then compared four policies on MobileNetV2 across buffer sizes from 64 KB to 8 MB. Keeping activations on-chip cut simulated cycles 2.1x and memory traffic 2.7x at 2 MB, rising to 3.7x and 7x at 8 MB. Their own policy, which adds cautious prefetching of upcoming weights, helped by up to 12.3% in specific layer windows but slowed the network by 3.5% at 2 MB. In one 512 KB case, cycle-accurate DRAM tools confirmed a 1.52% DRAM-cycle and 0.88% DRAM-energy reduction. All results are from simulation.
Keywords · edge AI accelerators · memory scheduling · prefetching · MobileNetV2 · trace-driven simulation · DRAM energy
Who did this work
- Concentration focus
- AI + Physical Intelligence
- Concentration focus
- AI + Physical Intelligence
- Concentration focus
- AI + Physical Intelligence
- Concentration focus
- AI + Physical Intelligence · AI + Health Tech
Students are not named. Year of study is as of fall 2026. Direction and goal come from each student's own program plan.
1Why memory, not math, slows the chip
A car's cameras feed networks that detect objects and lanes on every frame, on a chip inside the vehicle. A missed deadline is a safety problem, and the power budget competes with range and heat [1].
MobileNetV2 was designed to need little arithmetic. But every layer still moves activations and weights between off-chip DRAM and a small on-chip buffer. If that buffer is managed poorly, the chip spends most of its time waiting. The team asked one question: with the hardware held fixed, how much does the buffer policy alone change speed and memory traffic?
2Hold the hardware fixed, change one rule
- Built the simulator. A unified on-chip buffer, a 256 MAC-per-cycle compute array and a bandwidth-limited DRAM link, with sizes taken from configurations explored in SCALE-Sim [2].
- Fed it the real network. Layer-by-layer traces of all 141 MobileNetV2 layers [3].
- Compared four policies. Streaming everything from DRAM, pinning weights, keeping recently used tensors (LRU), and the team's priority policy, which adds cautious prefetching of the next layer's weights while reserving room for activations.
- Zoomed into three-layer windows. Swept buffer sizes from 64 KB to 8 MB to find the layer patterns where prefetching helps.
- Checked one case at DRAM level. Ran a 512 KB case through Ramulator and DRAMPower to confirm the traffic and energy change outside their own simulator.
3Keeping activations on-chip mattered most; prefetching helped only with room to spare
| Policy | Buffer | Cycles vs. baseline | DRAM traffic vs. baseline |
|---|---|---|---|
| Stream from DRAM (baseline) | Any | 10.64M cycles, 87.1% waiting | 162.2 MB |
| Pin weights | Any | About the same | About the same |
| Keep recent tensors (LRU) | 2 MB | 2.1x fewer | 2.7x less |
| Keep recent tensors (LRU) | 8 MB | 3.7x fewer, 9.8% waiting | 7x less |
| Priority (team's policy) | 64 KB–1 MB | 0.1–0.5% better than LRU | About the same as LRU |
| Priority (team's policy) | 2 MB | 3.5% worse than LRU | — |
In one 512 KB case the policy also cut DRAM reads by 2.23%. Cycle-accurate DRAM simulation (Ramulator) and an energy model (DRAMPower) confirmed a 1.52% drop in DRAM cycles and a 0.88% drop in DRAM energy for those windows.
4The paper and selected slides
Pages from the team's final paper and presentation. Student names are blurred.
5What this does not show
- All results come from simulation. Cycles are internally consistent units, not measured times on a real chip.
- The simulator moves whole tensors and does not model tiling, separate on-chip memories or fused layers.
- The DRAM energy check covers selected 512 KB windows, not the full network or the whole system.
- MobileNetV2's shortcut connections are approximated as a straight chain of layers.
6What this work shows the team can do
Researcher in AI Hardware and Embedded Systems
Did research at MIT on integrated photonics and AI-driven health sensing. 14+ journal papers and 5 patents. Builds wearable AI devices in industry.
The team designed the simulator, ran the experiments and wrote the paper. The Project Lead met with the team weekly and reviewed the research direction.
Computer architecture · ML systems · Embedded AI
A research paper with a clear question and an honest negative result works well as a writing sample.
7What the Project Lead observed
After the project, the Project Lead wrote an evaluation of each student and placed each one on a four-level scale. This section shows the highest level reached on the team. The notes below describe the team's project. Individual ratings are not published.
System-Level Insight
- 1Foundational Application
- 2Applied Reasoning
- 3Independent Problem Solving
- 4System-Level Insight
Level 4 of 4: anticipates broader implications, risks or downstream effects, and connects technical decisions to real-world impact.
- Framing the problem
- Grounded the research in a hardware bottleneck: lightweight convolutions cut computation, but not the latency and energy of moving activations and weights between DRAM and local memory.
- Decisions and trade-offs
- Refined the policy to keep LRU-style activation reuse while adding only conservative weight prefetching, and identified tiling as the natural next step for the accelerator design.
- Evidence and testing
- Organized a trace-driven method into one research story: comparisons with LRU, three-layer window analysis, and focused buffer cases at 192 KB and 512 KB.
- Limits and risk
- Connected scheduling choices to DRAM traffic, latency, energy and limited buffer capacity.
Where the Project Lead pushed further. State the validation boundary plainly: which findings come from the team's Python simulator and which were checked with Ramulator and DRAMPower.
Source: the Project Lead's written evaluations, June 2026.
References
- Lin, S.-C. et al. (2018). The architectural implications of autonomous driving: Constraints and acceleration. ASPLOS, 751–766.
- Raj, R. et al. (2025). SCALE-Sim v3: A modular cycle-accurate systolic accelerator simulator for end-to-end system analysis. ISPASS, 186–200.
- Sandler, M. et al. (2018). MobileNetV2: Inverted residuals and linear bottlenecks. CVPR.
Join the AI in Hardware program
This outcome came out of this project. Work with an AI hardware researcher on a real performance problem, and leave with research you can show.
