_{CONTRIBUTES TO  ·  FRONTIER LLM TRAINING & INFERENCE}

         

Open to research internships, PhD positions, and ML-systems / research-engineering roles in efficient training, sparse MoE, and world models.

   

_{Four that show the range: a subsystem designed, a parallelism strategy unblocked, layouts that used to be unconvertible, and a silent corruption caught.}

deepspeedai/DeepSpeed Introduced an affine IR for tensor-parallel shard geometry — executable offsets and strides with a specification document. Landed deliberately inert: its tests had to reproduce the existing conversion arithmetic before anything depended on it.
NVIDIA/Megatron-LM Unblocked HSDP in experimental Megatron-FSDP — removed the placement guard that rejected Partial gradients outright, added the Replicate→Partial transition, and gated DP-outer reduction on the last microbatch so gradients accumulate instead of reducing every backward.
deepspeedai/DeepSpeed Made three refused layouts convertible — GPTBigCode and Yuan shared-QK now describe themselves through segment lists and block selections, geometry that a single partition dimension cannot express.
vllm-project/vllm Four lines, one silent corruption. Disables cross-layer KV blocks under per-token-head quantization, where OffloadingConnector was corrupting the KV cache.

_{All twenty-one, newest first within each organisation.}

 _{9 merged}

NVIDIA/Megatron-LM Supports HFSDP deferred DP-outer gradient reduction in experimental Megatron-FSDP
NVIDIA/Megatron-LM Supports HSDP deferred DP-outer gradient reduction in experimental Megatron-FSDP
NVIDIA-NeMo/Automodel Collapses every guard into one per-class TieSupport (BOTH / TIED_ONLY / UNTIED_ONLY) + a from_pretrained flip check
NVIDIA-NeMo/Automodel Completes the guards for the remaining #2512 model families
NVIDIA-NeMo/Megatron-Bridge Makes the finetuning batch sampler epoch-aware, so checkpoint resume stops replaying the wrong data
NVIDIA-NeMo/Automodel Rejects tie_word_embeddings=True on separate-head families instead of silently mis-loading
NVIDIA-NeMo/Automodel Resolves tie_word_embeddings top-level-first, matching HF tying semantics
NVIDIA-NeMo/Automodel Re-ties lm_head to the active embed_tokens on the Gemma4 MoE path
NVIDIA-NeMo/Automodel Cherry-picks #2601 into r0.5.0

 _{4 merged}

vllm-project/vllm Disables cross-layer KV blocks under per-token-head quant — stops OffloadingConnector corrupting the KV cache
vllm-project/vllm Recovers the raw tail when the GPT-OSS Harmony parser ends non-terminal (Responses API)
vllm-project/vllm Returns raw output instead of dropping it when Harmony parsing ends in a non-terminal state
vllm-project/vllm Fixes the nightly Docker ImportError: AnthropicOutputConfig

 _{8 merged}

deepspeedai/DeepSpeed Keeps Yuan's o_proj bias whole on every rank — fixes a crash present since June 2024 and removes the last parameter universal-checkpoint conversion refused unconditionally
deepspeedai/DeepSpeed Describes the CodeGen fused-QKV layout as a block selection — the last layout whose geometry had no description, so conversion stops refusing it
deepspeedai/DeepSpeed Restore now takes its shard from the map the layer published — the same piece list conversion uses, run in the opposite direction, so the two sides can no longer describe one layout in two ways
deepspeedai/DeepSpeed Describes the GPTBigCode and Yuan shared-QK layouts so universal-checkpoint conversion stops refusing them — segment lists and block selections for layouts a single partition dimension cannot express
deepspeedai/DeepSpeed Emits affine maps from AutoTP layers, so conversion reads the map on the path a real job takes — covers contiguous splits, replicated parameters and uneven vocabulary heads across TP degrees
deepspeedai/DeepSpeed Carries the affine scale on the replicated map rather than the split — no in-tree layout scales a split, so the split constructors drop the argument
deepspeedai/DeepSpeed Introduces an affine IR for tensor-parallel shard geometry — executable offsets and strides with a specification document, landed inert so its tests had to reproduce the existing conversion arithmetic first
deepspeedai/DeepSpeed Avoids CUDA context init during import-time op checks — makes import deepspeed fork-safe

Learned Structure Shows Up Earlier in the Task's Own Basis: A Synchronisation Precursor in Grokking  ·  sole author
_{Holds activations and the metric fixed and varies only the readout basis: structure appears first in the task's own discrete-log basis in 60/60 transformers on modular multiplication across three primes. Extends to six finite groups via irreps. v3 corrects v1's causal claims and a metric artefact, and reports the negatives — memorisation precedes grokking in 179/179 runs, and synchronisation predicts it only weakly (ρ = 0.18).}

Does Depth Survive Composition, and Can a Router Use It? Adaptive Compute in Latent World Models
_{Pre-registered fixed-depth study — a 12-block early-exit predictor across nine DeepMind Control tasks × 8 seeds. Depth improved 10-step rollouts on six tasks, but on three a shallower exit beat the full stack on every seed. On cheetah the inversion traces to training, not the dynamics: supervising exits only at the first rollout step reversed it (0.87 → 1.15). A hindsight oracle beat the best fixed-depth mix by 10–37% at equal compute, yet no trained router beat a fixed schedule.}

Research

• We found the moment grokking actually begins — the grokking preprint in plain English; see the paper for the corrected v3 claims.
• When depth helps, hurts, or doesn't matter — a taxonomy of adaptive compute in latent world models.

From the trenches

• When resuming training made my model learn the wrong data — the epoch-aware sampler bug behind Megatron-Bridge #4601.
• When Docker cache lies — the BuildKit quirk behind a vLLM nightly build failure.

_{More at blog-blogachyuthan.vercel.app}

Building

• AD-LiST-JEPA — spatiotemporal JEPA world model for autonomous driving; predicts future BEV LiDAR embeddings without labels or contrastive pairs
• KAN-Multi — routing layer that selects among 6 function bases with zero supervision; +6.8% over MLP on CIFAR-100
• MoE-Bench — open diagnostic toolkit for expert collapse & routing entropy in sparse MoE LLMs (OLMoE, JetMoE, Qwen)

Focus  ·  Self-supervised learning · Sparse MoE architectures · Neural routing · World models · LiDAR perception

Stack

→ achyuthan-s.github.io

📫 as21154@nyu.edu  ·  achyuthan.sivasankar@gmail.com  ·  LinkedIn  ·  Portfolio  ·  Blog