A. Awan

Most-cited papers

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Marah Abdin, Sam Adé Jacobs, A. Awan, J. Aneja +85 more
2024arXiv.org2,593 citations
Semantic Scholar
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, A. Awan, Cheng Li +7 more
2022International Conference for High Performance Computing, Networking, Storage and Analysis730 citations Open access
Semantic Scholar
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
Samyam Rajbhandari, Cong-Long Li, Z. Yao, Min-Jia Zhang +4 more
2022International Conference on Machine Learning688 citations
Semantic Scholar
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
Connor Holmes, Masahiro Tanaka, Michael Wyatt, A. Awan +7 more
2024arXiv.org171 citations
Semantic Scholar
S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters
A. Awan, Khaled Hamidouche, J. Hashmi, D. Panda
2017ACM SIGPLAN Symposium on Principles & Practice of Parallel Programming145 citations
Semantic Scholar
DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales
Z. Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari +15 more
2023arXiv.org134 citations Open access
Semantic Scholar
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
Han-Lin Tang, Shaoduo Gan, A. Awan, Samyam Rajbhandari +5 more
2021International Conference on Machine Learning120 citations
Semantic Scholar
Scalable and Efficient MoE Training for Multitask Multilingual Models
Young Jin Kim, A. Awan, Alexandre Muzio, Andres Felipe Cruz Salinas +5 more
2021arXiv.org103 citations
Semantic Scholar
A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
Siddharth Singh, Olatunji Ruwase, A. Awan, Samyam Rajbhandari +2 more
2023International Conference on Supercomputing90 citations Open access
Semantic Scholar
An In-depth Performance Characterization of CPU- and GPU-based DNN Training on Modern Architectures
A. Awan, H. Subramoni, D. Panda
2017MLHPC@SC66 citations
Semantic Scholar
GEMS: GPU-Enabled Memory-Aware Model-Parallelism System for Distributed DNN Training
Arpan Jain, A. Awan, Asmaa Aljuhani, J. Hashmi +5 more
2020International Conference for High Performance Computing, Networking, Storage and Analysis56 citations
Semantic Scholar
Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?
A. Awan, Ching-Hsiang Chu, H. Subramoni, D. Panda
2017European MPI Users Group Meeting52 citations
Semantic Scholar
Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation
A. Awan, Jereon Bédorf, Ching-Hsiang Chu, H. Subramoni +1 more
2018IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing50 citations Open access
Semantic Scholar
Efficient Large Message Broadcast using NCCL and CUDA-Aware MPI for Deep Learning
A. Awan, Khaled Hamidouche, Akshay Venkatesh, D. Panda
2016European MPI Users Group Meeting50 citations
Semantic Scholar
NV-group: link-efficient reduction for distributed deep learning on modern dense GPU systems
Ching-Hsiang Chu, P. Kousha, A. Awan, Kawthar Shafie Khorassani +2 more
2020International Conference on Supercomputing44 citations
Semantic Scholar
1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB’s Convergence Speed
Cong-Long Li, A. Awan, Hanlin Tang, Samyam Rajbhandari +1 more
2021International Conference on High Performance Computing37 citations
Semantic Scholar
Performance Characterization of DNN Training using TensorFlow and PyTorch on Modern Clusters
Arpan Jain, A. Awan, Quentin Anthony, H. Subramoni +1 more
2019IEEE International Conference on Cluster Computing34 citations
Semantic Scholar
OC-DNN: Exploiting Advanced Unified Memory Capabilities in CUDA 9 and Volta GPUs for Out-of-Core DNN Training
A. Awan, Ching-Hsiang Chu, H. Subramoni, Xiao-Yi Lu +1 more
2018International Conference on High Performance Computing34 citations
Semantic Scholar
Privacy-aware searching with oblivious term matching for cloud storage
Z. Pervez, A. Awan, A. Khattak, Sungyoung Lee +1 more
2012Journal of Supercomputing34 citations
Semantic Scholar
DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies
S. Song, Bonnie Kruft, Min-Jia Zhang, Cong-Long Li +88 more
2023arXiv.org27 citations Open access
Semantic Scholar
Exploiting GPUDirect RDMA in Designing High Performance OpenSHMEM for NVIDIA GPU Clusters
Khaled Hamidouche, Akshay Venkatesh, A. Awan, H. Subramoni +2 more
2015IEEE International Conference on Cluster Computing26 citations
Semantic Scholar
Optimized large-message broadcast for deep learning workloads: MPI, MPI+NCCL, or NCCL2?
A. Awan, K. V. Manian, Ching-Hsiang Chu, H. Subramoni +1 more
2019Parallel Computing24 citations Open access
Semantic Scholar
CUDA Kernel Based Collective Reduction Operations on Large-scale GPU Clusters
Ching-Hsiang Chu, Khaled Hamidouche, Akshay Venkatesh, A. Awan +1 more
2016IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing23 citations
Semantic Scholar
Efficient and Scalable Multi-Source Streaming Broadcast on GPU Clusters for Deep Learning
Ching-Hsiang Chu, Xiao-Yi Lu, A. Awan, H. Subramoni +3 more
2017International Conference on Parallel Processing22 citations
Semantic Scholar
Scaling TensorFlow, PyTorch, and MXNet using MVAPICH2 for High-Performance Deep Learning on Frontera
Arpan Jain, A. Awan, H. Subramoni, D. Panda
2019Dynamic Languages Symposium21 citations
Semantic Scholar
Exploiting Hardware Multicast and GPUDirect RDMA for Efficient Broadcast
Ching-Hsiang Chu, Xiao-Yi Lu, A. Awan, H. Subramoni +2 more
2019IEEE Transactions on Parallel and Distributed Systems19 citations Open access
Semantic Scholar
Communication Profiling and Characterization of Deep-Learning Workloads on Clusters With High-Performance Interconnects
A. Awan, Arpan Jain, Ching-Hsiang Chu, H. Subramoni +1 more
2020IEEE Micro18 citations
Semantic Scholar
Designing Non-blocking Personalized Collectives with Near Perfect Overlap for RDMA-Enabled Clusters
H. Subramoni, A. Awan, Khaled Hamidouche, D. Pekurovsky +4 more
2015ISC High Performance17 citations
Semantic Scholar
DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention
Z. Yao, Xiaoxia Wu, Cong-Long Li, Minjia Zhang +5 more
2023arXiv.org11 citations Open access
Semantic Scholar
Efficient Training of Semantic Image Segmentation on Summit using Horovod and MVAPICH2-GDR
Quentin Anthony, A. Awan, Arpan Jain, H. Subramoni +1 more
2020IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum11 citations
Semantic Scholar
Intercloud message exchange middleware
M. B. Amin, W. A. Khan, A. Awan, Sungyoung Lee
2012International Conference on Ubiquitous Information Management and Communication11 citations
Semantic Scholar
MCR-DL: Mix-and-Match Communication Runtime for Deep Learning
Quentin Anthony, A. Awan, Jeff Rasley, Yuxiong He +4 more
2023IEEE International Parallel and Distributed Processing Symposium9 citations Open access
Semantic Scholar
High performance distributed deep learning: a beginner's guide
Dhabaleswar K. Dk Panda, A. Awan, H. Subramoni
2019ACM SIGPLAN Symposium on Principles & Practice of Parallel Programming9 citations
Semantic Scholar
CUDA-Aware OpenSHMEM: Extensions and Designs for High Performance OpenSHMEM on GPU Clusters
Khaled Hamidouche, Akshay Venkatesh, A. Awan, H. Subramoni +2 more
2016Parallel Computing9 citations
Semantic Scholar
HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training with TensorFlow
A. Awan, Arpan Jain, Quentin Anthony, H. Subramoni +1 more
2019ISC High Performance7 citations Open access
Semantic Scholar
CUDA M3: Designing Efficient CUDA Managed Memory-Aware MPI by Exploiting GDR and IPC
Khaled Hamidouche, A. Awan, Akshay Venkatesh, D. Panda
2016International Conference on High Performance Computing6 citations
Semantic Scholar
HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow
A. Awan, Arpan Jain, Quentin Anthony, H. Subramoni +1 more
2019arXiv.org5 citations
Semantic Scholar
GPU-Aware Design, Implementation, and Evaluation of Non-blocking Collective Benchmarks
A. Awan, Khaled Hamidouche, Akshay Venkatesh, Jonathan L. Perkins +2 more
2015European MPI Users Group Meeting5 citations
Semantic Scholar
Communication Profiling and Characterization of Deep Learning Workloads on Clusters with High-Performance Interconnects
A. Awan, Arpan Jain, Ching-Hsiang Chu, H. Subramoni +1 more
2019IEEE Symposium on High-Performance Interconnects3 citations
Semantic Scholar
A Case for Non-blocking Collectives in OpenSHMEM: Design, Implementation, and Performance Evaluation using MVAPICH2-X
A. Awan, Khaled Hamidouche, Ching-Hsiang Chu, D. Panda
2015Workshop on OpenSHMEM and Related Technologies3 citations
Semantic Scholar
On-demand Connection Management for OpenSHMEM and OpenSHMEM+MPI
Sourav Chakraborty, H. Subramoni, Jonathan L. Perkins, A. Awan +1 more
20152015 IEEE International Parallel and Distributed Processing Symposium Workshop3 citations
Semantic Scholar
An MPI-IO Compliant Java Based Parallel I/O Library
A. Awan, M. B. Amin, S. Hussain, A. Shafi +1 more
2013IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing1 citations
Semantic Scholar
Towards Efficient Support for Parallel I/O in Java HPC
A. Awan, Muhammad Sohaib Ayub, A. Shafi, Sungyoung Lee
2012International Conference on Parallel and Distributed Computing: Applications and Technologies1 citations
Semantic Scholar
A Novel Tensor-Expert Hybrid Parallelism Approach to Scale Mixture-of-Experts Training
Siddharth Singh, Olatunji Ruwase, A. Awan, Samyam Rajbhandari +2 more
2023arXiv.org0 citations
Semantic Scholar
Benchmarking in the Data Center: Expanding to the Cloud
S. Aseeri, A. Awan, Shahzeb Siddiqui, Sven Breuner +1 more
20220 citations
Semantic Scholar
An MPI-IO Compliant Java Based Parallel I/O Library
A. Awan, M. B. Amin, S. Hussain, A. Shafi +1 more
20132013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing0 citations
Semantic Scholar
Sim-Hadoop : 신뢰성 있고 효율적인 N-body 시뮬레이션을 위한 Hadoop 분산 파일 시스템과 병렬 I / O
마드 아, 승룡 이, 태충 정, A. Awan +2 more
20130 citations
Semantic Scholar