Vision and System Design Lab

Who We Are

Welcome to HKUST Vision and System Design Lab. The focus of our lab includes design, optimization and compression of artificial intelligence (AI) models, as well as architecture and design of AI chips/systems for energy-efficient training and inference of such AI models. Currently, we place strong focus on multimodal large foundation models for computer vision, vision-language, and medical applications. Our Lab also collaborates closely with InnoHK AI Chip Center for Smart Emerging Systems on co-design and co-optimization of AI systems across application, algorithm and hardware layers.

News

  • October 2024.  A 28nm 0.22μJ/Token Memory-Compute-Intensity-Aware CNN-Transformer Accelerator with Hybrid-Attention-Based Layer-Fusion and Cascaded Pruning for Semantic-Segmentation is accepted by ISSCC 2025. In this paper, we propose a hybrid attention mechanism coupled with a KV-weight-reused scheduler to fuse the layers between convolution and attention. Additionally, we introduce a cascaded feature map pruning strategy to achieve unified convolution-attention pruning.
  • September 2024.  Prof. Tim Cheng has been awarded the "CCF海外科技人物奖" in recognition of his outstanding achievements in the field of computer science and his contributions to the advancement of computer science in China.
  • September 2024.  RoLoRA: Finetuning Outlier-free Model with Rotation for Weight-Activation Quantization is accepted by EMNLP 2024. This paper presents RoLoRA, an efficient LLM fine-tuning method preserving the outlier-free characteristics brought by rotation operations for effective weight-activation quantization.
  • September 2024.  Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning is accepted by EMNLP 2024. In this paper, we introduce a coarse-to-fine pruner to maximize the input of effective CoT examples, thereby improving LLM mathematical reasoning capability.
  • September 2024.  Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision is accepted by TMLR. In this paper, we provide an analysis of the underlying difficulty of transformer quantization in the view of variation. A multi-crop knowledge distillation-based quantization method is proposed.
  • August 2024.  Efficient Quantization-aware Training with Adaptive Coreset Selection is accepted by TMLR. This work introduces Quantization-Aware Adaptive Coreset Selection (ACS), a method that uses error vector and disagreement scores to select crucial training samples, significantly improving the training efficiency.
  • August 2024.  Labeled-to-Unlabeled Distribution Alignment for Partially-Supervised Multi-Organ Medical Image Segmentation is accepted by Medical Image Analysis. In this paper, to address the problem of distribution mismatch, we propose a labeled-to-unlabeled distribution alignment framework that aligns feature distributions and enhances discriminative capability.
  • August 2024.  Vessel-Promoted OCT to OCTA Image Translation by Heuristic Contextual Constraints is accepted by Medical Image Analysis. We propose a novel framework, TransPro, that translates 3D OCT images into exclusive 3D OCTA images using an image translation pattern. Our main objective is to address two issues in existing image translation baselines, namely, the aimlessness in the translation process and incompleteness of the translated object.
  • June 2024.  Iterative Online Image Synthesis via Diffusion Model for Imbalanced Classification is accepted by MICCAI 2024. We introduce an Iterative Online Image Synthesis (IOIS) framework to address the class imbalance problem in medical image classification.
  • June 2024.  Rethinking Autoencoders for Medical Anomaly Detection from A Theoretical Perspective is accepted by MICCAI 2024. We for the first time provide a theoretical foundation for AE-based reconstruction methods in anomaly detection. By leveraging information theory, we elucidate the principles of these methods and reveal that the key to improving their performance lies in minimizing the information entropy of latent vectors.
  • May 2024.  Aligning Medical Images with General Knowledge from Large Language Models is early accepted by MICCAI 2024. We proposed ViP, a novel visual symptom-guided prompt learning framework for medical image analysis, which facilitates general knowledge transfer from CLIP. ViP mainly consists of two key components: a visual symptom generator and a dual-prompt network.
  • May 2024.  LENAS: Learning-based Neural Architecture Search and Ensemble for 3D Radiotherapy Dose Prediction is accepted by IEEE Transactions On Cybernetics. We proposed a novel learning-based ensemble approach named LENAS, which integrates neural architecture search with knowledge distillation for 3D radiotherapy dose prediction.
  • April 2024.  DoRA: Weight-Decomposed Low-Rank Adaptation is accepted by ICML 2024. We first introduced a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed LowRank Adaptation (DoRA).
  • March 2024.  BoNuS: Boundary Mining for Nuclei Segmentation with Partial Point Labels appears at IEEE Transactions on Medical Imaging. We proposed a novel boundary mining framework for nuclei segmentation, named BoNuS, which simultaneously learns nuclei interior and boundary information from the point labels.
  • March 2024.  Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformers is accepted by DAC 2024. This paper proposed a genetic LUT-Approximation algorithm namely GQA-LUT that can automatically determine the parameters with quantization awareness. GQA-LUT enables the employment of INT8-based LUT-Approximation that achieves an area savings of 81.3~81.7% and a power reduction of 79.3~80.2% compared to the high-precision FP/INT 32 alternatives.
  • December 2023.  AdaP-CIM: Compute-in-Memory Based Neural Network Accelerator using Adaptive Posit is accepted by DATE 2024.
  • November 2023.  CAE-GReaT: Convolutional-Auxiliary Efficient Graph Reasoning Transformer for Dense Image Predictions is accepted by International Journal of Computer Vision. We proposed an auxiliary and integrated network architecture, named Convolutional-Auxiliary Efficient Graph Reasoning Transformer, which joints strengths of both CNNs and ViT into a uniform framework.
  • November 2023.  Dynamic Sub-Cluster-Aware Network for Few-Shot Skin Disease Classification is accepted by IEEE Transactions on Neural Networks and Learning Systems. This paper addresses the problem of few-shot skin disease classification by introducing a novel approach called the Sub-Cluster-Aware Network (SCAN) that enhances accuracy in diagnosing rare skin diseases.
  • October 2023.  Nuclei segmentation with point annotations from pathology images via self-supervised learning and co-training is accepted by Medical Image Analysis. We proposed a weakly-supervised learning method for nuclei segmentation that only requires point annotations for training.
  • July 2023.  Compete to Win: Enhancing Pseudo Labels for Barely-supervised Medical Image Segmentation is accepted by IEEE Transactions on Neural Networks and Learning Systems. We proposed a novel Compete-to-Win method to enhance the pseudo label quality. In contrast to directly using one model's predictions as pseudo labels, our key idea is that high-quality pseudo labels should be generated by comparing multiple confidence maps produced by different networks to select the most confident one.
    • Read More

    Highlights

    Our publications

    Our Publications

    Our lab is at the forefront of cutting-edge AI research, covering a wide range of promising topics including AI chips/systems, compute-in-memory, electronic design automation, computer vision, vision-language, tiny machine learning, co-design, large foundational models, and medical image analysis.

    See our publications
    Our Research

    Our Research

    Our research aims to make advanced AI technologies more effective and efficient, enabling everyone to enjoy the convenience and pleasure brought by these transformative technologies that will shape the future.

    Browse our research
    Our Demos

    Our Demos

    Technology should never be limited to papers and slides. We strongly believe that our success in projects and demos is valuable and can drive potential solutions to practical problems.

    Watch our demos
    Our Team

    Our Team

    We are a team of passionate researchers who are dedicated to pushing the forefront of artificial intelligence accelerator. We are committed to creating an inclusive environment for research and acknowledge the importance of diverse knowledge backgrounds in the discovery process.

    Meet our team