電腦視覺與虛擬實境組
COMPUTER VISION & VR GROUP

Vision Group

INTELLIGENT TRANSPORTATION SYSTEM

ASSISTIVE ALARMING SYSTEM

PEDESTRIAN & VEHICLE DETECTION

VR Group

ACTION RECOGNITION

HUMAN-COMPUTER INTERACTION

AUGMENTED REALITY

MAC-RAD：透過從RGB視訊分離可重複使用資產模組實現模組化虛擬分身合成

MAC-RAD: Modular Avatar Composition via Reusable Assets Disentanglement from RGB Video

在本論文中，我們提出了一個兩階段模組化框架，用於從單目 RGB 影片中重建虛擬分身，旨在支援服裝級控制和虛擬分身組合。為此，我們定義了兩個階段，即解纏階段和組合階段。在解纏階段，我們的系統將輸入影片分解為具有語義意義的組件，包括皮膚紋理和一組帶有紋理的服裝網格，每個組件都與一個規範姿勢對齊，並由用戶提供的文字提示引導。此階段利用參數模型（例如 SMPL-X）進行姿勢和形狀估計，確保跨幀的一致性。

在組合階段，解纏後的組件將根據用戶的設定（例如體型和服裝選擇）重新組合成一個統一的、可動畫化的網格。產生的虛擬分身支援運動重新導向和渲染，同時支援靈活的服裝重組。

我們的框架強調模組化、可重複使用性和速度，允許快速創造虛擬分身，且僅需少量品質權衡。實驗結果展現了高度的視覺連貫性、服裝完整性以及對構圖的支援。作為未來的發展方向，我們設想在系統中擴展一個 CLIP 引導的服裝檢索模組。這將使用戶能夠通過自然語言描述進行直覺的、基於文字的虛擬分身編輯。

In this thesis, we present a two-stage modular framework for avatar reconstruction from monocular RGB videos, designed to support garment-level control and avatar composition. To this end, two stages are defined, namely, Disentanglement Stage and Composition Stage. In the Disentanglement Stage, our system decomposes input video into semantically meaningful components, including a skin texture and a set of textured clothing meshes, each aligned to a canonical pose and guided by user-provided textual prompts. This stage leverages parametric models (e.g., SMPL-X) for pose and shape estimation, ensuring consistency across frames.

In the Composition Stage, the disentangled components are reassembled into a unified, animatable mesh based on user-defined settings, such as body shape and clothing selection. The resulting avatar supports motion retargeting and rendering while enabling flexible garment recombination.

Our framework emphasizes modularity, reusability, and speed, allowing rapid avatar creation with only minor quality trade-offs. Experimental results demonstrate high visual coherence, garment integrity, and support for composition. As a future direction, we envision extending the system with a CLIP-guided garment retrieval module. This would enable intuitive, text-based avatar editing through natural language descriptions.

Keywords: Virtual Avatar, 3D Human Reconstruction, Modular Avatar Composition, Garment Disentanglement, Gaussian Splatting, Diffusion Model

AR遠端會議中的即時摘要與重播助理

ARM-RSPA: Augmented Reality Meeting with Real-Time Summarization and Playback Assistant

自疫情以後，遠端會議成為部分工作的常態，現在主流的會議系統如 Google Meet 與 Microsoft Teams 被廣泛使用。而隨著 AR/VR 技術的進步，虛擬會議能夠得到進一步的提升，並提供高度沉浸感與互動性的體驗給使用者。

傳統會議中，通常會有人整理會議並記錄以方便事後釐清內容；而現在視訊會議也有逐字稿等工具將講者的話轉錄下來，但在現在 VR/AR 的會議內，缺少記錄並輔助使用者的工具。因此我們提出了 ARM-RSPA，一個即時紀錄會議內容與虛擬環境狀態，並根據使用者要求摘要且重播關鍵的互動的助手系統。透過 AR 重播，ARM-RSPA 提供使用者更加方便且易於理解的方法。

在後面的實驗中，ARM-RSPA 展現出在會議中的運用，並成功幫助使用者理解講者所講解的內容。我們也探討 ARM-RSPA 與其他遠端會議的差別以及未來 AR 會議工具可能的發展。

Since the pandemic, remote meetings have become the norm for some work environments, with mainstream meeting systems like Google Meet and Microsoft Teams being widely used. With advancements in AR/VR technology, virtual meetings can be further enhanced to provide users with highly immersive and interactive experiences.

In traditional meetings, someone typically organizes and records the meeting to clarify content afterward. While video conferences now have tools like transcripts to record speakers' words, VR/AR meetings lack tools to help with the recording and assist users. To address this challenge, we develop ARM-RSPA in the research work, which is a real-time assistant system that records meeting content and virtual environment states, summarizes based on user requests, and replays key interactions. Through AR replay, ARM-RSPA provides users with a more convenient and easily understandable way to realize the content of the meetings.

To validate our research, we have conducted various experiments, where ARM-RSPA demonstrates its application in meetings and successfully helps users understand the content explained by speakers. We also discuss the differences between ARM-RSPA and other remote meeting solutions to highlight the high promise of our proposal system. Finally, we also mention our potential developments for other AR meeting tools in the future.

Keywords: VR/AR, Live Stream Summarization, LLM Assistant

基於急診室之心電圖影像與呼吸影片預測住院風險的
深度學習模型

Cardio-Respiratory Deep Learning Model for Predicting Hospital Admission from Emergency Department Electrocardiogram Images and Respiration Videos

急診室擁擠已成為現代醫療體系中的一項重大挑戰，不僅延誤病患的及時治療，也影響資源的最佳分配。臨床醫常依賴快速且主觀的「臨床直覺」評估來輔助分診與決定是否住院。我們假設深度學習模型同樣能利用病患的影像外觀，以更客觀且一致的方式預測住院需求。本研究提出一種創新的心肺雙模態深度學習模型，整合兩種非侵入式資料：從短程呼吸影片中擷取的呼吸波形影像，以及自心電圖（ECG）影像。為了強化模型對關鍵特徵的聚焦，我們引入注意力機制，並使用 Class-Balanced Loss 處理訓練資料中嚴重的類別不平衡。考量只有部分病患提供完整的 ECG 資料，我們先在外部資料集上以 ConvNeXt 架構預訓練 ECG特徵擷取器，以提升心電訊號的表示能力。對於呼吸模態，我們首先在無固定攝影機角度的複雜環境中，自動定位病患的胸部感興趣區域；接著透過 3D 卷積神經網路與雙向長短期記憶（BiLSTM）網路組合，估計呼吸訊號並將其轉換為影像格式，由端到端訓練的 ConvNeXt 模型進行特徵抽取。最後，我們將兩種模態的特徵向量與一個標示 ECG 資料可用性的二元指標串聯，並透過全連接層預測住院結果。
在臺大醫院（NTUH）資料集上的實驗結果顯示，融合 ECG 與呼吸模態的模型達到 AUROC 0.863 和準確率 0.814；僅使用呼吸影片的模型也獲得 AUROC 0.810 和準確率 0.788。這些成果證明本方法具備作為急診室客觀、非侵入式臨床決策支援工具的可行性。

Background: Predicting hospital admission at the emergency department (ED) triage can enable earlier interventions and more efficient resource allocation. Previous approaches, however, have primarily relied on structured data for admission prediction. The use of multimodal data with cardiac and respiratory signals may capture subtle clinical signs to improve the prediction of hospital admission at ED triage.This study aims to develop a reliable and accurate deep learning fusion model that integrates ECG images and respiration video to predict hospital admission at ED triage.We prospectively collected adult ED patient data from National Taiwan University Hospital (NTUH), including electrocardiogram (ECG) images and respiration video recorded during triage. The proposed model employs separate modality-specific branches with transfer learning for the ECG branch, explicit modality indicators to handle missing ECG inputs, and a spatial attention module to enhance cardiopulmonary feature extraction. Class-balanced loss was applied to mitigate the impact of dataset imbalance. Model performance was compared against a baseline Taiwan Triage Acuity Score (TTAS) model and single-modality variants. We further evaluated attention heatmaps generated by score-class activation mapping (score-CAM) for interpretability.The dataset comprised 118 patients (positive admission rate: 27%). The proposed ECG–respiration fusion model achieved an accuracy of 0.814, a recall of 0.749, a precision of 0.635, an F1-score of 0.680, an area under the receiver operating characteristic curve (AUROC) of 0.863, and an area under the precision-recall curve (AUPRC) of 0.675, outperforming the TTAS baseline and single-modality models. Ablation studies confirmed the contributions of spatial attention and class-balanced loss. Score-CAM visualizations revealed that the model consistently focused on physiologically relevant regions, such as respiratory waveform inflection points.The proposed multimodal deep learning-based model demonstrates strong potential for early hospital admission prediction in the ED, even under incomplete multimodal input conditions. Its integration of ECG and respiration video enhances both the predictive accuracy and interpretability of the existing triage system, making it suitable for addition to real-world triage workflows.

Keywords: Admission, Electrocardiography, Respiration, Deep Learning, Computer Vision

透過空間感知與動態類別平衡提升鳥瞰圖轉三維的佔據預測

HintOcc : Enhancing BEV-to-3D Reconstruction in Occupancy Prediction with Spatial-Awareness and Dynamic Class Balancing

隨著自動駕駛的快速發展，三維感知已成為智慧汽車的核心能力。基於攝 影機的系統必須準確理解環境的結構和語義，以支援運動規劃和決策等下游任 務。在複雜動態的交通場景中，如何高效且準確地偵測物體，是確保行車安全 的重要挑戰。 傳統方法利用三維體素表示進行編碼和解碼。雖然能達到良好性能，但常 伴隨龐大的記憶體消耗與高昂的計算成本。為提升效率，部分研究將三維體素 空間投影至二維鳥瞰圖表示，有效降低記憶體開銷並維持合理性能。然而，反 向過程中，即自 BEV 特徵重建回三維空間時，常因缺乏垂直訊息而導致準確 率下降。此外，現有的三維佔用率資料集普遍存在嚴重的類別不平衡問題，對 於如行人與摩托車等低出現頻率但對安全至關重要的目標，其表徵能力明顯不 足。 針對上述挑戰，我們提出 HintOcc，一個高效的三維佔據預測框架，旨在 強化 BEV 特徵的三維重建能力，並改善類別不平衡的問題。它能夠增強二維 鳥瞰圖特徵到三維體素空間重建的能力，同時提升現實世界數據集中代表性不 足類別的性能。首先，我們引入了二維垂直視圖分支，以結構提示形式提供高 度訊息，輔助網路對二維至三維特徵的重建過程。其次，我們採用可變形深度 可分離頭進行空間自適應解碼，同時降低參數開銷。最後，我們提出了一種批 次動態加權策略，根據每個訓練批次中類別的出現情況，自適應地強化稀有類 別的學習。 我們在 Occ3D-NuScenes 基準上評估了我們的方法。實驗結果表明，在相 當的計算限制條件下，HintOcc 的表現優於現有方法，並且與基準模型相比， 它提高了代表性不足類別的準確率。消融研究進一步驗證了每個組件在增強基 於二維鳥瞰圖特徵的三維佔用預測方面的有效性。

With the rapid advancement of autonomous driving, 3D perception has become essential for intelligent vehicles. In complex and dynamic traffic scenes, accurate and efficient object detection is critical for ensuring safety. Recent meth ods leverage bird’s-eye view (BEV) representations for their computational efficiency, but lifting 2D BEV features back into 3Dvoxel space remains a fundamental challenge due to the loss of vertical information during the encoding process. This often leads to degraded reconstruction performance. Additionally, class imbalance in real-world 3D occupancy datasets—where safety-critical classes such as pedestrians and bicycles are severely underrepresented compared to dominant categories like roads and buildings—significantly hinders the model’s performance on these rare classes. To address these challenges, we propose HintOcc, an effi cient framework that improves 2D BEV-to-3D reconstruc tion and alleviates class imbalance in 3D occupancy pre diction. HintOcc introduces (1) a vertical-view branch to recover height information lost during BEV encoding, (2) a deformable depthwise separable head for flexible and lightweight decoding, and (3) a batch-wise dynamic weight ing strategy to better emphasize rare classes during train ing. Evaluated on the Occ3D-NuScenes benchmark, Hin tOcc achieves competitive performance under similar com putational budgets, especially improving accuracy on un derrepresented classes.

Keywords: Deep Learning, Computer Vision, 3D Occupancy Network, Class Imbalance, Bird’s-Eye View (BEV).

UniMapOcc: A Unified Multi-Task Learning Framework for Joint 3D Occupancy Prediction and BEV Map Segmentation

佔據預測與鳥瞰圖地圖分割的統一多任務學習框架

隨著自駕車感知技術發展，僅辨識個別道路使用者已不足以完整描述複雜的駕駛環境。自駕車輛除了需精確掌握周遭的障礙物分布狀態之外，亦必須同步辨識路面結構資訊，以支援後續之路徑規劃與行為決策。三維語意佔據預測能以體素形式呈現場景中的空間佔用與語意資訊；鳥瞰圖地圖分割則描述可行駛區域、人行穿越道、停止線及道路分隔線等道路結構。這兩類資訊具有互補性，但許多現有方法通常採用各自獨立的模型，因而需要重複執行影像特徵擷取與視角轉換，亦未能充分利用兩項任務之間共享的空間資訊。

然而，將三維語意佔據預測與鳥瞰圖地圖分割整合於同一模型仍具有挑戰。前者著重體積空間中的幾何與語意理解，後者則依賴平面道路布局與精細邊界。直接使用為佔據預測所建立的特徵進行地圖解碼，可能無法充分保留道路結構資訊。此外，人行穿越道、停止線及道路分隔線等稀疏細長元素，也容易出現不完整預測與邊界誤差。

針對上述問題，本論文提出 UniMapOcc，一套以多視角環視影像為輸入的聯合感知架構，在單一模型中同時執行三維語意佔據預測與鳥瞰圖地圖分割。UniMapOcc 建立共享的三維體素與二維鳥瞰圖表示，並引入體素引導地圖精煉模組，利用三維幾何線索改善鳥瞰圖地圖特徵。此外，本論文結合多尺度特徵精煉與群組感知地圖增強，以提升模型對不同道路結構，尤其是稀疏細長地圖元素的辨識能力。

本論文使用 Occ3D-nuScenes 與其對應的 nuScenes 地圖標註進行評估。在平衡兩項任務表現的設定下，UniMapOcc 達到 39.40 的三維佔據預測 mIoU 與 49.97 的地圖分割 mIoU，分別較重現的 ProtoOcc 與 BEVFusion 基準提升 0.31 與 2.86 個百分點。此外，在以佔據預測為導向的設定下，三維佔據預測 mIoU 達 39.58，超越所有比較方法；在以地圖分割為導向的設定下，地圖分割 mIoU 進一步達 51.72，亦超越所有比較方法。實驗結果顯示，共享體素與鳥瞰圖表示搭配任務專屬的特徵精煉，能在維持三維場景理解能力的情況下同時提升道路地圖預測表現，並可依應用需求調整兩項任務之間的取捨。

With the advancement of autonomous driving perception, identifying individual road users alone is no longer sufficient to fully characterize complex driving environments. Autonomous vehicles must accurately perceive the distribution of surrounding obstacles while simultaneously recognizing road structure information to support subsequent path planning and behavioral decision-making. Three-dimensional semantic occupancy prediction represents spatial occupancy and semantic information in a voxelized form, whereas bird’s-eye-view (BEV) map segmentation describes road structures such as drivable areas, pedestrian crossings, stop lines, and dividers. Although these two types of information are complementary, many existing approaches employ separate models, resulting in redundant image feature extraction and view transformation while failing to fully exploit the spatial information shared between the two tasks.

However, integrating 3D semantic occupancy prediction and BEV map segmentation into a single model remains challenging. The former focuses on geometric and semantic understanding in volumetric space, whereas the latter relies on planar road layouts and fine-grained boundaries. Directly decoding maps from features constructed for occupancy prediction may not sufficiently preserve road structure information. Moreover, sparse and thin elements, such as pedestrian crossings, stop lines, and dividers, are particularly susceptible to incomplete predictions and boundary errors.

To address these challenges, this thesis proposes UniMapOcc, a unified perception framework that takes multi-view surround-camera images as input and jointly performs 3D semantic occupancy prediction and BEV map segmentation within a single model. UniMapOcc constructs shared 3D voxel and 2D BEV representations and introduces Voxel-Guided Map Refinement to improve BEV map features using 3D geometric cues. In addition, multi-scale feature refinement and Group-Aware Map Enhancement (GME) are incorporated to improve the model’s ability to recognize different road structures, particularly sparse and thin map elements.

UniMapOcc is evaluated on Occ3D-nuScenes and the corresponding nuScenes map annotations. Under a configuration that balances the performance of the two tasks, UniMapOcc achieves an mIoU of 39.40 for 3D occupancy prediction and 49.97 for BEV map segmentation, outperforming the reproduced ProtoOcc and BEVFusion baselines by 0.31 and 2.86 percentage points, respectively. Moreover, under the occupancy-oriented configuration, UniMapOcc achieves an occupancy mIoU of 39.58, surpassing all compared methods, while under the map-oriented configuration, the map mIoU further reaches 51.72, likewise surpassing all compared methods. These results indicate that shared voxel and BEV representations, together with task-specific feature refinement, can improve road map prediction performance while maintaining 3D scene understanding capability, and allow the trade-off between the two tasks to be adjusted according to application requirements.

Keywords: Computer Vision, 3D Occupancy Prediction, BEV Map Segmentation, Bird’s-Eye View (BEV), Multi-Task Learning

MP-Occ: 透過多原型查詢與高度感知鳥瞰圖融合提升三維佔據預測
MP-Occ: Enhancing 3D Occupancy Prediction with Multi-Prototype Queries and Height-Aware BEV Fusion

純視覺三維佔據預測使用多視角影像為輸入，旨在重建周遭空間中的幾何配置與語義資訊，並為自動駕駛系統提供重要的感知基礎。然而，現有基於鳥瞰圖與原型的方法仍存在幾項限制。首先，在將鳥瞰圖特徵展開至三維空間時，簡單的通道重塑難以提供明確的高度資訊，使不同高度層的特徵缺乏足夠區辨性。其次，鳥瞰圖分支通常只透過最終三維預測損失接受間接監督，限制了其學習具判別性語義表徵的效果。此外，單一類別原型難以完整描述同一語義類別內的幾何與外觀多樣性，進而限制模型對不同類別特徵的辨識能力。

為了解決上述問題，本研究提出 MP-Occ，一個兼顧預測準確度與推論效率的純視覺三維佔據預測框架。MP-Occ 包含三項主要設計。高度感知鳥瞰圖展開模組 (Height-Aware BEV Expansion Module) 在鳥瞰圖特徵展開至三維空間時加入可學習的高度嵌入，以提升不同高度層之間的特徵差異性。鳥瞰圖輔助頭 (BEV Auxiliary Head) 透過鳥瞰圖輔助損失 (BEV Auxiliary Loss) 直接監督鳥瞰圖分支，以強化其中間特徵的語義判別能力。多原型查詢解碼器 (Multi-Prototype Query Decoder) 則為每個類別配置多個原型，並透過多樣性損失降低同類原型收斂至相同表徵的風險，使模型能夠更細緻地描述類別內部的多樣性。

本研究在 Occ3D-nuScenes 資料集上進行實驗，以評估 MP-Occ 的效能。根據實驗結果，MP-Occ 在整體三維佔據預測表現與多個前景物件類別上皆取得改善。

關鍵字：電腦視覺、三維佔據預測、純視覺感知、多原型查詢、鳥瞰視角特徵融合、自動駕駛

Vision-centric 3D occupancy prediction reconstructs the geometry and semantics of the surrounding environment using only multi-view camera images, making it an important technique in autonomous driving perception. However, existing BEV- and prototype-based methods still have several limitations. First, when BEV features are expanded into 3D space, simple channel reshaping provides limited height information, making features at different height levels insufficiently distinguishable. Second, the BEV branch is usually supervised only indirectly through the final 3D prediction loss, which limits the learning of semantic features. In addition, a single class prototype is insufficient to fully describe the geometric and appearance diversity within the same semantic category, which can affect semantic classification accuracy.

To address these issues, we propose MP-Occ, a vision-centric 3D occupancy prediction framework that balances prediction accuracy and inference efficiency. MP-Occ contains three main designs. The Height-Aware BEV Expansion Module injects learnable height embeddings when expanding BEV features into 3D space, allowing the model to distinguish semantic differences across height levels. The BEV Auxiliary Head directly supervises the BEV branch through the BEV Auxiliary Loss, strengthening the semantic discriminability of its intermediate features. The Multi-Prototype Query Decoder assigns multiple prototypes to each class and uses a diversity loss to reduce the risk of same-class prototypes converging to identical representations, enabling the model to describe intra-class diversity in a more fine-grained manner.

We evaluate MP-Occ on the Occ3D-nuScenes dataset and compare it with existing methods. Experimental results show that MP-Occ improves overall 3D occupancy prediction performance and achieves gains on multiple foreground object classes, while introducing only minor additional inference time and memory cost. Ablation studies further demonstrate the individual contribution of each component, and visualization results show that prototypes of the same class maintain semantic consistency while producing different feature responses. These results demonstrate that MP-Occ effectively improves representation capacity for vision-centric 3D occupancy prediction while maintaining practical inference efficiency.

Keywords: Computer Vision, 3D Occupancy Prediction, Vision-Centric Perception, Multi-Prototype Queries, BEV Feature Fusion, Autonomous Driving