<\/p>\n
\uc601\uc0c1\uc774\ud574\ud2b9\ub860 ECE 704<\/span><\/p>\n Fall 2023<\/span> In this course, we will study the latest generative models (LLMs, open-source versions, finetuning, human feedback, reward model, image generation) as well as other hot AI topics.<\/p>\n<\/div>\n Objective:<\/span><\/b> Learning technical principles for the latest popular generative models in AI.<\/span><\/p>\n Class lectures:<\/span><\/b>\u00a0\uc218\u00a0(5-6)<\/span><\/p>\n Instructor: <\/span><\/b>\uc124\uc0c1\ud6c8, \uacf5\ud559\uad00<\/span>\u00a0404, 3290-3244, sull@korea.ac.kr <\/span><\/p>\n TA:<\/span><\/b> \uae40\uc7ac\ud604, \uacf5\ud559\uad00\u00a0438, 3290-3699, jhkim@mpeg.korea.ac.kr<\/span><\/p>\n Course materials:<\/strong><\/span><\/b> Selected papers from latest conferences and blogs<\/span><\/p>\n Prerequisite<\/span><\/b>: <\/span><\/b>machine learning, deep learning <\/span><\/p>\n Grading<\/span><\/b>: midterm (40%), final (40%), Homework\u00a0(10%), attendance (10%)<\/span><\/p>\n Reference<\/strong>:<\/p>\n Contents <\/strong>(tentative)<\/p>\n 2023 0906 <\/strong>Introduction<\/p>\n 2023 0913 <\/strong>Basics and Transformer<\/p>\n [L01-1] Attention Is All You Need, NIPS 2017 (Transformer) [L01-2] Language Models are Few-Shot Learners, NeurIPS 2020 (GPT3) <\/p>\n 2023 0920<\/strong> Reinforcement learning (RL), human feedback and Instruct GPT (1\/2)<\/p>\n [L01-3]Training language models to follow instructions with human feedback, NeurIPS 2022\u00a0 (InstructGPT)<\/p>\n [L01-3 ref1] Deep Reinforcement Learning from Human Preferences, NIPS 2017<\/p>\n [L01-3 ref2]: Fine-Tuning Language Models from Human Preferences, arXiv 2019<\/p>\n [L01-3 ref3] Proximal Policy Optimization Algorithms, arXiv 2017 (PPO)<\/p>\n Reference to RL: R. Sutton and A. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, 2018<\/p>\n <\/p>\n 2023 0927 <\/strong>Reinforcement learning (RL), human feedback and Instruct GPT (2\/2) and LLAMA 2<\/p>\n L01-0 lecture note (intro2 transformer, LLM, RL and Instruct GPT) post v2.2.pdf<\/a><\/p>\n <\/p>\n 2023 1004<\/strong> LLAMA 2, LLAMA 2-Chat<\/p>\n [L02-1] Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023 (Meta AI)<\/p>\n [L02-1 ref1] LLaMA: Open and Efficient Foundation Language Modelsl, 2023 (Meta AI)<\/p>\n [L02-1 ref2] GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023<\/p>\n L02-1 LLAMA 2 2023 2307.09288 post.pdf<\/a><\/p>\n <\/p>\n 2023 1011 <\/strong>LoRA<\/p>\n [L03] LoRA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS, ICLR 2022<\/p>\n L03 LoRA-Low-Rank Adaptation of LLM ICLR 2022 2106.09685 post.pdf<\/a><\/p>\n <\/p>\n 2023 1018 <\/strong>MEMIT<\/p>\n [L04-1] MEMIT: MASS-EDITING MEMORY IN A TRANSFORMER, ICLR 2023<\/p>\n [L04-1\u00a0 ref1]\u00a0 ROME: Locating and Editing Factual Associations in GPT, NeurIPS 2022<\/p>\n [L04-1 ref2] Transformer Feed-Forward Layers Are Key-Value Memories, 2021<\/p>\n L04 MEMIT Mass-editing memory T ICLR 2023 2210.07229 post.pdf (with additional comments in red and green for Figure 3)<\/a><\/p>\n <\/p>\n 2023 1025 <\/strong>Midterm exam<\/p>\n <\/p>\n 2023 1101 <\/strong>UniAD<\/p>\n [L05-1] UniAD: Planning-oriented Autonomous Driving, CVPR 2023<\/p>\n [L05-1 ref1]\u00a0 DETR: End-to-end object detection with transformers, ECCV, 2020<\/p>\n [L05-1 ref2] TrackFormer: Multi-Object Tracking with Transformers, CVPR 2022<\/p>\n [L05-1 ref3] BEVFormer: Learning Bird’s-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers, ECCV 2022<\/p>\n L05-1 Planning-oriented Autonomous Driving cvpr 2023 2212.10156 post.pdf<\/a><\/p>\n L05-0 Supple (query) short version post.pdf<\/a><\/p>\n <\/p>\n 2023 1108 <\/strong>Fundamentals: Training, Sampling, Acceleration, Guidance (1\/2)<\/p>\n [L06-1] Denoising Diffusion Models: A Generative Learning Big Bang, CVPR 2023 (Tutorial)<\/p>\n [L06-1 ref1] Deep Unsupervised Learning using Nonequilibrium Thermodynamics, ICML 2015<\/p>\n [L06-1 ref2] Denoising-diffusion-probabilistic-models (DDPM), NeurIPS 2020<\/p>\n <\/p>\n 2023 1115 <\/strong>Fundamentals: Training, Sampling, Acceleration, Guidance (2\/2)<\/p>\n [L06-1 ref3] SCORE-BASED GENERATIVE MODELING THROUGH STOCHASTIC DIFFERENTIAL EQUATIONS, ICLR 2021<\/p>\n Reference L06-1 cvpr2023-diffusion-tutorial-part-1 post.pdf<\/a><\/p>\n <\/p>\n 2023 1122\u00a0<\/strong>Stable Diffusion (1\/2)<\/p>\n [L07-1] High-Resolution Image Synthesis with Latent Diffusion Models,\u00a0 CVPR 2022<\/p>\n [L07-1 ref1] Taming Transformers for High-Resolution Image Synthesis, CVPR 2021<\/p>\n [L07-1 ref5] Auto-Encoding Variational Bayes (VAE), 2013 1312.6114<\/p>\n [L07-1 ref7]\u00a0 VQ-VAE neural-discrete-representation-learning, NIPS 2017<\/p>\n [L07-1 ref12]\u00a0 Alignments in Text-to-Image Generation (Tutorial on Vision Foundation Models), CVPR 2023<\/p>\n <\/p>\n 2023 1129\u00a0<\/strong>Stable diffusion (2\/2), DDIM<\/p>\n [L07-2-ref2] DENOISING DIFFUSION IMPLICIT MODELS (DDIM), ICLR 2021<\/p>\n
\nProfessor Sanghoon Sull
\nSchool of Electrical Engineering, Korea University<\/span><\/p>\n
\n\n
\n
\n[L01-1 ref1]\u00a0 https:\/\/jalammar.github.io\/illustrated-transformer\/<\/p>\n
\n[L01-2 ref2]\u00a0 https:\/\/jalammar.github.io\/illustrated-gpt2\/<\/p>\n
\nSimo Sarkka and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019 (Def. 3.9 (White noise), Def. 4.1 (Brownian motion)<\/p>\n