Deep Learning

Höfundar: Christopher M. Bishop; Hugh Bishop (Útgáfa: 0)
Deep Learning

Kaup valmöguleikar

This book offers a comprehensive introduction to the central ideas that underpin deep learning. It is intended both for newcomers to machine learning and for those already experienced in the field. Covering key concepts relating to contemporary architectures and techniques, this essential book equips readers with a robust foundation for potential future specialization. The field of deep learning is undergoing rapid evolution, and therefore this book focusses on ideas that are likely to endure the test of time.

The book is organized into numerous bite-sized chapters, each exploring a distinct topic, and the narrative follows a linear progression, with each chapter building upon content from its predecessors. This structure is well-suited to teaching a two-semester undergraduate or postgraduate machine learning course, while remaining equally relevant to those engaged in active research or in self-study. A full understanding of machine learning requires some mathematical background and so the book includes a self-contained introduction to probability theory.

However, the focus of the book is on conveying a clear understanding of ideas, with emphasis on the real-world practical value of techniques rather than on abstract theory. Complex concepts are therefore presented from multiple complementary perspectives including textual descriptions, diagrams, mathematical formulae, and pseudo-code. Chris Bishop is a Technical Fellow at Microsoft and is the Director of Microsoft Research AI4Science.

He is a Fellow of Darwin College Cambridge, a Fellow of the Royal Academy of Engineering, and a Fellow of the Royal Society. Hugh Bishop is an Applied Scientist at Wayve, a deep learning autonomous driving company in London, where he designs and trains deep neural networks. He completed his MPhil in Machine Learning and Machine Intelligence at Cambridge University. “Chris Bishop wrote a terrific textbook on neural networks in 1995 and has a deep knowledge of the field and its core ideas.

His many years of experience in explaining neural networks have made him extremely skillful at presenting complicated ideas in the simplest possible way and it is a delight to see these skills applied to the revolutionary new developments in the field. ” -- Geoffrey Hinton "With the recent explosion of deep learning and AI as a research topic, and the quickly growing importance of AI applications, a modern textbook on the topic was badly needed.

The "New Bishop" masterfully fills the gap, covering algorithms for supervised and unsupervised learning, modern deep learning architecture families, as well as how to apply all of this to various application areas. " – Yann LeCun “This excellent and very educational book will bring the reader up to date with the main concepts and advances in deep learning with a solid anchoring in probability. Theseconcepts are powering current industrial AI systems and are likely to form the basis of further advances towards artificial general intelligence.

Nánar um bókina

Útgefandi
Springer Nature
ISBN
9783031454684
Print ISBN
9783031454677
Format
Page Fidelity (PDF)
Útgáfa
0
Höfundar
Christopher M. Bishop; Hugh Bishop
Tungumál
English
Útgefið
2023-11-01
Prent takmörkun á líftíma
100
Prent takmörkun
2
Afritunar takmörkun
2

Kaflar

  • Preface
  • Goals of the book
  • Responsible use of technology
  • Structure of the book
  • References
  • Exercises
  • Mathematical notation
  • Acknowledgements
  • Contents
  • 1 The Deep Learning Revolution
  • 1.1. The Impact of Deep Learning
  • 1.1.1 Medical diagnosis
  • 1.1.2 Protein structure
  • 1.1.3 Image synthesis
  • 1.1.4 Large language models
  • 1.2. A Tutorial Example
  • 1.2.1 Synthetic data
  • 1.2.2 Linear models
  • 1.2.3 Error function
  • 1.2.4 Model complexity
  • 1.2.5 Regularization
  • 1.2.6 Model selection
  • 1.3. A Brief History of Machine Learning
  • 1.3.1 Single-layer networks
  • 1.3.2 Backpropagation
  • 1.3.3 Deep networks
  • 2 Probabilities
  • 2.1. The Rules of Probability
  • 2.1.1 A medical screening example
  • 2.1.2 The sum and product rules
  • 2.1.3 Bayes’ theorem
  • 2.1.4 Medical screening revisited
  • 2.1.5 Prior and posterior probabilities
  • 2.1.6 Independent variables
  • 2.2. Probability Densities
  • 2.2.1 Example distributions
  • 2.2.2 Expectations and covariances
  • 2.3. The Gaussian Distribution
  • 2.3.1 Mean and variance
  • 2.3.2 Likelihood function
  • 2.3.3 Bias of maximum likelihood
  • 2.3.4 Linear regression
  • 2.4. Transformation of Densities
  • 2.4.1 Multivariate distributions
  • 2.5. Information Theory
  • 2.5.1 Entropy
  • 2.5.2 Physics perspective
  • 2.5.3 Differential entropy
  • 2.5.4 Maximum entropy
  • 2.5.5 Kullback–Leibler divergence
  • 2.5.6 Conditional entropy
  • 2.5.7 Mutual information
  • 2.6. Bayesian Probabilities
  • 2.6.1 Model parameters
  • 2.6.2 Regularization
  • 2.6.3 Bayesian machine learning
  • Exercises
  • 3 Standard Distributions
  • 3.1. Discrete Variables
  • 3.1.1 Bernoulli distribution
  • 3.1.2 Binomial distribution
  • 3.1.3 Multinomial distribution
  • 3.2. The Multivariate Gaussian
  • 3.2.1 Geometry of the Gaussian
  • 3.2.2 Moments
  • 3.2.3 Limitations
  • 3.2.4 Conditional distribution
  • 3.2.5 Marginal distribution
  • 3.2.6 Bayes’ theorem
  • 3.2.7 Maximum likelihood
  • 3.2.8 Sequential estimation
  • 3.2.9 Mixtures of Gaussians
  • 3.3. Periodic Variables
  • 3.3.1 Von Mises distribution
  • 3.4. The Exponential Family
  • 3.4.1 Sufficient statistics
  • 3.5. Nonparametric Methods
  • 3.5.1 Histograms
  • 3.5.2 Kernel densities
  • 3.5.3 Nearest-neighbours
  • Exercises
  • 4 Single-layer Networks: Regression
  • 4.1. Linear Regression
  • 4.1.1 Basis functions
  • 4.1.2 Likelihood function
  • 4.1.3 Maximum likelihood
  • 4.1.4 Geometry of least squares
  • 4.1.5 Sequential learning
  • 4.1.6 Regularized least squares
  • 4.1.7 Multiple outputs
  • 4.2. Decision theory
  • 4.3. The Bias–Variance Trade-off
  • Exercises
  • 5 Single-layer Networks: Classification
  • 5.1. Discriminant Functions
  • 5.1.1 Two classes
  • 5.1.2 Multiple classes
  • 5.1.3 1-of-K coding
  • 5.1.4 Least squares for classification
  • 5.2. Decision Theory
  • 5.2.1 Misclassification rate
  • 5.2.2 Expected loss
  • 5.2.3 The reject option
  • 5.2.4 Inference and decision
  • 5.2.5 Classifier accuracy
  • 5.2.6 ROC curve
  • 5.3. Generative Classifiers
  • 5.3.1 Continuous inputs
  • 5.3.2 Maximum likelihood solution
  • 5.3.3 Discrete features
  • 5.3.4 Exponential family
  • 5.4. Discriminative Classifiers
  • 5.4.1 Activation functions
  • 5.4.2 Fixed basis functions
  • 5.4.3 Logistic regression
  • 5.4.4 Multi-class logistic regression
  • 5.4.5 Probit regression
  • 5.4.6 Canonical link functions
  • Exercises
  • 6 Deep Neural Networks
  • 6.1. Limitations of Fixed Basis Functions
  • 6.1.1 The curse of dimensionality
  • 6.1.2 High-dimensional spaces
  • 6.1.3 Data manifolds
  • 6.1.4 Data-dependent basis functions
  • 6.2. Multilayer Networks
  • 6.2.1 Parameter matrices
  • 6.2.2 Universal approximation
  • 6.2.3 Hidden unit activation functions
  • 6.2.4 Weight-space symmetries
  • 6.3. Deep Networks
  • 6.3.1 Hierarchical representations
  • 6.3.2 Distributed representations
  • 6.3.3 Representation learning
  • 6.3.4 Transfer learning
  • 6.3.5 Contrastive learning
  • 6.3.6 General network architectures
  • 6.3.7 Tensors
  • 6.4. Error Functions
  • 6.4.1 Regression
  • 6.4.2 Binary classification
  • 6.4.3 multiclass classification
  • 6.5. Mixture Density Networks
  • 6.5.1 Robot kinematics example
  • 6.5.2 Conditional mixture distributions
  • 6.5.3 Gradient optimization
  • 6.5.4 Predictive distribution
  • Exercises
  • 7 Gradient Descent
  • 7.1. Error Surfaces
  • 7.1.1 Local quadratic approximation
  • 7.2. Gradient Descent Optimization
  • 7.2.1 Use of gradient information
  • 7.2.2 Batch gradient descent
  • 7.2.3 Stochastic gradient descent
  • 7.2.4 Mini-batches
  • 7.2.5 Parameter initialization
  • 7.3. Convergence
  • 7.3.1 Momentum
  • 7.3.2 Learning rate schedule
  • 7.3.3 RMSProp and Adam
  • 7.4. Normalization
  • 7.4.1 Data normalization
  • 7.4.2 Batch normalization
  • 7.4.3 Layer normalization
  • Exercises
  • 8 Backpropagation
  • 8.1. Evaluation of Gradients
  • 8.1.1 Single-layer networks
  • 8.1.2 General feed-forward networks
  • 8.1.3 A simple example
  • 8.1.4 Numerical differentiation
  • 8.1.5 The Jacobian matrix
  • 8.1.6 The Hessian matrix
  • 8.2. Automatic Differentiation
  • 8.2.1 Forward-mode automatic differentiation
  • 8.2.2 Reverse-mode automatic differentiation
  • Exercises
  • 9 Regularization
  • 9.1. Inductive Bias
  • 9.1.1 Inverse problems
  • 9.1.2 No free lunch theorem
  • 9.1.3 Symmetry and invariance
  • 9.1.4 Equivariance
  • 9.2. Weight Decay
  • 9.2.1 Consistent regularizers
  • 9.2.2 Generalized weight decay
  • 9.3. Learning Curves
  • 9.3.1 Early stopping
  • 9.3.2 Double descent
  • 9.4. Parameter Sharing
  • 9.4.1 Soft weight sharing
  • 9.5. Residual Connections
  • 9.6. Model Averaging
  • 9.6.1 Dropout
  • Exercises
  • 10 Convolutional Networks
  • 10.1. Computer Vision
  • 10.1.1 Image data
  • 10.2. Convolutional Filters
  • 10.2.1 Feature detectors
  • 10.2.2 Translation equivariance
  • 10.2.3 Padding
  • 10.2.4 Strided convolutions
  • 10.2.5 Multi-dimensional convolutions
  • 10.2.6 Pooling
  • 10.2.7 Multilayer convolutions
  • 10.2.8 Example network architectures
  • 10.3. Visualizing Trained CNNs
  • 10.3.1 Visual cortex
  • 10.3.2 Visualizing trained filters
  • 10.3.3 Saliency maps
  • 10.3.4 Adversarial attacks
  • 10.3.5 Synthetic images
  • 10.4. Object Detection
  • 10.4.1 Bounding boxes
  • 10.4.2 Intersection-over-union
  • 10.4.3 Sliding windows
  • 10.4.4 Detection across scales
  • 10.4.5 Non-max suppression
  • 10.4.6 Fast region CNNs
  • 10.5. Image Segmentation
  • 10.5.1 Convolutional segmentation
  • 10.5.2 Up-sampling
  • 10.5.3 Fully convolutional networks
  • 10.5.4 The U-net architecture
  • 10.6. Style Transfer
  • Exercises
  • 11 Structured Distributions
  • 11.1. Graphical Models
  • 11.1.1 Directed graphs
  • 11.1.2 Factorization
  • 11.1.3 Discrete variables
  • 11.1.4 Gaussian variables
  • 11.1.5 Binary classifier
  • 11.1.6 Parameters and observations
  • 11.1.7 Bayes’ theorem
  • 11.2. Conditional Independence
  • 11.2.1 Three example graphs
  • 11.2.2 Explaining away
  • 11.2.3 D-separation
  • 11.2.4 Naive Bayes
  • 11.2.5 Generative models
  • 11.2.6 Markov blanket
  • 11.2.7 Graphs as filters
  • 11.3. Sequence Models
  • 11.3.1 Hidden variables
  • Exercises
  • 12 Transformers
  • 12.1. Attention
  • 12.1.1 Transformer processing
  • 12.1.2 Attention coefficients
  • 12.1.3 Self-attention
  • 12.1.4 Network parameters
  • 12.1.5 Scaled self-attention
  • 12.1.6 Multi-head attention
  • 12.1.7 Transformer layers
  • 12.1.8 Computational complexity
  • 12.1.9 Positional encoding
  • 12.2. Natural Language
  • 12.2.1 Word embedding
  • 12.2.2 Tokenization
  • 12.2.3 Bag of words
  • 12.2.4 Autoregressive models
  • 12.2.5 Recurrent neural networks
  • 12.2.6 Backpropagation through time
  • 12.3. Transformer Language Models
  • 12.3.1 Decoder transformers
  • 12.3.2 Sampling strategies
  • 12.3.3 Encoder transformers
  • 12.3.4 Sequence-to-sequence transformers
  • 12.3.5 Large language models
  • 12.4. Multimodal Transformers
  • 12.4.1 Vision transformers
  • 12.4.2 Generative image transformers
  • 12.4.3 Audio data
  • 12.4.4 Text-to-speech
  • 12.4.5 Vision and language transformers
  • Exercises
  • 13 Graph Neural Networks
  • 13.1. Machine Learning on Graphs
  • 13.1.1 Graph properties
  • 13.1.2 Adjacency matrix
  • 13.1.3 Permutation equivariance
  • 13.2. Neural Message-Passing
  • 13.2.1 Convolutional filters
  • 13.2.2 Graph convolutional networks
  • 13.2.3 Aggregation operators
  • 13.2.4 Update operators
  • 13.2.5 Node classification
  • 13.2.6 Edge classification
  • 13.2.7 Graph classification
  • 13.3. General Graph Networks
  • 13.3.1 Graph attention networks
  • 13.3.2 Edge embeddings
  • 13.3.3 Graph embeddings
  • 13.3.4 Over-smoothing
  • 13.3.5 Regularization
  • 13.3.6 Geometric deep learning
  • Exercises
  • 14 Sampling
  • 14.1. Basic Sampling Algorithms
  • 14.1.1 Expectations
  • 14.1.2 Standard distributions
  • 14.1.3 Rejection sampling
  • 14.1.4 Adaptive rejection sampling
  • 14.1.5 Importance sampling
  • 14.1.6 Sampling-importance-resampling
  • 14.2. Markov Chain Monte Carlo
  • 14.2.1 The Metropolis algorithm
  • 14.2.2 Markov chains
  • 14.2.3 The Metropolis–Hastings algorithm
  • 14.2.4 Gibbs sampling
  • 14.2.5 Ancestral sampling
  • 14.3. Langevin Sampling
  • 14.3.1 Energy-based models
  • 14.3.2 Maximizing the likelihood
  • 14.3.3 Langevin dynamics
  • Exercises
  • 15 Discrete Latent Variables
  • 15.1. K-means Clustering
  • 15.1.1 Image segmentation
  • 15.2. Mixtures of Gaussians
  • 15.2.1 Likelihood function
  • 15.2.2 Maximum likelihood
  • 15.3. Expectation–Maximization Algorithm
  • 15.3.1 Gaussian mixtures
  • 15.3.2 Relation to K-means
  • 15.3.3 Mixtures of Bernoulli distributions
  • 15.4. Evidence Lower Bound
  • 15.4.1 EM revisited
  • 15.4.2 Independent and identically distributed data
  • 15.4.3 Parameter priors
  • 15.4.4 Generalized EM
  • 15.4.5 Sequential EM
  • Exercises
  • 16 Continuous Latent Variables
  • 16.1. Principal Component Analysis
  • 16.1.1 Maximum variance formulation
  • 16.1.2 Minimum-error formulation
  • 16.1.3 Data compression
  • 16.1.4 Data whitening
  • 16.1.5 High-dimensional data
  • 16.2. Probabilistic Latent Variables
  • 16.2.1 Generative model
  • 16.2.2 Likelihood function
  • 16.2.3 Maximum likelihood
  • 16.2.4 Factor analysis
  • 16.2.5 Independent component analysis
  • 16.2.6 Kalman filters
  • 16.3. Evidence Lower Bound
  • 16.3.1 Expectation maximization
  • 16.3.2 EM for PCA
  • 16.3.3 EM for factor analysis
  • 16.4. Nonlinear Latent Variable Models
  • 16.4.1 Nonlinear manifolds
  • 16.4.2 Likelihood function
  • 16.4.3 Discrete data
  • 16.4.4 Four approaches to generative modelling
  • Exercises
  • 17 Generative Adversarial Networks
  • 17.1. Adversarial Training
  • 17.1.1 Loss function
  • 17.1.2 GAN training in practice
  • 17.2. Image GANs
  • 17.2.1 CycleGAN
  • Exercises
  • 18 Normalizing Flows
  • 18.1. Coupling Flows
  • 18.2. Autoregressive Flows
  • 18.3. Continuous Flows
  • 18.3.1 Neural differential equations
  • 18.3.2 Neural ODE backpropagation
  • 18.3.3 Neural ODE flows
  • Exercises
  • 19 Autoencoders
  • 19.1. Deterministic Autoencoders
  • 19.1.1 Linear autoencoders
  • 19.1.2 Deep autoencoders
  • 19.1.3 Sparse autoencoders
  • 19.1.4 Denoising autoencoders
  • 19.1.5 Masked autoencoders
  • 19.2. Variational Autoencoders
  • 19.2.1 Amortized inference
  • 19.2.2 The reparameterization trick
  • Exercises
  • 20 Diffusion Models
  • 20.1. Forward Encoder
  • 20.1.1 Diffusion kernel
  • 20.1.2 Conditional distribution
  • 20.2. Reverse Decoder
  • 20.2.1 Training the decoder
  • 20.2.2 Evidence lower bound
  • 20.2.3 Rewriting the ELBO
  • 20.2.4 Predicting the noise
  • 20.2.5 Generating new samples
  • 20.3. Score Matching
  • 20.3.1 Score loss function
  • 20.3.2 Modified score loss
  • 20.3.3 Noise variance
  • 20.3.4 Stochastic differential equations
  • 20.4. Guided Diffusion
  • 20.4.1 Classifier guidance
  • 20.4.2 Classifier-free guidance
  • Exercises
  • Appendix B. Calculus of Variations
  • Appendix C. Lagrange Multipliers
  • Bibliography
  • Index