📐 Vector Similarity: Inner Product vs Cosine Similarity

Inner Product (Dot Product)

Mathematical Definition:

A · B = Σ(A_i * B_i) = ||A|| * ||B|| * cos(θ)

What it measures:

• Combined effect of magnitude AND direction
• Sensitive to vector lengths
• Larger when vectors are both long and aligned

Cosine Similarity

Mathematical Definition:

cos(θ) = (A · B) / (||A|| * ||B||)

What it measures:

• Pure direction similarity (angle between vectors)
• Ignores vector magnitudes
• Range: [-1, 1] where 1 = same direction, -1 = opposite directions

🎯 AI Applications in Your Document:

Word Embeddings (NLP)

• Cosine Similarity used to find semantically similar words
• Words with similar meanings have similar vector directions
• Example: king - man + woman ≈ queen

Recommendation Systems

• Cosine Similarity between user and item vectors
• Finds users with similar preferences regardless of rating scale

Computer Vision

• Inner Product in CNN filters for feature detection
• Measures how much an image patch matches a filter pattern

🧩 Matrix Decompositions: Breaking Down Complexity

Why Decompose Matrices?

Matrix decompositions break complex matrices into simpler, interpretable components:

1. Singular Value Decomposition (SVD)

A = U × Σ × Vᵀ

• U: Left singular vectors (orthogonal basis for column space)
• Σ: Singular values (importance scores - diagonal matrix)
• Vᵀ: Right singular vectors (orthogonal basis for row space)

2. QR Decomposition

A = Q × R

• Q: Orthogonal matrix (orthonormal columns)
• R: Upper triangular matrix

3. Eigen Decomposition

A = Q × Λ × Q⁻¹

• Λ: Diagonal matrix of eigenvalues
• Q: Matrix of eigenvectors

4. LU Decomposition

A = L × U

• L: Lower triangular matrix
• U: Upper triangular matrix

🎯 AI Applications from Your Document:

SVD Applications:

• Image Compression: Keep only top k singular values
• Recommendation Systems: Collaborative filtering (Netflix Prize)
• NLP: Latent Semantic Analysis (LSA) for document similarity
• PCA: SVD is used to compute principal components

QR Decomposition:

• Solving Linear Systems: More numerically stable
• Least Squares Problems: Linear regression solutions
• Eigenvalue Computation: QR algorithm

Eigen Decomposition:

• PCA: Find directions of maximum variance
• Spectral Clustering: Graph partitioning using eigenvectors
• Markov Chains: Stationary distributions from eigenvectors

🔍 PCA & SVD: Dimensionality Reduction

Principal Component Analysis (PCA)

Mathematical Process:

1. Center the data: Subtract mean from each feature
2. Compute covariance matrix: Shows how features vary together
3. Eigen decomposition: Find eigenvectors (principal directions) and eigenvalues (variance explained)
4. Project data: Transform to new coordinate system

PCA finds:

• PC1: Direction of maximum variance
• PC2: Next orthogonal direction of maximum remaining variance
• And so on...

Singular Value Decomposition (SVD)

A = U × Σ × Vᵀ

• Directly gives principal components without explicit covariance calculation
• More numerically stable
• Works on any matrix (not just square)

🎯 AI Applications from Your Document:

PCA Applications:

• Data Visualization: Project high-D data to 2D/3D
• Feature Reduction: Remove redundant features
• Noise Reduction: Keep only important components
• Image Compression: Represent images with fewer components

SVD Applications:

• Recommendation Systems: Matrix factorization for collaborative filtering
• NLP: Latent Semantic Analysis for document similarity
• Image Processing: Low-rank approximations for compression
• Computer Vision: Structure from motion, facial recognition

🎯 Support Vector Machines (SVM)

Mathematical Foundation

SVM finds the optimal hyperplane that maximizes the margin between classes:

Hyperplane Equation:

w·x + b = 0

• w: Weight vector (normal to hyperplane)
• b: Bias term
• x: Input features

Margin Maximization:

• Finds hyperplane that maximizes distance to nearest points (support vectors)
• These support vectors define the decision boundary

Kernel Trick:

For non-linearly separable data, SVM uses kernels to project data to higher dimensions:

• Linear: w·x + b
• Polynomial: (γ·x·x' + r)^d
• RBF: exp(-γ·||x - x'||²)

🎯 AI Applications from Your Document:

Classification Tasks:

• Text Classification: Spam detection, sentiment analysis
• Image Recognition: Handwritten digit recognition
• Bioinformatics: Protein classification, cancer detection

Key Linear Algebra Concepts:

• Dot Products: Kernel computations rely on inner products
• Vector Norms: Distance calculations for margins
• Hyperplanes: Decision boundaries in high-dimensional spaces

Advantages:

• Effective in high-dimensional spaces
• Memory efficient (uses only support vectors)
• Versatile (different kernels for different problems)