PyTorch Interview Questions and Answers

Answer: Open-source machine learning framework developed by Facebook

PyTorch is an open-source machine learning framework developed by Facebook AI Research. It provides tensor computation and deep learning capabilities.

Answer: Training across multiple GPUs or machines

Distributed training trains model across multiple GPUs or machines. Speeds up training for large models/datasets. Use torch.distributed.

Answer: Multi-dimensional array similar to NumPy ndarray

A tensor is a multi-dimensional array, similar to NumPy ndarray but can run on GPUs. It is the fundamental data structure in PyTorch.

Answer: Simple multi-GPU training wrapper

DataParallel wraps model for simple multi-GPU training. Replicates model on each GPU, splits batch. Single-machine only. DistributedDataParallel preferred.

Answer: More efficient multi-GPU/multi-node training

DDP is more efficient than DataParallel. One process per GPU, gradient synchronization via ring-allreduce. Supports multi-node. Recommended for multi-GPU.

Answer: torch.tensor([1, 2, 3])

Use torch.tensor([1, 2, 3]) to create a tensor from a Python list. torch.Tensor() is also available but torch.tensor() is preferred.

Answer: 2D convolutional layer for images

nn.Conv2d applies 2D convolution over input signal (images). Fundamental for CNNs. Parameters: in_channels, out_channels, kernel_size.

Answer: Accumulating gradients over multiple batches before updating

Gradient accumulation accumulates gradients over multiple mini-batches before optimizer.step(). Simulates larger batch size with limited memory.

Answer: Reducing precision (float32 to int8) for faster inference

Quantization reduces model precision (typically float32 to int8) reducing size and increasing speed. May slightly decrease accuracy.

Answer: Max pooling layer reducing spatial dimensions

MaxPool2d applies max pooling, reducing spatial dimensions by taking maximum value in each window. Used for downsampling in CNNs.

Answer: tensor.size() or tensor.shape

Use tensor.size() or tensor.shape to get tensor dimensions. Both return the same result. tensor.size() returns torch.Size object.

Answer: TorchScript compiler for optimization and deployment

torch.jit compiles PyTorch models to TorchScript for optimization and deployment. Use torch.jit.script() or torch.jit.trace().

Answer: Automatic gradient computation system

Autograd is PyTorch's automatic differentiation engine that powers neural network training by automatically computing gradients.

Answer: Randomly zeros elements during training for regularization

Dropout randomly zeros elements with probability p during training. Prevents overfitting. Automatically disabled in eval mode.

Answer: Batch normalization for 4D inputs

BatchNorm2d applies batch normalization over 4D input (batch, channels, height, width). Stabilizes training and speeds convergence.

Answer: script analyzes code, trace records operations

jit.script analyzes Python code directly (handles control flow). jit.trace records operations on example input (can't handle dynamic control flow).

Answer: Adjusts learning rate during training

Learning rate scheduling adjusts learning rate during training. Common strategies: step decay, exponential decay, cosine annealing.

Answer: Removing unimportant weights/neurons to reduce size

Pruning removes unimportant weights or entire neurons/filters. Reduces model size and inference time. May require fine-tuning.

Answer: Training small model to mimic large model

Knowledge distillation trains smaller student model to mimic larger teacher model. Student learns from teacher's soft predictions.

Answer: Exports PyTorch models to ONNX format

torch.onnx exports PyTorch models to ONNX (Open Neural Network Exchange) format for interoperability with other frameworks and deployment.

Answer: Adam optimizer using adaptive learning rates

Adam (Adaptive Moment Estimation) uses adaptive learning rates for each parameter. Generally works well with default settings.

Answer: Limits gradient magnitude to prevent exploding gradients

Gradient clipping limits gradient magnitude. Prevents exploding gradients. Use torch.nn.utils.clip_grad_norm_() or clip_grad_value_().

Answer: Model serving framework for production deployment

TorchServe is PyTorch's model serving framework. Handles model management, versioning, metrics, RESTful APIs for production deployment.

Answer: Custom gradient computation by extending Function

Custom autograd Function extends torch.autograd.Function, implementing forward() and backward() for custom operations with gradients.