# Transformers & Attention (Data Science & ML)

> Attention mechanism, self-attention, multi-head attention, Transformer architecture, positional encoding

- 24 interview questions
- Senior
- [Interview Questions: Data Science & ML](https://sharpskill.dev/en/technologies/data-science/interview-questions.md)

## 1. What is the main advantage of the attention mechanism over RNNs for sequence processing?

**Answer**

The attention mechanism allows direct access to any position in the sequence, eliminating the sequential bottleneck of RNNs. Unlike RNNs that must propagate information step by step, attention computes direct connections between all positions, enabling massive parallelization and capturing long-range dependencies without gradient degradation.

## 2. In the attention mechanism, what do the Query (Q), Key (K) and Value (V) vectors represent?

**Answer**

Query represents what the token is looking for, Key represents what each token can offer as a match, and Value contains the information to retrieve. The attention score is computed between Q and K to determine relative importance, then used to weight the V. This analogy comes from information retrieval systems where a query is compared to keys to retrieve values.

## 3. What is the formula for scaled dot-product attention and why divide by the square root of dk?

**Answer**

The formula is Attention(Q,K,V) = softmax(QK^T / sqrt(dk)) * V. Dividing by sqrt(dk) is crucial because dot products of high-dimensional vectors tend to have large magnitudes, pushing softmax into regions with very small gradients. This normalization maintains stable variance of attention scores, ensuring efficient learning.

## 21 more questions available

- What is the fundamental difference between attention and self-attention?
- Why use multi-head attention rather than a single attention head?

Sign up for free: https://sharpskill.dev/en/login

## Other Data Science & ML interview topics

- [Python Basics](https://sharpskill.dev/en/technologies/data-science/interview-questions/python-basics.md): 25 questions, Junior
- [Python Object-Oriented Programming](https://sharpskill.dev/en/technologies/data-science/interview-questions/python-oop.md): 20 questions, Junior
- [Python Data Structures](https://sharpskill.dev/en/technologies/data-science/interview-questions/python-data-structures.md): 20 questions, Junior
- [Git Fundamentals](https://sharpskill.dev/en/technologies/data-science/interview-questions/git-fundamentals.md): 18 questions, Junior
- [SQL Basics](https://sharpskill.dev/en/technologies/data-science/interview-questions/sql-basics.md): 20 questions, Junior
- [NumPy Fundamentals](https://sharpskill.dev/en/technologies/data-science/interview-questions/numpy-fundamentals.md): 22 questions, Junior
- [Pandas Basics](https://sharpskill.dev/en/technologies/data-science/interview-questions/pandas-basics.md): 22 questions, Junior
- [Jupyter & Google Colab](https://sharpskill.dev/en/technologies/data-science/interview-questions/jupyter-colab.md): 16 questions, Junior
- [SQL Joins & Advanced Queries](https://sharpskill.dev/en/technologies/data-science/interview-questions/sql-joins-advanced.md): 22 questions, Mid-Level
- [Advanced Pandas](https://sharpskill.dev/en/technologies/data-science/interview-questions/pandas-advanced.md): 24 questions, Mid-Level
- [Visualization with Matplotlib & Seaborn](https://sharpskill.dev/en/technologies/data-science/interview-questions/matplotlib-seaborn.md): 20 questions, Mid-Level
- [Interactive Visualizations with Plotly](https://sharpskill.dev/en/technologies/data-science/interview-questions/plotly-interactive.md): 18 questions, Mid-Level
- [Descriptive Statistics](https://sharpskill.dev/en/technologies/data-science/interview-questions/statistics-descriptive.md): 20 questions, Mid-Level
- [Inferential Statistics](https://sharpskill.dev/en/technologies/data-science/interview-questions/statistics-inferential.md): 24 questions, Mid-Level
- [Web Scraping](https://sharpskill.dev/en/technologies/data-science/interview-questions/web-scraping.md): 18 questions, Mid-Level
- [BigQuery & Cloud Data](https://sharpskill.dev/en/technologies/data-science/interview-questions/bigquery-cloud.md): 18 questions, Mid-Level
- [Feature Engineering](https://sharpskill.dev/en/technologies/data-science/interview-questions/feature-engineering.md): 22 questions, Mid-Level
- [Supervised ML: Regression](https://sharpskill.dev/en/technologies/data-science/interview-questions/ml-supervised-regression.md): 24 questions, Mid-Level
- [Supervised ML: Classification](https://sharpskill.dev/en/technologies/data-science/interview-questions/ml-supervised-classification.md): 24 questions, Mid-Level
- [Decision Trees & Ensembles](https://sharpskill.dev/en/technologies/data-science/interview-questions/ml-trees-ensembles.md): 24 questions, Mid-Level
- [Unsupervised ML](https://sharpskill.dev/en/technologies/data-science/interview-questions/ml-unsupervised.md): 22 questions, Mid-Level
- [ML Pipelines & Validation](https://sharpskill.dev/en/technologies/data-science/interview-questions/ml-pipelines-validation.md): 22 questions, Mid-Level
- [Time Series & Forecasting](https://sharpskill.dev/en/technologies/data-science/interview-questions/time-series-forecasting.md): 22 questions, Mid-Level
- [Deep Learning Fundamentals](https://sharpskill.dev/en/technologies/data-science/interview-questions/deep-learning-fundamentals.md): 24 questions, Senior
- [TensorFlow & Keras](https://sharpskill.dev/en/technologies/data-science/interview-questions/tensorflow-keras.md): 22 questions, Senior
- [CNN & Image Classification](https://sharpskill.dev/en/technologies/data-science/interview-questions/cnn-image-classification.md): 24 questions, Senior
- [RNN & Sequences](https://sharpskill.dev/en/technologies/data-science/interview-questions/rnn-sequences.md): 22 questions, Senior
- [NLP & Hugging Face](https://sharpskill.dev/en/technologies/data-science/interview-questions/nlp-huggingface.md): 24 questions, Senior
- [GenAI & LangChain](https://sharpskill.dev/en/technologies/data-science/interview-questions/genai-langchain.md): 24 questions, Senior
- [MLOps & Deployment](https://sharpskill.dev/en/technologies/data-science/interview-questions/mlops-deployment.md): 24 questions, Senior

---

Source: SharpSkill (https://sharpskill.dev), tech interview preparation for your real stack.
HTML version of this page: https://sharpskill.dev/en/technologies/data-science/interview-questions/transformers-attention