Introduction to Neural Networks

This tutorial covers the basics of neural networks (aka “deep learning”), which is a technique within machine learning that tends to outperform other techniques when dealing with a large amount of data.

🎯 Goals:
- Introduce deep learning fundamentals through hands-on activities
- Provide the necessary background for the rest of the project
📖 Definitions:
- Artificial intelligence (AI) is a set of approaches to solving complex problems by imitating the brain’s ability to learn.
- Machine learning (ML) is the field of study that gives computers the ability to learn without being explicitly programmed (i.e. learning patterns instead of writing down rules.) Arguably, machine learning is now a subfield of AI.
🤔 Recap: Last week, we learned about using linear regression to predict the sale price of a house. We fit a function to the dataset:
- Input: above ground square feet
- Output: sale price
- Function type: linear
- Loss function: mean squared error
- Optimization algorithm: stochastic gradient descent
This week, we’ll work on a “classification” problem, which means that we have a category label for each data point, and we fit a function that can categorize inputs.

The MNIST dataset contains thousands of examples of handwritten numbers, with each digit labeled 0-9.

We’ll start with the MNIST problem in this notebook:

📓 Fitting MNIST with a multi-layer perceptron (MLP)

Next week, we’ll learn about other types of neural networks.

References:

Here are some recommendations for further reading:

Citation

BibTeX citation:

@online{foreman2025,
  author = {Foreman, Sam and Foreman, Sam and Ngom, Marieme and Zheng,
    Huihuo and Lusch, Bethany and Childers, Taylor},
  title = {Introduction to {Neural} {Networks}},
  date = {2025-07-15},
  url = {https://saforem2.github.io/hpc-bootcamp-2025/01-neural-networks/0-intro/},
  langid = {en}
}

For attribution, please cite this work as:

Foreman, Sam, Sam Foreman, Marieme Ngom, Huihuo Zheng, Bethany Lusch, and Taylor Childers. 2025. “Introduction to Neural Networks.” July 15, 2025. https://saforem2.github.io/hpc-bootcamp-2025/01-neural-networks/0-intro/.

--- title: "Introduction to Neural Networks" description: "A beginner's guide to understanding neural networks, their architecture, and how they function." categories: - ai - hpc date: 2025-07-15 date-modified: last-modified format: html: default gfm: toc: true author: - id: sf name: Sam Foreman orcid: 0000-0002-9981-0876 email: foremans@anl.gov affiliation: - name: '[ANL](https://www.anl.gov/)' city: Lemont state: IL url: https://alcf.anl.gov/about/people/sam-foreman - id: mn name: Marieme Ngom # orcid: 0000-0002-9981-0876 email: mngom@anl.gov affiliation: - name: '[ANL](https://www.anl.gov/)' city: Lemont state: IL url: https://alcf.anl.gov/about/people/marieme-ngom - id: hz name: Huihuo Zheng # orcid: 0000-0002-9981-0876 email: huihuo.zheng@anl.gov affiliation: - name: '[ANL](https://www.anl.gov/)' city: Lemont state: IL url: https://alcf.anl.gov/about/people/huihuo-zheng - id: bl name: Bethany Lusch email: blusch@anl.gov orcid: 0000-0002-9521-9990 # orcid: 0000-0002-9981-0876 affiliation: - name: '[ANL](https://www.anl.gov/)' city: Lemont state: IL url: https://alcf.anl.gov/about/people/bethany-lusch - id: tc name: Taylor Childers orcid: 0000-0002-0492-613X email: jchilders@anl.gov affiliation: - name: '[ANL](https://www.anl.gov/)' city: Lemont state: IL url: https://alcf.anl.gov/about/people/taylor-childers ---  This tutorial covers the basics of neural networks (aka "deep learning"), which is a technique within machine learning that tends to outperform other techniques when dealing with a large amount of data. - 🎯 **Goals**: - Introduce deep learning fundamentals through hands-on activities - Provide the necessary background for the rest of the project - 📖 **Definitions**: - _Artificial intelligence_ (AI) is a set of approaches to solving complex problems by imitating the brain's ability to learn. - _Machine learning_ (ML) is the field of study that gives computers the ability to learn without being explicitly programmed (i.e. learning patterns instead of writing down rules.) Arguably, machine learning is now a subfield of AI. - 🤔 **Recap**: Last week, we learned about using linear regression to predict the sale price of a house. We fit a function to the dataset: - Input: above ground square feet - Output: sale price - Function type: linear - Loss function: mean squared error - Optimization algorithm: stochastic gradient descent This week, we'll work on a "classification" problem, which means that we have a category label for each data point, and we fit a function that can categorize inputs. The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) contains thousands of examples of handwritten numbers, with each digit labeled 0-9. ::: {#fig-mnist-example} ![MNIST Task](../images/mnist_task.png) MNIST Data Sample ::: We'll start with the MNIST problem in this notebook: [📓 Fitting MNIST with a multi-layer perceptron (MLP)](../1-mnist/index.qmd) Next week, we'll learn about other types of neural networks. ## __References:__ - Here are some recommendations for further reading: - [tensorflow.org tutorials](https://www.tensorflow.org/tutorials) - [keras.io tutorials](https://keras.io/examples/) - [CS231n: Convolutional Neural Networks for Visual Recognition](http://cs231n.stanford.edu/) - [Deep Learning Specialization, Andrew Ng](https://www.coursera.org/specializations/deep-learning?utm_source=deeplearningai&utm_medium=institutions&utm_campaign=WebsiteCoursesDLSTopButton) - [PyTorch Challenge, Udacity](https://www.udacity.com/facebook-pytorch-scholarship) - [Deep Learning with Python](https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438) - [Keras Blog](https://blog.keras.io/) - [Hands-on ML book](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/) with [notebooks](https://github.com/ageron/handson-ml2).