HPC - Algorithms and Applications - Winter 16

Term

Winter 16/17

Lecturer

Prof. Dr. Michael Bader

Time and Place

lecture: Mon, 14-16 in room MI 02.07.023 (intro lecture on Wed, Oct 19, in the tutorial slot)
tutorial: Wed, 12-14, in room MI 02.07.023 (roughly every other week)

Audience

Elective topic in Informatics Bachelor/Master: students in mathematics or in any science or engineering discipline are welcome!

Tutorials

Alexander Pöppl, Valeriy Khakhutskyy

Exam

Fri, Feb 24, 13.30 in Interim 2

Semesterwochenstunden / ECTS Credits

3 SWS (2V + 1Ü) / 4 ECTS

TUMonline

TUMOnline

---

Announcements

• A Q&A session concerning the exam (lectures and tutorials) will take place on Feb 15, 2017 (Wed), 12:15-13:45, in seminar room MI 02.07.023 (tutorial slot)
• there will be no lecture on Mon, Oct 17 (opening day)
• the intro lecture is moved to Wed, Oct 19 (tutorial slot)

Content

The lecture will have a focus on parallel algorithms and implementation techniques in the field of numerical simulation and high performance computing, such as:

• linear algebra problems on dense and sparse matrices
• simulation on structured and unstructured meshes
• particle-based simulations (with long-range and short-range interactions)
• spectral methods (parallel FFT and related algorithms)
• Monte Carlo and statistical methods

(a.k.a. the seven dwarfs of HPC).

The accompanying tutorials will include practical assignments, and will concentrate on the programming of GPU and accelerator platforms.

Lecture Material

Lecture slides will be published/updated here after the lessons. See also the lecture from winter term 2015/16.

• Oct 19: Intro
• Oct 31, Nov 7: Fundamentals - Parallel Architectures, Models, and Languages
  • read the related paper: Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures (technical report by Williams et al.; published in Communications of the ACM 52 (4), 2009, p.65-76)
  • further info: Cannon's Algorithm as MPI examples - mpi_cannon.c (unsafe send/receive), mpi_cannon_sr.c (using MPI_Sendrecv), mpi_cannon_nbl.c (non-blocking communication)
• Oct 31, Nov 7, 21: Dwarf No. 1 - Dense Linear Algebra
  • further info: The GotoBLAS/BLIS Approach to Optimizing Matrix-Matrix Multiplication - Step-by-Step (by R. van de GeijnI)
  • read the related paper: article by Elmroth et al. in SIAM Review (access via TUM eAccess, if necessary)
  • chapter In-core dense matrices of the ScaLAPACK User's Guide
• Nov 14: Dwarf No. 2 - Sparse Linear Algebra: Application example (page rank) and data structures
  • further info: The PageRank Citation Ranking: Bringing Order to the Web (technical report by L. Page, L. Brin, R. Motwani, T. Winograd, 1999)
  • further info: The Anatomy of a Large-Scale Hypertextual Web Search Engine by S. Brin and L. Page (preprint; paper appeared in Computer Networks and ISDN Systems 30 (1-7), 1998)
  • further info: PageRank Beyond the Web - article in SIAM Review 57(3) by David Gleich
• Nov 28, Dec 5: Parallel Sparse Matrix-Vector Multiplication - Parallel SpMV, Cartesian Distribution
  • lecture material accompanying the book by R. Bisseling (compare the slides psc4_3.pdf, psc4_4.pdf, psc4_5.pdf, and psc4_6.pdf)
• Dec 5, 12, 19: Dwarf No. 5 - Structured Grids
  • articles by M. Frigo and V. Strumpen:
    Cache oblivious stencil operations (preprint);
    The memory behavior of cache oblivious stencil operations (preprint can be found via Google)
  • article by K. Datta et al. in SIAM Review (preprint)
• Jan 9, 16: Dwarf No. 6 - Unstructured Grids and Partitioning
  • additional material: article by Karypis and Kumar: A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs
  • additional material: article by Hendrickson and Leland: A Multilevel Algorithm for Partitioning Graphs (this article was awarded the "Test of Time Award" at the 2014 Supercomputing Conference, SC14)
• Jan 23, 30: Dwarf No. 4: N-body methods and Barnes-Hut and Fast Multipole algorithm
  • article by Barnes & Hut in Nature (article can be accessed via TUM ebib-access)
  • article by C.R. Anderson (Fast Multipole without multipoles (article can be accessed via TUM ebib-access)
  • article on Fast Multipole methods by C.R. Anderson
  • Maple worksheet: twobody.mw (also as PDF)
• Jan 30, Feb 6: Dwarf No. 5 revisited: Structured Grids and Space-filling Curves (contents that were only presented in these slides will not be part of the exam)
  • IPython Notebook worksheets: Hilbert_Plotter.ipynb, sfc_hilbert_plotter_adp.ipynb
  • Maple worksheets: hilbert_adap.mw (also as PDF);

Tutorials

Roughly every second week a two hour tutorial will take place (details at page top; days and time will be announced in TUMonline and in the lectures). The assignments and their solutions will be gradually posted here.

Date	Slides	Worksheet	Source	Source (solution)
Oct 24th	Organizational remarks Introduction to CUDA	-	-	-
Nov 11th	Introduction to CUDA	Worksheet 1	Exercise 1	included in Exercise 2
Nov 23rd	Dense LA in CUDA (part 2)	Worksheet 2	Template (Lecture) Template (Homework)	Solution 2
Nov 30th	Sparse LA in CUDA	Worksheet 3	Template (Lecture) Template (Homework)	Memory Accesses; Solution included in Exercise 4
Dec 14th	Solving the heat equation with CUDA	Worksheet 4	Template (Task 1.1a) Template (Task 1.1b) Template (Homework)	Solution 4
Jan 11th	The Shallow Water Equations and CUDA	Worksheet 5	Exercise 5	included in Exercise 6
Jan 25th	Further topics on SWE and CUDA	Worksheet 6	Exercise 6	-

Exam

End-Term Exam

• Written Exam on Feb 24th, 13:30-15:15 in Interim 2 (black building in front of Informatics building)
  • please be in the lecture room in time (by 13:15); the exam will start on 13:30, at the latest, and there will be announcements before the start!
• Helping material: One sheet of A4 paper (two-sided) with hand-written notes on it.
• The exam will extend over all topics discussed in the lectures and tutorials:
  • approx. 30% of the questions will deal with questions related to the tutorials; basic knowledge about GPU programming with CUDA is thus necessary

Repeat Exam

• Exam Review: on appointment in the week from Apr 24; please contact Alexander Pöppl
• The same rules as for the written exam will apply (see below)
• Written exam on Mon, Apr 10th, 2017, from 10:30 (MI Lecture Hall 2)
  • Please try to be in front of the room by 10:15 as the working time will start at 10.30. Announcements will be made prior to 10:30.
• Helping material: One sheet of A4 paper (two-sided) with hand-written notes on it.
• The exam will extend over all topics discussed in the lectures and tutorials:
  • approx. 30% of the questions will deal with questions related to the tutorials; basic knowledge about GPU programming with CUDA is thus required

Exam Preparation

• the following worksheet contains some example questions (with solutions) from previous exams:
  • exam questions with example solutions
  • note that this collection of exercises does not reflect the extent of assignments in the exam
  • note that the contents of the lecture may have slightly changed compared to previous years, such that exercises can have a slightly different focus

Literature and Online Material

• R.H. Bisseling: Parallel Scientific Computing - A structured approach using BSP and MPI, Oxford University Press, 2004.
  • Course notes on Rob Bisseling's lecture on Parallel Algorithms (based on the text book)
• V. Eijkhout: Introduction to High-Performance Scientific Computing (textbook, available as PDF on the website)
• T.G. Mattson, B.A. Sanders, B.L. Massingill: Patterns for Parallel Programming, Addison-Wesley, 2005
• G. Hager, G. Wellein: Introduction to High Performance Computing for Scientists and Engineers, Chapman & Hall/CRC Computational Science, 2010

(all available as ebooks from TUM library)

Books on CUDA

• D.B. Kirk, W.W. Hwu: Programming Massively Parallel Processors - A Hands-on Approach, Morgan-Kaufman, 2nd edition, 2013
• J. Sanders, E. Kandrot: CUDA by Example, Addison-Wesley, 2011

(both available as ebooks from TUM library)

Prerequisites

Helpful, but not strictly required is knowledge in:

• basics of numerical methods (e.g.: lecture IN0019 Numerical Programming or similar)
• basics of parallel programming (lecture Parallel Programming, HPC - Programming Paradigms and Scalability, or similar)

Most important is a certain interest in problems from scientific computing and numerical simulation!