This week you will..

Quick overview of the projects, report from Hackathon, discussion about batch normalization and hyperparameters search, first assignment using Tensorflow to create a small neural network. Small discussion about Tensorflow ideas and modalities and difference between 1.0 and 2.0 versions.

Learning Resources

Slides

The Deep Learning course will guide you through the mathematics background of machine learning approaches. We will start from a simple neural network and go through the different components of a network to understand and be able to create your own project.

The Objective of the Course

The aim of this course is to develop a deeper understanding of how and why neural network work. The first part will be devoted to understanding and implementing the basic behind most of the neural network approaches, the forward- and back-propagation, loss function, optimization, training, hyper-parameters tuning and analysis.

To gain a better understanding, we will implement those part in python (mostly using the numpy library). These methods already exists in popular frameworks (like Tensorflow or Pytorch, to cite a few), but using them without knowledge may be confusing.

During the course, we will have weekly discussion to deeper our understanding of the subject and also you will work on your own project.

Requirements and Motivation

In order to get the best out of this course, some previous knowledge is required. We expect the participant to have an understanding about the fundaments of mathematics (not being afraid of derivatives), linear algebra (mostly matrix multiplication) and python programming.

Based on the past semester, the estimated time is around 5 hours a week, ranging usually from 3 to 8 depending on the week's material. The project will start after 4 weeks of the course and will take some additional hours. However, assignment will decrease in the end of the course to leave you space for the project. Be sure to allocate enough time to manage to get through the whole course. If in doubt, ask us for advice.

The Project

Groups of students should be formed to work on a project. The project idea can come from the student, from a template or proposed from us. The project is needed in order to finish the course, and a final presentation will be given in the last week of the course.

The Course Material

The course will be held weekly and will constitute of an online session of 1 hour and a half. The material and slides for each session are found in each week's page.

ECTS

For further details about Certificates and ECTS please refer to the following page:

This week you will...

• Receive an introduction about the course and the people in it. A short overview of the course, contents and how it will work.

• Information about accounts, forum and contacts are provided.

Learning Resources

Until next week you should...

• Register in the Deep Learning Channel in the Mattermost Chat

• Register on Coursera and start the Neural Networks and Deep Learning course,

• Finish the first two weeks of the course

• Do the Programming Assignment on Logistic Regression

• Do the Programming Assignment on Python Numpy

The conditions to be met in order to receive a Certificate of Achievement and the ECTS are:

• Attendance to at least 80% of the classes (it is allowed to miss maximum 2 times)

• Delivery of the project with the needed documentation.

Attendance:

Since the weekly session will be on Zoom, please use your full name when you join the Zoom Meeting. The full name should be the same that you used to register at edu.opencampus.sh, because we have an automatic check.

Projects:

Check the Projects section to learn more about the projects.

Coursera:

Each weekly session is complemented with the videos and homework from the Coursera courses. Going through the video and doing the assignment allows you to learn and understand each session, so it is required for the course.

However, Coursera is indipendent from us and the completion of the Coursera assignment is NOT needed for the Opencampus Certificate. Completing all assignment will give you the Coursera Certificates (which is different)

Before the first class you should ...

• Register yourself in the Opencampus Mattermost Chat

• and for the , and enroll at least in the first course .

This week you will..

Get practical hints about initialization and regularization techniques to avoid overfitting and improving the training of a neural network.

Learning Resources

Slides

Until next week you should...

• Form the groups, decide the project and communicate it to the teacher.

• Do the Programming Assignment

This week you will..

See Mini-batches, Momentum, RMSProp and AdamOptimizer: an overview of optimization algorithm to train faster neural networks.

Learning Resources

Slides

This week you will...

• Check if everything worked with the tools we started using

• Have the first session with a small quiz and round of discussion.

• Discuss about python environment, dot product against element wise multiplication,

• Do you first exercise session training a small neural network recognizing cats!

Learning Resources

Slides

Useful (external) Resourcers

Until next week you should...

This week you will...

check weights initialization in the training and notebook example of planar data classification changing the number of hidden unit in a shallow network - only 1 hidden layer.

Learning Resources

Slides

Useful (external) Resources

Until next week you should...

• Finish both assignments

• Think about your project: prepare an idea and find other people willing to collaborate (there is time also next week, but please start)

This week you will...

• Explore deep neural network.

• First example of generalizing a neural network with L layers.

• Discussion and choice about the projects.

Learning Resources

Projects

Until next week you should...

• Create a group for your project!

• Do the Programming Assignments: Initialization, Regularization and Gradient Checking

Before the course start, you should be familiar with:

• our Opencampus.sh Mattermost Chat

• the Coursera platform and the Deep Learning Specialization

• the Tensorflow framework

What would be very beneficial for this course (but not mandatory, you will learn it anyway during the course):

• knowledge of numpy and opencv python packages

• knowing how to work with images in python

• knowing what a convolution is

This week you will..

• Get an overview about the different architectures

• Discuss the status of your project

This session is entirely dedicated to the presentation of the final project from the students. Schedule and timing will be decided and published during the course.

The conditions to be met in order to receive a Certificate of Achievement and the ECTS are:

1. Missing the weekly session maximum 2 times.

2. Presenting and uploading the project with the needed documentation.

Attendance:

Projects:

Check the Projects section to learn more about the projects.

Coursera:

Each weekly session is complemented with the videos and homework from the Coursera courses. Going through the video and doing the assignment allows you to learn and understand each session, so it is required for the course.

However, Coursera is indipendent from us and the completion of the Coursera assignment is NOT needed for the Opencampus Certificate. Completing all assignment will give you the Coursera Certificates (which is different)

This week you will..

• Discuss about CNN models

• Finalize groups and set objective for the project

• Check out the homework

Learning Resources

Until next week you should

This week you will..

• have a short talk about how to structure your idea for the project

• start to form the groups for the project

• discuss about advantages and limitations of CNN

• take a look at the first homework

Slides

Until next week you should

• check additional material on convolutional models

This week you will..

• Guidelines for the projects presentations, suggestions on what to put on them, baseline and human performances.

• Talk about how to structure the training, test and validation set, and more general on how to structure the whole project.

• What about using transfer learning, end-to-end approaches, divide the problem into smaller subproblems, or using multi-tasking? Sometimes the problem can be seen from another perspective.

• Since there were no programming assignment for this week, walkthrough an LSTM tutorial on time series.

Learning Resources

Slides

This week you will..

• Discuss about residual networks

• Discuss about training practices

Learning Resources

Until next week you should

• Check additional material on Transfer Learning

This week you will..

• Discuss about transfer learning

• Have a small internal discussion about your project and present us some insights

Learning Resources

Until next week you should

• Check out the videos for the third week of the coursera course

• Finish the homework on Car detection with YOLO

This week you will..

• Discuss the car detection algorithms

• Think about other detection/localization applications

Learning Resources

Until next week you should

• Prepare a small presentation about the state of your project.

• Finish the homework about Image Segmentation using U-Net

The presentation should contain:

• goal of the project

• state of the art (was it already done? If yes, how? If no, why?)

• your approach

• your dataset

• your problems (if any :D )

The conditions to be met in order to receive a Certificate of Achievement (and ECTS) are:

• Attendance to at least 80% of the classes (it is allowed to miss a maximum 2 classes)

• Delivery of a machine learning project based on the Hugging Face API and including a corresponding documentation.

Attendance:

If you attend via Zoom, please make sure to use your full name, which should be the same that you used to register at edu.opencampus.sh. Otherwise your attendance will not be recorded!

Projects:

Check the Projects section to learn more about the projects.

This week you will

• get an introduction to the course and possible projects,

• have a round of presentation

• get an overview of the resources for the course

• brainstorm ideas for the projects

Slides

Until next week you should

• Check the videos of the first week of the Convolutional Neural Network Course

• Finish the first homework Convolutional Model, Step by Step (only the first for this week)

• Write down and bring specific question (only if you have some)

This week you will..

• discuss about CNN, advantages and limitation after the course

• think about what can happens in the future

• hear about transformer in computer vision

Learning Resources

Until next week you should

• Prepare the presentation!

The Deep Learning for Computer Vision course continue the journey into neural network going into more details about Convolutional Neural Networks (CNN). This course is intended as a follow-up after the Deep Learning from Scratch course and aims at understanding and recreating complexing network architecture for fascinating and challenging projects.

The Objective of the Course

The aim of this course is to reach a deep level of understanding about how CNN works and why they are so powerful. During the course you will have to complete assignments which will give you an insight about what can be done, for example object detection, face recognition, neural style transfer.

During the course, we will have weekly discussion to deepen our understanding of the subject and time to work on your own project.

Requirements and Motivation

We assume you have knowledge about python, linear algebra and neural network. Ideally, you took the first course Deep Learning from Scratch (or a similar course) and you have done a small project. Having some experience is extremely beneficial in order to be able to keep up with the homeworks and the discussion, otherwise it tend to require a quite large amount of time.

The estimated time is around 5 to 10 hours a week distibuted between watching videos, doing homeworks and working on your project. Try to allocate enough time to manage to get through the whole course. If in doubt, ask us for advice.

The Project

Groups of students will be formed to work on a project. The project idea can come from any of the students, can be picked from a template, proposed from us or can be the continuation of a project from the last semesters. The project is needed in order to finish the course, and a final presentation will be given in the last week of the course.

The Course Material

The course will be held weekly and will constitute of an online session of 1 hour and a half. The material and slides for each session and will be uploaded on each week's page after the class.

ECTS

For further details about Certificates and ECTS please refer to the following page:

This week you will..

• Discuss about neural network and art generation

• Discuss about cost functions

Learning Resources

Check out Phil Wang's Repositories for more cool projects:

Until next week you should

• Work on your project - no fixed homework this week!

This week you will...

• get to know the mechanism underlying the self-attention approach.

• get to know the basics of prompt design and how to apply them in a playground.

• get to know examples for the application of transformer models.

Learning Resources

Additional resource explaining the Transformer model:

• Text: The Illustrated Transformer by Jay Alammar

• Video: Transformer Neural Networks - EXPLAINED!

Until next week you should...

• Watch video 3 (Multi Head Attention) and video 4 (Transformers) of the Rasa Series on Transformers & Attention.

• Note at least one question on the each of the videos!

• Complete chapter 2 of the Hugging Face course.

• Think about the project you want to do and/or get in contact with course mates about it.

This week you will...

• get all details about the structure and form of the course.

• get to know your fellow course participants.

• get a general introduction into the current use of transformer models and their basic idea.

Learning Resources

Until next week you should...

• watch video 1 (Self-Attention) and video 2 (Keys, Values, Queries) of the Rasa Series on Transformers & Attention.

• note at least one question on each of the videos of above.

• do chapter 1 of the Hugging Face course.

This week you will...

• learn how to fine-tune a Hugging Face model with your own data

Learning Resources

Until next week you should...

• complete chapter 4 (Sharing Models and Tokenizers) and chapter 5 (The Datasets Library) of the Hugging Face course

• do a literature review on the task of you project:

  • Search for transformer models applied to similar problems

  • Focus on the structure of the input and of the output

  • Are there pretrained models that you can use?

  • Which type of model is best suited? Do you need tokenization?

  • Do you need a type of embedding layer?

This section is currently re-written. More details on the GAN course will follow in this section soon.

Before the first class you should..

• Register yourself in the Opencampus Mattermost Chat

• Register yourself in Coursera and for the Natural Language Processing Specialization, and enroll at least in the first course Natural Language Processing with Classification and Vector Spaces.

This week you will...

• learn about the basic structure of the transformer models

• get to know what tokenization is about and why it is important.

Learning Resources

• To get an idea about the number of tokens a text or word is made off in the GPT models, you may want to checkout this site.

Until next week you should...

• complete chapter 3 (Fine-Tuning a Pretrained Model) of the Hugging Face course.

• meet with the other participants of the project you are interested in to get a more concrete idea of the project goal.

This week you will...

• insights on the benefits of the Hugging Face Datasets library

• discuss the literature review on project tasks

• get some ideas on how to visualize sequence data

Learning Resources

Until next class you should...

• complete chapter 6 (The Tokenizers Library) of the Hugging Face course

• look into the characteristics of you dataset and:

  • write down the specifics of how your data was collected

  • create filter variables to group your input data according to special characteristics

  • consider the following questions:

    • What are potential biases in your training data?

    • Are there outliers in the dataset?

    • Are the classes balanced? (If you deal with a classification task.)

This week you will...

• get insights on the preprocessing of different NLP and sequence classification tasks.

• get an idea on plausible hyperparameters to fit transformer models for different tasks.

• learn about different metrics to evaluate NLP models.

Learning Resources

Until next class you should...

• prepare your final presentation as described in week 8 here.

This is a brief overview of use cases of NLP. The goal is to show you what is possible with current NLP techniques and inspire you to use some of these applications for your own. This guide does not attempt to be comprehensive, so if you know of other interesting application we would be happy to tell us about them

1. Automatic text summarisation

Take one or more text documents and create a summary that represents the most important/relevant information from the original text. These summaries can either be “generic” (a general overview of the original text) or “query relevant” (a summary that only focuses on the text that is relevant to a picked topic). The summarisation process is either extractive (directly reproducing parts of the source text word-for-word) or abstractive (forming an internal semantic representation of the original content and using this to write the summary from scratch).

You can play around with a demo bot for this:

2. Question answering

This are tools that can answer questions asked in normal (natural) language.

It replies based on either text it saw during training or on some text you provide to it at the same time as asking the question. As with text summarisation, the answering process can either be extractive (directly quoting the source text) or abstractive (writing the answer based on an internal semantic representation of the original content).

3. Information extraction

The task of automatically extracting structured information from text documents.

Information extraction can facilitate further computation to be done on the previously unstructured data. There are two main types of information extraction: Named entity recognition and Relation extraction.

Named entity recognition allows you to identify all entities of a predefined category (e.g. Extract all cities; or extract all company names).

Relation extraction builds on top of named entity recognition. In addition to finding the entities, it allows you to detect the semantic relationships between them (e.g. Extract all countries and their capital cities; or extract all companies and the year they were founded in).

Here is a demo website where you can enter your text and see what subjects are extracted

4. Chat bots

Normal chatbots can hold conversations, answer your questions and carry out simple tasks (e.g. changing a setting in your account, placing an order or scheduling a meeting for you).

5. Text classification

The process of sorting pieces of text into one or more predefined categories. Examples of how this can be used include:

• Text sentiment classification;

• Spam filters;

• Determining whether the author is making a claim or not - as the first step in fact-checking;

• Analysing trends in social media monitoring.

6. Machine Translation

Translate from one language to another or let your text be rewritten.

Check out:

7. Write Code using natural language

Describe what you are trying to achieve, and let the AI draft the code for you (e.g. HTML, CSS, SQL query and Linux commands).

At present the tools that can do this are imperfect and can only really be used to write a first draft that you would need to review

For this week you should have gone through the lectures of week 1 of the first Coursera course on NLP, including the quiz and the assignment. https://www.coursera.org/learn/classification-vector-spaces-in-nlp/home/week/1

Let's start and dive into the fascinating world of NLP.

For an introduction we will look at how NLP was done a few years ago so that we can appreciate in the following courses what has changed in natural language processing for the better.

As a start we will do positive and negative sentiment analysis on Twitter tweets. Analyzing tweets is a huge topic in NLP. Hegdefonds use Twitter tweets to try to predict movement of stock prices, politician campaign managers analyze tweets to see how the sentiment about their candidate evolves. So we dive right to the heart of one of the use of NLP.

In this week we will count for each word in the dictionary how often it appears and positive respective negative tweets. We will use that dictionary to produce our input to a simple linear regression model and train it. So we won't use the words as the input to the model what we will do in later weeks. See the course videos for more details.

For the next week you should go through all the course videos, the assignment and the quiz of week 2 of course 1 in the NLP specialization. Take notes and notice if you have any questions about the material. In the next meeting we will discuss these.

https://www.coursera.org/learn/classification-vector-spaces-in-nlp/home/week/2

See you next week!

This week you will...

• present your project in the final presentation. :-)

The presentation should take about 15 minutes and include the following aspects that correspond to the content of a model card for the Hugging Face library:

• model description

• intended uses & limitations

• training data

• training procedure

• variable and Metrics

• evaluation results

Check this section on for more details.

Additionally, please also include into your presentation any things that you tried out and that didn't work, so we can all learn from your experiences.

Welcome to second course and to Natural Language Processing with Probabilistic Models! Congratulations that you have made it that far!

Let us start with some words on what probalistic models are. These are models that are based on the principle: "given these words what is the most likely next word". So a pretty reasonable approach

In this week we learn about autocorrect, minimum edit distance, and dynamic programming. At the end we build your own spellchecker to correct misspelled words!

For this week you should have gone through the lectures of week 4 of the first Coursera course on NLP and the assignment.

In this week we learn about one of the most important applications of NLP which is translating from one language to another.

Diese Woche werdet Ihr...

• Twitter ganz allgemein als tolle Resource für aktuelle Infos kennen lernen.

• Explizit das 'Twitterlehrerzimmer' als tolle Quelle für deutschsprachigen Content zum Thema kennen lernen

• Jan Vedder als einen 'Twitterlehrer' kennen lernen und seinen Online Kurs 'ChatGPT in der Schule' vorgestellt bekommen

Lernressourcen

Bis zur nächsten Woche solltet Ihr...

• Nehmt Euch Unterrichtsmaterialien oder eine Präsentation, die Ihr bereits genutzt habt. Sucht Euch einel Slide, um diese grafisch zu ergänzen.

• Generiert ein passendes Bild mit stable diffusion oder ggf. einem anderen Modell

Diese Woche werdet Ihr...

• Doris Weßels und Moritz Larsen kennen lernen

• von ihnen eine aktuelle Auswahl an KI-Tools gezeigt bekommen

Lernressourcen

Bis zur nächsten Woche solltet Ihr...

• (1) Erstellt eine Quiz-App zu einem Thema Eurer Wahl. Als Vorlage könnt Ihr folgenden Prompt nutzen:

  You are a quiz application, and we play together. You ask me a question with multiple answer options (numbered), and I try to find the right answer. The questions should be about linear algebra. At the beginning the difficulty level is easy. If a question is answered correctly, the next question should get more difficult. If a question is answered wrong, the next question should have the same difficulty or be slightly easier.

• (2) Bereitet Euch mit Hilfe eines Sprachmodells auf die nächste Woche vor, etwa mit folgendem initialen Prompt (fragt dann weiter nach Begriffen, die Ihr nicht versteht):

  Ich möchte verstehen wie die aktuellen Sprachmodelle wie ChatGPT funktionieren, habe aber keine Ahnung von maschinellem Lernen. Kannst Du mir auf einfache Art und Weise erklären wie sie funktionieren?

p.s.: Zur zweiten Aufgabe: Fragt das Sprachmodell gerne auch mal nach einer Lernressource auf YouTube.

We will talk about the general outline of the course, the final project and other organizational course related material. Also you get the chance to introduce yourself briefly and get to know your fellow students.

For the next week we will finally dive into the material. You should go through all the course videos, the assignment and the quiz of week 1 of course 1 in the NLP specialization.

For this week you should have gone through the lectures of week 2 of the first Coursera course on NLP, including the quiz and the assignment.

This week is very similiar to the first but instead of linear regression we will use naive bayes.

If you are not familiar with this machine learning algorithm these videos will give you a head start since in the coursera course naive bayes is just covered on the fly.

At the end you will be able to test your naive bayes model with your own tweets or other that you source from the internet.

For the next week you should go through all the course videos, the assignment and the quiz of week 3 of course 1 in the NLP specialization. Take notes and notice if you have any questions about the material. In the next meeting we will discuss these.

Have fun and see you next week.

Course dates:

• 19-04-2023 General Introduction

• 26-04-2023 ML Lifecycle Overview and Model Selection

• 03-05-2023 Data Definition and Collection

• 10-05-2023 From Feature Engineering to Data Storage

• 17-05-2023 Advanced Data Processing & Intro into Model Serving

• 24-05-2023 Model Infrastructure & Delivery

• 31-05-2023 Model Monitoring

• 07-06-2023 Project Presentations

Preparation

This week you will...

Learning Resources

Preparation

This week you will...

Learning Resources

Preparation

This week you will...

Learning Resources

For this week you should have gone through the lectures of week 3 of the first Coursera course on NLP and the assignment. https://www.coursera.org/learn/classification-vector-spaces-in-nlp/home/week/3

In this week we dive into representing our words as words vectors. Remember any ML Algorithm requires its input in a mathematical forms i.e in digits. The last two weeks we assiociated to each tweet two digits - one for posive sentiment and one for negative sentiment - and fed that vector in our models. From now on we will try a different approach indepedent from sentiment analysis ahd thus more general.

For that we encode each word as a vector. Thus we need a dictionary mapping each word to its corresponding vector. In general we do not know what is the best vector to represent a word. So we have also to learn that. Luckily there are a lot of pretrained word embeddings online and we can normally use one of those

For the next week you should go through all the course videos, the assignment and the quiz of week 4 of course 1 in the NLP specialization. Take notes and notice if you have any questions about the material. In the next meeting we will discuss these.

https://www.coursera.org/learn/classification-vector-spaces-in-nlp/home/week/4

To quote the NLP tutor: "And remember to have fun".

See you next week

Diese Woche werdet Ihr....

• lernen was Philipp Möller vom IQSH über die Einsatzmöglichkeiten von ChatGPT und von KI im Allgemeinen denkt.

Lernressourcen

Diese Woche werdet Ihr...

• durch die Diskussion der Übungsaufgaben der letzten Woche lernen, wie mit KI Ergebnisse bewertet werden können.

• einen Einblick erhalten, wie wissenschaftliche in diesem geforscht wird.

Lernressourcen

This week you will...

• get a basic idea of what MLOps is about

• get a first idea about possible projects you might want to work on throughout the course

Learning Resources

• Paper on the definition of DevOps

Until next week you should...

• register for the opencampus.sh program at Coursera (you will get an invitation link from opencampus.sh to do so) and enroll for the course Introduction to Machine Learning in Production

• complete week 1 and week 2 of the course Introduction to Machine Learning in Production, including the provided exercises/labs.

Diese Woche werdet Ihr...

• durch die Diskussion der Übungsaufgaben der letzten Woche lernen, wie eigene Texte mit z.B. Neuroflash entstehen.

• einen Einblick in die wissenschaftliche Arbeit von Vivian Gunser erhalten.

Lernressourcen

Bis zur nächsten Woche solltet Ihr...

Hier der Link zum . Die Aufabe ist, die folgenden beiden dortz vorgestellten Prompts zu nutzen;

1. Create a rubric for a 5-paragraph opinion essay based on the Texas Essential Knowledge and Skills for grade 10. The rubric should include 3 categories: 1. organization, 2. development of ideas, and 3. use of language/conventions. Each category is worth a maximum of 4 points for Exemplary, 3 points for Meeting, 2 points for Approaching, and 1 point for Beginning.

2. Act as an expert educator who is able to read information, analyze text and give supportive feedback based on a rubric that I will give you. Students were given the following assignment: [paste student assignment here] When you are ready, I will give you the rubric. I will then begin to give you the student work to evaluate and you will provide specific, constructive and meaningful feedback in a first-person, supportive voice. If the score given is less than [total points for the rubric] points, provide a brief paragraph on specific steps the student can do to improve the work and earn full credit based on the rubric. Do you understand?

This week you will...

• Get an overview of data problems that occur in machine learning applications.

• Learn some techniques to address these problems

• get to know about data quality

Learning Resources

• https://www.coursera.org/learn/introduction-to-machine-learning-in-production/home/week/3

• https://www.coursera.org/learn/machine-learning-data-lifecycle-in-production/home/week/1

Until next week you should..

• Complete the goal setting for your course project.

• complete week 2 and week 3 of the course https://www.coursera.org/learn/machine-learning-data-lifecycle-in-production/home/week/2, including the provided exercises/labs.

This week you will...

• learn about meta data from a machine learning pipe line

• learn about managing evolving data

• get an overview on enterprise storage solutions

Learning Resources

• Machine Learning Data Lifecycle in Production: Week 2 and Week 3

Until next week you should..

• complete week 4 of the course Machine Learning Data Lifecycle in Production and week 1 of the course Machine Learning Modeling Pipelines in Production; including the provided exercises/labs.

This week you will...

• Master Prompt Engineering

• Familiarize yourself with different prompting frameworks

Slides

Until next week you should...

• Go through the learning material below

• Apply and try the learned prompt engineering techniques on your project and report on your findings in the next session

Learning Resources

Prompt engineering is a relatively new discipline for developing and optimizing prompts(a.k.a the text inputs) to get the best out of large language models (LLMs) for a wide variety of tasks. This means that we manipulate the text input to the model with the goal to get the best or most desired output out of the model.

Prompt engineering skills generally help us to better understand the capabilities and limitations of LLMs just as they are very valuable to improve the capacity of LLMs on a wide range of common and complex tasks such as question answering and arithmetic reasoning.

We can also look at it, that Prompt Engineering is referring to methods on how to communicate with the LLM to steer its behavior towards desired outcomes. One key point of prompt engineering methods is that they don't touch/change the model weights. So the LLM is completely frozen and the only change is happening in the input values - the prompts.

Prompt engineering is a very empirical science and the effect of specific prompt engineering methods can vary a lot among models, thus requiring heavy experimentation and heuristics.

We will now look at different prompt engineering methods..

Basic Prompting

Zero-shot and few-shot learning are two most basic approaches for prompting the model, pioneered by many LLM papers and commonly used for benchmarking LLM performance.

Zero-Shot

Zero-shot learning is to simply feed the task text to the model and ask for the result.

Prompt:

Classify the text into neutral, negative or positive. 

Text: I think this course is amazing.
Sentiment:

Output:

positive

As you see for sophisticated LLMs and easy enough tasks this is already enough to achieve the aim.

Few-shot

Few-shot learning presents a set of demonstrations, each consisting of both input and desired output, on the target task. Normally they are high quality examples. As the model first sees good examples, it can better understand human intention and criteria for what kinds of answers are expected. Therefore, few-shot learning often leads to better performance than zero-shot. However, it comes at the cost of more token consumption.

Prompt:

A "whatpu" is a small, furry animal native to Tanzania. An example of a sentence that uses
the word whatpu is:
We were traveling in Africa and we saw these very cute whatpus.
To do a "farduddle" means to jump up and down really fast. An example of a sentence that uses
the word farduddle is:

Output_

When we won the game, we all started to farduddle in celebration.

For further improvements we first have to understand which are the elements of a prompt.

Elements of a Prompt

A prompt contains any of the following elements:

Instruction - a specific task or instruction you want the model to perform

Context - external information or additional context that can steer the model to better responses

Input Data - the input or question that we are interested to find a response for

Output Indicator - the type or format of the output.

You do not need all the four elements for a prompt and the format depends on the task at hand.

Best Practices

Usually, the more specific and relevant the context is to the task we are trying to perform, the better.

We should be very specific about the instruction and task we want the model to perform. The more descriptive and detailed the prompt is, the better the results. This is particularly important when we have a desired outcome or style of generation we are seeking. There aren't specific tokens or keywords that lead to better results. It's more important to have a good format and descriptive prompt. In fact, providing examples in the prompt is very effective to get desired output in specific formats.

Here a the golden 6 bullet points for good prompting:

1. Write as clearly and precisely as possible.

2. Provide as much context as possible/necessary.

3. Really use ChatGPT as a chat interaction.

4. Iterate until you are satisfied with the results.

5. Break down the task into individual steps.

6. Provide examples.

Next we will go to see specific prompting frameworks which increase the outputs especially for more complex tasks even more.

Ressources:

https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/

This week you will...

• get to know the course structure and you fellow course mates

• get an idea about possible projects and the offered default projects

Learning Resources

Until next week you should...

• watch Andrej Karpathy introduction to transformers

• watch this lecture on Contextual Word Representations Part 1

• decide on which project you want to do in this course

This week you will...

• Master the basics of attention and transformers

• Familiarize with advanced models

Slides

Learning Resources

• Stanford Lecture on transformers

• Rasa series on Attention

Until next week you should...

• create your project repo accroding to the instructions provided here

• complete your literature review according to the instructions provide in the template repo here

• create a ChatGPT account if you don´t already have one

• complete week 1 of the LLM Coursera course (Audit Mode)

This week you will...

• Gain a broad understanding of how language models have evolved, from early methods like n-grams to advanced transformer architectures.

• Understand the significance and limitations of word embeddings and recurrent neural networks, including LSTMs.

Slides

Learning Resources

• Stanford CS224N NLP with Deep Learning

• Stanford XCS224U: NLU

Until next week you should...

• Watch the last ten minutes of this Stanford lecture on Attention

• Watch the first 25 Minutes of this Stanford lecture on Attention

• Watch all 4 Videos of the Rasa Attention Series

• Complete the Notebook and play around with different embeddings

  • The linked Repo contains the notebook as well as two exemplary vector stores to play around with

• Optional: Watch this Stanford lecture on Transformers

In one interesting paper the authors emphasize the importance of a standardized taxonomy for LLM prompts targeted towards solving complex tasks and, subsequently, provide such a taxonomy, i.e., TELeR, which can be utilized by multiple independent researchers who are conducting such research studies in order to report their results using a single unified standard.

As developers we can look at the best level (i.e. level 6) and use all the listed promt details for our prompts we design for our task.

Diese Woche wirst Du...

• die anderen Kursteilnehmenden kennenlernen.

• den Ablauf, die Inhalte und die Ziele des Kurses kennenlernen.

• lernen wie die Zugänge zu Chat-Anwendungen wie ChatGPT, Bing und der OpenAI API aussehen und wie sie sich unterscheiden.

Lernressourcen

Bis zur nächsten Woche solltest Du...

• Dich bei Bing, ChatGPT und ggf. zur OpenAI-API angemeldet haben.

• mit Hilfe von ChatGPT und Bing zwei Multiple-Choice-Fragen (jeweils mit vier Antwortmöglichkeiten) auf Basis von Texten zu zwei unterschiedlichen Themenbereichen erstellen.

This week you will...

• know how to prepare the data for training LLMs.

• get a better technical understanding of how to train LLMs.

• learn about different alignment approaches such as RLHF and RLAIF using PPO, or DPO.

Learning Resources

• from the Hugging Face NLP course.

• by Andrej Karpathy explaining how to train a GPT from scratch.

Until next week you should...

• and of the course Generative AI with Large Language Models.

• specify which metrics you'll use to evaluate the model performance, and why you've chosen these metrics. Document it in the corresponding section of your project repository.

Self Consistency

The idea is to sample multiple, diverse reasoning paths through few-shot CoT, and use the generations to select the most consistent answer.

Generated Knowledge Prompting

First generate some knowledge given the question. Then answer the question with the knowledge as context in the second step

Tree of Thoughts (ToT)

ToT maintains a tree of thoughts, where thoughts represent coherent language sequences that serve as intermediate steps toward solving a problem. In plain english this means that multiple solution approaches are tried and one keeps kind of log what works or. If you have solved a Suduko you probalbly are familiar with this back and forth exploration and in one of the original ToT papers they reported huge success with this method for solving sudokus.

The ToT approach essentially enables an LM to self-evaluate the progress intermediate thoughts make towards solving a problem through a deliberate reasoning process. The LM's ability to generate and evaluate thoughts is then combined with search algorithms (e.g., breadth-first search and depth-first search) to enable systematic exploration of thoughts with lookahead and backtracking. While this leads to much better complex task solving capabilities it takes hugely more LM calls and takes quite some time..so if you want to build a chatbot this approach should not be used.

This week you will...

• ...

Slides

...

Learning Resources

• ...

Until next week you should...

• ...

Hybrid Format - Every Wednesday 18h00

Preparation

This week you will...

Learning Resources

This week you will..

• Check out the face recognition code

Learning Resources

Until next week you should

• Finish the homework on Art Generation with Neural Style Transfer

Great that you want to dive into the deep water in Machine Learning. These are exciting times with major advancements on a quarterly basis like ChatGPT, Whisper, StableDiffusion and so many more. Nevertheless all these exciting models were developed with solid ML knowledge which is what we want to acquire in this course.

This is a course which brings you from beginner to intermediate or even advanced. It is formally called Intermediate Machine Learning but following HuggingFace🤗 terms which we will use heavily in the course I like to call the course SmilingFace😊. This is meant ironically, because you will never laugh in the course😊. Okay joking aside the use of smileys during learning and practioning ML helps us to remember to have fun, laugh about our mistakes and take ourself not seriously as it was proposed by the HuggingFace🤗 community. Therefore we will use our 😊 heavily in this course.

On the next pages you can see what the content of each course week will be starting with what will happen during each of our course sessions. Then again the SmilingFace😊 will lead you to what else to do in the week. I have divided the course into three levels of course work:

😊

The part after one 😊 is mandatory for each course participant for a for successful participation

😊😊

The part after two 😊😊 is voluntary but recommended

😊😊😊

The part after three 😊😊😊 is completely voluntarily for the ones who really want to know

Remember the course instructor(me) is also fallible so please question me if you see something that does not kind of seem right for you. Also always ask questions especially if you don't fully understand something. This is really why we give this course so that you understand everything😊

Own contributions or suggestions for improving the course as well as feedback are always welcome😊

Let's dive right in!

This week you will...

• Learn about Retrieval augmented generation

• Agents

Learning Resources

• Watch OpenAIs Tips and Tricks on RAG and Finetuning

• Get to know Openai Function calling

Until next week you should...

• further investigate your dataset characteristics as described here.

• be prepared for a presentation of your dataset characteristics.

• do this short course by Deeplearning.AI and Weights & Biases on how to use the Weights & Biases framework to track and evaluate your model results.

Course session

Welcome and Introduction round

Introduction of the course, opencampus, the course instructor and the course participants

Tool Set-Up

• Coursera

• Colab

• Editor (VSCode)

• Virtual Environments

• Git/Github

Walk-through

PyTorch 101 (Lab 01)

A visual overview of the workflow in the Colab notebook you can get in the PyTorch diagram below:

To-do

😊

1. Watch the following introduction video to the PyTorch framework

1. Watch Week 1 of the Coursera Course

1. Go for your own through the Colab Notebook above (Pytorch101) and try to understand and repeat the steps for your own. Thereby you should also solve Task 1-3 in the notebook. You can create therefore a copy of the notebook in your Drive or download the notebook to work locally on it. Ensure that you sufficient computing resources available (i.e gpu) if you choose to work locally.

😊😊

Try to improve the accuracy in the PyTorch 101 notebook by tweaking the amount of layers and number of neurons

😊😊😊

Familiarize yourself with basic PyTorch Tutorials:

• https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html (First part)

• https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial2/Introduction_to_PyTorch.html

There are certain requirements which form the basis for a successful course participation. If you do not have the mandatory requirements listed below, cosidering enrolling into a more basic course of our offerings. Alternatively bring yourself up to speed. Under additional ressources on the left sidebar you find the necessary ressources. Since the course has a really high pace it will be absolutely necessary to straighten these basic requirements before the course!

Mandatory

Python

Here is a refresher notebook:

Math

Linear Algebra, Probability Theory (at least the basics)

Machine Learning

Basics:

• What is a neural network

• What is a forward/backprogragation

• What is a loss

• What is an activation function

Totally optional

You can set up your PC for local development. A guiding notebook is here:

Here different IDEs are presented and compared:

Course session

Kaggle

• Introduction

• Titanic

• Paddy

• Exploratory Data Analysis(EDA) for Paddy Disease Classification

Solutions exercise MLP

Presentation from the participants of the MLP from Coursera

Walk-through

PyTorchLightning

PyTorch 303 (Lab 03)

To-do

😊

Go for your own through the Colab Notebook above (PyTorch303) and try to understand and repeat the steps for your own.

Do Week 3 of the Coursera Course

Please register at kaggle.com and join the competition. Go through the Exploratory Data Analysis Notebook session and then train a Logistic regression as baseline model!

The main objective of this Kaggle competition is to develop a machine or deep learning-based model to classify the given paddy leaf images accurately. A training dataset of 10,407 (75%) labeled images across ten classes (nine disease categories and normal leaf) is provided. Moreover, the competition host also provides additional metadata for each image, such as the paddy variety and age. Your task is to classify each paddy image in the given test dataset of 3,469 (25%) images into one of the nine disease categories or a normal leaf.

So that is where we will be heading in the next session trying different tools and techniques.

EDA Notebook

Logistic regression (try first on your own but if your stuck look at the notebook below):

😊😊

Build an MLP in PyTorchLightning for Paddy Challenge on Kaggle

😊😊😊

Do your own EDA on the Paddy Challenge and/or look at other EDA notebooks from competitors. Make a final presentable EDA notebook

Transfer the CNN from the Coursera assignment to our Kaggle competition

Familiarize yourself with this PyTorch Tutorials:

This week you will..

• Follow a peer review process of the projects

• Discuss the main difference with U-Net

• Discuss about segmentations

Learning Resources

Until next week you should

• Check the videos of the fourth week of the coursera course

• finish the homework on Face Recognition

Every group will present their project. All details about timing and format will be discussed in the classroom and updated later here.

Course session

Explanatory Session Part 1

Self-attention and multihead attention

Hugging Face Introduction

Library and Walk-through of HuggingFace101

Explanatory Session Part 2

Transformer Encoder and Positional Encoding

To-do

😊

Go through this excellent site explaining Transformers:

Do Chapter 1 and Chapter 2 of the HuggingFace NLP course

Go through the TransformerHW1

😊😊

😊😊😊

Look closer at the Pytorch module nn.Transformer (documentation) and go through a tutorial on how to use it for next token prediction.

Chain of Thought is essentially a prompting approach that gives the model not only more or better context like the previous methods (e.g. few-shot prompting, 6 golden prompting rules..) but let model think longer to get the correct answer. More thinking here means more computation. As you can see in the figure below without chain-of-thought thinking the model has basically just one forward pas through the transformer model to get to the correct answer. That might be enough for simple tasks like sentiment analysis but not enough for more complex tasks. With this think step by step approach the model can use more computations to get to the correct result. This is actually not very different from who human are instructed to work step by step through complex problems.

Course session

Quiz

ML Basics recap

Solutions exercises

Presentation from the participants of the tasks from PyTorch101

Walk-through

PyTorch 202 (Lab 02)

To-do

😊

Go for your own through the Colab Notebook above (Pytorch202) and try to understand and repeat the steps for your own.

Do Week 2 of the Coursera Course

The notebook from the Coursera Course of Week 2 can be accessed here:

The redundancy between our notebooks and the Coursera notebooks is desired to reintroduce the concepts in a different way and hence enrich your learning experience!

😊😊

Try to improve the accuracy in the PyTorch 202 notebook by tweaking the amount of layers and number of neurons

😊😊😊

Familiarize yourself with basic PyTorch Tutorials:

• https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html (Second part)

• https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial3/Activation_Functions.html

• https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial4/Optimization_and_Initialization.html

Diese Woche werdet Ihr...

• durch die Diskussion der Prompts aus den Übungsaufgaben der letzten Woche lernen, wie man Prompts anreichern kann (bzw. muss) damit diese besser funktionieren.

• eine nicht-technische Einführung in die Funktionsweise von Sprachmodellen bekommen.

Lernressourcen

Bis zur nächsten Woche solltet Ihr...

wenn ihr an einem Projekt arbeitet:

• Euch ggf. erstmals im Team treffen, um das Projektziel genauer zu defnieren.

• eine "Literaturrecherche" durchführen. Das heißt, im Internet nach Beiträgen oder Posts suchen, in denen andere etwas Ähnliches gemacht haben und schaut wie dort vorgegangen wurde. Stellt eine Liste mit mindestens zwei oder drei Links mit ähnlichen Projekten zusammen.

alle:

• einen Veranstaltungshinweis in Form eines kurzen Blogs zu schreiben. Nehmt dazu an, Ihr wollt Werbung für eine Veranstaltung bei Euch in der Schule, Hochschule oder für Euch privat machen. Nutzt dazu aber nicht ChatGPT, sondern ein speziell auf Texterstellung ausgerichtetes Tool Eurer Wahl, wie etwa Neuroflash.

Course session

Solutions exercise CNN

Presentation from the participants of the CNN assignment from Coursera

Kaggle

• Homework presentation of Logistic Regression for Paddy Disease Classification

Walk-through

Basic CNN in PyTorch:

PyTorch 404

Basic CNN in PyTorchLightning:

To-do

😊

Go for your own through the Kaggle Notebook and PyTorch404 above and try to understand and repeat the steps for your own.

Do Week 4 of the Coursera Course

😊😊

Add the the test functionality and create a submission.csv and upload it to the leaderboard

😊😊😊

This week you will...

• get insights on the benefits of the Hugging Face Tokenizers library

• discuss the characteristics of the data you use in your project

Learning Resources

Until next class you should...

• complete at least two sections from chapter 7 (Main NLP Tasks) of the Hugging Face course

• calibrate a baseline model

Diese Woche werdet Ihr...

• noch einmal in die Forschungsarbeit von Jenni und Thorben (IPN) Einblicke erhalten.

• Es geht darum, welche Kompetenzen SchülerInnen und Lehrende zukünftig (nicht mehr) brauchen werden.

Lernressourcen

This week you will...

• get some additional ideas on models or areas of machine learning that were not covered in this course.

• get a quick overview of possible next steps in your machine learning journey and how to stay updated with the fast moving development in this field.

Learning Resources

Until next week you should...

• focus fully on your project again and prepare the project presentation for next week.

• consider the following requirements for the final presentation shown here.

This week you will...

• explore advanced training techniques designed to train large models efficiently, minimizing computational requirements.

• gain comprehensive insights into the key hyperparameters for effective model inference.

• discover the unique attributes of inference processes for streaming Large Language Models (LLMs).

Learning Resources

• Week 2 and week 3 of the course Generative AI with Large Language Models

Until next week you should...

• Focus on your project work and on the training of the final model.

Course session

Kaggle Finetuning

Presentation of experiments with the goal of improving the classification accuracy

Transfer Learning

Theory and Applications

To-do

😊

Watch the second half of Week 5 of the Coursera Course

Watch the following Seminar about Transformers:

😊😊

Go on using ideas discussed in this session and go on improving the accuracy on the Paddy Dataset

This week you will...

• get to know Weights & Biases a popular platform to evaluate deep learning models.

• understand different evaluation metrics.

• get a high level introduction into training LLMs.

Learning Resources

• short course by Deeplearning.AI and Weights & Biases on how to use the Weights & Biases framework to track and evaluate your model results

Until next week you should...

• watch this video on data collators.

• work through the section Training a causal language model from scratch from the Hugging Face NLP course (do not change to the TensorFlow version of the site).

• Decide on a baseline model and add a short description of your approach in the corresponding section of your project repository.

Preparation

This week you will...

Learning Resources

Course session

Faster Coding with ChatGPT, Stackoverflow and clever search

Solutions exercise RNN

Presentation from the participants of the RNN assignment from Coursera

Deep dive

• What are Embeddings?

• Reinforcements of and insights into RNNs beyond Coursera

Walk-through

PyTorch 505

Transfer Learning CNN in PyTorchLightning:

To-do

😊

Watch first half of Week 5 of the Coursera Course

!!!

Using all techniques learned build the best model you can to achieve an accury at least above 70%. You can use Transfer Learning, Augmentation and other tricks. You can also take inspiration from fellow notebooks on Kaggle. Good ideas will be rewarded by special achievement badges for the course. Have fun and push the accuracy!😊

!!!

Week 10 - Project Presentations

Week 5 - Agents

This week you will...

...

Until next week you should...

• ...

Diese Woche werdet Ihr...

• die Abschlusspräsentationen der Teilnehmenden sehen

• den Kurs abschließen.

Lernressource