Questions, answers, and feedback
- Teaching through Twitch? That’s interesting, I would like to know more when we have time to discuss this
- You can find more here: https://researchsoftwarehour.github.io/
- Teaching via Twitch research software hour
- motivation for taking this training
- difference between what we (also Carpentries) do and traditional academic teaching
- Have some “spare” time for open questions
- Take the prior knowledge into accounts
- But expect no “huge knowledge” before
- Known to unknown
- Keep it simple and finish on time
- Have enough breaks
- Make mistakes yourself, so people see that making mistakes is okay. Encourage failure.
- Focus more on practicals than the theory
- Motivation: if you teach people, you have a scaleable influence
- Customization for the audience is a challenge
- Pre-workshop survey should be used to prepare; without that, you need to be flexible as an instructor
- It’s important to create an environment where attendees feel comfortable to ask questions at their level
- Teach the building blocks
- Build confidence to continue from those building blocks
- Mixed audience is a challenge
- how do you connect to their background knowledge, research domain
- there will be people who already know, those who will learn and those who probably will not learn
- teaching philosophy:
- Focus on the practical. When learners leave a workshop they should be able to continue in their favourite environment. So that they are not stuck behind technical difficulties.
- Want learners to be troubleshoot themselves after the course and not depend on a cloud service.
- Show how to troubleshoot or search for answers while teaching to also teach searching for answers.
- Focus on getting an underlying understanding of the topic.
- Trying to explain things as simple as possible. Documentation is really important and we often check the documentation since we cannot remember everything.
- Training is an iterative process.
- Meeting learners at different levels is particularly challenging online.
- Trick: status chart (spreadsheet) for status and questions, equivalent to sticky notes. Helpers also write their name next to an entry so that we know that learners have been helped.
- Half-way into exercise start giving hints.
- As instructors jump between rooms to see how it is going.
- Be vulnerable as a teacher, show that you do not know everything, this helps also to create confidence in the learners
- we started discussing our interest in this workshop and also talking about our experiences with the Carpentries
- motivation to participate in today’s workshop:
- inspiration in online training +1
- need for next level workshop (following up software carpentries)
- next level is relatively more difficult to adjust to audience because of more diversity (how to define the prerequisites properly)
- Teaching philosophy
- follow the course material; adjust pace to audience; make use of exercises
- Enjoy teaching! While preparing for teaching and while performing it, I also enjoy learning new interesting things (also from learners and their unexpected questions), and organising my knowledge better. In addition, teaching can be very rewarding when seeing learners learn new things.
- It is imporant to make lessons interactive, and really involve the students. Just giving presentations does not work well.
Difference between Carpentry/CodeRefinery workshops and regular academic courses:
- largest difference to traditional academic teaching is the hands-on part
- one exercises the concepts at the same time that they’re taught
- requires different skills than normal lecturing
- also more demanding for particpants
- social environment: the code of conduct, makes people feel more safe than regular academic courses
- see people from widely different backgrounds, more diversity among instructors and learners
- more rewarding than regular teaching, participants get the wow effect when something works!
- Time scale: months vs days.
- Less time to adjust to different learner starting skill levels
- we never teach alone in carpentry/coderefinery
- trying to put oneself in the position of the learner to try to get a common ground
- use silly references to films or culture to create a light environment ->humor
- explain what you’re going to do, then do it
- use approaches that one sees working in other (non-educational) contexts in one’s teaching
Exercise: give feedback on teaching
- watch short video: https://www.youtube.com/watch?v=-ApVt04rB4U
- feedback on two axes: positive vs negative and content vs presentation
- (constructive) negative
- uses too much advanced jargon
- reading out loud while typing
- seemed passionate about the topic
- made a mistake and corrected it (shows real process of coding)
- asked if any questions at end
- (constructive) negative:
- could enlarge the font/screen a bit
- mobile phone distraction
- dismissive language: “this is really simple stuff”
- a bit too fast paced for someone not familiar with the topic
- using words like ‘simple’ and saying things like ‘even excel users could get this’ may intimidate students trying to learn a new concept
- (constructive) negative
- The jargon (functors, …) may be unfamiliar and could be explained.
- it was clear that the lesson started
- manually typed out the code
- made a mistake (relate-able) and said “my bad”
- (constructive) negative:
- I could follow along easier if you could increase the font size
- Best to focus on the lesson instead of picking up phone
- Using words like “simple” and “of course” may mean this to you, but not to all learners.
- No need to disparage Excel users; it does not contribute to the lesson and may scare away learners
- Starting the lesson clearly was good! There was no need to start angrily; inviting learners to sit down can be sufficient.
- links to prior knowledge
- try to explain more what you type in
- do not use jargon
- avoid throwing complex concepts into the air without explaining them
- tries to get the attention of the learners
- speaks clearly
- speaks while typing
- not always hidden behind the computer
- standing not sitting
- (constructive) negative:
- try to be more patient with learners
- try not to use dismissive language
- do not ignore errors, use them as an opportunity to teach
- passive agressive attitude in the beginning
- make screen readable
- slow down
- choose details to say and details to omit
- seems like a useful topic
- highlights important concepts
- live coding is a good teaching tool
- introduces concepts without prefacing it
- avoid words like ‘simple’ and ‘of course’
- too quick
- font size is very small
- making and owning up to mistakes creates a safe learning environment
- seemed passionate about the content
- vivid teaching style
- live teaching along with explanations
- dismissive language, “this is simple”
- try to talk slower, with more breaks
- text size far too small
- try to avoid checking phone, switch to silent before starting
- rambling delivery, should pause more
- sometimes a bit chaotic
- uses too much jargon
- aggressive to students at the start, starts on a bad note
- Is thorough but overloaded
- Good mastery of the content
- Try to reduce the jargon
- Keep focussed on teaching material
- Give an overview of the material before diving in
- you are very enthusastic
- live coding while explaining
- Avoid using diminuatives like “Simple”, “Easy”
- next time, speak a bit slower
- try to make the screen font a bit bigger
- better not talk negative “Even Excel users…”
15 minutes, end at xx:10
- Why does CodeRefinery target competent practitioner?
- Why is it important to be able to write code for attending a CodeRefinery workshop? For this exercise, we suggest to read Motivation and Demotivation from the Carpentries instructor training lesson.
Breakout room 1
What to do with novices?
- Talk to them and be honest about the fact that this workshop may not be as useful for them as they would want.
- It would also depend on the number of helpers you have available. If you have many people, some can focus on the novices.
- It makes sense that Code Refinery focuses on competent practitioners: if you teach defense in basketball, then trainees do need a basic competence in basketball for the training to make sense.
- There is a contract when you give pre-requisites for the course and people choose to participate. You can look at the communication strategy for your course if you see that you frequently have stragglers.
- Offer the option to audit a course and not participate in the workshop elements.
- Online courses also allow more space for people to join in their own way; not every participant needs to be actively engaged but may also “lurk”. Because it is online they do not take up as many resources.
- You cannot please everyone: using the 20/80 principle can help. Target the 80% that will benefit most from the course.
- Be very explicit about:
- What you are teaching
- Why you will be teaching it
- What you expect to be prerequisites
- also independently for each section/topic
How do you recognize them?
- They are checking their phone :phone:
- They may be very silent :shushing_face:
Breakout room 2
- How do you notice that somebody is not able to follow?
- Short feedback loops to recognize early whether learners follow.
- Design lesson so that it starts with basic concepts and exercises.
- Documentation lesson/concept can apply also non-coding.
- Cheat-sheets with basic concepts.
- Group novices and more advanced learners together in a group.
- Team exercises.
- Inclusivity approach: saying that novices are welcome to stay and watch and it is OK to not catch everything at once. ++
- Prep session where you can detect problems early or fixup session later.
- Note: Version control applies also to other work than coding, and I think that most CodeRefinery lessons are designed so that people who do not program can learn version control even without hearing the term before. But sure, they will not understand some of the concerns and terminology related to coding. Diclaimer: I might have a blind spot. (I agree with Matus here! So I share the same blind spot?)
- For 2-day workshops you reserve first morning on “theory” and concepts and getting the laptop configured and then later sessions are really hands on with everything working.
- Self-assessment can be tricky both ways
- novices maybe did not read the prerequites
- but also advanced learners who think they are novices.
Breakout room 3
- should be possible to separate the groups
- you will always have novices
- we should have talk common vocabulary
- one option is to pair novices with more experienced students
- if novices are ID’ed early on, possibly redirect to Carpentries
- helpers are the ones who will be most able to see who is a novice since they will help with questions
- if there is the opportunity, can do some screening via pre-workshop survey- including having students solve specific problem so that they can be redirected if needed.
- is it considered bad to have prerequisites (most non-novice courses do)?
- possibility to point part-novices to tutorials for missing skills ahead of time
- as a novice it can be helpful to be told if the course if above your level
- if there is an opportunity to talk to a potential student, asking a few basic questions can be helpful in figuring out what level they’re at
Breakout room 4
- You can’t keep all the participants happy all the time
- always feedback saying too fast and others too slow; as long as these groups are balanced, it is reasonably fine
- use different levels/types of exercises
- (almost) dedicate one helper to a particular learner (only possible if there are not many other questions for helpers)
- collaborative exercises (peer teaching) in break out rooms (put beginners and experts in teams so that they can peer learn)
- specify the prerequisites
- make it clear that a carpentry workshop is a good starting point before attending coderefinery, unless you have decent experience in coding
- Is SWC enough, or should you have been applying it a little bit first?
- provide some exercises to evaluate the level of the learners and based on that recommend whether to proceed with Code Refinery or rather go to the Carpentries first
- have recommended reading before joining workshop, something that gives a general picture of the content
Breakout room 5
- filtering process to avoid variation in levels (surveys ?)
- Ok if they leave after a day.
- Remind what we expect in terms of knowledge
- tell people that they could leave if they realize that the course is not suited
- Balance to be found - let’s be utilitarian in the approach (maximize benefits for the largest possible group)
- Choose a good title
https://coderefinery.github.io/instructor-training/04-running-workshops/workshop-requirements-inperson.md gives 404 ;-)
Exercise: online workshop differences
- What are the possible problems/disadvantages?
- What are the advantages?
- How do you have to prepare differently?
- What is your experience with online teaching?
- difficult to have social aspect
- more tiring
- as a teacher you do not see the learners, do not get much feedback (possible ‘solution’ (alternative): HackMD)
- 2 channels needed (audio-video presentation plus written Q&A), thus absolutely needed to have at least 2 instructors to be interactive. However, this should also be seen as an advantage
- participants have to use the 2 channels, which may be sometimes distracting or stressful, and cause connection issues. On the other hand, it’s awesome to be able to get back to a written Q&A document
- no travel
- easier to scale
- costs much less time, money, and environmental impact (in case of international or long-distance travels)
- allows for people sitting in, but not actively participating
- more inclusive (e.g. also people available part of the time, such as taking care of children or wet-lab work)
- allows recording of the session +1
- it’s awesome to be able to get back to a written Q&A document (and videos!)
- hybrid teaching (in person + online at once) +1
- HackMD advantage over GoogleDocs: less issues with bandwidth / many people editting; somewhat easier to to put oneself into view-only mode when bandwidth issues start
- It is more difficult to see “how it is going” online.
In a physical room you can judge a bit better.
- It takes also experience to be a helper to know about possible problems and to know when to “interrupt” and when not.
- Useful to have a host for breakoutroom management and each room should have a helper or chair making sure that everybody can speak up.
- Prep-session for helpers where we scroll through the material and exercises and point out all the possible pitfalls.
- difficult to track questions
- delayed interaction
- difficult to manage questions
- missing non-verbal signals
- larger group -> people that usually could not attend due to time/money reasons
- one is forced to structure existing material
- courses are more streamlined
- definitely more helpers are needed
- setup connection well in advance
- have someone has a host keeping track of the video conferencing platform
- none of us has experience with massive (online) workshop
- difficulty with individual questions (any alternative for sticky notes system?)
- bigger responsibility for the helpers in an online workshop
- live/on site lectures are “better”
- more focus on the main material
- Adjustment “on the fly” done during live session - is that possible in Online version ?
- Better preparation
- large class room means more “formal content”
What are the possible problems/disadvantages?
- bigger group less interaction
- feedback is different
What are the advantages?
- more reach
- online you can divide groups according to operating system so that they all see the same
- share the problem solving in small groups
How do you have to prepare differently?
What is your experience with online teaching?
Lesson design and development
Once I got into a pattern of making one-A4-page cheatsheets on different topcis. forces you to compress a lot and figure out what’s important, perhaps similar here.
Are your learner personas typically related to research domains (topics), or to expertise (novice-expert)?
- According to the TTT system,
A learner persona contains:
- person’s general background
- what they already know
- what they want to do
- any special needs
Reproducible Research - 1
- 1st year PhD student in ecology,
influence of melting on butterfly population in the Alps
develop some script to analyse the data, geospation analysis
want to collaborate with the neighbour university, want them to be able to rerun their analysis,
this other researcher
- systematically collect the data
- experimental design
- write data management plan
- data generation
- archiving etc
eg. fellow researcher wants to runrun the analysis with their data. They need to find and understand data and software
in this lesson the participants will learn how to descibe the data, by associating metadata, learn hwo to find the relevant ontolgies,
they will learn hwo to select the data repository that fits their purpose, how to deposit the data,
the participant will learn how to capture the dependencies of the software dependencies and how to document them
more specific outcome:
the end of this lessonlearnes will be able to use conda package manager to capture, document etc. the dependecies a Python package
- deposit data in dryad repository
- create environment.yml file to capture the dependencies
- two groups write readme file describing how to rerun the analysis, then switch and try to rerun others code based on readme
- add readme file to the github repository
Reproducible Research - 2
- Voluntary Graduate School course.
- University wide.
- 1st year PhD student who has some tabular data. They use code to process this.
- Create a reproducible workflow.
- Student gets a dataset, creates code to analyze the dataset, publishes the data and the code. Then other student tests the reproducibility of such published data and code.
- Evaluation of every step between research planning and publication of data/code.
- Create comprehensible data structure + file organization.
- Explain what is wrong with several different file organizations.
- Upload this dataset to Zenodo.
- Decide the metadata fields needed for this Zenodo upload.
Brainstorm what you want to cover
- Accessibility of your publication (Open access)
- Makes research trustworthy
- Makes your research results visible
- Improves your impact and helps your career
- Tool & tricks they can use
- New PhD student, no idea yet how science (publishing) works; wants to change the world
- Master student doing scientific research for her MSc thesis
Learners should learned by the end
- Understand that open access makes research results visible, trustworthy, improves impact and helps career
- Remember what tools & tricks they can use immediately
- Should have become open science enthusiasts
- End? Toy publication on Zenodo
- Start? Discussion for & against
- User of other people’s software which keeps crashing on use
- Writer/development of “brittle” code that crashes when they want to develop it further
- Dealing with error messages from APIs
- Unexpected behavior of their code in other ranges of input
- target audience: early programmers, first step into “robust code”
- requirements: already have experienced “working code”, to have something to see the errors.
- Being able to log errors
- Giving (better) control over the code
- Have experienced examples of situations when they need to deal with errors, and how to address this
- Know how to identify an error and how to approach an error
- How to look beyond a single error; look ahead
- Know how to use a debugger and basic troubleshooting mechanisms
- Understand a traceback
- Be able to read the actual error message and not be locked in expectation of what the error might be
- Be able to search for solutions
- Learn how to raise and report an issue when you find a problem in somebody else’s code
- How to make a minimal reproducible example for error reporting (including software & platform info)
- Knowing that testing exists; being aware that it is the next step in robust code development
- environment/version awareness and reporting
- Challenge: examples probably need to be language-specific
- Lesson length: 30 min
- learning personas
- Scientists writing documentation for their own software
- Developers explaining to users
- People with programming experience
- Victoria: is a Dutch scientist who just published a tool and wants to avoid taking too much time answering user questions so decided to document it
- Albert: 29yr old, never got PhD, tremble in left pinkie; likes sailing and cheeses. Writes code for old PhD supervisor. He wants to save time replying to too many questions.
- Making writing documentation easy/ier
- Save time
- Types of documentation
- target: Everyone
- tools: Markdown
- target: Users, Students,
- tools: Jupyter
- API Documentation
- target: developers, future maintainers
- tools: Doxygen, Sphinx etc
- Literate code
- target: developers, collaborators, competition, students
- Learning objectives
- Use a tool to create an API
- Markdown or something else
Feedback on day 1
Positive things (for example, one thing you can use):
- nice mix of teaching and exercises :+1:
- HackMD and great teachers, love the breakout groups!
- I enjoyed sharing experiences on teaching philosophy and best practices in the breakout rooms, and got several very useful ideas out of that.
- Good content and good group - enjoyable work session.
- HackMD is nice. I like the idea of moving beyond SCarpentry.
- Lots of practical ideas that we can implement when running and designing workshops, well organized
Things to be improved:
- I’m getting a lot of warnings on significant energy usage from hackmd… and it has become very slow
- If you refresh now, does it get any better?
- No it’s not improved. I can type again, but I still get the same warnings, and hackmd is the only tab I have open now.
- thanks for the info
- Maybe a bit long ? I am very tired now!
- Time it in the breakout rooms is sometimes a bit short
- share the link to the lesson to be taught so people can follow on their own just in case
Preparation for day 2
- Tomorrow, there will be “teaching practice” where each person gives a 5-minute lesson. You will give the lesson twice (to two different groups) and get feedback each time.
- Decide what kind of 5-minute lesson would you like to teach (live-style)
- If you already know, place your vote in the appropriate section at the top of these notes
- It can be anything, probably something that you would already give training about so that it’s not too much work for you. It can be a CodeRefinery lesson if you want to learn about those lessons.
Day 2 icebreaker
What do you want to get out of this workshop/what will you use from this workshop?
- More confidence towards teaching some of the CodeRefinery content
Overview of what’s in a CodeRefinery workshop
Practicing teaching and feedback
Demo of the demo
We present an example of what we expect for you to do next
Feedback on content:
- Nice introduction into the problem/tool.
- Put into context and remind learners that the approach is not specific to git
- I learned something!
- Explain what you mean by “a commit is good or bad” (if there is time)
- a quick reminder for Git commands ? (git clone for instance)
- missing a simple sentence saying: “git bisects just helps out do the checkouts in the optimal order… you have to do the work by hand”… (sort of)
- Go into detail about other ways to solve (e.g. pytest) which may be obscure for learners
Do exactly what we did above, for each person. Follow time carefully, better for everyone to do it a bit
What did you enjoy? What should we change/improve? What will you use that you learned here?
Very nice chance to get useful comments on teaching styles, especially focused on online delivery
I was expecting more technically advanced topics, but what was shown was more advanced than the Carpentry, but not as I much as I thought. Interesting balance!
The demo and feedback was very useful :+1:+1:
The first day was very similar to the Carpentries TTT. Now not everyone has gone through the Carpentries TTT. But if a big part of the participants have done it, then it would be nice to see it more shaped towards the Code Refinery workshop.
It might be useful to get a bit more time to prepare presentations (now some people were preparing presentations while others were presenting). +1
Regarding the previous point: maybe put this in the email for preparation! :+1:
The backwards lesson design and teaching practice/feedback were very useful, also the tips on how to prepare for/run a workshop +1
I agree that the backwards lesson was very useful, and it showed how difficult it is to properly design a lesson. Perhaps that could be discussed in more detail.
At times the way the instructors were presenting was a bit flat. You can see the instructors are very nice and that they really are into what they are saying. But sometimes I was missing a bit more of enthusiasm when providing the information +1
* come on, the instructors have been living in the Nordics for a long-enough time ;DD
The preparation documents were not addressed in the workshop.
I really do not like prose long text on the shared screen and someone basically reading it to me. I like the literature code refinery created but please do not read it out. It is like a bad powerpoint presentation. Either make it a reading material or “perform” it more or preferably replace with more exercise. The breakout sessions and team discussion were very nice, as was the general vibe and feeling. Your passion is paltable. +1
The teaching demoes & feedback sessions were super useful, and some fun too. Nice to do it twice! It could have been a tiny bit more time for the feedback, we needed more time when we started diving deeper into the interesting depths, details, suggestions…
In general good time keeping thanks a lot
Really good tec support, well done.
Lots of thought and effort went into the materials and in running the workshop, very much appreciated. Also it was nice you were always looking to improve your material based on things that came up in the workshop. +1