Questions and notes from workshop day 5

Icebreakers, day 5

What's the best documented project you've seen?

What makes a project particularly hard to use?

What tools do you use for writing code?

What did you have for breakfast?

Documentation

Overview: https://coderefinery.github.io/reproducible-research/intro/ Lesson material: https://coderefinery.github.io/documentation/

Motivation and wishlist

https://coderefinery.github.io/documentation/wishlist/

Motivation-1

Exercise

:::info

Exercise until xx:40

  1. For a person not familiar with markdown, it is not clear were to start doing the exercise. I would suggest adding a link or sthg like that.

    • this is a great resource to learn: https://commonmark.org/help/
    • we have more resources here: https://coderefinery.github.io/documentation/tools/#restructuredtext-and-markdown (but we jumped over that section, we should re-arrange material for better flow)
      • Thank you!

  2. Nice README examples:

    • https://github.com/plasma-umass/scalene
    • https://github.com/geopandas/geopandas#readme
    • https://github.com/IndrajeetPatil/ggstatsplot (Not sure if it is good example)
    • https://github.com/matiassingers/awesome-readme (For a list of good README examples)

  3. This link: picture uploaded about Sphinx link in the lesson material is not working at least on my PC

    • Weird, it used to be correct. https://www.sphinx-doc.org works though. (extra www)
    • Lesson updated

  4. The functions (Note, warning etc) isn't working for me. It seems to render markdown, just not these functions. I even tried to copy and paste it. I'm using the github website.

    • example:
      • https://github.com/bast/experiment
      • source: https://raw.githubusercontent.com/bast/experiment/main/README.md
    • You need to commit it and then you see it on the github project-site in the preview. Otherwise it doesn't work
    • note/warning/etc are Sphinx (MyST-markdown) specific things, and Github markdown rendering doesn't generally know about them so they look weird. never mind, I was thinking of something else.

  5. The warning and note functions don't work for me - I think it is because I'm using gitlab? Do they have a different markdown format?

    • Oh yes, Gitlab probably has a different Markdown renderer (It uses Github Flavoured Markdown GFM). Markdown (beyond basic HTML formatting) isn't standardized at all and generally isn't compatible(!).
    • GitLab has its own flavoured markdown called GitLab Flavoured Markdown GLFM (https://docs.gitlab.com/ee/user/markdown.html)

  6. In the GeoPandas example it says "Tests failing" on one of the badges. What does that mean (in general and for us if we want to use it)?

    • We'll see about automatic tests tomorrow morning! And it will be more precise
    • It means that one of their automatic tests to verify correctness isn't working.
    • If you ask me: I'd still probably use it. At least they track what works and doesn't, probably every big project has many things that aren't working and we don't know about it.

Sphinx and Markdown

https://coderefinery.github.io/documentation/sphinx/

We type along with Radovan now: https://coderefinery.github.io/documentation/sphinx/#exercise-sphinx-quickstart -> In preparation for the exercise

  1. The index file can be MyST-markdown also, but by default it's made in ReST so it's easier for this lesson to keep it like that.

    • please show how :-) that's interesting
    • basically a straight translation of the directives. You'll probably see in lessons, but it becomes {toctree} in triple-backquote block.

  2. Please slow down :D

    • thanks for informing!
    • (this was also pre-intro, you'll have time during the exercise time!)

:::info

Break until xx:12

(then return for exercise intro)

Then: Exercise until xx:40 (we will fill this in after break)

  1. A unrelated question. When trying to set up the whole conda environment initially I couldn't switch into it using VSCode. After experimenting for a bit I noticed I can do it with the normal Windows CMD, rechecking VSCode the default terminal is Powershell, after switching it CMD prompt it works. What is the difference between the two? Why does it not work with Powershell?

    • Hm... so these are the different ways of taking commands and running them in the operating system. They have different startup methods that can learn about conda and so on
    • Maybe miniconda was originally set up for CMD, so it sets up CMD so that future CMD:s will work for it. So switching to CMD then worked.
    • It's something we'd need to keep investigating

  2. On the website (after using xdr-..), it says that my somefeature.md and anotherfeature.md are not found, any ideas? (they show themselves, but when clicked I get the file not found error)

    • just checking (this might not be the reason): you have myst_parser installed (you are in the coderefinery conda environment)? the python -c "import myst_parser" worked for you?
    • I didn't have myst_parser installed it seems, need to check if things work.
      • because plain Sphinx does not know what to do with markdown files and it needs that extension to parse and render them. default for Sphinx are .rst files. But we prefer to show it with markdown. we believe this is easier and this is also what we use in our work.
    • Okay so I am in the (coderefinery) and have ran pip install myst_parser and checked with python -c "import myst_parser"; but doesn't seem to solve the rendering problem...
      • I would in this case try to remove the "_build" directory and re-run sphinx-build . _build again.
    • (thanks for the help ^^) It didn't work unfortunately tho
    • and the files are in the same folder as the index.rst? and you have activated the extension in conf.py?
    • They are in the same folder yes, and some-feature.md and another-feature.md are mentioned in the index.rst and in the conf.py the extension line has been altered to extensions = ['myst_parser'] ?

  3. For the excercise Sphinx autodoc, what if the example.py is in a subfolder (e.g. src)? Somehow Sphinx only creates the autodoc if the file is in the same folder as index.rst.

    • good question. we will find a solution (possibly later today) and show an example on how to do that.
      • Thanks!
    • You could try PYTHONPATH=src:$PYTHONPATH sphinx-build .... This sets PYTHONPATH so it's importable
    • Other things people might do is make it installable and install the code in the conda environment itself. Really depends a lot.

  4. How are those tools (e.g. Sphinx) compared to MkDocs?

    • we will answer on stream but in short: it's the same idea. there are a number of similar tools: https://coderefinery.github.io/documentation/tools/#html-static-site-generators
    • same idea: read markdown, produce HTML

  5. Excercise GH-Pages-1: when I clone the documentation-example repository, I can't find the .github/workflows/ folder. I have a .git folder, but no workflows inside. Shall we create it?

    • yes the folder is created when you create the file .github/workflows/documentation.yml, you do not have to create it separately

  6. What does the part in the square brackets do: ![alt text](image.png) ? (not sure how to format it as text. Sorry.)

    • It sets the text that is shown when the mouse hovers over the image, or if the browser cannot show the image.
    • The alt text is originally intended for automatic readers, for example for visually impaired. It should describe the image.
    • (after 0 and + there's a button for ´ but with SHIFT it starts a code block in markdown)
      • Great!
        • Thanks!

  7. I managed to link to a external python file and that worked fine! How can I refer to only one specific python function?

    • Where it says automodule, I think you can do specific functions and so on. I don't know off the top of my head but (surprise) Sphinx documentation is usually pretty good about it.
    • https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#directive-autofunction

  8. Can I create a Sphinx webpage without making it a public website? For example, if I want to create a tutorial for my colleagues, can I have a webpage like this that is "local"?

    • Yes, in this exercise we build .html and it's not shared anywhere. you could share it on a private web server - but you need to figure outwhat that web server is and the rest is probably easy-ish.

:::info

Exercise until xx:40, see above for info

Exercise feedback - how did it go?

  1. At TU Delft it went well, there is interest to look at Git pages

    • great!

  2. I managed to make it but got the error "Users/myname/code/doc-example/new-feature.md: WARNING: document isn't included in any toctree" and then got a bit stuck after that.

    • Is the new-feature.md in the same folder as index.rst?
    • A typo in filename can also give that error
    • If it's a warning, the rest works. What it means: it made the page new-feature.md but it isn't included in the Table of Contents. Maybe this is expected but maybe it could be added to a toctree directive somewhere

  3. The Github Actions part, I just copy and paste but I don't really know much about the commands. Not sure what it is used for.

    • We'll explain now. It is rather short and direct, it shows the outcome but there is plenty more to learn
    • (but also, many of us go copying around working Actions files rather than understand every bit of them...)

  4. I am confused abnout what sphinx-build . -W -b linkcheck _build does. From what I understand it should create a local web server, but I can't find where or how to find this.

    • It reads all documents and checks all links: does the URL still exist and return data? It's used to keep stuff up to date
    • sphinx-autobuild is an extension that does make a local web server and shows updates live as you write and save.

  5. Is it possible to create Github web pages without Github Action code, should I be unlucky that copy and paste does not work ?

    • yes! Sphinx works locally (like the first exercise). But you need to know where you'll be sharing it. The action builds + uses the Github Pages web server. Deploying this is a bit tricky... a different set of skills, I'd recommend asking it it doesn't work or trying to copy again something that works
    • (did I answer the right question?) Err ok. I thought maybe I can create Github pages using a click buttons approach (a shortcut/lazy way...) rather than typing complicated command lines (based on trial and error)...
    • Oh I see. Yeah, you basically need to program it to do things and deploy. It's good in the end.
    • (you could bulid locally and push to a gh-pages branch, and it would deploy that.)

  6. How often does Github Actions work the first time? yeah, like you don't need to go debugging it.

    • If you mean, first time after I write one, the answer is rarely. Maybe it's just me, but I always need to debug them for a while :smile:

  7. When using GitHub pages, it will also be available online for everyone right? Or is their a way to create a non-local private GitHub page?

    • yes, Github Pages (free version) is public for everyone. If you get one of the advanced plans, I think they can do authentication.
    • Or you can put it on your own web server and add whatever access control you want (that the server supports). This would be pretty common, too.
    • (remember we first build the Sphinx project locally, where no one else could see it. From that stage you can do what you'd like)

  8. How did you change the theme?

    • inside conf.py
    • You can find many awesome themes that could be used instead here: https://sphinx-themes.org/

  9. The link in the first step of exercise GH-Pages-1 doesn't work for me. How can I then get access to the template?

    • Are yo ulogged in to Github? The /generate part goes straight to a form that needs to be logged in
    • The project itself is here: https://github.com/coderefinery/documentation-example/ (after login there should be a Generate button, that goes to the /generate URL)
    • Thanks. Now it works. I was logged in with a different browser than I use for the notes.

  10. Thank you for a great morning!

    • thanks!

:::info

Lunch until xx:00 (12 CEST)

  1. I know that it is a little off-topic but does anybody know how we can include code from a file in mkdocs such as in the example given for sphinx where we used:

    :language: python
:emphasize-lines: 2-3
    • I don't know, but one of the big advantages of Sphinx is it's extreme extensibility and directive-based nature that allows this. mkdocs might have this, but there's much more that might be harder. (the complexity of Sphinx is also probably why it's not used as much as it should be...)

  2. How would I add code (with docstring) to the src folder now and make it appear in an API section on the documentation page?

    • The exercise Sphinx-4 goes over the basics
    • When there are new files, you might need to re-run sphinx-apidoc to regenerate the template files.
    • And make sure that the right autodoc directives are in the apidoc files
    • And make sure that the code is importable to generate the auto stuff
    • (can give more after lunch)+1 (struggling here as well)

  3. Instead of showing code blocks is it also possible to show the output of the code? I'm wondering if it is possible to use this approach to create an online tool of sorts (taking an input and generating an ouput based on that).

    • then the documentation would have to run some code in the background. that is possible. one approach are notebooks. I will look for some examples that do that ...

  4. Should I have made a new branch when I added the documentation.yml-file?

    • when I demonstrated it on stream, I committed directly to "main" but creating a new branch and adding on branch and creating pull request would be nicer in a collaborative setting

  5. Could you explain a bit the last part of the code to deploy Sphinx documentation to github pages? Specifically what those lines of code mean/do: 1.uses: peaceiris/actions-gh-pages@v3 2.github.ref == 'refs/heads/main' 3.github_token: ${{ secrets.GITHUB_TOKEN }} 4.force_orphan: true

    • (we will explain/reply later, now prepping for next lesson ...)
    • (1) says "use this other action to do some stuff for us": (this one: https://github.com/peaceiris/actions-gh-pages).
    • (2) says "if this is the main branch, do the deployment" (you don't want other pull requests deployed as the main documentation).
    • (3) tells the action (in (1)) to use the automatically included token to give permission to push (to the branch gh-pages).
    • (4) says "don't keep history of the gh-pages branch".

  6. A follow-up question to question 15: I am able to make autodoc API for my functions, but I fail to create hyperlinks to one specific function. I tried the :func:pythonmodule.func method, and also .. autofunction:: pythonmodule.func. Can please someone help me? Please see My Doc for the code example.

    • .. autofunction:: is the ReST syntax, not MyST. Try this: {autofunction} mandelbrot.mandelbrot_set
    • This is the equivalent MyST syntax, they can be always be translated between the two. Sometimes you see docs in only one or the other. (also, it's the main point of MyST-markdown: other markdowns don't generally have this power)
    • Thanks! I will try

  7. I have now used the tutorial here: https://sphinx-autoapi.readthedocs.io/en/latest/tutorials.html to generate an autoapi that points to the src. Would that be a good solution rather than making manual api.md?

    • ah that is even better! good find.

Jupyter notebooks

A tool to write and share executable notebooks and data visualization

https://coderefinery.github.io/jupyter/

  1. Is it possible to change folder viewer to a parent directory?

    • Part of the design of Jupyter is it captures the view in the directory it's started. You should restart in the higher directory
    • (this doesn't stop the Python processes inside from accessing stuff outside, though! it's just usability, not security)

  2. I think it is basically a table of contents, headings shows up there

    • ah thanks!

  3. How do I create a markdown cell again ? I miss it.

    • we will show again in a moment
    • m in command mode, there is also a drop down at the top

  4. Is there a link to these keyboard shortcut, I will later forget what to press to create/run cells ?

    • https://coderefinery.github.io/jupyter/interface/#keyboard-shortcuts

  5. Is there a way to increase the (code/markdown) font for my own notebook ? The one in my computer is a bit small for me to see.

    • Settings -> Theme has options for code/editor/UI adjustments seprately. You can also try adjusting font of whole web browser (control + or control -)

  6. How did you copy the table? When I try it just looks like text.

:::info

Exercise until xx:45

https://coderefinery.github.io/jupyter/first-notebook/#an-example-computational-notebook

How did the exercise go?

  1. I understand that press ESCAPE goes to "command" mode but is there a way for me to know if I am in command mode ?

    • If it shows you a cursor inside a cell then you are editing that cell.
    • You can try arrowkeys up and down to see whether it moves cursor or code selection (the blue vertical bar)

  2. Is there a way to go to command mode by mouse clicking ?

    • Yup! Click between hte blue vertical bar and the cell

  3. I cant seem to run the code cells. Instead of a number like this [1], I have a [*] and if I write print('hello') I do not see a result under the cell

    • please tell us what you see instead
    • Markdown cells and "raw" cells don't have the number next to them. Maybe your cell is not a code cell actually?
      • It seems like it was just very slow

  4. Is there a way to see what the python variable x, num_points are, like an "environment/variable tab" in MATLAB or RStudio ? Maybe a way to identify string list, pandas series vs dataframe.

    • There are differen "variable inspectors", let me see what I'd recommend...
    • Help -> Show Contextual Help. Then you click on anything and it shows the value + help. (there are others)
    • Under the extensions search "Variable Inspector"

Notebooks and version control

https://coderefinery.github.io/jupyter/version-control/

  1. Sorry, I have just joined the course today. I wonder whether we are going to cover/have covered pushing the Jupyter notebook to github without the outputs?

    • I'm not sure it's mentioned, but there is a git hook for clearing outputs: https://pypi.org/project/nbstripout/
    • By the various install things towards the bottom, git will remove output before committing. warning: it might be easy to forget on a new computer/clone.

  2. Is nbdime automatically installed when I download Jupyter

    • I think not. (How do I know if I have it ? Is it a python package ?)
      • how to know whether you have it: in your notebook you can try import nbdime - if you see an error, it is not there
    • It is not by default there as far as I know
    • Python package nbdime: pip install nbdime, conda install nbdime, or however you manage packages. it's included as part of the CodeRefinery conda environment

  3. When do you prefer using jupyter over just using github?

    • I think they serve different purposes. Github hosts static files, Jupyter lets you do computing. They can work well together.

  4. Is there a way to export Jupyter notebook as (PDF, .py file. HTML) using the kernel ? I am not so good with command lines...

    • Yep: File> Save and export notebook as. It should export it how it is here.
      • For PDF I think this is needed: https://nbconvert.readthedocs.io/en/latest/install.html#installing-tex
      • downloading $\LaTeX$ file and HTML file did work

:::info

Break until xx:07

:::

:::info

Exercise until xx:30

:::

Feedback, day 5

:::info

News for day 6

How was today for you?

Do you see value in the CodeRefinery video production (for youtube):

Comments on videos:

One good thing about today:

One thing to be improved for next time:

Any other comments:

  1. I've seen jupyter notebook being used in coding courses as material, so that the markdown parts and some code parts are fixed, non-editable, but students can edit certain code cells to do exercises and try out things. How is that done?
    • There's some metadata that can tell the editor "don't let this be editable". https://stackoverflow.com/questions/18162970/read-only-cells-in-ipython-jupyter-notebook
    • (but it's just a user interface thing. the user of course has access to the raw file so can do anything they want that way - this doesn't make it more secure.)

Funding

CodeRefinery is a project within the Nordic e-Infrastructure Collaboration (NeIC). NeIC is an organisational unit under NordForsk.

Privacy

Privacy policy

Follow us

Contact

support@coderefinery.org

Improve this page

Source code