Skip to content

Latest commit

 

History

History
638 lines (616 loc) · 23.8 KB

reproducible_research.org

File metadata and controls

638 lines (616 loc) · 23.8 KB

Reproducible with research with emacs (and org-mode)

1 Introduction [100%]

1.1 Introduction

What do I mean by reproducible research

  • Being able to reproduce experiments
  • Identify the state of the experiments (what has been done/what remains to be done)
  • Understand the process which leads to the current state

Why using emacs ?

  • Because you have all the tools to help you in this editor

Some pre-requisite

  • Linux ubuntu 16.04 / bash

2 Emacs basics [100%]

2.1 General informations

  • Global informations
    • EMACS = Editor MACroS
    • Development started during the 70s
    • Multiple variation GNU Emacs, XEmacs, AquaMacs…
    • Compatible Unix-Like (Linux, macos, …) + windows
    • Configuration = based on a dedicated language: emacs lisp
  • Global key concepts
    • 3 modifier keys = CTRL, MAJ, META
    • META = alt on PC, command on MAC
  • Commands = instructions to run. Can be associated with shortcuts!
  • minibuffer: where commands are launched/messages are indicated

2.2 Graphic overview with a legend

  • 1 file: build.gradle
  • 3 buffers: build.gradle, scratch, Messages
    • scratch is a special buffer for scratch notes
    • Messages is a special buffer which contains all the messages sent by emacs
  • 3 windows: 1 at the top, 2 at the bottom
  • 1 frame: emacs-bin@slemaguer-perso
  • 1 minibuffer indicating the message “Beginning of buffer”

assets/images/emacs_concepts.png

2.3 Major mode / minor mode

  • Emacs adaptation = modes
  • 2 kind of modes: major and minor
  • 1 buffer = 1 major mode + N minor modes
    • major mode = editing behavior (syntax coloring, indentation, …)
    • minor mode = specific adaptations (matching parenthesis, spell/syntax checking, …)
  • Help key = C-h m
    • List all the mode enabled in the buffer + some documentation
    • Example

2.4 Emacs configuration

  • Global
    • Configuration language = elisp (emacs lisp)
    • Programming language ⇒ a little bit of investment....
    • ....but powerful
  • Entry point = ~/.emacs or ~/.emacs.d/init.el
    • You can put everything there…
    • Or import other files
    • Not the focus of this course: a baseline configuration will be provided later

3 Hands-on emacs [100%]

3.1 Environment preparation

  • Pre-configuration
    • Package installation (ubuntu)
sudo apt-get install build-essential libglib2.0-dev imagemagick libgtk-3-dev libpoppler-dev libpoppler-glib-dev libxpm-dev libtiff-dev libjpeg-dev libgif-dev libncurses-dev
# Include profile variables
source ~/.profile
  • For the current session, write in the current terminal
source ~/.profile
echo $PATH # Check that the path is correct

3.2 Compilation/installation/running

bash emacs_install.sh -j 2 $HOME/environment/local
  • Running
emacs

3.3 The most important keys in the world ! (or maybe not but still)

  • Emacs relies massively on shortcuts (but are not mandatory !)
  • M-x = starting a command (that is the only mandatory one)
  • C-h m (mode), C-h k (keys), C-h f (function) = documentation shortcut
  • C-x f (open), C-x s (save), C-x w (save as) = open file/save buffer
  • C-<space> (set/unset selection mark), M-w (copy), C-w (cut), C-y (paste)
  • refcard = https://www.gnu.org/software/emacs/refcards/pdf/refcard.pdf

3.4 Having fun for 5min :D

3.4.1 Try to do this

  • Open some source code file (java, python, c, …)
  • Open some tex file
  • Query the documentation, …

3.4.2 Some feedback ?

  • What is easy ?
  • What is difficult ?
  • Do you have an idea of how to improve

3.4.3 Some questions ?

  • Don’t hesitate to be exhaustive or say if you don’t understand something

4 Hands-on configured emacs [100%]

4.1 Package management

(use-package spaceline-all-the-icons ;; The package to install
  :ensure t ;; Force the installation
  :after spaceline ;; Initialisation after the given package

  ;; Preconfiguration, *before* the package is loaded
  :init

  (setq spaceline-all-the-icons-separator-type 'wave
        spaceline-separator-dir-left '(left . left)
        spaceline-separator-dir-right '(right . right)
        spaceline-minor-modes-separator " ")

  ;; Configuration, *after* the package is loaded
  :config

  (spaceline-all-the-icons-theme)
  (require 'spaceline-config)
  (spaceline-helm-mode)
  (use-package info+
    :ensure t
    :init
    (spaceline-info-mode))

  (use-package fancy-battery
    :ensure t
    :init
    (add-hook 'after-init-hook #'fancy-battery-mode)
    (display-battery-mode -1))

  (spaceline-all-the-icons--setup-paradox)
  )
  • To list available packages use the command: list-packages

4.2 Install a baseline configuration

./assets/images/emacs_configured.png

4.3 Some main functionalities

  • Buffer management
    • Using C-x C-b to open the ibuffer list
  • File/directory management
    • Using C-x C-r to list the recently opened files
    • Using C-x C-d to list directories
  • VC/Project management (see the next section)
    • Main shortcut = <f4>
  • Agenda management (see the section about org-mode)
    • Main shortcut = <f12>

4.4 Having fun for 5min

  • Try to open some source code file (java, python, c, …)
  • Try to open some tex file
  • Try to query the documentation, …
  • Try to spot the differences

4.5 Bonus: configure a server/client architecture

bash server_adapt.sh -j 2 $HOME/environment/local
  • On the configuration part
    1. Uncomment the “server part” in the init.el file
  • Try to run the emacs command

5 Project/scm [100%]

5.1 Project management

The concept of a project is pretty basic - just a folder containing special file. Currently git, mercurial, darcs and bazaar repos are considered projects by default. So are lein, maven, sbt, scons, rebar and bundler projects.

  • Prefix : C-x p (see configuration file)
  • Some useful commands
    • C-c p f = open file in the project
    • C-c p h = open helm to manipulate projectile

5.2 Projectile - example

  • DEMO

5.3 Git

5.4 Magit - example

  • DEMO

5.5 Git-timemachine - example

  • DEMO

6 Org-mode - Introduction [100%]

6.1 General informations

Org mode is for keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system.

  • Also
    • spreadsheet
    • bibtex/reference
    • …

6.2 Task management - gtd principle

./assets/images/GTDcanonical.png

6.3 Task management - An example

6.4 Task management - Rendering

./assets/images/todo_rendering.png

6.5 Task management - Some important shortcuts

6.6 Dash

  • Allows you to have a global view of the project in progress
  • Command = org-dashboard-display
  • Shortcut = <f4> d
  • DEMO

6.7 Agenda

  • Allows you to visualize your “calendar”
  • Command = org-agenda
  • Shortcut = <f12>
  • DEMO

6.8 Capturing

  • Allows to interrupt briefly your workflow to add a reminder in your global todolist
  • To extend this nice feature, see section (Org-mode > Capturing) in the configuration file (~/.emacs.d/main.org)
  • DEMO

7 Org-mode - Writing documents [100%]

7.1 The header

#+TITLE: Reproducible with research with emacs (and org-mode)
#+AUTHOR: Sébastien Le Maguer
#+EMAIL: [email protected]
#+DATE: 12 May 2017
  • Can be adapted for the output (Example Latex)
# ##############################################################################################################################
#+STARTUP: beamer
#+STARTUP: oddeven
#+LaTeX_CLASS: beamer
#+LaTeX_CLASS_OPTIONS: [9pt]
#+SELECT_TAGS: export
#+EXCLUDE_TAGS: noexport
#+BEAMER_THEME: progressbar
#+LATEX_CMD: xelatex
#+LaTeX_HEADER: \hypersetup{
#+LaTeX_HEADER:   colorlinks = true,
#+LaTeX_HEADER:   linkcolor = black
#+LaTeX_HEADER: }
#+latex_header: \usepackage{subfig}
#+LaTeX_HEADER: \usepackage{bibentry}
#+LaTeX_HEADER: \usepackage{multimedia}
#+LaTeX_HEADER: \makeatletter
#+LaTeX_HEADER: \let\@mycite\@cite
#+LaTeX_HEADER: \def\@cite#1#2{{\hypersetup{linkcolor=green!60!black}[{#1\if@tempswa , #2\fi}]}}
#+LaTeX_HEADER: \makeatother
# ##############################################################################################################################

7.2 The body - architecture

* Header 1
** Header 2
*** Header 3
...
  • Lists
- list 1
  - list 11
    + list111 with other bullets
    + list112 with other bullets
  - list 12
    1. list121 as enumerate
    2. list122 as enumarate

7.3 The body - content formatting

/Emphasize/ (italics), *strongly* (bold), and */very strongly/* (bold italics).
  • Paragraph separated by an empty line
  • Horizontal rule is done using 5 dashes
  • Links are in brackets
  • Mathematical formula exactly like latex :
\begin{equation}
f_1(x) = \frac{1}{N} \sum_{r=1}^R log(x(r)^2)
\end{equation}

\begin{equation} f_1(x) = \frac{1}{N} ∑r=1^R log(x(r)^2) \end{equation}

7.4 The body - table

  • Command = org-table-create
  • Row separator = new line
  • Column separator = “|”
  • horizontal line = “|—+—|”
    • ”|” = first/end column indicator
    • ”-” = filler
    • ”+” = column separator

7.5 The body - images

#+CAPTION: (extracted from https://en.wikipedia.org/wiki/Getting_Things_Done)
#+NAME:   fig:gtd
#+ATTR_HTML: :width 75%
#+ATTR_LATEX: :width 0.75\linewidth
[[./assets/images/GTDcanonical.png]]
  • do you remember this slide “Task management - gtd principle” ?

7.6 The body - interesting shortcuts

  • More complete information at = http://orgmode.org/orgcard.pdf
  • In the text
    • C-c C-l create a link
  • Header
    • C-<return> create an item at the same level of the current header (no list)
    • M-<left/right-arrow> Change bullet and numbering type
  • List
    • M-<return> create an item at the same level in the list or header
    • Shift-<left/right-arrow> Change bullet and numbering type
  • In a table
    • <pipe><hypen> followed by tab = create a table or complete the row

7.7 Exporting

  • Different kind of exporter (HTML, reveal, LaTeX, Beamer, Markdown, …)
  • Command to export = C-c C-e then a dedicated buffer opens:
  • DEMO

7.8 Having fun for 5min

  • Let’s try to write a small document
  • Try to use everything we wrote before (a table, an image, …)
  • Export it in latex and/or HTML

8 Org-mode - Literate programming [100%]

8.1 What is literate programming

  • Introduced by D. Knuth (FIXME: ref)

Literate programming is a programming paradigm introduced by Donald Knuth in which a program is given as an explanation of the program logic in a natural language, such as English, interspersed with snippets of macros and traditional source code, from which a compilable source code can be generated.

8.2 How to handle it with org-mode

  • org-babel is handling that
  • Insert code using the following syntax
  • Including a script file using the following syntax
#+include: "~/work/courses/current/teaching/reproducible-research/assets/files/example_src.org" src org :eval never-export
  • Rendering results:

8.3 How to handle it with org-mode - actually working

  • org-babel is handling that
  • Insert code using the following syntax
  • Including a script file using the following syntax
#+include: "~/work/courses/current/teaching/reproducible-research/assets/files/example_src.org" src org :eval never-export
  • Rendering results:

8.4 Dealing with results/source/…

8.5 Deal - output

8.6 Having fun for 5min

  • Update the previous documents to add
    • A source for which you just want to have the source visible in the exported document
    • A source for which you just want to have the result visible in the exported document
    • A source for which you want to have both the source and the result visible in the exported document

9 Org-mode/emacs as a “pandoc” equivalent [100%]

9.1 But…why ?!

  • Problem: how to deal with massive data/time consuming experiments?
  • Solution:
    1. Design the experiments using workflow management tools (make, gradle, …)
    2. In the last experimental stage Format the results in a certain way (csv, ....)
    3. Design the org document assuming these results are available
    4. Call emacs to render the org-mode
  • Preliminary informations:
    • The default configuration is not used !
    • An emacs configuration is embedded in the project
    • Gradle is used to control the process
    • Scripts are based on the one proposed in https://github.com/nhoffman/org-export

9.2 A project example - architecture

.
├── build.gradle
├── gradle
├── gradlew
├── gradlew.bat
├── scripts
│   ├── org-export-beamer.el
│   ├── org-export-cli.el
│   ├── org-export-html.el
│   ├── org-export-latex.el
│   └── org-export-reveal.el
├── settings.gradle
└── src
    ├── assets
    ├── biblio.bib
    └── paper.org
  • We provide....
    • Gradle configuration (build.gradle, settings.gradle gradle*)
    • Emacs exporter scripts (scripts/*.el)
  • We need…
    • the source of the document (src/paper.org)
    • the optional bibliography (src/biblio.bib)
    • the assets which are downloaded into src/assets
  • The output will be put in the build directory

9.3 A project example - org part

  • see src/paper.org
  • Nothing different than before except.....
    • we assume the presence of the assets

9.4 A project example - gradle part

  • That is where the process is put into place
  • First retrieve the assets
task downloadAssets(type: Download) {
    src "http://www.coli.uni-saarland.de/~slemaguer/attachments/example_gradle_emacs/values.csv"
    dest "src/assets/values.csv"
    overwrite false
}
  • Then process the assets
task processAssetsLatex(type: Copy) {
    from fileTree('src').include('assets/**')
    into "$buildDir/${rootProject.name}/latex"
}
  • Finally generate the document (latex in this example)
task org2latex(type: Exec, dependsOn: "processAssetsLatex") {
    def orgFile = file("src/slides.org")
    def pdfFile = file("$buildDir/${rootProject.name}/latex/index.pdf")
    inputs.files orgFile, processAssetsLatex
    outputs.files pdfFile

    commandLine "emacs-bin", "-q", "--no-site-file", "--evaluate", "--script", "scripts/org-export-beamer.el", "--infile", orgFile, "--outfile", pdfFile
}

9.5 A project example - to run

  • To run just run the following command:
./gradlew b
  • Your turn:
    • create a project based on the current one
    • Try to generate some documentation

10 Conclusion [100%]

10.1 Summary

  • (Quick) Introduction to emacs
    • Main concepts: buffer, files, windows, commands, …
    • Main commands: M-x, C-h m, C-h k, …
    • Package management
    • A baseline configuration ⇒ you can modify it :D
  • (Quick) Introduction to org-mode
    • Todo lists + Agenda management
    • Writing documents using org-mode
    • Exporting documents using org-mode
  • Reproducible research
    • Litterate programming with org-mode
    • Invocating org-mode to generate documentation from a project

10.2 How to go further [some interesting links]