serhii.net

Format

• GBIF Infrastructure: Data processing has a detailed description of the flow
  • occurrences.txt is an improved/cleaned/formalized verbatim.txt
  • metadata
    • meta.xml has list of all colum data types etc.
      • for all files in the zip!
      • columns links lead to DCMI: DCMI Metadata Terms
    • metadata.xml is things like download doi, license, number of rows, etc.
• .zips are in Darwin format: FAQ
  • Because there are cases when both single and double quotes etc., and neither '/" as quotechar work.

  df = vx.read_csv(DS_LOCATION,convert="verbatim.hdf5",progress=True, sep="\t",quotechar=None,quoting=3,chunk_size=500_000)
  

Tools

• GBIF .zip parser lib:
  • BelgianBiodiversityPlatform/python-dwca-reader: 🐍 A Python package to read Darwin Core Archive (DwC-A) files.
  • Tried it, took a long time both for the zip and directory, so I gave up
• gbif/pygbif: GBIF Python client
  • API client, can also do graphs etc., neat!

Analysis

Things to try:

• limit number of columns through pd.read_csv.usecols()¹ to the ‘interesting’ ones
  • optionally take a smaller subset of the dataset and drop all NaNs
  • take column indexes from meta.xml
  • See if someone already did this:BelgianBiodiversityPlatform/python-dwca-reader: 🐍 A Python package to read Darwin Core Archive (DwC-A) files.

It all started with the menu bar disappearing on qutebrowser but not firefox:

Broke everything when trying to fix it, leading to not working vim bindings in lab. Now I have vim bindings back and can live without the menu I guess.

It took 4h of very frustrating trial and error that I don’t want to document anymore, but - the solution to get vim bindings inside jupyterlab was to use the steps for installing through jupyter of the extension for notebooks, not the recommended lab one.
Installation · lambdalisue/jupyter-vim-binding Wiki:

mkdir -p $(jupyter --data-dir)/nbextensions/vim_binding
jupyter nbextension install https://raw.githubusercontent.com/lambdalisue/jupyter-vim-binding/master/vim_binding.js --nbextensions=$(jupyter --data-dir)/nbextensions/vim_binding
jupyter nbextension enable vim_binding/vim_binding

I GUESS the issue was that previously I didn’t use --data-dir, and tried to install as-is, which led to permission hell. Me downgrading -lab at some point also helped maybe.

The recommended jupyterlab-vim package installed (through pip), was enabled, but didn’t do anything: jwkvam/jupyterlab-vim: Vim notebook cell bindings for JupyterLab.

Also, trying to install it in a clean virtualenv and then doing the same with pyenv was not part of the solution and made everything worse.

Useful bits

Getting paths for both -lab and classic:

> jupyter-lab paths
Application directory:   /home/sh/.local/share/jupyter/lab
User Settings directory: /home/sh/.jupyter/lab/user-settings
Workspaces directory: /home/sh/.jupyter/lab/workspaces

> jupyter --paths
config:
    /home/sh/.jupyter
    /home/sh/.local/etc/jupyter
    /usr/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /home/sh/.local/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /home/sh/.local/share/jupyter/runtime

Removing ALL packages I had locally:

pip uninstall --yes jupyter-black jupyter-client jupyter-console jupyter-core jupyter-events jupyter-lsp jupyter-server jupyter-server-terminals jupyterlab-pygments jupyterlab-server jupyterlab-vim jupyterlab-widgets
pip uninstall --yes jupyterlab nbconvert nbextension ipywidgets ipykernel nbclient nbclassic ipympl notebook 

To delete all extensions: jupyter lab clean --all

Related: 230606-1428 pip force reinstall package

Versions of everything

> pip freeze | ag "(jup|nb|ipy)"
ipykernel==6.23.1
ipython==8.12.2
ipython-genutils==0.2.0
jupyter-client==8.2.0
jupyter-contrib-core==0.4.2
jupyter-contrib-nbextensions==0.7.0
jupyter-core==5.3.0
jupyter-events==0.6.3
jupyter-highlight-selected-word==0.2.0
jupyter-nbextensions-configurator==0.6.3
jupyter-server==2.6.0
jupyter-server-fileid==0.9.0
jupyter-server-terminals==0.4.4
jupyter-server-ydoc==0.8.0
jupyter-ydoc==0.2.4
jupyterlab==3.6.4
jupyterlab-pygments==0.2.2
jupyterlab-server==2.22.1
jupyterlab-vim==0.16.0
nbclassic==1.0.0
nbclient==0.8.0
nbconvert==7.4.0
nbformat==5.9.0
scipy==1.9.3
widgetsnbextension==4.0.7

Bad vibes screenshot of a tiny part ofhistory | grep jup

“One of the 2.5 hours I’ll never get back”, Serhii H. (2023). Oil on canvas
Kitty terminal, scrot screenshotting tool, bash.

Docker image runs a Python script that uses print() a lot, but docker logs is silent because python print() uses buffered output, and it takes minutes to show.

Solution¹: tell python not to do that through an environment variable.

docker run --name=myapp -e PYTHONUNBUFFERED=1 -d myappimage

TIL about pip install packagename --force-reinstall¹

• Since then, we have witnessed an increased research interest into
• Technical developments have gradually found their way into
• comprehensive but not exhaustive review

…

(On a third thought, I realized how good ChatGPT is at suggesting this stuff, making this list basically useless. Good news though.)

I love Dia, and today I discovered that:

• It can do layers! That work as well as expected in this context
• To save an antialiased PNG, you have to explicitly pick “png (antialiased)” when exporting, it’s in the middle of the list and far away from all the other flavours of .png extensions

Before and after:

The Radar chart and its caveats: “radar or spider or web chart” (c)

… are best done in plotly:

• Radar charts in Python
• Polar charts in Python

For a log axis:

fig.update_layout(
    template=None,
    polar = dict(
        radialaxis = dict(type="log"),
    )

EDIT: for removing excessive margins, use

fig.update_layout( margin=dict(l=20, r=20, t=20, b=20), )

Trivial option: Label data points with Seaborn & Matplotlib | EasyTweaks.com

TL;DR

for i, label in enumerate (data_labels):
    ax.annotate(label, (x_position, y_position))

BUT! Overlapping texts are sad:

SO sent me to the library Home · Phlya/adjustText Wiki and it’s awesome

fig, ax = plt.subplots()
plt.plot(x, y, 'bo')
texts = [plt.text(x[i], y[i], 'Text%s' %i, ha='center', va='center') for i in range(len(x))]
# adjust_text(texts)
adjust_text(texts, arrowprops=dict(arrowstyle='->', color='red'))

Not perfect but MUCH cleaner:

More advanced tutorial: adjustText/Examples.ipynb at master · Phlya/adjustText · GitHub

Pypy doesn’t have the latest version, which has:

• min_arrow_len
• some other bits like expand
• more: adjustText/init.py at master · Phlya/adjustText · GitHub

How To Apply Conditional Formatting Across An Entire Row;

• Basics
  • $A$1 is a direct reference to A1, that won’t move if formula is applied to a range
  • ISBLANK(..) means cell is empty
  • AND(c1,c2,...,cN), OR(c1,c2,...,cN)
• Cond. formatting bits
  • If you need to pick the “current” cell, it’s the first cell of the range without $s
  • =$U1=1 is “if U of the current row is equal to 1” (then you can color the entire row green or whatever)
  • The order of the rules is precedence, that is rule N will overwrite N+1

This contains the entire list of all datasets I care about RE [230529-1413 Plants datasets taxonomy] for 230507-2308 230507-1623 Plants paper notes

• GBIF

  • Search
  • Seems to be a central place for all similar and not-similar datasets with a centralized API

• Pl@ntNet

  • FULL dataset: Pl@ntNet observations
    • 
    • https://www.gbif.org/occurrence/search?dataset_key=7a3679ef-5582-4aaa-81f0-8c2545cafc81
    • Contains num of species and yet another exploring thing:World flora: Species - Pl@ntNet identify
  • datamanager @ data
    • Not sure what this is but sounds interesting, maybe connected to downloading their stuff
    • 
  • OpenReview review of their paper: Pl@ntNet-300K: a plant image dataset with high label ambiguity and a long-tailed distribution | OpenReview

• https://flora-on.pt/

  • portuguese-only?

  • Flora-On: Flora de Portugal Interactiva. (2023). Sociedade Portuguesa de Botânica. www.flora-on.pt. Consulta efectuada em 29-5-2023.

  • Used in <@herediaLargeScalePlantClassification2017 (2017) z/d>

• iNaturalist-xxx

  • Has 2017-2018-… variants
    • iNaturalist 2018 Benchmark (Image Classification) | Papers With Code
  • train/val plants numbers in the paper, test sets unclear and needs to be downloaded to calculate
    • consensus (kaggle, paperswithcode etc.) seems to be train/val use for all of them
  • Competitions
    • All: visipedia/inat_comp: iNaturalist competition details
    • 2017:
      • (seems to be the coolest/most_used dataset)
      • Plantae, Insecta,Aves,Reptilia,Mammalia,Fungi,Amphibia,Mollusca,Animalia,Arachnida,Actinopterygii,Chromista
        • rows: supercategory (animalia, etc.), per-image attribution and license, GBIF (id, link, ..), others
        • Sample rows from dataset: iNat competition GBIF info - Google Sheets
      • <@vanhornINaturalistSpeciesClassification2018 (2018) z/d>
    • Added the rest to the spreadsheet, 2019 is the only one with missing per-category things

• INaturalist

  • Website is mirrored in GBIF
  • You can filter stuff in the website and get numbers, GBIF has a viewer that also can do things but not list of species
  • ‘research grade’ are the good observations: Help · iNaturalist
  • SPECIES!=IDENTIFIER! The latter seems to contain also families etc.
  • “iNaturalist Research-grade Observations”
    • On GBIF is the entire dataset: Occurrence search
      • This is the filter they used on iNaturalist: Observations · iNaturalist
    • Plants-only GBIF: Occurrence search
      • GBIF file with ‘species lst’: Download
        • It contains 129k records, but they aren’t just species, also families etc.
      • Corresponding iNat filter: Observations · iNaturalist
  • Plant seedlings dataset
    • https://github.com/TheSaintIndiano/Plant-Seedlings-Classification/blob/master/Seedlings.ipynb
    • num of images based on that doc: 390+ 611+ 231+ 496+ 221+ 475+ 287+ 385+ 221+ 654+ 516+ 263

• Flora Incognita

  • A lot of conflicting info!
    • Main site: Flora Incognita | EN – The Flora Incognita app – Interactive plant species identification
      • 16k+ plant types
  • 2020 Paper giving some numbers about the dataset Multi-view classification with convolutional neural networks | PLOS ONE
    • 
    • 775 classes
  • 2021 paper about app with some details about the num of plants and comparison to other datasets: The Flora Incognita app – Interactive plant species identification - Mäder - 2021 - Methods in Ecology and Evolution - Wiley Online Library
    • 4851 classes in table 1
  • Flora Capture: a citizen science application for collecting structured plant observations - PMC
  • Flowers, leaves or both? How to obtain suitable images for automated plant identification - PubMed is about the number of images for classification, and

    we curated a partly crowd-sourced image dataset, comprising 50,500 images of 101 species.

  • Paper using FI app, not FI dataset itself! Plant image identification application demonstrates high accuracy in Northern Europe | AoB PLANTS | Oxford Academic

    2494 observations with 3199 images from 588 species, 365 genera and 89 families

• PlantCLEF

  • PlantCLEF2021
    • PlantCLEF 2021 | ImageCLEF / LifeCLEF - Multimedia Retrieval in CLEF
    • 
    • TRAIN: 321270 herb sheets, 6316 field photos, 354 observations of both herbarium+field; TEST SET: “3,186 photos in the field related to 638 plant observations (about 5 pictures per plants on average).”
    • Dataset here: https://zenodo.org/record/3658343#.ZDe9x1qxVhE

REALLY NICE OVERVIEW PAPER with really good overview of the existing datasets! Frontiers | Plant recognition by AI: Deep neural nets, transformers, and kNN in deep embeddings

• Flavia

• Datasets | The Leaf Genie has list of leaf datasets! TODO

• Herbarium 2021

  • Huge ds and paper linking to smaller ones - preliminarily added them to the spreadsheet
  • [[2105.13808] The Herbarium 2021 Half-Earth Challenge Dataset](https://arxiv.org/abs/2105.13808
  • <@delutioHerbarium2021HalfEarth2021 (2021) z/d>)

Next steps

Spreadsheet

• Update it for all the sub-datasets if practical - e.g. web and friends if needed
• Done

Datasets

• Nice picture of who stole from whom
• Done