serhii.net

The Radar chart and its caveats: “radar or spider or web chart” (c)

… are best done in plotly:

• Radar charts in Python
• Polar charts in Python

For a log axis:

fig.update_layout(
    template=None,
    polar = dict(
        radialaxis = dict(type="log"),
    )

EDIT: for removing excessive margins, use

fig.update_layout( margin=dict(l=20, r=20, t=20, b=20), )

Trivial option: Label data points with Seaborn & Matplotlib | EasyTweaks.com

TL;DR

for i, label in enumerate (data_labels):
    ax.annotate(label, (x_position, y_position))

BUT! Overlapping texts are sad:

SO sent me to the library Home · Phlya/adjustText Wiki and it’s awesome

fig, ax = plt.subplots()
plt.plot(x, y, 'bo')
texts = [plt.text(x[i], y[i], 'Text%s' %i, ha='center', va='center') for i in range(len(x))]
# adjust_text(texts)
adjust_text(texts, arrowprops=dict(arrowstyle='->', color='red'))

Not perfect but MUCH cleaner:

More advanced tutorial: adjustText/Examples.ipynb at master · Phlya/adjustText · GitHub

Pypy doesn’t have the latest version, which has:

• min_arrow_len
• some other bits like expand
• more: adjustText/init.py at master · Phlya/adjustText · GitHub

How To Apply Conditional Formatting Across An Entire Row;

• Basics
  • $A$1 is a direct reference to A1, that won’t move if formula is applied to a range
  • ISBLANK(..) means cell is empty
  • AND(c1,c2,...,cN), OR(c1,c2,...,cN)
• Cond. formatting bits
  • If you need to pick the “current” cell, it’s the first cell of the range without $s
  • =$U1=1 is “if U of the current row is equal to 1” (then you can color the entire row green or whatever)
  • The order of the rules is precedence, that is rule N will overwrite N+1

This contains the entire list of all datasets I care about RE [230529-1413 Plants datasets taxonomy] for 230507-2308 230507-1623 Plants paper notes

• GBIF

  • Search
  • Seems to be a central place for all similar and not-similar datasets with a centralized API

• Pl@ntNet

  • FULL dataset: Pl@ntNet observations
    • 
    • https://www.gbif.org/occurrence/search?dataset_key=7a3679ef-5582-4aaa-81f0-8c2545cafc81
    • Contains num of species and yet another exploring thing:World flora: Species - Pl@ntNet identify
  • datamanager @ data
    • Not sure what this is but sounds interesting, maybe connected to downloading their stuff
    • 
  • OpenReview review of their paper: Pl@ntNet-300K: a plant image dataset with high label ambiguity and a long-tailed distribution | OpenReview

• https://flora-on.pt/

  • portuguese-only?

  • Flora-On: Flora de Portugal Interactiva. (2023). Sociedade Portuguesa de Botânica. www.flora-on.pt. Consulta efectuada em 29-5-2023.

  • Used in <@herediaLargeScalePlantClassification2017 (2017) z/d>

• iNaturalist-xxx

  • Has 2017-2018-… variants
    • iNaturalist 2018 Benchmark (Image Classification) | Papers With Code
  • train/val plants numbers in the paper, test sets unclear and needs to be downloaded to calculate
    • consensus (kaggle, paperswithcode etc.) seems to be train/val use for all of them
  • Competitions
    • All: visipedia/inat_comp: iNaturalist competition details
    • 2017:
      • (seems to be the coolest/most_used dataset)
      • Plantae, Insecta,Aves,Reptilia,Mammalia,Fungi,Amphibia,Mollusca,Animalia,Arachnida,Actinopterygii,Chromista
        • rows: supercategory (animalia, etc.), per-image attribution and license, GBIF (id, link, ..), others
        • Sample rows from dataset: iNat competition GBIF info - Google Sheets
      • <@vanhornINaturalistSpeciesClassification2018 (2018) z/d>
    • Added the rest to the spreadsheet, 2019 is the only one with missing per-category things

• INaturalist

  • Website is mirrored in GBIF
  • You can filter stuff in the website and get numbers, GBIF has a viewer that also can do things but not list of species
  • ‘research grade’ are the good observations: Help · iNaturalist
  • SPECIES!=IDENTIFIER! The latter seems to contain also families etc.
  • “iNaturalist Research-grade Observations”
    • On GBIF is the entire dataset: Occurrence search
      • This is the filter they used on iNaturalist: Observations · iNaturalist
    • Plants-only GBIF: Occurrence search
      • GBIF file with ‘species lst’: Download
        • It contains 129k records, but they aren’t just species, also families etc.
      • Corresponding iNat filter: Observations · iNaturalist
  • Plant seedlings dataset
    • https://github.com/TheSaintIndiano/Plant-Seedlings-Classification/blob/master/Seedlings.ipynb
    • num of images based on that doc: 390+ 611+ 231+ 496+ 221+ 475+ 287+ 385+ 221+ 654+ 516+ 263

• Flora Incognita

  • A lot of conflicting info!
    • Main site: Flora Incognita | EN – The Flora Incognita app – Interactive plant species identification
      • 16k+ plant types
  • 2020 Paper giving some numbers about the dataset Multi-view classification with convolutional neural networks | PLOS ONE
    • 
    • 775 classes
  • 2021 paper about app with some details about the num of plants and comparison to other datasets: The Flora Incognita app – Interactive plant species identification - Mäder - 2021 - Methods in Ecology and Evolution - Wiley Online Library
    • 4851 classes in table 1
  • Flora Capture: a citizen science application for collecting structured plant observations - PMC
  • Flowers, leaves or both? How to obtain suitable images for automated plant identification - PubMed is about the number of images for classification, and

    we curated a partly crowd-sourced image dataset, comprising 50,500 images of 101 species.

  • Paper using FI app, not FI dataset itself! Plant image identification application demonstrates high accuracy in Northern Europe | AoB PLANTS | Oxford Academic

    2494 observations with 3199 images from 588 species, 365 genera and 89 families

• PlantCLEF

  • PlantCLEF2021
    • PlantCLEF 2021 | ImageCLEF / LifeCLEF - Multimedia Retrieval in CLEF
    • 
    • TRAIN: 321270 herb sheets, 6316 field photos, 354 observations of both herbarium+field; TEST SET: “3,186 photos in the field related to 638 plant observations (about 5 pictures per plants on average).”
    • Dataset here: https://zenodo.org/record/3658343#.ZDe9x1qxVhE

REALLY NICE OVERVIEW PAPER with really good overview of the existing datasets! Frontiers | Plant recognition by AI: Deep neural nets, transformers, and kNN in deep embeddings

• Flavia

• Datasets | The Leaf Genie has list of leaf datasets! TODO

• Herbarium 2021

  • Huge ds and paper linking to smaller ones - preliminarily added them to the spreadsheet
  • [[2105.13808] The Herbarium 2021 Half-Earth Challenge Dataset](https://arxiv.org/abs/2105.13808
  • <@delutioHerbarium2021HalfEarth2021 (2021) z/d>)

Next steps

Spreadsheet

• Update it for all the sub-datasets if practical - e.g. web and friends if needed
• Done

Datasets

• Nice picture of who stole from whom
• Done

jupyter-black · PyPI

pip install jupyter-black

To load:

%load_ext jupyter_black

It will automatically format all correct python code in the cells!

NB works much, much better with jupyterlab, in the notebook version it first executes the cell, then does black and hides cell output. It does warn about that everywhere though.

Old code I wrote for making ds.corr() more readable, looked for it three times already ergo its place is here.

Basically: removes all small correlations, and optionally plots a colorful heatmap of that.

def plot_corr(res:pd.DataFrame):
        import seaborn as sns
        sns.heatmap(res, annot=True,fmt=".1f",cmap="coolwarm")
    
def get_biggest_corr(ds_corr: pd.DataFrame, limit: float=0.8, remove_diagonal=True, remove_nans=True,plot=False) -> pd.DataFrame:
	import numpy as np  # just in case
    res =  ds_corr[(ds_corr>limit) | (ds_corr<-limit)]
    if remove_diagonal:
        np.fill_diagonal(res.values, np.nan)
    if remove_nans:
        res = res.dropna(how='all', axis=0)
        res = res.dropna(how='all', axis=1)
    if plot:
        plot_corr(res)
    else:
        return res

Intro

I like seaborn but kept googling the same things and could never get any internal ‘consistency’ in it, which led to a lot of small unsystematic posts¹ but I felt I was going in circles. This post is an attempt to actually read the documentation and understand the underlying logic of it all.

I’ll be using the context of my “Informationsvisualisierung und Visual Analytics 2023” HSA course’s “Aufgabe 6: Visuelle Exploration multivariater Daten”, and the dataset given for that task: UCI Machine Learning Repository: Student Performance Data Set:

This data approach student achievement in secondary education of two Portuguese schools. The data attributes include student grades, demographic, social and school related features) and it was collected by using school reports and questionnaires

Goal:

• Mental picture of the different important architectural parts (figure/axis-level functions)
• Clarity about where are matplotlib things exposed
• Central place for the things I need every time I do seaborn stuff, that are currently distributed in many small posts

I’m not touching the seaborn.objects interface as the only place I’ve seen it mentioned is the official docu and I’m not sure it’s worth digging into for now.

Basics

An introduction to seaborn — seaborn 0.12.2 documentation

Themes and setting the (default) theme

# sets default theme that looks nice
# and used in all pics of the tutorial
sns.set_theme() 

• Themes: darkgrid (default), whitegrid, dark, white, and ticks.
• Refs:
  • Tutorial / list of themes: Controlling figure aesthetics — seaborn 0.12.2 documentation
  • seaborn.set_theme — seaborn 0.12.2 documentation

Links

• API reference — seaborn 0.12.2 documentation
  • is very logically built and is the best list of ‘what exists’
• Intro / tutorials:
  • An introduction to seaborn — seaborn 0.12.2 documentation
  • Overview of seaborn plotting functions — seaborn 0.12.2 documentation:
  • Controlling figure aesthetics — seaborn 0.12.2 documentation
  • Building structured multi-plot grids — seaborn 0.12.2 documentation

Figure-level vs. axes-level functions²

Overview of seaborn plotting functions — seaborn 0.12.2 documentation:

Basics

Functions can be:

• “axes-level”: They plot data onto a single matplotlib.axes.Axes object and return it
  • Contains the legend on the plot

  • The axes-level functions are written to act like drop-in replacements for matplotlib functions. While they add axis labels and legends automatically, they don’t modify anything beyond the axes that they are drawn into. That means they can be composed into arbitrarily-complex matplotlib figures with predictable results.

• “figure-level”: interface through a seaborn object that manages the figure
  • (usually a FacetGrid)
  • Each module has a single figure-level function that creates/accesses axes-level ones (through the kind=xxx parameter)
  • Have the col= and row= params that automatically create subplots!
  • They take care of their own legend

  • The figure-level functions wrap their axes-level counterparts and pass the kind-specific keyword arguments (such as the bin size for a histogram) down to the underlying function. That means they are no less flexible, but there is a downside: the kind-specific parameters don’t appear in the function signature or docstring

Special cases:

• sns.jointplot()³ has one plot with distributions around it and is a JointGrid
• sns.pairplot()⁴ “visualizes every pairwise combination of variables simultaneously” and is a PairGrid

In the pic above, the figure-level functions are the blocks on top, their axes-level functions - below. (TODO: my version of that pic with the kind=xxx bits added)

Customization

Figure-level

The returned seaborn.FacetGrid can be customized in some ways (all examples here from that documentation link).

FacetGrid customization params

g.map_dataframe(sns.scatterplot, x="total_bill", y="tip")
g.set_axis_labels("Total bill ($)", "Tip ($)")
g.set_titles(col_template="{col_name} patrons", row_template="{row_name}")
g.set(xlim=(0, 60), ylim=(0, 12), xticks=[10, 30, 50], yticks=[2, 6, 10])
g.tight_layout()
g.savefig("facet_plot.png")

Accessing underlying matplotlib objects

It’s possible to access the underlying matplotlib axes:

g = sns.FacetGrid(tips, col="sex", row="time", margin_titles=True, despine=False)
g.map_dataframe(sns.scatterplot, x="total_bill", y="tip")
g.figure.subplots_adjust(wspace=0, hspace=0)
for (row_val, col_val), ax in g.axes_dict.items():
    if row_val == "Lunch" and col_val == "Female":
        ax.set_facecolor(".95")
    else:
        ax.set_facecolor((0, 0, 0, 0))

And generally access matplotlib stuff:

• ax: The matplotlib.axes.Axes when no faceting variables are assigned.
• axes: An array of the matplotlib.axes.Axes objects in the grid.
• axes_dict: A mapping of facet names to corresponding matplotlib.axes.Axes.
• figure: Access the matplotlib.figure.Figure object underlying the grid (formerly fig)
• legend: The matplotlib.legend.Legend object, if present.

FacetGrid.set()

(Previously: 230515-2257 seaborn setting titles etc. with matplotlib set)

FacetGrid.set() is used from time to time in the tutorial (e.g. .set(title="My title"), especially in Building structured multi-plot grids) but never explicitly explained; in its documentation, there’s only “Set attributes on each subplot Axes”.

It sets attributes for each subplot’s matplotlib.axes.Axes. Useful ones are:

• title for plot title (set_title())
• xticks,yticks
• set_xlabel(), set_ylabel (but not sequentially as return value is not the ax)

Axis-level functions + adding them to a matplotlib Figure

Axis-level functions “can be composed into arbitrarily complex matplotlib figures”.

Practically:

fig, axs = plt.subplots(2)
sns.heatmap(..., ax=axs[0])
sns.heatmap(..., ax=axs[1])

Specifying figure sizes

Documentation has an entire section on it⁵, mostly reprasing and stealing screenshots from it.

Axis-level

For axis-level functions, the size of the plot is determined by the size of the Figure it is part of and the axes layout in that figure. You basically use what you would do in matplotlib, relevant being:

• the global rcParams: Customizing Matplotlib with style sheets and rcParams — Matplotlib 3.7.1 documentation
• or calling a method on the figure object (e.g. matplotlib.Figure.set_size_inches())

Figure-level functions

TL;DR they have FacetGrid’s’ height= and aspect=(ratio; 0.75 means 5 cells high, 4 cells wide) params that work per subplot.

Figure-level functions’ size has differences:

• the functions themselves have parameters to control the figure size (although these are actually parameters of the underlying FacetGrid that manages the figure)
• these parameters, height and aspect, work like this: width = height * aspect
  • by default, subplots are square
• The parameters correspond to the size of each subplot, not the overall figure

Modules

Blocks doing similar kinds of plots, each with a figure-level function and multiple axis-level ones. Listed in the API reference.⁶

• Distribution plots
  • displot is the figure-level interface
    • ! $\neq$ disTplot that is deprecated
    • 
  • histplot: Plot univariate or bivariate histograms to show distributions of datasets.
  • kdeplot :Plot univariate or bivariate distributions using kernel density estimation.
  • Less useful to me now:
    • ecdfplot: Plot empirical cumulative distribution functions.
    • rugplot: add ticks to axes with the distribution, usually in addition to other plots
• Categorical plots
  • seaborn.catplot is the figure-level interface
  • 
• Regression plots
  • seaborn.relplot
  • scatterplot (with kind="scatter"; the default)
  • lineplot (with kind="line")
• Matrix plots
  • Heatmap
  • Clustermap

And again, the already mentioned special cases, now with pictures:

• sns.jointplot()³ has one plot with distributions around it and is a JointGrid:
• sns.pairplot()⁴ “visualizes every pairwise combination of variables simultaneously” and is a PairGrid:
  

Design

Marks

The parameters for marks are described better in the tutorial than I ever could: Properties of Mark objects — seaborn 0.12.2 documentation:

• Coordinates
• Colors
• Marker/line styles
• Size
• Text
• Align, size, offset

TODO my main remaining question is where/how do I set this? Can this be done outside the seaborn.objects interface I don’t want to learn.

Markers

• Marker size Pass e.g. s=30 to the plotting function. (size= would be a column name)
• Marker style: you are infinitely flexible actually! And this even goes in the legend
• 

  sns.scatterplot(
  
      style="is_available",
      # marker=MarkerStyle("o", "left"),
      markers={True: MarkerStyle("o", "left"), False: MarkerStyle("o", "right")},
  )
  

  • See matplotlib Marker reference — Matplotlib 3.7.1 documentation

Individual questions/topics

Colors, palettes, themes etc

• SNS:
  • Big picture: Controlling figure aesthetics — seaborn 0.12.2 documentation
  • Theory+howto: Choosing color palettes — seaborn 0.12.2 documentation
    • seaborn.color_palette — seaborn 0.12.2 documentation
• Matplotlib:
  • A lot of theory + list of seaborn’s palettes: Choosing Colormaps in Matplotlib — Matplotlib 3.7.1 documentation
  • List of pre-existing ones: Colormap reference — Matplotlib 3.7.1 documentation

Setting theme and context

Controlling figure aesthetics — seaborn 0.12.2 documentation

There are five preset seaborn themes: dark, white, ticks, whitegrid, darkgrid. This picture contains the first four of the above in this order.

set_context()

Color palettes

The tutorial has this: Choosing color palettes — seaborn 0.12.2 documentation with both a theoretical basis about color and stuff, and the “how to set it in your plot”.

TL;DR sns.color_palette(PALETTE_NAME, NUM_COLORS, as_cmap=TRUE_IF_CONTINUOUS)

seaborn.color_palette() returns a list of colors or a continuous matplotlib ListedColormap colormap:

• Accepts as palette, among other things:

  • Name of a seaborn palette (deep, muted, bright, pastel, dark, colorblind)
  • Name of matplotlib colormap
  • ‘light:<color>’, ‘dark:<color>’, ‘blend:<color>,<color>’
  • A sequence of colors in any format matplotlib accepts

• n_colors: will truncate if it’s less than palette colors, will extend/cycle palette if it’s more

• as_cmap - whether to return a continuous ListedColormap

• desat

• You can do .as_hex() to get the list as hex colors.

• You can use it as context manager: with sns.color_palette(...): to temporarily change the current defaults.

Reversing palettes/colormaps

Matplotlib colormap + _r (tab10_r).

I needed a colormap where male is blue and female is orange, tab10 has these colors but in reversed order. This is how I got a colormap with the first two colors but reversed:

cm = sns.color_palette("tab10",2)[::-1]

First I generated a color_palette of 2 colors, then reversed the list of tuples it returned.

Individual plot types

Distributions

Plotting multiple distributions on the same subplot

• histplot has different approaches for plotting multiple= distributions on the same plot:
  • layer (default, make them overlap)
  • stack (one on top of the other)
  • dodge (multiple small columns for each distribution):
  • fill (this beauty):
• KDEplot can do this too! multiple=fill

Categorical

Pointplot

seaborn.pointplot

• Errorbars:
  • To make the errorbars not-overlap, dodge=True
  • You can control their width through errwidth=
  • Statistical estimation and error bars — seaborn 0.12.2 documentation has a really cool and thorough description of the types and theory:

    The error bars around an estimate of central tendency can show one of two general things: either the range of uncertainty about the estimate or the spread of the underlying data around it. These measures are related: given the same sample size, estimates will be more uncertain when data has a broader spread. But uncertainty will decrease as sample sizes grow, whereas spread will not.

Random

Heatmaps

• To order the rows/columns, you have to use pandas’s pd.sort_index()
• To annotate / add text to the cells: annot=True, fmt=".1f"
• To change the range of the colorbar/colormap , use vmin=/vmax=

Previously: Small unsystematic posts about seaborn: - Architecture-ish: - 230515-2257 seaborn setting titles etc. with matplotlib set - 230515-2016 seaborn things built on FacetGrid for easy multiple plots - Small misc: - 230428-2042 Seaborn basics - 230524-2209 Seaborn visualizing distributions and KDE plots)

Visualizing distributions of data — seaborn 0.12.2 documentation:

• TIL KDE means “Kernel Density Estimation”
• common_norm=True by default applies the same normalization to the entire distribution. False scales each independently. This is critical in many cases, esp. with stat="probability"

Generally: I read the seaborn documentation, esp. the high level architecture things, and a lot of things I’ve been asking myself since forever (e.g. 230515-2257 seaborn setting titles etc. with matplotlib set) have become much clearer - and will be its own post. I love seaborn and it’s honestly worth learning to use well and systematically.

Main classes:

ds = Dataset(...)
ds.set_format("pandas")

There’s cycler, a package:

• Composable cycles — cycler 0.11.0 documentation
• A tutorial for matplotlib: Styling with cycler — Matplotlib 3.7.1 documentation

It returns cycles of dicts, finite or infinite:

from cycler import cycler

# list of colors
pal = sns.color_palette("Paired")

# `cycler` is a finite cycle, cycler() is an infinite
cols = iter(cycler(color=pal)())

# every time you need a color
my_color = next(cols)