serhii.net

Overkill and suboptimal but good enough and makes my jupyter stuff more pleasant

class DDecorator(ContextDecorator):
    """To use:
    DD = DDecorator()
    with DD:
        your pl stuff with shown cols rows
    # or
    DD(your_df) -> display(your_df)
    """

    def __init__(self, ncols=-1, nrows=-1):
        self.ncols = ncols
        self.nrows = nrows

        def_state = pl.Config.state()
        # print(def_state)
        self.d_ncols = def_state["POLARS_FMT_MAX_COLS"]
        self.d_nrows = def_state["POLARS_FMT_MAX_ROWS"]

    def __call__(self, df, all: bool = False, ncols=None, nrows=None):
        ncols = self.ncols if ncols is None else ncols
        nrows = self.nrows if nrows is None else nrows
        if all:
            ncols = -1
            nrows = -1
        with pl.Config(set_tbl_cols=ncols, set_tbl_rows=nrows):
            display(df)

    def __enter__(self):
        pl.Config.set_tbl_rows(self.nrows)
        pl.Config.set_tbl_cols(self.ncols)
        return self

    def __exit__(self, *exc):
        # print("Finishing")
        # default vals
        pl.Config.set_tbl_rows(self.d_nrows)
        pl.Config.set_tbl_cols(self.d_ncols)
        return False


DD = DDecorator(nrows=30, ncols=30)


def describe(df):
    # with DDecorator(ncols=-1):
    with pl.Config(set_tbl_cols=-1):
        itables.show(
            df.describe(percentiles=None), fixedColumns={"start": 1}, scrollX=True
        )

Input

Read multiple CSV files with GLOBS!

Multiple - Polars user guide:

df = pl.read_csv("docs/assets/data/my_many_files_*.csv")

polars.read_csv — Polars documentation actually accepts lists as well!

Utils

Stack helper fn

Essentially transform a df with cols l_metric,t_metric,l_another,t_another into one with cols type(=l or t), metric,another.

def stack_types(
    df: pl.DataFrame,
    types: Sequence[str] = ("l", "t"),
    metrics: Sequence[str] = ("num_words", "num_chars"),
    id_cols: Sequence[str] = (),
    type_key="type",
) -> pl.DataFrame:
    """Stack prefixed metric columns into long format."""
    return pl.concat(
        [
            df.select(
                *id_cols,
                *(c(f"{t}_{m}").alias(m) for m in metrics),
                type=pl.lit(t),
            )
            for t in types
        ]
    ).rename({"type": type_key})

Bits

If then else otherwise nested

polars.when — Polars documentation

• otherwise() defaults to None

• The first one is picked if multiple conditions fit

# If multiple fit then the first one is picked
df.with_columns(
    pl.when(pl.col.foo > 2)
    .then(1)
    .when(pl.col.bar > 2)
    .then(4)
    .otherwise(-1)
    .alias("val")
)
df.with_columns(
...     when1 = pl.col.foo > 2,
...     then1 = 1,
...     when2 = pl.col.bar > 2,
...     then2 = 4,
...     otherwise = -1
... )

Filter where None is involved

polars.DataFrame.filter — Polars documentation TLDR: use e.g. ne_missing() instead of != : polars.Expr.eq_missing — Polars documentation

Polars exclude certain columns

https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.exclude.html: df.select(pl.exclude("ba")) for all cols except ba.G

select can take keyword args!

Select two cols except that col2 will be called new_col2:

df.select("some_col", new_col2="col2")

(oh so many pl.with_columns(...alias('new_col2')) will be avoided by this)

Strings

Format strings

.with_columns(
    pl.format(
        "{} [{}]",
        c(LLM_MODEL),
        c(THINKING).cast(pl.String).fill_null("None"),
    ).alias(MODEL_VARIANT)
)

# when ...
pl.concat_str(
	[c(LLM_MODEL), c(THINKING)],
	separator="/",
	ignore_nulls=True,  # single None doesn't None the entire row
).alias(MODEL_VARIANT)
# ...is not enough

# ...or even this to avoid the [{}] part if thinking is null
.with_columns(
	pl.when(c(THINKING).is_null())
	.then(c(LLM_MODEL))
	.otherwise(pl.format("{} [{}]", c(LLM_MODEL), c(THINKING)))
	.alias(MODEL_VARIANT)
)

Gotchas

Count, len and sum over binary values

# this will never work:
(pl.col(effect_col(m)) < 0).count().alias("n_worsened"),
(pl.col(effect_col(m)) < 0).len().alias("n_worsened"),
# do this insead
(pl.col(effect_col(m)) < 0).sum().alias("n_worsened"),

len counts the entire thing, count() only non-null boolean values (todo?) and basically all will become n regardless of condition.

Use sum().

Polars-statistics for tests/significance/bootstrapping

• GitHub - DataZooDE/polars-statistics: High-performance statistical testing and regression for Polars DataFrames, powered by Rust. · GitHub
• Has VERY good documentation and jupyter notebook examples!
  • e.g. polars-statistics/docs/examples/categorical-analysis.md has McNemar

Presentations

Why not put the various scattered advice I get into one place :)

0. Better less information properly conveyed than speeding through more information of which 0% will be retained
1. Never ask the audience a question they are not expecting
2. Pacing is not enough, when the slide is complex pause at the end to give people time to think
   • When doing dry runs, finishing reading a silde and going to the next won’t give a proper time estimate

Plots

• If you are planning to highlight or add markings to a plot on the slide, leave a color out of the plot itself
  • (for me: no red lines on plots!)

Pareto frontier - Wikipedia “is the set of choices where it is impossible to choose another option that is better in some way without being worse in another.”

Outperforming is called Pareto dominance: a solution A dominates (outperforms) B if A is no worse than B in every objective, and better than B in at least one objective

There are algorithms to calculate that

TODO: merge into 230525-1304 Seaborn how-to guide and use right terminology

Titles (figure-level)

(axis-level: sns.relplot, figure-level:sns.catplot)

Simple titles

sns.catplot(..)
plt.title("mytitle")

Titles overlapping facet text

Previously: 230515-2257 seaborn setting titles etc. with matplotlib set Key:

g = sns.catplot(...)

g.set_axis_labels("Width", "Equals (acc.)")

# # not needed if you do tight_layout()
# g.fig.subplots_adjust(top=0.85)

g.fig.suptitle("Masking and breaking words makes everything worse")

plt.tight_layout()
plt.show()

Fuller example from Change Axis Labels, Set Title and Figure Size to Plots with Seaborn - GeeksforGeeks:

  
# Plot scatter plot
g = sns.relplot(data = tips , x = "total_bill" , 
                y = "tip" , col = "time" , 
                hue = "day" , style = "day" , 
                kind = "scatter" )

# Title for the complete figure
g.fig.suptitle("Tips by time of day" ,
               fontsize = 'x-large' , 
               fontweight = 'bold' )

# Adjust subplots so that titles don't overlap
g.fig.subplots_adjust( top = 0.85 )

# Set x-axis and y-axis labels
g.set_axis_labels( "Tip" , "Total Bill (USD)" )

Previously: the venerable 260826-1558 Faker has providers and locales TIL: Mimesis Structured Data Generation - Mimesis

It’s much better at generating jobs, for one. EDIT: and colors

mim_locale = Locale.EN
generic = Generic(locale=mim_locale)
job = generic.person.occupation()

Supports locales incl. UKR as well! Providers: API - Mimesis

Config

Top-Level Chart Configuration — Vega-Altair 6.3.0.post4+gb2ca919d documentation All together with .properties on the Chart object itsel: alt.Chart().encode().properties().top_level_configuration_item()

Etc

• Encodings — Vega-Altair

  • Channels (x,y etc.): Channels — Vega-Altair
  • Encoding data types:
    • Shorthand syntax¹
      • Q: quantitative: ‘a continuous real-valued quantity’
      • Discrete/categorical:
        • O: ordinal (=sortable)
        • N: nominal (=not sortable)
      • T: temporal, G: geojson

• Faceted charts (=multiple side-by-side):

  • Layered & Multi-View Charts — Vega-Altair 6.3.0.post4+gb2ca919d documentation
  • Simple:
    • .encode(x=..,y=..,column=xxx,row=yyy

• Sorting

  • Sorting | Vega-Lite

• Rendering multiple altair charts in a single cell of a jupyter notebook

  graph[0].display()
  graph[1].display()
  

• Legends
  • Disable: alt.Color().legend(None) (! not inside Color()!)
  • Multiple legends
    • Displaying Two Different Legends in hconcat Chart Using Altair - GeeksforGeeks
    • Scale & Guide Resolution — Vega-Altair 6.3.0.post4+gb2ca919d documentation

    .encode(
    	alt.X("variable"),
    	alt.Y("count()"),
    	alt.Color("value:N", legend=alt.Legend(title="x")),
    	# TIL 
    	alt.Scale(scheme='blues') # make colors different too
    )
    #...
    alt.vconcat(*charts).resolve_scale(color="independent")
    

  • API: altair.Legend
• New lines to labels
  • passing a list of strings instead of string makes them newlines[^²]
  • labelExpr="replace(datum.label,'_','\\n')" doesn’t seem to work for newlines
    • Customizing Visualizations — Vega-Altair 6.3.0.post4+gb2ca919d documentation
      • takes us to d3-format | D3 by Observable for formatting a la python, :.2f only for numbers
    • more on this: Dynamic Behaviors with Parameters | Vega-Lite

Composing

• Rule — Vega-Altair 6.3.0.post4+gb2ca919d documentation
  • draw lines and ranges and candlesticks with y2/x2
  • Line Chart with Datum — Vega-Altair 6.3.0.post4+gb2ca919d documentation to highlight values

References

• Colors are valid CSS color names: [ CSS type - CSS | MDN

God the documentation is atrocious.

References

• Standard Colors — Rich 14.1.0 documentation
• Box constants (pass to box= to get a “border style”)

In no particular order bits I looked for but couldn’t find

Styles

Border styles

(E.g. for rules etc.): Panel(box=box.DOUBLE_EDGE).

Boxes types: Box — Rich 14.1.0 documentation

(Box — Rich 14.1.0 documentation says:

You can set the border style by importing one of the preset Box objects and setting the box argument in the table constructor.

That tells me nothing about what do I pass to Panel(..., border_style=), I think it’s only the colors etc. from normal style=, including e.g. ‘on blue’ . In the examples only “dim” and “not dim” are used…)

EDIT oh okay border_style='dim' also influences the colors of the title. Okay why not

How complexipy, ty, ruff and prek can improve vibe coding introduced me to GitHub - rohaquinlop/complexipy: Blazingly fast cognitive complexity analysis for Python, written in Rust. · GitHub and it’s really cool!

Show only the failed functions and suggest refactors:

uv run complexipy src --failed --suggest-refactors

UKR locale providers: Locale uk_UA — Faker 40.37.0 documentation Nice! (Note to self: mind inflection and agreement!)

Also Faker as providers, some quite smart ones, e.g. faker-healthcare-provider · PyPI that does languages (not UKR sadly) and matching realistic medical data.

Creating a provider is easy: Welcome to Faker’s documentation! — Faker 40.37.0 documentation