serhii.net

Documented worse than I’d like to.

Filters allow to do things to the records (structs that make up a log message later), be it change them in place or don’t let them pass.

You can pass a function in place of a Filter, it should:

• get a logging.LogRecord
• optionally change it in place
• decide whether to let it pass
• return 0 for no, non-zero for yes

The fields of a LogRecord are the same ones we name when doing formatting: name, lineno, msg and friends.

If your Filter tries to log something in a way that it’ll get filtered through it, you get recursion.

Sample of a filter that removes specific matches and gets added to a Handler:

def filter(record: logging.LogRecord) -> int:
	"""Filters away log records containing annoying stuff."""
	blacklist_condition = (
		(
			record.name == "lib_sdk.data"
			and "not available on your" in record.msg
		)
		or (
			record.name == "lib_sdk.data"
			and record.levelno == logging.WARNING
			and "which is legacy" in record.msg
		)
		or (
			record.name == "lib_sdk.data"
			and record.levelno == logging.WARNING
			and "created but without information" in record.msg
		)
	)
	if blacklist_condition:
		return 0
	else:
		return 1

sh = logging.StreamHandler()
sh.addFilter(filter)

Much better than what I had before (220914-2249 Python logging change level through context manager and operator magic).

One can go crazy here with regexes etc. but I shan’t.

My standard logging setup is logger=logging.getLogger(__package__) in my main runner file and .getLogger(__name__) for all other files.

I wanted to temporarily change the loglevel of a specific logger of a library. Logical thing is to use a context manager, and such things exist:

• Context-dependent logger in Python
• Logging Cookbook — Python 3.10.7 documentation

I liked the second one, but what I wanted is to change the loglevel of another logger.

Usage:

# inside somelib.data...
liblogger = logging.getLogger(__name__)

logger.info("Stuff")
liblogger.info("Stuff from the lib")
with LoggingContext(
	"somelib.data",
	level=logging.ERROR
):
	# very deep inside somelib.data...
	liblogger.warning("Useless warning")

liblogger.warning("Not useless warning")
logger.info("Stuff")

Idea:

• While inside the context, the loglevel of the logger used inside the library gets set to ERROR
  • I see only ERRORs from inside the library
  • I don’t see their useless warnings that would be logger.debug()s in my world
• Other loggers are unchanged
• On end of context everything goes back to normal

Second draft with operators!

But if I’m debugging I want these useless warnings!

After doing level=logging.ERROR if logger.level != logging.DEBUG else logging.getLogger('somelib_data').level oneliners I decided that I want the context manager to be flexible.

Ended up with this:

class LoggingContext:
    """Temporarily change the loglevel of a logger based on loglevels of
    other loggers or arbitrary conditions."""

    def __init__(
        self,
        logger_name: str,
        level_true: int,
        level_false: Optional[int] = None,
        l1: Union[logging.Logger, int] = logger,
        l2: Optional[int] = None,
        comp_fn: Optional[Callable] = lambda x, y: True,
    ):
        """Temporarily change logging level of a logger, optionally dependent
        on another logger's level.

        :param logger_name: Change the level of a logger with this name
            if None, the `level` new logger level will be used
        :param callable_for_unchanged: if set, will be used to compare
            main_logger_level to comparison logger level
            and if True, will leave everything unchanged.
        :param level_true: which loglevel to set in logger if condition is True
        :param level_false: loglevel to set if condition is False
            None means "don't change anything"
        :param l1: main logger whose effective loglevel we'll use, or a loglevel
            if None the global `logger` will be used
        :param l2: loglevel to compare l1 with
            if None will compare to the loglevel `level_true`
        :param comp_fn: callable taking two params, loglevels/ints l1 and l2,
            returning a boolean. Can be a lambda function or `operators` library
            operators (eq,neq etc.)
            If None will return True, ergo setting level_true always
        """
        self.other_logger = logging.getLogger(logger_name)

        # If it's a logger, get its effective level, if int - use that
        main_level = (
            l1.getEffectiveLevel() if isinstance(l1, logging.Logger) else l1
        )

        # Compare to l2 if it's there, otherwise to level_true
        effective_comparison_level = l2 if l2 else level_true

        # If callable is True, leave everything unchanged
        comparison_result = comp_fn(main_level, effective_comparison_level)

        # If we have no level_false, interpret it as "don't change anything"
        if comparison_result:
            self.level = level_true
        else:
            # 'None' here is a magic value "don't change anything"
            self.level = level_false

        logger.debug(
            f"{logger_name=}, {l1=}, {l2=}, "
            f"{level_true=}, {level_false=}, {comp_fn=}"
        )
        logger.debug(
            f"{self.other_logger=}, {self.level=}, {main_level=}, "
            f"{effective_comparison_level=}, {comparison_result=}"
        )

        if self.level is not None:
            logger.debug(f"Changing {logger_name=} to loglevel {self.level}")
        else:
            logger.debug(f"Leaving {logger_name=} unchanged.")

    def __enter__(self):
        if self.level is None:
            return None

        self.old_level = self.other_logger.level
        self.other_logger.setLevel(self.level)

    def __exit__(self, et, ev, tb):
        if self.level is None:
            return None
        else:
            self.other_logger.setLevel(self.old_level)

This changes the idea completely and brings some VERY non-intuitive dynamics with default values, not sure yet if it’s worth doing it like that for the sake of brevity but we’ll see.

• level_true, level_false are levels to use based on condition
• l1, l2 are the two loglevels we compare
• cond_fn is a Callable/lambda/… that does the condition and returns a boolean.
• Non-intuitive dynamics and default values. If omitted:
  • level_false means “no change to status quo”
  • l1 takes the global logger, which is probably a child of the logger we care about and inherits its effective loglevel
  • l2 becomes level_true
    • For cases like “change loglevel to X only if X is more/less/equal than/to our l1”

EXAMPLE SCENARIOS

• temporarily silence useless warnings of a library’s logger ‘other’:
  with LoggingContext('other', logging.ERROR):
• temporarily change loglevel of ‘other’, only if they’ll still be visible to me afterwards (=level higher than current one):
  with LoggingContext('other', logging.INFO, comp_fn=operators.le):
• temporarily change loglevel of ‘other’ to shut it up unless we’re in debug mode, in which case I want to see everything:

  with LoggingContext('other', logging.ERROR,
   l2=logging.DEBUG, comp_fn=operators.eq):
  

• if we’re at loglevel INFO or less, change ‘other’ to WARNING, if not - otherwise change it to ERROR

  from operators import le as less_or_equal
  
  with LoggingContext('other', level_true=logging.WARNING,
  level_false=logging.ERROR,
  l1=logger.level,  # just as demo, it's implicit everywhere
  l2=logging.INFO, comp_fn=less_or_equal):`
  

Initially it was lambdas, but I kept wishing for “can I just pass <= as a function?” and lo and behold - yes, through the operator library!

Fazit

That was fun, and TIL about operators. In any case - another function for my small library of snippets.

Best of all, my favourite python blog has an article about the topic:The Unknown Features of Python’s Operator Module | Martin Heinz | Personal Website & Blog

Let’s see if I end up using this utility function more than once.

Bonus

Another similar-ish snippet I wrote once and still love. You get pretty progress bars only if you have enough elements in your sequence for it to make sense:

def _tqdm(list_like: Sequence, iflarge: bool = False, lsize: int = 100, **kwargs):
    """Use tqdm if it's on, optionally based on length of list.

    Args:
        list_like: thing to iterate.
        iflarge (bool): If on, will use tqdm only for large lists
        lsize (int): anything more than this is 'large'
        **kwargs: get passed to tqdm as-is
    """

    if USE_TQDM:
        if not iflarge:
            return tqdm(list_like, **kwargs)
        else:
            # Count size only if it doesn't mean iterating an iterator
            if isinstance(list_like, Sequence) and len(list_like) > lsize:
                return tqdm(list_like, **kwargs)

    return list_like

Then, if the global USE_TQDM is true:

• for x in _tqdm(sth) is a vanilla tqdm
• for x in _tqdm(sth, True) becomes a tqdm only if we’re iterating through something larger than 100 elements.
• _tqdm(sth, True, 50, desc="DOCS") tqdm on 50+ elements with a label (how cool is that?)

And on the same topic:

def log(msg) -> None:
    """Use loglevel.debug if tqdm is used, loglevel.info otherwise."""
    if USE_TQDM:
        logger.debug(msg)
    else:
        logger.info(msg)

logger.info() destroy tqdms, so - if we’re using TQDM, log it as logger.debug(). We’ll still see it on that loglevel if we want to (or maybe we’re logging it to a file, who knows).

TODO

• I think the RIGHT way to solve this would be a logging.Filter object. Later.
• I want a stable workflow that logs everything to a logfile but shows only a subset on screen. This means setting loglevel DEBUG, and adding a handler of loglevel INFO for stdout and a FileHandler of same DEBUG level for a file.

In .ideavimrc I added these two:

nmap <leader><leader> :action CloseContent<cr>
nmap <C-S-T> :action ReopenClosedTab<cr>

First equal to my vim settings, second equal to the usual binding for it in “normal” browsers.

Python has a property function/decorator: Built-in Functions — Python 3.10.7 documentation.

Basically - you have a field and you want getter/setter functions on it.

Seen first in konfuzio_sdk, sample from there:

@property
def number_of_lines(self) -> int:
	"""Calculate the number of lines in Page."""
	return len(self.text.split('\n'))

Then you can run document.number_of_lines and it runs the function.

From OmegaConf source:

def fail() -> None:
	raise ValueError("Input list must be a list or a tuple of strings")

if not isinstance(dotlist, (list, tuple)):
	fail()

for arg in dotlist:
	if not isinstance(arg, str):
		fail()

I don’t know if I like this or not, but it’s interesting. But I did write similar things with a parametrized fail()

Was doing len(list(Path(".").iterdir())), shortened it to a truth-y list(...), then to a shorter any(Path(".")).iterdir().

Because I don’t need the length of (the elements in..) an iterator, I just need “does it have elements?”. I guess that’s why you can do any(Iterator) but not len(Iterator).

Gimp can open PDFs, if you select “open pages as images” instead of the default “as layers”, it will open each page as a separate image.

Then you can use burn/levels/… to improve quality of the scan of the document printed with a printer that’s low on toner.

Also - Goddammit Gimp interface - was looking for the burn tool. It’s hidden behind “Smudge”, had to use right click on it to get the full list. Hate this

OmegaConf is nice and has more features than YACS.

Merging (from the help)

conf = OmegaConf.merge(base_cfg, model_cfg, optimizer_cfg, dataset_cfg)

Bits I can’ find explicitly documented anywhere:

OmegaConf.merge() takes the first argument as “base”, and its keys should be a superset of keys in the next one or it errors out (from omegaconf.errors import ConfigKeyError).

It casts arguments automatically, if first argument’s key is a Path and the second is a str the merged one will be a Path(str_from_second_argument), beautiful!

New phone, need to set up again sync and friends to my VPS - I’ll document it this time.

This is part of the success story of “almost completely de-Google my life” that’s one of the better changes I ever did.

• Port Knocking
  • Port Knocker | F-Droid - Free and Open Source Android App Repository
  • Settings on server unchanged
• FreshRSS
  • FreshRSS - Google Reader compatible API implementation
  • Had remote access etc. enabled as described in FreshRSS
  • EasyRSS on Android:
    • Server: https://my.server/api/greader.php
    • Username: the one I use for web login
    • Password: the “API password’ I set in profile -> API management -> API password
• Nextcloud
  • Nextcloud | F-Droid - Free and Open Source Android App Repository
  • No issues logging in and setting up synchronization
  • Enabled backing up of my contacts to nextcloud (but not back prolly)
• DavX / dav / calendar / …
  • Nextcloud
    • Nextcloud — DAVx⁵
    • Worked as advertised, the nextcloud app opened the DAVx5 app
  • But actually I need Fastmail
    • Fastmail — DAVx⁵
    • Set up app password, logged it with my fastmail username and it
    • Worked

Goal: separate commands running separate taskwarrior reports/filters. But also usable to add tasks etc.

Previously (Day 728 - serhii.net) I used things like this in my zshrc:

th () {task s project.not:w sprint.not:s "$*"}

Found a better way:

## TASKWARRIOR
# All todos from both work and home
TW_WORK="rc.default.project:w rc.default.command:s"
TW_HOME="rc.default.project: rc.default.command:th"
# "Important tasks"
TW_I="rc.default.command:i"

# Work
alias s="task $TW_WORK"
# Home
alias t="task $TW_HOME"

# All pending tasks from all projects
alias ta="task rc.default.command:next"
# "Important" tags - report `i`
alias ti="task $TW_I"

This means: s runs taskwarrior and the s report, which shows work-only tasks; if I do s add whatever the task gets added automatically inside project:w.

For completeness, the code for each of these reports (~/.taskrc):

############
# REPORTS
############
report.s.description='Work tasks'
report.s.columns=id,project,tags,due.relative,description
report.s.labels=ID,Project,T,D,Desc
#report.s.sort=due+
report.s.sort=project-/,urgency+
report.s.filter=status:pending  -s
report.s.filter=status:pending ((project:w -s) or (+o or +a or +ACTIVE))

report.i.description='Important / priority'
report.i.columns=id,project,tags,due.relative,description
report.i.labels=ID,Project,T,D,Desc
report.i.sort=project-/,urgency+
report.i.filter=status:pending (+o or +a or +ACTIVE)

report.th.description='Home tasks'
report.th.columns=id,project,tags,due.relative,description
report.th.labels=ID,Project,T,D,Desc
report.th.sort=project-/,urgency+
report.th.filter=status:pending  -s
# report.th.filter=status:pending ((project.not:w project.not:l -srv -sd) or (+o or +a or +w or +ACTIVE))
report.th.filter=status:pending ((project.not:w project.not:l -srv -sd) or (+o or +a or +ACTIVE))

#Someday
report.sd.columns=id,start.age,depends,est,project,tags,sprint,recur,scheduled.countdown,due.relative,until.remaining,description,urgency
report.sd.labels=D,Active,Deps,E,Project,Tag,S,Recur,S,Due,Until,Description,Urg
report.sd.filter=status:pending (sprint:s or +sd)

# srv -- for continuously needed tasks like starting to work etc
report.srv.description='srv'
report.srv.columns=id,project,tags,pri,est,description,urgency
report.srv.labels=ID,Project,T,P,E,Description,U
report.srv.sort=urgency-
report.srv.filter=status:pending +srv

# Currently active task - for scripts
report.a.description='Currently active task'
report.a.columns=id,description #,project
report.a.labels=ID,D #,P
report.a.filter=+ACTIVE

report.next.filter=status:pending -srv -sd

urgency.user.tag.o.coefficient=10
urgency.user.tag.a.coefficient=5
urgency.user.tag.w.coefficient=3