In the middle of the desert you can say anything you want
How to see ping requests being recieved on the destination machine? - Super User:
Wireshark is too heavy duty for something so simple. Just use
tcpdump -nn icmp. Add andhost 1.2.3.4if you want to limit it to packets coming from 1.2.3.4.
I religiously do .realpath() pretty much every time I get a path from user input. Naively believing it also expands ~ etc.
Once I forgot and once I entered a non-expanded path myself: ~/this/
Then was tracking it as a bug, and found this bundle of joy:
/home/sh/me/dir~/me/dir/Checkpoints/checkpoint_288
It is in fact not illegal to create a directory called ~ in Unix.
And the things that used it as-is where there, and the things that were using it after a realpath were using another directory.
OK, I resolve()-d it - still the same.
TIL Path.resolve() takes care of symlinks and ..-like components, but not ~. So it should be Path.expanduser().resolve() from now on.
jq’s to_entries allows parsing key names as values/fiels:
``s__` jq ’to_entries' Input {“a”: 1, “b”: 2} Output [{“key”:“a”, “value”:1}, {“key”:“b”, “value”:2}]
Documented worse than I’d like to.
Filters allow to do things to the records (structs that make up a log message later), be it change them in place or don’t let them pass.
You can pass a function in place of a Filter, it should:
logging.LogRecordThe fields of a LogRecord are the same ones we name when doing formatting: name, lineno, msg and friends.
If your Filter tries to log something in a way that it’ll get filtered through it, you get recursion.
Sample of a filter that removes specific matches and gets added to a Handler:
def filter(record: logging.LogRecord) -> int:
"""Filters away log records containing annoying stuff."""
blacklist_condition = (
(
record.name == "lib_sdk.data"
and "not available on your" in record.msg
)
or (
record.name == "lib_sdk.data"
and record.levelno == logging.WARNING
and "which is legacy" in record.msg
)
or (
record.name == "lib_sdk.data"
and record.levelno == logging.WARNING
and "created but without information" in record.msg
)
)
if blacklist_condition:
return 0
else:
return 1
sh = logging.StreamHandler()
sh.addFilter(filter)
Much better than what I had before (220914-2249 Python logging change level through context manager and operator magic).
One can go crazy here with regexes etc. but I shan’t.
Goal: log everything to file, but show only part of the info on the screen. Previously: 220914-2249 Python logging change level through context manager and operator magic
My current understanding:
format = "[%(asctime)s %(name)s:%(lineno)s %(levelname)s]: %(message)s"
# Set it up, no handlers -> no default StreamHandler
# this loglevel is the one handlers will have access to!
logging.basicConfig(
level=logging.DEBUG,
handlers=[]
)
# Format, if we don't do this will be literally none
fmtr = logging.Formatter(fmt=format)
sh = logging.StreamHandler()
fh = logging.FileHandler("debug.log")
fh.setFormatter(fmtr)
sh.setFormatter(fmtr)
# Screen output set to whatever we want, fh to debug
sh.setLevel(loglevel)
fh.setLevel(logging.DEBUG)
# Add both handlers to root, both get propagated to logger etc.
logging.getLogger('').addHandler(sh)
logging.getLogger('').addHandler(fh)
Even though i did logger = logging.getLogger(__package__) at the very top of the file before the above bits, I can do logger.debug() etc. and it follows these settings. Nice.
My standard logging setup is logger=logging.getLogger(__package__) in my main runner file and .getLogger(__name__) for all other files.
I wanted to temporarily change the loglevel of a specific logger of a library. Logical thing is to use a context manager, and such things exist:
I liked the second one, but what I wanted is to change the loglevel of another logger.
Usage:
# inside somelib.data...
liblogger = logging.getLogger(__name__)
logger.info("Stuff")
liblogger.info("Stuff from the lib")
with LoggingContext(
"somelib.data",
level=logging.ERROR
):
# very deep inside somelib.data...
liblogger.warning("Useless warning")
liblogger.warning("Not useless warning")
logger.info("Stuff")
Idea:
logger.debug()s in my worldBut if I’m debugging I want these useless warnings!
After doing level=logging.ERROR if logger.level != logging.DEBUG else logging.getLogger('somelib_data').level oneliners I decided that I want the context manager to be flexible.
Ended up with this:
class LoggingContext:
"""Temporarily change the loglevel of a logger based on loglevels of
other loggers or arbitrary conditions."""
def __init__(
self,
logger_name: str,
level_true: int,
level_false: Optional[int] = None,
l1: Union[logging.Logger, int] = logger,
l2: Optional[int] = None,
comp_fn: Optional[Callable] = lambda x, y: True,
):
"""Temporarily change logging level of a logger, optionally dependent
on another logger's level.
:param logger_name: Change the level of a logger with this name
if None, the `level` new logger level will be used
:param callable_for_unchanged: if set, will be used to compare
main_logger_level to comparison logger level
and if True, will leave everything unchanged.
:param level_true: which loglevel to set in logger if condition is True
:param level_false: loglevel to set if condition is False
None means "don't change anything"
:param l1: main logger whose effective loglevel we'll use, or a loglevel
if None the global `logger` will be used
:param l2: loglevel to compare l1 with
if None will compare to the loglevel `level_true`
:param comp_fn: callable taking two params, loglevels/ints l1 and l2,
returning a boolean. Can be a lambda function or `operators` library
operators (eq,neq etc.)
If None will return True, ergo setting level_true always
"""
self.other_logger = logging.getLogger(logger_name)
# If it's a logger, get its effective level, if int - use that
main_level = (
l1.getEffectiveLevel() if isinstance(l1, logging.Logger) else l1
)
# Compare to l2 if it's there, otherwise to level_true
effective_comparison_level = l2 if l2 else level_true
# If callable is True, leave everything unchanged
comparison_result = comp_fn(main_level, effective_comparison_level)
# If we have no level_false, interpret it as "don't change anything"
if comparison_result:
self.level = level_true
else:
# 'None' here is a magic value "don't change anything"
self.level = level_false
logger.debug(
f"{logger_name=}, {l1=}, {l2=}, "
f"{level_true=}, {level_false=}, {comp_fn=}"
)
logger.debug(
f"{self.other_logger=}, {self.level=}, {main_level=}, "
f"{effective_comparison_level=}, {comparison_result=}"
)
if self.level is not None:
logger.debug(f"Changing {logger_name=} to loglevel {self.level}")
else:
logger.debug(f"Leaving {logger_name=} unchanged.")
def __enter__(self):
if self.level is None:
return None
self.old_level = self.other_logger.level
self.other_logger.setLevel(self.level)
def __exit__(self, et, ev, tb):
if self.level is None:
return None
else:
self.other_logger.setLevel(self.old_level)
This changes the idea completely and brings some VERY non-intuitive dynamics with default values, not sure yet if it’s worth doing it like that for the sake of brevity but we’ll see.
level_true, level_false are levels to use based on conditionl1, l2 are the two loglevels we comparecond_fn is a Callable/lambda/… that does the condition and returns a boolean.level_false means “no change to status quo”l1 takes the global logger, which is probably a child of the logger we care about and inherits its effective loglevell2 becomes level_true
l1”with LoggingContext('other', logging.ERROR):with LoggingContext('other', logging.INFO, comp_fn=operators.le):with LoggingContext('other', logging.ERROR,
l2=logging.DEBUG, comp_fn=operators.eq):
from operators import le as less_or_equal
with LoggingContext('other', level_true=logging.WARNING,
level_false=logging.ERROR,
l1=logger.level, # just as demo, it's implicit everywhere
l2=logging.INFO, comp_fn=less_or_equal):`
Initially it was lambdas, but I kept wishing for “can I just pass <= as a function?” and lo and behold - yes, through the operator library!
That was fun, and TIL about operators. In any case - another function for my small library of snippets.
Best of all, my favourite python blog has an article about the topic:The Unknown Features of Python’s Operator Module | Martin Heinz | Personal Website & Blog
Let’s see if I end up using this utility function more than once.
Another similar-ish snippet I wrote once and still love. You get pretty progress bars only if you have enough elements in your sequence for it to make sense:
def _tqdm(list_like: Sequence, iflarge: bool = False, lsize: int = 100, **kwargs):
"""Use tqdm if it's on, optionally based on length of list.
Args:
list_like: thing to iterate.
iflarge (bool): If on, will use tqdm only for large lists
lsize (int): anything more than this is 'large'
**kwargs: get passed to tqdm as-is
"""
if USE_TQDM:
if not iflarge:
return tqdm(list_like, **kwargs)
else:
# Count size only if it doesn't mean iterating an iterator
if isinstance(list_like, Sequence) and len(list_like) > lsize:
return tqdm(list_like, **kwargs)
return list_like
Then, if the global USE_TQDM is true:
for x in _tqdm(sth) is a vanilla tqdmfor x in _tqdm(sth, True) becomes a tqdm only if we’re iterating through something larger than 100 elements._tqdm(sth, True, 50, desc="DOCS") tqdm on 50+ elements with a label (how cool is that?)And on the same topic:
def log(msg) -> None:
"""Use loglevel.debug if tqdm is used, loglevel.info otherwise."""
if USE_TQDM:
logger.debug(msg)
else:
logger.info(msg)
logger.info() destroy tqdms, so - if we’re using TQDM, log it as logger.debug(). We’ll still see it on that loglevel if we want to (or maybe we’re logging it to a file, who knows).
In .ideavimrc I added these two:
nmap <leader><leader> :action CloseContent<cr>
nmap <C-S-T> :action ReopenClosedTab<cr>
First equal to my vim settings, second equal to the usual binding for it in “normal” browsers.
Python has a property function/decorator: Built-in Functions — Python 3.10.7 documentation.
Basically - you have a field and you want getter/setter functions on it.
Seen first in konfuzio_sdk, sample from there:
@property
def number_of_lines(self) -> int:
"""Calculate the number of lines in Page."""
return len(self.text.split('\n'))
Then you can run document.number_of_lines and it runs the function.
From OmegaConf source:
def fail() -> None:
raise ValueError("Input list must be a list or a tuple of strings")
if not isinstance(dotlist, (list, tuple)):
fail()
for arg in dotlist:
if not isinstance(arg, str):
fail()
I don’t know if I like this or not, but it’s interesting. But I did write similar things with a parametrized fail()
Was doing len(list(Path(".").iterdir())), shortened it to a truth-y list(...), then to a shorter any(Path(".")).iterdir().
Because I don’t need the length of (the elements in..) an iterator, I just need “does it have elements?”. I guess that’s why you can do any(Iterator) but not len(Iterator).