serhii.net

I seem to keep googling this. … and this is not final and magic and I should actually understand this on a deeper level.

Not today.

So.

TL;DR for copypaste

Reading lines in a file:

while IFS="" read -r p || [ -n "$p" ]
do
  printf '%s\n' "$p"
done < peptides.txt

For outputs of a command:

while read -r p; do
	echo $p;
done < <(echo "one\ntwo")

Easy option with cat

Otherwise: Easy option that I can memorize, both for lines in command and in file that will will skip the last line if it doesn’t have a trailing newline:

for word in $(cat peptides.txt); do echo $word; done

Same idea but with avoiding this bug:

cat peptides.txt | while read line ||  -n $line ;
do
   # do something with $line here
done

Correct option without cat

1. Same as first cat option above, same drawbacks, but no use of cat:

   while read p; do
     echo "$p"
   done <peptides.txt
   

2. Same as above but without the drawbacks:

   while IFS="" read -r p || [ -n "$p" ]
   do
     printf '%s\n' "$p"
   done < peptides.txt
   

3. This would make command read from stdin, 10 is arbitrary:

   while read -u 10 p; do
     ...
   done 10<peptides.txt
   

(All this from the same SO answer¹).

In general, if you’re using “cat” with only one argument, you’re doing something wrong (or suboptimal).

jq -r $stuff instead of quoted ‘correct’ values like

"one"
"two"
"three"

would return

one
two
three

Wanted to rename all tasks belonging to a certain project from a certain timeframe.

TL;DR

• Use filters to select tasks within a timeframe
• If you use hierarchical projects (pro:w.one.two) heavily and want to keep the children names:
  • Export them and use JSON parsing magic to get a unique list of project names
  • Bash loop to manually rename each of these project

Final command I used:

for p in $(task export "\(pro.is:w or pro:w.\) entry.after:2019-04-30 entry.before:2021-12-31"  | jq ".[].project" -r | sort | uniq);
	do task entry.after:2019-04-30 entry.before:2021-12-31 pro:$p mod pro:new_project_name$p;
done

Longer version

The task¹

Used project:w for work, now new work, makes sense to rename the previous one for cleaner separation.

To list all tasks created in certain dates (task all to cover tasks that aren’t just status:pending as by default):

task all pro:w entry.after:2019-04-30 entry.before:2021-12-31

1213 tasks. Wow.

Remembering when I was using sprints and renaming them at the end, pro:w covers pro:w.test and pro:whatever.

I was disciplined but wanted to cover all pro:w and pro:w.whatever but not pro:whatever just in case, so tested this, same result:

task all "\(pro.is:w or pro:w.\) entry.after:2019-04-30 entry.before:2021-12-31"

How to modify them?

The problem

Okay, got them. How to modify? Complexity: I need to change part of the project, so pro:w.one -> pro:old_w.one instead of changing all tasks’ project to pro:old_w

Attempts

Commands

There’s prepend ² but seems to work only for descriptions.

There’s t mod /from/to/ syntax³, couldn’t get it to work part of the project.

There’s regex⁴, but works only for filters if enabled

There’s json export but I don’t feel like parsing JSON, feels too close to day job :)

Listing projects

You can list projects like this:

# currently used
task projects

# all
task rc.list.all.projects=1 projects

This gives hope, if I get the list of projects I can just iterate through them and rename all of them individually.

Can’t find this documented, but task rc.list.all.projects=1 projects pro:w filters the projects by ones starting with w.

Format parses the hierarchy sadly

Project       Tasks
w              1107
  a               1
  aan             1

Can I rename the character used for hierarchy so that I get them as list of separate tags with dots in them? Not exposed through config from what I can see

…alright, JSON export it is

JSON export

It exists, and of course it accepts filters <3

task export "\(pro.is:w or pro:w.\) entry.after:2019-04-30 entry.before:2021-12-31" | wc -l

1215 lines - about the same ballpark as the number of tasks.

JSON output is an array of these objects:

  {
    "id": 0,
    "description": "write attn mechanism also on token features",
    "end": "20191016T143449Z",
    "entry": "20191016T120514Z",
    "est": "PT1H",
    "modified": "20200111T094548Z",
    "project": "w",
    "sprint": "2019-41",
    "status": "completed",
    "uuid": "d3f2b2ac-ec20-4d16-bd16-66b2e1e568f9",
    "urgency": 2
  },

Okay

> task export "\(pro.is:w or pro:w.\) entry.after:2019-04-30 entry.before:2021-12-31"  | jq ".[].project" | uniq


"w.lm"
"w.l.p"
"w.lm"
"w.lm"
"w.l.py"
"w.lm"
"w"

Proud that I wrote that from the first try, as trivial as it is. Thank you ExB for teaching me to parse JSONs.

The quotes - jq -r returns raw output⁵, so same as above but without quotes.

Final command to get the list of projects:

task export "\(pro.is:w or pro:w.\) entry.after:2019-04-30 entry.before:2021-12-31"  | jq ".[].project" -r | sort | uniq

(Remembering that uniq works only after sort)

And let’s make it a loop, final command:

for p in $(task export "\(pro.is:w or pro:w.\) entry.after:2019-04-30 entry.before:2021-12-31"  | jq ".[].project" -r | sort | uniq);
	do task entry.after:2019-04-30 entry.before:2021-12-31 pro:$p mod pro:new_project_name$p;
done

Nice but forgotten stuff:

• Taskwarrior - Context are neat, useless for me here, but neat.
• task summary
• Nice post about projects: Taskwarrior Projects | Random Geekery

Redirecting stuff

Had /dtb/days/day122.md-type posts, the older ones, and /dtb/days/1234-1234-my-title.md-type newer posts. They lived both in the same directory on disk, /content/dtb/days/.... The latter were converted from Obsidian, which meant (among other things) that deleting a page in Obsidian wouldn’t automatically delete the corresponding converted one in Hugo, and I couldn’t just rm -rf ..../days before each conversion because that would delete the older day234.md posts.

I wanted to put them in different folders on disk in ./content/, but keep the url structure serhii.net/dtb/post-name/ for both of them.

Solution was making all /dtb posts (incl. pages) use the section (dtb) in the permalink in config.yaml:

permalinks:
    dtb: '/:section/:filename'

Now they do, regardless of their location on disk.

Then I moved the old posts into ./content/dtb/old_days, kept the new ones in ./content/dtb/days

Lastly, this removes all converted posts (= all .mds except _index.md) before conversion so that no stray markdown posts are left:

find $OLD_DAYS | grep -v _index.md | xargs  rm 

Unsolved problems

Google still has serhii.net/dtb/days/... pages cached, and currently they’re available both from there and from /dtb/.... I can’t find a way to redirect all of the /dtb/days/... to /dtb/... except manually adding stuff to the frontmatter of each. I have scripts for that, but still ugly.

.htaccess is our friend.

" RewriteRule ^d/dtb(.*)$ /dtb$1 [R=301,NC,L]
RewriteRule ^dtb/days(.*)$ /dtb$1 [R=301,NC,L]

This is getting more and more bloated.

Generally, I see absolutely no reason not to rewrite this mess of build scripts in Python. obyde is a Python package, handling settings, file operations etc. is more intuitive to me in Python.

Instead I keep re-learning bash/zsh escape syntax every time, and I’m procrastinating doing error handling for the same reasons.

The only non-native thing would be rsync and git, which can be handled through a subprocess.

pytest-datafiles · PyPI is nice but returns a py.path instead of pathlib.Path.

Tried to write something to make it convert automatically.

ASSETS_DIR = Path(__file__).parent / "assets"

@pytest.fixture
def pfiles(datafiles):
    # Fixture that converts pytest-datafiles' py.path into a pathlib.Path
    return Path(str(datafiles))

@pytest.mark.datafiles(PROJ_DIR)
def test_read_meta_json(pfiles):
	assert do_sth_with_file(pfiles)

First nontrivial fixture I write, maybe a really bad idea to do it like that. This feels like a general use case and someone had to have had this problem

pytest-datafiles · PyPI allows copying files to a temporary directory, then they can be modified etc. Really neat!

Sample:

ASSETS_DIR = Path(__file__).parent / "assets"
PROJ_DIR = ASSETS_DIR / "project_dir"

konfdir =  pytest.mark.datafiles(PROJ_DIR)

@konfdir
def test_basedir_validity(datafiles):
	assert directory_is_valid(datafiles)

Also love this bit:

Note about maintenance: This project is maintained and bug reports or pull requests will be addressed. There is little activity because it simply works and no changes are required.

SADLY this means that returned path is py.path, I’m not the only one complaining about that¹

Pytest has newer native fixtures that use Pathlib (Temporary directories and files — pytest documentation) but datafiles hasn’t been moved to them.

A conftest.py file gets imported and run before all the other ones.

Pytest resolves all imports at the very beginning, I used conftest.py it to import a package so that it’ll be the one used by the imports in files that are imported in the tests (seeing that there’s a mypackage already imported, subsequent import mypackages are ignored)

(Can I think of this as something similar to an __init__.py?)

This looks really interesting! It’s not about the syntax, but about the basic design philosophies + examples of packages that use it.

What’s init for me? Designing for Python package imports | Towards Data Science

Other stuff I learned about __init__.py:

• You can use it to enforce import order ¹
• You can use it to declare package variables
• Automatically import modules from a package²

Stuff I discovered:

• You can set a breakpoint in pdb physically into an __init__.py, and for example look at the stack of what called it with w

Today, I ran this:

git commit -m "TICKETNAME Export of X generated with `name-of-some-utility`"

Commit message on gitlab was

"TICKETNAME Export of X generated with (Starting the export of data, wait till it downloads...)"

Clear but fascinating way it can break.

Do I want to get a clear picture of all the various levels of escaping, including globs, backticks, backslashes etc. happening in the shell?

Why doesn’t the # in git commit -m "Ticket #1231" result in a string with the 1234 commented out and a syntax error? I know it doesn’t but I wouldn’t be able to predict that behaviour without this knowledge. Would single quotes change much? How to actually comment the rest of the line this way?

What are the rules that decide whether a * gets expanded by the shell or passed to, say, scp as-is? Etc. etc. etc.

It’s all knowable and learnable, but I was never sure whether the ROI was worth it for me. Till now trial and error always worked in the rare instances I have to do something complex with bash scripts, but this is the first time it bites me in real life in an unexpected way.

I find this approach¹ brilliant (and of course it works with everything split in separate functions a la my last post: 211124-1744 argparse notes):

import argparse
import logging

parser = argparse.ArgumentParser()
parser.add_argument(
    '-d', '--debug',
    help="Print lots of debugging statements",
    action="store_const", dest="loglevel", const=logging.DEBUG,
    default=logging.WARNING,
)
parser.add_argument(
    '-v', '--verbose',
    help="Be verbose",
    action="store_const", dest="loglevel", const=logging.INFO,
)
args = parser.parse_args()    
logging.basicConfig(level=args.loglevel)

And TIL about dest= that will make my life much easier too by outsourcing more logic to argparse.