Prologue, nostalgia and oversharing

I worked at BxE for 2 years and 9 months, first part-time as Werkstudent, then “Junior researcher”¹ then finally “Machine learning engineer”.

This was my first “real” non-internship full-time job, first contact with German work culture and then there was the usual “only in a startup you can get 5 years of experience in 1 year” - it was an adventure and I learned a lot.

I love the German phrase “Zwischen den Jahren”² (lit. ‘between the years’), the time between Christmas and the New Year, for me always a time of limbo³ and especially reflection. Even more so if the ’time between years’ coincides with the time between jobs, with my last day at BxE being the 17.12.2021 and the next one starting on 01.01.2022.

A very specific kind of emptiness, not “I’m on vacation” but “I don’t have a job, I have no commitments or duties neither to my last workplace, nor to the upcoming one; I don’t even need to look for a job, there’s nothing I ‘should’ be doing”. Comparable to “I’ve just finished my final school exam, regardless of the results I’m as free as I’ll ever be until university starts”.

I spent the first week of that time sleeping, then had a lot of quality time with family and friends, and missed the traditional reviewing/thinking that I usually do at the end of important phases. There was no analysis and synthesis of the ’lessons learned’, not even diary-style bullet points. I thought I’d just forget about it and live my life, but nope.

Still it kept eating me, keeps eating me, the way unfinished business and disconnected memories that desperately want to be analyzed eat you - and 21 days in, I don’t think it will shut up, so I’ll give in.

Follows a rough list, in the order I remember them. ⁴

Things I learned how to do better

Soft skills

• Presenting
  • Both “how to effectively create slides” and “how to tell a story”
  • Graphs are awesome
    • Bonus points if they are easily reusable⁵
  • How to pick an appropriate abstraction level when explaining stuff
  • If I attempt to convey less, more gets understood at the end.
    • Especially in low-shared-context larger meetings or sprint reviews. High information density means everyone gets lost on slide #3.
• Mentoring / teaching / supporting
  • Mentoring a smart junior colleague was awesome and really satisfying, not sure who of us learned more
  • Still proud of “This is a series of tickets to learn $internal_tool, I’ll help you; please write the tutorials as you go”
• Communicating results and probabilities
  • Often it’s better to not communicate intermediate results or low-probability assestments
  • Make it as hard as possible to misunderstand you, even if the listener really wants to

Automatic document processing in Real Life

• Scientific papers and scenarios in ML tutorials live in a world where you rarely have:
  • street names containing months & dates, people named like “Christian Thomas”
  • tables in invoices showing incredible diversity and creativity and interesting design decisions
  • 50 annotation types, half of which overlap with each other
    • An ADDRESS prolly contains a STREET_NAME, and now you can’t train both together :(
  • “I had this invoice in my pocket, then my dog chewed it, yes these are traces of blood don’t ask where it comes from”
• Matching names/address with bad OCR to rows in databases compiled from multiple sources and OCR engines
  • Whatever str., Whateverstr., Whateverstraße, Whatever Straße, WhaIeveraBe, W̵h̶a̶t̴e̴v̶e̴r̴ ̶S̵t̷r̶a̵ß̷e̶, W̴̱̩̖̎͑ḧ̵͔̜́̌ą̸͉͌t̴̠̪̊̾͠e̷̖̥͌̇͗v̷͓͖͂̒ĕ̶͚̍r̴̳͊͝ ̶̯̲͔̄S̴̻̍̈́̾t̷͇͋̾r̵̡͂̓̏a̸͍͕͑͗̔ß̴̣́ę̸̥̠̌͌̆
  • Our Lady in Whatever General Hospital, OLWGH, W.G.H., OL-W. General Hospital
  • … human creativity knows no bounds!
• Measuring how changes in things like OCR or text flow influence performance of already trained models
  • train a network on data with bad OCR, improve OCR, your network now performs worse. Cry.
• NER F-Score metrics are only one part of the picture!

Working with data

• Parsing/converting a lot of random files, especially JSONs
• Per-word and per-char diffs, colums, sorting, etc.
• Exploratory data analysis (esp. with pandas)
• Generating synthetic data (simulate OCR errors, signatures, ..)

Software Engineering

• Prototyping
  • Writing converters and simple scripts to quickly test existing solutions
  • Quickly writing PoCs to solve a specific problem
    • …and productizing them afterwards if they work out
• Tools
  • Run/debug stuff on remote servers (pdb, sshfs, screen, port forwarding for jupyter/tensorboard etc.)
  • Jira/Confluence
  • Deployment, Docker, packaging, REST APIs
• Collaborating
  • Using git for version control when other people also work on the same project
  • Writing better tests, doing code reviews
• Generally organizing/structuring/implementing bigger projects, allocating/estimating time/energy/resources (others’ and my own)

Machine learning magic

• Tensorflow internals, writing layers/metrics/…, pytorch
• NLP: training/finetuning/evaluating NER, LMs
  • RNN / CRF / LSTM, Transformers and BERT
• Computer vision:
  • opencv, numpy, CNNs
  • Detectron2 internals
  • Document/layout parsing, table recognition, signatures/forms/…
• DVC and (very) rudimentary MLOps
• Analyzing, presenting and explaining the results
  • With graphs!

Things I learned about the universe

IT

Coding

• Smaller commits / PRs / merges are awesome
  • For bigger things, feature branches from feature branches and PRs when merging into the main feature branch is a way to do that
• Pair programming/reviewing (or mob-reviewing) teach you coding much faster, and spread the knowledge about a specific piece of code through the team much more effectively

Information flow

• Always better to ask for help earlier than later
  • Especially early during design stage if it’s not trivial
  • Especially if the code you’re working on becomes unexpectedly large and complex
    • Bad architecture that would benefit from a second set of eyes?
    • Genuinely hard thing best dealt with by splitting the workload?
• Having a clearly-defined mentor when dealing with unknown code/processes/… is crucial
  • Someone showing you a map of the forest, highlighting interesting bits etc. is better than you asking random people on Slack questions about individual trees
  • Having a set time (daily/weekly) for this is very helpful
• Experiments should be reproducible by default
  • You can understand code but you can’t get hyperparameters from someone’s bash history⁶

Not-IT

Ship of Theseus⁷

TL;DR Everything passes, this too shall pass

• There are few irreplaceable people. After watching my Nth “my God he’s the last person who knows X without him we’re dead” person leave without the office burning down I think i see a pattern there.
  • work might get harder, slower, you might need to dig deeper into JIRA and git blame, but there won’t be a full stop
  • Generally, pretty much anything is survivable: people leaving, plagues coming, etc.
• It’s surprising how better or worse can things get, how fast, and how hard it is to predict that
  • Winter 2020-2021 was bleak and lonely
  • Then not even a couple of months passed, and my team’s daily meeting is the highlight of my day. Everyone was working on really interesting problems and I was genuinely looking forward hearing about the updates on them, about the problems and solutions. Both my and others’ problems were of the “can’t wait till tomorrow so I can keep working on that really interesting thing” sort, mostly research/ML and not deadline-driven.

Home Office

I used to think that the places that see a drop in productivity after going remote just aren’t doing it right.

BxE was remote-first from before it was cool, all the processes were set up; my team was born after Corona, we were working together well.

Then we all met in real life for the first time.

• Seeing someone in 3D makes a world of difference, more than I thought possible, and one could feel that something changed for the better in all remote meetings after that
• We spent 4 hours whiteboard talking about a project we were starting, clearing up a surprising number of misunderstanding, mostly with definitions. Without that meeting, we’d have found them much later and fixing them would have been more painful.

Updated belief:

Working remotely doesn’t make much difference for the final result, unless you’re in the initial stages of planning something complex and creative all together (or are looking for an emotional connection to your team members).

Learning

• Learning new stuff in your area even if not needed for a concrete task you’re currently solving is important
  • Explicitly dedicating hours to (for example) reading papers helps follow these long-term goals in the context of short-term burning deadlines
• Know what you want: clearly formulated wishes (“I’d like to spend 20% of my time learning Hebrew”) are more likely to get attention than general ones (“I don’t think my Hebrew knowledge is enough, but there’s no time to fix that”)
  • Ask around - if 25 other people want to learn Hebrew for work, $companyName might organise a course for everyone!

People

• Asking people directly for feedback is almost a super-power
• Saying good things to people in private (“I love how you did X”) and in public (“with @Bob’s help figured out this bug”) makes everyone happier.
• Team names, logos, identities, freedom to design your slides work surprisingly well! (More token-y stuff like “let’s come up with our team values” predictably fares much worse, it’s a fine balance)
• If not forced, social-y stuff like “once a week you get matched to a random person to remotely drink a coffee” has its place.
  • You get a better picture of who else exists in the company, what problems/needs/skills they have
  • You also see them as ‘real’ people with real hobbies/interests that are genuinely interesting to talk to
  • You knew things like “partner management” exist, now you know why and what does it actually look like to do it

Mental health

• “External pressure VS pressure you put on yourself”
• You not burning out and staying sane is in the interest of both you and your company, but it’s your responsibility.
  • If you clearly communicate the problem people might offer solutions you had no idea existed
• Maintaining an identity, interests and a social circle outside of work is crucial
  • even if (especially if!) it requires effort
  • Especially if your need for social interaction is met and exceeded by the N hours of meetings per day
  • Even if your interests are remarkably close to your day job

Things I learnt about myself

• Areas where I can improve the most:
  • energy management, pressure/stress management
  • asking for help more
• Things that I loved and that really increased my quality of life:
  • Full control over my work computer, admin rights etc.
  • Home office (and coming to the office from time to time)
  • Flexible working time / honor system
• A real computer chair is more important than anything else, if it’s good you get tired much less

Things I learned about Leipzig

… something something friends you make along the way. I met a lot of wonderful people, connections (human ones, not LinkedIn) I hope will stay with me.

Leipzig is a small city and the world of AI/ML is an even smaller one - thank you BxE, we might meet again.